How to use transformWith() to ignore failed futures? - scala

I need to process sequence of transformations into one transformation, in order to if I have some failed Future it should be just ignored(I am trying to do it without recover or recoverWith, but smth going wrong, code fails when meet any failure)
type Transformation[T] = T => Future[T]
//in - Seq(trans1, trans2, trans3)
in.reduce[Transformation[T]](
(acc, el) =>
acc.andThen[Future[T]](
ft =>
ft.flatMap(el)
.transformWith[T](
t =>
t match {
case Failure(exception) => ft //this line is the most suspicious for me
case Success(value) => Future.successful(value)
}
)
)
)

The transformWith doesn't seem to provide you the possibility to recover with the original input value from the previous step, because it has to work with a Try. What is it supposed to do if the Try turns out to be a failure? By that time, it does not have the original input to fall back to, it only has a Throwable, not a T. So, transformWith seems insufficient.
If the pipeline is just a few transformations long, you can try foldLeft with fallbackTo:
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
type Transformation[T] = T => Future[T]
def chainTrafos[T](
trafos: List[Transformation[T]],
value: T
): Future[T] = {
trafos.foldLeft(Future { value })(
(f, t) => f.flatMap(x => t(x).fallbackTo(Future { x }))
)
}
val ta: Transformation[Int] = x => Future { x * x }
val tb: Transformation[Int] = x => Future.failed(new Error("oops"))
val tc: Transformation[Int] = x => Future { x - 58 }
println(Await.result(chainTrafos(List(ta, tb, tc), 10), 10.seconds))
prints
42

Related

Unwrapping the Functions without parameters in scala with Futures

I'm trying to create a transformation chain to define possible transformations for given functions:
type Transformation[T] = T => Future[T]
def transformationChain[T](chain: Seq[Transformation[T]]): Transformation[T] = {
}
val t1: Transformation[Int] = t => Future.successful(t + t)
val t2: Transformation[Int] = _ => Future.failed(new NoSuchElementException)
val t3: Transformation[Int] = t =>
if (t > 2) Future.successful(t * t)
else Future.failed(new NoSuchElementException)
val tc = transformationChain(Seq(t1, t2, t2, t3))
val tc2 = transformationChain(Seq(t2, t2, t2))
val tc3 = transformationChain(Seq(t2, t3, t1))
println(Await.result(tc(2), 5.seconds)) // 16
println(Await.result(tc2(2), 5.seconds)) // throw NoSuchElementException
println(Await.result(tc3(2), 5.seconds)) // 4
The problem is that I don't understand how to unwrap these functions in "transformationChain" method to send the result to every next function in the chain by calling them in the loop or recursively.
What you're described as Transformation (so function of A => F[B]) is called often Kleisli arrows.
Cats library has a data type, that makes operation on these kinds of functions easier. For example, it has a method andThen which allows the composition of these functions:
import cats.data.Kleisli
import cats.implicits._
val t1: Transformation[Int] = t => Future.successful(t + t)
val t2: Transformation[Int] = _ => Future.failed(new NoSuchElementException)
Kleisli(t1).andThen(Kleisli(t2))
The only problem is, that one of your transformations could return the failed future, which would short-circuit the whole chain. We can fix it with recoverWith.
So finally transformationChain could look like:
def transformationChain[T](chain: Seq[Transformation[T]]): Transformation[T] =
t =>
chain
//wrap the function in Kleisli and then use replace failed futures with succeeded
//future, that are passing value over
.map(Kleisli(_).recoverWith {
case _ => Kleisli(x => Future.successful(x))
})
.reduce(_ andThen _) //combine all elements with andThen
.apply(t)
It works fine for case 1 and 3 but fails for case 2 since it will just return passed value.
println(Await.result(tc(2), 5.seconds)) // 16
println(Await.result(tc3(2), 5.seconds)) // 4
println(Await.result(tc2(2), 5.seconds)) // 2
import scala.concurrent.{ ExecutionContext, Future }
import scala.util.Try
import ExecutionContext.Implicits.global
object Transformations {
type Transformation[T] = T => Future[T]
private object DummyException extends Exception
private val notReallyAFailedFuture: Future[Throwable] = Future.failed(DummyException)
def transformationChain[T](chain: Seq[Transformation[T]])(implicit ectx: ExecutionContext): Transformation[T] = t =>
if (chain.nonEmpty) {
val initialFut = Future.successful(t)
// resultFut will succeed if any of the transformations in the chain succeeded
// lastFailure will fail if all of the transformations succeeded, otherwise it has the last failure
val (resultFut: Future[T], lastFailure: Future[Throwable]) =
chain.foldLeft((Future.failed[T](DummyException), notReallyAFailedFuture)) { (acc, v) =>
val thisResult = acc._1.recoverWith {
case _ => initialFut
}.flatMap(v)
val lastFailure = thisResult.failed.recoverWith { case _ => acc._2 }
(thisResult.recoverWith { case _ => acc._1 }, lastFailure)
}
resultFut.recoverWith {
case _ =>
lastFailure.flatMap(Future.failed)
}
} else Future.successful(t) // What to do with an empty chain is unspecified
def main(args: Array[String]): Unit = {
import scala.concurrent.Await
import scala.concurrent.duration._
val t1: Transformation[Int] = t => Future.successful(t + t)
val t2: Transformation[Int] = _ => Future.failed(new NoSuchElementException)
val t3: Transformation[Int] = t =>
if (t > 2) Future.successful(t * t)
else Future.failed(new NoSuchElementException)
val tc1 = transformationChain(Seq(t1, t2, t2, t3))
val tc2 = transformationChain(Seq(t2, t2, t2))
val tc3 = transformationChain(Seq(t2, t3, t1))
println(Try(Await.result(tc1(2), 5.seconds)))
println(Try(Await.result(tc2(2), 5.seconds)))
println(Try(Await.result(tc3(2), 5.seconds)))
}
}
This implementation assumes that:
If multiple transformations fail, return the last failure
If the chain is empty, assume an identity transformation
transformationChain does now require an implicit ExecutionContext for scheduling the "glue" functions between the transformation futures. In Scala 2.13+, the scala.concurrent.ExecutionContext.parasitic context is actually a pretty good choice for executing these quick transformations (and is useful for basically nothing else).
In order to get all the printlns to execute, I've wrapped the Await.results in Try.
In the interest of brevity, there's some use of a failed Future to represent absence of a result.

How to transform a Seq[Try] to a Try[Seq]

With Futures there is an easy way to transform Seq[Future] to a Future[Seq]:
Future.sequence(seqOfFutures)
I could not find an analog thing with Try.
It works with foldLeft but what I really like would have something like Try.sequence(seqOfTry).
Is there a reason that such a function is not provided?
How is this done properly?
Semantics:
A List of the values on Success: Success(Seq(1,2,3,4))
For Failure there are 2 possibilities:
Fails on the fist Failure and returns it. This is handled by this question: listtryt-to-trylistt-in-scala
Gathers all Failures and returns a 'compound' Failure.
Is there also a solution for the 'compound' Failure?
As per Luis' suggestion Validated is designed for error accumulation so consider traverse like so
la.traverse(_.toEither.toValidatedNec)
lb.traverse(_.toEither.toValidatedNec)
which outputs
res2: cats.data.ValidatedNec[Throwable,List[Int]] = Invalid(Chain(java.lang.RuntimeException: boom, java.lang.RuntimeException: crash))
res3: cats.data.ValidatedNec[Throwable,List[Int]] = Valid(List(1, 2, 3))
where
import cats.syntax.traverse._
import cats.instances.list._
import cats.syntax.either._
import scala.util.{Failure, Success, Try}
val la: List[Try[Int]] = List(Success(1), Success(2), Failure(new RuntimeException("boom")), Success(3), Failure(new RuntimeException("crash")))
val lb: List[Try[Int]] = List(Success(1), Success(2), Success(3))
Without error accumulation we could just sequence like so
import cats.implicits._
la.sequence
which outputs
res0: scala.util.Try[List[Int]] = Failure(java.lang.RuntimeException: boom)
This is a solution to the second question.
case class CompoundError(errs: List[Throwable]) extends Throwable
def toTry[T](list: List[Try[T]]): Try[List[T]] =
list.partition(_.isSuccess) match {
case (res, Nil) =>
Success(res.map(_.get))
case (_, errs) =>
Failure(CompoundError(errs.collect { case Failure(e) => e }))
}
The partition operation separates the successes and failures, and the match returns the appropriate value depending on whether there are any failures or not.
Previous solution:
case class CompoundError(errs: List[Throwable]) extends Throwable
def toTry[T](list: List[Try[T]]): Try[List[T]] = {
val (res, errs) = list.foldLeft((List.empty[T], List.empty[Throwable])) {
case ((res, errs), item) =>
item match {
case Success(t) => (t :: res, errs)
case Failure(e) => (res, e :: errs)
}
}
errs match {
case Nil => Success(res.reverse)
case _ => Failure(CompoundError(errs.reverse))
}
}

unable to compile a code with list of Future

There are some suggestions on StackOverflow on how to handle a list of Futures but I want to try my own approach. But I am unable to compile the following code
I have a list of Futures.
I want to count how many of them passed or failed. I should get (2,1)
I store this in a tuple
The approach I want to take is go through each element of the list. The element of the list is Future[Int]. for each element, I call flatMap which calls the next cycle of recursion (I am assuming that if flatMap gets called then that particular future would have been successful so I increment pass count). Similarly, I want to call next cycle of recursion in recover and increment fail count but I am getting compilation error in it.
import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Failure, Success, Try}
import scala.concurrent.duration._
import scala.language.postfixOps
object ConcurrencyExample extends App {
type pass = Int
type fail = Int
val time = System.currentTimeMillis()
//use recursion to process each Future in the list
def segregate(l:List[Future[Int]]):Future[Tuple2[pass,fail]] = {
def go(l:List[Future[Int]],t:Tuple2[pass,fail]):Future[Tuple2[pass,fail]] = {
l match {
case Nil => Future{t}
//l is List of Future[Int]. flatMap each successful Future
//recover each failed Future
case l::ls => {
l flatMap (x => go(ls, (t._1 + 1, t._2)))
**l.recover({ case e => go(ls, (t._1 + 1, t._2))})**//I get error here
}
}
}
go(l,(0,0))
}
//hardcoded future
val futures2: List[Future[Int]] = List(Future {
1
}, Future {
2
}, Future {
throw new Exception("error")
})
val result = segregate(futures2)
result onComplete {
case Success(v) => println("pp:" + v)
case Failure(v) => println("fp:" + v)
}
Await.result(result,1000 millis)
}
#evan058 is correct about the signature of recover. But you can fix your program by changing recover to recoverWith.
recoverWith is to recover as flatMap is to map.
Here's the complete solution (with minor stylistic improvements):
import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Failure, Success, Try}
import scala.concurrent.duration._
import scala.language.postfixOps
object ConcurrencyExample extends App {
type pass = Int
type fail = Int
val time = System.currentTimeMillis()
//use recursion to process each Future in the list
def segregate[T](fs:List[Future[T]]):Future[(pass,fail)] = {
def go(fs:List[Future[T]],r:Future[(pass,fail)]):Future[(pass,fail)] = fs match {
case Nil => r
case l::ls =>
val fx = l.transform({_ => (1, 0)}, identity).recoverWith[(pass,fail)]({case _: Exception => Future(0, 1) })
for (x <- fx; t <- r; g <- go(ls, Future(t._1+x._1,t._2+x._2))) yield g
}
go(fs,Future((0,0)))
}
//hardcoded future
val futures2 = List(Future(1), Future(2), Future(throw new Exception("error")))
val result = segregate(futures2)
result onComplete {
case Success(v) => println(s"successes: ${v._1}, failures: ${v._2}")
case Failure(v) => v.printStackTrace()
}
Await.result(result,1000 millis)
}
If you look at the docs, the signature of recover is:
def recover[U >: T](pf: PartialFunction[Throwable, U])(implicit executor: ExecutionContext): Future[U]
You are calling recover on l which is a Future[Int] so recover is expecting a U >: Int.
However you are calling go again which has the return type Future[(pass, fail)] which is not >: Int.

scala How to wait for several Futures

I have seq
i want to do the iterate automatic
You could use iterator with flatMap. Here is a simplified example:
import scala.concurrent.Await
import scala.concurrent.Future
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
val functions: Seq[() => Future[Int]] = Seq(
() => Future(1),
() => Future(2),
() => Future(3),
() => Future(4),
() => Future(5)
)
def checkResult(result: Int): Boolean = result > 3
functions.iterator.flatMap { func =>
val result = Await.result(func(), 5.seconds)
if (checkResult(result)) Some(result) else None
}.next()
This code would return 4.
Update:
If you just care about the correct result and don't care whether or not all futures get executed, you could chain them with fallBackTo:
import scala.util._
functions.tail.foldLeft[Future[Int]](functions.head().filter(checkResult)) {
case (result, function) =>
result.fallbackTo(function().filter(checkResult))
}.onComplete {
case Success(result) => s"Got one: $result"
case Failure(exception) => s"Damn! Got NO result due to: $exception"
}
This would give you a Future that would complete with Success(4) with my example predicate x > 3.
UPDATE 2
As I found myself with some extra time, I took the liberty to combine some comments and suggestions in other answers (especially from Alexander Azarov) with a loan pattern approach:
import scala.annotation.tailrec
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
val functions: Seq[() => Future[Int]] = Seq(
() => Future(1),
() => Future(2),
() => Future(3),
() => Future(4),
() => Future(5)
)
def checkResult(result: Int): Boolean = result > 5
def withFirstValidResult[A, B](futureFuncs: Seq[() => Future[A]], predicate: A => Boolean, defaultValue: A)(op: (A) => B): Future[B] = {
#tailrec
#inline def find(remaining: Iterator[Future[A]]): Future[A] = {
if (remaining.hasNext) {
remaining.next().filter(predicate).recoverWith {
case _ =>
find(remaining)
}
} else {
Future.successful {
println(s"No valid result found, falling back to default: $defaultValue")
defaultValue
}
}
}
find(futureFuncs.iterator.map(_())).map(op)
}
You can now use this function with any kind of futures you want and provide an operation that should be executed with the first valid result or the defaultValue (though IMHO in case of no valid result I'd prefer a Failure with proper error handling myself):
withFirstValidResult(functions.iterator, checkResult, 0) { result =>
println(s"Got a valid result: $result")
}.onFailure {
case ex =>
println(s"Got NO valid result: $ex")
}
As an extra tip, if you want to speed up the function you can have the iterator lazily evaluate two functions at a time with futureFuncs.iterator.buffered. This way, while one future result is being evaluated, the iterator will automatically start the next future.
def firstF[A](seq: Seq[() => Future[A]], predicate: A => Boolean): Future[A] =
seq.head().filter(predicate).recoverWith {
case _ => firstF[A](seq.tail, predicate)
}
Method firstF will return the first Future that matches the condition specified. The resulting Future will be a Failure if none of the input Futures match. To return some default value otherwise, you would simply do
firstF(seq, predicate).recover { case _ => default }
Note 1 the code here demonstrates basic principles and does not distinguish between planned or unplanned exceptions.
Note 2 this code is not tail-recursive.
import scala.concurrent.{Await, Future}
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
def f(x:String) = Future { x }
val functions = Seq(
() => f("a"),
() => f("b"),
() => f(""),
() => f("d"),
() => f("e"))
functions.takeWhile( x => Await.result(x(), 5 seconds) != "")
This will give you
res0: Seq[() => scala.concurrent.Future[String]] = List(<function0>, <function0>)
Iam not sure if there is another approach but for my understanding if you want to decide based on the result of the preceeding future you need to wait.
EDIT:
Here is a recursive approach. But in my opinion the one with the iterator is good.
import scala.concurrent.{Await, Future}
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
def f(x:String) = Future { x }
val functions = Seq(
() => f("a"),
() => f("b"),
() => f(""),
() => f("d"),
() => f("e"))
check(functions, 0)
def check(list:Seq[() => Future[String]], pos:Int):String = {
if(list.size <= pos) return ""
val res = Await.result(list(pos)(), 5 seconds)
if(condition(res)) {
res
}
else check(list, pos+1)
}
def condition(c:String):Boolean = if(c == "d") true else false
It's tail recursive therefore your stack won't overflow that easy.

How to know which Future fail when doing Future.sequence?

Here is an example receive part of an actor I'm working on:
def receive = {
case "begin" =>
val listOfFutures: IndexedSeq[Future[Any]] = workers.map(worker => worker ? Work("test"))
val future: Future[IndexedSeq[Any]] = Future.sequence(listOfFutures)
future onComplete {
case Success(result) => println("Eventual result: "+result)
case Failure(ex) => println("Failure: "+ex.getMessage)
}
case msg => println("A message received: "+msg)
}
When ask fails for one of the workers (in case of a timeout), sequence future completes with failure. However I want to know which worker(s) have failed. Is there a more elegant way rather than simply mapping listOfFutures one by one without using Future.sequence ?
You can use the future's recover method to map or wrap the underlying exception:
import scala.concurrent.{Future, ExecutionContext}
case class WorkerFailed(name: String, cause: Throwable)
extends Exception(s"$name - ${cause.getMessage}", cause)
def mark[A](name: String, f: Future[A]): Future[A] = f.recover {
case ex => throw WorkerFailed(name, ex)
}
import ExecutionContext.Implicits.global
val f = (0 to 10).map(i => mark(s"i = $i", Future { i / i }))
val g = Future.sequence(f)
g.value // WorkerFailed: i = 0 - / by zero
Thanks to #O__ I have come with another solution that may a better fit some some cases.
case class WorkerDone(name: String)
case class WorkerFailed(name: String)
import ExecutionContext.Implicits.global
val f = (0 to 10).map {
i => Future {i/i; WorkerDone(s"worker$i")}.recover{
case ex => WorkerFailed(s"worker$i")
}
}
val futureSeq = Future.sequence(f)
futureSeq onComplete {
case Success(list) => list.collect {case result:WorkerFailed => result}.foreach {failed => println("Failed: "+failed.name)}
case Failure(ex) => println("Exception: "+ex.getMessage)
}
// just to make sure program doesn't end before onComplete is called.
Thread.sleep(2000L)
I'm not sure that if my example is a good practice, but my aim is to know which workers did fail no matter how did they fail.