Here is an example receive part of an actor I'm working on:
def receive = {
case "begin" =>
val listOfFutures: IndexedSeq[Future[Any]] = workers.map(worker => worker ? Work("test"))
val future: Future[IndexedSeq[Any]] = Future.sequence(listOfFutures)
future onComplete {
case Success(result) => println("Eventual result: "+result)
case Failure(ex) => println("Failure: "+ex.getMessage)
}
case msg => println("A message received: "+msg)
}
When ask fails for one of the workers (in case of a timeout), sequence future completes with failure. However I want to know which worker(s) have failed. Is there a more elegant way rather than simply mapping listOfFutures one by one without using Future.sequence ?
You can use the future's recover method to map or wrap the underlying exception:
import scala.concurrent.{Future, ExecutionContext}
case class WorkerFailed(name: String, cause: Throwable)
extends Exception(s"$name - ${cause.getMessage}", cause)
def mark[A](name: String, f: Future[A]): Future[A] = f.recover {
case ex => throw WorkerFailed(name, ex)
}
import ExecutionContext.Implicits.global
val f = (0 to 10).map(i => mark(s"i = $i", Future { i / i }))
val g = Future.sequence(f)
g.value // WorkerFailed: i = 0 - / by zero
Thanks to #O__ I have come with another solution that may a better fit some some cases.
case class WorkerDone(name: String)
case class WorkerFailed(name: String)
import ExecutionContext.Implicits.global
val f = (0 to 10).map {
i => Future {i/i; WorkerDone(s"worker$i")}.recover{
case ex => WorkerFailed(s"worker$i")
}
}
val futureSeq = Future.sequence(f)
futureSeq onComplete {
case Success(list) => list.collect {case result:WorkerFailed => result}.foreach {failed => println("Failed: "+failed.name)}
case Failure(ex) => println("Exception: "+ex.getMessage)
}
// just to make sure program doesn't end before onComplete is called.
Thread.sleep(2000L)
I'm not sure that if my example is a good practice, but my aim is to know which workers did fail no matter how did they fail.
Related
I need to process sequence of transformations into one transformation, in order to if I have some failed Future it should be just ignored(I am trying to do it without recover or recoverWith, but smth going wrong, code fails when meet any failure)
type Transformation[T] = T => Future[T]
//in - Seq(trans1, trans2, trans3)
in.reduce[Transformation[T]](
(acc, el) =>
acc.andThen[Future[T]](
ft =>
ft.flatMap(el)
.transformWith[T](
t =>
t match {
case Failure(exception) => ft //this line is the most suspicious for me
case Success(value) => Future.successful(value)
}
)
)
)
The transformWith doesn't seem to provide you the possibility to recover with the original input value from the previous step, because it has to work with a Try. What is it supposed to do if the Try turns out to be a failure? By that time, it does not have the original input to fall back to, it only has a Throwable, not a T. So, transformWith seems insufficient.
If the pipeline is just a few transformations long, you can try foldLeft with fallbackTo:
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
type Transformation[T] = T => Future[T]
def chainTrafos[T](
trafos: List[Transformation[T]],
value: T
): Future[T] = {
trafos.foldLeft(Future { value })(
(f, t) => f.flatMap(x => t(x).fallbackTo(Future { x }))
)
}
val ta: Transformation[Int] = x => Future { x * x }
val tb: Transformation[Int] = x => Future.failed(new Error("oops"))
val tc: Transformation[Int] = x => Future { x - 58 }
println(Await.result(chainTrafos(List(ta, tb, tc), 10), 10.seconds))
prints
42
With Futures there is an easy way to transform Seq[Future] to a Future[Seq]:
Future.sequence(seqOfFutures)
I could not find an analog thing with Try.
It works with foldLeft but what I really like would have something like Try.sequence(seqOfTry).
Is there a reason that such a function is not provided?
How is this done properly?
Semantics:
A List of the values on Success: Success(Seq(1,2,3,4))
For Failure there are 2 possibilities:
Fails on the fist Failure and returns it. This is handled by this question: listtryt-to-trylistt-in-scala
Gathers all Failures and returns a 'compound' Failure.
Is there also a solution for the 'compound' Failure?
As per Luis' suggestion Validated is designed for error accumulation so consider traverse like so
la.traverse(_.toEither.toValidatedNec)
lb.traverse(_.toEither.toValidatedNec)
which outputs
res2: cats.data.ValidatedNec[Throwable,List[Int]] = Invalid(Chain(java.lang.RuntimeException: boom, java.lang.RuntimeException: crash))
res3: cats.data.ValidatedNec[Throwable,List[Int]] = Valid(List(1, 2, 3))
where
import cats.syntax.traverse._
import cats.instances.list._
import cats.syntax.either._
import scala.util.{Failure, Success, Try}
val la: List[Try[Int]] = List(Success(1), Success(2), Failure(new RuntimeException("boom")), Success(3), Failure(new RuntimeException("crash")))
val lb: List[Try[Int]] = List(Success(1), Success(2), Success(3))
Without error accumulation we could just sequence like so
import cats.implicits._
la.sequence
which outputs
res0: scala.util.Try[List[Int]] = Failure(java.lang.RuntimeException: boom)
This is a solution to the second question.
case class CompoundError(errs: List[Throwable]) extends Throwable
def toTry[T](list: List[Try[T]]): Try[List[T]] =
list.partition(_.isSuccess) match {
case (res, Nil) =>
Success(res.map(_.get))
case (_, errs) =>
Failure(CompoundError(errs.collect { case Failure(e) => e }))
}
The partition operation separates the successes and failures, and the match returns the appropriate value depending on whether there are any failures or not.
Previous solution:
case class CompoundError(errs: List[Throwable]) extends Throwable
def toTry[T](list: List[Try[T]]): Try[List[T]] = {
val (res, errs) = list.foldLeft((List.empty[T], List.empty[Throwable])) {
case ((res, errs), item) =>
item match {
case Success(t) => (t :: res, errs)
case Failure(e) => (res, e :: errs)
}
}
errs match {
case Nil => Success(res.reverse)
case _ => Failure(CompoundError(errs.reverse))
}
}
Why this does not compile? I get the following error in the Success and Failure lines:
constructor cannot be instantiated to expected type; found :
scala.util.Success[T] required: Int
And the code:
val future = Future { 1 }
future.map {
case Success(s) => s
case Failure(f) => 0
}
Because map of a Future[Int] expects a function with domain Int, but instead you are trying to pass some function with domain Try[Int] into it.
The syntactically closest thing that works is onComplete:
import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Try, Success, Failure}
val future = Future { 1 }
future.onComplete {
case Success(s) => s
case Failure(f) => 0
}
However, this is probably not what you want, because the expressions s and 0 don't do anything, they are therefore not very interesting callbacks. Instead, you probably either want to transform the result:
future.transform {
case Success(s) => Success(s)
case Failure(f) => Success(0)
}
or recover from the failure with a 0:
future.recover {
case f: Exception => 0
}
Are you trying to do something like:
val future = Future {
1
}
future onComplete {
case Success(s) => s
case Failure(f) => 0
}
Suppose I have the following code:
val futureInt1 = getIntAsync1();
val futureInt2 = getIntAsync2();
val futureSum = for {
int1 <- futureInt1
int2 <- futureInt2
} yield (int1 + int2)
val sum = Await.result(futureSum, 60 seconds)
Now suppose one of getIntAsync1 or getIntAsync2 takes a very long time, and it leads to Await.result throwing exception:
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [60 seconds]
How am I supposed to know which one of getIntAsync1 or getIntAsync2 was still pending and actually lead to the timeout?
Note that here I'm merging 2 futures with zip, and this is a simple example for the question, but in my app I have this kind of code at different level (ie getIntAsync1 itself can use Future.zip or Future.sequence, map/flatMap/applicative)
Somehow what I'd like is to be able to log the pending concurrent operation stacktraces when a timeout occur on my main thread, so that I can know where are the bottlenexts on my system.
I have an existing legacy API backend which is not fully reactive yet and won't be so soon. I'm trying to increase response times by using concurrency. But since using this kind of code, It's become way more painful to understand why something takes a lot of time in my app. I would appreciate any tip you can provide to help me debugging such issues.
The key is realizing is that a the Future doesn't time out in your example—it's your calling thread which pauses for at most X time.
So, if you want to model time in your Futures you should use zipWith on each branch and zip with a Future which will contain a value within a certain amount of time. If you use Akka then you can use akka.pattern.after for this, together with Future.firstCompletedOf.
Now, even if you do, how do you figure out why any of your futures weren't completed in time, perhaps they depended on other futures which didn't complete.
The question boils down to: Are you trying to do some root-cause analysis on throughput? Then you should monitor your ExecutionContext, not your Futures. Futures are only values.
The proposed solution wraps each future from for block into TimelyFuture which requires timeout and name. Internally it uses Await to detect individual timeouts.
Please bear in mind this style of using futures is not intended for production code as it uses blocking. It is for diagnostics only to find out which futures take time to complete.
package xxx
import java.util.concurrent.TimeoutException
import scala.concurrent.{Future, _}
import scala.concurrent.duration.Duration
import scala.util._
import scala.concurrent.duration._
class TimelyFuture[T](f: Future[T], name: String, duration: Duration) extends Future[T] {
override def onComplete[U](ff: (Try[T]) => U)(implicit executor: ExecutionContext): Unit = f.onComplete(x => ff(x))
override def isCompleted: Boolean = f.isCompleted
override def value: Option[Try[T]] = f.value
#scala.throws[InterruptedException](classOf[InterruptedException])
#scala.throws[TimeoutException](classOf[TimeoutException])
override def ready(atMost: Duration)(implicit permit: CanAwait): TimelyFuture.this.type = {
Try(f.ready(atMost)(permit)) match {
case Success(v) => this
case Failure(e) => this
}
}
#scala.throws[Exception](classOf[Exception])
override def result(atMost: Duration)(implicit permit: CanAwait): T = {
f.result(atMost)
}
override def transform[S](ff: (Try[T]) => Try[S])(implicit executor: ExecutionContext): Future[S] = {
val p = Promise[S]()
Try(Await.result(f, duration)) match {
case s#Success(_) => ff(s) match {
case Success(v) => p.success(v)
case Failure(e) => p.failure(e)
}
case Failure(e) => e match {
case e: TimeoutException => p.failure(new RuntimeException(s"future ${name} has timed out after ${duration}"))
case _ => p.failure(e)
}
}
p.future
}
override def transformWith[S](ff: (Try[T]) => Future[S])(implicit executor: ExecutionContext): Future[S] = {
val p = Promise[S]()
Try(Await.result(f, duration)) match {
case s#Success(_) => ff(s).onComplete({
case Success(v) => p.success(v)
case Failure(e) => p.failure(e)
})
case Failure(e) => e match {
case e: TimeoutException => p.failure(new RuntimeException(s"future ${name} has timed out after ${duration}"))
case _ => p.failure(e)
}
}
p.future
}
}
object Main {
import scala.concurrent.ExecutionContext.Implicits.global
def main(args: Array[String]): Unit = {
val f = Future {
Thread.sleep(5);
1
}
val g = Future {
Thread.sleep(2000);
2
}
val result: Future[(Int, Int)] = for {
v1 <- new TimelyFuture(f, "f", 10 milliseconds)
v2 <- new TimelyFuture(g, "g", 10 milliseconds)
} yield (v1, v2)
val sum = Await.result(result, 1 seconds) // as expected, this throws exception : "RuntimeException: future g has timed out after 10 milliseconds"
}
}
If you are merely looking for informational metrics on which individual future was taking a long time (or in combination with others), your best bet is to use a wrapper when creating the futures to log the metrics:
object InstrumentedFuture {
def now() = System.currentTimeMillis()
def apply[T](name: String)(code: => T): Future[T] = {
val start = now()
val f = Future {
code
}
f.onComplete {
case _ => println(s"Future ${name} took ${now() - start} ms")
}
f
}
}
val future1 = InstrumentedFuture("Calculator") { /*...code...*/ }
val future2 = InstrumentedFuture("Differentiator") { /*...code...*/ }
You can check if a future has completed by calling its isComplete method
if (futureInt1.isComplete) { /*futureInt2 must be the culprit */ }
if (futureInt2.isComplete) { /*futureInt1 must be the culprit */ }
As a first approach i would suggest to lift your Future[Int] into Future[Try[Int]]. Something like that:
object impl {
def checkException[T](in: Future[T]): Future[Try[T]] =
in.map(Success(_)).recover {
case e: Throwable => {
Failure(new Exception("Error in future: " + in))
}
}
implicit class FutureCheck(s: Future[Int]) {
def check = checkException(s)
}
}
Then a small function to combine the results, something like that:
object test {
import impl._
val futureInt1 = Future{ 1 }
val futureInt2 = Future{ 2 }
def combine(a: Try[Int], b: Try[Int])(f: (Int, Int) => (Int)) : Try[Int] = {
if(a.isSuccess && b.isSuccess) {
Success(f(a.get, b.get))
}
else
Failure(new Exception("Error adding results"))
}
val futureSum = for {
int1 <- futureInt1.check
int2 <- futureInt2.check
} yield combine(int1, int2)(_ + _)
}
In futureSum you will have a Try[Int] with the integer or a Failure with the exception corresponding with the possible error.
Maybe this can be useful
val futureInt1 = getIntAsync1();
val futureInt2 = getIntAsync2();
val futureSum = for {
int1 <- futureInt1
int2 <- futureInt2
} yield (int1 + int2)
Try(Await.result(futureSum, 60 seconds)) match {
case Success(sum) => println(sum)
case Failure(e) => println("we got timeout. the unfinished futures are: " + List(futureInt1, futureInt2).filter(!_.isCompleted)
}
I have a Future[T] and I want to map the result, on both success and failure.
Eg, something like
val future = ... // Future[T]
val mapped = future.mapAll {
case Success(a) => "OK"
case Failure(e) => "KO"
}
If I use map or flatmap, it will only map successes futures. If I use recover, it will only map failed futures. onComplete executes a callback but does not return a modified future. Transform will work, but takes 2 functions rather than a partial function, so is a bit uglier.
I know I could make a new Promise, and complete that with onComplete or onSuccess/onFailure, but I was hoping there was something I was missing that would allow me to do the above with a single PF.
Edit 2017-09-18: As of Scala 2.12, there is a transform method that takes a Try[T] => Try[S]. So you can write
val future = ... // Future[T]
val mapped = future.transform {
case Success(_) => Success("OK")
case Failure(_) => Success("KO")
}
For 2.11.x, the below still applies:
AFAIK, you can't do this directly with a single PF. And transform transforms Throwable => Throwable, so that won't help you either. The closest you can get out of the box:
val mapped: Future[String] = future.map(_ => "OK").recover{case _ => "KO"}
That said, implementing your mapAll is trivial:
implicit class RichFuture[T](f: Future[T]) {
def mapAll[U](pf: PartialFunction[Try[T], U]): Future[U] = {
val p = Promise[U]()
f.onComplete(r => p.complete(Try(pf(r))))
p.future
}
}
Since Scala 2.12 you can use transform to map both cases:
future.transform {
case Success(_) => Try("OK")
case Failure(_) => Try("KO")
}
You also have transformWith if you prefer to use a Future instead of a Try. Check the documentation for details.
In a first step, you could do something like:
import scala.util.{Try,Success,Failure}
val g = future.map( Success(_):Try[T] ).recover{
case t => Failure(t)
}.map {
case Success(s) => ...
case Failure(t) => ...
}
where T is the type of the future result. Then you can use an implicit conversion to add this structure the Future trait as a new method:
implicit class MyRichFuture[T]( fut: Future[T] ) {
def mapAll[U]( f: PartialFunction[Try[T],U] )( implicit ec: ExecutionContext ): Future[U] =
fut.map( Success(_):Try[T] ).recover{
case t => Failure(t)
}.map( f )
}
which implements the syntax your are looking for:
val future = Future{ 2 / 0 }
future.mapAll {
case Success(i) => i + 0.5
case Failure(_) => 0.0
}
Both map and flatMap variants:
implicit class FutureExtensions[T](f: Future[T]) {
def mapAll[Target](m: Try[T] => Target)(implicit ec: ExecutionContext): Future[Target] = {
val promise = Promise[Target]()
f.onComplete { r => promise success m(r) }(ec)
promise.future
}
def flatMapAll[Target](m: Try[T] => Future[Target])(implicit ec: ExecutionContext): Future[Target] = {
val promise = Promise[Target]()
f.onComplete { r => m(r).onComplete { z => promise complete z }(ec) }(ec)
promise.future
}
}