Is it possible to asynchronously catch all errors given a list of futures?
I was thinking of something like:
Future { throw Exception } zip Future { throw Exception } recover ...
But I only receive a Throwable in the recover part. Any idea?
zip's docs clearly say it prefers errors from this over those from that (my emphasis):
def zip[U](that: Future[U]): Future[(T, U)] Zips the values of this
and that future, and creates a new future holding the tuple of their
results.
If this future fails, the resulting future is failed with the
throwable stored in this. Otherwise, if that future fails, the
resulting future is failed with the throwable stored in that.
You could implement your own function to combine two futures into Future[(Try[T], Try[U])], like:
def zipToTry[T,U](a: Future[T], b: Future[U])
(implicit executor: ExecutionContext)
: Future[(Try[T], Try[U])] = {
val t: Try[T] = a.map { Success(_) }.recover { Failure(_) }
val u: Try[U] = b.map { Success(_) }.recover { Failure(_) }
t.zip(u)
}
(i haven't tested this, but you get the idea)
Related
Considering a sequence of futures each returning Either[Status, Resp].
How would you propagate error status codes through a for comprehension which is using Future and not Either?
The code bellow does not work, since the parsing exception is not caught by .recover of the last future
The use case is Scala Play ActionRefiners which returns Future[Either[Status, TRequest[A]]].
def parseId(id: String):Future[Int] = {
Future.successful(Integer.parseInt(id))
}
def getItem(id: Int)(implicit ec: ExecutionContext): Future[Either[Status, String]] =
Future(Some("dummy res from db " + id)).transformWith {
case Success(opt) => opt match {
case Some(item) => Future.successful(Right(item))
case _ => Future.successful(Left(NotFound))
}
case Failure(_) => Future.successful(Left(InternalServerError))
}
(for {
id <- parseId("bad request")
resp <- getItem(id)
} yield resp).recover {
case _:NumberFormatException => Left(BadRequest)
}
I could move the .recover to parseId, but this makes the for comprehension very ugly - having to treat the Either[Status, id] in the middle
def parseId(id: String):Future[Either[Status, Int]] = {
Future.successful(Right(Integer.parseInt(id))).recover {
case _:NumberFormatException => Left(BadRequest)
}
}
Your exception is not caught because you are not throwing it inside the Future: Future.successful is immediately satisfied with the result of the expression you give it, if it throws an exception, it is executed on the current thread.
Try removing the .successful: Future(id.toInt) will do what you want.
Also, I would recommend to get rid of all the Eithers: these are highly overrated/overused, especially in the context of Future (that already wrap their result into Try anyhow), and just make the code more complicated and less readable without offering much benefit.
case class FailureReason(status: Status)
extends Exception(status.toString)
def notFound() = throw FailureReason(NotFound)
def internalError() = throw FailureReason(InternalError)
def badRequest() = throw FailureReason(BadRequest)
def parseId(id: String):Future[Int] = Future(id.toInt)
def getItem(id: Int): Future[String] = Future(Some("dummy"))
.map { _.getOrElse(notFound) }
.recover { _ => internalError }
// this is the same as your for-comprehension, just looking less ugly imo :)
parseId("foo").flatMap(getItem).recover {
case _: NumberFormatException => badRequest()
}
// if you still want `Either` in the end for some reason:
.map(Right.apply[Status, String])
.recover {
case _: NumberFormatException => Left(BadRequest) // no need for the first recover above if you do this
case FailureReason(status) => Left(status)
}
I have a method that may return Future - successful or failed or can even throw an exception. I can avoid this by putting try catch block on entire method and return Future all the time but i would like to avoid it for now. I have few problems with calling such method:
1) In caller code, if I use map I expect execution of a method and expect a Future or an exception which I tried to handle in following manner:
object ETLCoordinator {
private def getBusinessListFromModules(modulePaths: Iterable[File]) : Future[String] = {
implicit val ec = ExecutionContext.global
println("Inside getBusinessListFromModules..")
throw new java.lang.RuntimeException("failed to get businesses") //Exception is thrown before future was constructed
Future("ok")
}
def main(args: Array[String]) {
println("Inside Future Test..")
implicit val ec = ExecutionContext.global
val modulePaths = Iterable(new File("mdrqaint/MDR/QAINT/QAINTX"))
val fut1 = getBusinessListFromModules(modulePaths) //This is outside of try and which should be okay
try {
fut1.map { res =>
println("things after Successful fut1")
}.recover{
case t: Throwable => println("Failed future in fut1: "+ t.getMessage)
}
} catch {
case t: Throwable => println("Exception in fut1: "+ t.getMessage)
}
}
}
Output: ( No execution of recover or catch block above)
Inside Future Test..
Inside getBusinessListFromModules..
Exception in thread "main" java.lang.RuntimeException: failed to get businesses
But if I put val fut1 = getBusinessListFromModules(modulePaths) inside Try block then Exception is get caught in Catch block and I get output:
Inside Future Test..
Inside getBusinessListFromModules..
Exception in fut1: failed to get businesses
Why is this? I though Future execution happens upon calling some of its methods like map, flatmap, onSuccess, onComplete etc. In this case call to map is already inside Try block.
2) What is the better way to define and call such methods? Try/catch block in a caller or try/catch in method itself? or any other way. I tried wrapping the calling method in Future so I get Future[Future[String]] in caller. I was able to avoid all try-catch.
val fut1 = Future(getBusinessListFromModules(modulePaths))
//try {
fut1.map { res =>
res.map{ str =>
println("things after Successful fut1")
}.recover{
case t: Throwable => println("Failed in future of fut1: "+ t.getMessage)
}
println("things after Successful fut1 wrapper")
}.recover{
case t: Throwable => println("Failed to create future in fut1: "+ t.getMessage)
}
3) If there is another method inbetween which does delegation to getBusinessListFromModules but it itself is non-future method.
object ETLController {
private def getBusinessListFromModules(modulePaths: Iterable[File]) : Future[String] = {
implicit val ec = ExecutionContext.global
println("Inside getBusinessListFromModules..")
//throw new java.lang.RuntimeException("failed to get businesses")
Future("ok")
}
private def callGetBusList(modulePaths: Iterable[File]) : String = {
implicit val ec = ExecutionContext.global
val etlF = getBusinessListFromModules(modulePaths)
etlF onComplete {
case Success(itr) => {
println("Future getBusinessListFromModules success: "+ itr)
throw new java.lang.RuntimeException("RTE from callGetBusList")
}
case Failure(t) => {
println("Future getBusinessListFromModules throws an error")
}
}
"callGetBusList was a success"
}
def main(args: Array[String]) {
println("Inside Future Test..")
implicit val ec = ExecutionContext.global
val modulePaths = Iterable(new File("mdrqaint/MDR/QAINT/QAINTX"))
try {
val fut = Future(callGetBusList(modulePaths))
fut.map { res =>
println("successful future!")
}.recover{
case t: Throwable => println("Failed future: "+ t.getMessage)
}
} catch {
case t: Throwable => println("callGetBusList failed:" + t.getMessage)
}
}
}
Output: (no recover or catch block execution!)
Inside Future Test..
Inside getBusinessListFromModules..
Future getBusinessListFromModules success: ok
java.lang.RuntimeException: RTE from callGetBusList
at ..
successful future!
I even try double wrapping Future calls :
val fut = Future(Future(callGetBusList(modulePaths)))
fut.map { res =>
res.map { str =>
println("successful inner future! "+ str)
}.recover{
case t: Throwable => println("Failed inner future: "+ t.getMessage)
}
println("successful outer future!")
}.recover{
case t: Throwable => println("Failed outer future: "+ t.getMessage)
}
Output:
Future getBusinessListFromModules success: ok
java.lang.RuntimeException: RTE from callGetBusList
at
successful inner future! callGetBusList was a success
successful outer future!
I get "callGetBusList was a success" which seems like RuntimeException inside onComplete method got lost! How do I catch it in a final caller? What are the better practice to handle such future dependencies?
UPDATE:
based on #dk14 explanation, Opted to convert middle method to return Future and basically all methods to return some kind of Future and not a plain Exception.
object ETLController {
private def getBusinessListFromModules(modulePaths: Iterable[File]) : Future[String] = {
implicit val ec = ExecutionContext.global
println("Inside getBusinessListFromModules..")
Future {
Thread.sleep(2000)
throw new java.lang.RuntimeException("failed to get businesses")
"ok"
}
}
private def callGetBusList(modulePaths: Iterable[File]) : Future[String] = {
implicit val ec = ExecutionContext.global
val etlF = getBusinessListFromModules(modulePaths)
etlF map { itr =>
println("Future getBusinessListFromModules success: "+ itr)
throw new java.lang.RuntimeException("RTE from callGetBusList")
} recover {
case t: Throwable => {
println("Future callGetBusList throws an error: " + t.getMessage)
throw t
}
}
}
def main(args: Array[String]) {
println("Inside Future Test..")
implicit val ec = ExecutionContext.global
val modulePaths = Iterable(new File("mdrqaint/MDR/QAINT/QAINTX"))
val fut = callGetBusList(modulePaths)
fut.map { str =>
println("successful future! "+ str)
}.recover{
case t: Throwable => println("Failed future: "+ t.getMessage)
}
println("Active threads: " +Thread.activeCount())
sys.allThreads().foreach(t => t.join())
}
}
1) Futures are firing eagerly and they aren't referentially transparent.
Answers to the referenced question also contain some insights about Future's internal behavior, so I'd like to skip it here.
In order to manage side-effects concerning execution pools/queues/threads in a more predictable way, you could consider scalaz/monix/fs2 Task or iteratee/scalaz/cats Eval (more abstract lazy evaluation, and intended for sync stuff) + Cont (continuations are abstracting over subscriptions) as alternative. All are referentially transparent and start execution lazily "on-demand".
2) The best way is the one you don't like: to not throw exception outisde of Future context.
You might also consider flatMap to avoid Future[Future[T]]
3) Double wrapping Futures directly a-la Future(Future(...)) doesn't change anything. Your method is executed on val etlF = g... (in the same thread) no matter what it returns. Future("ok")'s content (lambda) is executed eagerly (with "small" unpredictable delay) on a different thread but [execution task is being submitted to the pool] still inside getBusinessListFromModules.
One workaround (not really recommended) is val etlF = Future(getBusinessListFromModules(...)).flatMap(identity) which would return you a future wrapping any exception coming directly from getBusinessListFromModules and indirectly from getBusinessListFromModules's internal Future.
It's better to refactor getBusinessListFromModules itself however, also introduce different exception types for different kinds of trouble (validation, sync vs async, so on) your method might get into.
P.S. There are ways to mix async and sync exception handling, but in practice it's hard to analyze and predict such mixed behavior (which you probably noticed already). And code gets ugly.
I was hoping code like follows would wait for both futures, but it does not.
object Fiddle {
val f1 = Future {
throw new Throwable("baaa") // emulating a future that bumped into an exception
}
val f2 = Future {
Thread.sleep(3000L) // emulating a future that takes a bit longer to complete
2
}
val lf = List(f1, f2) // in the general case, this would be a dynamically sized list
val seq = Future.sequence(lf)
seq.onComplete {
_ => lf.foreach(f => println(f.isCompleted))
}
}
val a = FuturesSequence
I assumed seq.onComplete would wait for them all to complete before completing itself, but not so; it results in:
true
false
.sequence was a bit hard to follow in the source of scala.concurrent.Future, I wonder how I would implement a parallel that waits for all original futures of a (dynamically sized) sequence, or what might be the problem here.
Edit: A related question: https://worldbuilding.stackexchange.com/questions/12348/how-do-you-prove-youre-from-the-future :)
One common approach to waiting for all results (failed or not) is to "lift" failures into a new representation inside the future, so that all futures complete with some result (although they may complete with a result that represents failure). One natural way to get that is lifting to a Try.
Twitter's implementation of futures provides a liftToTry method that makes this trivial, but you can do something similar with the standard library's implementation:
import scala.util.{ Failure, Success, Try }
val lifted: List[Future[Try[Int]]] = List(f1, f2).map(
_.map(Success(_)).recover { case t => Failure(t) }
)
Now Future.sequence(lifted) will be completed when every future is completed, and will represent successes and failures using Try.
And so, a generic solution for waiting on all original futures of a sequence of futures may look as follows, assuming an execution context is of course implicitly available.
import scala.util.{ Failure, Success, Try }
private def lift[T](futures: Seq[Future[T]]) =
futures.map(_.map { Success(_) }.recover { case t => Failure(t) })
def waitAll[T](futures: Seq[Future[T]]) =
Future.sequence(lift(futures)) // having neutralized exception completions through the lifting, .sequence can now be used
waitAll(SeqOfFutures).map {
// do whatever with the completed futures
}
A Future produced by Future.sequence completes when either:
all the futures have completed successfully, or
one of the futures has failed
The second point is what's happening in your case, and it makes sense to complete as soon as one of the wrapped Future has failed, because the wrapping Future can only hold a single Throwable in the failure case. There's no point in waiting for the other futures because the result will be the same failure.
This is an example that supports the previous answer. There is an easy way to do this using just the standard Scala APIs.
In the example, I am creating 3 futures. These will complete at 5, 7, and 9 seconds respectively. The call to Await.result will block until all futures have resolved. Once all 3 futures have completed, a will be set to List(5,7,9) and execution will continue.
Additionally, if an exception is thrown in any of the futures, Await.result will immediately unblock and throw the exception. Uncomment the Exception(...) line to see this in action.
try {
val a = Await.result(Future.sequence(Seq(
Future({
blocking {
Thread.sleep(5000)
}
System.err.println("A")
5
}),
Future({
blocking {
Thread.sleep(7000)
}
System.err.println("B")
7
//throw new Exception("Ha!")
}),
Future({
blocking {
Thread.sleep(9000)
}
System.err.println("C")
9
}))),
Duration("100 sec"))
System.err.println(a)
} catch {
case e: Exception ⇒
e.printStackTrace()
}
Even though it is quite old question But this is how I got it running in recent time.
object Fiddle {
val f1 = Future {
throw new Throwable("baaa") // emulating a future that bumped into an exception
}
val f2 = Future {
Thread.sleep(3000L) // emulating a future that takes a bit longer to complete
2
}
val lf = List(f1, f2) // in the general case, this would be a dynamically sized list
val seq = Future.sequence(lf)
import scala.concurrent.duration._
Await.result(seq, Duration.Inf)
}
This won't get completed and will wait till all the future gets completed. You can change the waiting time as per your use case. I have kept it to infinite and that was required in my case.
We can enrich Seq[Future[T]] with its own onComplete method through an implicit class:
def lift[T](f: Future[T])(implicit ec: ExecutionContext): Future[Try[T]] =
f map { Success(_) } recover { case e => Failure(e) }
def lift[T](fs: Seq[Future[T]])(implicit ec: ExecutionContext): Seq[Future[Try[T]]] =
fs map { lift(_) }
implicit class RichSeqFuture[+T](val fs: Seq[Future[T]]) extends AnyVal {
def onComplete[U](f: Seq[Try[T]] => U)(implicit ec: ExecutionContext) = {
Future.sequence(lift(fs)) onComplete {
case Success(s) => f(s)
case Failure(e) => throw e // will never happen, because of the Try lifting
}
}
}
Then, in your particular MWE, you can do:
val f1 = Future {
throw new Throwable("baaa") // emulating a future that bumped into an exception
}
val f2 = Future {
Thread.sleep(3000L) // emulating a future that takes a bit longer to complete
2
}
val lf = List(f1, f2)
lf onComplete { _ map {
case Success(v) => ???
case Failure(e) => ???
}}
This solution has the advantage of allowing you to call an onComplete on a sequence of futures as you would on a single future.
Create the Future with a Try to avoid extra hoops.
implicit val ec = ExecutionContext.global
val f1 = Future {
Try {
throw new Throwable("kaboom")
}
}
val f2 = Future {
Try {
Thread.sleep(1000L)
2
}
}
Await.result(
Future.sequence(Seq(f1, f2)), Duration("2 sec")
) foreach {
case Success(res) => println(s"Success. $res")
case Failure(e) => println(s"Failure. ${e.getMessage}")
}
I've been working up my answer to Is there a standard Scala function for running a block with a timeout?, and have run into a problem if an exception is thrown in a Future.
def runWithTimeout[T](timeoutMs: Long)(f: => T) : Option[T] = {
awaitAll(timeoutMs, future(f)).head.asInstanceOf[Option[T]]
}
So that
runWithTimeout(50) { "result" } should equal (Some("result"))
runWithTimeout(50) { Thread.sleep(100); "result" } should equal (None)
But if I throw an exception in my block it doesn't leak, but is swallowed - so that the following fails with "..no exception was thrown"
intercept[Exception] {
runWithTimeout(50) { throw new Exception("deliberate") }
}.getMessage should equal("deliberate")
Syserr has a stack trace with the message
<function0>: caught java.lang.Exception: deliberate
but I can't find where in the Scala runtime that is printed.
Apart from wrapping f in another block which catches exceptions and propagates them if thrown, is there any way to persuade awaitAll and/or Future to throw?
Short answer: no.
Exceptions don't do what you want when you're working in a threaded context, because you want to know about the exception in the caller, and the exception happens in the future's thread.
Instead, if you want to know what the exception was, you should return an Either[Exception,WhatYouWant]--of course, you have to catch that exception within the future and package it up.
scala> scala.actors.Futures.future{
try { Right("fail".toInt) } catch { case e: Exception => Left(e) }
}
res0: scala.actors.Future[Product with Serializable with Either[Exception,Int]] = <function0>
scala> res0() // Apply the future
res1: Product with Serializable with Either[Exception,Int] =
Left(java.lang.NumberFormatException: For input string: "fail")
Disclaimer: I work for Typesafe
Or.... you could use Akka and it would give you what you want without you having to go through hoops for it.
val f: Future[Int] = actor !!! message
Then
f.get
Will throw the exception that happened in the actor
f.await.exception
will give you an Option[Throwable]
Working my way through #Rex Kerr's suggestion, I've created
object Timeout {
val timeoutException = new TimeoutException
def runWithTimeout[T](timeoutMs: Long)(f: => T) : Either[Throwable, T] = {
runWithTimeoutIgnoreExceptions(timeoutMs)(exceptionOrResult(f)) match {
case Some(x) => x
case None => Left(timeoutException)
}
}
def runWithTimeout[T](timeoutMs: Long, default: T)(f: => T) : Either[Throwable, T] = {
val defaultAsEither: Either[Throwable, T] = Right(default)
runWithTimeoutIgnoreExceptions(timeoutMs, defaultAsEither)(exceptionOrResult(f))
}
def runWithTimeoutIgnoreExceptions[T](timeoutMs: Long)(f: => T) : Option[T] = {
awaitAll(timeoutMs, future(f)).head.asInstanceOf[Option[T]]
}
def runWithTimeoutIgnoreExceptions[T](timeoutMs: Long, default: T)(f: => T) : T = {
runWithTimeoutIgnoreExceptions(timeoutMs)(f).getOrElse(default)
}
private def exceptionOrResult[T](f: => T): Either[Throwable, T] =
try {
Right(f)
} catch {
case x => Left(x)
}
}
so that
#Test def test_exception {
runWithTimeout(50) { "result" }.right.get should be ("result")
runWithTimeout(50) { throw new Exception("deliberate") }.left.get.getMessage should be ("deliberate")
runWithTimeout(50) { Thread.sleep(100); "result" }.left.get should be (Timeout.timeoutException)
runWithTimeout(50, "no result") { "result" }.right.get should be ("result")
runWithTimeout(50, "no result") { throw new Exception("deliberate") }.left.get.getMessage should be ("deliberate")
runWithTimeout(50, "no result") { Thread.sleep(100); "result" }.right.get should be ("no result")
}
Again, I'm a bit of a Scala novice, so would welcome feedback.
scala.concurrent.ops.future includes exception handling.
So, instead of importing scala.actors.Futures.future, import scala.concurrent.ops.future instead.
That simple change in which import is there will cause the caller's call to .get to rethrow the exception. It works great!
Or use Future.liftTryTry, turns it from Future[Object] to Future[Try[Object]], and you can match on the Try[Object] and check for an exception case Throw(e) and log / exit gracefully
You need to override the method exceptionHandler in order to catch exceptions. So your option is to define your own future method so it creates a MyFutureActor with exceptionHandler.
EDIT: FutureActor is private, so subclassing it isn't possible.
Another option is to use links to know when exceptions happened.
However, I think Rex Kerr's approach is better - just wrap the function in something that will catch the Exception. Too bad future doesn't already do that.
I implemented a simple job processor that processes subjobs within futures (scala.actors.Futures). These futures themselves can create more futures for processing subjobs. Now, if one of these subjobs throws an exception, i want the job processor to reply with an error message for that job. I have a workaround solution for discovering failed subjobs, but i'm not sure if that's the best solution. Basically it works like this:
sealed trait JobResult
case class SuccessResult(content: String) extends JobResult
case class FailedResult(message: String) extends JobResult
for(subjob <- subjobs) yield {
future {
try {
SuccessResult(process(subjob))
} catch {
case e:Exception => FailedResult(e.getMessage)
}
}
}
The result at the top level is a recursive List of Lists of Lists... of JobResults. I recursively search the List for a failed Result and then return an error or the combined result depending on the types of results.
That works but i'm wondering if there's is a more elegant/easier solution for dealing with exceptions in futures?
The way you do it now, is essentially what scala.Either was designed for. See http://www.scala-lang.org/api/current/scala/Either.html
Modern scala futures are like Either in that they contain either a successful result or a Throwable. If you re-visit this code in scala 2.10, i think you'll find the situation quite pleasant.
Specifically, scala.concurrent.Future[T] technically only "is-a" Awaitable[T], but _.onComplete and Await.ready(_, timeout).value.get both present its result as a scala.util.Try[T], which is a lot like Either[Throwable, T] in that it's either the result or an exception.
Oddly, _.transform takes two mapping functions, one for T => U and one for Throwable => Throwable and (unless i'm missing something) there's no transformer that maps the future as Try[T] => Try[U]. Future's .map will allow you to turn a success into a failure by simply throwing an exception in the mapping function, but it only uses that for successes of the original Future. Its .recover, similarly can turn a failure into a success. If you wanted to be able to change successes to failures and vice-versa, you'd need to build something yourself that was a combination of _.map and _.recover or else use _.onComplete to chain to a new scala.concurrent.Promise[U] like so:
import scala.util.{Try, Success, Failure}
import scala.concurrent.{Future, Promise}
import scala.concurrent.ExecutionContext
def flexibleTransform[T,U](fut: Future[T])(f: Try[T] => Try[U])(implicit ec: ExecutionContext): Future[U] = {
val p = Promise[U]
fut.onComplete { res =>
val transformed = f(res)
p.complete(transformed)
}
p.future
}
which would be used like so:
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Await
import scala.concurrent.duration.Duration.Inf
def doIt() {
val a: Future[Integer] = Future {
val r = scala.util.Random.nextInt
if (r % 2 == 0) {
throw new Exception("we don't like even numbers")
} else if (r % 3 == 0) {
throw new Exception("we don't like multiples of three")
} else {
r
}
}
val b: Future[String] = flexibleTransform(a) {
case Success(i) =>
if (i < 0) {
// turn negative successes into failures
Failure(new Exception("we don't like negative numbers"))
} else {
Success(i.toString)
}
case Failure(ex) =>
if (ex.getMessage.contains("three")) {
// nevermind about multiples of three being a problem; just make them all a word.
Success("three")
} else {
Failure(ex)
}
}
val msg = try {
"success: " + Await.result(b, Inf)
} catch {
case t: Throwable =>
"failure: " + t
}
println(msg)
}
for { _ <- 1 to 10 } doIt()
which would give something like this:
failure: java.lang.Exception: we don't like even numbers
failure: java.lang.Exception: we don't like negative numbers
failure: java.lang.Exception: we don't like negative numbers
success: three
success: 1756800103
failure: java.lang.Exception: we don't like even numbers
success: 1869926843
success: three
failure: java.lang.Exception: we don't like even numbers
success: three
(or you could "pimp" Future into a RichFutureWithFlexibleTransform with an implicit def and make flexibleTransform a member function of that, dropping the fut param and simply using this)
(even better would be to take Try[T] => Future[U] and call it flexibleFlatMap so you could do async things in the transform)