Scala resolve multiple Futures and get a Map(String, AnyRef) - scala

I am currently trying to resolve multiple futures at once but as some of them may fail, I don't want to get a failure on all if one of them fails, instead, end up with a Map(String, AnyRef) (meaning a Map with the future name and the response converted to what a need).
Currently I have the following:
val fsResp = channelList.map {
channelRef => channelRef.ask(ReportStatus).mapTo[EventMessage]
}
Future.sequence(fsResp).onComplete{
case Success(resp: Seq[EventMessage]) =>
resp.foreach { event => Supervisor.foreach(_ ! event) }
val channels = loadConfiguredComponents()
.collect {
case ("processor" | "external", components) => components.map {
case (name, config: Channel) =>
(name, serializeDetails(config, resp.find(_.channel == ChannelName(name))))
}
}.flatten.toMap
val event = EventMessage(...)
Supervisor.foreach(_ ! event)
case Failure(exception) => originalSender ! replayError(exception.getMessage)
}
But this fails if any of those fails. So How can I end up with a Map(channelRef.path.name, event() | exception) ?
Thanks!

You can use fallbackTo in order to avoid a Failure. In this example I change Future[T] to Future[Option[T]] in order to fallback to None, and then remove None elements.
import scala.concurrent.ExecutionContext.Implicits.global
def method(value:Int) = { Thread.sleep(2000); println(value); value }
println("start")
val eventualNone = Future.successful(None)
val futures = List(Future(method(1)), Future(method(2)), Future(method(3)), Future(throw new RuntimeException))
val withoutFailures = futures.map(_.map(Option.apply).fallbackTo(eventualNone))
Future.sequence(withoutFailures).map(_.flatten).onComplete {
case Success(values) => println(values)
case Failure(ex:Throwable) => println("FAIL")
}
Thread.sleep(5000)
output
start
1
3
2
List(1, 2, 3)
Can be changed to Either[Throwable, T] instead of Option[T] if you want to know what fails.
This code always be Success (regarding the Future result), so you need to inspect your values in order to know if all futures fail.

To capture successful/failed values from the list of Futures, you can first apply map/recover to each of them, then use Future.sequence to transform the result list into a Future of List[Either[Throwable,EventMessage]], as shown in the following trivialized example:
import scala.concurrent.{Future, Await}
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
case class EventMessage(id: Int, msg: String)
val fsResp = List(
Future{EventMessage(1, "M1")}, Future{throw new Throwable()}, Future{EventMessage(3, "M3")}
)
val f = Future.sequence(
fsResp.map( _.map{ resp =>
// Do stuff with `resp`, add item to `Map()`, etc ...
Right(resp)
}.
recover{ case e: Throwable =>
// Log `exception` info, etc ...
Left(e)
} )
)
Await.result(f, Duration.Inf)
// f: scala.concurrent.Future[List[Product with Serializable with scala.util.
// Either[Throwable,EventMessage]]] = Future(Success(List(
// Right(EventMessage(1,M1)), Left(java.lang.Throwable), Right(EventMessage(3,M3))
// )))

Related

Error handling in list of scala futures - Apache Spark

I am having issues while handle exceptions in List of Scala futures. I am calling getQC_report(qcArgsThread,spark) method within ruuner method which process input file and saves in Hive table. Code below
import scala.util.{Failure, Success}
import scala.concurrent._
import scala.concurrent.duration._
val spark = SparkSession.builder.master("yarn").enableHiveSupport().getOrCreate()
var argsList: List[Array[String]] = List[Array[String]]()
for(ip_file <- INPUT_FILE.asScala.toList) {
var qcArgs:Array[String] = null
qcArgs = Array("input_file", ip_file,
"hiveDB",hiveDB,
"Outputhive_table",Outputhive_table)
argsList = qcArgs :: argsList
}
var pool = 0
def poolId = {
pool = pool + 1
pool
}
def runner(qcArgsThread: Array[String]) = Future {
sc.setLocalProperty("spark.scheduler.pool", poolId.toString)
getQC_report(qcArgsThread,spark)
}
val futures = argsList map(i => runner(i))
futures foreach(f => Await.ready(f, Duration.Inf))
futures.onComplete {
case Success(x) => {
println(s"\nresult = $x")
}
case Failure(e) => {
System.err.println("Failure happened!")
System.err.println(e.getMessage)
}
}
I am getting error in futures.onComplete line.
Error - Cannot resolve symbol onComplete.
Please help me in improving the code as I am new to using Scala Futures. Thanks!
The short answer is that because argsList is a List[Array[String]]
val futures = argsList map(i => runner(i))
will have the type List[Future[WhateverGetQC_ReportReturns]]. It specifically is not a Future, so has no onComplete method.
If you want to have a Future which completes when all the futures are completed, Future.sequence will convert a List[Future[T]] into a Future[List[T]]:
// replaces all code after val futures = argsList map ...
val allFutures = Future.sequence(futures)
val result: List[WhateverGetQC_ReportReturns] =
try {
Await.result(allFutures, Duration.Inf)
} catch {
case NonFatal(e) =>
System.err.println("Failure happened!")
System.err.println(e.getMessage)
}

Scala - Futures not starting

I want to start two workers on a Future method called extraire_phrases . I call them in my main, but it seems that the Promise is never fulfilled and I don't get anything at the end of my main, as if the workers don't start. Any ideas? Thanks a lot.
object Main {
val chemin_corpus:String = "src/corpus.txt"
val chemin_corpus_backup:String = "src/tartarinalpes.txt"
val chemin_dictionnaire:String = "src/dicorimes.dmp"
val chemin_dictionnaire_backup:String = "src/dicorimes2.dmp"
def main(args:Array[String]){
val quatrain = Promise[List[Phrase]]()
var grosPoeme = List[Phrase]()
Future {
val texte_1 = Phrases.extraire_phrases(chemin_corpus, chemin_dictionnaire)
val texte_2 = Phrases.extraire_phrases(chemin_corpus_backup, chemin_dictionnaire_backup)
texte_1.onComplete {
case Success(list) => {
val poeme = new DeuxVers(list)
poeme.ecrire :: grosPoeme
}
case Failure(ex) => {
quatrain.failure(LameExcuse("Error: " + ex.getMessage))
}
}
texte_2.onComplete {
case Success(lst) => {
val poeme2 = new DeuxVers(lst)
poeme2.ecrire :: grosPoeme
}
case Failure(ex) => {
quatrain.failure(LameExcuse("Error: " + ex.getMessage))
}
}
quatrain.success(grosPoeme)
}
println(quatrain.future)
println(grosPoeme)
}
}
Here is what I have in my console after execution:
Future(<not completed>)
List()
Even if I remove the Future { before val texte_1 it seems that none of them fire properly, texte_1 starts somehow, sometimes it works, sometimes not, and texte_2 never starts (never goes to completion). No failure either.
// Edit: Alvaro Carrasco's answer is the correct one. Thank both of you however for the help
Futures are executed asynchronously and your code won't "wait" for them to finish. onComplete will schedule some code to run when the future completes, but it won't force your program to wait for the result.
You need to thread the inner futures using map/flatMap/sequence so you end up with a single future at the end and then wait for it using Await.result(...).
You don't really need Promise here, as exceptions will caught by the future.
Something like this:
object Main {
val chemin_corpus:String = "src/corpus.txt"
...
def main(args:Array[String]){
...
val f1 = texte_1
.map {list =>
val poeme = new DeuxVers(list)
poeme.ecrire :: grosPoeme
}
val f2 = texte_2
.map {lst =>
val poeme2 = new DeuxVers(lst)
poeme2.ecrire :: grosPoeme
}
// combine both futures
val all = for {
res1 <- f1
res2 <- f2
} yield {
println(...)
}
// wait for the combined future
Await.result(all, 1.hour)
}
}
A solution with for-comprehension on Future. You need to change f1 and f2 to do what you need. f1 and f2 will be executed in parallel. for-comprehension gives elegant way to get the result of future(it's just syntactic sugar for compositions of operations with flatMap, filter and etc:
import scala.concurrent.{Await, Future}
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
val f1: Future[Seq[Int]] = Future {
// Do something here
Seq(1, 2, 3)
}.recover { case ex =>
// If Future fails, let's log an exception and return default value
println(s"Unable to complete f1: $ex")
Seq.empty[Int]
}
val f2: Future[Seq[Int]] = Future {
// Do something here
Seq(4, 5, 6)
}.recover { case ex =>
// If Future fails, let's log an exception and return default value
println(s"Unable to complete f2: $ex")
Seq.empty[Int]
}
// f1 and f2 have started
// we use for-comprehension on Future to get the result
val f = for {
seq1 <- f1
seq2 <- f2
} yield seq1 ++ seq2
// Blocking current thread and wait 1 seconds for the result
val r = Await.result(f, 1.seconds)
println(s"Result: $r")

NNTP client using akka streams in scala

I'm trying to implement an NNTP client that streams a list of commands to the server and parsing the results back. I'm facing several problems :
the NNTP protocol doesn't have an "unique" delimiter that could be use to frame results. Some commands return multi-line responses. How to handle that with streams ?
how to "map" the command issued with the server response and wait the end of server response before sending the next command ? (Throttling is not relevant here)
how to stop the stream processing on disconnection ? (Actually, the program never returns)
Here is my current implementation :
import akka.stream._
import akka.stream.scaladsl._
import akka.{ NotUsed, Done }
import akka.actor.ActorSystem
import akka.util.ByteString
import scala.concurrent._
import scala.concurrent.duration._
import java.nio.file.Paths
import scala.io.StdIn
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.Success
import scala.util.Failure
object AutomatedClient extends App {
implicit val system = ActorSystem("NewsClientTest")
implicit val materializer = ActorMaterializer()
// MODEL //
final case class Command(query: String)
final case class CommandResult(
resultCode: Int,
resultStatus: String,
resultList: Option[List[String]])
final case class ParseException(message: String) extends RuntimeException
// COMMAND HANDLING FUN //
// out ->
val sendCommand: Command => ByteString = c => ByteString(c.query + "\r\n")
// in <-
val parseCommandResultStatus: String => (Int, String) = s =>
(s.take(3).toInt, s.drop(3).trim)
val parseCommandResultList: List[String] => List[String] = l =>
l.foldLeft(List().asInstanceOf[List[String]]){
case (acc, ".") => acc
case (acc, e) => e.trim :: acc
}.reverse
val parseCommandResult: ByteString => Future[CommandResult] = b => Future {
val resultLines = b.decodeString("UTF-8").split("\r\n")
resultLines.length match {
case 0 => throw new ParseException("empty result")
case 1 =>
val (code, text) = parseCommandResultStatus(resultLines.head)
new CommandResult(code, text, None)
case _ =>
val (code, text) = parseCommandResultStatus(resultLines.head)
new CommandResult(code, text, Some(parseCommandResultList(resultLines.tail.toList)))
}
}
// STREAMS //
// Flows
val outgoing: Flow[Command, ByteString, NotUsed] = Flow fromFunction sendCommand
val incoming: Flow[ByteString, Future[CommandResult], NotUsed] = Flow fromFunction parseCommandResult
val protocol = BidiFlow.fromFlows(incoming, outgoing)
// Sink
val print: Sink[Future[CommandResult], _] = Sink.foreach(f =>
f.onComplete {
case Success(r) => println(r)
case Failure(r) => println("error decoding command result")
})
// Source
val testSource: Source[Command, NotUsed] = Source(List(
new Command("help"),
new Command("list"),
new Command("quit")
))
val (host, port) = ("localhost", 1119)
Tcp()
.outgoingConnection(host, port)
.join(protocol)
.runWith(testSource, print)
}
And here is the result output :
CommandResult(200,news.localhost NNRP Service Ready - newsmaster#localhost (posting ok),None)
CommandResult(100,Legal Commands,Some(List(article [<messageid>|number], authinfo type value, body [<messageid>|number], date, group newsgroup, head [<messageid>|number], help, last, list [active wildmat|active.times|counts wildmat], list [overview.fmt|newsgroups wildmat], listgroup newsgroup, mode reader, next, post, stat [<messageid>|number], xhdr field [range], xover [range], xpat field range pattern, xfeature useragent <client identifier>, xfeature compress gzip [terminator], xzver [range], xzhdr field [range], quit, 480 Authentication Required*, 205 Goodbye)))
We can see that the second CommandResult contains the result of "list" command and "quit" command.

How to retry failed Unmarshalling of a stream of akka-http requests?

I know it's possible to restart an akka-stream on error with a supervision strategy on the ActorMaterialzer
val decider: Supervision.Decider = {
case _: ArithmeticException => Supervision.Resume
case _ => Supervision.Stop
}
implicit val materializer = ActorMaterializer(
ActorMaterializerSettings(system).withSupervisionStrategy(decider))
val source = Source(0 to 5).map(100 / _)
val result = source.runWith(Sink.fold(0)(_ + _))
// the element causing division by zero will be dropped
// result here will be a Future completed with Success(228)
source: http://doc.akka.io/docs/akka/2.4.2/scala/stream/stream-error.html
I have the following use case.
/***
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-http-experimental" % "2.4.2",
"com.typesafe.akka" %% "akka-http-spray-json-experimental" % "2.4.2"
)
*/
import akka.http.scaladsl.unmarshalling.Unmarshal
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
import spray.json._
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import Uri.Query
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.scaladsl._
import scala.util.{Success, Failure}
import scala.concurrent.Await
import scala.concurrent.duration.Duration
import scala.concurrent.Future
object SO extends DefaultJsonProtocol {
implicit val system = ActorSystem()
import system.dispatcher
implicit val materializer = ActorMaterializer()
val httpFlow = Http().cachedHostConnectionPoolHttps[HttpRequest]("example.org")
def search(query: Char) = {
val request = HttpRequest(uri = Uri("https://example.org").withQuery(Query("q" -> query.toString)))
(request, request)
}
case class Hello(name: String)
implicit val helloFormat = jsonFormat1(Hello)
val searches =
Source('a' to 'z').map(search).via(httpFlow).mapAsync(1){
case (Success(response), _) => Unmarshal(response).to[Hello]
case (Failure(e), _) => Future.failed(e)
}
def main(): Unit = {
Await.result(searches.runForeach(_ => println), Duration.Inf)
()
}
}
Sometime a query will fail to unmarshall. I want to use a retry strategy on that single query
https://example.org/?q=v without restarting the whole alphabet.
I think it will be hard (or impossible) to implement it with a supervsior strategy, mostly because you want to retry "n" times (according to the discussion in comments), and I don't think you can track the number of times the element was tried when using supervision.
I think there are two ways to solve this issue. Either handle the risky operation as a separate stream or create a graph, which will do error handling. I will propose two solutions.
Note also that Akka Streams distinguishes between errors and failures, so if you wont' handle your failures they will eventually collapse the flow (if no strategy is intriduced), so in the example below I convert them to Either, which represent either success or error.
Separate stream
What you can do is to treat each alphabet letter as a separate stream and handle failures for each letter separately with the retry strategy, and some delay.
// this comes after your helloFormat
// note that the method is somehow simpler because it's
// using implicit dispatcher and scheduler from outside scope,
// you may also want to pass it as implicit arguments
def retry[T](f: => Future[T], delay: FiniteDuration, c: Int): Future[T] =
f.recoverWith {
// you may want to only handle certain exceptions here...
case ex: Exception if c > 0 =>
println(s"failed - will retry ${c - 1} more times")
akka.pattern.after(delay, system.scheduler)(retry(f, delay, c - 1))
}
val singleElementFlow = httpFlow.mapAsync[Hello](1) {
case (Success(response), _) =>
val f = Unmarshal(response).to[Hello]
f.recoverWith {
case ex: Exception =>
// see https://github.com/akka/akka/issues/20192
response.entity.dataBytes.runWith(Sink.ignore).flatMap(_ => f)
}
case (Failure(e), _) => Future.failed(e)
}
// so the searches can either go ok or not, for each letter, we will retry up to 3 times
val searches =
Source('a' to 'z').map(search).mapAsync[Either[Throwable, Hello]](1) { elem =>
println(s"trying $elem")
retry(
Source.single(elem).via(singleElementFlow).runWith(Sink.head[Hello]),
1.seconds, 3
).map(ok => Right(ok)).recover { case ex => Left(ex) }
}
// end
Graph
This method will integrate failures into the graph, and will allow for retries. This example makes all requests run in parallel and prefer to retry those which failed, but if you don't want this behaviour and run them one by one this is something you can also do I believe.
// this comes after your helloFormat
// you may need to have your own class if you
// want to propagate failures for example, but we will use
// right value to keep track of how many times we have
// tried the request
type ParseResult = Either[(HttpRequest, Int), Hello]
def search(query: Char): (HttpRequest, (HttpRequest, Int)) = {
val request = HttpRequest(uri = Uri("https://example.org").withQuery(Query("q" -> query.toString)))
(request, (request, 0)) // let's use this opaque value to count how many times we tried to search
}
val g = GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._
val searches = b.add(Flow[Char])
val tryParse =
Flow[(Try[HttpResponse], (HttpRequest, Int))].mapAsync[ParseResult](1) {
case (Success(response), (req, tries)) =>
println(s"trying parse response to $req for $tries")
Unmarshal(response).to[Hello].
map(h => Right(h)).
recoverWith {
case ex: Exception =>
// see https://github.com/akka/akka/issues/20192
response.entity.dataBytes.runWith(Sink.ignore).map { _ =>
Left((req, tries + 1))
}
}
case (Failure(e), _) => Future.failed(e)
}
val broadcast = b.add(Broadcast[ParseResult](2))
val nonErrors = b.add(Flow[ParseResult].collect {
case Right(x) => x
// you may also handle here Lefts which do exceeded retries count
})
val errors = Flow[ParseResult].collect {
case Left(x) if x._2 < 3 => (x._1, x)
}
val merge = b.add(MergePreferred[(HttpRequest, (HttpRequest, Int))](1, eagerComplete = true))
// #formatter:off
searches.map(search) ~> merge ~> httpFlow ~> tryParse ~> broadcast ~> nonErrors
merge.preferred <~ errors <~ broadcast
// #formatter:on
FlowShape(searches.in, nonErrors.out)
}
def main(args: Array[String]): Unit = {
val source = Source('a' to 'z')
val sink = Sink.seq[Hello]
source.via(g).toMat(sink)(Keep.right).run().onComplete {
case Success(seq) =>
println(seq)
case Failure(ex) =>
println(ex)
}
}
Basically what happens here is we run searches through httpFlow and then try to parse the response, we
then broadcast the result and split errors and non-errors, the non errors go to sink, and errors get sent
back to the loop. If the number of retries exceed the count, we ignore the element, but you can also do
something else with it.
Anyway I hope this gives you some idea.
For the streams solution above, any retries for the last element in the stream won't execute. That's because when the upstream completes after sending the last element the merge will also complete. After that the only output came come from the non-retry outlet but since the element goes to retry that gets completed too.
If you need all input elements to generate an output you'll need an extra mechanism to stop the upstream complete from reaching the process&retry graph. One possibility is to use a BidiFlow which monitors the input and output from the process&retry graph to ensure all the required outputs have been generated (for the observed inputs) before propagating the oncomplete. In the simple case that could just be counting input and output elements.

Scala: List[Future] to Future[List] disregarding failed futures

I'm looking for a way to convert an arbitrary length list of Futures to a Future of List. I'm using Playframework, so ultimately, what I really want is a Future[Result], but to make things simpler, let's just say Future[List[Int]] The normal way to do this would be to use Future.sequence(...) but there's a twist... The list I'm given usually has around 10-20 futures in it, and it's not uncommon for one of those futures to fail (they are making external web service requests).
Instead of having to retry all of them in the event that one of them fails, I'd like to be able to get at the ones that succeeded and return those.
For example, doing the following doesn't work:
import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.Success
import scala.util.Failure
val listOfFutures = Future.successful(1) :: Future.failed(new Exception("Failure")) ::
Future.successful(3) :: Nil
val futureOfList = Future.sequence(listOfFutures)
futureOfList onComplete {
case Success(x) => println("Success!!! " + x)
case Failure(ex) => println("Failed !!! " + ex)
}
scala> Failed !!! java.lang.Exception: Failure
Instead of getting the only the exception, I'd like to be able to pull the 1 and 3 out of there. I tried using Future.fold, but that apparently just calls Future.sequence behind the scenes.
The trick is to first make sure that none of the futures has failed. .recover is your friend here, you can combine it with map to convert all the Future[T] results to Future[Try[T]]] instances, all of which are certain to be successful futures.
note: You can use Option or Either as well here, but Try is the cleanest way if you specifically want to trap exceptions
def futureToFutureTry[T](f: Future[T]): Future[Try[T]] =
f.map(Success(_)).recover { case x => Failure(x)}
val listOfFutures = ...
val listOfFutureTrys = listOfFutures.map(futureToFutureTry(_))
Then use Future.sequence as before, to give you a Future[List[Try[T]]]
val futureListOfTrys = Future.sequence(listOfFutureTrys)
Then filter:
val futureListOfSuccesses = futureListOfTrys.map(_.filter(_.isSuccess))
You can even pull out the specific failures, if you need them:
val futureListOfFailures = futureListOfTrys.map(_.filter(_.isFailure))
Scala 2.12 has an improvement on Future.transform that lends itself in an anwser with less codes.
val futures = Seq(Future{1},Future{throw new Exception})
// instead of `map` and `recover`, use `transform`
val seq = Future.sequence(futures.map(_.transform(Success(_))))
val successes = seq.map(_.collect{case Success(x)=>x})
successes
//res1: Future[Seq[Int]] = Future(Success(List(1)))
val failures = seq.map(_.collect{case Failure(x)=>x})
failures
//res2: Future[Seq[Throwable]] = Future(Success(List(java.lang.Exception)))
I tried Kevin's answer, and I ran into a glitch on my version of Scala (2.11.5)... I corrected that, and wrote a few additional tests if anyone is interested... here is my version >
implicit class FutureCompanionOps(val f: Future.type) extends AnyVal {
/** Given a list of futures `fs`, returns the future holding the list of Try's of the futures from `fs`.
* The returned future is completed only once all of the futures in `fs` have been completed.
*/
def allAsTrys[T](fItems: /* future items */ List[Future[T]]): Future[List[Try[T]]] = {
val listOfFutureTrys: List[Future[Try[T]]] = fItems.map(futureToFutureTry)
Future.sequence(listOfFutureTrys)
}
def futureToFutureTry[T](f: Future[T]): Future[Try[T]] = {
f.map(Success(_)) .recover({case x => Failure(x)})
}
def allFailedAsTrys[T](fItems: /* future items */ List[Future[T]]): Future[List[Try[T]]] = {
allAsTrys(fItems).map(_.filter(_.isFailure))
}
def allSucceededAsTrys[T](fItems: /* future items */ List[Future[T]]): Future[List[Try[T]]] = {
allAsTrys(fItems).map(_.filter(_.isSuccess))
}
}
// Tests...
// allAsTrys tests
//
test("futureToFutureTry returns Success if no exception") {
val future = Future.futureToFutureTry(Future{"mouse"})
Thread.sleep(0, 100)
val futureValue = future.value
assert(futureValue == Some(Success(Success("mouse"))))
}
test("futureToFutureTry returns Failure if exception thrown") {
val future = Future.futureToFutureTry(Future{throw new IllegalStateException("bad news")})
Thread.sleep(5) // need to sleep a LOT longer to get Exception from failure case... interesting.....
val futureValue = future.value
assertResult(true) {
futureValue match {
case Some(Success(Failure(error: IllegalStateException))) => true
}
}
}
test("Future.allAsTrys returns Nil given Nil list as input") {
val future = Future.allAsTrys(Nil)
assert ( Await.result(future, 100 nanosecond).isEmpty )
}
test("Future.allAsTrys returns successful item even if preceded by failing item") {
val future1 = Future{throw new IllegalStateException("bad news")}
var future2 = Future{"dog"}
val futureListOfTrys = Future.allAsTrys(List(future1,future2))
val listOfTrys = Await.result(futureListOfTrys, 10 milli)
System.out.println("successItem:" + listOfTrys);
assert(listOfTrys(0).failed.get.getMessage.contains("bad news"))
assert(listOfTrys(1) == Success("dog"))
}
test("Future.allAsTrys returns successful item even if followed by failing item") {
var future1 = Future{"dog"}
val future2 = Future{throw new IllegalStateException("bad news")}
val futureListOfTrys = Future.allAsTrys(List(future1,future2))
val listOfTrys = Await.result(futureListOfTrys, 10 milli)
System.out.println("successItem:" + listOfTrys);
assert(listOfTrys(1).failed.get.getMessage.contains("bad news"))
assert(listOfTrys(0) == Success("dog"))
}
test("Future.allFailedAsTrys returns the failed item and only that item") {
var future1 = Future{"dog"}
val future2 = Future{throw new IllegalStateException("bad news")}
val futureListOfTrys = Future.allFailedAsTrys(List(future1,future2))
val listOfTrys = Await.result(futureListOfTrys, 10 milli)
assert(listOfTrys(0).failed.get.getMessage.contains("bad news"))
assert(listOfTrys.size == 1)
}
test("Future.allSucceededAsTrys returns the succeeded item and only that item") {
var future1 = Future{"dog"}
val future2 = Future{throw new IllegalStateException("bad news")}
val futureListOfTrys = Future.allSucceededAsTrys(List(future1,future2))
val listOfTrys = Await.result(futureListOfTrys, 10 milli)
assert(listOfTrys(0) == Success("dog"))
assert(listOfTrys.size == 1)
}
I just came across this question and have another solution to offer:
def allSuccessful[A, M[X] <: TraversableOnce[X]](in: M[Future[A]])
(implicit cbf: CanBuildFrom[M[Future[A]], A, M[A]],
executor: ExecutionContext): Future[M[A]] = {
in.foldLeft(Future.successful(cbf(in))) {
(fr, fa) ⇒ (for (r ← fr; a ← fa) yield r += a) fallbackTo fr
} map (_.result())
}
The idea here is that within the fold you are waiting for the next element in the list to complete (using the for-comprehension syntax) and if the next one fails you just fallback to what you already have.
You can easily wraps future result with option and then flatten the list:
def futureToFutureOption[T](f: Future[T]): Future[Option[T]] =
f.map(Some(_)).recover {
case e => None
}
val listOfFutureOptions = listOfFutures.map(futureToFutureOption(_))
val futureListOfOptions = Future.sequence(listOfFutureOptions)
val futureListOfSuccesses = futureListOfOptions.flatten
You can also collect successful and unsuccessful results in different lists:
def safeSequence[A](futures: List[Future[A]]): Future[(List[Throwable], List[A])] = {
futures.foldLeft(Future.successful((List.empty[Throwable], List.empty[A]))) { (flist, future) =>
flist.flatMap { case (elist, alist) =>
future
.map { success => (elist, alist :+ success) }
.recover { case error: Throwable => (elist :+ error, alist) }
}
}
}
If you need to keep failed futures for some reason, e.g., logging or conditional processing, this works with Scala 2.12+. You can find working code here.
val f1 = Future(1)
val f2 = Future(2)
val ff = Future.failed(new Exception())
val futures: Seq[Future[Either[Throwable, Int]]] =
Seq(f1, f2, ff).map(_.transform(f => Success(f.toEither)))
val sum = Future
.sequence(futures)
.map { eithers =>
val (failures, successes) = eithers.partitionMap(identity)
val fsum = failures.map(_ => 100).sum
val ssum = successes.sum
fsum + ssum
}
assert(Await.result(sum, 1.second) == 103)