The objective of this code is to take a sequence of futures, process them with Future.sequence, generate another sequence of futures and process them again with another Future.sequence.
The problem is that it doesn't print anything. What's wrong with this code?
object TestFutures extends App {
def future (i:Int) = Future { i }
def future2 (i:Int) = Future { i * 20 }
val futureResult = (1 to 10).map {
x => future(x)
}
var futureInts = Seq[Future[Int]]()
Future.sequence(futureResult).map{ list =>
list.foreach( y => futureInts = futureInts :+ future2(y))
}
Future.sequence(futureInts).map { list2 =>
list2.foreach( z => println(z))
}
Thread.sleep(5000)
}
It does work, you just have a race condition between the Thread.sleep and the futures finishing execution. You can wait on the futures completion with Await.result:
import scala.concurrent.{Await, Future}
import scala.concurrent.ExecutionContext.Implicits._
import scala.concurrent.duration._
def future(i: Int) = Future.successful(i)
def future2(i: Int) = Future.successful(i * 20)
val futureResult = (1 to 10).map(x => future(x))
val firstResult =
Future
.sequence(futureResult)
.flatMap(list => Future.sequence(list.map(future2))
val sequence = Await.result(firstResult, 10 seconds)
sequence.foreach(println)
Note you should not be synchronously blocking on futures in production code, this is merely for the demonstration that the future does do what you want it to.
Related
I am learning Future in Scala and have the following code snippet. I am generating random marks based on the first letter in the name.
For the following scenario, I expect a list to be printed by the onComplete method. But it does not print anything.
def randomMark(name:String) = name(0) match {
case 'R'|'A'|'J'|'S' => Thread.sleep(500); 99
case _ => Thread.sleep(500); 80
}
import scala.concurrent._
import concurrent.ExecutionContext.Implicits.global
val returns = Future sequence List( Future(randomMark("Rob")), Future(randomMark("Andy")), Future(randomMark("George")) )
Thread.sleep(550)
returns onComplete { e => { val y1 = e.getOrElse("Error"); println(y1) } }
//This println statement does not execute. I expect a list List(99,99,80) to be printed
Can someone please help me to understand why the functional literal I supply for onComplete method does not execute?
Thanks!
Most likely because you need to wait for the result because onComplete is an async operation.
import scala.concurrent.duration._
import scala.concurrent._
def randomMark(name:String) = name(0) match {
case 'R'|'A'|'J'|'S' => Thread.sleep(500); 99
case _ => Thread.sleep(500); 80
}
import scala.concurrent._
import concurrent.ExecutionContext.Implicits.global
val returns = Future sequence List( Future(randomMark("Rob")), Future(randomMark("Andy")), Future(randomMark("George")) )
Thread.sleep(550)
returns onComplete { e => { val y1 = e.getOrElse("Error"); println(y1) } }
println("Waiting futures to be completed")
Await.ready(returns, 5.seconds)
println("Futures to be completed")
Print's out:
Waiting futures to be completed
List(99, 99, 80)
Futures to be completed
Scatie: https://scastie.scala-lang.org/SWv18p8RTtuo7nMNHNHMoQ
Since callbacks are executed asynchronously, you'll need to wait for your callback to complete before the program exits. In this case, if you want to wait for an onComplete callback, you need to signal its completion somehow. In the following example I'm using a Promise:
def randomMark(name:String) = name(0) match {
case 'R'|'A'|'J'|'S' => Thread.sleep(500); 99
case _ => Thread.sleep(500); 80
}
import scala.concurrent._
import scala.concurrent.duration._
import ExecutionContext.Implicits.global
val returns = Future sequence List( Future(randomMark("Rob")), Future(randomMark("Andy")), Future(randomMark("George")) )
val returns: scala.concurrent.Future[List[Int]] = Future(<not completed>)
val p = Promise[Unit]()
returns onComplete { e => { val y1 = e.getOrElse("Error"); println(y1); p.success(()) } }
Await.ready(p.future, 5.seconds)
But, you could, instead use andThen as follows:
def randomMark(name:String) = name(0) match {
case 'R'|'A'|'J'|'S' => Thread.sleep(500); 99
case _ => Thread.sleep(500); 80
}
import scala.concurrent._
import scala.concurrent.duration._
import ExecutionContext.Implicits.global
val returns =
Future sequence List( Future(randomMark("Rob")), Future(randomMark("Andy")), Future(randomMark("George")) ) andThen
{ case e => { val y1 = e.getOrElse("Error"); println(y1) } }
Await.ready(returns, 5.seconds)
As mentioned in the jump-start guide, mapN will run all the futures in parallel, so I created the below simple Scala program, but a sample run shows diff to be 9187 ms and diffN to be 9106 ms. So it looks like that the mapN is also running the futures sequentially, isn't it? Please let me know if I am missing something?
package example
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
import java.time.LocalDateTime
import java.time.Duration
import scala.util.Failure
import scala.util.Success
import java.time.ZoneOffset
import cats.instances.future._
import cats.syntax.apply._
object FutureEx extends App {
val before = LocalDateTime.now()
val sum = for {
a <- getA
b <- getB
c <- getC
} yield (a + b + c)
sum onComplete {
case Failure(ex) => println(s"Error: ${ex.getMessage()}")
case Success(value) =>
val after = LocalDateTime.now()
println(s"before: $before")
println(s"after: $after")
val diff = getDiff(before, after)
println(s"diff: $diff")
println(s"sum: $value")
}
// let the above finish
Thread.sleep(20000)
val beforeN = LocalDateTime.now()
val usingMapN = (getA, getB, getC).mapN(add)
usingMapN onComplete {
case Failure(ex) => println(s"Error: ${ex.getMessage()}")
case Success(value) =>
val afterN = LocalDateTime.now()
println(s"beforeN: $beforeN")
println(s"afterN: $afterN")
val diff = getDiff(beforeN, afterN)
println(s"diffN: $diff")
println(s"sum: $value")
}
def getA: Future[Int] = {
println("inside A")
Thread.sleep(3000)
Future.successful(2)
}
def getB: Future[Int] = {
println("inside B")
Thread.sleep(3000)
Future.successful(3)
}
def getC: Future[Int] = {
println("inside C")
Thread.sleep(3000)
Future.successful(4)
}
def add(a: Int, b: Int, c: Int) = a + b + c
def getDiff(before: LocalDateTime, after: LocalDateTime): Long = {
Duration.between(before.toInstant(ZoneOffset.UTC), after.toInstant(ZoneOffset.UTC)).toMillis()
}
}
Because you have sleep outside Future it should be like:
def getA: Future[Int] = Future {
println("inside A")
Thread.sleep(3000)
2
}
So you start async Future with apply - Future.successful on the other hand returns pure value, meaning you execute sleep in same thread.
The time is going before mapN is ever called.
(getA, getB, getC).mapN(add)
This expression is creating a tuple (sequentially) and then calling mapN on it. So it is calling getA then getB then getC and since each of them has a 3 second delay, it takes 9 seconds.
There are some suggestions on StackOverflow on how to handle a list of Futures but I want to try my own approach. But I am unable to compile the following code
I have a list of Futures.
I want to count how many of them passed or failed. I should get (2,1)
I store this in a tuple
The approach I want to take is go through each element of the list. The element of the list is Future[Int]. for each element, I call flatMap which calls the next cycle of recursion (I am assuming that if flatMap gets called then that particular future would have been successful so I increment pass count). Similarly, I want to call next cycle of recursion in recover and increment fail count but I am getting compilation error in it.
import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Failure, Success, Try}
import scala.concurrent.duration._
import scala.language.postfixOps
object ConcurrencyExample extends App {
type pass = Int
type fail = Int
val time = System.currentTimeMillis()
//use recursion to process each Future in the list
def segregate(l:List[Future[Int]]):Future[Tuple2[pass,fail]] = {
def go(l:List[Future[Int]],t:Tuple2[pass,fail]):Future[Tuple2[pass,fail]] = {
l match {
case Nil => Future{t}
//l is List of Future[Int]. flatMap each successful Future
//recover each failed Future
case l::ls => {
l flatMap (x => go(ls, (t._1 + 1, t._2)))
**l.recover({ case e => go(ls, (t._1 + 1, t._2))})**//I get error here
}
}
}
go(l,(0,0))
}
//hardcoded future
val futures2: List[Future[Int]] = List(Future {
1
}, Future {
2
}, Future {
throw new Exception("error")
})
val result = segregate(futures2)
result onComplete {
case Success(v) => println("pp:" + v)
case Failure(v) => println("fp:" + v)
}
Await.result(result,1000 millis)
}
#evan058 is correct about the signature of recover. But you can fix your program by changing recover to recoverWith.
recoverWith is to recover as flatMap is to map.
Here's the complete solution (with minor stylistic improvements):
import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Failure, Success, Try}
import scala.concurrent.duration._
import scala.language.postfixOps
object ConcurrencyExample extends App {
type pass = Int
type fail = Int
val time = System.currentTimeMillis()
//use recursion to process each Future in the list
def segregate[T](fs:List[Future[T]]):Future[(pass,fail)] = {
def go(fs:List[Future[T]],r:Future[(pass,fail)]):Future[(pass,fail)] = fs match {
case Nil => r
case l::ls =>
val fx = l.transform({_ => (1, 0)}, identity).recoverWith[(pass,fail)]({case _: Exception => Future(0, 1) })
for (x <- fx; t <- r; g <- go(ls, Future(t._1+x._1,t._2+x._2))) yield g
}
go(fs,Future((0,0)))
}
//hardcoded future
val futures2 = List(Future(1), Future(2), Future(throw new Exception("error")))
val result = segregate(futures2)
result onComplete {
case Success(v) => println(s"successes: ${v._1}, failures: ${v._2}")
case Failure(v) => v.printStackTrace()
}
Await.result(result,1000 millis)
}
If you look at the docs, the signature of recover is:
def recover[U >: T](pf: PartialFunction[Throwable, U])(implicit executor: ExecutionContext): Future[U]
You are calling recover on l which is a Future[Int] so recover is expecting a U >: Int.
However you are calling go again which has the return type Future[(pass, fail)] which is not >: Int.
I have a couple of futures. campaignFuture returns a List[BigInt] and I want to be able to call the second future profileFuture for each of the values in the list returned from the first one. The second future can only be called when the first one is complete. How do I achieve this in Scala?
campaignFuture(1923).flatMap?? (May be?)
def campaignFuture(advertiserId: Int): Future[List[BigInt]] = Future {
val campaignHttpResponse = getCampaigns(advertiserId.intValue())
parseProfileIds(campaignHttpResponse.entity.asString)
}
def profileFuture(profileId: Int): Future[List[String]] = Future {
val profileHttpResponse = getProfiles(profileId.intValue())
parseSegmentNames(profileHttpResponse.entity.asString)
}
A for comprehension is here not applicable because we have a mix of List's and Future's.
So, your friends are map and flatMap:
To react on Future result
import scala.concurrent.{Future, Promise, Await}
import scala.concurrent.duration.Duration
import scala.concurrent.ExecutionContext.Implicits.global
def campaignFuture(advertiserId: Int): Future[List[BigInt]] = Future {
List(1, 2, 3)
}
def profileFuture(profileId: Int): Future[List[String]] = {
// delayed Future
val p = Promise[List[String]]
Future {
val delay: Int = (math.random * 5).toInt
Thread.sleep(delay * 1000)
p.success(List(s"profile-for:$profileId", s"delayed:$delay sec"))
}
p.future
}
// Future[List[Future[List[String]]]
val listOfProfileFuturesFuture = campaignFuture(1).map { campaign =>
campaign.map(id => profileFuture(id.toInt))
}
// React on Futures which are done
listOfProfileFuturesFuture foreach { campaignFutureRes =>
campaignFutureRes.foreach { profileFutureRes =>
profileFutureRes.foreach(profileListEntry => println(s"${new Date} done: $profileListEntry"))
}
}
// !!ONLY FOR TESTING PURPOSE - THIS CODE BLOCKS AND EXITS THE VM WHEN THE FUTURES ARE DONE!!
println(s"${new Date} waiting for futures")
listOfProfileFuturesFuture.foreach{listOfFut =>
Await.ready(Future.sequence(listOfFut), Duration.Inf)
println(s"${new Date} all futures done")
System.exit(0)
}
scala.io.StdIn.readLine()
To get the result of all Futures at once
import scala.concurrent.{Future, Await}
import scala.concurrent.duration.Duration
import scala.concurrent.ExecutionContext.Implicits.global
def campaignFuture(advertiserId: Int): Future[List[BigInt]] = Future {
List(1, 2, 3)
}
def profileFuture(profileId: Int): Future[List[String]] = Future {
List(s"profile-for:$profileId")
}
// type: Future[List[Future[List[String]]]]
val listOfProfileFutures = campaignFuture(1).map { campaign =>
campaign.map(id => profileFuture(id.toInt))
}
// type: Future[List[List[String]]]
val listOfProfileFuture = listOfProfileFutures.flatMap(s => Future.sequence(s))
// print the result
//listOfProfileFuture.foreach(println)
//scala.io.StdIn.readLine()
// wait for the result (THIS BLOCKS INFINITY!)
Await.result(listOfProfileFuture, Duration.Inf)
we use Future.sequence to convert a List[Future[T]] to Future[List[T]].
flatMap to get a Future[T] from Future[Future[T]]
if you need wait for the result (BLOCKING!) use Await to wait for the result
Suppose I have several futures and need to wait until either any of them fails or all of them succeed.
For example: Let there are 3 futures: f1, f2, f3.
If f1 succeeds and f2 fails I do not wait for f3 (and return failure to the client).
If f2 fails while f1 and f3 are still running I do not wait for them (and return failure)
If f1 succeeds and then f2 succeeds I continue waiting for f3.
How would you implement it?
You could use a for-comprehension as follows instead:
val fut1 = Future{...}
val fut2 = Future{...}
val fut3 = Future{...}
val aggFut = for{
f1Result <- fut1
f2Result <- fut2
f3Result <- fut3
} yield (f1Result, f2Result, f3Result)
In this example, futures 1, 2 and 3 are kicked off in parallel. Then, in the for comprehension, we wait until the results 1 and then 2 and then 3 are available. If either 1 or 2 fails, we will not wait for 3 anymore. If all 3 succeed, then the aggFut val will hold a tuple with 3 slots, corresponding to the results of the 3 futures.
Now if you need the behavior where you want to stop waiting if say fut2 fails first, things get a little trickier. In the above example, you would have to wait for fut1 to complete before realizing fut2 failed. To solve that, you could try something like this:
val fut1 = Future{Thread.sleep(3000);1}
val fut2 = Promise.failed(new RuntimeException("boo")).future
val fut3 = Future{Thread.sleep(1000);3}
def processFutures(futures:Map[Int,Future[Int]], values:List[Any], prom:Promise[List[Any]]):Future[List[Any]] = {
val fut = if (futures.size == 1) futures.head._2
else Future.firstCompletedOf(futures.values)
fut onComplete{
case Success(value) if (futures.size == 1)=>
prom.success(value :: values)
case Success(value) =>
processFutures(futures - value, value :: values, prom)
case Failure(ex) => prom.failure(ex)
}
prom.future
}
val aggFut = processFutures(Map(1 -> fut1, 2 -> fut2, 3 -> fut3), List(), Promise[List[Any]]())
aggFut onComplete{
case value => println(value)
}
Now this works correctly, but the issue comes from knowing which Future to remove from the Map when one has been successfully completed. As long as you have some way to properly correlate a result with the Future that spawned that result, then something like this works. It just recursively keeps removing completed Futures from the Map and then calling Future.firstCompletedOf on the remaining Futures until there are none left, collecting the results along the way. It's not pretty, but if you really need the behavior you are talking about, then this, or something similar could work.
You can use a promise, and send to it either the first failure, or the final completed aggregated success:
def sequenceOrBailOut[A, M[_] <: TraversableOnce[_]](in: M[Future[A]] with TraversableOnce[Future[A]])(implicit cbf: CanBuildFrom[M[Future[A]], A, M[A]], executor: ExecutionContext): Future[M[A]] = {
val p = Promise[M[A]]()
// the first Future to fail completes the promise
in.foreach(_.onFailure{case i => p.tryFailure(i)})
// if the whole sequence succeeds (i.e. no failures)
// then the promise is completed with the aggregated success
Future.sequence(in).foreach(p trySuccess _)
p.future
}
Then you can Await on that resulting Future if you want to block, or just map it into something else.
The difference with for comprehension is that here you get the error of the first to fail, whereas with for comprehension you get the first error in traversal order of the input collection (even if another one failed first). For example:
val f1 = Future { Thread.sleep(1000) ; 5 / 0 }
val f2 = Future { 5 }
val f3 = Future { None.get }
Future.sequence(List(f1,f2,f3)).onFailure{case i => println(i)}
// this waits one second, then prints "java.lang.ArithmeticException: / by zero"
// the first to fail in traversal order
And:
val f1 = Future { Thread.sleep(1000) ; 5 / 0 }
val f2 = Future { 5 }
val f3 = Future { None.get }
sequenceOrBailOut(List(f1,f2,f3)).onFailure{case i => println(i)}
// this immediately prints "java.util.NoSuchElementException: None.get"
// the 'actual' first to fail (usually...)
// and it returns early (it does not wait 1 sec)
Here is a solution without using actors.
import scala.util._
import scala.concurrent._
import java.util.concurrent.atomic.AtomicInteger
// Nondeterministic.
// If any failure, return it immediately, else return the final success.
def allSucceed[T](fs: Future[T]*): Future[T] = {
val remaining = new AtomicInteger(fs.length)
val p = promise[T]
fs foreach {
_ onComplete {
case s # Success(_) => {
if (remaining.decrementAndGet() == 0) {
// Arbitrarily return the final success
p tryComplete s
}
}
case f # Failure(_) => {
p tryComplete f
}
}
}
p.future
}
You can do this with futures alone. Here's one implementation. Note that it won't terminate execution early! In that case you need to do something more sophisticated (and probably implement the interruption yourself). But if you just don't want to keep waiting for something that isn't going to work, the key is to keep waiting for the first thing to finish, and stop when either nothing is left or you hit an exception:
import scala.annotation.tailrec
import scala.util.{Try, Success, Failure}
import scala.concurrent._
import scala.concurrent.duration.Duration
import ExecutionContext.Implicits.global
#tailrec def awaitSuccess[A](fs: Seq[Future[A]], done: Seq[A] = Seq()):
Either[Throwable, Seq[A]] = {
val first = Future.firstCompletedOf(fs)
Await.ready(first, Duration.Inf).value match {
case None => awaitSuccess(fs, done) // Shouldn't happen!
case Some(Failure(e)) => Left(e)
case Some(Success(_)) =>
val (complete, running) = fs.partition(_.isCompleted)
val answers = complete.flatMap(_.value)
answers.find(_.isFailure) match {
case Some(Failure(e)) => Left(e)
case _ =>
if (running.length > 0) awaitSuccess(running, answers.map(_.get) ++: done)
else Right( answers.map(_.get) ++: done )
}
}
}
Here's an example of it in action when everything works okay:
scala> awaitSuccess(Seq(Future{ println("Hi!") },
Future{ Thread.sleep(1000); println("Fancy meeting you here!") },
Future{ Thread.sleep(2000); println("Bye!") }
))
Hi!
Fancy meeting you here!
Bye!
res1: Either[Throwable,Seq[Unit]] = Right(List((), (), ()))
But when something goes wrong:
scala> awaitSuccess(Seq(Future{ println("Hi!") },
Future{ Thread.sleep(1000); throw new Exception("boo"); () },
Future{ Thread.sleep(2000); println("Bye!") }
))
Hi!
res2: Either[Throwable,Seq[Unit]] = Left(java.lang.Exception: boo)
scala> Bye!
For this purpose I would use an Akka actor. Unlike the for-comprehension, it fails as soon as any of the futures fail, so it's a bit more efficient in that sense.
class ResultCombiner(futs: Future[_]*) extends Actor {
var origSender: ActorRef = null
var futsRemaining: Set[Future[_]] = futs.toSet
override def receive = {
case () =>
origSender = sender
for(f <- futs)
f.onComplete(result => self ! if(result.isSuccess) f else false)
case false =>
origSender ! SomethingFailed
case f: Future[_] =>
futsRemaining -= f
if(futsRemaining.isEmpty) origSender ! EverythingSucceeded
}
}
sealed trait Result
case object SomethingFailed extends Result
case object EverythingSucceeded extends Result
Then, create the actor, send a message to it (so that it will know where to send its reply to) and wait for a reply.
val actor = actorSystem.actorOf(Props(new ResultCombiner(f1, f2, f3)))
try {
val f4: Future[Result] = actor ? ()
implicit val timeout = new Timeout(30 seconds) // or whatever
Await.result(f4, timeout.duration).asInstanceOf[Result] match {
case SomethingFailed => println("Oh noes!")
case EverythingSucceeded => println("It all worked!")
}
} finally {
// Avoid memory leaks: destroy the actor
actor ! PoisonPill
}
This question has been answered but I am posting my value class solution (value classes were added in 2.10) since there isn't one here. Please feel free to criticize.
implicit class Sugar_PimpMyFuture[T](val self: Future[T]) extends AnyVal {
def concurrently = ConcurrentFuture(self)
}
case class ConcurrentFuture[A](future: Future[A]) extends AnyVal {
def map[B](f: Future[A] => Future[B]) : ConcurrentFuture[B] = ConcurrentFuture(f(future))
def flatMap[B](f: Future[A] => ConcurrentFuture[B]) : ConcurrentFuture[B] = concurrentFutureFlatMap(this, f) // work around no nested class in value class
}
def concurrentFutureFlatMap[A,B](outer: ConcurrentFuture[A], f: Future[A] => ConcurrentFuture[B]) : ConcurrentFuture[B] = {
val p = Promise[B]()
val inner = f(outer.future)
inner.future onFailure { case t => p.tryFailure(t) }
outer.future onFailure { case t => p.tryFailure(t) }
inner.future onSuccess { case b => p.trySuccess(b) }
ConcurrentFuture(p.future)
}
ConcurrentFuture is a no overhead Future wrapper that changes the default Future map/flatMap from do-this-then-that to combine-all-and-fail-if-any-fail. Usage:
def func1 : Future[Int] = Future { println("f1!");throw new RuntimeException; 1 }
def func2 : Future[String] = Future { Thread.sleep(2000);println("f2!");"f2" }
def func3 : Future[Double] = Future { Thread.sleep(2000);println("f3!");42.0 }
val f : Future[(Int,String,Double)] = {
for {
f1 <- func1.concurrently
f2 <- func2.concurrently
f3 <- func3.concurrently
} yield for {
v1 <- f1
v2 <- f2
v3 <- f3
} yield (v1,v2,v3)
}.future
f.onFailure { case t => println("future failed $t") }
In the example above, f1,f2 and f3 will run concurrently and if any fail in any order the future of the tuple will fail immediately.
You might want to checkout Twitter's Future API. Notably the Future.collect method. It does exactly what you want: https://twitter.github.io/scala_school/finagle.html
The source code Future.scala is available here:
https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Future.scala
You can use this:
val l = List(1, 6, 8)
val f = l.map{
i => future {
println("future " +i)
Thread.sleep(i* 1000)
if (i == 12)
throw new Exception("6 is not legal.")
i
}
}
val f1 = Future.sequence(f)
f1 onSuccess{
case l => {
logInfo("onSuccess")
l.foreach(i => {
logInfo("h : " + i)
})
}
}
f1 onFailure{
case l => {
logInfo("onFailure")
}