I have the following code:
// Start async functions
val async1: Future[Seq[Int]] = ...
val async2: Future[Seq[Int]] = ...
val async3: Future[Seq[Int]] = ...
// Wait for completion
(for {
a1 <- async1
a2 <- async2
a3 <- async3
} yield (a1, a2, a3)).map {
// Use the results
}
I want to improve this to handle a variable amount of async functions (and not necessarily calling each of them every time). What I have done so far is:
// Start the async functions ?
val asyncs: Seq[Future[Seq[Int]] = otherList.filter(x => someCondition).map(x => asyncFunc(x))
// Wait for the functions to finish ?
(for (seqInt <- asyncs) yield seqInt).map {
case results => // <-- problem here
// Use the results
}
The problem I am having is that the results are of type Future[Seq[Int]], but I expected they would be of type (Seq[Int], Seq[Int], Seq[Int]) like in the first snippet.
In the end I would like to do is kickoff a dynamic amount of async functions which all have the same Future return type, wait for them all to finish, then use all of their results together.
Future.sequence is the key part I was missing (thanks for the comment)
// Create a list of Futures
val asyncs: Seq[Future[Seq[Int]] = otherList.filter(x => someCondition).map(x => asyncFunc(x))
// Use Future.sequence to to execute them and return a list of sequence of integers
Future.sequence(asyncs).map{
case results => // Use the results List[Seq[Int]]
}.recover {
case error => // Oh no!
}
Related
Say I have the following operations that must proceed in order:
Get blog post
Post analytics
Forward blog post
In code it may look like this:
val blogPostFut: Future[BlogPost] = blogService.getPost(postId)
val afterAnalytics: Future[BlogPost] = blogPostFut.flatMap(blogPost =>
val ignoredResponse: Future[Analytics] = analyticsService.sendAnalytics(blogPost)
ignoredResponse.map(_ => blogPost) // <-- THIS BOTHERS ME
)
val finalValue: Future[ForwardResult] = afterAnalytics.flatMap(blogPost =>
forwardService.forward(blogPost)
)
I am bothered that, in order to ensure proper ordering of execution, I have to pass forward blogPost within ignoredResponse in order to ensure it is available for step 3.
I'd love if I could do something like this:
blogPostFut.magicalFlatMap(analyticsService.sendAnalytics)
Where magicalFlatMap might be implemented like so:
// pseudocode
def magicalFlatMap[A,B](f: A => Future[B]): Future[A] = f().map(_ => this.value)
Does magicalFlatMap exist in either the Scala stdlib or in Cats? Is it possible to map a Future for side effects while automatically retaining the value of the original Future and strict ordering of operations?
magicalFlatMap seems to be cats.FlatMap#flatTap
https://github.com/typelevel/cats/blob/master/core/src/main/scala/cats/FlatMap.scala#L150
Try Future.andThen for side-effects
for {
blogPost <- blogService.getPost(postId).andThen { case Success(post) => analyticsService.sendAnalytics(post) }
finalValue <- forwardService.forward(blogPost)
} yield {
finalValue
}
Here is a dummy example
val result = for {
v1 <- Future(1)
v2 <- Future(v1 + 2).andThen { case Success(v) => println(v) }
v3 <- Future(v1 + v2)
} yield {
v3
}
result.foreach(println)
which should output
3
4
We could also do
for {
blogPost <- blogService.getPost(postId)
_ <- analyticsService.sendAnalytics(blogPost)
finalValue <- forwardService.forward(blogPost)
} yield {
finalValue
}
however in this case failure in analyticsService.sendAnalytics(blogPost) would short-circuit the whole for-comprehension which might not be desirable.
Testing this I can see that it works:
def twoHtmlFutures = Action { request =>
val async1 = as1.index(embed = true)(request) // Future[Result]
val async2 = as2.index(embed = true)(request) // Future[Result]
val async1Html = async1.flatMap(x => Pagelet.readBody(x)) // Future[Html]
val async2Html = async2.flatMap(x => Pagelet.readBody(x)) // Future[Html]
val source1 = Source.fromFuture(async1Html) // Source[Html, NotUsed]
val source2 = Source.fromFuture(async2Html) // Source[Html, NotUsed]
val merged = source1.merge(source2) // Source[Html, NotUsed]
Ok.chunked(merged)
}
But trying to put it into a For Comprehension is not working for me. This is what I tried:
def twoHtmlFutures2 = Action.async { request =>
val async1 = as1.index(embed = true)(request)
val async2 = as2.index(embed = true)(request)
for {
async1Res <- async1 // from Future[Result] to Result
async2Res <- async2 // from Future[Result] to Result
async1Html <- Pagelet.readBody(async1Res) // from Result to Html
async2Html <- Pagelet.readBody(async2Res) // from Result to Html
} yield {
val source1 = single(async1Html) // from Html to Source[Html, NotUsed]
val source2 = single(async2Html) // from Html to Source[Html, NotUsed]
val merged = source1.merge(source2) // Source[Html, NotUsed]
Ok.chunked(merged)
}
}
But this just jumps on-screen at the same time rather than at different times (streamed) as the first example does. Any helpers out there to widen my eyelids?Thanks
Monads are a sequencing shape and Futures models this as causal dependence (first-this-future-completes-then-that-future-completes):
val x = Future(something).map(_ => somethingElse) or Future(something).flatMap(_ => Future(somethingElse)
However, there's a little trick one can do in for comprehensions:
def twoHtmlFutures = Action { request =>
Ok.chunked(
Source.fromFutureSource(
for { _ <- Future.unit // For Scala version <= 2.11 use Future.successful(())
async1 = as1.index(embed = true)(request) // Future[Result]
async2 = as2.index(embed = true)(request) // Future[Result]
async1Html = async1.flatMap(x => Pagelet.readBody(x)) // Future[Html]
async2Html = async2.flatMap(x => Pagelet.readBody(x)) // Future[Html]
source1 = Source.fromFuture(async1Html) // Source[Html, NotUsed]
source2 = Source.fromFuture(async2Html) // Source[Html, NotUsed]
} yield source1.merge(source2) // Source[Html, NotUsed]
)
)
I describe this technique in greater detail in this blogpost.
An alternate solution to your problem could be:
def twoHtmlFutures = Action { request =>
Ok.chunked(
Source.fromFuture(as1.index(embed = true)(request)).merge(Source.fromFuture(as2.index(embed = true)(request))).mapAsyncUnordered(2)(b => Pagelet.readBody(b))
)
}
For comprehension and flatMap (which is its desugared version) are used to sequence things.
In the context of Future, this means that in a for comprehension, each of the statement is started only once the previous one has successfully ended.
In your case, you want two Futures run in parallel. This is not what flatMap (or for comprehension) is.
What your code do is the following:
do the first index call
when that's over, do the second index call
when that's over, do the first readBody
when that's over, do the second readBody
when that's over create two (synchronous) sources with the values from the two previous steps, merge them, and start returning the merged source as chunked response.
What your previous code did was
do the first index call
when that's over, do the first readBody
in the meantime, do the same for the second index and readBody
in the meantime, create a source that will output an element when the first readBody yields a result
in the meantime, do the same for the second
merge these two sources, and start all at once to give the merged output as a chunked response.
So, in this case, you start your chunked response just after receiving the request (but with nothing inside yet, waiting for the Futures to be resolved), while in the former case, you wait at each computation for the previous one to be over, even if you don't need its result to go on.
What you should remember, is that you should use flatMap on Future, only if you need the result from a previous computation or if you wish for another computation to be over before doing something else. The same goes for for comprehension, which is just a nice-looking way of chaining flatMaps.
Here's what your proposed for comprehension looks like after it's been desugared (courtesy of IntelliJ's "Desugar Scala code ..." menu option):
async1.flatMap((async1Res: Nothing) =>
async2.flatMap((async2Res: Nothing) =>
Pagelet.readBody(async1Res).flatMap((async1Html: Nothing) =>
Pagelet.readBody(async2Res).map((async2Html: Nothing) =>
Ok.chunked(merged)))))
As you can see, the nesting, and the concluding flatMap/map pair, are very different from your original code plan.
As a general rule, every <- in a single for comprehension is turned into a flatMap() except for the final one, which is a map(), and each is nested inside the previous.
What is the best way to go from
Map[String, Future[A]]
to
Map[String, A]
where A is the result of the corresponding future's execution?
This won't compile:
val results = for {
(key, future) <- myMap
result <- future
} yield (key, result)
as I can't mix futures and iterables in the same for comprehension.
If you convert it into a Seq[Future[(String,A)]], you can then use Future.fold to get it back a single Future[Map[...]]:
def transform[A](m: Map[String, Future[A]]): Future[Map[String, A]] = {
val seq: Seq[Future[(String, A)]] = m.toSeq.map { case (key, f) =>
f.map(i => key -> i)
}
Future.fold(seq)(Map.empty[String, A])(_ + _)
}
Then redeem the single future as normal.
Something like this perhaps:
map.mapValues { Await.result(_, 5 seconds) }
Dima already gave an answer using Await. However it will raise an exception when the Future fails.
You can further wrap the types inside as a Try and then do a .collect to filter only for the successful Futures (check out the official API for it).
import scala.util.{ Try, Success }
val results = myMap
.map {
case (key, value) => key => Try(Await.result(value, 5.seconds))
}
.collect {
case (key, Success(value)) => key -> value
}
With the call above, you automatically discard failing futures and only collect successful ones.
Given a method that returns a Future like this...
def myMethod(name: String, count: Int, default: Boolean): Future[Unit] = {
...
}
... I need to invoke it N times, so I've defined a list of tuples containing the parameters to be passed:
val paramList = List(
("text1", 22, true),
("text2", 55, true),
("text3", 77, false)
)
How do I invoke myMethod with Future.traverse?
Future.traverse(paramList)(myMethod _).tupled(/* how do I pass the current tuple here? */)
Two possible approaches:
val futuresFromSequence: Future[List[Unit]] = Future.sequence(paramList.map {
case (a,b,c) => myMethod(a,b,c)
})
val futuresFromTraverse: Future[List[Unit]] = Future.traverse(paramList)(x => x match {
case(a,b,c) => myMethod(a,b,c)
})
Note that traverse takes some collection and a function from an item of that collection to a future, sequence instead takes a list of futures and folds it to a future of list.
If you want to stick with the tupled syntax (which I personally dislike):
Future.traverse(paramList)(x => (myMethod _).tupled(x))
As noted in the comments you may want to run them in serial (although it's not 100% clear from the question), in that case you can use foldLeft and flatMap to chain future execution:
myList.foldLeft(Future(List.empty[Unit]))((prevFuture, currentTuple) => {
for {
prev <- prevFuture
curr <- (myMethod _).tupled(currentTuple)
} yield prev :+ curr
})
Where basically the first generator waits for the future in the accumulator to complete before launching the new one, this also returns a Future[List[Unit]].
This is a follow-up to my previous question
Suppose I want to create a future with my function but don't want to start it immediately (i.e. I do not want to call val f = Future { ... // my function}.
Now I see it can be done as follows:
val p = promise[Unit]
val f = p.future map { _ => // my function here }
Is it the only way to create a future with my function w/o executing it?
You can do something like this
val p = Promise[Unit]()
val f = p.future
//... some code run at a later time
p.success {
// your function
}
LATER EDIT:
I think the pattern you're looking for can be encapsulated like this:
class LatentComputation[T](f: => T) {
private val p = Promise[T]()
def trigger() { p.success(f) }
def future: Future[T] = p.future
}
object LatentComputation {
def apply[T](f: => T) = new LatentComputation(f)
}
You would use it like this:
val comp = LatentComputation {
// your code to be executed later
}
val f = comp.future
// somewhere else in the code
comp.trigger()
You could always defer creation with a closure, you'll not get the future object right ahead, but you get a handle to call later.
type DeferredComputation[T,R] = T => Future[R]
def deferredCall[T,R](futureBody: T => R): DeferredComputation[T,R] =
t => future {futureBody(t)}
def deferredResult[R](futureBody: => R): DeferredComputation[Unit,R] =
_ => future {futureBody}
If you are getting too fancy with execution control, maybe you should be using actors instead?
Or, perhaps, you should be using a Promise instead of a Future: a Promise can be passed on to others, while you keep it to "fulfill" it at a later time.
It's also worth giving a plug to Promise.completeWith.
You already know how to use p.future onComplete mystuff.
You can trigger that from another future using p completeWith f.
You can also define a function that creates and returns the Future, and then call it:
val double = (value: Int) => {
val f = Future { Thread.sleep(1000); value * 2 }
f.onComplete(x => println(s"Future return: $x"))
f
}
println("Before future.")
double(2)
println("After future is called, but as the future takes 1 sec to run, it will be printed before.")
I used this to executes futures in batches of n, something like:
// The functions that returns the future.
val double = (i: Int) => {
val future = Future ({
println(s"Start task $i")
Thread.sleep(1000)
i * 2
})
future.onComplete(_ => {
println(s"Task $i ended")
})
future
}
val numbers = 1 to 20
numbers
.map(i => (i, double))
.grouped(5)
.foreach(batch => {
val result = Await.result( Future.sequence(batch.map{ case (i, callback) => callback(i) }), 5.minutes )
println(result)
})
Or just use regular methods that return futures, and fire them in series using something like a for comprehension (sequential call-site evaluation)
This well known problem with standard libraries Future: they are designed in such a way that they are not referentially transparent, since they evaluate eagerly and memoize their result. In most use cases, this is totally fine and Scala developers rarely need to create non-evaluated future.
Take the following program:
val x = Future(...); f(x, x)
is not the same program as
f(Future(...), Future(...))
because in the first case the future is evaluated once, in the second case it is evaluated twice.
The are libraries which provide the necessary abstractions to work with referentially transparent asynchronous tasks, whose evaluation is deferred and not memoized unless explicitly required by the developer.
Scalaz Task
Monix Task
fs2
If you are looking to use Cats, Cats effects works nicely with both Monix and fs2.
this is a bit of a hack, since it have nothing to do with how future works but just adding lazy would suffice:
lazy val f = Future { ... // my function}
but note that this is sort of a type change as well, because whenever you reference it you will need to declare the reference as lazy too or it will be executed.