Blocking bad, async good, but is blocking within a future still blocking? This is something that I keep coming back to; consider following pseudo-code:
def queryName(id:Id):Future[String]
def queryEveryonesNames:Future[Seq[String]] = {
val everyonesIds:Future[Seq[Id]] = getIds
val everyonesNames:Future[Seq[Future[String]]] = {
everyonesIds.map(seq.map(id=>queryName(id)))
}
// I'm trying to understand the impact of what I'll do below
everyonesNames.map(seq=>seq.map(fut=>blocking(fut, 1 s)))
}
queryEveryonesNames
In the last line I turned Future[Seq[Future[String]]] (notice future within future) into Future[Seq[String]] by blocking on the inner future.
Blocking on a future within a future feels redundant, at least here, yet having a future within a future feels redundant as well.
Can you propose a smarter way of getting rid of the inner future?
Do you think blocking on a future inside a future is bad? If so why and under what circumstances?
Yes, future blocking is blocking, you should avoid that, as resources will be blocked to wait for a result, even if they are in another thread.
If I understood correctly, your question is how to convert Future[Seq[Future[String]]] into Future[Seq[String]] in a non-blocking way.
You can do that with for-comprehensions:
val in = Future[Seq[Future[String]]]
val m = for( a <- in ) // a is Seq[Future[String]]
yield ( Future.sequence(a)) // yields m = Future[Future[Seq[String]]]
val result = for(a <- m; b <- a) yield (b) // yields Future[Seq[String]]
EDIT:
Or just:
val result = in.flatMap(a => Future.sequence(a))
Related
So I was reading the "Scala with Cats" book, and there was this sentence which I'm going to quote down here:
Note that Scala’s Futures aren’t a great example of pure functional programming because they aren’t referentially transparent.
And also, an example is provided as follows:
val future1 = {
// Initialize Random with a fixed seed:
val r = new Random(0L)
// nextInt has the side-effect of moving to
// the next random number in the sequence:
val x = Future(r.nextInt)
for {
a <- x
b <- x
} yield (a, b)
}
val future2 = {
val r = new Random(0L)
for {
a <- Future(r.nextInt)
b <- Future(r.nextInt)
} yield (a, b)
}
val result1 = Await.result(future1, 1.second)
// result1: (Int, Int) = (-1155484576, -1155484576)
val result2 = Await.result(future2, 1.second)
// result2: (Int, Int) = (-1155484576, -723955400)
I mean, I think it's because of the fact that r.nextInt is never referentially transparent, right? since identity(r.nextInt) would never be equal to identity(r.nextInt), does this mean that identity is not referentially transparent either? (or Identity monad, to have better comparisons with Future). If the expression being calculated is RT, then the Future would also be RT:
def foo(): Int = 42
val x = Future(foo())
Await.result(x, ...) == Await.result(Future(foo()), ...) // true
So as far as I can reason about the example, almost every function and Monad type should be non-RT. Or is there something special about Future? I also read this question and its answers, yet couldn't find what I was looking for.
You are actually right and you are touching one of the pickiest points of FP; at least in Scala.
Technically speaking, Future on its own is RT. The important thing is that different to IO it can't wrap non-RT things into an RT description. However, you can say the same of many other types like List, or Option; so why folks don't make a fuss about it?
Well, as with many things, the devil is in the details.
Contrary to List or Option, Future is typically used with non-RT things; e.g. an HTTP request or a database query. Thus, the emphasis folks give in showing that Future can't guarantee RT in those situations.
More importantly, there is only one reason to introduce Future on a codebase, concurrency (not to be confused with parallelism); otherwise, it would be the same as Try. Thus, controlling when and how those are executed is usually important.
Which is the reason why cats recommends the use of IO for all use cases of Future
Note: You can find a similar discussion on this cats PR and its linked discussions: https://github.com/typelevel/cats/pull/4182
So... the referential transparency simply means that you should be able to replace the reference with the actual thing (and vice versa) without changing the overall symatics or behaviour. Like mathematics is.
So, lets say you have x = 4 and y = 5, then x + y, 4 + y, x + 5, and 4 + 5 are pretty much the same thing. And can be replaced with each otherwhenever you want.
But... just look at following two things...
val f1 = Future { println("Hi") }
val f2 = f1
val f1 = Future { println("Hi") }
val f2 = Future { println("Hi") }
You can try to run it. The behaviour of these two programs is not going to be the same.
Scala Future are eagerly evaluated... which means that there is no way to actually write Future { println("Hi") } in your code without executing it as a seperate behaviour.
Keep in mind that this is not just linked to having side effects. Yes, the example which I used here with println was a side effect, but that was just to make the behaviour difference obvious to notice.
Even if you use something to suspend the side effect inside the Future, you will endup with two suspended side effects instead of one. And once these suspended side effects are passed to the interpreater, the same action will happen twice.
In following example, even if we suspend the print side-effect by wrapping it up in an IO, the expansive evaluation part of the program can still cause different behavours even if everything in the universe is exactly same for two cases.
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
// cpu bound
// takes around 80 miliseconds
// we have only 1 core
def veryExpensiveComputation(input: Int): Int = ???
def impl1(): Unit = {
val f1 = Future {
val result = veryExpensiveComputation(10)
IO {
println(result)
result
}
}
val f2 = f1
val f3 = f1
val futures = Future.sequence(Seq(f1, f2, f3))
val ios = Await.result(futures, 100 milli)
}
def impl2(): Unit = {
val f1 = Future {
val result = veryExpensiveComputation(10)
IO {
println(result)
result
}
}
val f2 = Future {
val result = veryExpensiveComputation(10)
IO {
println(result)
result
}
}
val f3 = Future {
val result = veryExpensiveComputation(10)
IO {
println(result)
result
}
}
val futures = Future.sequence(Seq(f1, f2, f3))
val ios = Await.result(futures, 100 milli)
}
The first impl will cause only 1 expensive computation, but the second will trigger 3 expensive computations. And thus the program will fail with timeout in the second example.
If properly written with IO or ZIO (without Future), it with fail with timeout in both implementations.
I've got what seems like a fairly simple situation here:
I create a sequence of futures with yield of signature Future[(String, Date)]
I use Future.sequence to turn the IndexedSeq[Future[(String, Date)]] into a Future[IndexedSeq[(String, Date)]]
I call onComplete on that future to do something with each future after all of them have completed.
I'm doing this because the first wave are database inserts, I can't do the updateCalculatedField call until all inserts are completed, because updateCalculatedField involves aggregate functions and I don't want to introduce a race condition. Here's the code:
val initialInserts = for(i <- 0 until responseData.numValues()) yield { ... }
val res = Future.sequence(initialInserts)
res.onComplete{
case Success(seq) =>
for((i, d) <- seq)
updateCalculatedField(i, d)
case Failure(e) => e.printStackTrace()
}
What exactly am I doing wrong here? The code within the onComplete call never executes, and the program exits with no complaints. I've checked, and res.size is equal to the amount of rows in my database after the program exits... so the callback should have been executed. I'm at a loss here.
Thanks.
I want to create a list of Future, each of which could pass or fail and collate results from successful Future. How can I do this?
val futures2:List[Future[Int]] = List(Future{1}, Future{2},Future{throw new Exception("error")})
Questions
1) I want to wait for each future to finish
2) I want to collect sum of return values from each success future and ignore the ones which failed (so I should get 3).
One thing that you need to understand is that... Avoid trying to "get" values from a Future or Futures.
You can keep on operating in the Futuristic land.
val futureList = List(
Future(1),
Future(2),
Future(throw new Exception("error"))
)
// addd 1 to futures
// map will propagate errors to transformed futures
// only successful futures will result in +1, rest will stay with errors
val tranformedFutureList = futureList
.map(future => future.map(i => i + 1))
// print values of futures
// simimlar to map... for each will work only with successful futures
val unitFutureList = futureList
.map(future => future.foreach(i => println(i)))
// now lets give you sum of your "future" values
val sumFuture = futureList
.foldLeft(Future(0))((facc, f) => f.onComplete({
case Success(i) => facc.map(acc => acc + i)
case Failure(ex) => facc
})
And since OP (#Manu Chanda) asked about "getting" a value from a Promise, I am adding some bits about what Promise are in Scala.
So... first lets talk how to think about a Future in Scala.
If you see a Future[Int] then try to think of it as an ongoing computation which is "supposed to produce" an Int. Now that computation can successfully complete and result in a Success[Int] or a throw an exception and result in a Failure[Throwable]. And thus you see the functions such as onComplete, recoverWith, onFailure which seem like talking about a computation.
val intFuture = Future {
// all this inside Future {} is going to run in some other thread
val i = 5;
val j = i + 10;
val k = j / 5;
k
}
Now... what is a Promise.
Well... as the name indicates... a Promise[Int] is a promise of an Int value... nothing more.
Just like when a parent promises a certain toy to their child. Note that in this case... the parent has not necessarily started working on getting that toy, they have just promised that they will.
To complete the promise... they will first have to start working to complete it... got to market... buy from shop... come back home.Or... sometimes... they are busy so... they will ask someone else to bring that toy and keep doing their work... that other guy will try to bring that toy to parent (he may fail to buy it) and then they will complete the promise with whatever result they got from him.
So... basically a Promise wraps a Future inside of it. And that "wrapped" Future "value" can be considered as the value of the Promise.
so...
println("Well... The program wants an 'Int' toy")
// we "promised" our program that we will give it that int "toy"
val intPromise = Promise[Int]()
// now we can just move on with or life
println("Well... We just promised an 'Int' toy")
// while the program can make plans with how will it play with that "future toy"
val intFuture = intPromise.future
val plusOneIntFuture = intFuture.map(i => i + 1)
plusOneIntFuture.onComplete({
case Success(i) => println("Wow... I got the toy and modified it to - " + i)
case Failure(ex) => println("I did not get they toy")
})
// but since we at least want to try to complete our promise
println("Now... I suppose we need to get that 'Int' toy")
println("But... I am busy... I can not stop everything else for that toy")
println("ok... lets ask another thread to get that")
val getThatIntFuture = Future {
println("Well... I am thread 2... trying to get the int")
val i = 1
println("Well... I am thread 2... lets just return this i = 1 thingy")
i
}
// now lets complete our promise with whatever we will get from this other thread
getThatIntFuture.onComplete(intTry => intPromise.complete(intTry))
The above code will result in following output,
Well... The program wants an 'Int' toy
Well... We just promised an 'Int' toy
Now... I suppose we need to get that 'Int' toy
But... I am busy... I can not stop everything else for that toy
Well... I am thread 2... trying to get the int
Well... I am thread 2... lets just return this i = 1 thingy
Wow... I got the toy and modified it to - 2
Promise don't help you in "getting" a value from a Future. Asynchronous processes (or Future in Scala) are just running in another timeline... you can not "get" their "value" in your time-line unless you work on aligning your timeline with the process's time-line itself.
I have a for comprehension like this using an image library that supports concurrent operations https://github.com/sksamuel/scrimage :
for (
file <- myDirectory.listFiles;
image <- AsyncImage(file);
scaled <- image.scale(0.5)
// possibly more ops here?
)
{
scaled.writer(Format.PNG).write(new File(dirOutput + file.getName))
}
here is the definition for write():
def write(out: OutputStream)(implicit executionContext: ExecutionContext): Future[Unit] = Future {
writer.write(out)
}
What I want to do is wait until all my images in my directory finish resizing before my program shutdowns. From what I gleaned from other posts on SO is to basically stuff all these Futures into a list of Futures and and use Await for that to finish... Can anyone help me out here?
The problem you are encountering is that you are mixing two different monadic comprehensions, i.e. listFiles returns a List[File] while image.scale returns a Future[Image]. While both can be used in a for-comprehension, they cannot be mixed. Also, image <- AsyncImage(file) should be an assignment, since it just returns an instance:
// start the scaling operations (this is a List comprehension)
val scaleOps: List[Future[Unit]] = for {
file <- myDirectory.listFiles
image = AsyncImage(file)
// Now transform the image into a Future[Unit] with a nested Future comprehension
ops = for {
scaled <- image.scale(0.5)
written <- scaled.writer(Format.PNG).write(new File(dirOutput + file.getName))
} yield { written }
} yield { ops }
// collect and await the results
Await.result(Future.sequence(scaleOps), 10 minutes)
In the outer comprehension <- always produces an item from a List and yield returns the final list.
In the inner comprehension <- always produces the value of the Future, allowing you to use it as input for another call producing a Future, hence producing a Future[Unit] out of the image.
If I am making 3 different types of requests in a single Action, how should I handle things if they all have their own execution contexts, is there a best practise for this type of scenerio?
In an action I need to do the following:
Make db calls using slick to mysql
Make redis calls
Make memcache calls
So say my action looks like this where I have 3 different execution contexts, please correct me where I have erred:
def userProfile = Action.async {
Future {
// db
}(Contexts.myDbEC)
Future {
// redis calls
}(Contexts.myRedisEC)
Future {
// memcache calls
}(Contexts.myMemcacheEC)
}
Is it correct that I have to use Action.asynch in this case? Why is that?
What if I need the result of one execution context with the other, how will I handle this?
How will I combine the results when the Future code executes in the block, do I have to define my variables outside of the future calls or is there a way to get a return value?
Usage of Action.async is correct if you wanted to make your Action async which then should return a Future.
For your second question assuming you want to retrieve result of one future and combine with another one? - you can compose futures using flatMap or foreach.
Going with your example you can collect future results and use for comprehension to collect the result.
val future1 = future{ //db call }
val future2 = future{ //redis call}
val future3 = future{//memcache call}
val res = for{
r1 <- future1
r2 <- future2
r3 <- future3
} yield(r1+r2+r3)
note: it's important to run those futures outside for comprehension as above to run them in parallel. If you did following then they will run sequentially.
val res = for{
r1 <- future{//db call}
r2 <- future{//redis call}
r3 <- future{//memcache call}
}