I'm attempting to implement an API method (AsyncWriteJournal.asyncWriteMessages – part of Akka 2.4's persistence API) that requires me to return a Future[Seq[Try[Unit]]]. The idea being that a number of operations can be performed any of which may individually succeed or fail, the future can be used to await completion of all the operations. The future itself can optionally succeed even there's a certain class of failure in individual operations.
The underlying operations I'm using (Rediscala) return Futures but I'm stuck trying to transform a Seq[Future[Unit]] into the required return type. I can do it using Promise on each operation and tying them together with Future.sequence but then the resulting future fails if any of the operations fails which is not the correct behavior
Is this what you wanted?
def transform(ops: Seq[Future[Unit]])(implicit ec: ExecutionContext): Future[Seq[Try[Unit]]] =
Future.sequence(ops.map (
_.map(Success(_)).recover {
case ex => Failure(ex)
}
))
What about if you turn a possibly failing future into a future that will succeed in any case, and then use Future.sequence?
Related
I am trying to get my head around Scala's promise and future constructs.
I've been reading Futures and Promises in Scala Documentation and am a bit confused as I've got a feeling that the concepts of promises and futures are mixed up.
In my understanding a promise is a container that we could populate
value in a later point. And future is some sort of an asynchronous
operation that would complete in a different execution path.
In Scala we can obtain a result using the attached callbacks to future.
Where I'm lost is how promise has a future?
I have read about these concepts in Clojure too, assuming that promise and future have some generic common concept, but it seems like I was wrong.
A promise p completes the future returned by p.future. This future is
specific to the promise p. Depending on the implementation, it may be
the case that p.future eq p.
val p = promise[T]
val f = p.future
You can think of futures and promises as two different sides of a pipe.
On the promise side, data is pushed in, and on the future side, data can be pulled out.
And future is some sort of an asynchronous operation that would complete in a different execution path.
Actually, a future is a placeholder object for a value that may be become available at some point in time, asynchronously. It is not the asynchronous computation itself.
The fact that there is a future constructor called future that returns such a placeholder object and spawns an asynchronous computation that completes this placeholder object does not mean that the asynchronous computation is called a future. There are also other future constructors/factory methods.
But the point I do not get is how promise has a future?
To divide promises and futures into 2 separate interfaces was a design decision. You could have these two under the same interface Future, but that would then allow clients of futures to complete them instead of the intended completer of the future. This would cause unexpected errors, as there could be any number of contending completers.
E.g. for the asynchronous computation spawned by the future construct, it would no longer be clear whether it has to complete the promise, or if the client will do it.
Futures and promises are intended to constrain the flow of data in the program.
The idea is to have a future client that subscribes to the data to act on it once the data arrives.
The role of the promise client is to provide that data.
Mixing these two roles can lead to programs that are harder to understand or reason about.
You might also ask why the Promise trait does not extend Future. This is another design decision to discourage programmers from blindly passing Promises to clients where they should upcast the Promise to Future (this upcast is prone to be left out, whereas having to explicitly call future on the promise ensures you call it every time). In other words, by returning a promise you are giving the right to complete it to somebody else, and by returning the future you are giving the right to subscribe to it.
EDIT:
If you would like to learn more about futures, Chapter 4 in the Learning Concurrent Programming in Scala book describes them in detail. Disclaimer: I'm the author of the book.
The difference between the two is that futures are usually centered around the computation while promises are centered around data.
It seems your understanding matches this, but let me explain what I mean:
In both scala and clojure futures are (unless returned by some other function/method) created with some computation:
// scala
future { do_something() }
;; clojure
(future (do-something))
In both cases the "return-value" of the future can only be read (without blocking) only after the computation has terminated. When this is the case is typically outside the control of the programmer, as the computation gets executed in some thread (pool) in the background.
In contrast in both cases promises are an initially empty container, which can later be filled (exactly once):
// scala
val p = promise[Int]
...
p success 10 // or failure Exception()
;; clojure
(def p (promise))
(deliver p 10)
Once this is the case it can be read.
Reading the futures and promises is done through deref in clojure (and realized? can be used to check if deref will block). In scala reading is done through the methods provided by the Future trait. In order to read the result of a promise we thus have to obtain an object implementing Future, this is done by p.future. Now if the trait Future is implemented by a Promise, then p.future can return this and the two are equal. This is purely a implementation choice and does not change the concepts. So you were not wrong!
In any case Futures are mostly dealt with using callbacks.
At this point it might be worthwhile to reconsider the initial characterization of the two concepts:
Futures represent a computation that will produce a result at some point. Let's look at one possible implementation: We run the code in some thread(pool) and once its done, we arrange use the return value to fulfill a promise. So reading the result of the future is reading a promise; This is clojure's way of thinking (not necessarily of implementation).
On the other hand a promise represents a value that will be filled at some point. When it gets filled this means that some computation produced a result. So in a way this is like a future completing, so we should consume the value in the same way, using callbacks; This is scala's way of thinking.
Note that under the hood Future is implemented in terms of Promise and this Promise is completed with the body you passed to your Future:
def apply[T](body: =>T): Future[T] = impl.Future(body) //here I have omitted the implicit ExecutorContext
impl.Future is an implementation of Future trait:
def apply[T](body: =>T)(implicit executor: ExecutionContext): scala.concurrent.Future[T] =
{
val runnable = new PromiseCompletingRunnable(body)
executor.prepare.execute(runnable)
runnable.promise.future
}
Where PromiseCompletingRunnable looks like this:
class PromiseCompletingRunnable[T](body: => T) extends Runnable {
val promise = new Promise.DefaultPromise[T]()
override def run() = {
promise complete {
try Success(body) catch { case NonFatal(e) => Failure(e) }
}
} }
So you see even though they are seperate concepts that you can make use of independently in reality you can't get Future without using Promise.
I'm a spark Scala Programmer. I have a spark job that has sub-tasks which to complete the whole job. I wanted to use Future to complete the subtasks in parallel. On completion of the whole job I have to return the whole job response.
What I heard about scala Future is once the main thread executed and stopped the remaining threads will be killed and also you will get empty response.
I have to use Await.result to collect the results. But all the blogs are telling that you should avoid Await.result and it's a bad practice.
Is using Await.result is correct way of doing or not in my case?
def computeParallel(): Future[String] = {
val f1 = Future { "ss" }
val f2 = Future { "sss" }
val f3 = Future { "ssss" }
for {
r1 <- f1
r2 <- f2
r3 <- f3
} yield (r1 + r2 + r3)
}
computeParallel().map(result => ???)
To my understanding, we have to use Future in webservice kind of application where it has one process always running that won't be exited. But in my case, once logic execution(scala program) is complete it will exit.
Can I use Future to my problem or not?
Using futures in Spark is probably not advisable except in special cases, and simply parallelizing computation isn't one of them (giving a non-blocking wrapper to blocking I/O (e.g. making requests to an outside service) is quite possibly the only special case).
Note that Future doesn't guarantee parallelism (whether and how they're executed in parallel depends on the ExecutionContext in which they're run), just asynchrony. Also, in the event that you're spawning computation-performing futures inside a Spark transformation (i.e. on the executor, not the driver), chances are that there won't be any performance improvement, since Spark tends to do a good job of keeping the cores on the executors busy, all spawning those futures does is contend for cores with Spark.
Broadly, be very careful about combining parallelism abstractions like Spark RDDs/DStreams/Dataframes, actors, and futures: there are a lot of potential minefields where such combinations can violate guarantees and/or conventions in the various components.
It's also worth noting that Spark has requirements around serializability of intermediate values and that futures aren't generally serializable, so a Spark stage can't result in a future; this means that you basically have no choice but to Await on the futures spawned in a stage.
If you still want to spawn futures in a Spark stage (e.g. posting them to a web service), it's probably best to use Future.sequence to collapse the futures into one and then Await on that (note that I have not tested this idea: I'm assuming that there's an implicit CanBuildFrom[Iterator[Future[String]], String, Future[String]] available):
def postString(s: String): Future[Unit] = ???
def postStringRDD(rdd: RDD[String]): RDD[String] = {
rdd.mapPartitions { strings =>
// since this is only get used for combining the futures in the Await, it's probably OK to use the implicit global execution context here
implicit val ectx = ???
Await.result(strings.map(postString))
}
rdd // Pass through the original RDD
}
My scenario is, I have multiple API's call to be made and each of them will require to use data from previous call.
Now let's suppose I made a API call with future, and return response of the first call, I am not able to call next API from within Oncomplete of first call.
You can link Futures with flatMap function
def firstApiCall(): Future[FirstRes]
def secondApiCall(firstRes: FirstRes): Future[SecondRes]
def combinedCall()(implicit ec: ExecutionContext): Future[SecondRes] = firstApiCall.flatMap(res => secondApiCall(res))
You could use flatMap or the cleaner alternative to sequencing a bunch of of futures, which is to use the for yield syntax. It's just compile time sugar that will internally transform itself into flatMap and map and withFilter method calls, but it's generally cleaner and easier to read.
import scala.concurrent.ExecutionContext.Implicits.global
for {
first <- firstApiCall()
second <- secondApiCall(first)
} yield second
As I know, Future is read-only and Promise is write-once data structure.
We need a Promise to complete a Future
For example,
object Lie extends Throwable
val lie = Future { throw Lie }
val guess = Promise[String]()
lie.onComplete { case Success(s) => guess.success("I knew it was true!")
case Failure(t) => guess.failure("I knew it was lie")}
// return type: Unit
guess.future.map(println)
// res12: scala.concurrent.Future[Unit] = List()
// I knew it was lie!
// Requires Promise to chain Future with exception
But, I can't understand why we need to have both Future and Promise
I guess Promise is required because of Future.onComplete signature
Since Future.onComplete return type is Unit,Future with possible exceptions cannot be chained
I assume Promise was introduced to overcome this limitation
But why not just change the signature of Future.onComplete ?
Changing the return type of Future.onComplete as Future[T] will enable chaining on Future with exception
and then, Future does not need Promise
For example, code above can be changed into
val lie = Future { throw Lie }
lie.onComplete {
case Success(s) => "I knew it was true!"
case Failure(t) => "I knew it was lie!"
}.map(println)
//onComplete return type is Future[String]
My question is
1) am I right? does Future not need Promise , If onComplete signature is changed from Unit to Future[T]?
2) Why Future and Promise are divided in the first place ?
UDPATE
Thanks to the repliers, Now I understand the purpose of Promise. It wasn't actually for Future chaining
If I may, can I ask you
Why onComplete returns Unit ??
It can actually return Future[T] to enable chaining Future easily
For example
Future { throw Error }.onComplete {
case Success(s) => "Success"
case Failure(t) => throw Error
}.onComplete {
case Success(s) => "Success"
case Failure(t) => throw Error
}. ...
Future.apply[T](block: => T): Future[T] is syntactic sugar for Future.unit.map(_ => block)[1]
A Future represents a value which may or may not be currently available.
A Promise represents the obligation to provide such a value at some point.
Having separate entities for Future (for reads) and Promise (for writes) means that it is easy to reason about capabilities:
When a Future is a parameter, it is a request to have some value at some point and when it is used as a return type, it's a response which may not be currently available.
When a Promise is a parameter, it is the "consumption" of responsibility of producing some value at some point, and when it is used as a return type it is the "production" of responsibility to produce the value at some point.
All in all, being able to reason about capabilities, especially in asynchronous, or even concurrent programs, is extremely valuable.
Most of the time Promises need not be used, since that is transparently handled by the Future-combinators—but when integrating with third party software or networking libraries it can be extremely useful.
For more information about interesting new features in Scala 2.12, have a look here.
1: Where Future.unit
is defined as:
val unit: Future[Unit] = Future.successful(())
am I right? does Future not need Promise , If onComplete signature is
changed from Unit to Future[T]?
You're mixing things up a little. Let's clarify.
A Future[T] represents a computation which will complete in the future. That is, you pass Future.apply a function which will execute on a thread assigned by some ExecutionContext you define.
Now, on the other hand, a Promise[T] is a way to create a Future[T], without actually creating a Future[T]. A good example for this would be the Future.successful method (which will internally consume Promise.successful):
def successful[T](result: T): Future[T] = Promise.successful(result).future
This requires no ExecutionContext and no queuing if any additional resources. It's merely a convenience wrapper that allows you to "artificially" create a Future[T].
No.
Futurecannot cover the use cases of Promise. Promise has its own value. You cannot replace Future with Promise.
Future represents a computation which is available at a later point of time. You can get the result using onComplete once the future execution is complete and you can compose futures using map, flatMap, recover and recoverWith.
Promise is one time write container, clients can subscribe to the container. When subscribed client gets a future to wait until value is written into that container called promise.
Future is read-only
You cannot use future as one time write container. You can only read from future.
But Promise is different.
What if you want to give user something, which you do not have right now but you think you will have it soon ?
That means you are promising user something which you do not have right now.
So you want to keep the user waiting till you have the thing which you will give it to user. Thats when you do p.future and generate a future so that user can wait for the result using the future.
Once you have the result which you promised to give the user, You give to user by making the future a success or failure when something bad happens (i.e by doing p.complete).
Even if onComplete return type is changed to Future[T]. Future cannot act like or serve purpose of Promise.
Creating future using Future companion object
Future can be created by using Future.apply also. In this case future created would be complete once the computation inside the future finishes.
Future is used to subscribe to the result of the time-consuming computation, whereas Promise can be a publish and subscribe model.
I am trying to get my head around Scala's promise and future constructs.
I've been reading Futures and Promises in Scala Documentation and am a bit confused as I've got a feeling that the concepts of promises and futures are mixed up.
In my understanding a promise is a container that we could populate
value in a later point. And future is some sort of an asynchronous
operation that would complete in a different execution path.
In Scala we can obtain a result using the attached callbacks to future.
Where I'm lost is how promise has a future?
I have read about these concepts in Clojure too, assuming that promise and future have some generic common concept, but it seems like I was wrong.
A promise p completes the future returned by p.future. This future is
specific to the promise p. Depending on the implementation, it may be
the case that p.future eq p.
val p = promise[T]
val f = p.future
You can think of futures and promises as two different sides of a pipe.
On the promise side, data is pushed in, and on the future side, data can be pulled out.
And future is some sort of an asynchronous operation that would complete in a different execution path.
Actually, a future is a placeholder object for a value that may be become available at some point in time, asynchronously. It is not the asynchronous computation itself.
The fact that there is a future constructor called future that returns such a placeholder object and spawns an asynchronous computation that completes this placeholder object does not mean that the asynchronous computation is called a future. There are also other future constructors/factory methods.
But the point I do not get is how promise has a future?
To divide promises and futures into 2 separate interfaces was a design decision. You could have these two under the same interface Future, but that would then allow clients of futures to complete them instead of the intended completer of the future. This would cause unexpected errors, as there could be any number of contending completers.
E.g. for the asynchronous computation spawned by the future construct, it would no longer be clear whether it has to complete the promise, or if the client will do it.
Futures and promises are intended to constrain the flow of data in the program.
The idea is to have a future client that subscribes to the data to act on it once the data arrives.
The role of the promise client is to provide that data.
Mixing these two roles can lead to programs that are harder to understand or reason about.
You might also ask why the Promise trait does not extend Future. This is another design decision to discourage programmers from blindly passing Promises to clients where they should upcast the Promise to Future (this upcast is prone to be left out, whereas having to explicitly call future on the promise ensures you call it every time). In other words, by returning a promise you are giving the right to complete it to somebody else, and by returning the future you are giving the right to subscribe to it.
EDIT:
If you would like to learn more about futures, Chapter 4 in the Learning Concurrent Programming in Scala book describes them in detail. Disclaimer: I'm the author of the book.
The difference between the two is that futures are usually centered around the computation while promises are centered around data.
It seems your understanding matches this, but let me explain what I mean:
In both scala and clojure futures are (unless returned by some other function/method) created with some computation:
// scala
future { do_something() }
;; clojure
(future (do-something))
In both cases the "return-value" of the future can only be read (without blocking) only after the computation has terminated. When this is the case is typically outside the control of the programmer, as the computation gets executed in some thread (pool) in the background.
In contrast in both cases promises are an initially empty container, which can later be filled (exactly once):
// scala
val p = promise[Int]
...
p success 10 // or failure Exception()
;; clojure
(def p (promise))
(deliver p 10)
Once this is the case it can be read.
Reading the futures and promises is done through deref in clojure (and realized? can be used to check if deref will block). In scala reading is done through the methods provided by the Future trait. In order to read the result of a promise we thus have to obtain an object implementing Future, this is done by p.future. Now if the trait Future is implemented by a Promise, then p.future can return this and the two are equal. This is purely a implementation choice and does not change the concepts. So you were not wrong!
In any case Futures are mostly dealt with using callbacks.
At this point it might be worthwhile to reconsider the initial characterization of the two concepts:
Futures represent a computation that will produce a result at some point. Let's look at one possible implementation: We run the code in some thread(pool) and once its done, we arrange use the return value to fulfill a promise. So reading the result of the future is reading a promise; This is clojure's way of thinking (not necessarily of implementation).
On the other hand a promise represents a value that will be filled at some point. When it gets filled this means that some computation produced a result. So in a way this is like a future completing, so we should consume the value in the same way, using callbacks; This is scala's way of thinking.
Note that under the hood Future is implemented in terms of Promise and this Promise is completed with the body you passed to your Future:
def apply[T](body: =>T): Future[T] = impl.Future(body) //here I have omitted the implicit ExecutorContext
impl.Future is an implementation of Future trait:
def apply[T](body: =>T)(implicit executor: ExecutionContext): scala.concurrent.Future[T] =
{
val runnable = new PromiseCompletingRunnable(body)
executor.prepare.execute(runnable)
runnable.promise.future
}
Where PromiseCompletingRunnable looks like this:
class PromiseCompletingRunnable[T](body: => T) extends Runnable {
val promise = new Promise.DefaultPromise[T]()
override def run() = {
promise complete {
try Success(body) catch { case NonFatal(e) => Failure(e) }
}
} }
So you see even though they are seperate concepts that you can make use of independently in reality you can't get Future without using Promise.