I have following pseudo code.
Invoke fetch, fetchRecordDetail, upload and notifyUploaded functions in sequence. Each function returns a future event but the first functions returns Option[T], going forward(fetchRecordDetail, upload and notifyUploaded) calls I need to carry only Some[T] type and ignore None.
Unfortunately I was able to achieve the following output with too many Await.ready calls.
Expected output
notified List(UploadResult(a_detail_uploaded), UploadResult(c_detail_uploaded))
Code
def fetch(id: String): Future[Option[Record]] = Future {
Thread sleep 100
if (id != "b" && id != "d") {
Some(Record(id))
} else None
}
def fetchRecordDetail(record: Record): Future[RecordDetail] = Future {
Thread sleep 100
RecordDetail(record.id + "_detail")
}
def upload(recordDetail: RecordDetail): Future[UploadResult] = Future {
Thread sleep 100
UploadResult(recordDetail.id + "_uploaded")
}
def notifyUploaded(results: Seq[UploadResult]): Future[Unit] = Future{ println("notified " + results)}
val result: Future[Unit] = //Final call to 'notifyUploaded' goes here
Await.ready(result, Duration.Inf)
Can someone help to improvise this code by avoiding Await.ready calls.
val ids: Seq[String] = Seq("a", "b", "c", "d")
def filterSome(s:String) = fetch(s) map ((s, _)) collect { case (s, Some(v)) => v }
val validData = ids map filterSome
Await.ready(Future.sequence(validData), Duration.Inf)
val records = validData.map(_.value.get.toOption)
val recordDetails = records.flatten map fetchRecordDetail
Await.ready(Future.sequence(recordDetails), Duration.Inf)
val uploadResult = recordDetails.map(_.value.get.toOption).flatten map upload
Await.ready(Future.sequence(uploadResult), Duration.Inf)
val seqUploadResult = uploadResult.map(_.value.get.toOption)
val result: Future[Unit] = notifyUploaded(seqUploadResult.flatten)
Await.ready(result, Duration.Inf)
This appears to work.
Future.sequence(ids.map(fetch)) //fetch Recs
.map(_.flatten) //remove None
.flatMap(rs=> Future.sequence(rs.map(fetchRecordDetail))) //fetch Details
.flatMap(ds=> Future.sequence(ds.map(upload))) //upload
.flatMap(notifyUploaded) //notify
It returns a Future[Unit] which you could Await() on, but I don't know why.
Something like that is what do you want?:
for {
f1 <- validData
f2 <- recordDetails
f3 <- seqUploadResult
}yield f3
onComplete(notifyUploaded(seqUploadResult.flatten))
Related
I want to start two workers on a Future method called extraire_phrases . I call them in my main, but it seems that the Promise is never fulfilled and I don't get anything at the end of my main, as if the workers don't start. Any ideas? Thanks a lot.
object Main {
val chemin_corpus:String = "src/corpus.txt"
val chemin_corpus_backup:String = "src/tartarinalpes.txt"
val chemin_dictionnaire:String = "src/dicorimes.dmp"
val chemin_dictionnaire_backup:String = "src/dicorimes2.dmp"
def main(args:Array[String]){
val quatrain = Promise[List[Phrase]]()
var grosPoeme = List[Phrase]()
Future {
val texte_1 = Phrases.extraire_phrases(chemin_corpus, chemin_dictionnaire)
val texte_2 = Phrases.extraire_phrases(chemin_corpus_backup, chemin_dictionnaire_backup)
texte_1.onComplete {
case Success(list) => {
val poeme = new DeuxVers(list)
poeme.ecrire :: grosPoeme
}
case Failure(ex) => {
quatrain.failure(LameExcuse("Error: " + ex.getMessage))
}
}
texte_2.onComplete {
case Success(lst) => {
val poeme2 = new DeuxVers(lst)
poeme2.ecrire :: grosPoeme
}
case Failure(ex) => {
quatrain.failure(LameExcuse("Error: " + ex.getMessage))
}
}
quatrain.success(grosPoeme)
}
println(quatrain.future)
println(grosPoeme)
}
}
Here is what I have in my console after execution:
Future(<not completed>)
List()
Even if I remove the Future { before val texte_1 it seems that none of them fire properly, texte_1 starts somehow, sometimes it works, sometimes not, and texte_2 never starts (never goes to completion). No failure either.
// Edit: Alvaro Carrasco's answer is the correct one. Thank both of you however for the help
Futures are executed asynchronously and your code won't "wait" for them to finish. onComplete will schedule some code to run when the future completes, but it won't force your program to wait for the result.
You need to thread the inner futures using map/flatMap/sequence so you end up with a single future at the end and then wait for it using Await.result(...).
You don't really need Promise here, as exceptions will caught by the future.
Something like this:
object Main {
val chemin_corpus:String = "src/corpus.txt"
...
def main(args:Array[String]){
...
val f1 = texte_1
.map {list =>
val poeme = new DeuxVers(list)
poeme.ecrire :: grosPoeme
}
val f2 = texte_2
.map {lst =>
val poeme2 = new DeuxVers(lst)
poeme2.ecrire :: grosPoeme
}
// combine both futures
val all = for {
res1 <- f1
res2 <- f2
} yield {
println(...)
}
// wait for the combined future
Await.result(all, 1.hour)
}
}
A solution with for-comprehension on Future. You need to change f1 and f2 to do what you need. f1 and f2 will be executed in parallel. for-comprehension gives elegant way to get the result of future(it's just syntactic sugar for compositions of operations with flatMap, filter and etc:
import scala.concurrent.{Await, Future}
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
val f1: Future[Seq[Int]] = Future {
// Do something here
Seq(1, 2, 3)
}.recover { case ex =>
// If Future fails, let's log an exception and return default value
println(s"Unable to complete f1: $ex")
Seq.empty[Int]
}
val f2: Future[Seq[Int]] = Future {
// Do something here
Seq(4, 5, 6)
}.recover { case ex =>
// If Future fails, let's log an exception and return default value
println(s"Unable to complete f2: $ex")
Seq.empty[Int]
}
// f1 and f2 have started
// we use for-comprehension on Future to get the result
val f = for {
seq1 <- f1
seq2 <- f2
} yield seq1 ++ seq2
// Blocking current thread and wait 1 seconds for the result
val r = Await.result(f, 1.seconds)
println(s"Result: $r")
Given the following methods...
def doSomething1: Future[Int] = { ... }
def doSomething2: Future[Int] = { ... }
def doSomething3: Future[Int] = { ... }
... and the following for-comprehension:
for {
x <- doSomething1
y <- doSomething2
z <- doSomething3
} yield x + y + z
The three methods run in parallel, but in my case doSomething2 MUST run after doSomething1 has finished. How do I run the three methods in sequence?
EDIT
As suggested by Philosophus42, here below is a possible implementation of doSomething1:
def doSomething1: Future[Int] = {
// query the database for customers younger than 40;
// `find` returns a `Future` containing the number of matches
customerService.find(Json.obj("age" -> Json.obj("$lt" -> 40)))
}
... so the Future is created by an internal call to another method.
EDIT 2
Perhaps I simplified the use case too much... and I'm sorry. Let's try again and go closer to the real use-case. Here are the three methods:
for {
// get all the transactions generated by the exchange service
transactions <- exchange.orderTransactions(orderId)
//for each transaction create a log
logs <- Future.sequence(tansactions.map { transaction =>
for {
// update trading order status
_ <- orderService.findAndUpdate(transaction.orderId, "Executed")
// create new log
log <- logService.insert(Log(
transactionId => transaction.id,
orderId => transaction.orderId,
...
))
} yield log
})
} yield logs
What I'm trying to do is to create a log for each transaction associated with an order. logService.insert gets invoked many times even if transactions just contains one entry.
Comment on your post
First, how does the code inside doSomethingX look like? Even more irrated, that with your given code, the futures run parallel.
Answer
In order to make the Future execution sequential, just use
for {
v1 <- Future { ..block1... }
v2 <- Future { ..block2... }
} yield combine(v1, v2)
The reason this works, is that the statement Future { ..body.. } starts asynchronous computation, at that point in time the statement is evaluated.
With the above for-comprehension desugared
Future { ..block1.. }
.flatMap( v1 =>
Future { ..block>.. }
.map( v2 => combine(v1,v2) )
)
it is obvious, that
if Future{ ...block1... } has it's result available,
the flatMap method is triggered, which
then triggers execution of Future { ...block2... }.
Thus Future { ...block2... } is executed after Future { ...block1... }
Additional information
A Future
Future {
<block>
}
immediately triggers execution of contained block via the ExecutionContext
Snippet 1:
val f1 = Future { <body> }
val f2 = Future { <otherbody> }
The two computations are running parallel (in case your ExecutionContext is setup this way), as the two values are evaluated immediately.
Snippet 2:
The construct
def f1 = Future { ..... }
will start execution of the future, once f1 is called
Edit:
j3d, I'm still confused, why your code does not work as expected, if your statement is correct, that the Future is created within the computeSomethingX methods.
Here is a code snippet that proves, that computeSomething2 is executed after computeSomething1
import scala.concurrent.{Await, Future}
import scala.concurrent.duration._
object Playground {
import scala.concurrent.ExecutionContext.Implicits.global
def computeSomething1 : Future[Int] = {
Future {
for (i <- 1 to 10) {
println("computeSomething1")
Thread.sleep(500)
}
10
}
}
def computeSomething2 : Future[String] = {
Future {
for(i <- 1 to 10) {
println("computeSomething2")
Thread.sleep(800)
}
"hello"
}
}
def main(args: Array[String]) : Unit = {
val resultFuture: Future[String] = for {
v1 <- computeSomething1
v2 <- computeSomething2
} yield v2 + v1.toString
// evil "wait" for result
val result = Await.result(resultFuture, Duration.Inf)
println( s"Result: ${result}")
}
}
with output
computeSomething1
computeSomething1
computeSomething1
computeSomething1
computeSomething1
computeSomething1
computeSomething1
computeSomething1
computeSomething1
computeSomething1
computeSomething2
computeSomething2
computeSomething2
computeSomething2
computeSomething2
computeSomething2
computeSomething2
computeSomething2
computeSomething2
computeSomething2
Result: hello10
Edit 2
If you want them to be executed in parallel, create the futures beforehand (here f1 and f2)
def main(args: Array[String]) : Unit = {
val f1 = computeSomething1
val f2 = computeSomething2
val resultFuture: Future[String] = for {
v1 <- f1
v2 <- f2
} yield v2 + v1.toString
// evil "wait" for result
val result = Await.result(resultFuture, Duration.Inf)
println( s"Result: ${result}")
}
I see two variants to achieve this:
First:
Make sure that the Futures are created inside the for comprehension. This means your functions should be defined like this: def doSomething1: Future[Int] = Future { ... }. In that case the for comprehension should execute the Futures in sequence.
Second:
Use the map function of the Future that you need to complete before the others start:
doSomething1.map{ i =>
for {
y <- doSomething2
z <- doSomething3
} yield i + y + z
}
I have two external processes to be run sequentially:
val antProc = Process(Seq(antBatch","everythingNoJunit"), new File(scriptDir))
val bossProc = Process(Seq(bossBatch,"-DcreateConnectionPools=true"))
val f: Future[Process] = Future {
println("Run ant...")
antProc.run
}
f onSuccess {
case proc => {
println("Run boss...")
bossProc.run
}
}
The result is:
Run ant...
Process finished with exit code 0
How do I run antProc until completion, then bossProc?
The following method seems to achieve the purpose. However, it's not a Future approach.
antProc.!<
bossProc.!<
You should be able to do something like this:
val antProc = Process(Seq(antBatch,"everythingNoJunit"), new File(scriptDir))
val bossProc = Process(Seq(bossBatch,"-DcreateConnectionPools=true"))
val antFut: Future[Process] = Future {
antProc.run
}
val bossFut: Future[Process] = Future {
bossProc.run
}
val aggFut = for{
antRes <- antFut
bossRes <- bossFut
} yield (antRes, bossRes)
aggFut onComplete{
case tr => println(tr)
}
The result of the aggFut will be a tuple consisting of the ant result and the boss result.
Also, be sure your vm that is running this is not exiting before the async callbacks can occur. If your execution context contains daemon threads then it might exit before completion.
Now if you want bossProc to run after antProc, the code would look like this
val antProc = Process(Seq(antBatch,"everythingNoJunit"), new File(scriptDir))
val bossProc = Process(Seq(bossBatch,"-DcreateConnectionPools=true"))
val antFut: Future[Process] = Future {
antProc.run
}
val aggFut = for{
antRes <- antFut
bossRes <- Future {bossProc.run}
} yield (antRes, bossRes)
aggFut onComplete{
case tr => println(tr)
}
I'm looking for a way to convert an arbitrary length list of Futures to a Future of List. I'm using Playframework, so ultimately, what I really want is a Future[Result], but to make things simpler, let's just say Future[List[Int]] The normal way to do this would be to use Future.sequence(...) but there's a twist... The list I'm given usually has around 10-20 futures in it, and it's not uncommon for one of those futures to fail (they are making external web service requests).
Instead of having to retry all of them in the event that one of them fails, I'd like to be able to get at the ones that succeeded and return those.
For example, doing the following doesn't work:
import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.Success
import scala.util.Failure
val listOfFutures = Future.successful(1) :: Future.failed(new Exception("Failure")) ::
Future.successful(3) :: Nil
val futureOfList = Future.sequence(listOfFutures)
futureOfList onComplete {
case Success(x) => println("Success!!! " + x)
case Failure(ex) => println("Failed !!! " + ex)
}
scala> Failed !!! java.lang.Exception: Failure
Instead of getting the only the exception, I'd like to be able to pull the 1 and 3 out of there. I tried using Future.fold, but that apparently just calls Future.sequence behind the scenes.
The trick is to first make sure that none of the futures has failed. .recover is your friend here, you can combine it with map to convert all the Future[T] results to Future[Try[T]]] instances, all of which are certain to be successful futures.
note: You can use Option or Either as well here, but Try is the cleanest way if you specifically want to trap exceptions
def futureToFutureTry[T](f: Future[T]): Future[Try[T]] =
f.map(Success(_)).recover { case x => Failure(x)}
val listOfFutures = ...
val listOfFutureTrys = listOfFutures.map(futureToFutureTry(_))
Then use Future.sequence as before, to give you a Future[List[Try[T]]]
val futureListOfTrys = Future.sequence(listOfFutureTrys)
Then filter:
val futureListOfSuccesses = futureListOfTrys.map(_.filter(_.isSuccess))
You can even pull out the specific failures, if you need them:
val futureListOfFailures = futureListOfTrys.map(_.filter(_.isFailure))
Scala 2.12 has an improvement on Future.transform that lends itself in an anwser with less codes.
val futures = Seq(Future{1},Future{throw new Exception})
// instead of `map` and `recover`, use `transform`
val seq = Future.sequence(futures.map(_.transform(Success(_))))
val successes = seq.map(_.collect{case Success(x)=>x})
successes
//res1: Future[Seq[Int]] = Future(Success(List(1)))
val failures = seq.map(_.collect{case Failure(x)=>x})
failures
//res2: Future[Seq[Throwable]] = Future(Success(List(java.lang.Exception)))
I tried Kevin's answer, and I ran into a glitch on my version of Scala (2.11.5)... I corrected that, and wrote a few additional tests if anyone is interested... here is my version >
implicit class FutureCompanionOps(val f: Future.type) extends AnyVal {
/** Given a list of futures `fs`, returns the future holding the list of Try's of the futures from `fs`.
* The returned future is completed only once all of the futures in `fs` have been completed.
*/
def allAsTrys[T](fItems: /* future items */ List[Future[T]]): Future[List[Try[T]]] = {
val listOfFutureTrys: List[Future[Try[T]]] = fItems.map(futureToFutureTry)
Future.sequence(listOfFutureTrys)
}
def futureToFutureTry[T](f: Future[T]): Future[Try[T]] = {
f.map(Success(_)) .recover({case x => Failure(x)})
}
def allFailedAsTrys[T](fItems: /* future items */ List[Future[T]]): Future[List[Try[T]]] = {
allAsTrys(fItems).map(_.filter(_.isFailure))
}
def allSucceededAsTrys[T](fItems: /* future items */ List[Future[T]]): Future[List[Try[T]]] = {
allAsTrys(fItems).map(_.filter(_.isSuccess))
}
}
// Tests...
// allAsTrys tests
//
test("futureToFutureTry returns Success if no exception") {
val future = Future.futureToFutureTry(Future{"mouse"})
Thread.sleep(0, 100)
val futureValue = future.value
assert(futureValue == Some(Success(Success("mouse"))))
}
test("futureToFutureTry returns Failure if exception thrown") {
val future = Future.futureToFutureTry(Future{throw new IllegalStateException("bad news")})
Thread.sleep(5) // need to sleep a LOT longer to get Exception from failure case... interesting.....
val futureValue = future.value
assertResult(true) {
futureValue match {
case Some(Success(Failure(error: IllegalStateException))) => true
}
}
}
test("Future.allAsTrys returns Nil given Nil list as input") {
val future = Future.allAsTrys(Nil)
assert ( Await.result(future, 100 nanosecond).isEmpty )
}
test("Future.allAsTrys returns successful item even if preceded by failing item") {
val future1 = Future{throw new IllegalStateException("bad news")}
var future2 = Future{"dog"}
val futureListOfTrys = Future.allAsTrys(List(future1,future2))
val listOfTrys = Await.result(futureListOfTrys, 10 milli)
System.out.println("successItem:" + listOfTrys);
assert(listOfTrys(0).failed.get.getMessage.contains("bad news"))
assert(listOfTrys(1) == Success("dog"))
}
test("Future.allAsTrys returns successful item even if followed by failing item") {
var future1 = Future{"dog"}
val future2 = Future{throw new IllegalStateException("bad news")}
val futureListOfTrys = Future.allAsTrys(List(future1,future2))
val listOfTrys = Await.result(futureListOfTrys, 10 milli)
System.out.println("successItem:" + listOfTrys);
assert(listOfTrys(1).failed.get.getMessage.contains("bad news"))
assert(listOfTrys(0) == Success("dog"))
}
test("Future.allFailedAsTrys returns the failed item and only that item") {
var future1 = Future{"dog"}
val future2 = Future{throw new IllegalStateException("bad news")}
val futureListOfTrys = Future.allFailedAsTrys(List(future1,future2))
val listOfTrys = Await.result(futureListOfTrys, 10 milli)
assert(listOfTrys(0).failed.get.getMessage.contains("bad news"))
assert(listOfTrys.size == 1)
}
test("Future.allSucceededAsTrys returns the succeeded item and only that item") {
var future1 = Future{"dog"}
val future2 = Future{throw new IllegalStateException("bad news")}
val futureListOfTrys = Future.allSucceededAsTrys(List(future1,future2))
val listOfTrys = Await.result(futureListOfTrys, 10 milli)
assert(listOfTrys(0) == Success("dog"))
assert(listOfTrys.size == 1)
}
I just came across this question and have another solution to offer:
def allSuccessful[A, M[X] <: TraversableOnce[X]](in: M[Future[A]])
(implicit cbf: CanBuildFrom[M[Future[A]], A, M[A]],
executor: ExecutionContext): Future[M[A]] = {
in.foldLeft(Future.successful(cbf(in))) {
(fr, fa) ⇒ (for (r ← fr; a ← fa) yield r += a) fallbackTo fr
} map (_.result())
}
The idea here is that within the fold you are waiting for the next element in the list to complete (using the for-comprehension syntax) and if the next one fails you just fallback to what you already have.
You can easily wraps future result with option and then flatten the list:
def futureToFutureOption[T](f: Future[T]): Future[Option[T]] =
f.map(Some(_)).recover {
case e => None
}
val listOfFutureOptions = listOfFutures.map(futureToFutureOption(_))
val futureListOfOptions = Future.sequence(listOfFutureOptions)
val futureListOfSuccesses = futureListOfOptions.flatten
You can also collect successful and unsuccessful results in different lists:
def safeSequence[A](futures: List[Future[A]]): Future[(List[Throwable], List[A])] = {
futures.foldLeft(Future.successful((List.empty[Throwable], List.empty[A]))) { (flist, future) =>
flist.flatMap { case (elist, alist) =>
future
.map { success => (elist, alist :+ success) }
.recover { case error: Throwable => (elist :+ error, alist) }
}
}
}
If you need to keep failed futures for some reason, e.g., logging or conditional processing, this works with Scala 2.12+. You can find working code here.
val f1 = Future(1)
val f2 = Future(2)
val ff = Future.failed(new Exception())
val futures: Seq[Future[Either[Throwable, Int]]] =
Seq(f1, f2, ff).map(_.transform(f => Success(f.toEither)))
val sum = Future
.sequence(futures)
.map { eithers =>
val (failures, successes) = eithers.partitionMap(identity)
val fsum = failures.map(_ => 100).sum
val ssum = successes.sum
fsum + ssum
}
assert(Await.result(sum, 1.second) == 103)
Is there an alternative pattern for having success and failure closures in Scala?
There's nothing wrong with this convention that's similar to what node.js libraries normally do, but I'm just wondering if there's another way to do this in Scala.
for example:
def performAsyncAction(n: BigInt,
success: (BigInt) => Unit,
failure: FunctionTypes.Failure): Unit = {
Then the call to the function
performAsyncAction(10,
{(x: BigInt) =>
/* Code... */
},
{(t: Throwable) =>
e.printStackTrace()
})
Thanks
It sounds like you want a Future. See the AKKA implementation here.
A Future is a functional construct that lets you specify a block of code to be executed asynchronously and then you can grab the result once it's completed:
import akka.actor.ActorSystem
import akka.dispatch.Await
import akka.dispatch.Future
import akka.util.duration._
implicit val system = ActorSystem("FutureSystem")
val future = Future {
1 + 1
}
val result = Await.result(future, 1 second)
println(result) // prints "2"
You can specify on-failure behavior with the onFailure method (there's also onComplete and onSuccess):
val future = Future {
throw new RuntimeException("error")
}.onFailure {
case e: RuntimeException => println("Oops! We failed with " + e)
}
// will print "Oops! We failed with java.lang.RuntimeException: error"
But the best part is that Futures are Monads, so you can create pipelines of asynchronous actions using things like map and flatMap:
val f1 = Future { "hello" }
val f2 = f1.map(_ + " world")
val f3 = f2.map(_.length)
val result = Await.result(f3, 1 second)
println(result) // prints "11"
Or use them in for-comprehensions:
val f1 = Future { "hello" }
val f2 = Future { " " }
val f3 = Future { "world" }
val f4 =
for (
a <- f1;
b <- f2;
c <- f3
) yield {
a + b + c
}
val result = Await.result(f4, 1 second)
println(result) // prints "hello world"