This question already has answers here:
Flaky onSuccess of Future.sequence
(2 answers)
Closed 3 years ago.
I'm experimenting with futures. So i created a huge list of random numbers and then separated it in 3 groups to run them parallelly with some code
val itemsInGroup = 500000
val numbers: List[Int] = 1.to(1500000).map(v => Random.nextInt(20)).toList
val groups: List[List[Int]] = numbers.grouped(itemsInGroup).toList.take(3)
val future = Future.sequence(groups.map(gr => Future[Int] {countSum(gr)}))
future andThen {
case Success(threeNumbers) => println(threeNumbers)
}
what makes countSum is not too much important, just to take time i use this code
case class Person(name: String, age: Int) {
def age10: Int = age - age % 10
}
def countSum(lst: List[Int]): Int = lst.map(v => Person("John", v).age10).sum
as a result of a future i print list of 3 numbers. The problem is it doesnt work every time. Sometimes andThen works sometimes not, and if i change itemsInGroup value to small amounts, it works more frequently than big amounts of elements. So i suspect that there's a kind of implicit timeout or something otherwise i can't explain such phenomen.
Please, your tips appreciated
UPD Actually so much code were not needed, even simple example
val ft = Future {
Thread.sleep(10)
10
}
ft andThen {
case Success(value) => println("Here i work")
}
works the same way - sometimes works sometimes not, the more delay the less possibility that it will complete
Your main app thread is probably being terminated before your future completes. Try the following code.
import scala.concurrent.{Await, Future}
import scala.concurrent.duration.Duration
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Success, Failure}
val future = Future {
Thread.sleep(10)
10
}
val result = future andThen {
case Success(value) => println("Here i work")
case Failure(ex) => println(s"Error: ${ex.getMessage}")
}
println(Await.result(result, Duration.Inf)) // Only if you are sure it will ever finish, still it is recommended to use an appropriate timeout.
Related
I have a situation where I need to run a bunch of operations in parallel.
All operations have the same return value (say a Seq[String]).
Its possible that some of the operations may fail, and others successfully return results.
I want to return both the successful results, and any exceptions that happened, so I can log them for debugging.
Is there a built-in way, or easy way through any library (cats/scalaz) to do this, before I go and write my own class for doing this?
I was thinking of doing each operation in its own future, then checking each future, and returning a tuple of Seq[String] -> Seq[Throwable] where left value is the successful results (flattened / combined) and right is a list of any exceptions that occurred.
Is there a better way?
Using Await.ready, which you mention in a comment, generally loses most benefits from using futures. Instead you can do this just using the normal Future combinators. And let's do the more generic version, which works for any return type; flattening the Seq[String]s can be added easily.
def successesAndFailures[T](futures: Seq[Future[T]]): Future[(Seq[T], Seq[Throwable])] = {
// first, promote all futures to Either without failures
val eitherFutures: Seq[Future[Either[Throwable, T]]] =
futures.map(_.transform(x => Success(x.toEither)))
// then sequence to flip Future and Seq
val futureEithers: Future[Seq[Either[Throwable, T]]] =
Future.sequence(eitherFutures)
// finally, Seq of Eithers can be separated into Seqs of Lefts and Rights
futureEithers.map { seqOfEithers =>
val (lefts, rights) = seqOfEithers.partition(_.isLeft)
val failures = lefts.map(_.left.get)
val successes = rights.map(_.right.get)
(successes, failures)
}
}
Scalaz and Cats have separate to simplify the last step.
The types can be inferred by the compiler, they are shown just to help you see the logic.
Calling value on your Future returns an Option[Try[T]]. If the Future has not completed then the Option is None. If it has completed then it's easy to unwrap and process.
if (myFutr.isCompleted)
myFutr.value.map(_.fold( err: Throwable => //log the error
, ss: Seq[String] => //process results
))
else
// do something else, come back later
Sounds like a good use-case for the Try idiom (it's basically similar to the Either monad).
Example of usage from the doc:
import scala.util.{Success, Failure}
val f: Future[List[String]] = Future {
session.getRecentPosts
}
f onComplete {
case Success(posts) => for (post <- posts) println(post)
case Failure(t) => println("An error has occurred: " + t.getMessage)
}
It actually does a little bit more than what you asked because it is fully asynchronous. Does it fit your use-case?
I'd do it this way:
import scala.concurrent.{Future, ExecutionContext}
import scala.util.Success
def eitherify[A](f: Future[A])(implicit ec: ExecutionContext): Future[Either[Throwable, A]] = f.transform(tryResult => Success(tryResult.toEither))
def eitherifyF[A, B](f: A => Future[B])(implicit ec: ExecutionContext): A => Future[Either[Throwable, B]] = { a => eitherify(f(a)) }
// here we need some "cats" magic for `traverse` and `separate`
// instead of `traverse` you can use standard `Future.sequence`
// there is no analogue for `separate` in the standard library
import cats.implicits._
def myProgram[A, B](values: List[A], asyncF: A => Future[B])(implicit ec: ExecutionContext): Future[(List[Throwable], List[B])] = {
val appliedTransformations: Future[List[Either[Throwable, B]]] = values.traverse(eitherifyF(asyncF))
appliedTransformations.map(_.separate)
}
This question already has answers here:
Wait until all Future.onComplete callbacks are executed
(2 answers)
Closed 5 years ago.
I am trying to understand higher order functions in scala futures. I wrote this piece of code
import scala.concurrent.Future
import scala.util.{Failure, Success}
import scala.concurrent.ExecutionContext.Implicits.global
object App30 extends App {
def getMilk(): String = {
val ans = "5 gallons of milk"
ans
}
def getFlour(): String =
{
val ans= "2 oz of flour"
ans
}
val milkFuture = Future { getMilk() }
val flourFuture = Future { getFlour() }
val resultFut = {
for {
milk <- milkFuture
flour <- flourFuture
result = milk + flour
} yield result
}
resultFut.onComplete{
case Success(answer) => println("The result of getting ingridents is " + answer)
case Failure(exception) => println("could not access future value")
}
}
My issue is that I am not able to get the required string 'the result of getting ...' . I see that the debugger stops with onComplete and does not proceed to successful case.
May I know where I am going wrong? If this was a silly mistake, kindly point it out and I will take down the question after that. Else, provide with an explanation on what concept is amiss as this will help newcomers.
Thanks
You have to explicitly block on the future, otherwise the main thread might terminate before its completion:
import scala.concurrent.{Await, Future}
import scala.concurrent.duration._
Await.result(resultFut, 5 seconds)
Note such methods should definitely not be used with production code, it is merely for testing the output of the Future. Usually, you'll have some event loop running endlessly in the background where the future will have a chance to complete, unless something bad happens.
I am a newbie to scala futures and I have a doubt regarding the return value of scala futures.
So, generally syntax for a scala future is
def downloadPage(url: URL) = Future[List[Int]] {
}
I want to know how to access the List[Int] from some other method which calls this method.
In other words,
val result = downloadPage("localhost")
then what should be the approach to get List[Int] out of the future ?
I have tried using map method but not able to do this successfully.`
The case of Success(listInt) => I want to return the listInt and I am not able to figure out how to do that.
The best practice is that you don't return the value. Instead you just pass the future (or a version transformed with map, flatMap, etc.) to everyone who needs this value and they can add their own onComplete.
If you really need to return it (e.g. when implementing a legacy method), then the only thing you can do is to block (e.g. with Await.result) and you need to decide how long to await.
You need to wait for the future to complete to get the result given some timespan, here's something that would work:
import scala.concurrent.duration._
def downloadPage(url: URL) = Future[List[Int]] {
List(1,2,3)
}
val result = downloadPage("localhost")
val myListInt = result.result(10 seconds)
Ideally, if you're using a Future, you don't want to block the executing thread, so you would move your logic that deals with the result of your Future into the onComplete method, something like this:
result.onComplete({
case Success(listInt) => {
//Do something with my list
}
case Failure(exception) => {
//Do something with my error
}
})
I hope you already solved this since it was asked in 2013 but maybe my answer can help someone else:
If you are using Play Framework, it support async Actions (actually all Actions are async inside). An easy way to create an async Action is using Action.async(). You need to provide a Future[Result]to this function.
Now you can just make transformations from your Future[List[Int]] to Future[Result] using Scala's map, flatMap, for-comprehension or async/await. Here an example from Play Framework documentation.
import play.api.libs.concurrent.Execution.Implicits.defaultContext
def index = Action.async {
val futureInt = scala.concurrent.Future { intensiveComputation() }
futureInt.map(i => Ok("Got result: " + i))
}
You can do something like that. If The wait time that is given in Await.result method is less than it takes the awaitable to execute, you will have a TimeoutException, and you need to handle the error (or any other error).
import scala.concurrent._
import ExecutionContext.Implicits.global
import scala.util.{Try, Success, Failure}
import scala.concurrent.duration._
object MyObject {
def main(args: Array[String]) {
val myVal: Future[String] = Future { silly() }
// values less than 5 seconds will go to
// Failure case, because silly() will not be done yet
Try(Await.result(myVal, 10 seconds)) match {
case Success(extractedVal) => { println("Success Happened: " + extractedVal) }
case Failure(_) => { println("Failure Happened") }
case _ => { println("Very Strange") }
}
}
def silly(): String = {
Thread.sleep(5000)
"Hello from silly"
}
}
The best way I’ve found to think of a Future is a box that will, at some point, contain the thing that you want. The key thing with a Future is that you never open the box. Trying to force open the box will lead you to blocking and grief. Instead, you put the Future in another, larger box, typically using the map method.
Here’s an example of a Future that contains a String. When the Future completes, then Console.println is called:
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
object Main {
def main(args:Array[String]) : Unit = {
val stringFuture: Future[String] = Future.successful("hello world!")
stringFuture.map {
someString =>
// if you use .foreach you avoid creating an extra Future, but we are proving
// the concept here...
Console.println(someString)
}
}
}
Note that in this case, we’re calling the main method and then… finishing. The string’s Future, provided by the global ExecutionContext, does the work of calling Console.println. This is great, because when we give up control over when someString is going to be there and when Console.println is going to be called, we let the system manage itself. In constrast, look what happens when we try to force the box open:
val stringFuture: Future[String] = Future.successful("hello world!")
val someString = Future.await(stringFuture)
In this case, we have to wait — keep a thread twiddling its thumbs — until we get someString back. We’ve opened the box, but we’ve had to commandeer the system’s resources to get at it.
It wasn't yet mentioned, so I want to emphasize the point of using Future with for-comprehension and the difference of sequential and parallel execution.
For example, for sequential execution:
object FuturesSequential extends App {
def job(n: Int) = Future {
Thread.sleep(1000)
println(s"Job $n")
}
val f = for {
f1 <- job(1)
f2 <- job(2)
f3 <- job(3)
f4 <- job(4)
f5 <- job(5)
} yield List(f1, f2, f3, f4, f5)
f.map(res => println(s"Done. ${res.size} jobs run"))
Thread.sleep(6000) // We need to prevent main thread from quitting too early
}
And for parallel execution (note that the Future are before the for-comprehension):
object FuturesParallel extends App {
def job(n: Int) = Future {
Thread.sleep(1000)
println(s"Job $n")
}
val j1 = job(1)
val j2 = job(2)
val j3 = job(3)
val j4 = job(4)
val j5 = job(5)
val f = for {
f1 <- j1
f2 <- j2
f3 <- j3
f4 <- j4
f5 <- j5
} yield List(f1, f2, f3, f4, f5)
f.map(res => println(s"Done. ${res.size} jobs run"))
Thread.sleep(6000) // We need to prevent main thread from quitting too early
}
I have n different sources to, say, gets rates of USD to EUR. Let n = 3 and the sources be Google, Yahoo, MyRates with corresponding methods:
def getYahooRate:Double = ???
def getGoogleRate:Double = ???
def getMyRate:Double = ???
I want to query the rate of USD to EUR in such a way that all n sources are polled in parallel and the first response to be received is immediately returned. If none reply within a specified time-frame, then an exception is thrown.
What is the canonical way to implement this using Scala (and if necessary Akka)?
Is there any library method that does most of this?
EDIT: Here is what I have tried. Some comments on the code would be appreciated:
This is somewhat like a parallel version of trycatch from this SO question. The code for the below method is based on this SO answer
type unitToT[T] = ()=>T
def trycatchPar[B](list:List[unitToT[B]], timeOut:Long):B = {
if (list.isEmpty) throw new Exception("call list must be non-empty")
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent._
import scala.concurrent.duration._
import scala.util.Failure
import scala.util.Success
val p = promise[B]
val futures = list.map(l => Future{l()})
futures foreach {
_ onComplete {
case s # Success(_) => {
// Arbitrarily return the first success
p tryComplete s
}
case s # Failure(_) =>
}
}
Await.result(p.future, timeOut millis)
}
You can use Future.firstCompletedOf
val first = Future.firstCompletedOf(futures)
Await.result(first, timeOut.millis)
My code which uses mapAync(1) doesn't work what I want it to do. But when I changed the mapAsync(1) to map by using Await.result, it works. So I have a question.
Does the following (A) Use map and (B) use mapAsync(1) yield the same result at anytime?
// (A) Use map
someSource
.map{r =>
val future = makeFuture(r) // returns the same future if r is the same
Await.result(future, Duration.Inf)
}
// (B) Use mapAsync(1)
someSource
.mapAsync(1){r =>
val future = makeFuture(r) // returns the same future if r is the same
future
}
Actually, I want to paste my real code, but it is too long to paste and has some dependencies of my original stages.
While semantically the type of both streams ends up being the same (Source[Int, NotUsed]), the style displayed in example (A) is very very bad – please don't block (Await) inside streams.
Such cases are exactly the use case for mapAsync. Your operation returns a Future[T], and you want to push that value downwards through the stream once the future completes. Please note that there is no blocking in mapAsync, it schedules a callback to push the value of the future internally and does so once it completes.
To answer your question about "do they do the same thing?", technically yes but the first one will cause performance issues in the threadpool you're running on, avoid map+blocking when mapAsync can do the job.
These calls are semantically very similar, although blocking by using Await is probably not a good idea. The type signature of both these calls is, of course, the same (Source[Int, NotUsed]), and in many cases these calls will produce the same results (blocking aside). The following, for example, which includes scheduled futures and a non-default supervision strategy for failures, gives the same results for both map with an Await inside and mapAsync:
import akka.actor._
import akka.stream.ActorAttributes.supervisionStrategy
import akka.stream.Supervision.resumingDecider
import akka.stream._
import akka.stream.scaladsl._
import scala.concurrent._
import scala.concurrent.duration._
import scala.language.postfixOps
object Main {
def main(args: Array[String]) {
implicit val system = ActorSystem("TestSystem")
implicit val materializer = ActorMaterializer()
import scala.concurrent.ExecutionContext.Implicits.global
import system.scheduler
def makeFuture(r: Int) = {
akka.pattern.after(2 seconds, scheduler) {
if (r % 3 == 0)
Future.failed(new Exception(s"Failure for input $r"))
else
Future(r + 100)
}
}
val someSource = Source(1 to 20)
val mapped = someSource
.map { r =>
val future = makeFuture(r)
Await.result(future, Duration.Inf)
}.withAttributes(supervisionStrategy(resumingDecider))
val mappedAsync = someSource
.mapAsyncUnordered(1) { r =>
val future = makeFuture(r)
future
}.withAttributes(supervisionStrategy(resumingDecider))
mapped runForeach println
mappedAsync runForeach println
}
}
It is possible that your upstream code is relying on the blocking behaviour in your map call in some way. Can you produce a concise reproduction of the issue that you are seeing?