My code which uses mapAync(1) doesn't work what I want it to do. But when I changed the mapAsync(1) to map by using Await.result, it works. So I have a question.
Does the following (A) Use map and (B) use mapAsync(1) yield the same result at anytime?
// (A) Use map
someSource
.map{r =>
val future = makeFuture(r) // returns the same future if r is the same
Await.result(future, Duration.Inf)
}
// (B) Use mapAsync(1)
someSource
.mapAsync(1){r =>
val future = makeFuture(r) // returns the same future if r is the same
future
}
Actually, I want to paste my real code, but it is too long to paste and has some dependencies of my original stages.
While semantically the type of both streams ends up being the same (Source[Int, NotUsed]), the style displayed in example (A) is very very bad – please don't block (Await) inside streams.
Such cases are exactly the use case for mapAsync. Your operation returns a Future[T], and you want to push that value downwards through the stream once the future completes. Please note that there is no blocking in mapAsync, it schedules a callback to push the value of the future internally and does so once it completes.
To answer your question about "do they do the same thing?", technically yes but the first one will cause performance issues in the threadpool you're running on, avoid map+blocking when mapAsync can do the job.
These calls are semantically very similar, although blocking by using Await is probably not a good idea. The type signature of both these calls is, of course, the same (Source[Int, NotUsed]), and in many cases these calls will produce the same results (blocking aside). The following, for example, which includes scheduled futures and a non-default supervision strategy for failures, gives the same results for both map with an Await inside and mapAsync:
import akka.actor._
import akka.stream.ActorAttributes.supervisionStrategy
import akka.stream.Supervision.resumingDecider
import akka.stream._
import akka.stream.scaladsl._
import scala.concurrent._
import scala.concurrent.duration._
import scala.language.postfixOps
object Main {
def main(args: Array[String]) {
implicit val system = ActorSystem("TestSystem")
implicit val materializer = ActorMaterializer()
import scala.concurrent.ExecutionContext.Implicits.global
import system.scheduler
def makeFuture(r: Int) = {
akka.pattern.after(2 seconds, scheduler) {
if (r % 3 == 0)
Future.failed(new Exception(s"Failure for input $r"))
else
Future(r + 100)
}
}
val someSource = Source(1 to 20)
val mapped = someSource
.map { r =>
val future = makeFuture(r)
Await.result(future, Duration.Inf)
}.withAttributes(supervisionStrategy(resumingDecider))
val mappedAsync = someSource
.mapAsyncUnordered(1) { r =>
val future = makeFuture(r)
future
}.withAttributes(supervisionStrategy(resumingDecider))
mapped runForeach println
mappedAsync runForeach println
}
}
It is possible that your upstream code is relying on the blocking behaviour in your map call in some way. Can you produce a concise reproduction of the issue that you are seeing?
Related
I have a small test of fs2 streams, process elements, waiting and then write them to a file.
I'm getting a type error saying and I could not figure out what it does mean:
Error : required: fs2.Stream[[x]cats.effect.IO[x],Unit] => fs2.Stream[[+A]cats.effect.IO[A],Unit],
found : [F[_]]fs2.Pipe[F,Byte,Unit]
import java.nio.file.Paths
import cats.effect.{Blocker, ExitCode, IO, IOApp, Timer}
import fs2.Stream
import fs2.io
import fs2.concurrent.Queue
import scala.concurrent.duration._
import scala.util.Random
class StreamTypeIntToDouble(q: Queue[IO, Int])(implicit timer: Timer[IO]) {
import core.Processing._
val blocker: Blocker =
Blocker.liftExecutionContext(
scala.concurrent.ExecutionContext.Implicits.global
)
def storeInQueue: Stream[IO, Unit] = {
Stream(1, 2, 3)
.covary[IO]
.evalTap(n => IO.delay(println(s"Pushing $n to Queue")))
.metered(Random.between(1, 20).seconds)
.through(q.enqueue)
}
def getFromQueue: Stream[IO, Unit] = {
q.dequeue
.evalMap(n => IO.delay(println(s"Pulling from queue $n")))
.through(
io.file
.writeAll(Paths.get("file.txt"), blocker)
)
}
}
object Five extends IOApp {
override def run(args: List[String]): IO[ExitCode] = {
val program = for {
q <- Queue.bounded[IO, Int](10)
b = new StreamTypeIntToDouble(q)
_ <- b.storeInQueue.compile.drain.start
_ <- b.getFromQueue.compile.drain
} yield ()
program.as(ExitCode.Success)
}
}
There are a couple of issues here, and the first is the most confusing. writeAll is polymorphic in its context F[_], but it requires a ContextShift instance for F (as well as Sync). You don't currently have a ContextShift[IO] in scope, so the compiler won't infer that the F for writeAll should be IO. If you add something like this:
implicit val ioContextShift: ContextShift[IO] =
IO.contextShift(scala.concurrent.ExecutionContext.Implicits.global)
…then the compiler will infer IO as you'd expect.
My suggestion for cases like this is to skip the type inference. Writing it out with the type parameter is only marginally more verbose:
.through(
io.file
.writeAll[IO](Paths.get("file.txt"), blocker)
)
…and it means that you'll get useful error messages for things like missing type class instances.
Once you've fixed that issue there are a couple of others. The next is that using evalMap in this context means that you'll have a stream of () values. If you change it to evalTap, the logging side effects will still happen appropriately, but you won't lose the actual values of the stream you call it on.
The last issue is that writeAll requires a stream of bytes, while you've given it a stream of Ints. How you want to deal with that discrepancy depends on the intended semantics, but for the sake of example something like .map(_.toByte) would make it compile.
I am learning akka/scala and am trying to read only those Futures that succeeded from a Seq[Future[Int]] but cant get anything to work.
I simulated an array of 10 Future[Int] some of which fail depending on the value FailThreshold takes (all fail for 10 and none fail for 0).
I then try to read them into an ArrayBuffer (could not find a way to return immutable structure with the values).
Also, there isn't a filter on Success/Failure so had to run an onComplete on each future and update buffer as a side-effect.
Even when the FailThreshold=0 and the Seq has all Future set to Success, the array buffer is sometimes empty and different runs return array of different sizes.
I tried a few other suggestions from the web like using Future.sequence on the list but this throws exception if any of future variables fail.
import akka.actor._
import akka.pattern.ask
import scala.concurrent.{Await, Future, Promise}
import scala.concurrent.duration._
import scala.util.{Timeout, Failure, Success}
import concurrent.ExecutionContext.Implicits.global
case object AskNameMessage
implicit val timeout = Timeout(5, SECONDS)
val FailThreshold = 0
class HeyActor(num: Int) extends Actor {
def receive = {
case AskNameMessage => if (num<FailThreshold) {Thread.sleep(1000);sender ! num} else sender ! num
}
}
class FLPActor extends Actor {
def receive = {
case t: IndexedSeq[Future[Int]] => {
println(t)
val b = scala.collection.mutable.ArrayBuffer.empty[Int]
t.foldLeft( b ){ case (bf,ft) =>
ft.onComplete { case Success(v) => bf += ft.value.get.get }
bf
}
println(b)
}
}
}
val system = ActorSystem("AskTest")
val flm = (0 to 10).map( (n) => system.actorOf(Props(new HeyActor(n)), name="futureListMake"+(n)) )
val flp = system.actorOf(Props(new FLPActor), name="futureListProcessor")
// val delay = akka.pattern.after(500 millis, using=system.scheduler)(Future.failed( throw new IllegalArgumentException("DONE!") ))
val delay = akka.pattern.after(500 millis, using=system.scheduler)(Future.successful(0))
val seqOfFtrs = (0 to 10).map( (n) => Future.firstCompletedOf( Seq(delay, flm(n) ? AskNameMessage) ).mapTo[Int] )
flp ! seqOfFtrs
The receive in FLPActor mostly gets
Vector(Future(Success(0)), Future(Success(1)), Future(Success(2)), Future(Success(3)), Future(Success(4)), Future(Success(5)), Future(Success(6)), Future(Success(7)), Future(Success(8)), Future(Success(9)), Future(Success(10)))
but the array buffer b has varying number of values and empty at times.
Can someone please point me to gaps here,
why would the array buffer have varying sizes even when all Future have resolved to Success,
what is the correct pattern to use when we want to ask different actors with TimeOut and use only those asks that have successfully returned for further processing.
Instead of directly sending the IndexedSeq[Future[Int]], you should transform to Future[IndexedSeq[Int]] and then pipe it to the next actor. You don't send the Futures directly to an actor. You have to pipe it.
HeyActor can stay unchanged.
After
val seqOfFtrs = (0 to 10).map( (n) => Future.firstCompletedOf( Seq(delay, flm(n) ? AskNameMessage) ).mapTo[Int] )
do a recover, and use Future.sequence to turn it into one Future:
val oneFut = Future.sequence(seqOfFtrs.map(f=>f.map(Some(_)).recover{ case (ex: Throwable) => None})).map(_.flatten)
If you don't understand the business with Some, None, and flatten, then make sure you understand the Option type. One way to remove values from a sequence is to map values in the sequence to Option (either Some or None) and then to flatten the sequence. The None values are removed and the Some values are unwrapped.
After you have transformed your data into a single Future, pipe it over to FLPActor:
oneFut pipeTo flp
FLPActor should be rewritten with the following receive function:
def receive = {
case printme: IndexedSeq[Int] => println(printme)
}
In Akka, modifying some state in the main thread of your actor from a Future or the onComplete of a Future is a big no-no. In the worst case, it results in race conditions. Remember that each Future runs on its own thread, so running a Future inside an actor means you have concurrent work being done in different threads. Having the Future directly modify some state in your actor while the actor is also processing some state is a recipe for disaster. In Akka, you process all changes to state directly in the primary thread of execution of the main actor. If you have some work done in a Future and need to access that work from the main thread of an actor, you pipe it to that actor. The pipeTo pattern is functional, correct, and safe for accessing the finished computation of a Future.
To answer your question about why FLPActor is not printing out the IndexedSeq correctly: you are printing out the ArrayBuffer before your Futures have been completed. onComplete isn't the right idiom to use in this case, and you should avoid it in general as it isn't good functional style.
Don't forget the import akka.pattern.pipe for the pipeTo syntax.
I am a newbie to scala futures and I have a doubt regarding the return value of scala futures.
So, generally syntax for a scala future is
def downloadPage(url: URL) = Future[List[Int]] {
}
I want to know how to access the List[Int] from some other method which calls this method.
In other words,
val result = downloadPage("localhost")
then what should be the approach to get List[Int] out of the future ?
I have tried using map method but not able to do this successfully.`
The case of Success(listInt) => I want to return the listInt and I am not able to figure out how to do that.
The best practice is that you don't return the value. Instead you just pass the future (or a version transformed with map, flatMap, etc.) to everyone who needs this value and they can add their own onComplete.
If you really need to return it (e.g. when implementing a legacy method), then the only thing you can do is to block (e.g. with Await.result) and you need to decide how long to await.
You need to wait for the future to complete to get the result given some timespan, here's something that would work:
import scala.concurrent.duration._
def downloadPage(url: URL) = Future[List[Int]] {
List(1,2,3)
}
val result = downloadPage("localhost")
val myListInt = result.result(10 seconds)
Ideally, if you're using a Future, you don't want to block the executing thread, so you would move your logic that deals with the result of your Future into the onComplete method, something like this:
result.onComplete({
case Success(listInt) => {
//Do something with my list
}
case Failure(exception) => {
//Do something with my error
}
})
I hope you already solved this since it was asked in 2013 but maybe my answer can help someone else:
If you are using Play Framework, it support async Actions (actually all Actions are async inside). An easy way to create an async Action is using Action.async(). You need to provide a Future[Result]to this function.
Now you can just make transformations from your Future[List[Int]] to Future[Result] using Scala's map, flatMap, for-comprehension or async/await. Here an example from Play Framework documentation.
import play.api.libs.concurrent.Execution.Implicits.defaultContext
def index = Action.async {
val futureInt = scala.concurrent.Future { intensiveComputation() }
futureInt.map(i => Ok("Got result: " + i))
}
You can do something like that. If The wait time that is given in Await.result method is less than it takes the awaitable to execute, you will have a TimeoutException, and you need to handle the error (or any other error).
import scala.concurrent._
import ExecutionContext.Implicits.global
import scala.util.{Try, Success, Failure}
import scala.concurrent.duration._
object MyObject {
def main(args: Array[String]) {
val myVal: Future[String] = Future { silly() }
// values less than 5 seconds will go to
// Failure case, because silly() will not be done yet
Try(Await.result(myVal, 10 seconds)) match {
case Success(extractedVal) => { println("Success Happened: " + extractedVal) }
case Failure(_) => { println("Failure Happened") }
case _ => { println("Very Strange") }
}
}
def silly(): String = {
Thread.sleep(5000)
"Hello from silly"
}
}
The best way I’ve found to think of a Future is a box that will, at some point, contain the thing that you want. The key thing with a Future is that you never open the box. Trying to force open the box will lead you to blocking and grief. Instead, you put the Future in another, larger box, typically using the map method.
Here’s an example of a Future that contains a String. When the Future completes, then Console.println is called:
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
object Main {
def main(args:Array[String]) : Unit = {
val stringFuture: Future[String] = Future.successful("hello world!")
stringFuture.map {
someString =>
// if you use .foreach you avoid creating an extra Future, but we are proving
// the concept here...
Console.println(someString)
}
}
}
Note that in this case, we’re calling the main method and then… finishing. The string’s Future, provided by the global ExecutionContext, does the work of calling Console.println. This is great, because when we give up control over when someString is going to be there and when Console.println is going to be called, we let the system manage itself. In constrast, look what happens when we try to force the box open:
val stringFuture: Future[String] = Future.successful("hello world!")
val someString = Future.await(stringFuture)
In this case, we have to wait — keep a thread twiddling its thumbs — until we get someString back. We’ve opened the box, but we’ve had to commandeer the system’s resources to get at it.
It wasn't yet mentioned, so I want to emphasize the point of using Future with for-comprehension and the difference of sequential and parallel execution.
For example, for sequential execution:
object FuturesSequential extends App {
def job(n: Int) = Future {
Thread.sleep(1000)
println(s"Job $n")
}
val f = for {
f1 <- job(1)
f2 <- job(2)
f3 <- job(3)
f4 <- job(4)
f5 <- job(5)
} yield List(f1, f2, f3, f4, f5)
f.map(res => println(s"Done. ${res.size} jobs run"))
Thread.sleep(6000) // We need to prevent main thread from quitting too early
}
And for parallel execution (note that the Future are before the for-comprehension):
object FuturesParallel extends App {
def job(n: Int) = Future {
Thread.sleep(1000)
println(s"Job $n")
}
val j1 = job(1)
val j2 = job(2)
val j3 = job(3)
val j4 = job(4)
val j5 = job(5)
val f = for {
f1 <- j1
f2 <- j2
f3 <- j3
f4 <- j4
f5 <- j5
} yield List(f1, f2, f3, f4, f5)
f.map(res => println(s"Done. ${res.size} jobs run"))
Thread.sleep(6000) // We need to prevent main thread from quitting too early
}
I have n different sources to, say, gets rates of USD to EUR. Let n = 3 and the sources be Google, Yahoo, MyRates with corresponding methods:
def getYahooRate:Double = ???
def getGoogleRate:Double = ???
def getMyRate:Double = ???
I want to query the rate of USD to EUR in such a way that all n sources are polled in parallel and the first response to be received is immediately returned. If none reply within a specified time-frame, then an exception is thrown.
What is the canonical way to implement this using Scala (and if necessary Akka)?
Is there any library method that does most of this?
EDIT: Here is what I have tried. Some comments on the code would be appreciated:
This is somewhat like a parallel version of trycatch from this SO question. The code for the below method is based on this SO answer
type unitToT[T] = ()=>T
def trycatchPar[B](list:List[unitToT[B]], timeOut:Long):B = {
if (list.isEmpty) throw new Exception("call list must be non-empty")
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent._
import scala.concurrent.duration._
import scala.util.Failure
import scala.util.Success
val p = promise[B]
val futures = list.map(l => Future{l()})
futures foreach {
_ onComplete {
case s # Success(_) => {
// Arbitrarily return the first success
p tryComplete s
}
case s # Failure(_) =>
}
}
Await.result(p.future, timeOut millis)
}
You can use Future.firstCompletedOf
val first = Future.firstCompletedOf(futures)
Await.result(first, timeOut.millis)
I'm confused about how Scala Future works. Can someone explain what's wrong with code:
import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits.global
val numbers = List(1,2,3,4,5,6,7,8,9,10)
def showNumbers() = {
numbers
}
val futueNumber = future {
showNumbers.filter((a: Int) => a % 2 == 0)
}
futueNumber onSuccess {
case resultList => resultList
}
futueNumber
I expect to receive a result:
List(1,2,3,4,5,6,7,8,9,10)
But REPL outputs:
res1: scala.concurrent.Future[List[Int]] = scala.concurrent.impl.Promise$DefaultPromise#3644febc
Please, give me a short hint, how to fix this
The simple answer is that you need to await the future for you example to work:
val result = Await.result(futureNumber, 1.second)
However, the purpose of future's is not to block on asynchronous work, as Await does, but instead to use the result of the future when becomes available.
The key to using future is understanding that it does not actually hold the value of your computation, but rather is promise that at some point in the future, it will provide that value.
The usual use case is a request coming into a server that requires some work, which may take some time and itself wait on IO, like this:
def receiveMsg(msg: String, caller: Caller) = {
doSomeAsyncWork(msg).map(response => caller.send(response)
}
I.e. you are called with an object that has a send method that you want to call when doSomeAsyncWork is done, but you don't want to block the current thread until that future completes. Instead you map the future, which means that response => caller.send(response) will be called for you when doSomeAsyncWork is done.
futureNumber is of type Future[List[Int]], since you defined it as such in
val futureNumber = future {
showNumbers.filter((a: Int) => a % 2 == 0)
}
Therefore, when you print futureNumber, that is what gets printed. The variable does not get reassigned to the future value! If you want to print the value of the future, you need to do it inside the onSucess callback:
futureNumber onSuccess {
case resultList => println(resultList)
}
Alternatively, you can block waiting for the future with
import scala.concurrent.ExecutionContext.Implicits.global
import duration._
val theList = scala.concurrent.Await.result(futureNumber, 10 seconds)
println(theList)
Although this might not work if done in the REPL.
The whole point of the Future type is that it acts as a marker that alerts you to the presence of concurrency and should help you handle it better.