Scala Nested Futures - scala

I have a couple of futures. campaignFuture returns a List[BigInt] and I want to be able to call the second future profileFuture for each of the values in the list returned from the first one. The second future can only be called when the first one is complete. How do I achieve this in Scala?
campaignFuture(1923).flatMap?? (May be?)
def campaignFuture(advertiserId: Int): Future[List[BigInt]] = Future {
val campaignHttpResponse = getCampaigns(advertiserId.intValue())
parseProfileIds(campaignHttpResponse.entity.asString)
}
def profileFuture(profileId: Int): Future[List[String]] = Future {
val profileHttpResponse = getProfiles(profileId.intValue())
parseSegmentNames(profileHttpResponse.entity.asString)
}

A for comprehension is here not applicable because we have a mix of List's and Future's.
So, your friends are map and flatMap:
To react on Future result
import scala.concurrent.{Future, Promise, Await}
import scala.concurrent.duration.Duration
import scala.concurrent.ExecutionContext.Implicits.global
def campaignFuture(advertiserId: Int): Future[List[BigInt]] = Future {
List(1, 2, 3)
}
def profileFuture(profileId: Int): Future[List[String]] = {
// delayed Future
val p = Promise[List[String]]
Future {
val delay: Int = (math.random * 5).toInt
Thread.sleep(delay * 1000)
p.success(List(s"profile-for:$profileId", s"delayed:$delay sec"))
}
p.future
}
// Future[List[Future[List[String]]]
val listOfProfileFuturesFuture = campaignFuture(1).map { campaign =>
campaign.map(id => profileFuture(id.toInt))
}
// React on Futures which are done
listOfProfileFuturesFuture foreach { campaignFutureRes =>
campaignFutureRes.foreach { profileFutureRes =>
profileFutureRes.foreach(profileListEntry => println(s"${new Date} done: $profileListEntry"))
}
}
// !!ONLY FOR TESTING PURPOSE - THIS CODE BLOCKS AND EXITS THE VM WHEN THE FUTURES ARE DONE!!
println(s"${new Date} waiting for futures")
listOfProfileFuturesFuture.foreach{listOfFut =>
Await.ready(Future.sequence(listOfFut), Duration.Inf)
println(s"${new Date} all futures done")
System.exit(0)
}
scala.io.StdIn.readLine()
To get the result of all Futures at once
import scala.concurrent.{Future, Await}
import scala.concurrent.duration.Duration
import scala.concurrent.ExecutionContext.Implicits.global
def campaignFuture(advertiserId: Int): Future[List[BigInt]] = Future {
List(1, 2, 3)
}
def profileFuture(profileId: Int): Future[List[String]] = Future {
List(s"profile-for:$profileId")
}
// type: Future[List[Future[List[String]]]]
val listOfProfileFutures = campaignFuture(1).map { campaign =>
campaign.map(id => profileFuture(id.toInt))
}
// type: Future[List[List[String]]]
val listOfProfileFuture = listOfProfileFutures.flatMap(s => Future.sequence(s))
// print the result
//listOfProfileFuture.foreach(println)
//scala.io.StdIn.readLine()
// wait for the result (THIS BLOCKS INFINITY!)
Await.result(listOfProfileFuture, Duration.Inf)
we use Future.sequence to convert a List[Future[T]] to Future[List[T]].
flatMap to get a Future[T] from Future[Future[T]]
if you need wait for the result (BLOCKING!) use Await to wait for the result

Related

Does cats mapN run all the futures in parallel?

As mentioned in the jump-start guide, mapN will run all the futures in parallel, so I created the below simple Scala program, but a sample run shows diff to be 9187 ms and diffN to be 9106 ms. So it looks like that the mapN is also running the futures sequentially, isn't it? Please let me know if I am missing something?
package example
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
import java.time.LocalDateTime
import java.time.Duration
import scala.util.Failure
import scala.util.Success
import java.time.ZoneOffset
import cats.instances.future._
import cats.syntax.apply._
object FutureEx extends App {
val before = LocalDateTime.now()
val sum = for {
a <- getA
b <- getB
c <- getC
} yield (a + b + c)
sum onComplete {
case Failure(ex) => println(s"Error: ${ex.getMessage()}")
case Success(value) =>
val after = LocalDateTime.now()
println(s"before: $before")
println(s"after: $after")
val diff = getDiff(before, after)
println(s"diff: $diff")
println(s"sum: $value")
}
// let the above finish
Thread.sleep(20000)
val beforeN = LocalDateTime.now()
val usingMapN = (getA, getB, getC).mapN(add)
usingMapN onComplete {
case Failure(ex) => println(s"Error: ${ex.getMessage()}")
case Success(value) =>
val afterN = LocalDateTime.now()
println(s"beforeN: $beforeN")
println(s"afterN: $afterN")
val diff = getDiff(beforeN, afterN)
println(s"diffN: $diff")
println(s"sum: $value")
}
def getA: Future[Int] = {
println("inside A")
Thread.sleep(3000)
Future.successful(2)
}
def getB: Future[Int] = {
println("inside B")
Thread.sleep(3000)
Future.successful(3)
}
def getC: Future[Int] = {
println("inside C")
Thread.sleep(3000)
Future.successful(4)
}
def add(a: Int, b: Int, c: Int) = a + b + c
def getDiff(before: LocalDateTime, after: LocalDateTime): Long = {
Duration.between(before.toInstant(ZoneOffset.UTC), after.toInstant(ZoneOffset.UTC)).toMillis()
}
}
Because you have sleep outside Future it should be like:
def getA: Future[Int] = Future {
println("inside A")
Thread.sleep(3000)
2
}
So you start async Future with apply - Future.successful on the other hand returns pure value, meaning you execute sleep in same thread.
The time is going before mapN is ever called.
(getA, getB, getC).mapN(add)
This expression is creating a tuple (sequentially) and then calling mapN on it. So it is calling getA then getB then getC and since each of them has a 3 second delay, it takes 9 seconds.

How do I measure elapsed time inside Cats IO effect?

I'd like to measure elapsed time inside IO container. It's relatively easy to do with plain calls or futures (e.g. something like the code below)
class MonitoringComponentSpec extends FunSuite with Matchers with ScalaFutures {
import scala.concurrent.ExecutionContext.Implicits.global
def meter[T](future: Future[T]): Future[T] = {
val start = System.currentTimeMillis()
future.onComplete(_ => println(s"Elapsed ${System.currentTimeMillis() - start}ms"))
future
}
def call(): Future[String] = Future {
Thread.sleep(500)
"hello"
}
test("metered call") {
whenReady(meter(call()), timeout(Span(550, Milliseconds))) { s =>
s should be("hello")
}
}
}
But not sure how to wrap IO call
def io_meter[T](effect: IO[T]): IO[T] = {
val start = System.currentTimeMillis()
???
}
def io_call(): IO[String] = IO.pure {
Thread.sleep(500)
"hello"
}
test("metered io call") {
whenReady(meter(call()), timeout(Span(550, Milliseconds))) { s =>
s should be("hello")
}
}
Thank you!
Cats-effect has a Clock implementation that allows pure time measurement as well as injecting your own implementations for testing when you just want to simulate the passing of time. The example from their documentation is:
def measure[F[_], A](fa: F[A])
(implicit F: Sync[F], clock: Clock[F]): F[(A, Long)] = {
for {
start <- clock.monotonic(MILLISECONDS)
result <- fa
finish <- clock.monotonic(MILLISECONDS)
} yield (result, finish - start)
}
In cats effect 3, you can use .timed. Like,
import cats.effect.IO
import cats.effect.unsafe.implicits.global
import cats.implicits._
import scala.concurrent.duration._
val twoSecondsLater = IO.sleep(2.seconds) *> IO.println("Hi")
val elapsedTime = twoSecondsLater.timed.map(_._1)
elapsedTime.unsafeRunSync()
// would give something like this.
Hi
res0: FiniteDuration = 2006997667 nanoseconds

Handling dependency on Future completion in Scala Play (2.6) EssentialFilter

Suppose I have an EssentialFilter called AbcFilter whose purpose is to add a header to each request. However, to decide what it needs to add, it has to make a call to a library that returns a Future. The way I have currently implemented it makes use of Await, since I need the Future to complete so I know how to construct the header I need to add. Here is the implementation:
import play.api.libs.concurrent.Execution.Implicits.defaultContext
import play.api.mvc._
import scala.concurrent.duration.{Duration, _}
import scala.concurrent.{Await, Future}
class AbcFilter () extends EssentialFilter {
def apply(nextFilter: EssentialAction) = new EssentialAction {
def apply(requestHeader: RequestHeader) = {
// This call returns a future
val future: Future[String] = asyncLibrary.doSomething(requestHeader)
//Is there any way to avoid using Await here?
val futureResult = Await.result[String](future, Duration(5, SECONDS))
val newHeader = "New-Header-Name"-> futureResult
//I do need the future to complete in order to create this header!
val data = requestHeader.headers.toSimpleMap.toSeq :+ newHeader
val newHeaders = new Headers(_headers = data)
nextFilter(requestHeader.withHeaders(newHeaders))
}
}
}
Is there a way to implement this without the Await?
To avoid using Await, stay within the context of a Future:
class AbcFilter () extends EssentialFilter {
def apply(nextFilter: EssentialAction) = new EssentialAction {
def apply(requestHeader: RequestHeader) = {
val newHeaderFut: Future[(String, String)] = asyncLibrary.doSomething(requestHeader)
.map(s => "New-Header-Name" -> s)
val newHeaders: Future[Headers] = newHeaderFut.map(requestHeader.headers.add)
val accumulator: Accumulator[ByteString, Result] = nextFilter(requestHeader)
accumulator.mapFuture { result =>
newHeaders.map(h => result.withHeaders(h.headers: _*))
}
}
}
}
I was able to solve this by using Filter instead of EssentialFilter and using flatMap. The flatMap could be applied to the first future we get in the below code sample, but hopefully this provides an idea of the pattern.
class AbcFilter () extends Filter{
def apply(nextFilter: RequestHeader => Future[Result])
(requestHeader: RequestHeader): Future[Result] = {
val newHeaderFut: Future[(String, String)] = asyncLibrary.doSomething(requestHeader)
.map(s => "New-Header-Name" -> s)
val newHeaders: Future[Headers] = newHeaderFut.map(requestHeader.headers.add)
newHeaders.flatMap { h => { nextFilter(requestHeader.withHeaders(newHeaders))
}
}
}
}
} }

Sequence of two Future.sequence doesn't work

The objective of this code is to take a sequence of futures, process them with Future.sequence, generate another sequence of futures and process them again with another Future.sequence.
The problem is that it doesn't print anything. What's wrong with this code?
object TestFutures extends App {
def future (i:Int) = Future { i }
def future2 (i:Int) = Future { i * 20 }
val futureResult = (1 to 10).map {
x => future(x)
}
var futureInts = Seq[Future[Int]]()
Future.sequence(futureResult).map{ list =>
list.foreach( y => futureInts = futureInts :+ future2(y))
}
Future.sequence(futureInts).map { list2 =>
list2.foreach( z => println(z))
}
Thread.sleep(5000)
}
It does work, you just have a race condition between the Thread.sleep and the futures finishing execution. You can wait on the futures completion with Await.result:
import scala.concurrent.{Await, Future}
import scala.concurrent.ExecutionContext.Implicits._
import scala.concurrent.duration._
def future(i: Int) = Future.successful(i)
def future2(i: Int) = Future.successful(i * 20)
val futureResult = (1 to 10).map(x => future(x))
val firstResult =
Future
.sequence(futureResult)
.flatMap(list => Future.sequence(list.map(future2))
val sequence = Await.result(firstResult, 10 seconds)
sequence.foreach(println)
Note you should not be synchronously blocking on futures in production code, this is merely for the demonstration that the future does do what you want it to.

In akka-stream how to create a unordered Source from a futures collection

I need to create an akka.stream.scaladsl.Source[T, Unit] from a collection of Future[T].
E.g., having a collection of futures returning integers,
val f1: Future[Int] = ???
val f2: Future[Int] = ???
val fN: Future[Int] = ???
val futures = List(f1, f2, fN)
how to create a
val source: Source[Int, Unit] = ???
from it.
I cannot use Future.sequence combinator, since then I would wait for each future to complete before getting anything from the source. I want to get results in any order as soon as any future completes.
I understand that Source is a purely functional API and it should not run anything before somehow materializing it. So, my idea is to use an Iterator (which is lazy) to create a source:
Source { () =>
new Iterator[Future[Int]] {
override def hasNext: Boolean = ???
override def next(): Future[Int] = ???
}
}
But that would be a source of futures, not of actual values. I could also block on next using Await.result(future) but I'm not sure which tread pool's thread will be blocked. Also this will call futures sequentially, while I need parallel execution.
UPDATE 2: it turned out there was a much easier way to do it (thanks to Viktor Klang):
Source(futures).mapAsync(1)(identity)
UPDATE: here is what I've got based on #sschaef answer:
def futuresToSource[T](futures: Iterable[Future[T]])(implicit ec: ExecutionContext): Source[T, Unit] = {
def run(actor: ActorRef): Unit = {
futures.foreach { future =>
future.onComplete {
case Success(value) =>
actor ! value
case Failure(NonFatal(t)) =>
actor ! Status.Failure(t) // to signal error
}
}
Future.sequence(futures).onSuccess { case _ =>
actor ! Status.Success(()) // to signal stream's end
}
}
Source.actorRef[T](futures.size, OverflowStrategy.fail).mapMaterializedValue(run)
}
// ScalaTest tests follow
import scala.concurrent.ExecutionContext.Implicits.global
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
"futuresToSource" should "convert futures collection to akka-stream source" in {
val f1 = Future(1)
val f2 = Future(2)
val f3 = Future(3)
whenReady {
futuresToSource(List(f1, f2, f3)).runFold(Seq.empty[Int])(_ :+ _)
} { results =>
results should contain theSameElementsAs Seq(1, 2, 3)
}
}
it should "fail on future failure" in {
val f1 = Future(1)
val f2 = Future(2)
val f3 = Future.failed(new RuntimeException("future failed"))
whenReady {
futuresToSource(List(f1, f2, f3)).runWith(Sink.ignore).failed
} { t =>
t shouldBe a [RuntimeException]
t should have message "future failed"
}
}
Creating a source of Futures and then "flatten" it via mapAsync:
scala> Source(List(f1,f2,fN)).mapAsync(1)(identity)
res0: akka.stream.scaladsl.Source[Int,Unit] = akka.stream.scaladsl.Source#3e10d804
One of the easiest ways to feed a Source is through an Actor:
import scala.concurrent.Future
import akka.actor._
import akka.stream._
import akka.stream.scaladsl._
implicit val system = ActorSystem("MySystem")
def run(actor: ActorRef): Unit = {
import system.dispatcher
Future { Thread.sleep(100); actor ! 1 }
Future { Thread.sleep(200); actor ! 2 }
Future { Thread.sleep(300); actor ! 3 }
}
val source = Source
.actorRef[Int](0, OverflowStrategy.fail)
.mapMaterializedValue(ref ⇒ run(ref))
implicit val m = ActorMaterializer()
source runForeach { int ⇒
println(s"received: $int")
}
The Actor is created through the Source.actorRef method and made available through the mapMaterializedValue method. run simply takes the Actor and sends all the completed values to it, which can then be accessed through source. In the example above, the values are sent directly in the Future, but this can of course be done everywhere (for example in the onComplete call on the Future).