Akka: how to manage a double http request? - scala

In my actor I have two requests at different addresses like this:
http.singleRequest(HttpRequest(uri = encodedUri).addHeader(Accept(MediaTypes.`application/json`)))
I need that these two requests return both a value. As normal futures I would expect something like this:
val response: Future[SomeData] = for {
r1 <- firstRequest
r2 <- secondRequest
} yield {
// merge the results of these two responses
}
response onComplete {
case Success(body) => sndr ! Something(body)
case Failure(message) => BadRequest(message.toString)
}
In this part of the documentation:
http://doc.akka.io/docs/akka/2.4/scala/http/client-side/request-level.html
It is suggested to use pipeTo to self to manage the single request, instead of using native onComplete/map/etc.
How can I apply that for multiple requests like in my case, where I need to wait for 2 or more to be completed?

Simple and straight foward
val f1 = Future { //request1 }
val f2 = Future { //request2 }
val resultF = f1 zip f2
resultF pipeTo self
Current actor will get the result as a message and message will be a tuple (f1Result, f2Result)
if result resultF fails current actor gets failure message wrapped in akka.actor.Status.Failure
In the method f1 and f2 are independent futures
In case f2 depends on the value of f1 use flatMap
val resultF = f1.flatMap { f1Result => createF2(f1Result) }
//alternatively we can use for comprehension
resultF pipeTo self
Example
import akka.actor.Actor
import akka.actor.Status.Failure
import scala.concurrent.Future
import akka.pattern.pipe
object ManagerActor {
case object Exec
}
class ManagerActor extends Actor {
import ManagerActor._
implicit val dispather = context.dispatcher
override def receive: Receive = {
case Exec =>
val f1 = Future { 1 }
val f2 = Future { 2 }
val resultF = f1 zip f2
resultF pipeTo self
case result: (Int, Int) => //be careful about type erasure
println(s"""result:$result""")
case Failure(th) =>
println(s"""msg: ${th.getMessage}""")
}
}
to run
object Main {
def main(args: Array[String]): Unit = {
val system = ActorSystem()
val actor = system.actorOf(Props[ManagerActor])
actor ! ManagerActor.Exec
Thread.sleep(1000000)
}
}
We can use Future.sequence instead to zip to generalise this to any number of http requests.

Related

In akka-stream how to create a unordered Source from a futures collection

I need to create an akka.stream.scaladsl.Source[T, Unit] from a collection of Future[T].
E.g., having a collection of futures returning integers,
val f1: Future[Int] = ???
val f2: Future[Int] = ???
val fN: Future[Int] = ???
val futures = List(f1, f2, fN)
how to create a
val source: Source[Int, Unit] = ???
from it.
I cannot use Future.sequence combinator, since then I would wait for each future to complete before getting anything from the source. I want to get results in any order as soon as any future completes.
I understand that Source is a purely functional API and it should not run anything before somehow materializing it. So, my idea is to use an Iterator (which is lazy) to create a source:
Source { () =>
new Iterator[Future[Int]] {
override def hasNext: Boolean = ???
override def next(): Future[Int] = ???
}
}
But that would be a source of futures, not of actual values. I could also block on next using Await.result(future) but I'm not sure which tread pool's thread will be blocked. Also this will call futures sequentially, while I need parallel execution.
UPDATE 2: it turned out there was a much easier way to do it (thanks to Viktor Klang):
Source(futures).mapAsync(1)(identity)
UPDATE: here is what I've got based on #sschaef answer:
def futuresToSource[T](futures: Iterable[Future[T]])(implicit ec: ExecutionContext): Source[T, Unit] = {
def run(actor: ActorRef): Unit = {
futures.foreach { future =>
future.onComplete {
case Success(value) =>
actor ! value
case Failure(NonFatal(t)) =>
actor ! Status.Failure(t) // to signal error
}
}
Future.sequence(futures).onSuccess { case _ =>
actor ! Status.Success(()) // to signal stream's end
}
}
Source.actorRef[T](futures.size, OverflowStrategy.fail).mapMaterializedValue(run)
}
// ScalaTest tests follow
import scala.concurrent.ExecutionContext.Implicits.global
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
"futuresToSource" should "convert futures collection to akka-stream source" in {
val f1 = Future(1)
val f2 = Future(2)
val f3 = Future(3)
whenReady {
futuresToSource(List(f1, f2, f3)).runFold(Seq.empty[Int])(_ :+ _)
} { results =>
results should contain theSameElementsAs Seq(1, 2, 3)
}
}
it should "fail on future failure" in {
val f1 = Future(1)
val f2 = Future(2)
val f3 = Future.failed(new RuntimeException("future failed"))
whenReady {
futuresToSource(List(f1, f2, f3)).runWith(Sink.ignore).failed
} { t =>
t shouldBe a [RuntimeException]
t should have message "future failed"
}
}
Creating a source of Futures and then "flatten" it via mapAsync:
scala> Source(List(f1,f2,fN)).mapAsync(1)(identity)
res0: akka.stream.scaladsl.Source[Int,Unit] = akka.stream.scaladsl.Source#3e10d804
One of the easiest ways to feed a Source is through an Actor:
import scala.concurrent.Future
import akka.actor._
import akka.stream._
import akka.stream.scaladsl._
implicit val system = ActorSystem("MySystem")
def run(actor: ActorRef): Unit = {
import system.dispatcher
Future { Thread.sleep(100); actor ! 1 }
Future { Thread.sleep(200); actor ! 2 }
Future { Thread.sleep(300); actor ! 3 }
}
val source = Source
.actorRef[Int](0, OverflowStrategy.fail)
.mapMaterializedValue(ref ⇒ run(ref))
implicit val m = ActorMaterializer()
source runForeach { int ⇒
println(s"received: $int")
}
The Actor is created through the Source.actorRef method and made available through the mapMaterializedValue method. run simply takes the Actor and sends all the completed values to it, which can then be accessed through source. In the example above, the values are sent directly in the Future, but this can of course be done everywhere (for example in the onComplete call on the Future).

How to add elements to Source dynamically?

I have example code to generate an unbound source and working with it:
object Main {
def main(args : Array[String]): Unit = {
implicit val system = ActorSystem("Sys")
import system.dispatcher
implicit val materializer = ActorFlowMaterializer()
val source: Source[String] = Source(() => {
Iterator.continually({ "message:" + ThreadLocalRandom.current().nextInt(10000)})
})
source.runForeach((item:String) => { println(item) })
.onComplete{ _ => system.shutdown() }
}
}
I want to create class which implements:
trait MySources {
def addToSource(item: String)
def getSource() : Source[String]
}
And I need use it with multiple threads, for example:
class MyThread(mySources: MySources) extends Thread {
override def run(): Unit = {
for(i <- 1 to 1000000) { // here will be infinite loop
mySources.addToSource(i.toString)
}
}
}
And expected full code:
object Main {
def main(args : Array[String]): Unit = {
implicit val system = ActorSystem("Sys")
import system.dispatcher
implicit val materializer = ActorFlowMaterializer()
val sources = new MySourcesImplementation()
for(i <- 1 to 100) {
(new MyThread(sources)).start()
}
val source = sources.getSource()
source.runForeach((item:String) => { println(item) })
.onComplete{ _ => system.shutdown() }
}
}
How to implement MySources?
One way to have a non-finite source is to use a special kind of actor as the source, one that mixes in the ActorPublisher trait. If you create one of those kinds of actors, and then wrap with a call to ActorPublisher.apply, you end up with a Reactive Streams Publisher instance and with that, you can use an apply from Source to generate a Source from it. After that, you just need to make sure your ActorPublisher class properly handles the Reactive Streams protocol for sending elements downstream and you are good to go. A very trivial example is as follows:
import akka.actor._
import akka.stream.actor._
import akka.stream.ActorFlowMaterializer
import akka.stream.scaladsl._
object DynamicSourceExample extends App{
implicit val system = ActorSystem("test")
implicit val materializer = ActorFlowMaterializer()
val actorRef = system.actorOf(Props[ActorBasedSource])
val pub = ActorPublisher[Int](actorRef)
Source(pub).
map(_ * 2).
runWith(Sink.foreach(println))
for(i <- 1 until 20){
actorRef ! i.toString
Thread.sleep(1000)
}
}
class ActorBasedSource extends Actor with ActorPublisher[Int]{
import ActorPublisherMessage._
var items:List[Int] = List.empty
def receive = {
case s:String =>
if (totalDemand == 0)
items = items :+ s.toInt
else
onNext(s.toInt)
case Request(demand) =>
if (demand > items.size){
items foreach (onNext)
items = List.empty
}
else{
val (send, keep) = items.splitAt(demand.toInt)
items = keep
send foreach (onNext)
}
case other =>
println(s"got other $other")
}
}
With Akka Streams 2 you can use a sourceQueue : How to create a Source that can receive elements later via a method call?
As I mention in this answer, the SourceQueue is the way to go, and since Akka 2.5 there is a handy method preMaterialize which eliminates the need to create a composite source first.
I give an example in my other answer.

Return Future Message from an Actor

I have an existing API which returns a Future. Now introducing an Actor for one of the use cases and trying to continue using same service API from it. From below you can see MyService.saveValues return future.
object MyActor {
implicit val ec = scala.concurrent.ExecutionContext.Implicits.global
implicit val timeout = Timeout(1 second)
case class SampleMessage(list: List[String])
val actorSystem = //get actorSystem that was created at the startup
def saveMessage(list: List[String]) ={
val res = (actorSystem.actorOf(Props[MyActor]) ? SaveMyMessage(list) ).mapTo[Future[\/[Throwable,String]]
//res map{ r=>
//}
}
}
class MyActor extends Actor {
import MyActor._
def receive = {
case SaveMyMessage(list) =>
val originalSender = sender
val res : Future[\/[Throwable,String] ] = MyService.saveValues(list)
originalSender ! res
}
}
As you can see in def saveMessage I am using ask to wait for a result from an actor. However ask also creates its own future so result (val res) in saveMessage becomes Future[Future[T]] which looks annoying. What's the best way to deal with this scenario?
pipeTo forwards the result of a Future to an ActorRef.
import akka.pattern.pipe
val originalSender = sender
val res : Future[\/[Throwable,String] ] = MyService.saveValues(list)
res pipeTo originalSender
If saveValues throws, your future never complete and will end up timing something out though.
If you ever end up with Future[Future[A]], but want Future[A] as a result of sequencing/traversing or something, you can always "flatMap that shit."
import ExecutionContext.Implicits.global
val foo: Future[Future[Int]] = ???
val bar: Future[Int] = foo.flatMap(identity)
If you already depend on scalaz (and scalaz.contrib if on scalaz 7.0.x), you can use join defined in monad syntax.
import scalaz.syntax.monad._
import scalaz.contrib.std.scalaFuture._
val bar2: Future[Int] = foo.join

how can we get the result from various actors and then send the result from all the actors to one response in play 2.2.0?

case object getValues
case class Data(a: Any, b:Any)
class Testing extends Actor {
def receive = {
case "do something" => {
val result2 = { User.getList }
sender ! result2
}
case _ => println("done")
}
}
class Testing2 extends Actor {
def receive = {
case getValues =>
val result2 = { User.getMap }
}yield Data(a,b)
val child = context.actorOf(Props[Testing], "child")//creating actor
implicit val askTimeout = Timeout(1.second)//giving timeout
val r = child ? "do something"
val res = for {
a <- r
} yield (a)
res map (p=>sender ! Data(p, result2))//sending response to the sender
}
}
//Controller
object Application extends Controller {
def testingActor2 = Action.async {
val system = ActorSystem("ActorSystem")
val actor1 = system.actorOf(Props[Testing2], "Testing2")
implicit val askTimeout = Timeout(1.second)
val res = actor1 ? getValues
implicit val formats = DefaultFormats
val result = for {
r1 <- res
} yield r1
result map (r => { println(r); Ok("got result") })
}
i am not able to get the result of both actors to controller. Please tell me how to use actors if i have to take their result and then send with the response to the page .
val f1 = Future(1) //ask first actor
val f2 = Future(2) //ask second actor
Future.sequence(List(f1, f2)).map(list => {
//we got List[Int] here
list
})
One issue I can see with your code, and it might not be the entire problem, is that in Testing2, you are closing over the sender which is mutable and can lead to issues. The reason is that by the time the async map is run on the Future, the sender could have changed to a different ActorRef or could be nullified (switched to the dead letter actor ref) either way you won't get the response back to where you intended for it to go. You can fix that simply by doing the following:
...
val res = for {
a <- r
} yield (a)
val originator = sender
res foreach (p=>originator ! Data(p, result2))//sending response to the sender
Or you could import the pipeTo pattern and do it like this:
import akka.pattern.pipeTo
...
val res = for {
a <- r
} yield (a)
res map (p=> Data(p, result2)) pipeTo sender

How to wait for several Futures?

Suppose I have several futures and need to wait until either any of them fails or all of them succeed.
For example: Let there are 3 futures: f1, f2, f3.
If f1 succeeds and f2 fails I do not wait for f3 (and return failure to the client).
If f2 fails while f1 and f3 are still running I do not wait for them (and return failure)
If f1 succeeds and then f2 succeeds I continue waiting for f3.
How would you implement it?
You could use a for-comprehension as follows instead:
val fut1 = Future{...}
val fut2 = Future{...}
val fut3 = Future{...}
val aggFut = for{
f1Result <- fut1
f2Result <- fut2
f3Result <- fut3
} yield (f1Result, f2Result, f3Result)
In this example, futures 1, 2 and 3 are kicked off in parallel. Then, in the for comprehension, we wait until the results 1 and then 2 and then 3 are available. If either 1 or 2 fails, we will not wait for 3 anymore. If all 3 succeed, then the aggFut val will hold a tuple with 3 slots, corresponding to the results of the 3 futures.
Now if you need the behavior where you want to stop waiting if say fut2 fails first, things get a little trickier. In the above example, you would have to wait for fut1 to complete before realizing fut2 failed. To solve that, you could try something like this:
val fut1 = Future{Thread.sleep(3000);1}
val fut2 = Promise.failed(new RuntimeException("boo")).future
val fut3 = Future{Thread.sleep(1000);3}
def processFutures(futures:Map[Int,Future[Int]], values:List[Any], prom:Promise[List[Any]]):Future[List[Any]] = {
val fut = if (futures.size == 1) futures.head._2
else Future.firstCompletedOf(futures.values)
fut onComplete{
case Success(value) if (futures.size == 1)=>
prom.success(value :: values)
case Success(value) =>
processFutures(futures - value, value :: values, prom)
case Failure(ex) => prom.failure(ex)
}
prom.future
}
val aggFut = processFutures(Map(1 -> fut1, 2 -> fut2, 3 -> fut3), List(), Promise[List[Any]]())
aggFut onComplete{
case value => println(value)
}
Now this works correctly, but the issue comes from knowing which Future to remove from the Map when one has been successfully completed. As long as you have some way to properly correlate a result with the Future that spawned that result, then something like this works. It just recursively keeps removing completed Futures from the Map and then calling Future.firstCompletedOf on the remaining Futures until there are none left, collecting the results along the way. It's not pretty, but if you really need the behavior you are talking about, then this, or something similar could work.
You can use a promise, and send to it either the first failure, or the final completed aggregated success:
def sequenceOrBailOut[A, M[_] <: TraversableOnce[_]](in: M[Future[A]] with TraversableOnce[Future[A]])(implicit cbf: CanBuildFrom[M[Future[A]], A, M[A]], executor: ExecutionContext): Future[M[A]] = {
val p = Promise[M[A]]()
// the first Future to fail completes the promise
in.foreach(_.onFailure{case i => p.tryFailure(i)})
// if the whole sequence succeeds (i.e. no failures)
// then the promise is completed with the aggregated success
Future.sequence(in).foreach(p trySuccess _)
p.future
}
Then you can Await on that resulting Future if you want to block, or just map it into something else.
The difference with for comprehension is that here you get the error of the first to fail, whereas with for comprehension you get the first error in traversal order of the input collection (even if another one failed first). For example:
val f1 = Future { Thread.sleep(1000) ; 5 / 0 }
val f2 = Future { 5 }
val f3 = Future { None.get }
Future.sequence(List(f1,f2,f3)).onFailure{case i => println(i)}
// this waits one second, then prints "java.lang.ArithmeticException: / by zero"
// the first to fail in traversal order
And:
val f1 = Future { Thread.sleep(1000) ; 5 / 0 }
val f2 = Future { 5 }
val f3 = Future { None.get }
sequenceOrBailOut(List(f1,f2,f3)).onFailure{case i => println(i)}
// this immediately prints "java.util.NoSuchElementException: None.get"
// the 'actual' first to fail (usually...)
// and it returns early (it does not wait 1 sec)
Here is a solution without using actors.
import scala.util._
import scala.concurrent._
import java.util.concurrent.atomic.AtomicInteger
// Nondeterministic.
// If any failure, return it immediately, else return the final success.
def allSucceed[T](fs: Future[T]*): Future[T] = {
val remaining = new AtomicInteger(fs.length)
val p = promise[T]
fs foreach {
_ onComplete {
case s # Success(_) => {
if (remaining.decrementAndGet() == 0) {
// Arbitrarily return the final success
p tryComplete s
}
}
case f # Failure(_) => {
p tryComplete f
}
}
}
p.future
}
You can do this with futures alone. Here's one implementation. Note that it won't terminate execution early! In that case you need to do something more sophisticated (and probably implement the interruption yourself). But if you just don't want to keep waiting for something that isn't going to work, the key is to keep waiting for the first thing to finish, and stop when either nothing is left or you hit an exception:
import scala.annotation.tailrec
import scala.util.{Try, Success, Failure}
import scala.concurrent._
import scala.concurrent.duration.Duration
import ExecutionContext.Implicits.global
#tailrec def awaitSuccess[A](fs: Seq[Future[A]], done: Seq[A] = Seq()):
Either[Throwable, Seq[A]] = {
val first = Future.firstCompletedOf(fs)
Await.ready(first, Duration.Inf).value match {
case None => awaitSuccess(fs, done) // Shouldn't happen!
case Some(Failure(e)) => Left(e)
case Some(Success(_)) =>
val (complete, running) = fs.partition(_.isCompleted)
val answers = complete.flatMap(_.value)
answers.find(_.isFailure) match {
case Some(Failure(e)) => Left(e)
case _ =>
if (running.length > 0) awaitSuccess(running, answers.map(_.get) ++: done)
else Right( answers.map(_.get) ++: done )
}
}
}
Here's an example of it in action when everything works okay:
scala> awaitSuccess(Seq(Future{ println("Hi!") },
Future{ Thread.sleep(1000); println("Fancy meeting you here!") },
Future{ Thread.sleep(2000); println("Bye!") }
))
Hi!
Fancy meeting you here!
Bye!
res1: Either[Throwable,Seq[Unit]] = Right(List((), (), ()))
But when something goes wrong:
scala> awaitSuccess(Seq(Future{ println("Hi!") },
Future{ Thread.sleep(1000); throw new Exception("boo"); () },
Future{ Thread.sleep(2000); println("Bye!") }
))
Hi!
res2: Either[Throwable,Seq[Unit]] = Left(java.lang.Exception: boo)
scala> Bye!
For this purpose I would use an Akka actor. Unlike the for-comprehension, it fails as soon as any of the futures fail, so it's a bit more efficient in that sense.
class ResultCombiner(futs: Future[_]*) extends Actor {
var origSender: ActorRef = null
var futsRemaining: Set[Future[_]] = futs.toSet
override def receive = {
case () =>
origSender = sender
for(f <- futs)
f.onComplete(result => self ! if(result.isSuccess) f else false)
case false =>
origSender ! SomethingFailed
case f: Future[_] =>
futsRemaining -= f
if(futsRemaining.isEmpty) origSender ! EverythingSucceeded
}
}
sealed trait Result
case object SomethingFailed extends Result
case object EverythingSucceeded extends Result
Then, create the actor, send a message to it (so that it will know where to send its reply to) and wait for a reply.
val actor = actorSystem.actorOf(Props(new ResultCombiner(f1, f2, f3)))
try {
val f4: Future[Result] = actor ? ()
implicit val timeout = new Timeout(30 seconds) // or whatever
Await.result(f4, timeout.duration).asInstanceOf[Result] match {
case SomethingFailed => println("Oh noes!")
case EverythingSucceeded => println("It all worked!")
}
} finally {
// Avoid memory leaks: destroy the actor
actor ! PoisonPill
}
This question has been answered but I am posting my value class solution (value classes were added in 2.10) since there isn't one here. Please feel free to criticize.
implicit class Sugar_PimpMyFuture[T](val self: Future[T]) extends AnyVal {
def concurrently = ConcurrentFuture(self)
}
case class ConcurrentFuture[A](future: Future[A]) extends AnyVal {
def map[B](f: Future[A] => Future[B]) : ConcurrentFuture[B] = ConcurrentFuture(f(future))
def flatMap[B](f: Future[A] => ConcurrentFuture[B]) : ConcurrentFuture[B] = concurrentFutureFlatMap(this, f) // work around no nested class in value class
}
def concurrentFutureFlatMap[A,B](outer: ConcurrentFuture[A], f: Future[A] => ConcurrentFuture[B]) : ConcurrentFuture[B] = {
val p = Promise[B]()
val inner = f(outer.future)
inner.future onFailure { case t => p.tryFailure(t) }
outer.future onFailure { case t => p.tryFailure(t) }
inner.future onSuccess { case b => p.trySuccess(b) }
ConcurrentFuture(p.future)
}
ConcurrentFuture is a no overhead Future wrapper that changes the default Future map/flatMap from do-this-then-that to combine-all-and-fail-if-any-fail. Usage:
def func1 : Future[Int] = Future { println("f1!");throw new RuntimeException; 1 }
def func2 : Future[String] = Future { Thread.sleep(2000);println("f2!");"f2" }
def func3 : Future[Double] = Future { Thread.sleep(2000);println("f3!");42.0 }
val f : Future[(Int,String,Double)] = {
for {
f1 <- func1.concurrently
f2 <- func2.concurrently
f3 <- func3.concurrently
} yield for {
v1 <- f1
v2 <- f2
v3 <- f3
} yield (v1,v2,v3)
}.future
f.onFailure { case t => println("future failed $t") }
In the example above, f1,f2 and f3 will run concurrently and if any fail in any order the future of the tuple will fail immediately.
You might want to checkout Twitter's Future API. Notably the Future.collect method. It does exactly what you want: https://twitter.github.io/scala_school/finagle.html
The source code Future.scala is available here:
https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Future.scala
You can use this:
val l = List(1, 6, 8)
val f = l.map{
i => future {
println("future " +i)
Thread.sleep(i* 1000)
if (i == 12)
throw new Exception("6 is not legal.")
i
}
}
val f1 = Future.sequence(f)
f1 onSuccess{
case l => {
logInfo("onSuccess")
l.foreach(i => {
logInfo("h : " + i)
})
}
}
f1 onFailure{
case l => {
logInfo("onFailure")
}