I have an akka Flow[I, O] that is out of my control because its coming in from some third-party code. I need to react to whenever an input element does not produce an output element (for example, because an exception was thrown in some part of the flow). For that, I need the input element that produced the failure. I do not find any API on the flow or similar that allows me to register a handler or react to it in any way. How can I do that?
You want to Resume rather than Stop when the akka streams flow throws an exception. After collecting all successful elements, you can Seq#diff to tell what elements are dropped due to exception thrown.
import scala.concurrent.ExecutionContext
import scala.util.{Failure, Success}
object Exception {
case class MyException(n: Int) extends RuntimeException
def main(args: Array[String]): Unit = {
implicit val system: ActorSystem = ActorSystem("Exception")
implicit val ec: ExecutionContext = system.dispatcher
val decider: Supervision.Decider = {
case _: MyException => Supervision.Resume
case _ => Supervision.Stop
}
val flow = Flow[Int]
.map(n =>
if (n % 2 == 1) throw MyException(n)
else n
)
val in = 1 to 10
val outFuture = Source(in)
.via(flow)
.withAttributes(ActorAttributes.supervisionStrategy(decider))
.runWith(Sink.seq)
outFuture.onComplete {
case Success(out) =>
println("dropped elements are " + (in.diff(out)))
case Failure(_) =>
println("unknown failure")
}
}
}
The console outputs are:
dropped elements are Vector(1, 3, 5, 7, 9)
Reference: How to get object that caused failure in Akka Streams?
Related
There are some suggestions on StackOverflow on how to handle a list of Futures but I want to try my own approach. But I am unable to compile the following code
I have a list of Futures.
I want to count how many of them passed or failed. I should get (2,1)
I store this in a tuple
The approach I want to take is go through each element of the list. The element of the list is Future[Int]. for each element, I call flatMap which calls the next cycle of recursion (I am assuming that if flatMap gets called then that particular future would have been successful so I increment pass count). Similarly, I want to call next cycle of recursion in recover and increment fail count but I am getting compilation error in it.
import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Failure, Success, Try}
import scala.concurrent.duration._
import scala.language.postfixOps
object ConcurrencyExample extends App {
type pass = Int
type fail = Int
val time = System.currentTimeMillis()
//use recursion to process each Future in the list
def segregate(l:List[Future[Int]]):Future[Tuple2[pass,fail]] = {
def go(l:List[Future[Int]],t:Tuple2[pass,fail]):Future[Tuple2[pass,fail]] = {
l match {
case Nil => Future{t}
//l is List of Future[Int]. flatMap each successful Future
//recover each failed Future
case l::ls => {
l flatMap (x => go(ls, (t._1 + 1, t._2)))
**l.recover({ case e => go(ls, (t._1 + 1, t._2))})**//I get error here
}
}
}
go(l,(0,0))
}
//hardcoded future
val futures2: List[Future[Int]] = List(Future {
1
}, Future {
2
}, Future {
throw new Exception("error")
})
val result = segregate(futures2)
result onComplete {
case Success(v) => println("pp:" + v)
case Failure(v) => println("fp:" + v)
}
Await.result(result,1000 millis)
}
#evan058 is correct about the signature of recover. But you can fix your program by changing recover to recoverWith.
recoverWith is to recover as flatMap is to map.
Here's the complete solution (with minor stylistic improvements):
import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Failure, Success, Try}
import scala.concurrent.duration._
import scala.language.postfixOps
object ConcurrencyExample extends App {
type pass = Int
type fail = Int
val time = System.currentTimeMillis()
//use recursion to process each Future in the list
def segregate[T](fs:List[Future[T]]):Future[(pass,fail)] = {
def go(fs:List[Future[T]],r:Future[(pass,fail)]):Future[(pass,fail)] = fs match {
case Nil => r
case l::ls =>
val fx = l.transform({_ => (1, 0)}, identity).recoverWith[(pass,fail)]({case _: Exception => Future(0, 1) })
for (x <- fx; t <- r; g <- go(ls, Future(t._1+x._1,t._2+x._2))) yield g
}
go(fs,Future((0,0)))
}
//hardcoded future
val futures2 = List(Future(1), Future(2), Future(throw new Exception("error")))
val result = segregate(futures2)
result onComplete {
case Success(v) => println(s"successes: ${v._1}, failures: ${v._2}")
case Failure(v) => v.printStackTrace()
}
Await.result(result,1000 millis)
}
If you look at the docs, the signature of recover is:
def recover[U >: T](pf: PartialFunction[Throwable, U])(implicit executor: ExecutionContext): Future[U]
You are calling recover on l which is a Future[Int] so recover is expecting a U >: Int.
However you are calling go again which has the return type Future[(pass, fail)] which is not >: Int.
Im very new to akka and futures in scala. Im using spray to get the output of a URL and return a Future[String] to another object. Here is the object that is making the HTTP request.
object ActionsService {
private implicit val formats = DefaultFormats
implicit val system = ActorSystem()
import system.dispatcher
val pipeline = sendReceive ~> unmarshal[String]
def getActions: Future[String] ={
val out = getOutput("http://www.google.com")
out
}
def getOutput(url: String): Future[String] ={
val response = pipeline (Get (url) )
response
}
def shutdown(code: Int): Unit = {
IO(Http).ask(Http.CloseAll)(1.second).await
system.shutdown()
}
}
And here is the main method in the other object where I am trying to shutdown the actor system.
import ExecutionContext.Implicits.global
def main(args: Array[String]) {
val test = ActionsService.getActions
test.onComplete {
case Success(x) => println(x)
case Failure(y) => println(y)
}
ActionsService.system.shutdown()
For some reason the akka system wont shutdown. Ive also tried to shut down the akka system using my shutdown method but I get an Exception in thread "main" akka.pattern.AskTimeoutException: Timed out error (which also occurs in the main method when the shutdown method is called).
akka version is 2.2.3
You didn't specify version of Akka you're using but in last version you need to call
system.terminate()
instead and wait for termination like this (ass an example)
Await.ready(system.whenTerminated, Duration.Inf)
The answer above did not end up working for me so I had to block the thread by adding a sleep period for the main method before the shutdown method is called.
def main(args: Array[String]) {
val test = ActionsService.getActions
Thread.sleep(700)
test.onComplete {
case Success(x) => println(x)
case Failure(y) => println(y)
}
CorporateActionsService.system.shutdown()
}
I need to create an akka.stream.scaladsl.Source[T, Unit] from a collection of Future[T].
E.g., having a collection of futures returning integers,
val f1: Future[Int] = ???
val f2: Future[Int] = ???
val fN: Future[Int] = ???
val futures = List(f1, f2, fN)
how to create a
val source: Source[Int, Unit] = ???
from it.
I cannot use Future.sequence combinator, since then I would wait for each future to complete before getting anything from the source. I want to get results in any order as soon as any future completes.
I understand that Source is a purely functional API and it should not run anything before somehow materializing it. So, my idea is to use an Iterator (which is lazy) to create a source:
Source { () =>
new Iterator[Future[Int]] {
override def hasNext: Boolean = ???
override def next(): Future[Int] = ???
}
}
But that would be a source of futures, not of actual values. I could also block on next using Await.result(future) but I'm not sure which tread pool's thread will be blocked. Also this will call futures sequentially, while I need parallel execution.
UPDATE 2: it turned out there was a much easier way to do it (thanks to Viktor Klang):
Source(futures).mapAsync(1)(identity)
UPDATE: here is what I've got based on #sschaef answer:
def futuresToSource[T](futures: Iterable[Future[T]])(implicit ec: ExecutionContext): Source[T, Unit] = {
def run(actor: ActorRef): Unit = {
futures.foreach { future =>
future.onComplete {
case Success(value) =>
actor ! value
case Failure(NonFatal(t)) =>
actor ! Status.Failure(t) // to signal error
}
}
Future.sequence(futures).onSuccess { case _ =>
actor ! Status.Success(()) // to signal stream's end
}
}
Source.actorRef[T](futures.size, OverflowStrategy.fail).mapMaterializedValue(run)
}
// ScalaTest tests follow
import scala.concurrent.ExecutionContext.Implicits.global
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
"futuresToSource" should "convert futures collection to akka-stream source" in {
val f1 = Future(1)
val f2 = Future(2)
val f3 = Future(3)
whenReady {
futuresToSource(List(f1, f2, f3)).runFold(Seq.empty[Int])(_ :+ _)
} { results =>
results should contain theSameElementsAs Seq(1, 2, 3)
}
}
it should "fail on future failure" in {
val f1 = Future(1)
val f2 = Future(2)
val f3 = Future.failed(new RuntimeException("future failed"))
whenReady {
futuresToSource(List(f1, f2, f3)).runWith(Sink.ignore).failed
} { t =>
t shouldBe a [RuntimeException]
t should have message "future failed"
}
}
Creating a source of Futures and then "flatten" it via mapAsync:
scala> Source(List(f1,f2,fN)).mapAsync(1)(identity)
res0: akka.stream.scaladsl.Source[Int,Unit] = akka.stream.scaladsl.Source#3e10d804
One of the easiest ways to feed a Source is through an Actor:
import scala.concurrent.Future
import akka.actor._
import akka.stream._
import akka.stream.scaladsl._
implicit val system = ActorSystem("MySystem")
def run(actor: ActorRef): Unit = {
import system.dispatcher
Future { Thread.sleep(100); actor ! 1 }
Future { Thread.sleep(200); actor ! 2 }
Future { Thread.sleep(300); actor ! 3 }
}
val source = Source
.actorRef[Int](0, OverflowStrategy.fail)
.mapMaterializedValue(ref ⇒ run(ref))
implicit val m = ActorMaterializer()
source runForeach { int ⇒
println(s"received: $int")
}
The Actor is created through the Source.actorRef method and made available through the mapMaterializedValue method. run simply takes the Actor and sends all the completed values to it, which can then be accessed through source. In the example above, the values are sent directly in the Future, but this can of course be done everywhere (for example in the onComplete call on the Future).
Here is an example receive part of an actor I'm working on:
def receive = {
case "begin" =>
val listOfFutures: IndexedSeq[Future[Any]] = workers.map(worker => worker ? Work("test"))
val future: Future[IndexedSeq[Any]] = Future.sequence(listOfFutures)
future onComplete {
case Success(result) => println("Eventual result: "+result)
case Failure(ex) => println("Failure: "+ex.getMessage)
}
case msg => println("A message received: "+msg)
}
When ask fails for one of the workers (in case of a timeout), sequence future completes with failure. However I want to know which worker(s) have failed. Is there a more elegant way rather than simply mapping listOfFutures one by one without using Future.sequence ?
You can use the future's recover method to map or wrap the underlying exception:
import scala.concurrent.{Future, ExecutionContext}
case class WorkerFailed(name: String, cause: Throwable)
extends Exception(s"$name - ${cause.getMessage}", cause)
def mark[A](name: String, f: Future[A]): Future[A] = f.recover {
case ex => throw WorkerFailed(name, ex)
}
import ExecutionContext.Implicits.global
val f = (0 to 10).map(i => mark(s"i = $i", Future { i / i }))
val g = Future.sequence(f)
g.value // WorkerFailed: i = 0 - / by zero
Thanks to #O__ I have come with another solution that may a better fit some some cases.
case class WorkerDone(name: String)
case class WorkerFailed(name: String)
import ExecutionContext.Implicits.global
val f = (0 to 10).map {
i => Future {i/i; WorkerDone(s"worker$i")}.recover{
case ex => WorkerFailed(s"worker$i")
}
}
val futureSeq = Future.sequence(f)
futureSeq onComplete {
case Success(list) => list.collect {case result:WorkerFailed => result}.foreach {failed => println("Failed: "+failed.name)}
case Failure(ex) => println("Exception: "+ex.getMessage)
}
// just to make sure program doesn't end before onComplete is called.
Thread.sleep(2000L)
I'm not sure that if my example is a good practice, but my aim is to know which workers did fail no matter how did they fail.
I have an existing API which returns a Future. Now introducing an Actor for one of the use cases and trying to continue using same service API from it. From below you can see MyService.saveValues return future.
object MyActor {
implicit val ec = scala.concurrent.ExecutionContext.Implicits.global
implicit val timeout = Timeout(1 second)
case class SampleMessage(list: List[String])
val actorSystem = //get actorSystem that was created at the startup
def saveMessage(list: List[String]) ={
val res = (actorSystem.actorOf(Props[MyActor]) ? SaveMyMessage(list) ).mapTo[Future[\/[Throwable,String]]
//res map{ r=>
//}
}
}
class MyActor extends Actor {
import MyActor._
def receive = {
case SaveMyMessage(list) =>
val originalSender = sender
val res : Future[\/[Throwable,String] ] = MyService.saveValues(list)
originalSender ! res
}
}
As you can see in def saveMessage I am using ask to wait for a result from an actor. However ask also creates its own future so result (val res) in saveMessage becomes Future[Future[T]] which looks annoying. What's the best way to deal with this scenario?
pipeTo forwards the result of a Future to an ActorRef.
import akka.pattern.pipe
val originalSender = sender
val res : Future[\/[Throwable,String] ] = MyService.saveValues(list)
res pipeTo originalSender
If saveValues throws, your future never complete and will end up timing something out though.
If you ever end up with Future[Future[A]], but want Future[A] as a result of sequencing/traversing or something, you can always "flatMap that shit."
import ExecutionContext.Implicits.global
val foo: Future[Future[Int]] = ???
val bar: Future[Int] = foo.flatMap(identity)
If you already depend on scalaz (and scalaz.contrib if on scalaz 7.0.x), you can use join defined in monad syntax.
import scalaz.syntax.monad._
import scalaz.contrib.std.scalaFuture._
val bar2: Future[Int] = foo.join