Time outs and crash when using actors from a mapping function - scala

Question

I want to do something like this:
For every item in the collection, ask an actor and block while waiting for the response for some timeout interval, if the time out exception is thrown, I want to move on to the next item.
Here's the code pattern:
implicit val timeout: akka.util.Timeout = 3 seconds
collection.foreach { item =>
val future = (actor ? Request(msg = item)).mapTo[Response]
future.onComplete {
case Failure(ex) => // log ex
case Success(resp) => // use resp
}
Await.result(future, 3 seconds)
}
The actor is itself calling other actors which might take longer than my 3 second time out.
This doesn't work as expected: after the first item times out, the whole thing crashes and stops. There are some dead letter notifications, I suppose that's because when the actors that my actor is calling finish, the original sender is invalid (it took more than 3 seconds). So my question is how do I tell it to just forget the time out item and continue with the rest as if nothing happened?

Answer 1

#stefanobaghino is right. See here, as written in documentation if a future contains an exception then Await.result throws is so that it can be handled properly.
Here you are matching Failure case of future but you are not recovering from it. A better approach would be like following -
collection.foreach { item =>
val future = (actor ? Request(msg = item)).mapTo[Response]
future.recover {
case ex: Exception =>
// log ex
Response(ex.message) // some other object of type Response
}
val response = Await.result(future, 3 seconds)
// use response here
}
After reading answer by #Dimitri I tried logging timestamps in milliseconds to see where it was causing lag in whole process and i found rather strange behavior. I observed that whenever there were dead-letters there was huge lag in even starting processing of next message to actor. Not sure why this is happening. Below is the code i tried to check it -
package com.lightbend.akka.sample
import akka.actor.{ Actor, ActorLogging, ActorRef, ActorSystem, Props }
import akka.pattern.{ ask, pipe, AskTimeoutException }
import scala.concurrent.Await
import scala.concurrent.duration._
import scala.io.StdIn
import scala.util.{ Try, Success, Failure }
import scala.concurrent.ExecutionContext.Implicits.global
import java.util.concurrent.TimeoutException
object AkkaQuickStart {
class NumberActor extends Actor {
override def receive: Receive = {
case (num: Int, startAt: Long) =>
println("B " + num.toString + " : " + System.currentTimeMillis().toString + " : " + (System.currentTimeMillis() - startAt).toString)
Thread.sleep(500 * num)
sender() ! "OK"
}
}
def main(args: Array[String]): Unit = {
implicit val timeout: akka.util.Timeout = 1 seconds
val numActor = ActorSystem("system").actorOf(Props(new NumberActor()))
val range = (1 to 5) ++ (4 to 1 by -1)
println(range)
def lag(implicit startAt: Long): String = (System.currentTimeMillis() - startAt).toString
range.map { r =>
implicit val startAt = System.currentTimeMillis()
println("A " + r.toString + " : " + System.currentTimeMillis().toString + " : " + lag)
val future = (numActor ? (r, startAt))
.recover {
case ex: AskTimeoutException =>
println("E " + r.toString + " : " + System.currentTimeMillis().toString + " : " + lag)
"Ask timeout"
}
.mapTo[String]
future.onComplete{
case Success(reply) =>
println("C " + r.toString + " : " + System.currentTimeMillis().toString + " : " + lag + " : success " + reply)
case Failure(reply) =>
println("C " + r.toString + " : " + System.currentTimeMillis().toString + " : " + lag + " : failure")
}
Try(Await.result(future, 1 seconds)) match {
case Success(reply) =>
println("D " + r.toString + " : " + System.currentTimeMillis().toString + " : " + lag + " : " + reply)
case Failure(ex) =>
println("D " + r.toString + " : " + System.currentTimeMillis().toString + " : " + lag + " : Await timeout ")
}
}
}
}
I tried different combinations of Ask timeout and Await timeout and found following lags in starting processing of actor message sent at the end of iteration -
Ask timeout = 1 Await Timeout = 1 => 3000 - 4500 ms causes dead-letters
Ask timeout = 1 Await Timeout = 3 => 3000 - 4500 ms causes dead-letters
Ask timeout = 3 Await Timeout = 1 => 3000 - 4500 ms causes dead-letters
Ask timeout = 3 Await timeout = 3 => 0 - 500 ms does not cause dead-letters
I am not sure but a guess is that dispatcher takes time in handling dead-letters and thus can not start processing messages of our Actor. May be some more experienced can explain it.

Answer 2

#stefanobaghino #Tarun Thanks for your help, I think I got it now.
So the thing is there are 2 timeouts that can cause an exception:
The Ask (?) timeout throws akka.pattern.AskTimeoutException if we have to wait longer than the actor takes to respond.
The Await.result throws java.util.concurrent.TimeoutException if we don't wait long enough for the future to finish.
Both of these can cause the whole thing to crash. For the first one, as you mentioned, we can add recover to return some default value. For the second one, we also should catch and handle the exception.
You can see different behaviours when changing the two timeouts and removing the recover/Try in the following example:
object Example {
class NumberActor extends Actor {
override def receive: Receive = {
case num: Int =>
Thread.sleep(250 * num)
sender() ! "OK"
}
}
def main(): Unit = {
implicit val timeout: akka.util.Timeout = 1 seconds
val numActor = ActorSystem("system").actorOf(Props(new NumberActor()))
val range = (1 to 5) ++ (4 to 1 by -1)
println(range)
range.map { r =>
val future = (numActor ? r)
.recover { case ex: TimeoutException => "FAIL" }
.mapTo[String]
Try(Await.result(future, 1 seconds)) match {
case Success(reply) => println(reply)
case Failure(ex) => println(ex)
}
}
}
}