Resolving Akka futures from ask in the event of a failure - scala

I am calling an Actor using the ask pattern within a Spray application, and returning the result as the HTTP response. I map failures from the actor to a custom error code.
val authActor = context.actorOf(Props[AuthenticationActor])
callService((authActor ? TokenAuthenticationRequest(token)).mapTo[LoggedInUser]) { user =>
complete(StatusCodes.OK, user)
}
def callService[T](f: => Future[T])(cb: T => RequestContext => Unit) = {
onComplete(f) {
case Success(value: T) => cb(value)
case Failure(ex: ServiceException) => complete(ex.statusCode, ex.errorMessage)
case e => complete(StatusCodes.InternalServerError, "Unable to complete the request. Please try again later.")
//In reality this returns a custom error object.
}
}
This works correctly when the authActor sends a failure, but if the authActor throws an exception, nothing happens until the ask timeout completes. For example:
override def receive: Receive = {
case _ => throw new ServiceException(ErrorCodes.AuthenticationFailed, "No valid session was found for that token")
}
I know that the Akka docs say that
To complete the future with an exception you need send a Failure message to the sender. This is not done automatically when an actor throws an exception while processing a message.
But given that I use asks for a lot of the interface between the Spray routing actors and the service actors, I would rather not wrap the receive part of every child actor with a try/catch. Is there a better way to achieve automatic handling of exceptions in child actors, and immediately resolve the future in the event of an exception?
Edit: this is my current solution. However, it's quite messy to do this for every child actor.
override def receive: Receive = {
case default =>
try {
default match {
case _ => throw new ServiceException("")//Actual code would go here
}
}
catch {
case se: ServiceException =>
logger.error("Service error raised:", se)
sender ! Failure(se)
case ex: Exception =>
sender ! Failure(ex)
throw ex
}
}
That way if it's an expected error (i.e. ServiceException), it's handled by creating a failure. If it's unexpected, it returns a failure immediately so the future is resolved, but then throws the exception so it can still be handled by the SupervisorStrategy.

If you want a way to provide automatic sending of a response back to the sender in case of an unexpected exception, then something like this could work for you:
trait FailurePropatingActor extends Actor{
override def preRestart(reason:Throwable, message:Option[Any]){
super.preRestart(reason, message)
sender() ! Status.Failure(reason)
}
}
We override preRestart and propagate the failure back to the sender as a Status.Failure which will cause an upstream Future to be failed. Also, it's important to call super.preRestart here as that's where child stopping happens. Using this in an actor looks something like this:
case class GetElement(list:List[Int], index:Int)
class MySimpleActor extends FailurePropatingActor {
def receive = {
case GetElement(list, i) =>
val result = list(i)
sender() ! result
}
}
If I was to call an instance of this actor like so:
import akka.pattern.ask
import concurrent.duration._
val system = ActorSystem("test")
import system.dispatcher
implicit val timeout = Timeout(2 seconds)
val ref = system.actorOf(Props[MySimpleActor])
val fut = ref ? GetElement(List(1,2,3), 6)
fut onComplete{
case util.Success(result) =>
println(s"success: $result")
case util.Failure(ex) =>
println(s"FAIL: ${ex.getMessage}")
ex.printStackTrace()
}
Then it would properly hit my Failure block. Now, the code in that base trait works well when Futures are not involved in the actor that is extending that trait, like the simple actor here. But if you use Futures then you need to be careful as exceptions that happen in the Future don't cause restarts in the actor and also, in preRestart, the call to sender() will not return the correct ref because the actor has already moved into the next message. An actor like this shows that issue:
class MyBadFutureUsingActor extends FailurePropatingActor{
import context.dispatcher
def receive = {
case GetElement(list, i) =>
val orig = sender()
val fut = Future{
val result = list(i)
orig ! result
}
}
}
If we were to use this actor in the previous test code, we would always get a timeout in the failure situation. To mitigate that, you need to pipe the results of futures back to the sender like so:
class MyGoodFutureUsingActor extends FailurePropatingActor{
import context.dispatcher
import akka.pattern.pipe
def receive = {
case GetElement(list, i) =>
val fut = Future{
list(i)
}
fut pipeTo sender()
}
}
In this particular case, the actor itself is not restarted because it did not encounter an uncaught exception. Now, if your actor needed to do some additional processing after the future, you can pipe back to self and explicitly fail when you get a Status.Failure:
class MyGoodFutureUsingActor extends FailurePropatingActor{
import context.dispatcher
import akka.pattern.pipe
def receive = {
case GetElement(list, i) =>
val fut = Future{
list(i)
}
fut.to(self, sender())
case d:Double =>
sender() ! d * 2
case Status.Failure(ex) =>
throw ex
}
}
If that behavior becomes common, you can make it available to whatever actors need it like so:
trait StatusFailureHandling{ me:Actor =>
def failureHandling:Receive = {
case Status.Failure(ex) =>
throw ex
}
}
class MyGoodFutureUsingActor extends FailurePropatingActor with StatusFailureHandling{
import context.dispatcher
import akka.pattern.pipe
def receive = myReceive orElse failureHandling
def myReceive:Receive = {
case GetElement(list, i) =>
val fut = Future{
list(i)
}
fut.to(self, sender())
case d:Double =>
sender() ! d * 2
}
}

Related

Scala Akka Actor - Dead Letters encountered

I am using Scala with Akka actors. I know that an actor has a mailbox. So any communication with the actor is serialized. I have an actor that does a certain job--say it downloads an image.
class DownloadImageActor(implicit val injector: Injector) extends Actor with Injectable {
val imageDownloadService = inject[ImageDownloadService]
implicit val ec = inject[ExecutionContext]
override def receive: Receive = {
case DownloadImage(jobId, imageUrl) =>
imageDownloadService.downloadImage(imageUrl).onComplete {
case Success(image) =>
sender() ! ImageDownloadSuccess(imageUrl, image, jobId)
case Failure(e) =>
sender() ! ImageDownloadFail(imageUrl, e, jobId)
}
}
}
As you can see, the actor downloads the image in an async fashion. imageDownloadService.downloadImage returns a Future on whose completion a message is sent to the sender. Now this is where I get the dead letters encountered message.
Where have I gone wrong?
EDIT #1
The parent actor which sends message to the download actor
class ParentActor(implicit val injector : Injector) extends Actor with Injectable {
val downloadImageActor = inject[ActorRef](identified by "ImageDownloadActor")
override def receive: Receive = {
case DownloadImages(urls, _id) => urls.foreach(url =>
downloadImageActor ! DownloadImage(id, imageUrl = url)
)
case ImageDownloadSuccess(image : Image) =>
}
}
Don't know if any other problem, but the method you use sender in Future is wrong, you need to assign it to a new variable, then combine the onComplete callback it forms a closure which will not be override by other actor handle.
In your code, need to add lineA, and replace lineB, lineC with lineD, lineE. Or you may want to have a look for pipeTo function of Future.
class DownloadImageActor(implicit val injector : Injector) extends Actor with Injectable{
val imageDownloadService = inject[ImageDownloadService]
implicit val ec = inject[ExecutionContext]
override def receive: Receive = {
case DownloadImage(jobId, imageUrl) =>
val client = sender // lineA
imageDownloadService.downloadImage(imageUrl).onComplete {
//case Success(image) => sender() ! ImageDownloadSuccess(imageUrl, image, jobId) // lineB
//case Failure(e) => sender() ! ImageDownloadFail(imageUrl,e, jobId) // lineC
case Success(image) => client ! ImageDownloadSuccess(imageUrl, image, jobId) // lineD
case Failure(e) => client ! ImageDownloadFail(imageUrl,e, jobId) // lineE
}
}
}

What's the Akka-typed equivalent to pipeTo?

I'm currently trying to rewrite an existing untyped actor into a typed one. Since the actor is talking to a MySQL database using ScalikeJDBC, and since I'd like to have that done asynchronously, I'm dealing with Futures coming out of a separate (non-actor) repository class.
With untyped Akka, in an actor's receive method, I could do this:
import akka.pattern.pipe
val horseList : Future[Seq[Horse]] = horseRepository.listHorses(...)
horseList pipeTo sender()
And the sender actor would eventually receive a list of horses. I can't figure out how to do this inside a Behaviour, like:
val behaviour : Behavior[ListHorses] = Behaviors.receive {
(ctx,msg) => msg match {
case ListHorses(replyTo) =>
val horseListF : Future[Seq[Horse]] = horseRepository.listHorses(...)
// -> how do I make horseListF's content end up at replyTo? <-
Behaviors.same
}
}
The pipe pattern doesn't work (as it expects an untyped ActorRef), and so far I haven't found anything else in the akka-actor-typed (2.5.12) dependency I'm using to make this work.
How do I do this?
In Akka 2.5.22 (maybe earlier) there is context.pipeToSelf:
def pipeToSelf[Value](future: Future[Value])(mapResult: Try[Value] => T): Unit
You still have to provide a pattern match for Success and Failure, which in my code I've reduced with this sugar:
def mapPipe[A, T](success: A => T, failure: Throwable => T): Try[A] => T = {
case Success(value) => success(value)
case Failure(e) => failure(e)
}
Resulting in a call like this:
case class Horses(horses: Seq[Horse]) extends Command
case class HorseFailure(e: Throwable) extends Command
...
context.pipeToSelf(horseList) {
mapPipe(Horses,HorseFailure)
}
You can simply send a message to replyTo when the future completes successfully:
case ListHorses(replyTo) =>
horseRepository.listHorses(...) foreach { horses => replyTo ! horses }
Behaviors.same
Or if you want to handle errors as well:
case ListHorses(replyTo) =>
horseRepository.listHorses(...) onComplete {
case Success(horses) => replyTo ! horses
case Failure(e) => // error handling
}
Behaviors.same
In order for this to work, you need an ExecutionContext. It usually makes sense to use the same one as the actor, so you will have to make it available to onComplete or foreach first:
implicit val ec = ctx.executionContext

Akka Supervisor Strategy - Correct Use Case

I have been using Akka Supervisor Strategy to handle business logic exceptions.
Reading one of the most famous Scala blog series Neophyte, I found him giving a different purpose for what I have always been doing.
Example:
Let's say I have an HttpActor that should contact an external resource and in case it's down, I will throw an Exception, for now a ResourceUnavailableException.
In case my Supervisor catches that, I will call a Restart on my HttpActor, and in my HttpActor preRestart method, I will call do a schedulerOnce to retry that.
The actor:
class HttpActor extends Actor with ActorLogging {
implicit val system = context.system
override def preRestart(reason: Throwable, message: Option[Any]): Unit = {
log.info(s"Restarting Actor due: ${reason.getCause}")
message foreach { msg =>
context.system.scheduler.scheduleOnce(10.seconds, self, msg)
}
}
def receive = LoggingReceive {
case g: GetRequest =>
doRequest(http.doGet(g), g.httpManager.url, sender())
}
A Supervisor:
class HttpSupervisor extends Actor with ActorLogging with RouterHelper {
override val supervisorStrategy =
OneForOneStrategy(maxNrOfRetries = 5) {
case _: ResourceUnavailableException => Restart
case _: Exception => Escalate
}
var router = makeRouter[HttpActor](5)
def receive = LoggingReceive {
case g: GetRequest =>
router.route(g, sender())
case Terminated(a) =>
router = router.removeRoutee(a)
val r = context.actorOf(Props[HttpActor])
context watch r
router = router.addRoutee(r)
}
}
What's the point here?
In case my doRequest method throws the ResourceUnavailableException, the supervisor will get that and restart the actor, forcing it to resend the message after some time, according to the scheduler. The advantages I see is the fact I get for free the number of retries and a nice way to handle the exception itself.
Now looking at the blog, he shows a different approach in case you need a retry stuff, just sending messages like this:
def receive = {
case EspressoRequest =>
val receipt = register ? Transaction(Espresso)
receipt.map((EspressoCup(Filled), _)).recover {
case _: AskTimeoutException => ComebackLater
} pipeTo(sender)
case ClosingTime => context.system.shutdown()
}
Here in case of AskTimeoutException of the Future, he pipes the result as a ComebackLater object, which he will handle doing this:
case ComebackLater =>
log.info("grumble, grumble")
context.system.scheduler.scheduleOnce(300.millis) {
coffeeSource ! EspressoRequest
}
For me this is pretty much what you can do with the strategy supervisor, but in a manually way, with no built in number of retries logic.
So what is the best approach here and why? Is my concept of using akka supervisor strategy completely wrong?
You can use BackoffSupervisor:
Provided as a built-in pattern the akka.pattern.BackoffSupervisor implements the so-called exponential backoff supervision strategy, starting a child actor again when it fails, each time with a growing time delay between restarts.
val supervisor = BackoffSupervisor.props(
Backoff.onFailure(
childProps,
childName = "myEcho",
minBackoff = 3.seconds,
maxBackoff = 30.seconds,
randomFactor = 0.2 // adds 20% "noise" to vary the intervals slightly
).withAutoReset(10.seconds) // the child must send BackoffSupervisor.Reset to its parent
.withSupervisorStrategy(
OneForOneStrategy() {
case _: MyException => SupervisorStrategy.Restart
case _ => SupervisorStrategy.Escalate
}))

Execution context for futures in Actors

I have a Actor, and on some message I'm running some method which returns Future.
def receive: Receive = {
case SimpleMessge() =>
val futData:Future[Int] = ...
futData.map { data =>
...
}
}
Is it possible to pass actual context to wait for this data? Or Await is the best I can do if I need this data in SimpleMessage?
If you really need to wait for the future to complete before processing the next message, you can try something like this:
object SimpleMessageHandler{
case class SimpleMessage()
case class FinishSimpleMessage(i:Int)
}
class SimpleMessageHandler extends Actor with Stash{
import SimpleMessageHandler._
import context._
import akka.pattern.pipe
def receive = waitingForMessage
def waitingForMessage: Receive = {
case SimpleMessage() =>
val futData:Future[Int] = ...
futData.map(FinishSimpleMessage(_)) pipeTo self
context.become(waitingToFinish(sender))
}
def waitingToFinish(originalSender:ActorRef):Receive = {
case SimpleMessage() => stash()
case FinishSimpleMessage(i) =>
//Do whatever you need to do to finish here
...
unstashAll()
context.become(waitingForMessage)
case Status.Failure(ex) =>
//log error here
unstashAll()
context.become(waitingForMessage)
}
}
In this approach, we process a SimpleMessage and then switch handling logic to stash all subsequent SimpleMessages received until we get a result from the future. When we get a result, failure or not, we unstash all of the other SimpleMessages we have received while waiting for the future and go on our merry way.
This actor just toggles back and forth between two states and that allows you to only fully process one SimpleMessage at a time without needing to block on the Future.

Akka matching Failures, and recovery

Stuff I need help with is in bold.
I have an actor that is flying multiple spray HttpRequests, the requests are paginated and the actor makes sure it writes the results in sequence into a database (sequence is important to resume crawlers). I explain this because I don't want to explore other patterns of concurrency at the moment. The actor needs to recover from timeouts without restarting.
in my actor I have the following :
case f : Failure => {
system.log.error("faiure")
system.log.error(s"$f")
system.shutdown()
}
case f : AskTimeoutException => {
system.log.error("faiure")
system.log.error(s"$f")
system.shutdown()
}
case msg # _ => {
system.log.error("Unexpected message in harvest")
system.log.error(s"${msg}")
system.shutdown()
}
but I can't match correctly :
[ERROR] [11/23/2013 14:58:10.694] [Crawler-akka.actor.default-dispatcher-3] [ActorSystem(Crawler)] Unexpected message in harvest
[ERROR] [11/23/2013 14:58:10.694] [Crawler-akka.actor.default-dispatcher-3] [ActorSystem(Crawler)] Failure(akka.pattern.AskTimeoutException: Timed out)
My dispatches look as follows :
abstract class CrawlerActor extends Actor {
private implicit val timeout: Timeout = 20.seconds
import context._
def dispatchRequest(node: CNode) {
val reqFut = (System.requester ? CrawlerRequest(node,Get(node.url))).map(r=> CrawlerResponse(node,r.asInstanceOf[HttpResponse]))
reqFut pipeTo self
}
class CrawlerRequester extends Actor {
import context._
val throttler = context.actorOf(Props(classOf[TimerBasedThrottler],System.Config.request_rate),"throttler")
throttler ! SetTarget(Some(IO(Http).actorRef))
def receive : Receive = {
case CrawlerRequest(type_,request) => {
throttler forward request
}
}
}
Once I find the correct way of matching, is there anyway I can get my hands on the CrawlerRequest that the timeout occurred with ? it contains some state I need to figure out how to recover.
This situation occurs if you use pipeTo to respond to message that sent by tell.
For example:
in actorA: actorB ! message
in actorB: message => doStuff pipeTo sender
in actorA: receives not 'scala.util.Failure', but 'akka.actor.Status.Failure'
The additional logic in pipeTo is to transform Try's Failure into akka's actor Failure (akka.actor.Status.Failure). This works fine when you use ask pattern, because temporary ask actor handle akka.actor.Status.Failure for you, but does not work well with tell.
Hope this short answer helps :)
Good luck!
Need to type out the full path of the Failure case class, (or import it I guess).
case f: akka.actor.Status.Failure => {
system.log.error("faiure")
system.log.error(s"${f.cause}")
system.shutdown()
}
That just leaves getting to the request associated with the timeout. Seems a map and pipe with a custom failure handler is needed at point request dispatch. Looking into it now.
The following trampolines the timeout into the actor.
case class CrawlerRequestTimeout(request: CrawlerRequest)
abstract class CrawlerActor extends Actor {
private implicit val timeout: Timeout = 20.seconds
import context._
def dispatchRequest(node: CNode) {
val req = CrawlerRequest(node,Get(node.url))
val reqFut = (System.requester ? req).map(r=> CrawlerResponse(node,r.asInstanceOf[HttpResponse]))
reqFut onFailure {
case te: akka.pattern.AskTimeoutException => self ! CrawlerRequestTimeout(req)
}
reqFut pipeTo self
}
}
with a match of :
case timeout : CrawlerRequestTimeout => {
println("boom")
system.shutdown()
}
Need to find a way of suppressing the exception though, it's still firing. Perhaps suppression isn't really a concern, verifying.
No, suppression is a concern, or the exception trickles down to the msg # _, need to put in a case class to absorb the redundant failure message.
ok, so getting rid of the pipeto gets rid of the exception entering the client actor. It's also a lot easier to read :D
abstract class CrawlerActor extends Actor {
private implicit val timeout: Timeout = 20.seconds
import context._
def dispatchRequest(node: CNode) {
val req = CrawlerRequest(node,Get(node.url))
val reqFut = (System.requester ? req)
reqFut onFailure {
case te: akka.pattern.AskTimeoutException => self ! CrawlerRequestTimeout(req)
}
reqFut onSuccess {
case r: HttpResponse => self ! CrawlerResponse(node,r)
}
}
}
If I understand correctly, you currently don't succeed in matching the AskTimeoutException.
If so, you should match case Failure(AskTimeoutException) => ... instead of case f : AskTimeoutException => ....