Child actors, futures and exceptions - scala

Question

I'm currently working on an application with a signup process. This signup process will, at some point, communicate with external systems in an asynchronous manner. To keep this question concise, I'm showing you two important actors I've written:
SignupActor.scala
class SignupActor extends PersistentFSM[SignupActor.State, Data, DomainEvt] {
private val apiActor = context.actorOf(ExternalAPIActor.props(new HttpClient))
// At a certain point, a CreateUser(data) message is sent to the apiActor
}
ExternalAPIActor.scala
class ExternalAPIActor(apiClient: HttpClient) extends Actor {
override def preRestart(reason: Throwable, message: Option[Any]) = {
message.foreach(context.system.scheduler.scheduleOnce(3 seconds, self, _))
super.preRestart(reason, message)
}
def receive: Receive = {
case CreateUser(data) =>
Await.result(
apiClient.post(data)
.map(_ => UserCreatedInAPI())
.pipeTo(context.parent),
Timeout(5 seconds).duration
)
}
}
This setup seems to work as expected. When there is an issue with the external API (such as a timeout or network problems), the Future returned by HttpClient::post fails and will result in an exception thanks to Await.result. This, in turn thanks to the SupervisorStrategy of the SignupActor parent actor, will restart the ExternalAPIActor where we can re-send the last message to itself with a small delay to avoid deadlock.
I see a couple of issues with this setup:
Within the receive method of ExternalAPIActor, blocking occurs. As far as I understand, blocking within Actors is considered an anti-pattern.
The delay used to re-send the message is static. If the API is unavailable for longer periods of time, we will keep on sending HTTP requests every 3 seconds. I'd like some kind of exponential backoff mechanism here instead.
To continue on with the latter, I've tried the following in the SignupActor:
SignupActor.scala
val supervisor = BackoffSupervisor.props(
Backoff.onFailure(
ExternalAPIActor.props(new HttpClient),
childName = "external-api",
minBackoff = 3 seconds,
maxBackoff = 30 seconds,
randomFactor = 0.2
)
)
private val apiActor = context.actorOf(supervisor)
Unfortunately, this doesn't seem to do anything at all -- the preRestart method of ExternalAPIActor isn't called at all. When replacing Backoff.onFailure with Backoff.onStop, the preRestart method is called, but without any kind of exponential backoff at all.
Given the above, my questions are as follows:
Is using Await.result the recommended (the only?) way to make sure exceptions thrown in a Future returned from services called within actors are caught and handled accordingly? An especially important part of my particular use case is the fact that messages shouldn't be dropped but retried when something went wrong. Or is there some other (idiomatic) way that exceptions thrown in asynchronous contexts should be handled within Actors?
How would one use the BackoffSupervisor as intended in this case? Again: it is very important that the message responsible for the exception is not dropped, but retried until a N-number of times (to be determined by the maxRetries argument of SupervisorStrategy.

Answer 1

Is using Await.result the recommended (the only?) way to make sure
exceptions thrown in a Future returned from services called within
actors are caught and handled accordingly?
No. Generally that's not how you want to handle failures in Akka. A better alternative is to pipe the failure to your own actor, avoiding the need to use Await.result at all:
def receive: Receive = {
case CreateUser(data) =>
apiClient.post(data)
.map(_ => UserCreatedInAPI())
.pipeTo(self)
case Success(res) => context.parent ! res
case Failure(e) => // Invoke retry here
}
This would mean no restart is required to handle failure, they are all part of the normal flow of your actor.
An additional way to handle this can be to create a "supervised future". Taken from this blog post:
object SupervisedPipe {
case class SupervisedFailure(ex: Throwable)
class SupervisedPipeableFuture[T](future: Future[T])(implicit executionContext: ExecutionContext) {
// implicit failure recipient goes to self when used inside an actor
def supervisedPipeTo(successRecipient: ActorRef)(implicit failureRecipient: ActorRef): Unit =
future.andThen {
case Success(result) => successRecipient ! result
case Failure(ex) => failureRecipient ! SupervisedFailure(ex)
}
}
implicit def supervisedPipeTo[T](future: Future[T])(implicit executionContext: ExecutionContext): SupervisedPipeableFuture[T] =
new SupervisedPipeableFuture[T](future)
/* `orElse` with the actor receive logic */
val handleSupervisedFailure: Receive = {
// just throw the exception and make the actor logic handle it
case SupervisedFailure(ex) => throw ex
}
def supervised(receive: Receive): Receive =
handleSupervisedFailure orElse receive
}
This way, you only pipe to self once you get a Failure, and otherwise send it to the actor the message was meant to be sent to, avoiding the need for the case Success I added to the receive method. All you need to do is replace supervisedPipeTo with the original framework provided pipeTo.

Answer 2

Alright, I've done some more thinking and tinkering and I've come up with the following.
ExternalAPIActor.scala
class ExternalAPIActor(apiClient: HttpClient) extends Actor with Stash {
import ExternalAPIActor._
def receive: Receive = {
case msg # CreateUser(data) =>
context.become(waitingForExternalServiceReceive(msg))
apiClient.post(data)
.map(_ => UserCreatedInAPI())
.pipeTo(self)
}
def waitingForExternalServiceReceive(event: InputEvent): Receive = LoggingReceive {
case Failure(_) =>
unstashAll()
context.unbecome()
context.system.scheduler.scheduleOnce(3 seconds, self, event)
case msg:OutputEvent =>
unstashAll()
context.unbecome()
context.parent ! msg
case _ => stash()
}
}
object ExternalAPIActor {
sealed trait InputEvent
sealed trait OutputEvent
final case class CreateUser(data: Map[String,Any]) extends InputEvent
final case class UserCreatedInAPI() extends OutputEvent
}
I've used this technique to prevent the original message from being lost in case there is something wrong with the external service we're calling. During the process of a request to an external service, I switch context, waiting for either a response of a failure and switch back afterwards. Thanks to the Stash trait, I can make sure other requests to external services aren't lost as well.
Since I have multiple actors in my application calling external services, I abstracted the waitingForExternalServiceReceive to its own trait:
WaitingForExternalService.scala
trait WaitingForExternalServiceReceive[-tInput, +tOutput] extends Stash {
def waitingForExternalServiceReceive(event: tInput)(implicit ec: ExecutionContext): Receive = LoggingReceive {
case akka.actor.Status.Failure(_) =>
unstashAll()
context.unbecome()
context.system.scheduler.scheduleOnce(3 seconds, self, event)
case msg:tOutput =>
unstashAll()
context.unbecome()
context.parent ! msg
case _ => stash()
}
}
Now, the ExternalAPIActor can extend this trait:
ExternalAPIActor.scala
class ExternalAPIActor(apiClient: HttpClient) extends Actor with WaitingForExternalServiceReceive[InputEvent,OutputEvent] {
import ExternalAPIActor._
def receive: Receive = {
case msg # CreateUser(data) =>
context.become(waitingForExternalServiceReceive(msg))
apiClient.post(data)
.map(_ => UserCreatedInAPI())
.pipeTo(self)
}
}
object ExternalAPIActor {
sealed trait InputEvent
sealed trait OutputEvent
final case class CreateUser(data: Map[String,Any]) extends InputEvent
final case class UserCreatedInAPI() extends OutputEvent
}
Now, the actor won't get restarted in case of failures/errors and the message isn't lost. What's more, the entire flow of the actor now is non-blocking.
This setup is (most probably) far from perfect, but it seems to work exactly as I need it to.