Handling exceptions in preStart in actors - scala

I have a service that has a supervisor responsible for build children actors and handling their exceptions.
ServiceMain -> Supervisor -> DiscoveryActor
To build the DiscoveryActor I call the following
Await.result(supervisor ? (Props[Discovery], "Discovery"), Duration.create(60, SECONDS)) match {
case actor: ActorRef =>
discoveryActor = actor
case ex: Exception =>
logger.error("Failed to initialize discovery actor", ex)
sys.exit(1)
}
Which gets handled by this code in Supervisor
def receive = {
case p: Props => sender() ! context.actorOf(p)
case (p: Props, s: String) => sender() ! context.actorOf(p, s)
}
It is expected of the DiscoveryActor to throw an exception in preStart() if it can't research a host as configured. The exception ApiConnectionException is thrown by the actor and is captured by akka and turned into a ActorInitializationException.
I have tired catching this exception in the Await and with the superviseStrategy as below
override val supervisorStrategy =
AllForOneStrategy() {
case _: Exception => Escalate
}
But neither of these have managed to catch it, what I am attempting to do is catch such exception and exit the application.
If anyone can point out where I am going wrong or what I have missed then I shall be very grateful!

I simplified your code a bit just to get straight to the root of the problem. You can copy it and paste it into your editor. It uses ScalaTest suite.
SupervisorStrategy defined in Supervisor actor does catch an exception thrown by Discovery actor in preStart method. You might want to have a closer look at your own code.
Your Await block is trying to catch an exception, but it's not possible in this context. The Exception is thrown by Discovery actor, not send as a message. Ask pattern (?) you use simply awaits for a message to arrive though. Only use of SupervisorStrategy can get you back a thrown exception. Instead of escalating exception in your Supervisor you could send a message to your application Guardian actor saying that initialization failed so the application should exit. Or do it directly in your Supervisor.
import java.util.concurrent.TimeUnit
import akka.actor.SupervisorStrategy.Escalate
import akka.actor._
import akka.pattern.ask
import akka.testkit.{ImplicitSender, TestKit}
import akka.util.Timeout
import org.scalatest.{BeforeAndAfterAll, FunSuiteLike, Matchers}
import scala.concurrent.Await
abstract class ActorSuite(systemName: String)
extends TestKit(ActorSystem(systemName))
with FunSuiteLike
with ImplicitSender
with Matchers
with BeforeAndAfterAll {
override def afterAll {
TestKit.shutdownActorSystem(system)
}
}
class FailingActorInitializationSuite extends ActorSuite("failing-system") {
test("run it") {
val supervisor = system.actorOf(Props[Supervisor])
var discoveryActor: ActorRef = null
implicit val timeout = Timeout(60, TimeUnit.SECONDS)
Await.result(
supervisor ?(Props[Discovery], "Discovery"), timeout.duration) match {
case actor: ActorRef =>
discoveryActor = actor
}
}
}
class Supervisor extends Actor with ActorLogging {
override val supervisorStrategy =
AllForOneStrategy() {
case e: Exception =>
log.error(s"Caught an exception [${e.getCause.getMessage}] and escalating")
Escalate
}
override def receive: Receive = {
case (p: Props, s: String) => sender() ! context.actorOf(p, s)
}
}
class Discovery extends Actor {
override def preStart(): Unit = {
super.preStart()
throw new RuntimeException("Can't create")
}
override def receive: Actor.Receive = {
case _ =>
}
}

Related

How to restart a killed actor?

I have a supervisor and a child actor that looks as the following:
SupervisorActor.scala
import akka.actor.{Actor, ActorLogging, Props, Terminated}
object SupervisorActor {
def props: Props = Props(new SupervisorActor)
object KillFoo
object TerminateFoo
object RestartFoo
}
final class SupervisorActor extends Actor with ActorLogging {
import com.sweetsoft.SupervisorActor._
log.info(s"Start the actor ${self}")
private val foo = context.actorOf(FooActor.props, "FOOCHILD")
context.watch(foo)
override def receive: Receive = {
case KillFoo =>
context.stop(foo)
case Terminated(actor) =>
log.info(s"Kill actor $actor")
case TerminateFoo =>
log.info("Terminate FooActor")
foo ! new IllegalArgumentException
case RestartFoo =>
}
}
FooActor.scala
import akka.actor.SupervisorStrategy.{Escalate, Restart, Resume, Stop}
import akka.actor.{Actor, ActorLogging, OneForOneStrategy, Props}
import scala.concurrent.duration._
object FooActor {
def props: Props = Props(new FooActor)
}
final class FooActor extends Actor with ActorLogging {
log.info(s"Start the actor ${ self }")
override val supervisorStrategy: OneForOneStrategy =
OneForOneStrategy(maxNrOfRetries = 1, withinTimeRange = 1.minute) {
case _: ArithmeticException => Resume
case _: NullPointerException => Restart
case _: IllegalArgumentException =>
println("STOP exception raised.")
Stop
case _: Exception => Escalate
}
override def receive: Receive = {
case _ => log.info("I got killed.")
}
}
and Main.scala
import akka.actor.ActorSystem
import com.sweetsoft.SupervisorActor.{TerminateFoo}
import scala.concurrent.Future
import scala.io.StdIn
object Main extends App {
val system = ActorSystem("monarch")
implicit val dispatcher = system.dispatcher
try {
// Create top level supervisor
val supervisor = system.actorOf(SupervisorActor.props, "mupervisor")
// Exit the system after ENTER is pressed
Future {
Thread.sleep(2000)
supervisor ! TerminateFoo
}
StdIn.readLine()
} finally {
system.terminate()
}
}
After the FooActor got killed, I would like to restart it manually again, like I did:
private val foo = context.actorOf(FooActor.props, "FOOCHILD")
How to do it?
I am thinking about to create a function, that is going to create the FooActor and after it got killed, just call the function to start a new FooActor.
There are a few problems with the code. supervisorStrategy should be in the SupervisorActor as it is responsible for supervision, not the child actor itself.
foo ! new IllegalArgumentException won't cause the child actor to terminate as actors can accept any object as a message and Exception derived objects are not treated specially. It'll just print "I got killed." but otherwise ignore the message and keep running. In particular, it won't invoke supervisorStrategy
handler.
You can either:
send the pre-defined system PoisonPill message to the child actor
to gracefully stop it (i.e. after processing all pending messages)
send the pre-defined system Kill message to the child actor to stop
it immediately after processing the current message and ignore the
other queued messages if any.
forward TerminateFoo message to it
and throw an exception in its handler. In this case the child
object's destiny is decided by supervisorStrategy handler for
IllegalArgumentException exception type (i.e. Stop in your case).
.
override def receive: Receive = {
case TerminateFoo =>
log.info("Stopping FooActor by throwing an unhandled exception for supervisorStrategy to process it")
throw new IllegalArgumentException
case m => log.info(s"I received a message $m")
}
See https://doc.akka.io/docs/akka/current/actors.html#stopping-actors for details

Error logging propagation when using Akka Supervision

Here's the thing am kinda stuck at. I have a SupervisorActor which creates Actor A and B and so on. There are no child actors to ActorA or ActorB.Lets say both Actor A and B hit Database and get SQL exception. This is propagated to the SupervisorActor up the chain. When I catch SQL exception, I also need to Log that Actor A had a SQL exception. But how can I achieve this?
1 way I could think is my Actor A logs it and throws an exception above the call stack. But I would need a try-catch block in my code. Which kinda defeats the purpose here.
Creating
Another way, I can think of is Actor A and B create a new child Actor A1 which would send it up the chain but that's not an option coz that's a common library without actors.
Is there a way to achieve something similar to :
Yes! I wanted to know if we can achieve something like:
try{
saveUser()
}
catch {
case b: BatchUpdateException =>
logger.error("We received a BatchUpdateException when trying to save the user")
throw b
}
case e: Exception =>
logger.error("Some other exception occured ")
throw e
}
try{
saveSeller()
}
catch {
case b: BatchUpdateException =>
logger.error("We received a BatchUpdateException when trying to save the Seller details")
throw b
}
case e: Exception =>
logger.error("Some other exception occured ")
throw e
}
PS: Am not sure if supervision strategy is the right approach to achieve what I am trying to achieve. I am trying to explore new possibilities.
Centralizing fault-handling logic within a supervisor actor using SupervisorStrategy is a better approach than scattering/duplicating it across individual actors. In particular, expressing the Exception-handling logic as the decider parameter of type PartialFunction[Throwable, Directive] helps improve code maintainability.
When I catch SQL exception, I also need to Log that Actor A had a SQL exception. But how can I achieve this?
Within the supervisor actor, you can always log Exceptions from individual child actors within the supervisor actor by including the corresponding actor references via sender. Below is a trivialized example of a supervisor actor logging actor-specific Exceptions from a couple of child actors and taking corresponding Resume/Stop/Escalate actions:
import akka.actor.{Actor, ActorSystem, Props, ActorLogging}
import akka.actor.OneForOneStrategy
import akka.actor.SupervisorStrategy._
import scala.concurrent.duration._
import java.sql.SQLException
implicit val system = ActorSystem("system")
implicit val ec = system.dispatcher
case class CreateWorker(props: Props, name: String)
case class BogusQuery(ex: Exception)
def doQuery(q: BogusQuery) = throw q.ex
class MySupervisor extends Actor with ActorLogging {
override val supervisorStrategy =
OneForOneStrategy(maxNrOfRetries = 5, withinTimeRange = 1.minute) {
case e: SQLException =>
log.error(s"Supervisor: $e from $sender; Resuming!")
Resume
case e: NullPointerException =>
log.error(s"Supervisor: $e from $sender; Stopping!")
Stop
case _: Exception =>
log.error(s"Supervisor: Unknown exception from $sender; Escalating!")
Escalate
}
def receive = {
case w: CreateWorker => sender ! context.actorOf(w.props, w.name)
}
}
class MyWorker extends Actor with ActorLogging {
def receive = {
case q: BogusQuery =>
log.info(s"$self: Received '$q'!")
doQuery(q)
case x =>
log.error(s"$self: Unknown value '${x}'!")
}
}
val supervisor = system.actorOf(Props[MySupervisor], "supervisor")
supervisor ! CreateWorker(Props[MyWorker], "workerA")
supervisor ! CreateWorker(Props[MyWorker], "workerB")
val workerA = system.actorSelection("/user/supervisor/workerA")
val workerB = system.actorSelection("/user/supervisor/workerB")
workerA ! BogusQuery(new SQLException)
// [INFO] [<timestamp>] [<dispatcher>] [akka://system/user/supervisor/workerA]
// Actor[akka://system/user/supervisor/workerA#-2129514903]:
// Received 'BogusQuery(java.sql.SQLException)'!
// [ERROR] [<timestamp>] [<dispatcher>] [akka://system/user/supervisor]
// Supervisor: java.sql.SQLException from
// Actor[akka://system/user/supervisor/workerA#-2129514903]; Resuming!
// [WARN] [<timestamp>] [<dispatcher>] [akka://system/user/supervisor/workerA] null
workerB ! BogusQuery(new NullPointerException)
// [ERROR] [<timestamp>] [<dispatcher>] [akka://system/user/supervisor]
// Supervisor: java.lang.NullPointerException from
// Actor[akka://system/user/supervisor/workerB#-1563197689]; Stopping!
// [ERROR] [<timestamp>] [<dispatcher>] [akka://system/user/supervisor/workerB] null
// java.lang.NullPointerException ...

How to stop an Actor reloading on exception

In a scenario where an exception is thrown in an Actor receive I want to prevent this actor from being reloaded. I understood that the correct way to do this is by overriding supervisorStrategy but this does not work as shown in the example below:
class MyActor extends Actor {
println("Created new actor")
def receive = {
case msg =>
println("Received message: " + msg)
throw new Exception()
}
override val supervisorStrategy = OneForOneStrategy() {
case _: Exception => Stop
}
}
val system = ActorSystem("Test")
val actor = system.actorOf(Props(new MyActor()))
actor ! "Hello"
When I run this code "Created new actor" is output twice showing that the Actor is reloaded again after the exception.
What is the correct way to prevent the Actor being reloaded?
When an actor overrides the default supervisor strategy, that strategy applies to that actor's children. Your actor is using the default supervisor strategy, which restarts actors when they throw an exception. Define a parent for your actor and override the supervisor strategy in that parent.
class MyParent extends Actor {
override val supervisorStrategy = OneForOneStrategy() {
case _: Exception => Stop
}
val child = context.actorOf(Props[MyActor])
def receive = {
case msg =>
println(s"Parent received the following message and is sending it to the child: $msg")
child ! msg
}
}
class MyActor extends Actor {
println("Created new actor")
def receive = {
case msg =>
println(s"Received message: $msg")
throw new Exception()
}
}
val system = ActorSystem("Test")
val actor = system.actorOf(Props[MyParent])
actor ! "Hello"
In the above example, a MyActor is created as a child of MyParent. When the latter receives the "Hello" message, it sends the same message to the child. The child is stopped when it throws the exception, and "Created new actor" is therefore printed only once.

Akka - test supervision strategy

I have the following scenario: the parent supervisor actor creates a child for each message using a factory (function) passed in the constructor.
class supervisorActor(childActorMaker: ActorRefFactory => ActorRef)
extends Actor with ActorLogging{
def receive: Receive = {
case "testThis" =>
val childActor = childActorMaker(context)
childActor!"messageForChild"
}
override val supervisorStrategy =
OneForOneStrategy() {
case _ => Stop
}
}
class childActor extends Actor {
def receive:Receive = {
case _ => /** whatever **/
}
}
In the test I override the child Receive to force an exception. My expectation is the child actor to be stopped every time because of the Supervision Strategy i set.
"When the child Actor throws an exception the Supervisor Actor " should " " +
" stop it" in {
val childActorRef = TestActorRef(new childActor() {
override def receive = {
case msg: String => throw new IllegalArgumentException("kaboom")
}
})
watch(childActorRef)
val maker = (_: ActorRefFactory) => childActorRef
val supervisorActorRef = system.actorOf(Props(new supervisorActor(maker)))
supervisorActorRef!"testThis"
expectTerminated(childActorRef, 1.second)
}
I would expect the child actor to be stoped because of the supervisorStrategy Stop.
Instead I get this error:
java.lang.AssertionError: assertion failed: timeout (1 second) during expectMsg: Terminated
Any idea of why is this happening? Thank you
It seems that the childActorRef is not created with supervisorActorRef's context (I mean ActorContext of supervisorActor you created in the test code).
So childActor(childActorRef) is not a child of supervisorActor(supervisorActorRef) in your test code. That's why supervisor strategy of supervisorActor doesn't serve the purpose.

Resolving Akka futures from ask in the event of a failure

I am calling an Actor using the ask pattern within a Spray application, and returning the result as the HTTP response. I map failures from the actor to a custom error code.
val authActor = context.actorOf(Props[AuthenticationActor])
callService((authActor ? TokenAuthenticationRequest(token)).mapTo[LoggedInUser]) { user =>
complete(StatusCodes.OK, user)
}
def callService[T](f: => Future[T])(cb: T => RequestContext => Unit) = {
onComplete(f) {
case Success(value: T) => cb(value)
case Failure(ex: ServiceException) => complete(ex.statusCode, ex.errorMessage)
case e => complete(StatusCodes.InternalServerError, "Unable to complete the request. Please try again later.")
//In reality this returns a custom error object.
}
}
This works correctly when the authActor sends a failure, but if the authActor throws an exception, nothing happens until the ask timeout completes. For example:
override def receive: Receive = {
case _ => throw new ServiceException(ErrorCodes.AuthenticationFailed, "No valid session was found for that token")
}
I know that the Akka docs say that
To complete the future with an exception you need send a Failure message to the sender. This is not done automatically when an actor throws an exception while processing a message.
But given that I use asks for a lot of the interface between the Spray routing actors and the service actors, I would rather not wrap the receive part of every child actor with a try/catch. Is there a better way to achieve automatic handling of exceptions in child actors, and immediately resolve the future in the event of an exception?
Edit: this is my current solution. However, it's quite messy to do this for every child actor.
override def receive: Receive = {
case default =>
try {
default match {
case _ => throw new ServiceException("")//Actual code would go here
}
}
catch {
case se: ServiceException =>
logger.error("Service error raised:", se)
sender ! Failure(se)
case ex: Exception =>
sender ! Failure(ex)
throw ex
}
}
That way if it's an expected error (i.e. ServiceException), it's handled by creating a failure. If it's unexpected, it returns a failure immediately so the future is resolved, but then throws the exception so it can still be handled by the SupervisorStrategy.
If you want a way to provide automatic sending of a response back to the sender in case of an unexpected exception, then something like this could work for you:
trait FailurePropatingActor extends Actor{
override def preRestart(reason:Throwable, message:Option[Any]){
super.preRestart(reason, message)
sender() ! Status.Failure(reason)
}
}
We override preRestart and propagate the failure back to the sender as a Status.Failure which will cause an upstream Future to be failed. Also, it's important to call super.preRestart here as that's where child stopping happens. Using this in an actor looks something like this:
case class GetElement(list:List[Int], index:Int)
class MySimpleActor extends FailurePropatingActor {
def receive = {
case GetElement(list, i) =>
val result = list(i)
sender() ! result
}
}
If I was to call an instance of this actor like so:
import akka.pattern.ask
import concurrent.duration._
val system = ActorSystem("test")
import system.dispatcher
implicit val timeout = Timeout(2 seconds)
val ref = system.actorOf(Props[MySimpleActor])
val fut = ref ? GetElement(List(1,2,3), 6)
fut onComplete{
case util.Success(result) =>
println(s"success: $result")
case util.Failure(ex) =>
println(s"FAIL: ${ex.getMessage}")
ex.printStackTrace()
}
Then it would properly hit my Failure block. Now, the code in that base trait works well when Futures are not involved in the actor that is extending that trait, like the simple actor here. But if you use Futures then you need to be careful as exceptions that happen in the Future don't cause restarts in the actor and also, in preRestart, the call to sender() will not return the correct ref because the actor has already moved into the next message. An actor like this shows that issue:
class MyBadFutureUsingActor extends FailurePropatingActor{
import context.dispatcher
def receive = {
case GetElement(list, i) =>
val orig = sender()
val fut = Future{
val result = list(i)
orig ! result
}
}
}
If we were to use this actor in the previous test code, we would always get a timeout in the failure situation. To mitigate that, you need to pipe the results of futures back to the sender like so:
class MyGoodFutureUsingActor extends FailurePropatingActor{
import context.dispatcher
import akka.pattern.pipe
def receive = {
case GetElement(list, i) =>
val fut = Future{
list(i)
}
fut pipeTo sender()
}
}
In this particular case, the actor itself is not restarted because it did not encounter an uncaught exception. Now, if your actor needed to do some additional processing after the future, you can pipe back to self and explicitly fail when you get a Status.Failure:
class MyGoodFutureUsingActor extends FailurePropatingActor{
import context.dispatcher
import akka.pattern.pipe
def receive = {
case GetElement(list, i) =>
val fut = Future{
list(i)
}
fut.to(self, sender())
case d:Double =>
sender() ! d * 2
case Status.Failure(ex) =>
throw ex
}
}
If that behavior becomes common, you can make it available to whatever actors need it like so:
trait StatusFailureHandling{ me:Actor =>
def failureHandling:Receive = {
case Status.Failure(ex) =>
throw ex
}
}
class MyGoodFutureUsingActor extends FailurePropatingActor with StatusFailureHandling{
import context.dispatcher
import akka.pattern.pipe
def receive = myReceive orElse failureHandling
def myReceive:Receive = {
case GetElement(list, i) =>
val fut = Future{
list(i)
}
fut.to(self, sender())
case d:Double =>
sender() ! d * 2
}
}