Error logging propagation when using Akka Supervision - scala

Here's the thing am kinda stuck at. I have a SupervisorActor which creates Actor A and B and so on. There are no child actors to ActorA or ActorB.Lets say both Actor A and B hit Database and get SQL exception. This is propagated to the SupervisorActor up the chain. When I catch SQL exception, I also need to Log that Actor A had a SQL exception. But how can I achieve this?
1 way I could think is my Actor A logs it and throws an exception above the call stack. But I would need a try-catch block in my code. Which kinda defeats the purpose here.
Creating
Another way, I can think of is Actor A and B create a new child Actor A1 which would send it up the chain but that's not an option coz that's a common library without actors.
Is there a way to achieve something similar to :
Yes! I wanted to know if we can achieve something like:
try{
saveUser()
}
catch {
case b: BatchUpdateException =>
logger.error("We received a BatchUpdateException when trying to save the user")
throw b
}
case e: Exception =>
logger.error("Some other exception occured ")
throw e
}
try{
saveSeller()
}
catch {
case b: BatchUpdateException =>
logger.error("We received a BatchUpdateException when trying to save the Seller details")
throw b
}
case e: Exception =>
logger.error("Some other exception occured ")
throw e
}
PS: Am not sure if supervision strategy is the right approach to achieve what I am trying to achieve. I am trying to explore new possibilities.

Centralizing fault-handling logic within a supervisor actor using SupervisorStrategy is a better approach than scattering/duplicating it across individual actors. In particular, expressing the Exception-handling logic as the decider parameter of type PartialFunction[Throwable, Directive] helps improve code maintainability.
When I catch SQL exception, I also need to Log that Actor A had a SQL exception. But how can I achieve this?
Within the supervisor actor, you can always log Exceptions from individual child actors within the supervisor actor by including the corresponding actor references via sender. Below is a trivialized example of a supervisor actor logging actor-specific Exceptions from a couple of child actors and taking corresponding Resume/Stop/Escalate actions:
import akka.actor.{Actor, ActorSystem, Props, ActorLogging}
import akka.actor.OneForOneStrategy
import akka.actor.SupervisorStrategy._
import scala.concurrent.duration._
import java.sql.SQLException
implicit val system = ActorSystem("system")
implicit val ec = system.dispatcher
case class CreateWorker(props: Props, name: String)
case class BogusQuery(ex: Exception)
def doQuery(q: BogusQuery) = throw q.ex
class MySupervisor extends Actor with ActorLogging {
override val supervisorStrategy =
OneForOneStrategy(maxNrOfRetries = 5, withinTimeRange = 1.minute) {
case e: SQLException =>
log.error(s"Supervisor: $e from $sender; Resuming!")
Resume
case e: NullPointerException =>
log.error(s"Supervisor: $e from $sender; Stopping!")
Stop
case _: Exception =>
log.error(s"Supervisor: Unknown exception from $sender; Escalating!")
Escalate
}
def receive = {
case w: CreateWorker => sender ! context.actorOf(w.props, w.name)
}
}
class MyWorker extends Actor with ActorLogging {
def receive = {
case q: BogusQuery =>
log.info(s"$self: Received '$q'!")
doQuery(q)
case x =>
log.error(s"$self: Unknown value '${x}'!")
}
}
val supervisor = system.actorOf(Props[MySupervisor], "supervisor")
supervisor ! CreateWorker(Props[MyWorker], "workerA")
supervisor ! CreateWorker(Props[MyWorker], "workerB")
val workerA = system.actorSelection("/user/supervisor/workerA")
val workerB = system.actorSelection("/user/supervisor/workerB")
workerA ! BogusQuery(new SQLException)
// [INFO] [<timestamp>] [<dispatcher>] [akka://system/user/supervisor/workerA]
// Actor[akka://system/user/supervisor/workerA#-2129514903]:
// Received 'BogusQuery(java.sql.SQLException)'!
// [ERROR] [<timestamp>] [<dispatcher>] [akka://system/user/supervisor]
// Supervisor: java.sql.SQLException from
// Actor[akka://system/user/supervisor/workerA#-2129514903]; Resuming!
// [WARN] [<timestamp>] [<dispatcher>] [akka://system/user/supervisor/workerA] null
workerB ! BogusQuery(new NullPointerException)
// [ERROR] [<timestamp>] [<dispatcher>] [akka://system/user/supervisor]
// Supervisor: java.lang.NullPointerException from
// Actor[akka://system/user/supervisor/workerB#-1563197689]; Stopping!
// [ERROR] [<timestamp>] [<dispatcher>] [akka://system/user/supervisor/workerB] null
// java.lang.NullPointerException ...

Related

How to restart a killed actor?

I have a supervisor and a child actor that looks as the following:
SupervisorActor.scala
import akka.actor.{Actor, ActorLogging, Props, Terminated}
object SupervisorActor {
def props: Props = Props(new SupervisorActor)
object KillFoo
object TerminateFoo
object RestartFoo
}
final class SupervisorActor extends Actor with ActorLogging {
import com.sweetsoft.SupervisorActor._
log.info(s"Start the actor ${self}")
private val foo = context.actorOf(FooActor.props, "FOOCHILD")
context.watch(foo)
override def receive: Receive = {
case KillFoo =>
context.stop(foo)
case Terminated(actor) =>
log.info(s"Kill actor $actor")
case TerminateFoo =>
log.info("Terminate FooActor")
foo ! new IllegalArgumentException
case RestartFoo =>
}
}
FooActor.scala
import akka.actor.SupervisorStrategy.{Escalate, Restart, Resume, Stop}
import akka.actor.{Actor, ActorLogging, OneForOneStrategy, Props}
import scala.concurrent.duration._
object FooActor {
def props: Props = Props(new FooActor)
}
final class FooActor extends Actor with ActorLogging {
log.info(s"Start the actor ${ self }")
override val supervisorStrategy: OneForOneStrategy =
OneForOneStrategy(maxNrOfRetries = 1, withinTimeRange = 1.minute) {
case _: ArithmeticException => Resume
case _: NullPointerException => Restart
case _: IllegalArgumentException =>
println("STOP exception raised.")
Stop
case _: Exception => Escalate
}
override def receive: Receive = {
case _ => log.info("I got killed.")
}
}
and Main.scala
import akka.actor.ActorSystem
import com.sweetsoft.SupervisorActor.{TerminateFoo}
import scala.concurrent.Future
import scala.io.StdIn
object Main extends App {
val system = ActorSystem("monarch")
implicit val dispatcher = system.dispatcher
try {
// Create top level supervisor
val supervisor = system.actorOf(SupervisorActor.props, "mupervisor")
// Exit the system after ENTER is pressed
Future {
Thread.sleep(2000)
supervisor ! TerminateFoo
}
StdIn.readLine()
} finally {
system.terminate()
}
}
After the FooActor got killed, I would like to restart it manually again, like I did:
private val foo = context.actorOf(FooActor.props, "FOOCHILD")
How to do it?
I am thinking about to create a function, that is going to create the FooActor and after it got killed, just call the function to start a new FooActor.
There are a few problems with the code. supervisorStrategy should be in the SupervisorActor as it is responsible for supervision, not the child actor itself.
foo ! new IllegalArgumentException won't cause the child actor to terminate as actors can accept any object as a message and Exception derived objects are not treated specially. It'll just print "I got killed." but otherwise ignore the message and keep running. In particular, it won't invoke supervisorStrategy
handler.
You can either:
send the pre-defined system PoisonPill message to the child actor
to gracefully stop it (i.e. after processing all pending messages)
send the pre-defined system Kill message to the child actor to stop
it immediately after processing the current message and ignore the
other queued messages if any.
forward TerminateFoo message to it
and throw an exception in its handler. In this case the child
object's destiny is decided by supervisorStrategy handler for
IllegalArgumentException exception type (i.e. Stop in your case).
.
override def receive: Receive = {
case TerminateFoo =>
log.info("Stopping FooActor by throwing an unhandled exception for supervisorStrategy to process it")
throw new IllegalArgumentException
case m => log.info(s"I received a message $m")
}
See https://doc.akka.io/docs/akka/current/actors.html#stopping-actors for details

How to stop an Actor reloading on exception

In a scenario where an exception is thrown in an Actor receive I want to prevent this actor from being reloaded. I understood that the correct way to do this is by overriding supervisorStrategy but this does not work as shown in the example below:
class MyActor extends Actor {
println("Created new actor")
def receive = {
case msg =>
println("Received message: " + msg)
throw new Exception()
}
override val supervisorStrategy = OneForOneStrategy() {
case _: Exception => Stop
}
}
val system = ActorSystem("Test")
val actor = system.actorOf(Props(new MyActor()))
actor ! "Hello"
When I run this code "Created new actor" is output twice showing that the Actor is reloaded again after the exception.
What is the correct way to prevent the Actor being reloaded?
When an actor overrides the default supervisor strategy, that strategy applies to that actor's children. Your actor is using the default supervisor strategy, which restarts actors when they throw an exception. Define a parent for your actor and override the supervisor strategy in that parent.
class MyParent extends Actor {
override val supervisorStrategy = OneForOneStrategy() {
case _: Exception => Stop
}
val child = context.actorOf(Props[MyActor])
def receive = {
case msg =>
println(s"Parent received the following message and is sending it to the child: $msg")
child ! msg
}
}
class MyActor extends Actor {
println("Created new actor")
def receive = {
case msg =>
println(s"Received message: $msg")
throw new Exception()
}
}
val system = ActorSystem("Test")
val actor = system.actorOf(Props[MyParent])
actor ! "Hello"
In the above example, a MyActor is created as a child of MyParent. When the latter receives the "Hello" message, it sends the same message to the child. The child is stopped when it throws the exception, and "Created new actor" is therefore printed only once.

Handling exceptions in preStart in actors

I have a service that has a supervisor responsible for build children actors and handling their exceptions.
ServiceMain -> Supervisor -> DiscoveryActor
To build the DiscoveryActor I call the following
Await.result(supervisor ? (Props[Discovery], "Discovery"), Duration.create(60, SECONDS)) match {
case actor: ActorRef =>
discoveryActor = actor
case ex: Exception =>
logger.error("Failed to initialize discovery actor", ex)
sys.exit(1)
}
Which gets handled by this code in Supervisor
def receive = {
case p: Props => sender() ! context.actorOf(p)
case (p: Props, s: String) => sender() ! context.actorOf(p, s)
}
It is expected of the DiscoveryActor to throw an exception in preStart() if it can't research a host as configured. The exception ApiConnectionException is thrown by the actor and is captured by akka and turned into a ActorInitializationException.
I have tired catching this exception in the Await and with the superviseStrategy as below
override val supervisorStrategy =
AllForOneStrategy() {
case _: Exception => Escalate
}
But neither of these have managed to catch it, what I am attempting to do is catch such exception and exit the application.
If anyone can point out where I am going wrong or what I have missed then I shall be very grateful!
I simplified your code a bit just to get straight to the root of the problem. You can copy it and paste it into your editor. It uses ScalaTest suite.
SupervisorStrategy defined in Supervisor actor does catch an exception thrown by Discovery actor in preStart method. You might want to have a closer look at your own code.
Your Await block is trying to catch an exception, but it's not possible in this context. The Exception is thrown by Discovery actor, not send as a message. Ask pattern (?) you use simply awaits for a message to arrive though. Only use of SupervisorStrategy can get you back a thrown exception. Instead of escalating exception in your Supervisor you could send a message to your application Guardian actor saying that initialization failed so the application should exit. Or do it directly in your Supervisor.
import java.util.concurrent.TimeUnit
import akka.actor.SupervisorStrategy.Escalate
import akka.actor._
import akka.pattern.ask
import akka.testkit.{ImplicitSender, TestKit}
import akka.util.Timeout
import org.scalatest.{BeforeAndAfterAll, FunSuiteLike, Matchers}
import scala.concurrent.Await
abstract class ActorSuite(systemName: String)
extends TestKit(ActorSystem(systemName))
with FunSuiteLike
with ImplicitSender
with Matchers
with BeforeAndAfterAll {
override def afterAll {
TestKit.shutdownActorSystem(system)
}
}
class FailingActorInitializationSuite extends ActorSuite("failing-system") {
test("run it") {
val supervisor = system.actorOf(Props[Supervisor])
var discoveryActor: ActorRef = null
implicit val timeout = Timeout(60, TimeUnit.SECONDS)
Await.result(
supervisor ?(Props[Discovery], "Discovery"), timeout.duration) match {
case actor: ActorRef =>
discoveryActor = actor
}
}
}
class Supervisor extends Actor with ActorLogging {
override val supervisorStrategy =
AllForOneStrategy() {
case e: Exception =>
log.error(s"Caught an exception [${e.getCause.getMessage}] and escalating")
Escalate
}
override def receive: Receive = {
case (p: Props, s: String) => sender() ! context.actorOf(p, s)
}
}
class Discovery extends Actor {
override def preStart(): Unit = {
super.preStart()
throw new RuntimeException("Can't create")
}
override def receive: Actor.Receive = {
case _ =>
}
}

Resolving Akka futures from ask in the event of a failure

I am calling an Actor using the ask pattern within a Spray application, and returning the result as the HTTP response. I map failures from the actor to a custom error code.
val authActor = context.actorOf(Props[AuthenticationActor])
callService((authActor ? TokenAuthenticationRequest(token)).mapTo[LoggedInUser]) { user =>
complete(StatusCodes.OK, user)
}
def callService[T](f: => Future[T])(cb: T => RequestContext => Unit) = {
onComplete(f) {
case Success(value: T) => cb(value)
case Failure(ex: ServiceException) => complete(ex.statusCode, ex.errorMessage)
case e => complete(StatusCodes.InternalServerError, "Unable to complete the request. Please try again later.")
//In reality this returns a custom error object.
}
}
This works correctly when the authActor sends a failure, but if the authActor throws an exception, nothing happens until the ask timeout completes. For example:
override def receive: Receive = {
case _ => throw new ServiceException(ErrorCodes.AuthenticationFailed, "No valid session was found for that token")
}
I know that the Akka docs say that
To complete the future with an exception you need send a Failure message to the sender. This is not done automatically when an actor throws an exception while processing a message.
But given that I use asks for a lot of the interface between the Spray routing actors and the service actors, I would rather not wrap the receive part of every child actor with a try/catch. Is there a better way to achieve automatic handling of exceptions in child actors, and immediately resolve the future in the event of an exception?
Edit: this is my current solution. However, it's quite messy to do this for every child actor.
override def receive: Receive = {
case default =>
try {
default match {
case _ => throw new ServiceException("")//Actual code would go here
}
}
catch {
case se: ServiceException =>
logger.error("Service error raised:", se)
sender ! Failure(se)
case ex: Exception =>
sender ! Failure(ex)
throw ex
}
}
That way if it's an expected error (i.e. ServiceException), it's handled by creating a failure. If it's unexpected, it returns a failure immediately so the future is resolved, but then throws the exception so it can still be handled by the SupervisorStrategy.
If you want a way to provide automatic sending of a response back to the sender in case of an unexpected exception, then something like this could work for you:
trait FailurePropatingActor extends Actor{
override def preRestart(reason:Throwable, message:Option[Any]){
super.preRestart(reason, message)
sender() ! Status.Failure(reason)
}
}
We override preRestart and propagate the failure back to the sender as a Status.Failure which will cause an upstream Future to be failed. Also, it's important to call super.preRestart here as that's where child stopping happens. Using this in an actor looks something like this:
case class GetElement(list:List[Int], index:Int)
class MySimpleActor extends FailurePropatingActor {
def receive = {
case GetElement(list, i) =>
val result = list(i)
sender() ! result
}
}
If I was to call an instance of this actor like so:
import akka.pattern.ask
import concurrent.duration._
val system = ActorSystem("test")
import system.dispatcher
implicit val timeout = Timeout(2 seconds)
val ref = system.actorOf(Props[MySimpleActor])
val fut = ref ? GetElement(List(1,2,3), 6)
fut onComplete{
case util.Success(result) =>
println(s"success: $result")
case util.Failure(ex) =>
println(s"FAIL: ${ex.getMessage}")
ex.printStackTrace()
}
Then it would properly hit my Failure block. Now, the code in that base trait works well when Futures are not involved in the actor that is extending that trait, like the simple actor here. But if you use Futures then you need to be careful as exceptions that happen in the Future don't cause restarts in the actor and also, in preRestart, the call to sender() will not return the correct ref because the actor has already moved into the next message. An actor like this shows that issue:
class MyBadFutureUsingActor extends FailurePropatingActor{
import context.dispatcher
def receive = {
case GetElement(list, i) =>
val orig = sender()
val fut = Future{
val result = list(i)
orig ! result
}
}
}
If we were to use this actor in the previous test code, we would always get a timeout in the failure situation. To mitigate that, you need to pipe the results of futures back to the sender like so:
class MyGoodFutureUsingActor extends FailurePropatingActor{
import context.dispatcher
import akka.pattern.pipe
def receive = {
case GetElement(list, i) =>
val fut = Future{
list(i)
}
fut pipeTo sender()
}
}
In this particular case, the actor itself is not restarted because it did not encounter an uncaught exception. Now, if your actor needed to do some additional processing after the future, you can pipe back to self and explicitly fail when you get a Status.Failure:
class MyGoodFutureUsingActor extends FailurePropatingActor{
import context.dispatcher
import akka.pattern.pipe
def receive = {
case GetElement(list, i) =>
val fut = Future{
list(i)
}
fut.to(self, sender())
case d:Double =>
sender() ! d * 2
case Status.Failure(ex) =>
throw ex
}
}
If that behavior becomes common, you can make it available to whatever actors need it like so:
trait StatusFailureHandling{ me:Actor =>
def failureHandling:Receive = {
case Status.Failure(ex) =>
throw ex
}
}
class MyGoodFutureUsingActor extends FailurePropatingActor with StatusFailureHandling{
import context.dispatcher
import akka.pattern.pipe
def receive = myReceive orElse failureHandling
def myReceive:Receive = {
case GetElement(list, i) =>
val fut = Future{
list(i)
}
fut.to(self, sender())
case d:Double =>
sender() ! d * 2
}
}

Akka matching Failures, and recovery

Stuff I need help with is in bold.
I have an actor that is flying multiple spray HttpRequests, the requests are paginated and the actor makes sure it writes the results in sequence into a database (sequence is important to resume crawlers). I explain this because I don't want to explore other patterns of concurrency at the moment. The actor needs to recover from timeouts without restarting.
in my actor I have the following :
case f : Failure => {
system.log.error("faiure")
system.log.error(s"$f")
system.shutdown()
}
case f : AskTimeoutException => {
system.log.error("faiure")
system.log.error(s"$f")
system.shutdown()
}
case msg # _ => {
system.log.error("Unexpected message in harvest")
system.log.error(s"${msg}")
system.shutdown()
}
but I can't match correctly :
[ERROR] [11/23/2013 14:58:10.694] [Crawler-akka.actor.default-dispatcher-3] [ActorSystem(Crawler)] Unexpected message in harvest
[ERROR] [11/23/2013 14:58:10.694] [Crawler-akka.actor.default-dispatcher-3] [ActorSystem(Crawler)] Failure(akka.pattern.AskTimeoutException: Timed out)
My dispatches look as follows :
abstract class CrawlerActor extends Actor {
private implicit val timeout: Timeout = 20.seconds
import context._
def dispatchRequest(node: CNode) {
val reqFut = (System.requester ? CrawlerRequest(node,Get(node.url))).map(r=> CrawlerResponse(node,r.asInstanceOf[HttpResponse]))
reqFut pipeTo self
}
class CrawlerRequester extends Actor {
import context._
val throttler = context.actorOf(Props(classOf[TimerBasedThrottler],System.Config.request_rate),"throttler")
throttler ! SetTarget(Some(IO(Http).actorRef))
def receive : Receive = {
case CrawlerRequest(type_,request) => {
throttler forward request
}
}
}
Once I find the correct way of matching, is there anyway I can get my hands on the CrawlerRequest that the timeout occurred with ? it contains some state I need to figure out how to recover.
This situation occurs if you use pipeTo to respond to message that sent by tell.
For example:
in actorA: actorB ! message
in actorB: message => doStuff pipeTo sender
in actorA: receives not 'scala.util.Failure', but 'akka.actor.Status.Failure'
The additional logic in pipeTo is to transform Try's Failure into akka's actor Failure (akka.actor.Status.Failure). This works fine when you use ask pattern, because temporary ask actor handle akka.actor.Status.Failure for you, but does not work well with tell.
Hope this short answer helps :)
Good luck!
Need to type out the full path of the Failure case class, (or import it I guess).
case f: akka.actor.Status.Failure => {
system.log.error("faiure")
system.log.error(s"${f.cause}")
system.shutdown()
}
That just leaves getting to the request associated with the timeout. Seems a map and pipe with a custom failure handler is needed at point request dispatch. Looking into it now.
The following trampolines the timeout into the actor.
case class CrawlerRequestTimeout(request: CrawlerRequest)
abstract class CrawlerActor extends Actor {
private implicit val timeout: Timeout = 20.seconds
import context._
def dispatchRequest(node: CNode) {
val req = CrawlerRequest(node,Get(node.url))
val reqFut = (System.requester ? req).map(r=> CrawlerResponse(node,r.asInstanceOf[HttpResponse]))
reqFut onFailure {
case te: akka.pattern.AskTimeoutException => self ! CrawlerRequestTimeout(req)
}
reqFut pipeTo self
}
}
with a match of :
case timeout : CrawlerRequestTimeout => {
println("boom")
system.shutdown()
}
Need to find a way of suppressing the exception though, it's still firing. Perhaps suppression isn't really a concern, verifying.
No, suppression is a concern, or the exception trickles down to the msg # _, need to put in a case class to absorb the redundant failure message.
ok, so getting rid of the pipeto gets rid of the exception entering the client actor. It's also a lot easier to read :D
abstract class CrawlerActor extends Actor {
private implicit val timeout: Timeout = 20.seconds
import context._
def dispatchRequest(node: CNode) {
val req = CrawlerRequest(node,Get(node.url))
val reqFut = (System.requester ? req)
reqFut onFailure {
case te: akka.pattern.AskTimeoutException => self ! CrawlerRequestTimeout(req)
}
reqFut onSuccess {
case r: HttpResponse => self ! CrawlerResponse(node,r)
}
}
}
If I understand correctly, you currently don't succeed in matching the AskTimeoutException.
If so, you should match case Failure(AskTimeoutException) => ... instead of case f : AskTimeoutException => ....