I have some actors that when their preRestart() is invoked, the "message" is None. I don't understand why this is the case.
Context
My application is a Play web application that uses Akka. My error handling strategy is to send an error notification to the origin of the request, so details can be sent down to the user. I do this by hooking into preRestart() as follows:
trait NotifyRequesterErrorHandling { this: Actor with ActorLogging =>
override def preRestart(reason: Throwable, message: Option[Any]) {
notifyRequester(message, reason)
}
protected def notifyRequester(message: Option[Any], reason: Throwable) {
message match {
case Some(m) =>
val who = getRequester(m)
who ! SearchError(reason, m)
case None =>
val error = if (reason == null) "No exception" else reason toString()
ReportError("NotificationError", Some(reason), "", s"${this.context.self.path}: Attempted to notify requester but no previous message")
}
}
def getRequester(message: Any): ActorRef
}
Problem
In my logs, I am seeing a lot of the "Attempted to notify requester but no previous message" error logs. Usually it occurs for all the actors in my system. This is likely because I have one top-level actor, which is responsible for all the other actors (they are all children).
In the logs, the reason parameter does contain a throwable, though.
I am also using a one-for-all strategy. So, basically, whenever all the actors are restarted, I get lots of these errors.
Possible Explanations aka guesses
After all actors are restarted a new instance of each actor is created, and thus there is no previous message
When actors are restarted as one-for-all, all of their messages have been processed and their queue is empty. This means that there is no previous message
The documentation for preRestart states: "message - optionally the current message the actor processed when failing, if applicable"
I.e. it is only applicable to the failing actor. Not the other ones who are just restarted and not failing.
Related
I have the below Actor which receives two messages:
class SampleActor extends Actor {
def receive = {
case "hello" => println("Hello Message")
case "world"=>println("World Message")
case _=>println("Unhandled message")
}
}
}
I want to test if the actor is able to receive the messages "hello" and "world". The examples I have seen do not fulfill this requirement.
The one here and here has the actor sending back a message and using expectMsg() they check the response. But my requirement is to see if the Actor is able to receive the message or not, and not whether the actor is able to respond back with a specific message.
The example here changes the state of the actor and they check the state to see if the actor is able to receive the message. Again, not what I wanted.
So, how do I just test if my actor is able to handle the two messages through TestKit?
I have an Actor, which acts as a Supervisor, but also needs to "return" data to the caller, wether this is an actor or not shouldn't matter.
I am asking my Supervisor, let's call him SV.
SV Processes the message I send to him, and sends back a response.
val system = ActorSystem("ActorSystem")
val sv = system.actorOf(Props[SV], name = "SV")
sv ? msg
And SV's recieve method looks like this:
def receive = {
case msg => (someChild ? msg).pipeTo(sender)
...
}
This all works jolly fine.
The problem is when the child throws an exception, this exception is caught by the supervisor strategy.
override def supervisorStrategy = OneForOneStrategy () {
case e : Throwable => {
val newResponse = someNewResponse
sender ! newResponse
...
}
}
sender is not longer a reference to whoever called SV in the first place, and I can't seem to work out how to send a message back to the askee, and get back to my original flow.
One of the three Actor rules is: “An Actor can send a finite number of messages to other Actors it knows.” The last two words are critical here: if the supervisor has not been introduced to the original sender somehow and the failure (exception) itself does not contain the sender’s reference either, then the supervisor cannot possibly send a message to the sender. Either you catch all exceptions in the child actor, wrap them together with the sender in your own exception type and then rethrow, or the original message needs to travel via the supervisor to the child and back, so that the supervisor can see that a reply is outstanding when a failure occurs.
Are you using the supervisor strategy and exceptions to control the flow or your data? Consider using the type system (Option type or a Failure on the response Future) for "exception" cases in your child actors and handle the response flow without exceptions.
The supervisor strategy is for unhandled exceptions.
When an unhandled exception occurs you lose the ability to respond with a message to the sender. Unless you wrap the sender in the unhandled exception like Roland Kuhn suggests.
Let the supervisor strategy instead handle how the actor system should respond to your unhandled exceptions by mapping them to a Directive.
I'm trying to respond to the sender of message when the receiving actor dies due to that message. If I Restart the actor on failure I get
preRestart(reason: Throwable, message: Option[Any])
but now I'm committed to restarting.
If I Stop the actor, i only get
postStop()
with no knowledge what stopped myself.
Meanwhile in the supervisor, I only get the Throwable and no indication of what caused it.
I suppose, I can dig through the DeadLetters post actor termination, but that seems like a noisy approach, since I'd have to listen to all of dead letters and somewhere correlate the termination with the deadletter event stream.
UPDATE: DeadLetter really doesn't seem to be an option. The message that caused the death doesn't even go to DeadLetters, it just disappears.
Is there a mechanism I am overlooking?
According to this thread on the Akka Users List, there isn't a mechanism in the actor supervision death cycle to accomplish this. Moreover, the documentation explicitly states that the message is dropped:
What happens to the Message
If an exception is thrown while a message is being processed (i.e. taken out of its mailbox and handed over to the current behavior), then this message will be lost. It is important to understand that it is not put back on the mailbox. So if you want to retry processing of a message, you need to deal with it yourself by catching the exception and retry your flow.
The ideal solution would be to use a dedicated actor for dangerous operations and have the initiator monitor the death of that actor to determine failure.
As my scenario arose from something considered safe but that had a bug in it, the separate actor option would have been after the fact. To avoid wrapping all code paths in try/catch but be able to guard more complicated and critical flows, I ended up with creating a wrapper for receive that let's me intercept exceptions:
object SafeReceive {
def apply(receive: Receive)(recover: Any => PartialFunction[Throwable, Unit]): Receive =
new Receive {
override def isDefinedAt(x: Any): Boolean = receive.isDefinedAt(x)
override def apply(v1: Any): Unit = try {
receive(v1)
} catch recover(v1)
}
}
which I can use for select actors like this:
def receive = SafeReceive {
case ... => ...
} {
msg => {
case e: Exception =>
sender ! OperationFailed(msg, e)
throw e
}
}
I am using the redis-react-nb library with my akka application. Unfortunately, when a connection to redis lost the library will stop itself with context stop self
This means that my application receives a termination notification like so:
def receive: Actor.Receive = {
case Terminated(actorRef) =>
...
}
So I'm wondering what to do here. Ideally I want to restart all children actors so they will all be initialised with a new redis connection - basically resetting the actor system to its initial state.
Am I missing something conceptually? Or is there a way for an actor to restart all of it's children without receiving an exception? self ! RestartAll or something similar?
Another way to look at this is to say that the actor you are talking about depends on the redis actor, it cannot continue its work without it. In that case it would be appropriate to escalate that as a failure to its parent and let that do the restart. If you do not handle the Terminated message, then a DeathPactException will be thrown, but you will have to install an appropriate supervisorStrategy in the parent since the default for that one is to terminate the child. I would recommend defining and throwing a RedisFailedException when you receive the Terminated message.
One of the possible solutions is take a look into implementation of SupervisorStrategy.Restart. Unfortunately the implementation uses InternalActorRef:
final def restartChild(child: ActorRef, cause: Throwable, suspendFirst: Boolean): Unit = {
val c = child.asInstanceOf[InternalActorRef]
if (suspendFirst) c.suspend()
c.restart(cause)
}
So I assume there is no built-in solution and you should do it yourself.
def receive = {
case Terminated(actorRef) => self ! RestartAll
case RestartAll =>
context.children.foreach(_ ! PoisonPill)
//start all necessary actors again
}
I have a supervising Akka actor which uses a router to forward messages to worker actors.
I have a class which wraps the supervisor and when I call a method on that class it "asks" the supervisor to do something and then I use Await.result(theFuture) to wait for the result (I cannot continue without the result).
If the workers throw an exception, I want to restart the worker which threw the exception, and I want the exception to be caught by the code which calls the wrapper class.
I passed a OneForOneStrategy to the router constructor, which returns RESTART in the case of an Exception. In the postRestart method of the worker, I log the restart, so I can validate that the worker is actually restarted.
When the worker throws an exception, it gets restarted, but the exception disappears. The Future which is the result of asking the supervisor, contains an exception, but it is a akka.pattern.AskTimeoutException, which is thrown after just 5 seconds rather than 20 seconds, which is the implicit timeout that I have lingering around. The exception actually occurs less than a second after the worker starts.
Question 1: how can I get the exception from the worker in the code which calls my wrapper class?
Also, the receive method of the worker is like this:
def receive = {
case r: Request =>
val response = ??? //throws an exception sometimes
sender ! response
}
Something is logging the exception to the console, but it isn't my code. The stack trace is:
[ERROR] [02/11/2013 21:34:20.093] [MySystem-akka.actor.default-dispatcher-9]
[akka://MySystem/user/MySupervisor/MyRouter/$a] Something went wrong!
at myApp.Worker.$$anonfun$receive$1.applyOrElse(Source.scala:169)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:425)
at akka.actor.ActorCell.invoke(ActorCell.scala:386)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:230)
at akka.dispatch.Mailbox.run(Mailbox.scala:212)
at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:502)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
Line 169 of Source.scala is the line val response = ??? shown in the listing of the receive method above.
Question 2: who is logging that exception to the console, and how can I stop it?
1)
try somethingThatCanFail() catch {
case e: Exception => sender ! Status.Failure(e); throw e
}
The "tell failure" causes the caller to get a Failure containing the exception. Throwing "e" causes the oneForOneStrategy to be called which restarts the worker.
2)
It is the actor system itself that logs the failure, and the only way
to quiet it down is to filter out things by creating and configuring
your own LoggingAdapter as described here
http://doc.akka.io/docs/akka/2.1.0/scala/logging.html There is a
ticket for changing this
https://www.assembla.com/spaces/akka/tickets/2824 but it is targeted
for Akka 2.2
Answered by https://groups.google.com/forum/#!topic/akka-user/fenCvYu3HYE
In order to be notified of one of your children failing, you need to
First watch the child
Then you will be sent a Terminated() when the actor dies with a reference to it.
Something like:
class ParentActor extends Actor {
// this is sample of how to watch for death of one of your children
val childActor = context.actorOf(Props[SomeService], "SomeService")
val dyingChild = context.watch(context.actorOf(childActor))
def receive = {
case Terminated(`dyingChild`) =>
println("dyingChild died")
case Terminated(terminatedActor) =>
println(s"This child just died $terminatedActor")
}
}
Hope this helps.