I'm trying to respond to the sender of message when the receiving actor dies due to that message. If I Restart the actor on failure I get
preRestart(reason: Throwable, message: Option[Any])
but now I'm committed to restarting.
If I Stop the actor, i only get
postStop()
with no knowledge what stopped myself.
Meanwhile in the supervisor, I only get the Throwable and no indication of what caused it.
I suppose, I can dig through the DeadLetters post actor termination, but that seems like a noisy approach, since I'd have to listen to all of dead letters and somewhere correlate the termination with the deadletter event stream.
UPDATE: DeadLetter really doesn't seem to be an option. The message that caused the death doesn't even go to DeadLetters, it just disappears.
Is there a mechanism I am overlooking?
According to this thread on the Akka Users List, there isn't a mechanism in the actor supervision death cycle to accomplish this. Moreover, the documentation explicitly states that the message is dropped:
What happens to the Message
If an exception is thrown while a message is being processed (i.e. taken out of its mailbox and handed over to the current behavior), then this message will be lost. It is important to understand that it is not put back on the mailbox. So if you want to retry processing of a message, you need to deal with it yourself by catching the exception and retry your flow.
The ideal solution would be to use a dedicated actor for dangerous operations and have the initiator monitor the death of that actor to determine failure.
As my scenario arose from something considered safe but that had a bug in it, the separate actor option would have been after the fact. To avoid wrapping all code paths in try/catch but be able to guard more complicated and critical flows, I ended up with creating a wrapper for receive that let's me intercept exceptions:
object SafeReceive {
def apply(receive: Receive)(recover: Any => PartialFunction[Throwable, Unit]): Receive =
new Receive {
override def isDefinedAt(x: Any): Boolean = receive.isDefinedAt(x)
override def apply(v1: Any): Unit = try {
receive(v1)
} catch recover(v1)
}
}
which I can use for select actors like this:
def receive = SafeReceive {
case ... => ...
} {
msg => {
case e: Exception =>
sender ! OperationFailed(msg, e)
throw e
}
}
Related
I am trying to implement an Akka actor which receives a message and then waits to receive another message without looping. The reason that I am trying to implement it this way is because I require the object received in the first case statement in the second case statement. Below is an example of what I mean.
def receive: Receive = {
case s # start =>
receive {
case e # end =>
println(f"Received ${s} and ${e}")
}
}
I am aware that this might not be the ideal pattern. However, I am still new to Akka and Scala and am not aware of any other way to implement this. All suggestions are welcome.
This is a good way to achieve this:
Create a second receive method that just waits for the second message. Use context.become to set this method as the receive function when the first message arrives. The s value can be a parameter of this method.
When the second message arrives, it is handled by the new receive method. Use context.become to restore the original receive method once the message is processed.
You can use the Stash support to stash any other messages that come when the second receive method is active. When an unexpected message arrives, stash the message. When the correct message has been processed and the original receive handler has been restored, call unstashAll to put them back on the queue. These messages will be re-sent to the actor in the original order to be processed by the main receive method.
It might look like this:
def receive: Receive = {
case s # start =>
context.become(endReceive(s))
// Other cases
}
def endReceive(s: ???): Receive = {
case e # end =>
println(f"Received ${s} and ${e}")
context.become(receive)
unstashAll()
case _ =>
stash()
}
I have an actor p: ParentActor which I want to do some cleanup jobs when it is stopped by making use of the postStop method. One of the jobs involves sending a message to a child actor c: ChildActor. After that, c and then p should be stopped. However, once context.stop(p) has been called c seems to be immediately stopped and is unable to receive messages.
Below is an example of what I'd like to do:
class ParentActor extends Actor {
...
override def postStop {
logger.info("Shutdown message received. Sending message to child...")
val future = c ? "doThing"
logger.info("Waiting for child to complete task...")
Await.result(future, Duration.Inf)
context.stop(c)
logger.info("Child stopped. Stopping self. Bye!")
context.stop(self)
}
}
Results in error message:
[ERROR] [SetSystem-akka.actor.default-dispatcher-5] [akka://SetSystem/user/ParentActor] Recipient[Actor[akka://SetSystem/user/ParentActor/ChildActor#70868360]] had already been terminated. Sender[Actor[akka://SetSystem/user/ParentActor#662624985]] sent the message of type "java.lang.String".
An alternative would be to send p a message saying to shutdown, and have the above actions take place as a result of that, but using the built in stopping functionality seems better.
PS. This is a new application so design alternatives are welcome.
When an actor A is stopped, its children are indeed stopped before A's postStop hook is called. The sequence of events when an actor is stopped is as follows (from the official documentation):
Termination of an actor proceeds in two steps: first the actor suspends its mailbox processing and sends a stop command to all its children, then it keeps processing the internal termination notifications from its children until the last one is gone, finally terminating itself (invoking postStop, dumping mailbox, publishing Terminated on the DeathWatch, telling its supervisor).
Overriding the parent's postStop won't help you because your desired shutdown procedure includes sending a message to a child and waiting for a reply. When the parent is terminated, the child is stopped before the parent's postStop is run.
As you mentioned, sending a specific message to the ParentActor to initiate the shutdown is another approach. Something like the following would work:
import akka.pattern.pipe
class ParentActor extends Actor {
...
def receive = {
case Shutdown =>
(c ? DoYourThing).mapTo[ThingIsDone].pipeTo(self)
case ThingIsDone =>
logger.info("Child has finished its task.")
context.stop(self)
case ...
}
}
Note that one should avoid using Await in an actor. Instead, pipe the result of a Future to an actor.
I have an Actor, which acts as a Supervisor, but also needs to "return" data to the caller, wether this is an actor or not shouldn't matter.
I am asking my Supervisor, let's call him SV.
SV Processes the message I send to him, and sends back a response.
val system = ActorSystem("ActorSystem")
val sv = system.actorOf(Props[SV], name = "SV")
sv ? msg
And SV's recieve method looks like this:
def receive = {
case msg => (someChild ? msg).pipeTo(sender)
...
}
This all works jolly fine.
The problem is when the child throws an exception, this exception is caught by the supervisor strategy.
override def supervisorStrategy = OneForOneStrategy () {
case e : Throwable => {
val newResponse = someNewResponse
sender ! newResponse
...
}
}
sender is not longer a reference to whoever called SV in the first place, and I can't seem to work out how to send a message back to the askee, and get back to my original flow.
One of the three Actor rules is: “An Actor can send a finite number of messages to other Actors it knows.” The last two words are critical here: if the supervisor has not been introduced to the original sender somehow and the failure (exception) itself does not contain the sender’s reference either, then the supervisor cannot possibly send a message to the sender. Either you catch all exceptions in the child actor, wrap them together with the sender in your own exception type and then rethrow, or the original message needs to travel via the supervisor to the child and back, so that the supervisor can see that a reply is outstanding when a failure occurs.
Are you using the supervisor strategy and exceptions to control the flow or your data? Consider using the type system (Option type or a Failure on the response Future) for "exception" cases in your child actors and handle the response flow without exceptions.
The supervisor strategy is for unhandled exceptions.
When an unhandled exception occurs you lose the ability to respond with a message to the sender. Unless you wrap the sender in the unhandled exception like Roland Kuhn suggests.
Let the supervisor strategy instead handle how the actor system should respond to your unhandled exceptions by mapping them to a Directive.
I have some actors that when their preRestart() is invoked, the "message" is None. I don't understand why this is the case.
Context
My application is a Play web application that uses Akka. My error handling strategy is to send an error notification to the origin of the request, so details can be sent down to the user. I do this by hooking into preRestart() as follows:
trait NotifyRequesterErrorHandling { this: Actor with ActorLogging =>
override def preRestart(reason: Throwable, message: Option[Any]) {
notifyRequester(message, reason)
}
protected def notifyRequester(message: Option[Any], reason: Throwable) {
message match {
case Some(m) =>
val who = getRequester(m)
who ! SearchError(reason, m)
case None =>
val error = if (reason == null) "No exception" else reason toString()
ReportError("NotificationError", Some(reason), "", s"${this.context.self.path}: Attempted to notify requester but no previous message")
}
}
def getRequester(message: Any): ActorRef
}
Problem
In my logs, I am seeing a lot of the "Attempted to notify requester but no previous message" error logs. Usually it occurs for all the actors in my system. This is likely because I have one top-level actor, which is responsible for all the other actors (they are all children).
In the logs, the reason parameter does contain a throwable, though.
I am also using a one-for-all strategy. So, basically, whenever all the actors are restarted, I get lots of these errors.
Possible Explanations aka guesses
After all actors are restarted a new instance of each actor is created, and thus there is no previous message
When actors are restarted as one-for-all, all of their messages have been processed and their queue is empty. This means that there is no previous message
The documentation for preRestart states: "message - optionally the current message the actor processed when failing, if applicable"
I.e. it is only applicable to the failing actor. Not the other ones who are just restarted and not failing.
I am using the redis-react-nb library with my akka application. Unfortunately, when a connection to redis lost the library will stop itself with context stop self
This means that my application receives a termination notification like so:
def receive: Actor.Receive = {
case Terminated(actorRef) =>
...
}
So I'm wondering what to do here. Ideally I want to restart all children actors so they will all be initialised with a new redis connection - basically resetting the actor system to its initial state.
Am I missing something conceptually? Or is there a way for an actor to restart all of it's children without receiving an exception? self ! RestartAll or something similar?
Another way to look at this is to say that the actor you are talking about depends on the redis actor, it cannot continue its work without it. In that case it would be appropriate to escalate that as a failure to its parent and let that do the restart. If you do not handle the Terminated message, then a DeathPactException will be thrown, but you will have to install an appropriate supervisorStrategy in the parent since the default for that one is to terminate the child. I would recommend defining and throwing a RedisFailedException when you receive the Terminated message.
One of the possible solutions is take a look into implementation of SupervisorStrategy.Restart. Unfortunately the implementation uses InternalActorRef:
final def restartChild(child: ActorRef, cause: Throwable, suspendFirst: Boolean): Unit = {
val c = child.asInstanceOf[InternalActorRef]
if (suspendFirst) c.suspend()
c.restart(cause)
}
So I assume there is no built-in solution and you should do it yourself.
def receive = {
case Terminated(actorRef) => self ! RestartAll
case RestartAll =>
context.children.foreach(_ ! PoisonPill)
//start all necessary actors again
}