I have an actor which receive Int and accumulate result to the buffer. Some times actor can throw exception. What is the best way to restore previous state of buffer into actor after exception ?
BR.
In order to control what happens you need to set the supervisor strategy on the parent actor.
override val supervisorStrategy =
OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1 minute) {
case e: Exception => {
println(s"got exception ~ ${e.getMessage()}")
Resume // Or Continue, Escalate, Restart, Stop
}
}
If you would like to handle the exception then resume the actor then your handler would return Resume
When resuming, the actor state remains unchanged.
In addition to manub's answer, you should also consider separating
the part that accumulates the value, and
the part that may fail
into two actors as follows:
The accumulating actor should create a child actor. This child actor does the job that can potentially fail, and then sends the result of its job as a message back to its parent. The parent actor receives the result of the job and does the accumulating, or a notification that the child actor has failed. This way, you can, for example, restart failed jobs using various supervision strategies.
For more information on such an approach, see for example this answer Handling Faults in Akka actors
In order to be able to recover the state of a stateful actor you should consider Akka Persistence. You will require a durable store (Cassandra or LevelDB for example) to persist state, and the framework will handle restoring the state for you (provided that you have a PersistentActor).
http://doc.akka.io/docs/akka/current/scala/persistence.html explains in detail how to achieve this.
You can set the Supervisor strategy to Resume. When your Actor throws an exception, the Supervisor will handle the exception and your Actor will throw away the current message and move on to the next one. (Newer versions of Akka let you set a maximum number of retires on the same message)
If instead, you would like your Actor to continue processing the message that caused the exception, you can use a try-catch-finally block.
Related
I have a question about Akka Timeouts.
For example, I have the Supervisor actor which sends a message to DBActor to start reading the database. The Supervisor sets a timeout of 5 seconds.
The DBActor starts reading the database; it gets the ResultSet, and then enters a loop to iterate the ResultSet rows and build the return value.
Then the Supervisor experiences a Timeout exception and returns a value from the .recover block
My question is ; does the DBActor continue to loop through the ResultSet after the TimeoutException ? Because that would be a resource leak.
The DBActor is independent of the supervisor and will keep processing the records until it is finished or is told to stop. It does not matter that the supervisor has given up on the current request because the DBActor does not know this. If you want the process to stop early, the supervisor will have to send another message to the DBActor to tell it to abort the operation. The supervisor will also have to deal with the case where the results arrive after the timeout has expired.
Also note that Scala cannot "leak" resources as such because it implements garbage collection. If resources are accumulating somewhere it is because a reference to that resource is being held by an object.
I am trying to use a hierarchy of Akka actors to handle per user state. There is a parent actor that owns all the children, and handles the get-or-create in the correct way (see a1, a2):
class UserActorRegistry extends Actor {
override def Receive = {
case msg# DoPerUserWork(userId, _) =>
val perUserActor = getOrCreateUserActor(userId)
// perUserActor is live now, but will it receive "msg"?
perUserActor.forward(msg)
}
def getOrCreateUserActor(userId: UserId): ActorRef = {
val childName = userId.toActorName
context.child(childName) match {
case Some(child) => child
case None => context.actorOf(Props(classOf[UserActor], userId), childName)
}
}
In order to reclaim memory, the UserActors expire after a period of idleness (i.e. a timer triggers the child actor to call context.stop(self)).
My problem is that I think I have a race condition between the "getOrCreateUserActor" and the child actor receiving the forwarded message -- if the child expires in that window then the forwarded message will be lost.
Is there any way I can either detect this edge case, or refactor the UserActorRegistry to preclude it?
I can see two problems with your current design that open yourself up to the race condition you mention:
1) Having the termination condition (timer sending a poison pill) go directly to the child actor. By taking this approach, the child can certainly be terminated on a separate thread (within the dispatcher) while at the same time, a message has been setup to be forwarded to it in the UserActorRegistry actor (on a different thread within the dispatcher).
2) Using a PoisonPill to terminate the child. A PoisonPill is for a graceful stop, allowing for other messages in the mailbox to be processed first. In your case, you are terminating due to inactivity, which seems to indicate no other messages already in the mailbox. I see a PoisonPill as wrong here because in your case, another message might be sent after the PosionPill and that message would surely be lost after the PoisonPill is processed.
So I'm going to suggest that you delegate the termination of the inactive children to the UserActorRegistry as opposed to doing it in the children themselves. When you detect the condition of inactivity, send a message to the instance of UserActorRegistry indicating that a particular child needs to be terminated. When you receive that message, terminate that child via stop instead of sending a PoisonPill. By using the single mailbox of the UserActorRegistry which is processed in a serial manner, you can help ensure that a child is not about to be terminated in parallel while you are about to send it a message.
Now, there is a complication here that you have to deal with. Stopping an actor is asynchronous. So if you call stop on a child, it might not be completely stopped when you are processing a DoPerUserWork message and thus might send it a message that will be lost because it's in the process of stopping. You can solve this by keeping some internal state (a List) that represents children that are in the process of being stopped. When you stop a child, add its name to that list and then setup DeathWatch (via context watch child) on it. When you receive the Terminated event for that child, remove it's name from the list of children being terminated. If you receive work for a child while its name is in that list, requeue it for re-processing, maybe up to a max number of times so as to not try and reprocess forever.
This is not a perfect solution; it's just an identification of some of the issues with your approach and a push in the right direction for solving some of them. Let me know if you want to see the code for this and I'll whip something together.
Edit
In response to your second comment. I don't think you'll be able to look at a child ActorRef and see that it's currently shutting down, thus the need for that list of children that are in the process of being shutdown. You could enhance the DoPerUserWork message to contain a numberOfAttempts:Int field and increment this and send back to self for reprocessing if you see the target child is currently shutting down. You could then use the numberOfAttempts to prevent re-queuing forever, stopping at some max number of attempts. If you don't feel completely comfortable relying on DeathWatch, you could add a time-to-live component to the items in the list of children shutting down. You could then prune items as you encounter them if they are in the list but have been in there too long.
I'm just starting to learn Akka Actors in Scala. My understanding is that messages received by an Actor are queued in an Actor's mailbox, and processed one at a time. By processing messages one at a time, concurrency issues (race conditions, deadlocks) are mitigated.
But what happens if the Actor creates a future to do the work associated with a message? Since the future is async, the Actor could begin processing the next several messages while the future associated with the prior message is still running. Wouldn't this potentially create race conditions? How can one safely use futures within an Actor's receive() method to do long running tasks?
The safest way to use futures within an actor is to only ever use pipeTo on the future and send its result as a message to an actor (possibly the same actor).
import akka.pattern.pipe
object MyActor {
def doItAsynchronously(implicit ec: ExecutionContext): Future[DoItResult] = {
/* ... */
}
}
class MyActor extends Actor {
import MyActor._
import context.dispatcher
def receive = {
case DoIt =>
doItAsynchronously.pipeTo(self)
case DoItResult =>
// Got a result from doing it
}
}
This ensures that you won't mutate any state within the actor.
Remember two things
The notion behind the term Future(a special actor) is that we can create an actor for any result while it(the result) is still being computed, started or finished but we can't have an address for that special actor or future.
Suppose I want to buy something (my result is buying something, and the process it to initiate steps to start buying procedure) we can create an actor for the result (buy) but if there is any problem and we can't buy the thing we will have an exception instead of the result.
Now how the above two things fit is explained below-
Say we need to compute the factorial of 1 billion we may think that it will take a lot of time to compute but we get the Future immediately it will not take time to produce the Future (" Since the future is async, the Actor could begin processing the next several messages while the future associated with the prior message is still running."). Next, we can pass this future, store it and assign it.
Hope so you understand what I'm trying to say.
Src : https://www.brianstorti.com/the-actor-model/
If you need to mutate state in futures without blocking incoming messages you might want to reconsider redesigning your actor model. I would introduce separate actors for each task you would use those futures on. After all, an actor's main task is to maintain its state without letting it escape thus providing safe concurrency. Define an actor for those long running task whose responsibility is only to take care of that.
Instead of taking care of the state manually you might want to consider using akka's FSM so you get a much cleaner picture of what changes when. I presonally prefer this approach to ugly variables when I'm dealing with more complex systems.
I'm currently sending a message to an Akka actor every second for it to perform a task.
If that task (function) is still running when a new message is received by the actor, I would like the actor to do nothing. Basically, I want the actor function to be run only if it's not already running.
What's a good way to do this? Should I use Akka actors or do it another way?
Cheers
Actors process a message at a time. The situation you describe cannot happen.
Akka actors process their messages asynchronously one-by-one, so you can only drop/ignore "expired" messages to avoid extra processing and OutOfMemoryException-s due to actor's Mailbox overflow.
You can ignore expired (more than 1 second old in your case) messages inside your actor:
case class DoWork(createdTime: Long = System.currentTimeMillis)
final val messageTimeout = 1000L // one second
def receive = {
case DoWork(createdTime) =>
if((System.currentTimeMillis - createdTime) < messageTimeout) { doWork() }
}
Or you can create a custom Mailbox which can drop expired messages internally.
Of course, as Robin Green already mentioned, actors in general are not supposed to run long-running operations internally so this approach can be suitable only if your actor doesn't need to process other kind of messages (they will not be processed in time). And in case of high CPU demands you may consider to move your actor on a separate dispatcher.
Newbie question of Akka - I'm reading over Akka Essentials, could someone please explain the difference between Akka Stop/Poison Pill vs. Kill ? The book offers just a small explaination "Kill is synchronous vs. Poison pill is asynchronous." But in what way? Does the calling actor thread lock during this time? Are the children actors notified during kill, post-stop envoked, etc? Example uses of one concept vs. the other?
Many thanks!
Both stop and PoisonPill will terminate the actor and stop the message queue. They will cause the actor to cease processing messages, send a stop call to all its children, wait for them to terminate, then call its postStop hook. All further messages are sent to the dead letters mailbox.
The difference is in which messages get processed before this sequence starts. In the case of the stop call, the message currently being processed is completed first, with all others discarded. When sending a PoisonPill, this is simply another message in the queue, so the sequence will start when the PoisonPill is received. All messages that are ahead of it in the queue will be processed first.
By contrast, the Kill message causes the actor to throw an ActorKilledException which gets handled using the normal supervisor mechanism. So the behaviour here depends on what you've defined in your supervisor strategy. The default is to stop the actor. But the mailbox persists, so when the actor restarts it will still have the old messages except for the one that caused the failure.
Also see the 'Stopping an Actor', 'Killing an Actor' section in the docs:
http://doc.akka.io/docs/akka/snapshot/scala/actors.html
And more on supervision strategies:
http://doc.akka.io/docs/akka/snapshot/scala/fault-tolerance.html
Use PoisonPill whenever you can. It is put on the mailbox and is consumed like any other message. You can also use "context.stop(self)" from within an actor.
PoisonPill asynchronously stops the actor after it’s done with all messages that were received into mailbox, prior to PoisonPill.
You can use both actor stop and poison pill to stop processing of actors, and kill to terminate the actor in its entirety.
x.stop is a call you make in akka receive method, will only replace actor state with new actor after calling postStop.
x ! PoisonPill is a method that you pass to actor to stop processing when the actor is running(Recommended). will also replace actor state after calling postStop.
x.kill will terminate the actor and will remove the actor in the actor Path and replace the entire actor with a new actor.