In my scala application I have an actor A that is the child of the guardian actor. The guardian actor has been given a one for all strategy to escalate on children shutdown.
Under various circumstances, the actor A calls context stop self that usually results in the entire application exiting, which is the desired behavior.
However, I recently observed a scenario where context stop self resulted in an error
Failed to run termination callback, due to [Futures timed out after [5000 milliseconds]]
..and the actor system did not shutdown.
The application does sometimes have hundreds of thousands of actors under Actor A, and I wonder if shutting them down caused some sort of timeout in the termination procedure.
Can anyone help me understand why the behavior described above would happen and should I be using context system shutdown instead to shutdown the system?
Related
I know actors are persistent, They are always alive sitting idle as long as there is nothing to do(no messages in the mailbox), So who monitors the mailbox ?
Is it the Actor itself ? but it doesn't have a thread assigned from dispatcher until its having a message.?
Or, is there any background daemon running monitoring mailboxes of each actors?
As #Sarvesh Kumar Singh mentioned, actor is not by default persistent. You can create persistent actor by extending PersistentActor: Akka Persistence.
In fact, when a actor(cell) is created in the actor system hierarchy, its mailbox(Mailbox is a subclass of Runnable) will be attached to the dispatcher, when the actor is started, the dispatcher will then execute the mailbox runnable(defined in the run method) by polling message from the mailbox queue, if there's a message, it will trigger the defined actor receive method to actually handle the message. Message looping(during this run, the messages in the queue will the be handled into batch, if handling this batch of batch doesn't pass the limit of dispatcher throughput configuration) is also done in the run method by a recursive call of processMailBox defined in mailbox. At the end of each round, the mail box will attach itself to the dispatcher thread pool for execution if there's messages the actor mail box is scheduled for execution the the loop continues. If not, the loop is broken, the next run will be scheduled when there's message sent to the actor, at this moment, message will be dispatched via the dispatcher attached with the actor cell when you call actorRef ! message, this will make the dispatcher to schedule the message handling, check this out: Dispatch::sendMessage and Dispatcher::dispatch. As explained in Dispatchers, dispatcher's throughput can be configured to determine the maximum number of messages to be processed per actor before the thread jumps to the next actor.
I'm new to akka-actor and confused with some problems:
when I create an actorSystem, and use actorOf(Props(classOf[AX], ...)) to create actor in main method, how many instances are there for my actor AX?
If the answer to Q1 was just one, does this mean whatever data-structure I created in the AX actor class's definition will only appear in one thread and I should not concern about concurrency problems?
What if one of my actor's action (one case in receive method) is a time consuming task and would take quite long time to finish? Will my single Actor instance not responding until it finish that task?
If the answer to Q3 is right, what I am supposed to do to prevent my actor from not responding? Should I start another thread and send another message back to it until finish the task? Is there a best practice there I should follow?
yes, the actor system will only create 1 actor instance for each time you call the 'actorOf' method. However, when using a Router it is possible to create 1 router which spreads the load to any number of actors. So in that case it is possible to construct multiple instances, but 'normally' using actorOf just creates 1 instance.
Yes, within an actor you do not have to worry about concurrency because Akka guarantees that any actor only processes 1 message at the time. You must take care not to somehow mutate the state of the actor from code outside the actor. So whenever exposing the actor state, always do this using an immutable class. Case classes are excellent for this. But also be ware of modifying the actor state when completing a Future from inside the actor. Since the Future runs on it's own thread you could have a concurrency issue when the Future completes and the actor is processing a next message at the same time.
The actor executes on 1 thread at the time, but this might be a different thread each time the actor executes.
Akka is a highly concurrent and distributed framework, everything is asynchronous and non-blocking and you must do the same within your application. Scala and Akka provide several solutions to do this. Whenever you have a time consuming task within an actor you might either delegate the time consuming task to another actor just for this purpose, use Futures or use Scala's 'async/await/blocking'. When using 'blocking' you give a hint to the compiler/runtime a blocking action is done and the runtime might start additional thread to prevent thread starvation. The Scala Concurrent programming book is an excellent guide to learn this stuff. Also look at the concurrent package ScalaDocs and Neophyte's Guide to Scala.
If the actor really has to wait for the time consuming task to complete, then yes, your actor can only respond when that's finished. But this is a very 'request-response' way of thinking. Try to get away from this. The actor could also respond immediately indicating the task has started and send an additional message once the task has been completed.
With time consuming tasks always be sure to use a different threadpool so the ActorSystem will not be blocked because all of it's available threads are used up by time consuming tasks. For Future's you can provide a separate ExecutionContext (do not use the ActorSystem's Dispatch context for this!), but via Akka's configuration you can also configure certain actors to run on a different thread pool.
See 3.
Success!
one instance (if you declare a router in your props then (maybe) more than one)
Yes. This is one of the advantages of actors.
Yes. An Actor will process messages sequentially.
You can use scala.concurrent.Future (do not use actor state in the future) or delegate the work to a child actor (the main actor can manage the state and can respond to messages). Future or child-actor depends on use case.
It is said:
Akka ensures that each instance of an actor runs in its own lightweight thread and that messages are processed one at a time.
Can you please explain what is the reason of processing messages one at a time in an Actor?
This way we can guarantee thread safety inside an Actor.
Because an actor will only ever handle one message at any given time, we can guarantee that accessing the actor's local state is safe to access, even though the Actor itself may be switching Threads which it is executing on. Akka guarantees that the state written while handling message M1 are visible to the Actor once it handles M2, even though it may now be running on a different thread (normally guaranteeing this kind of safety comes at a huge cost, Akka handles this for you).
It also originates from the original Actor model description, which is an concurrency abstraction, described as actors who can only one by one handle messages and respond to these by performing one of these actions: send other messages, change it's behaviour or create new actors.
I am confused by behavior I am seeing in Akka. Briefly, I have a set of actors performing scientific calculations (star formation simulation). They have some state. When an error occurs such that one or more enter an invalid state, I want to restart the whole set to start over. I also want to do this if a single calc (over the entire set) takes too long (there is no way to predict in advance how long it may run).
So, there is the set of Simulation actors at the bottom of the tree, then a Director above them (that creates them via a Router, and sends them messages via that Router as well). There is one more Director level above that to create Directors on different machines and collect results from them all.
I handle the timeout case by using the Akka Scheduler to create a one-time timeout event, in the local Director, when the simulation is started. When the Director gets this event, if all its Simulation actors have not finished, it does this:
children ! Broadcast(Kill)
where children is the Router that owns/created them - this sends a Kill to all the children (SimulActors).
What I thought would occur is that all the child actors would be restarted. However, their preRestart() hook method is never called. I see the Kill message received, but that's it.
I must be missing something fundamental here. I have read the Akka docs on this topic and I have to say I find them less than clear (especially the page on Supervisors). I would really appreciate either a thorough explanation of the Kill/restart process, or just some other references (Google wasn't very helpful).
Note
If the child of a router terminates, the router will not automatically
spawn a new child. In the event that all children of a router have
terminated the router will terminate itself.
Taken from the akka docs.
I would consider using a supervision strategy - akka has behavior built in for killing all actors (all for one strategy) and you can define the specific strategy - eg restart.
I think a more idiomatic way to run this would be to have the actors throw x exception if they're not done after a period of time and then the supervisor handle that via supervision strategy.
You could throw a not done exception from the child and then define the behaviour like so:
override val supervisorStrategy =
AllForOneStrategy(maxNrOfRetries = 0) {
case _: NotDoneException ⇒ Stop
case _: Exception ⇒ Restart
}
It's important to understand that a restart means stopping the old actor and creating a new separate object/Actor
References:
http://doc.akka.io/docs/akka/snapshot/scala/fault-tolerance.html
http://doc.akka.io/docs/akka/snapshot/general/supervision.html
Newbie question of Akka - I'm reading over Akka Essentials, could someone please explain the difference between Akka Stop/Poison Pill vs. Kill ? The book offers just a small explaination "Kill is synchronous vs. Poison pill is asynchronous." But in what way? Does the calling actor thread lock during this time? Are the children actors notified during kill, post-stop envoked, etc? Example uses of one concept vs. the other?
Many thanks!
Both stop and PoisonPill will terminate the actor and stop the message queue. They will cause the actor to cease processing messages, send a stop call to all its children, wait for them to terminate, then call its postStop hook. All further messages are sent to the dead letters mailbox.
The difference is in which messages get processed before this sequence starts. In the case of the stop call, the message currently being processed is completed first, with all others discarded. When sending a PoisonPill, this is simply another message in the queue, so the sequence will start when the PoisonPill is received. All messages that are ahead of it in the queue will be processed first.
By contrast, the Kill message causes the actor to throw an ActorKilledException which gets handled using the normal supervisor mechanism. So the behaviour here depends on what you've defined in your supervisor strategy. The default is to stop the actor. But the mailbox persists, so when the actor restarts it will still have the old messages except for the one that caused the failure.
Also see the 'Stopping an Actor', 'Killing an Actor' section in the docs:
http://doc.akka.io/docs/akka/snapshot/scala/actors.html
And more on supervision strategies:
http://doc.akka.io/docs/akka/snapshot/scala/fault-tolerance.html
Use PoisonPill whenever you can. It is put on the mailbox and is consumed like any other message. You can also use "context.stop(self)" from within an actor.
PoisonPill asynchronously stops the actor after it’s done with all messages that were received into mailbox, prior to PoisonPill.
You can use both actor stop and poison pill to stop processing of actors, and kill to terminate the actor in its entirety.
x.stop is a call you make in akka receive method, will only replace actor state with new actor after calling postStop.
x ! PoisonPill is a method that you pass to actor to stop processing when the actor is running(Recommended). will also replace actor state after calling postStop.
x.kill will terminate the actor and will remove the actor in the actor Path and replace the entire actor with a new actor.