Does the exception generated by a function within the child actor have to be caught and thrown by the child actor explicitly or will the supervisor strategy directives(escalate) take care of percolating the generated exception to the supervisor actor behind the scenes?
My supervisor strategy:
override val supervisorStrategy =
OneForOneStrategy(maxNrOfRetries = 5, withinTimeRange = 5 minute) {
case _: ArithmeticException ⇒ Resume
case _: NullPointerException ⇒ Restart
case _: IllegalArgumentException ⇒ Stop
case _: IOException ⇒ Stop
case _: Exception ⇒ Restart
}
And some of the operations within the child actor could possibly throw an IOException. Should I put a try catch block in the child actor to catch it and then throw it so that the supervisor can catch it? Or will akka take care of it behind the scenes?
In case you have nothing to lose, let it crash. It should be the default behaviour. But unfortunately sometimes you need to allocate resources or you need to information about what happened when things went wrong.
Personally I usually catch all the exceptions I would like to inform about what is going on when I am creating endpoints or web applications.
Most of the time it's best to "let it crash" and choose the correct supervisor strategy for your situation. If you want to keep the internal state, just choose resume. Cleaning up resources before restarting can be done in the preRestart method.
Akka has sophisticated ways of handling exceptions, why not use them? There is no reason to catch an exception and then throw it again.
Related
I am trying to implement fault tolerance within my actor system for my Scala project, to identify errors and handle them. I am using Classic actors. Each supervisor actor has 5 child actors, if one of these child actors fails, I want to restart that actor, and log the error, and as I mentioned, handle the problem that caused this actor to fail.
I implemented a One-For-One SupervisorStrategy in my Supervisor actor class as such:
override val supervisorStrategy =
OneForOneStrategy(maxNrOfRetries = 5, withinTimeRange = 1.minute) {
case e: ArithmeticException =>
logger.error(s"Supervisor: $e from $sender; Restarting!")
Restart
case e: NullPointerException =>
logger.error(s"Supervisor: $e from $sender; Restarting!")
Restart
case e: IllegalArgumentException =>
logger.error(s"Supervisor: $e from $sender; Restarting!")
Restart
case _: Exception =>
logger.error(s"Supervisor: Unknown exception from $sender; Escalating!")
Restart
}
I added a throw new NullPointerException in one of the methods I know is called in every actor, as the system never/rarely fails. But, in the log file, I do not get what I expect, but the following:
13:41:19.893 [ClusterSystem-akka.actor.default-dispatcher-21] ERROR akka.actor.OneForOneStrategy - null
java.lang.NullPointerException: null
I have looked at so many examples, as well as the Akka documentation for fault tolerance for both types and classic actors, but I cannot get this to work.
EDIT
After doing some more research, I saw that a potential reason for the error I am getting could be caused by an infinite cycle in the child actor, whereby it throws an error, restarts, throws an error etc.
So I took a different approach, where in the supervisor actor, I added a variable sentError = false, then in one of the supervisor methods that tells the child actors to start working, I added the following:
if(!errorSent){
errorSent = true
actor ! new NullPointerException
}
And when a child actor receives an error, they throw it. I took this approach so that this only ever happens once, to avoid the cycle. However, unfortunately this did not change the output, and I still got the error mentioned above.
In our project, we need to do a task which listens to a queue and process the coming messages in a loop, which is never finished. The code is looking like:
def processQueue = {
while(true) {
val message = queue.next();
processMessage(message) match {
case Success(_) => ...
case _ => ...
}
}
}
So we want to run it in a separate thread.
I can imagine two ways to do it, one is to use Thread as what we do in Java:
new Thread(new Runnable() { processQueue() }).start();
Another way is use Future (as we did now):
Future { processQueue }
I just wonder if is it correct to use Future in this case, since as I know(which might be wrong), Future is mean to be running some task which will finish or return a result in some time of the future. But our task is never finished.
I also wonder what's the best solution for this in scala.
A Future is supposed to a value that will eventually exist, so I don't think it makes much sense to create one that will never be fulfilled. They're also immutable, so passing information to them is a no-no. And using some externally referenced queue within the Future sounds like dark road to go down.
What you're describing is basically an Akka Actor, which has it's own FIFO queue, with a receive method to process messages. It would look something like this:
import akka.actor._
class Processor extends Actor {
def receive = {
case msg: String => processMessage(msg) match {
case Success(x) => ...
case _ => ...
}
case otherMsg # Message(_, _) => {
// process this other type of message..
}
}
}
Your application could create a single instance of this Processor actor with an ActorSystem (or some other elaborate group of these actors):
val akkaSystem = ActorSystem("myActorSystem")
val processor: ActorRef = akkaSystem.actorOf(Props[Processor], "Processor")
And send it messages:
processor ! "Do some work!"
In short, it's a better idea to use a concurrency framework like Akka than to create your own for processing queues on separate threads. The Future API is most definitely not the way to go.
I suggest perusing the Akka Documentation for more information.
If you are just running one thread (aside from the main thread), it won't matter. If you do this repeatedly, and you really want lots of separate threads, you should use Thread since that is what it is for. Futures are built with the assumption that they'll terminate, so you might run out of pool threads.
I am developing a server which is an intermediate server between 2 other endpoints.
So I implement a spray route which triggers a computation, and this computation invokes an actor which invokes a spray client.
The following pattern applies:
Client => ...=> (Spray Route => Service Actor => Spray Client) =>... => distant Server.
I am developing the exception management and I have a hard time understanding what to do.
When the distant server sends me BadRequest, or any error code, I want to throw an exception, which leads answering to the client the same kind of errors.
I guess my question is general about exception handling and excalation.
I naively believed that when I throw an exception in a future, the future could call the case Failure:
def receive = {
case SendRequest => {
val s = sender()
val call = for {
request <- ComputeRequest
result <- CallSprayClient ? request
} yield result
call onComplete {
case Success(succ) => s ! succ
case Failure(e) = throw e
}
}
}
The thing is, when ComputeRequest or CallSprayClient throw an exception, the Failure case of my callback is not triggered.
I looked at the supervision pattern, but it seems that the exception or message causing the error is not propagated neither.
In my particular case, depending on the exception, I would like to send a different http response to my client, hence the need to escalate.
What development pattern should I apply?
Thanks.
I am currently evaluating javascript scripts using Rhino in a restful service. I wish for there to be an evaluation time out.
I have created a mock example actor (using scala 2.10 akka actors).
case class Evaluate(expression: String)
class RhinoActor extends Actor {
override def preStart() = { println("Start context'"); super.preStart()}
def receive = {
case Evaluate(expression) ⇒ {
Thread.sleep(100)
sender ! "complete"
}
}
override def postStop() = { println("Stop context'"); super.postStop()}
}
Now I run use this actor as follows:
def run {
val t = System.currentTimeMillis()
val system = ActorSystem("MySystem")
val actor = system.actorOf(Props[RhinoActor])
implicit val timeout = Timeout(50 milliseconds)
val future = (actor ? Evaluate("10 + 50")).mapTo[String]
val result = Try(Await.result(future, Duration.Inf))
println(System.currentTimeMillis() - t)
println(result)
actor ! PoisonPill
system.shutdown()
}
Is it wise to use the ActorSystem in a closure like this which may have simultaneous requests on it?
Should I make the ActorSystem global, and will that be ok in this context?
Is there a more appropriate alternative approach?
EDIT: I think I need to use futures directly, but I will need the preStart and postStop. Currently investigating.
EDIT: Seems you don't get those hooks with futures.
I'll try and answer some of your questions for you.
First, an ActorSystem is a very heavy weight construct. You should not create one per request that needs an actor. You should create one globally and then use that single instance to spawn your actors (and you won't need system.shutdown() anymore in run). I believe this covers your first two questions.
Your approach of using an actor to execute javascript here seems sound to me. But instead of spinning up an actor per request, you might want to pool a bunch of the RhinoActors behind a Router, with each instance having it's own rhino engine that will be setup during preStart. Doing this will eliminate per request rhino initialization costs, speeding up your js evaluations. Just make sure you size your pool appropriately. Also, you won't need to be sending PoisonPill messages per request if you adopt this approach.
You also might want to look into the non-blocking callbacks onComplete, onSuccess and onFailure as opposed to using the blocking Await. These callbacks also respect timeouts and are preferable to blocking for higher throughput. As long as whatever is way way upstream waiting for this response can handle the asynchronicity (i.e. an async capable web request), then I suggest going this route.
The last thing to keep in mind is that even though code will return to the caller after the timeout if the actor has yet to respond, the actor still goes on processing that message (performing the evaluation). It does not stop and move onto the next message just because a caller timed out. Just wanted to make that clear in case it wasn't.
EDIT
In response to your comment about stopping a long execution there are some things related to Akka to consider first. You can call stop the actor, send a Kill or a PosionPill, but none of these will stop if from processing the message that it's currently processing. They just prevent it from receiving new messages. In your case, with Rhino, if infinite script execution is a possibility, then I suggest handling this within Rhino itself. I would dig into the answers on this post (Stopping the Rhino Engine in middle of execution) and setup your Rhino engine in the actor in such a way that it will stop itself if it has been executing for too long. That failure will kick out to the supervisor (if pooled) and cause that pooled instance to be restarted which will init a new Rhino in preStart. This might be the best approach for dealing with the possibility of long running scripts.
I want to use Akka actors as a mailbox only, i.e. I want to create n threads, which each create 1 remote actor.
Each thread obtains a reference to all the remote actors of the other threads, such that they can send messages to eachother via their respective actors.
The actor is defined as follows:
case class Receive
case class GroupReceive(id: Symbol)
case class GroupMsg[T](id: Symbol, msg: T)
class FooParActor(val distributor: Distributor) extends Actor
with Stash {
import context._
val globalRank: Int = distributor.globalRank
def doReceive(realSender: ActorRef, ID: Symbol) {
unstashAll()
become({
case GroupMsg(ID, msg) =>
realSender ! msg
unbecome()
case GroupMsg(otherId, msg) =>
println(globalRank + ": stashing " + otherId)
unbecome()
case x => sys.error("bad msg: " + x)
}, discardOld = false)
}
def receive = {
case GroupReceive(id) =>
doReceive(sender, id)
case GroupMsg(id, x) =>
stash()
case x => sys.error("bad msg: " + x)
}
}
To read a message, the owner-thread sends GroupReceive('someSymbol) to his local actor, which in turn forwards a GroupMsg to the thread. The code from the thread's point of view to read a message looks like this:
def groupRcv[T](id:Symbol) = Await.result(aref ? GroupReceive(id), timeout.duration).asInstanceOf[T]
where aref is a reference to the local actor of this thread.
I sometimes experience deadlocks(timeouts of 5 seconds) with the above pattern, even with extremely simple usage and small messages. I narrowed the problem down to actors stalling after receiving the GroupReceive(id) message, but before entering the first case of doReceive(...): case GroupMsg(ID, msg) =>.
I made printout-traces to check that the actors actually have messages in the stash before they go to the doReceive call, and it seems that for some reason, they just don't handle them. Can the code I presented above go to a state where a GroupMsg() gets lost from a FooParActor's stash? Or is there any other way the actor can go to a deadlock after receiving a GroupReceive() message?
You are using Await.result() but not sharing where you do that: if you call groupRcv on a thread on which your actors are supposed to be running then you can of course have starvation (i.e. the target actor does not have a thread available to run on, so it will never complete the request).
It seems that you are mixing thread-based concurrency with actors in an unhealthy fashion, but since you only hint at it and don’t show the code I can only give you the broad advice to not do that. When programming actors, forget about threads; those are managed by Akka. In particular do not misuse Akka’s threads (i.e. the Await.result would probably work on your own external thread pool, although there is nearly always a better alternative).
In the end, if you are using actors just to make “a thread with a mailbox” then Akka cannot help you and you will run into all the usual traditional concurrency pitfalls.