Akka Circuit Breaker sharing between actors - scala

I have a shared external resource (say a file store) which a pool of actors is using. Each time a new request is made to the file store a new actor is created to fill the request with a reference to the external system passed in.
The current approach where I create a circuit breaker per actor defeats the purpose as a new actor is created for each 'request' which performs a sequence of operations on this external resource.
Not ideal - too many CB instances;
class MySharedResourceActor(externalResourceRef: ExtSystem) extends Actor with ActorLogging {
val breaker = new CircuitBreaker(context.system.scheduler,
maxFailures = 5,
callTimeout = 10.seconds,
resetTimeout = 1.minute).onOpen(notifyMeOnOpen())
def receive = {
case SomeExternalOp =>
breaker.withSyncCircuitBreaker(dangerousCallToExternalSystem()) pipeTo sender()
}
}
Better Approach - pass in a CB ref;
class MySharedResourceActor(externalResourceRef: ExtSystem, val breaker: CircuitBreaker) extends Actor with ActorLogging {
def receive = {
case SomeExternalOp =>
breaker.withSyncCircuitBreaker(dangerousCallToExternalSystem()) pipeTo sender()
}
}
Is it safe to pass in a Circuit-breaker reference from the parent actor which also maintains a reference to the external system and share this circuit breaker between multiple actors in a router pool, dynamically created or otherwise?

Yes it's safe to follow this approach. We share circuit breakers across related actors (pooled or otherwise) that are making http calls to the same host. If you didn't do this, and let each instance have it's own breaker, even if they were long lived instances, each one would need to hit the fail threshold separately before the breaker opened and I doubt this is the behavior you want. By sharing, it allows multiple actors to contribute stats (fails, successes) into the breaker so that the breaker is representative of all calls that have gone into the resource.
In looking at Akka's code, they are using atomics inside of the circuit breaker to represent state and handle state transitions, so they should be safe to use in multiple actors.

Related

Declaring Actor state variables as mutable ones

I am fairly new with Akka framework and Concurrency concepts. And from Akka docs, I understood that only one message in the Actor mailbox would be processed at a time. So single thread would be processing Actor's state at a time. And my doubt is that, so declaring an Actor state/data variable as mutable - 'Var'(Only when 'Val' doesn't fit), will not cause inconsistent Actor states in the case of Concurrency.
I am using Scala for development. In the following Master actor, details of workers is stored in a mutable variable 'workers'. Will it be a problem with concurrency?
class Master extends PersistentActor with ActorLogging {
...
private var workers = Map[String, WorkerState]()
...
}
I think what you are doing is fine. As you said, one of the fundamental guarantees of Akka actors is that a single actor will be handling one message at a time, so there will not be inconsistent Actor states.
Akka actors conceptually each have their own light-weight thread,
which is completely shielded from the rest of the system. This means
that instead of having to synchronize access using locks you can just
write your actor code without worrying about concurrency at all.
http://doc.akka.io/docs/akka/snapshot/general/actors.html
Also, it is a good thing that you're using a var instead of a val with a mutable map :)
Another way to consider coding situations like these is to alter the actor's "state" after each message handled. Eg.:
class Master extends PersistentActor with ActorLogging {
type MyStateType = ... // eg. Map[String, WorkerState], or an immutable case class - of course, feel free to just inline the type...
def receive = handle(initState) // eg. just inline a call to Map.empty
def handle(state: MyStateType): Actor.Receive = LoggingReceive {
case MyMessageType(data) =>
... // processing data - build new state
become(handle(newState))
case ... // any other message types to be handled, etc.
}
... // rest of class implementation
}
While it is true that there is still mutable state happening here (in this case, it is the state of the actor as a whole - it becomes effectively a "non-finite state machine"), it feels better contained/hidden (to me, at least), and the "state" (or "workers") available to the actor for any given message is treated as entirely immutable.

Spray.io - delegate processing to another actor(s)

I implement a REST service using Spray.io framework. Such service must receive some "search" queries, process them and send result back to the client(s). The code that perfrom searching located in separate actor - SearchActor, so after receiving (JSON) query from user, i re-send (using ask pattern) this query to my SearchActor. But what i don't really understand it's how i must implement interaction between spray.io route actor and my SearchActor.
I see here several variants but which one is more correct and why?
Create one instance of SearchActor at startup and send every request to this actor
For every request create new instance of SearchActor
Create pool of SearchActor actors at startup and send requests to this pool
You're not forced to use the ask pattern. In fact, it will create a thread for each of your request and this is probably not what you want. I would recommend that you use a tell instead. You do this by spawning a new Actor for each request (less expensive than a thread), that has the RequestContext in one of its constructor fields. You will use this context to give the response back, typically with its complete method.
Example code.
class RESTActor extends HttpService {
val route = path("mypath") ~ post {
entity(as[SearchActor.Search]) { search => ctx =>
SearchActor(ctx) ! search
}
}
}
case class SearchActor(ctx: RequestContext) {
def receive = {
case msg: Search => //... search process
case msg: Result => ctx.complete(msg) // sends back reply
}
}
Variant #1 is out of question after the initial implementation - you would want to scale out, so single blocking actor is bad.
Variants #2 and #3 are not very different - creation of new actor is cheap and has minimal overhead. As your actors may die often (i.e. backend is not available), i would say that #2 is the way to go.
Concrete implementation idea is shown at http://techblog.net-a-porter.com/2013/12/ask-tell-and-per-request-actors/

Creating an in-memory storage using Akka actors

I'm trying to familiarize myself with Akka using a pet project. This is basically a tournament-style server, having a pool of players. On each round, it should fetch two random players from the pool, play them against each other, then update a scoreboard. The whole thing sits behind an HTTP server. Players are added via an HTTP Post message, and games are triggered via a 'Play' message that's to be sent from a JS client.
The way I thought of implementing it is as follows:
a PlayerPool actor handling two messages, 'AddPlayer' and 'Play'
a Round actor that takes a message with two players, plays a round, then updates the
GameResults actor, containing a list of played rounds and the winner of each round. It also sports the 'GetScore' message which returns the scoreboard.
The HTTP layer is a Spray Routing actor, exposing calls for 'GetScore', 'AddPlayer' and 'Play', talking to the PlayerPool and GameResults actors.
My problem is that two separate actors are talking to GameResults - both Spray directly, and the Round actor, and apparently two separate instances of GameResults are being created, so that while Round adds a result to one instance, Spray is reading the scoreboard from another instance. Obviously I'm missing something very basic and before attempting to resolve the issue, I'd like to understand what's the canonic way of doing this in Akka? basically this problem can be reduced to an actor that holds a state across different calls.
Would be glad to have someone point me in the right direction.
code snippet example how to pass messages from spray to your game result actor :
hope this helps
object SomeApp extends App{
//create a actor system
val yourSystem = ActorSystem("Your-System")
//init spray actor
val restHandler = yourSystem.actorOf(Props[RestHandler], name = "Rest-spray")
//init play pool actor
val playerPoolActor = yourSystem.actorOf(Props[PlayerPool], name = "playerPool")
//INIT ONLY 1 game result actor
val gameResultsActor = yourSystem.actorOf(Props[GameResults], name = "gameResults")
//spray listen to all IP on port 90210
IO(Http)(yourSystem) ! Http.Bind(restHandler, interface = "0.0.0.0" , port = 90210)
class RestHandler extends Actor with RestHandlerRoute {
def actorRefFactory = context
//nice to hold the route in a diffrent trait
def receive = runRoute(someRoute)
}
}
//only thing the trait is doing is configuring the routing and send message to relavant actor
trait RestHandlerRoute extends HttpService{me: Actor =>
import SomeApp._
val someRoute =
path("getGameResults") { get { respondWithMediaType(`text/html`) { complete {
//case the user requested http://someIP:90210/getGameResults spray will send a message to the game result actor
gameResultsActor ! getResult
} ~
}

How to represent that an Akka actor is in a long running operation using FSM-s

My actor that is described with an FSM is waiting for a trigger (in Idle state). When it gets it it starts processing some data (and goes to Running state) and when it is done it goes back to Idle state.
If I understand the FSM model correctly, from this point of view there were two events:
processing started (Idle->Running) and processing completed (Running->Idle).
But from the actor point of view there was only one message.
One possibility is to delegate the processing itself to another actor. So I can forward the triggering event and go to Running state, then when got the result I go to Idle.
It has the advantage that the FSM itself can quickly react to requests (e.g. asking what's the current state), but it makes the design more complicated.
Another one is to send a completed message to self when the actor has completed processing, which will trigger the Running -> Idle transition, but it looks a bit strange for me.
What other options do I have?
Note: there are several other states with various transitions so I'd like to stick to the FSM model.
Since it seems like you have an actor that needs to do processing and make transitions to an FSM, I suggest you follow these guidelines (some basic code outline follows this list):
Actor A receives the message in question that will trigger some processing
Actor A transitions a separate FSM actor, say F, (akka.actor.FSM) to the appropriate state. Actor A on startup would spawn the specific FSM to track the state for the corresponding context (e.g. for all transactions or state per transaction or some other context). The code outline below uses all processing or completed transactions as the context for the example but may need to be changed.
Actor A then triggers whatever processing should be triggered for the message. Remember actors shouldn't block generally, but here is an answer that provides more guidelines on when an Akka actor may block.
Alternative: If you can trigger long running processing without blocking and ensure you receive the necessary events after processing stages by the other party, then you could eliminate the front Actor A and just have the FSM actor F. You should look at onTransition in this case.
So my code outline suggestion is based on what I understood from the question is:
/* Events */
sealed trait MyEvents
case class ProcessingStarted(txnId: Long) extends MyEvents
case class ProcessingFinished(txnId: Long, result: Result) extends MyEvents
/* Valid states for your FSM */
sealed trait MyStates
case object Idle extends MyStates
/* Constructor arguments could be anything, I randomly chose a Long for a transaction ID which may be specific to a job */
case object Processing extends MyStates
/* define more case classes or objects depending on the rest of the states */
/* Valid internal state data types for FSM */
sealed trait MyDataTypes
case object Uninitialized extends MyDataTypes
case class StateData(processingIds: Seq[Long], resultMap: Map[Long, Result]) extends MyDataTypes
import akka.actor.{ Actor, ActorRef, FSM }
import scala.concurrent.duration._
class ActorF extends FSM[MyStates, MyDataTypes] {
startWith(Idle, Uninitialized)
when(Idle) {
case Event(ProcessingStarted(txnId), Uninitialized) =>
goto(Processing) using StateData(Seq(txnId), Map.empty[Long, Result])
case Event(ProcessingStarted(txnId), StateData(emptyIds, resultMap)) =>
goto(Processing) using StateData(Seq(txnId), resultMap)
}
when(Processing) {
case Event(ProcessingFinished(txnId, result), StateData(processingIds, resultMap)) => {
val remainingIds = processingIds diff Seq(txnId)
val newResultMap = resultMap + (txnId -> result)
if (remainingIds.isEmpty) {
goto(Idle) using StateData(remainingIds, newResultMap)
} else {
stay using StateData(remainingIds, newResultMap)
}
}
}
initialize()
}
// inside Actor A do something like this for creating the FSM actor (just like other actors)
val f = system.actorOf(Props(classOf[ActorF], this))
// send an event message to it just like other types of actors
f ! ProcessingStarted(txnId)
If you choose to trigger off non-blocking processing requests to other parts of your code you can use onTransition to add the trigger code in as necessary. You may also want to update the MyEvents case classes to a different tense. The event naming was used above was to show that something else was responsible for triggering that off (e.g. Actor A the received the initial message to do some things).
Also take note of Supervision capabilities of Akka which could be used here to supervise the actors in question.
For more details read the following which might help further with building your FSM, testing it, using non-blocking methods to trigger long-running processing externally. All of which might be useful for your needs:
Scala docs for Akka 2.2.x: FSMs
Scala docs for Akka 2.2.x: Testing
Scala docs for Akka 2.2.x: Akka IO for non-blocking
Scala docs for Akka 2.2.x: ZeroMQ

Akka control threadpool threads

Potentially a very silly question--
Is it possible to customize Akka/Scala actors such that you control the threads that are used by the actors? e.g. can you initialize your own set of threads to be used in the thread pool, or otherwise control/modify the threads?
In Akka, the thread pool is managed via a MessageDispatcher instance. You can set the dispatcher you want to actors easily:
class MyActor( dispatcher: MessageDispatcher ) extends Actor {
self.dispatcher = dispatcher
...
}
To provide your own dispatcher, you can extend akka.dispatch.MessageDispatcher (see existing dispatchers implementation for examples). Here you can play directly with the threads.
Of course, it's dangerous to put business logic inside a dispatcher because it may break the actor model and increase the number of concurrency bugs...
I tried to understand it myself, but seams that guys in Akka don't want thread management to be exposed to public.
ThreadPoolConfig - the class that is responsible for creation of ExecutorService instances is a case class with method createExecutorService() declared final!
final def createExecutorService(threadFactory: ThreadFactory): ExecutorService = {
flowHandler match {
case Left(rejectHandler) ⇒
val service = new ThreadPoolExecutor(...)
service
case Right(bounds) ⇒
val service = new ThreadPoolExecutor(...)
new BoundedExecutorDecorator(service, bounds)
}
}
So, I don't see an easy ways to provide your own ExecutorService.