Akka Cluster - Recover data from a crashed actor - scala

Question

Let's suppose I have a master with a n slaves, where the slaves are located on other computers.
Master actor looks like
class MasterActor extends Actor {
val router: ActorRef = // ... initialized router to contact the slaves
override def receive: Receive = {
case work: DoWork => router ! work
}
}
While my slaves look like
class SlaveActor extends Actor {
override def receive: Receive = {
case work: DoWork => // Some logic that takes a couple of seconds
}
}
In the case where a machine where some slaves are hosted crashes, or if the app has been stopped while processing some work, I expect a fallback mechanism that enable the system (I suppose the master) to be aware of that failure and then redistribute the lost work to the other slaves.
I have understood the principle of supervision in Akka, that enables master to get notified when a child actor is unreachable, but how can I get back the specific instance of work that the actor had to redistribute it ?
As I started using Akka very recently, my approach is perhaps not adapted to the best practices and should I solve this situation in another way ?
Thanks !

Answer 1

Now this approach won't work in every scenario but how about storing the child actor work to a file system. You can create provide an id to each child and save the work in a file based on key -> value pair, where child actor id is the key and value is the work in progress.
Now, what data has to be saved, that you have to decide but this is one of the simplest approach that you can give a try.
Hope this helps!

Answer 2

It sounds like what you want is Akka Persistence. The main model for this is event sourcing
A rough outline would be to persist one event when receiving a DoWork and persist a second event when finishing the work. On recovery, the state is then a list of work not finished (which will, if you get the protocol right be at most one piece of work), along these lines:
case class DidWork(dw: DoWork)
class SlaveActor extends PersistentActor {
val persistenceId: String = ???
var workToDo: List[DoWork] = Nil
val receiveRecover: Receive = {
case dw: DoWork => workToDo = dw :: workToDo
case DidWork(dw: DoWork) => workToDo = workToDo.filter(_ != dw)
case SnapshotOffer(_, snapshot: List[DoWork]) => workToDo = snapshot
}
val snapshotInterval = ???
val receiveCommand: Receive = {
case dw: DoWork =>
persist(dw) { event =>
workToDo = dw :: workToDo
context.system.eventStream.publish(event)
if (lastSequenceNr % snapshotInterval == 0 && lastSequenceNr != 0)
saveSnapshot(workToDo)
doWork
}
}
private[this] def doWork: Unit = {
workToDo.reverse.foreach { dw: DoWork =>
// do the work
persist(DidWork(dw)) { event =>
workToDo = workToDo.tail
context.system.eventStream.publish(event)
if (lastSequenceNr % snapshotInterval == 0 && lastSequenceNr != 0)
saveSnapshot(workToDo)
}
}
}
}

Answer 3

If you want to make sure that the master knows about workers going away you can watch them (see docs https://doc.akka.io/docs/akka/current/actors.html#lifecycle-monitoring-aka-deathwatch ). If you keep track of what workload you have assigned to what worker, that is yet not completed you can implement resending. However you must likely consider the case where the master or it's actor system goes away as well, and how to recover if that happens.
The Akka distributed workers sample covers exactly this so it could be a good source of inspiration: https://developer.lightbend.com/guides/akka-distributed-workers-scala/