How to persist actor state with high rate of messages - scala

Question

I want to start using actors with heavy rate messages . actor's last state is very important
I was following the persistence example shown at here http://doc.akka.io/docs/akka/2.3.9/scala/persistence.html#event-sourcing
I tried to send a heavy load of messages
for (i <-0 to 100000){
persistentActor ! Cmd("foo"+i)
}
and using the persistAsync like this
val receiveCommand: Receive = {
case Cmd(data) =>
persistAsync(Evt(s"${data}-${numEvents}"))(updateState)
case "snap" => saveSnapshot(state)
case "print" => println(state)
}
before shutdown I added Thread.sleep(150000) to make sure that all persists . at first all seems to be ok , however re-running the app shows that some are going to dead-letter
> [INFO] [02/03/2015 15:35:18.187]
> [example-akka.actor.default-dispatcher-3]
> [akka://example/user/persistentActor-4-scala] Message
> [java.lang.String] from Actor[akka://example/deadLetters] to
> Actor[akka://example/user/persistentActor-4-scala#1206460640] was not
> delivered. [1] dead letters encountered. This logging can be turned
> off or adjusted with configuration settings 'akka.log-dead-letters'
> and 'akka.log-dead-letters-during-shutdown'. [INFO] [02/03/2015
> 15:35:18.192] [example-akka.actor.default-dispatcher-3]
> [akka://example/user/persistentActor-4-scala] Message
> [akka.persistence.Recover] from
> Actor[akka://example/user/persistentActor-4-scala#1206460640] to
> Actor[akka://example/user/persistentActor-4-scala#1206460640] was not
> delivered. [2] dead letters encountered. This logging can be turned
> off or adjusted with configuration settings 'akka.log-dead-letters'
> and 'akka.log-dead-letters-during-shutdown'.
or getting something like :
----------
[INFO] [02/03/2015 15:54:32.732] [example-akka.actor.default-dispatcher-11] [akka://example/user/persistentActor-4-scala] Message [akka.persistence.JournalProtocol$ReplayedMessage] from Actor[akka://example/deadLetters] to Actor[akka://example/user/persistentActor-4-scala#-973984210] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [02/03/2015 15:54:32.735] [example-akka.actor.default-dispatcher-3] [akka://example/user/persistentActor-4-scala] Message [akka.persistence.JournalProtocol$ReplayedMessage] from Actor[akka://example/deadLetters] to Actor[akka://example/user/persistentActor-4-scala#-973984210] was not delivered. [10] dead letters encountered, no more dead letters will be logged. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
A fatal error has been detected by the Java Runtime Environment:
SIGSEGV (0xb) at pc=0x00007fa2a3e06b6a, pid=18870, tid=140335801857792
JRE version: Java(TM) SE Runtime Environment (7.0_71-b14) (build 1.7.0_71-b14)
Java VM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode linux-amd64 compressed oops)
Problematic frame:
V [libjvm.so+0x97bb6a] Unsafe_GetNativeByte+0xaa
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
An error report file with more information is saved as:
/home/tadmin/projects/akka-sample-persistence-scala/hs_err_pid18870.log
If you would like to submit a bug report, please visit:
http://bugreport.sun.com/bugreport/crash.jsp
========================================================================
how can I persist a state of an actor that should handle heavy rate of messages ?

Answer 1

I suspect that the default database, levelDB, is the problem, and is most certainly what is core dumping. Are you by any chance writing to the db with more than one actor concurrently? In my experience, I've seen it core dump in that scenario. You could try it in shared mode, but I just plugged in a different database and the problems went away. In my case I used Cassandra.
I would try an in-memory journal plugin for akka-persistence. It is very easy to swap in. If the problems go away, then you know that levelDB is the problem. If so, go with a different database.