Akka Actors app hangs under high volume - scala

Question

I am working on a load generator app using akka actors. The app worked fine for few million requests but when increasing the load to more than say 10 million requests or run the load for a duration ( instead of number of requests) using a infinite loop the application hangs. Below is a simplified implementation and it just prints the command being tested. I also notice that the stats are not logged or the app does'nt shutdown when the time is over. I use the schedulers to dump stats every 30 secs and shutdown the app after 2hrs. Tested with small interval and do not see the processing of the "stats" and "Shutdown" messages.
Any idea what may be causing the application to hang ?
import akka.actor._
import akka.util.duration._
import akka.routing.RoundRobinRouter
import com.test.redload.util.CommandGenerator
import org.apache.log4j.Logger
import akka.util.Duration
class LoadWorker extends Actor {
val log = Logger.getLogger(this.getClass().getName())
def receive = {
case "PUT" => sender ! PUT
case "GET" => sender ! GET
case "DELETE" => sender ! DELETE
case "POST" => sender ! POST
case "HEAD" => sender ! HEAD
}
def PUT():Boolean = {println("PUT");return true}
def GET():Boolean = {println("GET");return true}
def DELETE():Boolean = {println("DELETE");return true}
def POST():Boolean = {println("POST");return true}
def HEAD():Boolean = {println("HEAD");return true}
}
class LoadGenerator(nrOfWorkers:Int, noOfMessages:Int) extends Actor {
val log = Logger.getLogger(this.getClass().getName())
val start:Long = System.currentTimeMillis
var noOfMessageRcvd:Int = 0
val r = new CommandGenerator// <- is basically are list implementation that iterates and returns the next command
r.addCommand("PUT",5) r.addCommand("GET",2) r.addCommand("DELETE",2)
r.addCommand("POST",2) r.addCommand("HEAD",1) r.addCommand("LBRPOP",1)
val loadRouter = context.actorOf(Props[LoadWorker].withRouter(RoundRobinRouter(nrOfWorkers)),name ="loadRouter")
def receive = {
case "start" => {
if(noOfMessages > 1) {
for( i <- 0 until noOfMessages) loadRouter ! r.getRandomCommand()
} else {
log.info("Time bound Load run..")
//for( i <- 0 until 10000000) { //<- For any number greater than few millions that app hangs after few messages
while(true){loadRouter ! r.getRandomCommand() //<- with while loop the app hangs as soon as it begins
}
}
}
case true => {
noOfMessageRcvd +=1
if(noOfMessages == noOfMessageRcvd){
self ! "shutdown"
}
}
case "stats" => {
logStats()
}
case "shutdown" => {
logStats()
log.info("Shutting Down!")
context.system.shutdown()
}
}
def logStats(){
var duration = (System.currentTimeMillis - start)/1000
if( duration > 0) {
log.info(noOfMessageRcvd+" messages processed in "+duration +" seconds "
+ "at "+ noOfMessageRcvd/duration +" TPS" )
} else {
log.info(noOfMessageRcvd+" messages processed in less than a second ")
}
}
}
object RedLoad extends App{
val log = Logger.getLogger(this.getClass().getName())
val system = ActorSystem("LoadGeneratorApp");
// -1 is if we want to run for a period of time and > 1 the run will end after the messages are procesed
val lg = system.actorOf(Props(new LoadGenerator(100,-1)),"LG")
//Log the stats every 30 seconds
system.scheduler.schedule(0 seconds,30 seconds,lg,"stats")
//Shutdown the load run after 2 hours, if no of message is > -1 then it will shutdown after
//all messages are processed
system.scheduler.scheduleOnce(2 hours,lg,"shutdown")
lg ! "start"
log.info("Started..")
}

Answer 1

Well, your actor can't process more than one message at a time, and you just set it to be busy forever sending messages. Working as designed. Remove the endless loop and send batches of messages to the loadRouter and send yourself continuation-messages to keep sending more messages.
case SendBatch =>
(1 to batchSize) foreach { router ! message }
self ! SendBatch