How to compile a sequence of results using ask pattern - scala

Say I have a worker actor that receives a message, does a bit of processing and returns a result. And I have a sequence of messages that need to be converted into a sequence of results:
object Test {
case class Message(str: String)
case class Result(str: String)
class Worker extends Actor {
def receive = {
case Message(data) =>
println("Sleeping: " + data)
Thread.sleep(10000)
val result = Result(data + " - result")
println("Sending result: " + result)
sender ! result
}
}
def test(messages: Seq[Message]): Future[Seq[Result]] = {
val worker = ActorSystem().actorOf(Props(new Worker))
val results = messages.map { m =>
implicit val timeout = Timeout(20 seconds)
println("Sending: " + m)
val result = worker ? m
result.asInstanceOf[Future[Result]]
}
Future.sequence(results)
}
def main(args: Array[String]): Unit = {
val messages: Seq[Message] = args.map(Message(_))
test(messages).foreach { r =>
println("Result: " + r)
}
}
}
If I run the above with just "message-1" as an argument it runs fine giving the the output is below:
Sending: Message(message-1)
Sleeping: message-1
Sending result: Result(message-1 - result)
Result: ArraySeq(Result(message-1 - result))
However say I do it with: "message-1" "message-2" "message-3" then the last message ends up being sent to deadLetters:
Sending: Message(message-1) Sending: Message(message-2) Sleeping:
message-1 Sending: Message(message-3)
Sending result: Result(message-1 - result)
Sleeping: message-2
Sending result: Result(message-2 - result)
Sleeping: message-3
Sending result: Result(message-3 - result)
[INFO] [07/15/2016 09:07:49.832]
[default-akka.actor.default-dispatcher-2] [akka://default/deadLetters]
Message [util.Tester$Result] from
Actor[akka://default/user/$a#1776546850] to
Actor[akka://default/deadLetters] was not delivered. [1] dead letters
encountered. This logging can be turned off or adjusted with
configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
I am guessing this is because my calling thread has gone out of scope by the time the last message is sent. How can correctly collect all results into a sequence?
Note that changing my test method to below gives the same results:
def test(messages: Seq[Message]): Future[Seq[Result]] = {
val worker = ActorSystem().actorOf(Props(new Worker))
Future.traverse(messages) { m =>
implicit val timeout = Timeout(20 seconds)
println("Sending: " + m)
val result = worker ? m
result.asInstanceOf[Future[Result]]
}
}

The dumb answer is:
Future.traverse(messages)(m => actor ? m).map(_.asInstanceOf[Result])
But it might be better to send data all at once:
class Worker extends Actor {
def receive = {
case Message(data) =>
// Convert data into result
...
sender ! result
case seq: Seq[Message] =>
...
sender ! results
}
}

Seems to be because my timeout was set too low. Should have been large enough to cover all the work - for example 40 seconds.

Related

Scala Akka Actors: How to send the result of an Http response back to the sender?

I'm trying to execute the following Scala code inside an Akka Actor.
class FilteringService(implicit timeout: Timeout) extends Actor {
def receive: PartialFunction[Any, Unit] = {
case GetProfiles ⇒
val requester = sender
def getProfiles = {
var result = new Array[Profile](0)
println("[GET-PROFILES] Entered, making request")
val req = Get("http://localhost:9090/profiles")
implicit val profileFormat = jsonFormat16(Profile)
val responseFuture: Future[HttpResponse] = Http().singleRequest(req)
println("[GET-PROFILES] Entered, request sent")
responseFuture.onComplete {
case Success(response) =>
println("[RES - SUCCESS] Request returned with " + response.status)
val responseAsProfiles = Unmarshal(response.entity).to[Array[Profile]]
responseAsProfiles.onComplete {
println("[UNMARSH - SUCCESS] Unmarshaling Done!")
_.get match {
case profiles: Array[Profile] =>
println("[UNMARSH - SUCCESS] Sending Profiles message to " + sender())
requester ! profiles
println("[UNMARSH - SUCCESS] Message sent to " + sender())
case _ => println("error")
}
}
case Failure(_) =>
sys.error("something wrong")
//return Future[Array[Profile]]
}
}
println("[RECEIVE] Message GetProfiles received from " + sender().toString())
getProfiles
println("[RECEIVE] Message GetProfiles invoked")
}
When the Actor receives the message "GetProfiles":
1- it sends a request to a remote server, so the result of operation is a Future[HttpResponse]
2- in case of success it retrieves the response (a JSON array) and asks for unmarshalling the object to Array[Profile]. (It's not important the Profile model). The result of Unmarshall method is a Future[Array[Profile]]
3- In case of success, I want to send the result back to the original sender!
I managed to do this, but it's a trick because I'm saving the sender in a variable, that is visible in the scope (requester).
I know that there is the pipe pattern, so I could send the responseAsProfiles object back to the sender in theory, but the object is created inside the onComplete method of the responseFuture object (we have to wait it, of course!)
So that's all!
How could I send the result back to the sender using the pipe pattern in this case?
Thanks in advance!!!
General idea is that you compose futures using map and flatMap and try to avoid using onComplete as much as possible.
See if you can convert your code to following smaller pieces and then compose:
def getRawProfileData(): Future[HttpResponse] = {
// ... here you make http request
}
def unmarshalProfiles(response: HttpResponse): Future[List[Profile]] = {
// ... unmarshalling logic
}
def getProfiles(): Future[List[Profile]] = getRawProfileData().flatMape(unmarshalProfiles)
// now from receive block
case GetProfiles ⇒ getProfiles().pipeTo(sender())

Processing unknown amount of actors results on single timeout

I am looking to expand the following code to work for an unknown amount of actor ask requests.
implicit val timeout = Timeout(100 millis)
val sendRequestActor = context.actorOf(Props(new SendRequest(request)), "Send_Request_".concat(getActorNumber))
val sendRequestActor2 = context.actorOf(Props(new SendRequest(request)), "Send_Request_".concat(getActorNumber))
val a1 = ask(sendRequestActor, Request).fallbackTo(Future.successful(RequestTimeout))
val a2 = ask(sendRequestActor2, Request).fallbackTo(Future.successful(RequestTimeout))
val result = for {
r1 <- a1
r2 <- a2
} yield(r1, r2)
val r = Await.result(result, 100 millis)
r match {
case (b: SuccessResponse, b2: SuccessResponse) => {
//Process Results
}
case (b: SuccessResponse, b2: RequestTimeout) => {
//Process Results
}
case (b: RequestTimeout, b2: SuccessResponse) => {
//Process Results
}
case (b: RequestTimeout, b2: RequestTimeout) => {
//Process Results
}
case _ => {}
}
I am trying to send out requests to a List of recipients(gotten from a previous database call). The number of recipients will vary each time this function is called. Recipients have a maximum of 100 milliseconds to respond before I want to time out their requests and record a RequestTimeout. The SendRequest actor will reply with SuccessResponse if the recipients respond. I am assuming I will have to change the val result for-loop to process a list, but I am unsure of how to structure everything so that I will wait the minimum amount of time(either when all actors return or when the timeout hits, whichever is lower). I do not need everything in a single return value like the example, I am fine with a list of results and matching type on each iteration.
Any help would be appreciated, please let me know if I can provide any other information.
Thank you
Edit:
Calling Class:
case object GetResponses
def main(args: Array[String]) {
val route = {
get {
complete {
//stuff
val req_list = List(req1,req2,req3)
val createRequestActor = system.actorOf(Props(new SendAll(req_list)), "Get_Response_Actor_" + getActorNumber)
val request_future = ask(createRequestActor, GetResponses).mapTo[List[Any]]
Thread.sleep(1000)
println(request_future)
//more stuff
}
}
}
Http().bindAndHandle(route, "localhost", 8080)
}
Updated Sending Class:
class SendAll(requests: List[Request]) extends Actor {
import context.{become,dispatcher}
var numProcessed = 0
var results: List[Any] = List()
requests.foreach(self ! _)
implicit val timeout = Timeout(100 millis)
def receive = {
case r: RequestMsg =>
val sendRequestActor = context.actorOf(Props(new SendRequest(r)), "Send_Request_".concat(getActorNumber))
(sendRequestActor ? Request).pipeTo(self)
case s: SuccessResponse =>
println("Got Success")
results = results :+ s
println(results.size + " == " + requests.size)
if(results.size == requests.size) {
println("Before done")
become(done)
}
case akka.actor.Status.Failure(f) =>
println("Got Failed")
results = results :+ RequestTimeout
if(results.size == requests.size) {
become(done)
}
case m =>
println("Got Other")
}
def done: Receive = {
case GetResponses =>
println("Done")
sender ! results
case _ => {
println("Done as well")
}
}
}
Output
Got Success
1 == 3
Got Success
2 == 3
Got Success
3 == 3
Before done
Future(<not completed>)
I would pass the list of requests to the actor, then pipe the responses from the child actors to self instead of using Await.result. For example:
class Handler(requests: List[RequestMsg]) extends Actor {
import context.{become, dispatcher}
var numProcessed = 0
var results: List[Any] = List()
requests.foreach(self ! _)
implicit val timeout = Timeout(100.millis)
def receive = {
case r: RequestMsg =>
val sendRequestActor = context.actorOf(Props(new SendRequest(r)), "Send_Request".concat(getActorNumber))
(sendRequestActor ? Request).pipeTo(self)
case s: SuccessResponse =>
println(s"response: $s")
results = results :+ s
if (results.size == requests.size)
become(done)
case akka.actor.Status.Failure(f) =>
println("a request failed or timed out")
results = results :+ RequestTimeout
if (results.size == requests.size)
become(done)
case m =>
println(s"Unhandled message received while processing requests: $m")
sender ! NotDone
}
def done: Receive = {
case GetResponses =>
println("sending responses")
sender ! results
}
}
You would instantiate an actor for every list of requests:
val requests1 = List(RequestMsg("one"), RequestMsg("two"), RequestMsg("three"))
val handler1 = system.actorOf(Props(new Handler(requests1)))
In this example--following the principle that an actor should have a distinct, limited sphere of responsibility--the actor simply coordinates requests and responses; it doesn't perform any processing on the collected responses. The idea is that another actor would send this actor a GetResponses messages in order to get the responses and process them (or this actor would proactively send the results to a processing actor).
The simplest solution is put all your actor refs into the List map it to List[Future] and use Future.sequence to obtain Future[List].
val route = {
get {
val listActorRefs = List(actorRef1, actorRef2, ...)
val futureListResponses = Future.sequence(listActorRefs.map(_ ? Request))
onComplete(futureListResponses) {
case Success(listResponse) => ...
complete(...)
case Failure(exception) => ...
}
}
}
A better solution is avoid a lot of actor' asks, prepare some ResponseCollector actor which will send all your message (I suggest to look at BroadcastPool) and schedule one message for itself to stop waiting and return result.

Akka Router increment counter on message arrival from routees

I'm trying to keep counting on each successful import. But here is a problem - Counter works if the router receives a message from its parent but if I'm trying to send a message from its children it receives it but doesn't update the global variable that is out of the scope.
I know it sounds complicated. Let me show you the code.
Here is the router
class Watcher(size: Int) extends Actor {
var router = {
val routees = Vector.fill(size) {
val w = context.actorOf(
Props[Worker]
)
context.watch(w)
ActorRefRoutee(w)
}
Router(RoundRobinRoutingLogic(), routees)
}
var sent = 0
override def supervisorStrategy(): SupervisorStrategy = OneForOneStrategy(maxNrOfRetries = 100) {
case _: DocumentNotFoundException => {
Resume
}
case _: Exception => Escalate
}
override def receive: Receive = {
case container: MessageContainer =>
router.route(container, sender)
case Success =>
sent += 1
case GetValue =>
sender ! sent
case Terminated(a) =>
router.removeRoutee(a)
val w = context.actorOf(Props[Worker])
context.watch(w)
router = router.addRoutee(w)
case undef =>
println(s"${this.getClass} received undefinable message: $undef")
}
}
Here is the worker
class Worker() extends Actor with ActorLogging {
var messages = Seq[MessageContainer]()
var received = 0
override def receive: Receive = {
case container: MessageContainer =>
try {
importMessage(container.message, container.repo)
context.parent ! Success
} catch {
case e: Exception =>
throw e
}
case e: Error =>
log.info(s"Error occurred $e")
sender ! e
case undef => println(s"${this.getClass} received undefinable message: $undef")
}
}
So on supervisor ? GetValue I get 0 but suppose to have 1000.The strangest thing is that when I debug it with the breakpoint right on the case Success => ... the value is incremented every time the new message arrives. But supervisor ? GetValue still returns 0.
Let's assume I want to count on case container: MessageContainer => ... and it will magically work; I'll get desirable number, but it doesn't show if I actually imported anything. What's going on?
Here is the test case.
#Test
def testRouter(): Unit = {
val system = ActorSystem("RouterTestSystem")
// val serv = AddressFromURIString("akka.tcp://master#host:1334")
val supervisor = system.actorOf(Props(new Watcher(20)))//.withDeploy(akka.actor.Deploy(scope = RemoteScope(serv))))
val repo = coreSession.getRepositoryName
val containers = (0 until num)
.map(_ => MessageContainer(MessageFactory.generate("/"), repo))
val watch = Stopwatch.createStarted()
(0 until num).par
.foreach( i => {
supervisor ! containers.apply(i)
})
implicit val timeout = Timeout(60 seconds)
val future = supervisor ? GetValue
val result = Await.result(future, timeout.duration).asInstanceOf[Int]
val speed = result / (watch.elapsed(TimeUnit.MILLISECONDS) / 1000.0)
println(f"Import speed: $speed%.2f")
assertEquals(num, result)
}
Can you please explained it in details. Why is it happening? Why only on message received from the children? Another approach?
Well... there can be many potential problems hidden in the parts of code that you have not shared. But, for the sake of this discussion I will assume that everything else is fine and we will just discuss problems with your shared code.
Now, let me explain a bit about Actors. To put things simply, every actor has a mailbox (where it keeps messages in the sequence they were received) and processes them one by one in the order they were received. Since the mailbox is used like a Queue we will refer to it as a Queue in this discussion.
Also... I don't know what this container.apply(i) is going to return... so I will refer to the return value of that container.apply(1) as MessageContainer__1
In your test runner you are first creating an instance of Watcher,
val supervisor = system.actorOf(Props(new Watcher(20)))
Now, lets say that you are sending these 2 messages (num = 2) to supervisor,
So supervisor's mailbox will look something like,
Queue(MessageContainer__0, MessageContainer__1)
Then you send it another message GetValue so the mailbox will look like,
Queue(MessageContainer__0, MessageContainer__1, GetValue)
Now the actor will process the first message and pass it to the workers, the mail-box will look like,
Queue(MessageContainer__1, GetValue)
Now even if your worker is ultra-fast and instantaneous in sending the reply the mailbox will look like,
Queue(MessageContainer__1, GetValue, Success)
And now since your worker super-ultra-fast and instantaneously replies with a Success, the state after passing the second MessageContainer will look like,
Queue(GetValue, Success, Success)
And... here is the root of your problem. The Supervisor sees the GetValue massage before any Success messages, no matter how fast your workers are.
And thus it will process GetValue and reply with current value of sent which is 0.

Scala Akka Play, Future doesn't return

I'm using Scala 2.10, Akka 2.1 and Play 2.1. When I send an http request to my backend, I ask one actor to compute something. The idea is to return the result of the calculation if it returns before a timeout, otherwise a different string. See code below.
val futureInt: Future[Int] = ask(testActor, Calculate(number.toInt)).mapTo[Int]
val timeoutFuture = play.api.libs.concurrent.Promise.timeout("Oops", 2.seconds)
Async {
Future.firstCompletedOf(Seq(futureInt, timeoutFuture)).map {
case i: Int => Ok("Got result " + i)
case t: String => Ok("timeout expired")
}
}
The actor is as follows:
class TestActor() extends Actor {
def receive = {
case Calculate(tonumber: Int) =>
for (a <- 1 to tonumber) {
val c: Double = scala.math.pow(a, 2)
println("a: " + a + ", c: " + c)
}
12345 // hardcoded value to return when the calculation finishes
case _ =>
println("bah")
}
}
My problem is that even if the actor finishes before the timeout, nothing is "returned" by the Future and so the timeout always expires. What am I doing wrong? Thanks a lot.
From http://doc.akka.io/docs/akka/snapshot/scala/actors.html
Using ask will send a message to the receiving Actor as with tell, and the receiving actor must reply with sender ! reply in order to complete the returned Future with a value.
and
Warning
To complete the future with an exception you need send a Failure message to the sender. This is not done automatically when an actor throws an exception while processing a message.
So instead of "returning" like you would in a usual scala function, do something along the lines of
def receive = {
case Calculate(tonumber: Int) =>
...
sender ! 12345
case _ =>
sender ! akka.actor.Status.Failure(new InvalidArgumentException)
}

Akka Actors app hangs under high volume

I am working on a load generator app using akka actors. The app worked fine for few million requests but when increasing the load to more than say 10 million requests or run the load for a duration ( instead of number of requests) using a infinite loop the application hangs. Below is a simplified implementation and it just prints the command being tested. I also notice that the stats are not logged or the app does'nt shutdown when the time is over. I use the schedulers to dump stats every 30 secs and shutdown the app after 2hrs. Tested with small interval and do not see the processing of the "stats" and "Shutdown" messages.
Any idea what may be causing the application to hang ?
import akka.actor._
import akka.util.duration._
import akka.routing.RoundRobinRouter
import com.test.redload.util.CommandGenerator
import org.apache.log4j.Logger
import akka.util.Duration
class LoadWorker extends Actor {
val log = Logger.getLogger(this.getClass().getName())
def receive = {
case "PUT" => sender ! PUT
case "GET" => sender ! GET
case "DELETE" => sender ! DELETE
case "POST" => sender ! POST
case "HEAD" => sender ! HEAD
}
def PUT():Boolean = {println("PUT");return true}
def GET():Boolean = {println("GET");return true}
def DELETE():Boolean = {println("DELETE");return true}
def POST():Boolean = {println("POST");return true}
def HEAD():Boolean = {println("HEAD");return true}
}
class LoadGenerator(nrOfWorkers:Int, noOfMessages:Int) extends Actor {
val log = Logger.getLogger(this.getClass().getName())
val start:Long = System.currentTimeMillis
var noOfMessageRcvd:Int = 0
val r = new CommandGenerator// <- is basically are list implementation that iterates and returns the next command
r.addCommand("PUT",5) r.addCommand("GET",2) r.addCommand("DELETE",2)
r.addCommand("POST",2) r.addCommand("HEAD",1) r.addCommand("LBRPOP",1)
val loadRouter = context.actorOf(Props[LoadWorker].withRouter(RoundRobinRouter(nrOfWorkers)),name ="loadRouter")
def receive = {
case "start" => {
if(noOfMessages > 1) {
for( i <- 0 until noOfMessages) loadRouter ! r.getRandomCommand()
} else {
log.info("Time bound Load run..")
//for( i <- 0 until 10000000) { //<- For any number greater than few millions that app hangs after few messages
while(true){loadRouter ! r.getRandomCommand() //<- with while loop the app hangs as soon as it begins
}
}
}
case true => {
noOfMessageRcvd +=1
if(noOfMessages == noOfMessageRcvd){
self ! "shutdown"
}
}
case "stats" => {
logStats()
}
case "shutdown" => {
logStats()
log.info("Shutting Down!")
context.system.shutdown()
}
}
def logStats(){
var duration = (System.currentTimeMillis - start)/1000
if( duration > 0) {
log.info(noOfMessageRcvd+" messages processed in "+duration +" seconds "
+ "at "+ noOfMessageRcvd/duration +" TPS" )
} else {
log.info(noOfMessageRcvd+" messages processed in less than a second ")
}
}
}
object RedLoad extends App{
val log = Logger.getLogger(this.getClass().getName())
val system = ActorSystem("LoadGeneratorApp");
// -1 is if we want to run for a period of time and > 1 the run will end after the messages are procesed
val lg = system.actorOf(Props(new LoadGenerator(100,-1)),"LG")
//Log the stats every 30 seconds
system.scheduler.schedule(0 seconds,30 seconds,lg,"stats")
//Shutdown the load run after 2 hours, if no of message is > -1 then it will shutdown after
//all messages are processed
system.scheduler.scheduleOnce(2 hours,lg,"shutdown")
lg ! "start"
log.info("Started..")
}
Well, your actor can't process more than one message at a time, and you just set it to be busy forever sending messages. Working as designed. Remove the endless loop and send batches of messages to the loadRouter and send yourself continuation-messages to keep sending more messages.
case SendBatch =>
(1 to batchSize) foreach { router ! message }
self ! SendBatch