Scala-redis subscribes to * but receives zero messages - scala

Integrating redis with my Scala application using Akka but for some reason it does not receive any messages. I can confirm that redis does have a ton of traffic on its side by opening the redis-cli on the command line.
After a pSubscribe it receives: subscribed to * and count = 1
My guess is that it might be related to the way Akka is set up to receive callbacks. I had to strip out Scala Actors in the scala-redis lib and replace them with Akka actors due to some conflicts.
Here's the code:
The Subscriber Actor
class Subscriber(client: RedisClient) extends Actor {
var callback: PubSubMessage => Any = { m => }
def receive: Receive = {
case Subscribe(channels) =>
client.subscribe(channels.head, channels.tail: _*)(callback)
case pSubscribe(channels) =>
client.pSubscribe(channels.head, channels.tail: _*)(callback)
case pSubscribeAll(channels) =>
Logger.info("Subscribing to all channels")
client.pSubscribe(channels.head, channels.tail: _*)(callback)
case Register(cb) =>
Logger.info("Callback is registered")
callback = cb
case Unsubscribe(channels) =>
client.unsubscribe(channels.head, channels.tail: _*)
case UnsubscribeAll =>
client.unsubscribe
}
}
Initializing the Subscriber
class RelaySub extends Actor {
// important config values
val system = ActorSystem("pubsub")
val conf = play.api.Play.current.configuration
val relayPubHost = conf.getString("relays.redis.host").get
val relayPubPort = conf.getInt("relays.redis.port").get
val rs = new RedisClient(relayPubHost, relayPubPort)
val s = system.actorOf(Props(new Subscriber(rs)))
s ! Register(callback)
s ! pSubscribeAll(Array("*"))
Logger.info("Engine Relay Subscriber has started up")
def receive: Receive = {
case Register(callback) =>
}
def callback(pubsub: PubSubMessage) = pubsub match {
case S(channel, no) => Logger.info("subscribed to " + channel + " and count = " + no)
case U(channel, no) => Logger.info("unsubscribed from " + channel + " and count = " + no)
case M(channel, msg) =>
msg match {
// exit will unsubscribe from all channels and stop subscription service
case "exit" =>
Logger.info("unsubscribe all ... no handler yet ;)")
// message "+x" will subscribe to channel x
case x if x startsWith "+" =>
Logger.info("subscribe to ... no handler yet ;)")
// message "-x" will unsubscribe from channel x
case x if x startsWith "-" =>
Logger.info("unsubscribe from ... no handler yet ;)")
// other message receive
case x =>
Logger.info("Engine: received redis message")
val channelVars = channel.split(".").toArray[String]
if(channelVars(0)!=Engine.instanceID)
channelVars(1) match {
case "relay" =>
EngineSyncLocal.constructRelay(channel, msg)
case _ =>
Logger.error("Engine: received unknown redis message")
}
}
}
}
Thanks for your help!

I found the problem. It appears to be a bug in the scala-redis client.
I added some logging in the consumer class and began receiving Engine: weird message errors which means that it doesn't recognize the incoming traffic. I'll contact the author and put in a pull request.
The code:
class Consumer(fn: PubSubMessage => Any) extends Runnable {
def start () {
val myThread = new Thread(this) ;
myThread.start() ;
}
def run {
whileTrue {
asList match {
case Some(Some(msgType) :: Some(channel) :: Some(data) :: Nil) =>
Logger.info("Engine: redis traffic")
msgType match {
case "subscribe" | "psubscribe" => fn(S(channel, data.toInt))
case "unsubscribe" if (data.toInt == 0) =>
fn(U(channel, data.toInt))
break
case "punsubscribe" if (data.toInt == 0) =>
fn(U(channel, data.toInt))
break
case "unsubscribe" | "punsubscribe" =>
fn(U(channel, data.toInt))
case "message" | "pmessage" =>
fn(M(channel, data))
case x => throw new RuntimeException("unhandled message: " + x)
}
case _ => Logger.error("Engine: weird redis message")
}
}
}
}

case x => throw new RuntimeException("unhandled message: " + x)
}
case Some(Some("pmessage")::Some(pattern)::Some(channel):: Some(message)::Nil)=>
fn(M(channel, message))
asList match is missing a case

Related

getting dead letter from Actor when trying to get back the response from it

I have ParentActor and 2 ChildActors here is my code
Class ParentActor extends Actor {
val mongoActor = context.of.....
val esActor = context.of ............
def receive {
case InserInMongo(obj) =>
val mFuture = ask(mongoActor, InsertDataInMongo(object)).mapTo[Boolean]
mFuture.onComplete {
case Success(resultMongo) =>
sender ! resultMongo
case Failure(e) =>
sender ! akka.actor.Status.Failure(e)
throw e
}
case InserInES(obj) =>
val eFuture = ask(esActor, InsertDataInES(object)).mapTo[Boolean]
eFuture.onComplete {
case Success(resultES) =>
sender ! resultES
case Failure(e) =>
sender ! akka.actor.Status.Failure(e)
throw e
}
}
}
here is the calling code
class Demo {
val mongoFuture = ask(parentActor, InsertInMongo(obj))
.mapTo[Boolean]
.recover {
case e =>
println("error in mongo: " + e)
false
}
val esFuture = ask(parentActor, InsertInES(obj))
.mapTo[Boolean]
.recover {
case e =>
println("error in ES: " + e)
false
}
val f = Future.sequence(List(mongoFuture, esFuture))
val result: Seq[Boolean] = Await.result(f, Duration.Inf)
log.info ("result {}",result)
}
everything works fine but if i get some MongoException in mongoChildActor i am unable to get the parent actor result for ES
here is what i am getting
17:19:45.782 [MyActorSystem-akka.actor.default-dispatcher-4] INFO akka.actor.DeadLetterActorRef - Message [java.lang.Boolean] from Actor[akka://MyActorSystem/user/ParentActor#1383701267] to Actor[akka://MyActorSystem/deadLetters] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
All i want is if i got exception in mongo i got the exception and then it should proceed with ES and return its result (either its successful value or exception)
Don't throw exceptions in receive, and don't close over sender (sender is a function, that is only valid in the context of receive, you are using it inside .onComplete, on a different thread).
Something like this should do what you want:
import akka.pattern.pipe
def receive: Receive = {
case InserInMongo(obj) =>
ask(mongoActor, InsertDataInMongo(object)).pipeTo(sender)
case InserInES(obj) =>
ask(esActor, InsertDataInES(object)).pipeTo(sender)
}
Or, even easier:
def receive: Receive {
case msg: InsertDataInMongo =>
mongoActor.forward(msg)
case msg: InsertDataInES =>
esActor.forward(msg)
}

akka- how to ensure all responses of dynamic number of actors are returned to parent actor?

I need to create variable number of actors each time my program starts and then must ensure all responses are return after a period of time. This
link gives a good idea for fixed number of actors but what about dynamic number?
This is my code that creates actor and passes messages to them:
ruleList = ...
val childActorList: Iterable[ActorRef] = ruleList.map(ruleItem =>
context.actorOf(DbActor.props(ruleItem.parameter1, ruleItem.parameter2)))
implicit val timeout = Timeout(10.second)
childActorList.foreach(childActor =>
childActor ? (tempTableName, lastDate)
)
Updated-1
According to #Raman Mishra guides , I updated my code as bellow, this is the code in parent actor:
override val supervisorStrategy: SupervisorStrategy = {
OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 10 seconds) {
case exp: SQLException => //Resume;
throw exp
case exp:AskTimeoutException => throw exp
case other: Exception => throw other
}
}
override def receive: Receive = {
case Start(tempTableName, lastDate) => {
implicit val timeout = Timeout(10.second)
ruleList.foreach { ruleItem =>
val childActor = context.actorOf(DbActor.props(ruleItem._1, query = ruleItem._2))
ask(childActor, (tempTableName, lastDate)).mapTo[Seq[Int]]
onComplete {
lastDate)).mapTo[Seq[Int]] onComplete {
case util.Success(res) => println("done" + res + ruleItem._2)
case util.Failure(exp: AskTimeoutException) => println("Failed query:" + ruleItem._2); throw exp
case other => println(other)
}
}
And in child actor:
case (brokerTableName, lastDate) => {
Logger("Started query by actor" + self.path.name + ':' +
val repo = new Db()
val res = repo.getAggResult(query = (brokerTableName, lastDate))
val resWrapper = res match {
case elem: Future[Any] => elem
case elem:Any => Future(elem)
}
resWrapper pipeTo self
}
case res:List[Map[Any, Any]] => {
// here final result is send to parent actor
repo.insertAggresults(res, aggTableName) pipeTo context.parent
}
Now, whenever I run main app, first, parent actor starts and create child actors and send messages to them using ask method. Child actors do their tasks but the problem here is child actors response never returns back to parent actor and in every run of app, AskTimeoutException occurs. I doubt if the use of onComplete method is correct or not. Any help will be appreciated.
"Updated-2"
I found out the problem is in using context.parent instead of sender(). Also, when I pipe to sender, first part of my result, and the sender ask for second part, the problem is resolved but I don't know what happens here, why Can't I pipe to self and return the final result to parent?
This is the last code:
In parent actor:
override def receive: Receive = {
case Start(tempTableName, lastDate) => {
println("started: called by remote actor")
implicit val timeout = Timeout(5 second)
ruleList.foreach { ruleItem =>
val childActor = context.actorOf(DbActor.props(ruleItem._1, query = ruleItem._2))
ask(childActor, Broker(tempTableName, lastDate)) onComplete {
// (childActor ? Broker(tempTableName, lastDate)).mapTo[Seq[Int]] onComplete {
case util.Success(res: List[Map[Any, Any]]) => (childActor ? res) onComplete {
case util.Success(res: Seq[Any]) => println("Successfull- Num,ber of documents:" + res.length + " " + ruleItem._2)
case util.Failure(exp: AskTimeoutException) => println("Failed for writing - query:" + ruleItem._2); throw exp
}
case util.Failure(exp: AskTimeoutException) => println("Failed for reading - query :" + ruleItem._2); throw exp
case other => println(other)
}
}
}
}
In child actor:
case (brokerTableName, lastDate) => {
Logger("Started query by actor" + self.path.name + ':' +
val repo = new Db()
val res = repo.getAggResult(query = (brokerTableName, lastDate))
val resWrapper = res match {
case elem: Future[Any] => elem
case elem:Any => Future(elem)
}
resWrapper pipeTo sender()
}
case res:List[Map[Any, Any]] => {
// here final result is send to parent actor
repo.insertAggresults(res, aggTableName) pipeTo sender()
}
The reason that replying to sender() works where replying to context.parent does not is that ask creates an temporary actor to handle the response. You need to reply to this temporary actor: the sender (which is different from the parent).
Also it's not clear whether the getAggResult method is blocking. If so this will not help (see here).

Scala warning a type was inferred to be 'AnyVal'

I have this actor with the following recieve code:
override def receive: Receive = {
case RefreshAccessTokenRequest(accountId, Some(refreshToken), _, _) =>
concurApi.refreshAccessToken(accountId, refreshToken) onComplete {
case Success(refreshTokenResult) =>
context.parent ! ConcurResultCallbackInformation(accountId, Right(refreshTokenResult))
accountService.updateAccountOAuth2Credentials(accountId, refreshTokenResult.toOAuth2Credentials) recover {
case ex => logger.error("Error updating account Id: " + accountId + " with new refresh token. message: " + ex.getMessage)
}
context.stop(self)
case Failure(t) =>
context.parent ! ConcurResultCallbackInformation(accountId, Left(t))
context.stop(self)
}
case Cancel => context.stop(self)
case x => logger.error(s"ConcurAccessTokenActor: Got an unknown message $x")
}
I am getting a warning on the bolded part which says:
Warning:(18, 42) a type was inferred to be AnyVal; this may indicate a programming error.
case Success(refreshTokenResult) =>
this is the refreshAccessToken function:
def refreshAccessToken(accountId: String, refreshToken: String)(implicit context: ExecutionContext) : Future[ConcurRefreshTokenResult] = {
val body = s"client_id=${sandboxClientId}"
val futureRes = ws.url(sandboxRefreshTokenFullUrl).withHeaders("Content-Type" -> "application/x-www-form-urlencoded").post(body)
futureRes.flatMap { res =>
res.status match {
case 200 =>
val extract = Try {
res.body.fromJson[ConcurRefreshTokenResult]
}.recover { case ex =>
throw ex
}
Promise.fromTry(extract).future
case 400 =>
Future.failed(new RuntimeException(s"refresh Access Token failed with status 400, and body: ${res.body}"))
case x =>
Future.failed(new RuntimeException(s"refresh Access Token failed with status $x, and body: ${res.body}"))
}
}
}
Any ideas?
Ok I got it..
The warning came from this code part:
case Success(refreshTokenResult) =>
context.parent ! ConcurResultCallbackInformation(accountId, Right(refreshTokenResult))
accountService.updateAccountOAuth2Credentials(accountId, refreshTokenResult.toOAuth2Credentials) recover {
case ex => logger.error("Error updating account Id: " + accountId + " with new refresh token. message: " + ex.getMessage)
}
context.stop(self)
the 'updateAccountOAuth2Credentials' function returns Future[Boolean] yet the recover part returns Unit so it interperts it to AnyVal
I think the only thing that is confusing is the location of the warning arrow since this function appears in the middle between 2 other functions, and has no effects on the return type of the entire case block

Processing unknown amount of actors results on single timeout

I am looking to expand the following code to work for an unknown amount of actor ask requests.
implicit val timeout = Timeout(100 millis)
val sendRequestActor = context.actorOf(Props(new SendRequest(request)), "Send_Request_".concat(getActorNumber))
val sendRequestActor2 = context.actorOf(Props(new SendRequest(request)), "Send_Request_".concat(getActorNumber))
val a1 = ask(sendRequestActor, Request).fallbackTo(Future.successful(RequestTimeout))
val a2 = ask(sendRequestActor2, Request).fallbackTo(Future.successful(RequestTimeout))
val result = for {
r1 <- a1
r2 <- a2
} yield(r1, r2)
val r = Await.result(result, 100 millis)
r match {
case (b: SuccessResponse, b2: SuccessResponse) => {
//Process Results
}
case (b: SuccessResponse, b2: RequestTimeout) => {
//Process Results
}
case (b: RequestTimeout, b2: SuccessResponse) => {
//Process Results
}
case (b: RequestTimeout, b2: RequestTimeout) => {
//Process Results
}
case _ => {}
}
I am trying to send out requests to a List of recipients(gotten from a previous database call). The number of recipients will vary each time this function is called. Recipients have a maximum of 100 milliseconds to respond before I want to time out their requests and record a RequestTimeout. The SendRequest actor will reply with SuccessResponse if the recipients respond. I am assuming I will have to change the val result for-loop to process a list, but I am unsure of how to structure everything so that I will wait the minimum amount of time(either when all actors return or when the timeout hits, whichever is lower). I do not need everything in a single return value like the example, I am fine with a list of results and matching type on each iteration.
Any help would be appreciated, please let me know if I can provide any other information.
Thank you
Edit:
Calling Class:
case object GetResponses
def main(args: Array[String]) {
val route = {
get {
complete {
//stuff
val req_list = List(req1,req2,req3)
val createRequestActor = system.actorOf(Props(new SendAll(req_list)), "Get_Response_Actor_" + getActorNumber)
val request_future = ask(createRequestActor, GetResponses).mapTo[List[Any]]
Thread.sleep(1000)
println(request_future)
//more stuff
}
}
}
Http().bindAndHandle(route, "localhost", 8080)
}
Updated Sending Class:
class SendAll(requests: List[Request]) extends Actor {
import context.{become,dispatcher}
var numProcessed = 0
var results: List[Any] = List()
requests.foreach(self ! _)
implicit val timeout = Timeout(100 millis)
def receive = {
case r: RequestMsg =>
val sendRequestActor = context.actorOf(Props(new SendRequest(r)), "Send_Request_".concat(getActorNumber))
(sendRequestActor ? Request).pipeTo(self)
case s: SuccessResponse =>
println("Got Success")
results = results :+ s
println(results.size + " == " + requests.size)
if(results.size == requests.size) {
println("Before done")
become(done)
}
case akka.actor.Status.Failure(f) =>
println("Got Failed")
results = results :+ RequestTimeout
if(results.size == requests.size) {
become(done)
}
case m =>
println("Got Other")
}
def done: Receive = {
case GetResponses =>
println("Done")
sender ! results
case _ => {
println("Done as well")
}
}
}
Output
Got Success
1 == 3
Got Success
2 == 3
Got Success
3 == 3
Before done
Future(<not completed>)
I would pass the list of requests to the actor, then pipe the responses from the child actors to self instead of using Await.result. For example:
class Handler(requests: List[RequestMsg]) extends Actor {
import context.{become, dispatcher}
var numProcessed = 0
var results: List[Any] = List()
requests.foreach(self ! _)
implicit val timeout = Timeout(100.millis)
def receive = {
case r: RequestMsg =>
val sendRequestActor = context.actorOf(Props(new SendRequest(r)), "Send_Request".concat(getActorNumber))
(sendRequestActor ? Request).pipeTo(self)
case s: SuccessResponse =>
println(s"response: $s")
results = results :+ s
if (results.size == requests.size)
become(done)
case akka.actor.Status.Failure(f) =>
println("a request failed or timed out")
results = results :+ RequestTimeout
if (results.size == requests.size)
become(done)
case m =>
println(s"Unhandled message received while processing requests: $m")
sender ! NotDone
}
def done: Receive = {
case GetResponses =>
println("sending responses")
sender ! results
}
}
You would instantiate an actor for every list of requests:
val requests1 = List(RequestMsg("one"), RequestMsg("two"), RequestMsg("three"))
val handler1 = system.actorOf(Props(new Handler(requests1)))
In this example--following the principle that an actor should have a distinct, limited sphere of responsibility--the actor simply coordinates requests and responses; it doesn't perform any processing on the collected responses. The idea is that another actor would send this actor a GetResponses messages in order to get the responses and process them (or this actor would proactively send the results to a processing actor).
The simplest solution is put all your actor refs into the List map it to List[Future] and use Future.sequence to obtain Future[List].
val route = {
get {
val listActorRefs = List(actorRef1, actorRef2, ...)
val futureListResponses = Future.sequence(listActorRefs.map(_ ? Request))
onComplete(futureListResponses) {
case Success(listResponse) => ...
complete(...)
case Failure(exception) => ...
}
}
}
A better solution is avoid a lot of actor' asks, prepare some ResponseCollector actor which will send all your message (I suggest to look at BroadcastPool) and schedule one message for itself to stop waiting and return result.

Combining Spray Routing + Actor Pattern Matching

Following the Akka Cluster documentation, I have the Worker Dial-in example running.
http://doc.akka.io/docs/akka/snapshot/java/cluster-usage.html
So I've trying to integrate that with a spray routing.
My idea is to have a cluster behind the scenes and through a http rest, call that service.
So I have the following code.
object Boot extends App {
val port = if (args.isEmpty) "0" else args(0)
val config =
ConfigFactory
.parseString(s"akka.remote.netty.tcp.port=$port")
.withFallback(ConfigFactory.parseString("akka.cluster.roles = [frontend]"))
.withFallback(ConfigFactory.load())
val system = ActorSystem("ClusterSystem", config)
val frontend = system.actorOf(Props[TransformationFrontend], name = "frontend")
implicit val actSystem = ActorSystem()
IO(Http) ! Http.Bind(frontend, interface = config.getString("http.interface"), port = config.getInt("http.port"))
}
class TransformationFrontend extends Actor {
var backends = IndexedSeq.empty[ActorRef]
var jobCounter = 0
implicit val timeout = Timeout(5 seconds)
override def receive: Receive = {
case _: Http.Connected => sender ! Http.Register(self)
case HttpRequest(GET, Uri.Path("/job"), _, _, _) =>
jobCounter += 1
val backend = backends(jobCounter % backends.size)
val originalSender = sender()
val future : Future[TransformationResult] = (backend ? new TransformationJob(jobCounter + "-job")).mapTo[TransformationResult]
future onComplete {
case Success(s) =>
println("received from backend: " + s.text)
originalSender ! s.text
case Failure(f) => println("error found: " + f.getMessage)
}
case job: TransformationJob if backends.isEmpty =>
sender() ! JobFailed("Service unavailable, try again later", job)
case job: TransformationJob =>
jobCounter += 1
backends(jobCounter % backends.size) forward job
case BackendRegistration if !backends.contains(sender()) =>
println("backend registered")
context watch sender()
backends = backends :+ sender()
case Terminated(a) =>
backends = backends.filterNot(_ == a)
}
}
But what I really want to do is to combining the spray routing with those pattern matching.
Instead of writing my GET like the above, I would like to write like this:
path("job") {
get {
respondWithMediaType(`application/json`) {
complete {
(backend ? new TransformationJob(jobCounter + "-job")).mapTo[TransformationResult]
}
}
}
}
But extending my Actor with this class, I have to do the following
def receive = runRoute(defaultRoute)
How can I combine this approach with my TransformationFrontend Actor pattern matching methods? BackendRegistration, Terminated, TransformationJob?
You can compose PartialFunctions like Receive with PartialFunction.orElse:
class TransformationFrontend extends Actor {
// ...
def myReceive: Receive = {
case job: TransformationJob => // ...
// ...
}
def defaultRoute: Route =
get {
// ...
}
override def receive: Receive = runRoute(defaultRoute) orElse myReceive
}
That said, it often makes sense to split up functionality into several actors (as suggested in the comment above) if possible.