getting akka dead letters when using pipeTo between two actors - scala

i have a use case in which i have a actor hierarchy
parent -> childABC -> workerchild
Now the worker child works and send its result to its parent(childABC which is a child of parent) and that child actor(childABC) send the result back to parent actor I am using pipeTo and getting dead letters here is my code
parent actor:
final case object GetFinalValue
class MyActor extends Actor{
import context.dispatcher
import akka.pattern.pipe
val log = LoggerFactory.getLogger(this.getClass)
val myManageActor = context.actorOf(Props[ManagerMyActor],"Managemyactor")
implicit val timeout = Timeout(ReadTimeIntervalValue.getInterval(), SECONDS)
override def receive: Receive = {
case GetFinalValue=>
ask(myManageActor,GetValue).pipeTo(sender())
case message =>
log.warn(" Unhandled message received : {}", message)
unhandled(message)
}
}
childABC (acc to example I gave above)
final case object GetValue
class ManagerMyActor extends Actor{
import context.dispatcher
import akka.pattern.pipe
val log = LoggerFactory.getLogger(this.getClass)
val myTokenActor = context.actorOf(Props[TokenMyActor2],"toknMyActor2")
implicit val timeout = Timeout(ReadTimeIntervalValue.getInterval(), SECONDS)
override def receive: Receive = {
case GetValue=>
ask(myTokenActor,CalculateValue).pipeTo(sender())
case message =>
log.warn(" Unhandled message received : {}", message)
unhandled(message)
}
}
child actor:
final case object CalculateValue
class TokenMyActor2 extends Actor{
import context.dispatcher
import akka.pattern.pipe
val log = LoggerFactory.getLogger(this.getClass)
override def receive: Receive = {
case CalculateValue=>
val future = Future{ "get the string"
}
val bac = future.map{result =>
sender ! result
}//.pipeTo(sender())
case message =>
log.warn("Actor MyActor: Unhandled message received : {}", message)
unhandled(message)
}
}
def main(args: Array[String]): Unit = {
implicit val timeout = Timeout(ReadTimeIntervalValue.getInterval(), SECONDS)
val myActor = system.actorOf(Props[MyActor],"myActor")
val future = ask(myActor, GetFinalValue).mapTo[String]
future.map {str =>
log.info ("string is {}",str)
}
Here are the logs:
[INFO] [akkaDeadLetter][01/12/2021 19:17:22.000] [api-akka.actor.default-dispatcher-5] [akka://api/deadLetters] Message [java.lang.String] from Actor[akka://api/user/myActor/Managemyactor/toknMyActor2#1239397461] to Actor[akka://api/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://api/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][01/12/2021 19:17:41.989] [api-akka.actor.default-dispatcher-7] [akka://api/deadLetters] Message [akka.actor.Status$Failure] from Actor[akka://api/user/myActor#1829301550] to Actor[akka://api/deadLetters] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://api/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][01/12/2021 19:17:41.996] [api-akka.actor.default-dispatcher-7] [akka://api/deadLetters] Message [akka.actor.Status$Failure] from Actor[akka://api/user/myActor/Managemyactor#-269929265] to Actor[akka://api/deadLetters] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://api/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
Please guide me where am I mistaken, or pipeTo should not be used like this? if so what should i do to make it work

Not sure if it's intended or not but ask(myManageActor,GetValue).pipeTo(sender()) can be implemented as forward.
class MyActor extends Actor {
lazy val myManageActor: ActorRef = ???
override def receive: Receive = {
case GetFinalValue =>
myManageActor.forward(GetValue)
}
}
forward is the same as tell but it preserves the original sender of the messages.
This can be applied to MyActor and ManagerMyActor.
In the case of TokenMyActor2, you should not use
future.map{ result =>
sender ! result
}
as it it breaks akka context encapsulation, as specified in docs
When using future callbacks, such as onComplete, or map such as
thenRun, or thenApply inside actors you need to carefully avoid
closing over the containing actor’s reference, i.e. do not call
methods or access mutable state on the enclosing actor from within the
callback. This would break the actor encapsulation and may introduce
synchronization bugs and race conditions because the callback will be
scheduled concurrently to the enclosing actor. Unfortunately there is
not yet a way to detect these illegal accesses at compile time. See
also: Actors and shared mutable state
You should instead rely on Future(???).pipeTo(sender()), which is safe to use with sender().
After applying these changes, the code does work as expected
case object GetFinalValue
case object GetValue
case object CalculateValue
class MyActor extends Actor {
private val myManageActor: ActorRef =
context.actorOf(Props[ManagerMyActor], "myManageActor")
override def receive: Receive = { case GetFinalValue =>
myManageActor.forward(GetValue)
}
}
class ManagerMyActor extends Actor {
private val myTokenActor =
context.actorOf(Props[TokenMyActor2], "toknMyActor2")
override def receive: Receive = { case GetValue =>
myTokenActor.forward(CalculateValue)
}
}
class TokenMyActor2 extends Actor {
import context.dispatcher
override def receive: Receive = { case CalculateValue =>
val future = Future { "get the string" }
future.pipeTo(sender())
}
}
implicit val timeout = Timeout(3, SECONDS)
implicit val system = ActorSystem("adasd")
import system.dispatcher
val myActor = system.actorOf(Props[MyActor], "myActor")
val future = ask(myActor, GetFinalValue).mapTo[String]
future.foreach { str =>
println(s"got $str")
}
Produces got get the string.
As a final note, I'd advise not to use ask pattern within actors. The basic functionality of ask can be easily achieved with just tell and forward. Also the code is shorter and not overloaded with constant need of implicit val timeout

Just to add on top of the great post by #IvanStanislavciuc. You already noticed that you lose the reference to the sender in futures. A simple solution for that will be to keep it up front.
It means that changing in MyActor:
ask(myManageActor,GetValue).pipeTo(sender()) // Won't work
into:
val originalSender = sender()
ask(myTokenActor,CalculateValue).pipeTo(originalSender)
In ManagerMyActor, Change:
ask(myTokenActor,CalculateValue).pipeTo(sender()) // Won't work
into:
val originalSender = sender()
ask(myManageActor,GetValue).pipeTo(originalSender)
And in TokenMyActor2:
val originalSender = sender()
Future{ "get the string" }.pipeTo(originalSender)
Code run at Scastie.

Related

Actor supervised by BackoffSupervisor loses stashed messages after restart

I have an actor with stash usage. Sometimes, when it crashes, it loses all stashed messages. I found that it depends on what supervision logic I use.
I wrote a simple example.
An actor with the stash:
case object WrongMessage
case object TestMessage
case object InitialMessage
class TestActor extends Actor with Stash {
override def receive: Receive = uninitializedReceive
def uninitializedReceive: Receive = {
case TestMessage =>
println(s"stash test message")
stash()
case WrongMessage =>
println(s"wrong message")
throw new Throwable("wrong message")
case InitialMessage =>
println(s"initial message")
context.become(initializedReceive)
unstashAll()
}
def initializedReceive: Receive = {
case TestMessage =>
println(s"test message")
}
}
In the following code, TestActor never receives stashed TestMessage:
object Test1 extends App {
implicit val system: ActorSystem = ActorSystem()
val actorRef = system.actorOf(BackoffSupervisor.props(Backoff.onFailure(
Props[TestActor], "TestActor", 1 seconds, 1 seconds, 0
).withSupervisorStrategy(OneForOneStrategy()({
case _ => SupervisorStrategy.Restart
}))))
actorRef ! TestMessage
Thread.sleep(5000L)
actorRef ! WrongMessage
Thread.sleep(5000L)
actorRef ! InitialMessage
}
But this code works well:
class SupervisionActor extends Actor {
val testActorRef: ActorRef = context.actorOf(Props[TestActor])
override def supervisorStrategy: SupervisorStrategy = OneForOneStrategy()({
case _ => SupervisorStrategy.Restart
})
override def receive: Receive = {
case message => testActorRef forward message
}
}
object Test2 extends App {
implicit val system: ActorSystem = ActorSystem()
val actorRef = system.actorOf(Props(classOf[SupervisionActor]))
actorRef ! TestMessage
Thread.sleep(5000L)
actorRef ! WrongMessage
Thread.sleep(5000L)
actorRef ! InitialMessage
}
I looked into sources and found that actor supervision uses
LocalActorRef.restart method which backed by system dispatcher logic, but BackoffSupervisor simply creates a new actor after termination of the old one. Is there any way to work around it?
I'm not sure one can make restart under BackoffSupervisor properly send stashed messages without some custom re-implementation effort.
As you've already pointed out that BackoffSupervisor does its own restart that bypasses the standard actor lifecycle. In fact, it's explicitly noted in the BackoffOnRestartSupervisor source code:
Whatever the final Directive is, we will translate all Restarts to our
own Restarts, which involves stopping the child.
In case you haven't read about this reported issue, it has a relevant discussion re: problem with Backoff.onFailure.
Backoff.onStop would also give the wanted BackoffSupervisor feature, but unfortunately it has its own use cases and won't be triggered by an exception.

Akka broadcast ask doesn't deliver to all routees

I am learning to use broadcast messages in Akka routers. Is there a way to receive responses for ask from all the routees to the router?
I have this sample code.
Master.scala
object Master {
case object brdcst
}
class Master extends Actor {
implicit val timeout = Timeout(5 seconds)
val router: ActorRef = context.actorOf (RoundRobinPool (3).props(Props[Worker]), "router")
override def receive: Receive = {
case brdcst => {
val future = router ? Broadcast(brdcst)
val result = Await.result(future, timeout.duration)
println("result = " + result)
}
}
}
object MasterTest extends App {
val actorSystem = ActorSystem("ActorSystem")
val actor = actorSystem.actorOf(Props[master], "root")
actor ! brdcst
}
Worker.scala
class Worker extends Actor {
val routee = context.actorOf(Props[Worker], "routee")
override def receive: Receive = {
case brdcst => sender() ! self.path.name
}
}
This code gives the following output
result = $a
[INFO] [10/16/2018 21:47:07.484] [ActorSystem-akka.actor.default-dispatcher-2] [akka://ActorSystem/deadLetters] Message [java.lang.String] from Actor[akka://ActorSystem/user/root/router/$a#340358688] to Actor[akka://ActorSystem/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior, then [Actor[akka://ActorSystem/deadLetters]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [10/16/2018 21:47:07.504] [ActorSystem-akka.actor.default-dispatcher-10] [akka://ActorSystem/deadLetters] Message [java.lang.String] from Actor[akka://ActorSystem/user/root/router/$b#-151225340] to Actor[akka://ActorSystem/deadLetters] was not delivered. [2] dead letters encountered. If this is not an expected behavior, then [Actor[akka://ActorSystem/deadLetters]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
So the reply is coming only from one routee. How to get responses from all the routees? (Maybe as a list like result = [$a, $b, $c])
ask (?) creates an internal actor to handle replies. This internal actor handles only a single reply and is automatically shut down: this is the reason you get only the first response, while the responses from the other two routees go to dead letters.
To get the desired behavior, use tell (!) and collect the responses from the routees. For example:
class Master extends Actor {
val numRoutees = 3
val router = context.actorOf(RoundRobinPool(numRoutees).props(Props[Worker]), "router")
def handleMessages(replies: Set[String] = Set()): Receive = {
case brdcst =>
router ! Broadcast(brdcst)
case reply: String =>
val updatedReplies = replies + reply
if (updatedReplies.size == numRoutees) {
println("result = " + updatedReplies.mkString("[", ",", "]"))
}
become(handleMessages(updatedReplies))
}
def receive = handleMessages
}
In the above example, the master encodes the routees' replies as part of its state, using become.
Also, don't use Await in actors.

when an actor contains an async method, this will lead to dead letter error with the Ask time out exception

I use ask mode send a request to an actor called actor-A in a function called fun-F. The actor will get an ID generated by another system in an async way, when this is completed, I will forward a message contains this ID to another actor called actor-B, the actor-B will do some DB operations and then send back the DB operation result in a message to the sender, since in my case I use forward mode, so the actor-B recognize the sender as fun-F, the akka will give the fun-F a temporary actor name, so the returned value should be delivered to the temp actor.
My question is:
If I use sync-method to get the ID from another system, then forward this message to the actor-B, after actor-B's DB operation, the result can be delivered to the value of fun-F, and the fun-F is defined as temporary actor Actor[akka://ai-feedback-service/temp/$b] by the akka framework runtime.
If I use async-method to get the ID from another system, when it completed, I will forward the message in the oncompleted {} code block in another call back thread, the DB operation in actor-B is handled successfully, but the returned value cannot be delivered to the value defined in the fun-F, and in this case the fun-F is defined as Actor[akka://ai-feedback-service/deadLetters] by the akka framwork runtime. So the actor-B lose its way and do not know how to get back or where should this message be delivered, and this will cause an Ask time out exception throws in my log.
How can I handled this issue? or how can I avoid this dead letter ask time out exception?
Below is my code:
// this is the so-called fun-F [createFeedback]
def createFeedback(query: String,
response: String,
userId: Long,
userAgent: String,
requestId: Long,
errType: Short,
memo: String): Future[java.lang.Long] = {
val ticket = Ticket(userId,
requestId,
query,
response,
errType,
userAgent,
memo)
val issueId = (jiraActor ? CreateJiraTicketSignal(ticket))
.mapTo[CreateFeedbackResponseSignal].map{ r =>
r.issueId.asInstanceOf[java.lang.Long]
}
issueId
}
//this is the so-called actor-A [jiraActor]
//receive method are run in its parent actor for some authorization
//in this actor only override the handleActorMsg method to deal msg
override def handleActorMsg(msg: ActorMsgSignal): Unit = {
msg match {
case s:CreateJiraTicketSignal =>
val issueId = createIssue(cookieCache.cookieContext.flag,
cookieCache.cookieContext.cookie,
s.ticket)
println(s">> ${sender()} before map $issueId")
issueId.map{
case(id:Long) =>
println(s">> again++issueId = $id ${id.getClass}")
println(s">>> $self / ${sender()}")
println("again ++ jira action finished")
dbActor.forward(CreateFeedbackSignal(id,s.ticket))
case(message:String) if(!s.retry) =>
self ! CreateJiraTicketSignal(s.ticket,true)
case(message:String) if(s.retry) =>
log.error("cannot create ticket :" + message)
}
println(s">> after map $issueId")
}
//this is the so-called actor-B [dbActor]
override def receive: Receive = {
case CreateFeedbackSignal(issueId:Long, ticket:Ticket) =>
val timestampTicks = System.currentTimeMillis()
val description: String = Json.obj("question" -> ticket.query,
"answer" -> ticket.response)
.toString()
dao.createFeedback(issueId,
ticket.usrId.toString,
description,
FeedbackStatus.Open.getValue
.asInstanceOf[Byte],
new Timestamp(timestampTicks),
new Timestamp(timestampTicks),
ticket.usrAgent,
ticket.errType,
ticket.memo)
println(s">> sender = ${sender()}")
sender() ! (CreateFeedbackResponseSignal(issueId))
println("db issue id is " + issueId)
println("db action finished")
}
To avoid the dead letters issue, do the following:
For every request, use an identifier (probably the requestId) that you can associate with the ultimate target for the request. That is, tie the requestId that you're passing to the createFeedback method to the caller (ActorRef) of that method, then pass this id through your messaging chain. You can use a map to hold these associations.
Change CreateFeedbackResponseSignal(issueId) to include the requestId from the Ticket class: CreateFeedbackResponseSignal(requestId, issueId).
When dealing with the asynchronous result of a Future from inside an actor, pipe the result of the Future to self instead of using a callback.
With this approach, the result of createIssue will be sent to jiraActor when the result is available. jiraActor then sends that result to dbActor.
jiraActor will be the sender in dbActor. When jiraActor receives the result from dbActor, jiraActor can look up the reference to the target in its internal map.
Below is a simple example that mimics your use case and is runnable in ScalaFiddle:
import akka.actor._
import akka.pattern.{ask, pipe}
import akka.util.Timeout
import language.postfixOps
import scala.concurrent._
import scala.concurrent.duration._
case class Signal(requestId: Long)
case class ResponseSignal(requestId: Long, issueId: Long)
object ActorA {
def props(actorB: ActorRef) = Props(new ActorA(actorB))
}
class ActorA(dbActor: ActorRef) extends Actor {
import context.dispatcher
var targets: Map[Long, ActorRef] = Map.empty
def receive = {
case Signal(requestId) =>
val s = sender
targets = targets + (requestId -> s)
createIssue(requestId).mapTo[Tuple2[Long, Long]].pipeTo(self) // <-- use pipeTo
case ids: Tuple2[Long, Long] =>
println(s"Sending $ids to dbActor")
dbActor ! ids
case r: ResponseSignal =>
println(s"Received from dbActor: $r")
val target = targets.get(r.requestId)
println(s"In actorA, sending to: $target")
target.foreach(_ ! r)
targets = targets - r.requestId
}
}
class DbActor extends Actor {
def receive = {
case (requestId: Long, issueId: Long) =>
val response = ResponseSignal(requestId, issueId)
println(s"In dbActor, sending $response to $sender")
sender ! response
}
}
val system = ActorSystem("jiratest")
implicit val ec = system.dispatcher
val dbActor = system.actorOf(Props[DbActor])
val jiraActor = system.actorOf(Props(new ActorA(dbActor)))
val requestId = 2L
def createIssue(requestId: Long): Future[(Long, Long)] = {
println(s"Creating an issue ID for requestId[$requestId]")
Future((requestId, 99L))
}
def createFeedback(): Future[Long] = {
implicit val timeout = Timeout(5.seconds)
val res = (jiraActor ? Signal(requestId)).mapTo[ResponseSignal]
res.map(_.issueId)
}
createFeedback().onComplete { x =>
println(s"Done: $x")
}
Running the above code in ScalaFiddle results in the following output:
Creating an issue ID for requestId[2]
Sending (2,99) to dbActor
In dbActor, sending ResponseSignal(2,99) to Actor[akka://jiratest/user/$b#-710097339]
Received from dbActor: ResponseSignal(2,99)
In actorA, sending to: Some(Actor[akka://jiratest/temp/$a])
Done: Success(99)

Akka ask returns nothing when tell is used in future

The following code uses 2 asks and initially seems like it supposed to print "This should be printed in actor1 but most of the time wont because of a race condition with the receive method finishing".
However there seems to be a race condition between the 'receive' method finishing and the future finishing and thus nothing is printed.
Is this the intended behavior by akka? is this a bug?
I am trying to avoid using 'ask' as much as possible and using 'tell' instead, but sometimes it's a must.
import akka.actor._
import akka.routing.SmallestMailboxPool
import akka.util.Timeout
import akka.pattern.ask
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Failure, Success}
object ActorsAskingStuff extends App {
val system = ActorSystem("abc-system")
val actor1 = system.actorOf(Props[Actor1].withRouter(SmallestMailboxPool(1)), "actor1")
actor1 ! 5
}
class Actor1 extends Actor {
implicit val timeout = Timeout(60 seconds)
val actor2: ActorRef = context.actorOf(Props[Actor2].withRouter(SmallestMailboxPool(1)), "actor2")
override def receive: Receive = {
case _ =>
println("actor1 was called")
actor2 ? "hello" onComplete {
case Success(a) => println(a)
case Failure(b) => println("FAILURE")
}
}
}
class Actor2 extends Actor {
implicit val timeout = Timeout(6 seconds)
val actor3: ActorRef = context.actorOf(Props[Actor3].withRouter(SmallestMailboxPool(1)), "actor3")
override def receive: Actor.Receive = {
case _ =>
println("actor2 was called")
actor3 ? "hello" map {
_ =>
println("Actor2 completed the future")
sender ! "This should be printed in actor1 but most of the time wont because of a race condition with the receive method finishing"
}
// uncomment this to make it work
//Thread.sleep(100)
}
}
class Actor3 extends Actor {
override def receive: Actor.Receive = {
case _ =>
println("actor3 was called")
sender ! "I'm actor3"
}
}
You are closing over the sender when you do this:
actor3 ? "hello" map {
_ =>
println("Actor2 completed the future")
sender ! "This should be printed in actor1 but most of the time wont because of a race condition with the receive method finishing"
}
At that point the sender could have changed. In order to avoid closing over the sender you can rewrite like this:
val originator = sender()
actor3 ? "hello" map {
_ =>
println("Actor2 completed the future")
originator ! "This should be printed in actor1 but most of the time wont because of a race condition with the receive method finishing"
}
Once you go into a Future based callback (like onComplete or map for example) at that point, the actor thinks it's done precessing the current message and moves onto the next message in the mailbox (if there is one). When this happens, the sender() which is just a def that returns the value of a var that is subject to change will return either a new ActorRef (more messages) or nothing (deadletter) if no messages.
You can checkout this post for more info:
sender inside a future

How to send iterables between actors or from an actor to a Future?

A future from the main method of a program sends a msg to its actor asking for an iterable object. The actor then creates another future that asks for the iterable object (say an ArrayBuffer) from a remote actor. After receiving the ArrayBuffer from the remote actor, how would the actor send it back to the first future in the main method? It seems creating a local alias of sender and creating a separate case class to represent the iterable does not prevent dead letters from being encountered.
Here is a sample code:
case class SequenceObject(sqnce:Seq[someArrayBuffer])
//...
implicit val timeout = Timeout(10 seconds)
val fut1: Future[Any] = myActor ? iNeedAnArrayBufferObject
fut1.onSuccess {
case listOfItems: SequenceObject => {
//do sth with listofItems.sqnce
}
class myActor extends Actor {
implicit val timeout = Timeout(1 seconds)
def receive = {
case a: iNeedAnArrayBufferObject => {
val originalSender = sender
val fut: Future[Any] = (remoteActor ? a)
fut.onSuccess {
case list: SequenceObject => {
originalSender ! SequenceObject(list.sqnce)
}
}
The remote actor code is:
class ServerActorClass extends Actor {
def receive = {
case a: iNeedAnArrayBufferObject => {
val closer = sender()
closer ! SequenceObject(ArrayBufferObject[information])
}
}
The above does not seem to work. The remote actor and the local actor can communicate and messages are received correctly. However, the iterable object is never send back to fut1. Why is that? Thanks in advance.
Check pipeTo pattern in Ask: Send-And-Receive-Future section