ExecutionContext causes Akka dead letter - scala

For some reason I have to use gRPC and Akka at the same time. When this actor is started as a top actor, nothing goes wrong (in this little demo). But when it becomes a child actor, it cannot receive any messages, and the following is logged:
[default-akka.actor.default-dispatcher-6] [akka://default/user/Grpc] Message [AkkaMessage.package$GlobalStart] from Actor[akka://default/user/TrackerCore#-808631363] to Actor[akka://default/user/Grpc#-1834173068] was not delivered. [1] dead letters encountered.
The example core:
class GrpcActor() extends Actor {
val ec = scala.concurrent.ExecutionContext.global
val service = grpcService.bindService(new GrpcServerImpl(), ec)
override def receive: Receive = {
case GlobalStart() => {
println("GlobalStart")
}
...
}
}
I tried to create a new ExecutionContext like:
scala.concurrent.ExecutionContext.fromExecutor(Executors.newFixedThreadPool(10))
Why is this happening, and how do can I debug a dead letters problem like this (no exception is thrown)?
Update:
Sorry I didn't list everything here. I used normal Main method to test GrpcActor as top actor, and ScalaTest to test it as child actor, which is a mistake.
class GrpcActorTest extends FlatSpec with Matchers{
implicit val system = ActorSystem()
val actor: ActorRef = system.actorOf(Props[GrpcActor])
actor ! GlobalStart()
}
It is this empty test suite that active shutdown the whole actor system. But the problem is with this line
val service = grpcService.bindService(new GrpcServerImpl(), ec)
the delivery of GlobalStart() was delayed after the shutdown.
Without that line, message can be delivered before the shutdown.
Is this a normal behavior?
(My guess: it happened that the GlobalStart() was queued after the shutdown message with that line, which did some heavy work and made the difference in time)

One way to address the problem is to make service a lazy val:
class GrpcActor extends Actor {
...
lazy val service = grpcService.bindService(new GrpcServerImpl(), ec)
...
}
A lazy val is useful for long-running operations: in this case, it defers the initialization of service until it is used for the first time. Without the lazy modifier, service is initialized when the actor is created.
An alternative approach is to add a Thread.sleep in your test to prevent the actor system from shutting down before the actor has fully initialized:
class GrpcActorTest extends FlatSpec with Matchers {
...
actor ! GlobalStart()
Thread.sleep(5000) // or whatever length of time is needed to initialize the actor
}
(As a side note, consider using the Akka Testkit for your actor tests.)

Add a supervisor strategy to its parent, add println to the actor lifecycle. There is something that kill your actor . Finally, if you provide a complete example maybe I can say more :)

Related

Ensure message order in test when mixing futures with actor messages

I'm testing an actor that uses an asnychronous future-based API. The actor uses the pipe pattern to send a message to itself when a future completes:
import akka.pattern.pipe
// ...
// somewhere in the actor's receive method
futureBasedApi.doSomething().pipeTo(self)
In my test I mock the API so I control future completion via promises. However, this is interleaved with other messages sent directly to the actor:
myActor ! Message("A")
promiseFromApiCall.success(Message("B"))
myActor ! Message("C")
Now I'm wondering how I can guarantee that the actor receives and
processes message B between message A and C in my test because message B is actually sent in another thread, so I can't control the order
in which the actor's mailbox receives the messages.
I thought about several possible solutions:
sleep after each message for a few milliseconds to make another
order very unlikely
wait for the actor to acknowledge each message, although
acknowledgement is only required for testing
send message B directly to the actor to simulate completion of the
future and write a separate test that ensures that the pipe pattern
is properly used (the test above would not fail if the actor would
not pipe the result message to itself)
I don't really like either of these options but I tend to use the last
one. Is there another better way I can enforce a certain message order in the tests?
Clarification: The question is not how to deal with the fact that messages might be received in random order in production. Controlling the order in the test is essential to make sure that the actor can actually deal with different message orders.
One idea is to define a flag in your actor that indicates whether the actor has received message B. When the actor receives message C, the actor can stash that message C if the flag is false, then unstash it once the actor receives message B. For example:
class MyActor extends Actor with Stash {
def receiveBlock(seenMsgB: Boolean, seenMsgC: Boolean): Receive = {
case MakeApiCall =>
callExternalApi().mapTo[MessageB].pipeTo(self)
case m: MessageB if seenMsgC => // assume msg C has been stashed
unstashAll()
// ...do something with msg B
become(receiveBlock(true, seenMsgC)) // true, true
case m: MessageB if !seenMsgC =>
// ...do something with message B
become(receiveBlock(true, seenMsgC)) // true, false
case m: MessageC if seenMsgB =>
// ...do something with message C
context.become(receiveBlock(seenMsgB, true)) // true, true
case m: MessageC if !seenMsgB =>
stash()
context.become(receiveBlock(seenMsgB, true)) // false, true
case ...
}
def receive = receiveBlock(false, false)
}
After reading a lot more about akka, I finally found a better solution: Replacing the actor mailbox with one I can observe in the tests. This way I can wait until the actor receives a new message after I complete the promise. Only then the next message is sent. The code for this TestingMailbox is given at the end of the post.
Update: In Akka Typed this can be achieved very elegantly with a BehaviorInterceptor. Just wrap the Behavior under test with a custom interceptor that forwards all messages and signals but lets you observe them.
The mailbox solution for untyped Akka is given below.
The actor can be configured like this:
actorUnderTest = system.actorOf(Props[MyActor]).withMailbox("testing-mailbox"))
I have to make sure the "testing-mailbox" is known by the actor system by providing a configuration:
class MyTest extends TestKit(ActorSystem("some name",
ConfigFactory.parseString("""{
testing-mailbox = {
mailbox-type = "my.package.TestingMailbox"
}
}""")))
with BeforeAndAfterAll // ... and so on
With this being set up, I can change my test like this:
myActor ! Message("A")
val nextMessage = TestingMailbox.nextMessage(actorUnderTest)
promiseFromApiCall.success(Message("B"))
Await.ready(nextMessage, 3.seconds)
myActor ! Message("C")
With a little helper method, I can even write it like this:
myActor ! Message("A")
receiveMessageAfter { promiseFromApiCall.success(Message("B")) }
myActor ! Message("C")
And this is my custom mailbox:
import akka.actor.{ActorRef, ActorSystem}
import akka.dispatch._
import com.typesafe.config.Config
import scala.concurrent.{Future, Promise}
object TestingMailbox {
val promisesByReceiver =
scala.collection.concurrent.TrieMap[ActorRef, Promise[Any]]()
class MessageQueue extends UnboundedMailbox.MessageQueue {
override def enqueue(receiver: ActorRef, handle: Envelope): Unit = {
super.enqueue(receiver, handle)
promisesByReceiver.remove(receiver).foreach(_.success(handle.message))
}
}
def nextMessage(receiver: ActorRef): Future[Any] =
promisesByReceiver.getOrElseUpdate(receiver, Promise[Any]).future
}
class TestingMailbox extends MailboxType
with ProducesMessageQueue[TestingMailbox.MessageQueue] {
import TestingMailbox._
def this(settings: ActorSystem.Settings, config: Config) = this()
final override def create(owner: Option[ActorRef],
system: Option[ActorSystem]) =
new MessageQueue()
}
If it is so important to order messages you should use ask (?) which returns Future and chain them even if you dont expect any response from an actor.

What happens to unmatched messages with Akka?

The Akka documentation says that the mailbox is NOT scanned for messages. Each message is handled after another (FIFO by default) in the order of their arrival. However, when I send a message from an actor to another which is not matched be the receiving actor, it is neither moved to the deadletters actor (which would appear in the log I suppose) nor does it block handling the next message in the mailbox which arrives one second later and can be handled correctly.
What does happen to the unmatched message from the mailbox?
I am using Scala 2.10.4 and Akka 2.4-SNAPSHOT in sbt.
package main.scala
import akka.actor._
class SRActor(dest: ActorRef) extends Actor with ActorLogging {
dest ! A
dest ! B
context.stop(self)
override def receive = {
case _ => {
log.info("Finally got something")
}
}
}
class SRActorReceiver extends Actor with ActorLogging {
override def receive = {
case B =>
log.info("Finally got B")
}
}
Actor creation:
package main.scala
import akka.actor._
case object A
case object B
object ErrorApp extends App {
// SR: Send nowhere received
var system6 = ActorSystem("ErrorActorSystem")
val srActorReceiver = system6.actorOf(Props(classOf[SRActorReceiver]), "sractorreceiver")
val sractor = system6.actorOf(Props(classOf[SRActor], srActorReceiver), "sractor")
// wait until actors have finished
Thread.sleep(1000)
system6.shutdown
Copied from docs
Please note that the Akka Actor receive message loop is exhaustive,
which is different compared to Erlang and the late Scala Actors. This
means that you need to provide a pattern match for all messages that
it can accept and if you want to be able to handle unknown messages
then you need to have a default case as in the example above.
Otherwise an akka.actor.UnhandledMessage(message, sender, recipient)
will be published to the ActorSystem's EventStream.
There is also unhandled method in Actor trait that you can override. (docs)
def unhandled(message: Any): Unit
User overridable callback.
Is called when a message isn't handled by the current behavior of the actor by default it fails with either a akka.actor.DeathPactException (in case of an unhandled akka.actor.Terminated message) or publishes an akka.actor.UnhandledMessage to the actor's system's akka.event.EventStream

Scala and Akka - Testing actors as a system with Akka Testkit

In my Scala application say I have Actor A and Actor B. I want to devise a test case in ScalaTest that would allow me to send a message to Actor A and see what message it sends to Actor B in order to see if A is properly processing it's data and sending the right message to B. How would one test this? It took me a long time to get this cooked up on my own...but it does seem to mostly work.
class A extends Actor { ... }
class B extends Actor { ... }
class C(p: TestProbe) extends B {
override def receive = {
LoggingReceive {
case x =>
println(x.toString)
p.ref ! x
}
}
}
case class MsgToB(...)
// Spec class which extends TestKit
"A" should {
"send the right message to B" {
val p = TestProbe()
val a = TestActorRef[A]
val c = TestActorRef(Props(new C(p)))
// Assume A has a reference to C. Not shown here.
a ! msg
// Assert messages
p.expectMsgType[MsgToB]
}
}
Is this the best means of doing this? Is there a better practice?
To me it sounds like what you want is to test the behaviour of actor A in isolation. In order to do this, you need to be able to control how actor A gets its reference to actor B. For example, you could provide the reference in the actor's constructor:
import akka.actor.{Actor, ActorRef, Props}
class A(refToB: ActorRef) extends Actor { ... }
object A {
def props(refToB: ActorRef): Props = Props(new A(refToB))
}
There are alternative ways you can pass the reference to actor B to actor A, but using the constructor is arguably the easiest choice. In the example above, we also provide a method for creating the correct Props for the actor.
Now that you can control the reference to actor B, you can replace the actor reference with test probe in tests.
import akka.testkit.TestProbe
// Initialise a test probe
val probe = TestProbe()
// Actor A with reference to actor B replaced with the test probe
val a = system.actorOf(A.props(probe.ref))
// Send a message to actor A
a ! someMessage
// Verify that the probe received a correct response from actor A
p.expectMsgType[MsgToB]
Notice that I created the actor using the actor system from the TestKit instead of using the TestActorRef. This means that the actor message processing will be asynchronous instead of synchronous. Personally, I've found the asynchronous testing style to be a better fit because it better represents how the actor is run in a production system. Asynchronous testing is also recommended in the official documentation.

Shutting down an Akka Actor system from an Actor

I know I can use system.shutdown() outside of an actor system to stop the actor system and make system.awaitTermination() stop blocking execution on its thread.
I want to trigger the actor system shutdown from within one of my Actors. I thought I should be able to call context.system.shutdown() from within an Actor's receive method, however when I do this nothing seems to happen, system.awaitTermination() (on the main thread) just keeps on blocking.
Any ideas?
How to program an Actor to kill self
class WatchActor extends Actor {
val child = context.actorOf(Props.empty, "child")
context.watch(child) // <-- this is the only call needed for registration
var lastSender = system.deadLetters
def receive = {
case "kill" =>
context.stop(child); lastSender = sender()
case Terminated(`child`) => lastSender ! "finished"
}
}
At this url there is a good explain.
http://doc.akka.io/docs/akka/snapshot/scala/actors.html
jfm
You should not do that: it is like a snake biting its own tail.
system.awaitTermination() will keep blocking because you are precisely waiting for it to end!
You can perfectly call context.system.shutdown() within an actor, but you cannot call system.awaitTermination() in the actor system context. And also it does not make a lot of sense: why would you wait since there is anyway nothing to execute afterwards, the system being down?
Blocking on the system to shut down makes sense outside of it only, if you want to execute further instructions after it has been stopped.
Main Thread:
val l = new CountDownLatch(1)
val s = ActorSystem("system", config)
...
//create some actors and send some messages to them
...
l.await()
s.shutdown()
s.awaitTermination()
Some where in def receive: Actor.Receive:
l.countDown()

Assert order of messages received using Akka TestProbe

We have an actor that we are writing unit tests for, and as part of the tests we want to assert that certain messages are sent to another actor in a certain order. In our unit tests, the actor receiving the messages is represented by an Akka TestProbe, which gets injected into the actor under test when it is created.
It is no problem to assert that the messages were sent to the test probe, however we have been struggling to work out a way to assert they are sent in the correct order (we could not find any suitable methods for doing this in the documentation). Any ideas how we achieve this?
Below is a minimal implementation that highlights the problem.
Implementation
case class Message(message: String)
case class ForwardedMessage(message: String)
class ForwardingActor(forwardTo: ActorRef) extends Actor {
def receive = {
case Message(message) =>
forwardTo ! ForwardedMessage(message)
}
}
Unit Test
class ForwardMessagesInOrderTest extends TestKit(ActorSystem("testSystem"))
with WordSpecLike
with MustMatchers {
"A forwarding actor" must {
val forwardingReceiver = TestProbe()
val forwardingActor = system.actorOf(Props(new ForwardingActor(forwardingReceiver.ref)))
"forward messages in the order they are received" in {
forwardingActor ! Message("First message")
forwardingActor ! Message("Second message")
// This is the closest way we have found of achieving what we are looking for, it asserts
// that both messages were received, but doesn't assert correct order. The test will pass
// regardless which way round we put the messages below.
forwardingReceiver.expectMsgAllOf(
ForwardedMessage("Second message"),
ForwardedMessage("First message"))
}
}
}
I'm going to suggest two changes to your test spec. First, when creating the actor under test, use a TestActorRef like so:
val forwardingActor = TestActorRef(new ForwardingActor(forwardingReceiver.ref))
Using a TestActorRef will assure that the CallingThreadDispatcher is used, removing any complications from testing async code (which an actor is). Once you do that, you can change your assertions to:
forwardingReceiver.expectMsg(ForwardedMessage("First message"))
forwardingReceiver.expectMsg(ForwardedMessage("Second message"))
These assertions are inherently In-Order, so if things came in out of this order, they will fail. This should fix your issues.