How to properly test actor behavior in case of failure - scala

It is very important to know how a system might behave in case a particular actor fails - how does the recovery run, are the snapshots fetched properly etc. Surprisingly, I have not found any advice on it in the docs
An approach I have seen in some examples is to have a special Fail msg and let the actor throw an Exception on this msg.
class MyActor extends Actor {
override def receive =
...
case Fail => throw new Exception("kaboom")
}
I hope never to use it as it mixes production code and test code and would probably generate some reasonable WTFs in unsuspecting readers.
How would I induce an arbitrary actor to fail properly?
Thank you

After some thought, I have implemented a quick solution. The test package contains a trait for failing which is then mixed into actors for failure testing:
object Failing{
case object Fail
}
trait Failing {
def failOnMsg: PartialFunction[Any, Unit] = {
case Fail => throw new Exception("kaboom!")
}
}
then in the test the PF is prepended to the original receive
val props = Props(new MyActor(arg) with Failing {
override def receive = failOnMsg orElse super.receive
})
val actor = system.actorOf(props)
actor ! Fail
It is still better than what I had before, but brings some boilerplate. I hope there is a better solution.

Related

How to supervise actors in Akka and handle exceptions

I am trying to improve the error handling with the actors in my system. Sometimes, when processing data, something goes wrong, and I need to stop and restart the actor, as well as log some information regarding this failure.
I have a Supervisor, which has 5 actors working for it. So I need to be able to supervise all of them. I found this link:
https://doc.akka.io/docs/akka/current/typed/fault-tolerance.html
regarding this, but I don't think it is very clear on where to implement the code:
Behaviors.supervise(behavior).onFailure[IllegalStateException](SupervisorStrategy.restart)
Where exactly is this code supposed to go?
Thanks
You can think of this supervisor as another behavioiur which wraps your behaviour inside of it.
Lets say you want to have following HelloWorld actor.
object HelloWorldActor {
sealed trait Command
case class HelloCommand(name: String) extends Command
def apply(): Behavior[Command] =
Behaviors.receiveMessage[Command] {
case msg: HelloCommand =>
println(s"Hello ${msg.name}")
Behaviors.same
}
}
Now, you can "wrap" this "behaviour" with a "supervisor"
object SupervisedHelloWorldActor {
sealed trait Command
case class HelloCommand(name: String) extends Command
def apply(): Behavior[Command] =
Behaviors.supervise(
Behaviors.receiveMessage[Command] {
case HelloCommand(name) =>
println(s"Hello ${name}")
Behaviors.same
}
).onFailure(onFailure[IllegalStateException](SupervisorStrategy.restart))
}

Test Message Adapters in Akka Typed

I am using Akka Typed (version 2.6.8) and I developed an actor that uses a message adapter.
object EncoderClient {
sealed trait Command
final case class KeepASecret(secret: String) extends Command
private final case class WrappedEncoderResponse(response: Encoded) extends Command
def apply(encoder: ActorRef[Encode]): Behavior[Command] =
Behaviors.setup { context =>
val encoderResponseMapper: ActorRef[Encoded] =
context.messageAdapter(response => WrappedEncoderResponse(response))
Behaviors.receiveMessage {
case KeepASecret(secret) =>
encoder ! Encode(secret, encoderResponseMapper)
Behaviors.same
case WrappedEncoderResponse(response) =>
context.log.info(s"I will keep a secret for you: ${response.payload}")
Behaviors.same
}
}
}
I want to test the effect of creation of the Message Adapter. I see that there is a class in the testkit library, MessageAdapter that seems to be a perfect fit to my needs.
However, I can't find anywhere an example of how to use it. Any help?
The MessageAdapter effect in the behavior testkit only signals that an adapter was created, as a result of a message or actor start. It won't help you with verifying that your adapter works.
You can inject a test probe for the encoder, expect a Encode message to get a hold of the message adapter ActorRef in the message. Something like this:
val probe = createTestProbe[Encode]()
val actor = spawn(EncoderClient(probe.ref))
actor ! KeepASecret("hush")
val encodeRequest = probe.receiveMessage()
encodeRequest ! Encoded(...)
// ... verify whatever happens on adapted response arriving...
I found a solution to my problem. If you're interested in testing if a message adapter is created by an actor for a particular type T, then you can use the BehaviorTestKit.expectEffectPF method.
It follows an example:
"EncoderClient" should "send an encoding request to the encode" in {
val encoder = TestInbox[Encode]()
val client = BehaviorTestKit(EncoderClient(encoder.ref))
client.run(KeepASecret("My secret"))
client.expectEffectPF {
case MessageAdapter(clazz, _) if clazz == classOf[Encoded] =>
}
}
If the object clazz is not of the tested type, the test fails.
Instead, if you're interested in testing that the message adapter works correctly, then, use the strategy suggested by johanandren.

Assert order of messages received using Akka TestProbe

We have an actor that we are writing unit tests for, and as part of the tests we want to assert that certain messages are sent to another actor in a certain order. In our unit tests, the actor receiving the messages is represented by an Akka TestProbe, which gets injected into the actor under test when it is created.
It is no problem to assert that the messages were sent to the test probe, however we have been struggling to work out a way to assert they are sent in the correct order (we could not find any suitable methods for doing this in the documentation). Any ideas how we achieve this?
Below is a minimal implementation that highlights the problem.
Implementation
case class Message(message: String)
case class ForwardedMessage(message: String)
class ForwardingActor(forwardTo: ActorRef) extends Actor {
def receive = {
case Message(message) =>
forwardTo ! ForwardedMessage(message)
}
}
Unit Test
class ForwardMessagesInOrderTest extends TestKit(ActorSystem("testSystem"))
with WordSpecLike
with MustMatchers {
"A forwarding actor" must {
val forwardingReceiver = TestProbe()
val forwardingActor = system.actorOf(Props(new ForwardingActor(forwardingReceiver.ref)))
"forward messages in the order they are received" in {
forwardingActor ! Message("First message")
forwardingActor ! Message("Second message")
// This is the closest way we have found of achieving what we are looking for, it asserts
// that both messages were received, but doesn't assert correct order. The test will pass
// regardless which way round we put the messages below.
forwardingReceiver.expectMsgAllOf(
ForwardedMessage("Second message"),
ForwardedMessage("First message"))
}
}
}
I'm going to suggest two changes to your test spec. First, when creating the actor under test, use a TestActorRef like so:
val forwardingActor = TestActorRef(new ForwardingActor(forwardingReceiver.ref))
Using a TestActorRef will assure that the CallingThreadDispatcher is used, removing any complications from testing async code (which an actor is). Once you do that, you can change your assertions to:
forwardingReceiver.expectMsg(ForwardedMessage("First message"))
forwardingReceiver.expectMsg(ForwardedMessage("Second message"))
These assertions are inherently In-Order, so if things came in out of this order, they will fail. This should fix your issues.

Failing a scalatest when akka actor throws exception outside of the test thread

I've had a situation come up and bite me a few times where I'm testing an Actor and the Actor throws an exception unexpectedly (due to a bug), but the test still passes. Now most of the time the exception in the Actor means that whatever the test is verifying won't come out properly so it the test fails, but in rare cases that's not true. The exception occurs in a different thread than the test runner so the test runner knows nothing about it.
One example is when I'm using a mock to verify some dependency gets called, and due to a mistake in the Actor code I call an unexpected method in the mock. That causes the mock to throw an exception which blows up the actor but not the test. Sometimes this can even cause downstream tests to fail mysteriously because of how the Actor blew up. For example:
// using scala 2.10, akka 2.1.1, scalatest 1.9.1, easymock 3.1
// (FunSpec and TestKit)
class SomeAPI {
def foo(x: String) = println(x)
def bar(y: String) = println(y)
}
class SomeActor(someApi: SomeAPI) extends Actor {
def receive = {
case x:String =>
someApi.foo(x)
someApi.bar(x)
}
}
describe("problem example") {
it("calls foo only when it receives a message") {
val mockAPI = mock[SomeAPI]
val ref = TestActorRef(new SomeActor(mockAPI))
expecting {
mockAPI.foo("Hi").once()
}
whenExecuting(mockAPI) {
ref.tell("Hi", testActor)
}
}
it("ok actor") {
val ref = TestActorRef(new Actor {
def receive = {
case "Hi" => sender ! "Hello"
}
})
ref.tell("Hi", testActor)
expectMsg("Hello")
}
}
"problemExample" passes, but then downstream "ok actor" fails for some reason I don't really understand... with this exception:
cannot reserve actor name '$$b': already terminated
java.lang.IllegalStateException: cannot reserve actor name '$$b': already terminated
at akka.actor.dungeon.ChildrenContainer$TerminatedChildrenContainer$.reserve(ChildrenContainer.scala:86)
at akka.actor.dungeon.Children$class.reserveChild(Children.scala:78)
at akka.actor.ActorCell.reserveChild(ActorCell.scala:306)
at akka.testkit.TestActorRef.<init>(TestActorRef.scala:29)
So, I can see ways of catching this sort of thing by examining the logger output in afterEach handlers. Definitely doable, although a little complicated in cases where I actually expect an exception and that's what I'm trying to test. But is there any more direct way of handling this and making the test fail?
Addendum: I have looked at the TestEventListener and suspect there's maybe something there that would help, but I can't see it. The only documentation I could find was about using it to check for expected exceptions, not unexpected ones.
Thinking in Actors there is also another solution: failures travel to the supervisor, so that is the perfect place to catch them and feed them into the test procedure:
val failures = TestProbe()
val props = ... // description for the actor under test
val failureParent = system.actorOf(Props(new Actor {
val child = context.actorOf(props, "child")
override val supervisorStrategy = OneForOneStrategy() {
case f => failures.ref ! f; Stop // or whichever directive is appropriate
}
def receive = {
case msg => child forward msg
}
}))
You can send to the actor under test by sending to failureParent and all failures—expected or not—go to the failures probe for inspection.
Other than examining the logs, I can think of two ways to fail tests when an actor crashes:
Ensure that no Terminated message is received
Check the TestActorRef.isTerminated property
The latter option is deprecated, so I'll ignore it.
Watching Other Actors from Probes describes how to setup a TestProbe. In this case it might look something like:
val probe = TestProbe()
probe watch ref
// Actual test goes here ...
probe.expectNoMessage()
If the actor dies due to an exception it will generate the Terminated message. If that happens during the test and you expect something else, the test will fail. If it happens after your last message expectation, then the expectNoMessage() should fail when Terminated is received.
Okay, I've had a little time to play with this. I've got a nice solution that uses an event listener and filter to catch errors. (Checking isTerminated or using TestProbes is probably good in more focused cases but seems awkward when trying to make something to mix into any old test.)
import akka.actor.{Props, Actor, ActorSystem}
import akka.event.Logging.Error
import akka.testkit._
import com.typesafe.config.Config
import org.scalatest._
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.mock.EasyMockSugar
import scala.collection.mutable
trait AkkaErrorChecking extends ShouldMatchers {
val system:ActorSystem
val errors:mutable.MutableList[Error] = new mutable.MutableList[Error]
val errorCaptureFilter = EventFilter.custom {
case e: Error =>
errors += e
false // don't actually filter out this event - it's nice to see the full output in console.
}
lazy val testListener = system.actorOf(Props(new akka.testkit.TestEventListener {
addFilter(errorCaptureFilter)
}))
def withErrorChecking[T](block: => T) = {
try {
system.eventStream.subscribe(testListener, classOf[Error])
filterEvents(errorCaptureFilter)(block)(system)
withClue(errors.mkString("Akka error(s):\n", "\n", ""))(errors should be('empty))
} finally {
system.eventStream.unsubscribe(testListener)
errors.clear()
}
}
}
You can just use withErrorChecking inline at specific spots, or mix it into a Suite and use withFixture to do it globally across all tests, like this:
trait AkkaErrorCheckingSuite extends AkkaErrorChecking with FunSpec {
override protected def withFixture(test: NoArgTest) {
withErrorChecking(test())
}
}
If you use this in my original example, then you will get the first test "calls foo only when it receives a message" to fail, which is nice because that's where the real failure is. But the downstream test will still fail as well due to the system blowing up. To fix that, I went a step further and used a fixture.Suite to instance a separate TestKit for each test. That solves lots of other potential test isolation issues when you have noisy actors. It requires a little more ceremony declaring each test but I think it's well worth it. Using this trait with my original example I get the first test failing and the second one passing which is just what I want!
trait IsolatedTestKit extends ShouldMatchers { this: fixture.Suite =>
type FixtureParam = TestKit
// override this if you want to pass a Config to the actor system instead of using default reference configuration
val actorSystemConfig: Option[Config] = None
private val systemNameRegex = "[^a-zA-Z0-9]".r
override protected def withFixture(test: OneArgTest) {
val fixtureSystem = actorSystemConfig.map(config => ActorSystem(systemNameRegex.replaceAllIn(test.name, "-"), config))
.getOrElse (ActorSystem (systemNameRegex.replaceAllIn(test.name, "-")))
try {
val errorCheck = new AkkaErrorChecking {
val system = fixtureSystem
}
errorCheck.withErrorChecking {
test(new TestKit(fixtureSystem))
}
}
finally {
fixtureSystem.shutdown()
}
}
}

How can we override the react construct of scala actors?

The usual way of detecting the type of message received in scala actors is by
loop{
react{
case x: String =>
}
}
However, I want to know how can we override this implementation of react construct so that we can do implicit logging of the message that is received.
I am trying to implement a use case mentioned below ->
1. Before a message is matched to any case class, I want to write a log statement on console / file showing the occurance of the message.
2. We can log these messages explicitly by println() / log4j logging . However, I want to design a generic logger for scala actors which will log all the messages sent or received.
Any help in this regard will be helpful.
Thanks in advance
First, be aware that the Scala actors library is being deprecated in favor of Akka. So this answer won't be helpful very long (though the other actors library will continue to be available for a while--and since it's open source potentially forever if people want to maintain it).
Anyway, the react method is defined in scala.actors.Actor. Simply fail to import it, or hide it with your own. Your own what?
Well, the method just takes a PartialFunction[Any,Unit]. So, your should also:
def react(pf: PartialFunction[Any,Unit]): Nothing = { /*how?;*/ Actor.react(/*what?*/) }
You really only have access to the partial function, and you have to defer to Actor.react to do what you want. So you need to wrap pf in another PartialFunction that performs your logging. So you can
val qf = new PartialFunction[Any,Unit] {
def isDefinedAt(a: Any) = pf.isDefinedAt(a)
def apply(a: Any): Unit = {
log(a) // Maybe add more logic to know what a is
pf(a)
}
}
If you want to see messages that come in and get examined but are not actually consumed, you could do more with isDefinedAt also.
So, obviously enough, I hope, /*how?*/ is the above to define (create) qf, and /*what?*/ is just qf.
If you want to know whether a is a case class, the answer is that you cannot (by design). A case class is just syntactic sugar on top of ordinary Scala features; it's just there to save you typing. See, for instance, this question.
However, you can get pretty close by pattern matching for Product and checking whether it has a copy method:
case class M(i: Int)
val a: Any = M(5)
scala> a match {
case p: Product if p.getClass.getMethods.exists(_.getName=="copy") => println("Yes")
case _ => println("No")
}
Yes
If you really want to get fancy, check if copy has the same number and type of parameters as the constructor.
//Following is a code for a logReact Method that does the same thing as react but also logs the message received hope this works for you
import scala.actors.Actor;
import scala.actors.Actor._
trait ActorLogging extends Actor {
def logReact(handler: PartialFunction[Any, Unit]): Nothing = {
val handler2: PartialFunction[Any, Unit] = {
case x =>
println("Inside Logs -- with message recieved -- " + x.toString);
handler.apply(x);
}
super.react(handler2)
}
}
class sumAct extends Actor with ActorLogging {
def act() {
loop {
logReact {
case a: Int =>
println("Inside actor Sum Act Received the message -- " + a)
exit;
}
}
}
}
object ActorLog {
def main(args: Array[String]): Unit = {
var s: sumAct = new sumAct;
s.start();
s ! 1.toInt;
}
}