Akka Actor Test: Automatic reply with a TestProbe - scala

I am trying to get a test probe to reply with an acknowledgement, whenever it receive any message .
I wrote the following code in my test but it does not work:
val chgtWriter = new TestProbe(system) {
def receive: Receive = {
case m => println("receive messagereplying with ACK"); sender() ! ACK
}
}
Is there a way to do that. The actor that is actually sending the message to the test probe is definitely running on another thread than the TestThread. Below you can see the full test as currently crafted.
feature("The changeSetActor periodically fetch new change set following a schedule") {
scenario("A ChangeSetActor fetch new changeset from a Fetcher Actor that return a full and an empty ChangeSet"){
Given("a ChangeSetActor with a schedule of fetching a message every 10 seconds, a ChangeFetcher and a ChangeWriter")
val chgtFetcher = TestProbe()
val chgtWriter = new TestProbe(system) {
def receive: Receive = {
case m => println("receive message {} replying with ACK"); sender() ! ACK
}
}
val fromTime = Instant.now().truncatedTo(ChronoUnit.SECONDS)
val chgtActor = system.actorOf(ChangeSetActor.props(chgtWriter.ref, chgtFetcher.ref, fromTime))
When("all are started")
Then("The Change Fetcher should receive at least 3 messages from the ChangeSetActor within 40 seconds")
var changesetSNum = 1
val received = chgtFetcher.receiveWhile( 40 seconds) {
case FetchNewChangeSet(m) => {
println(s"received: FetchNewChangeSet(${m}")
if (changesetSNum == 1) {
chgtFetcher.reply(NewChangeSet(changeSet1))
changesetSNum += 1
}
else
chgtFetcher.reply(NoAvailableChangeSet)
}
}
received.size should be (3)
}
}
The changeSetActor is fully tested and works. The test hang with the ChangeWriter. It never receive a message in the receive method.
EDIT1(Following #Jakko anser)
The Auto Pilots is as follow:
val probe = TestProbe()
probe.setAutoPilot(new TestActor.AutoPilot {
def run(sender: ActorRef, msg: Any): TestActor.AutoPilot =
msg match {
case "stop" ⇒ TestActor.NoAutoPilot
case x ⇒ **testActor.tell(x, sender)**; TestActor.KeepRunning
}
})
Although all the explanation given so far is way clear, what is confusing in the official example is the reference "testActor". Who is testActor here ? there is no variable declaration of that name at that point.

You can script your test probes using Auto Pilots. For example:
import akka.testkit._
val probe = TestProbe()
probe.setAutoPilot(new TestActor.AutoPilot {
def run(sender: ActorRef, msg: Any): TestActor.AutoPilot = {
println("receive messagereplying with ACK")
sender ! ACK
TestActor.KeepRunning
}
})
In the above example, we set up a test probe with an automatic message handler, Auto Pilot. The auto pilot will be automatically triggered when the probe receives a message. In this example, the auto pilot will print a message and respond back to the sender.
After a message has been handled, the auto pilot can decide how the next incoming message will be handled. It can either set up a different auto pilot, reuse the existing auto pilot (TestActor.KeepRunning), or disable the auto pilot completely (TestActor.NoAutoPilot). In this example, the same auto pilot will be used for handling all the incoming messages.
You can still use test probe assertions as usual even with the auto pilot attached to the probe.
The testActor in the official documentation refers to the actor you are writing tests against. For example, in your case the actor could be the ChangeSetActor assigned to value chgtActor. Since all you really want to do is to respond back to the sender from probe, it is enough for the test probe auto pilot to respond back to the sender and not care about the testActor.

Related

Akka test - wait for actor initialization

case class FeatureFilter(s3Client: AmazonS3) extends Actor with ActorLogging {
override def preStart(): Unit = {
self ! Initialize
}
override def receive: Receive = {
case Initialize =>
// long running operaton
val tryfile = S3Connection(s3Client).downloadObject(...)
tryfile match {
case Success(file) =>
context.become(active(file))
case Failure(exception) =>
self ! PoisonPill
}
}
def active(file: File): Receive = {
case Query(key) =>
// do some processing and reply to sender
}
}
I am using below test for above actor:
"an actor" should {
// mocked S3 client
val client = ...
"test for presence of keys" in {
val actor = system.actorOf(Props(FeatureFilter(client)))
for (i <- 1 to 100) {
actor ! Query("test_string")
expectMsg(SomeMessage)
}
}
}
The above test fails with
java.lang.AssertionError: assertion failed: timeout (3 seconds) during expectMsg while waiting ...
I think this is because when the message actor ! Query("test_string") is sent to actor, it's handler is still receive, and so it doesn't respond, and hence the timeout.
But I even tried adding the handler for Query(key) in the receive method (just like in active method). Still I am getting the same error.
Could someone please point what is the issue here ?
Also when I move the S3 download task to preStart(), still the issue remains same. Isn't preStart() a blocking call ? How would the code in the test proceed until the preStart() is completed ?
akka stash sounds like the way you looking for. In case of any message that the actor support but is unhandled add on stash and unapply all if active is reached.
look at actor stash for documentation and example usage
may your code would look like
case msg => stash()
...
unstashAll()
context.become(active(file))

Send message to actor after restart from Supervisor

I am using BackoffSupervisor strategy to create a child actor that has to process some message. I want to implement a very simple restart strategy, in which in case of exception:
Child propagates failing message to supervisor
Supervisor restarts child and sends the failing message again.
Supervisor gives up after 3 retries
Akka persistence is not an option
So far what I have is this:
Supervisor definition:
val childProps = Props(new SenderActor())
val supervisor = BackoffSupervisor.props(
Backoff.onFailure(
childProps,
childName = cmd.hashCode.toString,
minBackoff = 1.seconds,
maxBackoff = 2.seconds,
randomFactor = 0.2
)
.withSupervisorStrategy(
OneForOneStrategy(maxNrOfRetries = 3, loggingEnabled = true) {
case msg: MessageException => {
println("caught specific message!")
SupervisorStrategy.Restart
}
case _: Exception => SupervisorStrategy.Restart
case _ ⇒ SupervisorStrategy.Escalate
})
)
val sup = context.actorOf(supervisor)
sup ! cmd
Child actor that is supposed to send the e-mail, but fails (throws some Exception) and propagates Exception back to supervisor:
class SenderActor() extends Actor {
def fakeSendMail():Unit = {
Thread.sleep(1000)
throw new Exception("surprising exception")
}
override def receive: Receive = {
case cmd: NewMail =>
println("new mail received routee")
try {
fakeSendMail()
} catch {
case t => throw MessageException(cmd, t)
}
}
}
In the above code I wrap any exception into custom class MessageException that gets propagated to SupervisorStrategy, but how to propagate it further to the new child to force reprocessing? Is this the right approach?
Edit. I attempted to resent the message to the Actor on preRestart hook, but somehow the hook is not being triggered:
class SenderActor() extends Actor {
def fakeSendMail():Unit = {
Thread.sleep(1000)
// println("mail sent!")
throw new Exception("surprising exception")
}
override def preStart(): Unit = {
println("child starting")
}
override def preRestart(reason: Throwable, message: Option[Any]): Unit = {
reason match {
case m: MessageException => {
println("aaaaa")
message.foreach(self ! _)
}
case _ => println("bbbb")
}
}
override def postStop(): Unit = {
println("child stopping")
}
override def receive: Receive = {
case cmd: NewMail =>
println("new mail received routee")
try {
fakeSendMail()
} catch {
case t => throw MessageException(cmd, t)
}
}
}
This gives me something similar to following output:
new mail received routee
caught specific message!
child stopping
[ERROR] [01/26/2018 10:15:35.690]
[example-akka.actor.default-dispatcher-2]
[akka://example/user/persistentActor-4-scala/$a/1962829645] Could not
process message sample.persistence.MessageException:
Could not process message <stacktrace>
child starting
But no logs from preRestart hook
The reason that the child's preRestart hook is not invoked is because Backoff.onFailure uses BackoffOnRestartSupervisor underneath the covers, which replaces the default restart behavior with a stop-and-delayed-start behavior that is consistent with the backoff policy. In other words, when using Backoff.onFailure, when a child is restarted, the child's preRestart method is not called because the underlying supervisor actually stops the child, then starts it again later. (Using Backoff.onStop can trigger the child's preRestart hook, but that's tangential to the present discussion.)
The BackoffSupervisor API doesn't support the automatic resending of a message when the supervisor's child restarts: you have to implement this behavior yourself. An idea for retrying messages is to let the BackoffSupervisor's supervisor handle it. For example:
val supervisor = BackoffSupervisor.props(
Backoff.onFailure(
...
).withReplyWhileStopped(ChildIsStopped)
).withSupervisorStrategy(
OneForOneStrategy(maxNrOfRetries = 3, loggingEnabled = true) {
case msg: MessageException =>
println("caught specific message!")
self ! Error(msg.cmd) // replace cmd with whatever the property name is
SupervisorStrategy.Restart
case ...
})
)
val sup = context.actorOf(supervisor)
def receive = {
case cmd: NewMail =>
sup ! cmd
case Error(cmd) =>
timers.startSingleTimer(cmd.id, Replay(cmd), 10.seconds)
// We assume that NewMail has an id field. Also, adjust the time as needed.
case Replay(cmd) =>
sup ! cmd
case ChildIsStopped =>
println("child is stopped")
}
In the above code, the NewMail message embedded in the MessageException is wrapped in a custom case class (in order to easily distinguish it from a "normal"/new NewMail message) and sent to self. In this context, self is the actor that created the BackoffSupervisor. This enclosing actor then uses a single timer to replay the original message at some point. This point in time should be far enough in the future such that the BackoffSupervisor can potentially exhaust SenderActor's restart attempts, so that the child can have ample opportunity to get in a "good" state before it receives the resent message. Obviously this example involves only one message resend regardless of the number of child restarts.
Another idea is to create a BackoffSupervisor-SenderActor pair for every NewMail message, and have the SenderActor send the NewMail message to itself in the preStart hook. One concern with this approach is the cleaning up of resources; i.e., shutting down the BackoffSupervisors (which will, in turn, shut down their respective SenderActor children) when the processing is successful or when the child restarts are exhausted. A map of NewMail ids to (ActorRef, Int) tuples (in which the ActorRef is a reference to a BackoffSupervisor actor, and the Int is the number of restart attempts) would be helpful in this case:
class Overlord extends Actor {
var state = Map[Long, (ActorRef, Int)]() // assuming the mail id is a Long
def receive = {
case cmd: NewMail =>
val childProps = Props(new SenderActor(cmd, self))
val supervisor = BackoffSupervisor.props(
Backoff.onFailure(
...
).withSupervisorStrategy(
OneForOneStrategy(maxNrOfRetries = 3, loggingEnabled = true) {
case msg: MessageException =>
println("caught specific message!")
self ! Error(msg.cmd)
SupervisorStrategy.Restart
case ...
})
)
val sup = context.actorOf(supervisor)
state += (cmd.id -> (sup, 0))
case ProcessingDone(cmdId) =>
state.get(cmdId) match {
case Some((backoffSup, _)) =>
context.stop(backoffSup)
state -= cmdId
case None =>
println(s"${cmdId} not found")
}
case Error(cmd) =>
val cmdId = cmd.id
state.get(cmdId) match {
case Some((backoffSup, numRetries)) =>
if (numRetries == 3) {
println(s"${cmdId} has already been retried 3 times. Giving up.")
context.stop(backoffSup)
state -= cmdId
} else
state += (cmdId -> (backoffSup, numRetries + 1))
case None =>
println(s"${cmdId} not found")
}
case ...
}
}
Note that SenderActor in the above example takes a NewMail and an ActorRef as constructor arguments. The latter argument allows the SenderActor to send a custom ProcessingDone message to the enclosing actor:
class SenderActor(cmd: NewMail, target: ActorRef) extends Actor {
override def preStart(): Unit = {
println(s"child starting, sending ${cmd} to self")
self ! cmd
}
def fakeSendMail(): Unit = ...
def receive = {
case cmd: NewMail => ...
}
}
Obviously the SenderActor is set up to fail every time with the current implementation of fakeSendMail. I'll leave the additional changes needed in SenderActor to implement the happy path, in which SenderActor sends a ProcessingDone message to target, to you.
In the good solution that #chunjef provides, he alert about the risk of schedule a job resend before the backoff supervisor has started the worker
This enclosing actor then uses a single timer to replay the original message at some point. This point in time should be far enough in the future such that the BackoffSupervisor can potentially exhaust SenderActor's restart attempts, so that the child can have ample opportunity to get in a "good" state before it receives the resent message.
If this happens, the scenario will be jobs going to dead letters and no further progress will be done.
I've made a simplified fiddle with this scenario.
So, the schedule delay should be larger than the maxBackoff, and this could represent an impact in job completion time.
A possible solution to avoid this scenario is making the worker actor to send a message to his father when is ready to work, like here.
The failed child actor is available as the sender in your supervisor strategy. Quoting https://doc.akka.io/docs/akka/current/fault-tolerance.html#creating-a-supervisor-strategy:
If the strategy is declared inside the supervising actor (as opposed
to within a companion object) its decider has access to all internal
state of the actor in a thread-safe fashion, including obtaining a
reference to the currently failed child (available as the sender of
the failure message).
Sending emails is a dangerous operation with some third party software in your case. Why not to apply Circuit Breaker pattern and skip the sender actor entirely? Also, you can still have an actor (with some Backoff Supervisor) and Circuit Breaker inside it (if that makes sense for you).

Akka Router increment counter on message arrival from routees

I'm trying to keep counting on each successful import. But here is a problem - Counter works if the router receives a message from its parent but if I'm trying to send a message from its children it receives it but doesn't update the global variable that is out of the scope.
I know it sounds complicated. Let me show you the code.
Here is the router
class Watcher(size: Int) extends Actor {
var router = {
val routees = Vector.fill(size) {
val w = context.actorOf(
Props[Worker]
)
context.watch(w)
ActorRefRoutee(w)
}
Router(RoundRobinRoutingLogic(), routees)
}
var sent = 0
override def supervisorStrategy(): SupervisorStrategy = OneForOneStrategy(maxNrOfRetries = 100) {
case _: DocumentNotFoundException => {
Resume
}
case _: Exception => Escalate
}
override def receive: Receive = {
case container: MessageContainer =>
router.route(container, sender)
case Success =>
sent += 1
case GetValue =>
sender ! sent
case Terminated(a) =>
router.removeRoutee(a)
val w = context.actorOf(Props[Worker])
context.watch(w)
router = router.addRoutee(w)
case undef =>
println(s"${this.getClass} received undefinable message: $undef")
}
}
Here is the worker
class Worker() extends Actor with ActorLogging {
var messages = Seq[MessageContainer]()
var received = 0
override def receive: Receive = {
case container: MessageContainer =>
try {
importMessage(container.message, container.repo)
context.parent ! Success
} catch {
case e: Exception =>
throw e
}
case e: Error =>
log.info(s"Error occurred $e")
sender ! e
case undef => println(s"${this.getClass} received undefinable message: $undef")
}
}
So on supervisor ? GetValue I get 0 but suppose to have 1000.The strangest thing is that when I debug it with the breakpoint right on the case Success => ... the value is incremented every time the new message arrives. But supervisor ? GetValue still returns 0.
Let's assume I want to count on case container: MessageContainer => ... and it will magically work; I'll get desirable number, but it doesn't show if I actually imported anything. What's going on?
Here is the test case.
#Test
def testRouter(): Unit = {
val system = ActorSystem("RouterTestSystem")
// val serv = AddressFromURIString("akka.tcp://master#host:1334")
val supervisor = system.actorOf(Props(new Watcher(20)))//.withDeploy(akka.actor.Deploy(scope = RemoteScope(serv))))
val repo = coreSession.getRepositoryName
val containers = (0 until num)
.map(_ => MessageContainer(MessageFactory.generate("/"), repo))
val watch = Stopwatch.createStarted()
(0 until num).par
.foreach( i => {
supervisor ! containers.apply(i)
})
implicit val timeout = Timeout(60 seconds)
val future = supervisor ? GetValue
val result = Await.result(future, timeout.duration).asInstanceOf[Int]
val speed = result / (watch.elapsed(TimeUnit.MILLISECONDS) / 1000.0)
println(f"Import speed: $speed%.2f")
assertEquals(num, result)
}
Can you please explained it in details. Why is it happening? Why only on message received from the children? Another approach?
Well... there can be many potential problems hidden in the parts of code that you have not shared. But, for the sake of this discussion I will assume that everything else is fine and we will just discuss problems with your shared code.
Now, let me explain a bit about Actors. To put things simply, every actor has a mailbox (where it keeps messages in the sequence they were received) and processes them one by one in the order they were received. Since the mailbox is used like a Queue we will refer to it as a Queue in this discussion.
Also... I don't know what this container.apply(i) is going to return... so I will refer to the return value of that container.apply(1) as MessageContainer__1
In your test runner you are first creating an instance of Watcher,
val supervisor = system.actorOf(Props(new Watcher(20)))
Now, lets say that you are sending these 2 messages (num = 2) to supervisor,
So supervisor's mailbox will look something like,
Queue(MessageContainer__0, MessageContainer__1)
Then you send it another message GetValue so the mailbox will look like,
Queue(MessageContainer__0, MessageContainer__1, GetValue)
Now the actor will process the first message and pass it to the workers, the mail-box will look like,
Queue(MessageContainer__1, GetValue)
Now even if your worker is ultra-fast and instantaneous in sending the reply the mailbox will look like,
Queue(MessageContainer__1, GetValue, Success)
And now since your worker super-ultra-fast and instantaneously replies with a Success, the state after passing the second MessageContainer will look like,
Queue(GetValue, Success, Success)
And... here is the root of your problem. The Supervisor sees the GetValue massage before any Success messages, no matter how fast your workers are.
And thus it will process GetValue and reply with current value of sent which is 0.

Can actors read messages under a certain condition?

I have this situation:
ActorA sends ActorB start/stop messages every 30-40 seconds
ActorA sends ActorB strings to print (always)
ActorB must print the strings he receive, but only if ActorA sent just a start message
Now i wonder if i can do the following things:
Can ActorB read messages only under a certain condition (if a boolean is set as true) without losing the messages he receives while that boolean is set as false?
Can ActorB read a start/stop message from ActorA before the other string messages? I'd like to have this situation: ActorA sends a start message to ActorB, ActorB start printing the strings he received before the start messages (and that is still receiving) and then stop as soon as it receives a stop messages?
I don't know if I explained it well.
EDIT: Thank you, the answers are great, but I still have some doubts.
Does the become mantain the order of the messages? I mean, if i have "Start-M1-M2-Stop-M3-M4-M5-Start-M6-M7-Stop", will the printing order be "M1-M2" and then "M3-M4-M5-M6-M7" or could M6 be read before M3, M4 and M5 (if M6 is received just after the become)?
Can I give a higher priority to start/stop messages? If ActorB receives "M1-M2-M3", and then it receives a stop message while it is printing "M1", i want that ActorB saves again M2 and M3.
You can exactly solve your problem with the Stash trait and the become/unbecome functionality of Akka. The idea is the following:
When you receive a Stop message then you switch to a behaviour where you stash all messages which are not Start. When you receive a Start message, then you switch to a behaviour where you print all received messages and additionally you unstash all messages which have arrived in the meantime.
case object Start
case object Stop
case object TriggerStateChange
case object SendMessage
class ActorB extends Actor with Stash {
override def receive: Receive = {
case Start =>
context.become(printingBehavior, false)
unstashAll()
case x => stash()
}
def printingBehavior: Receive = {
case msg: String => println(msg)
case Stop => context.unbecome()
}
}
class ActorA(val actorB: ActorRef) extends Actor {
var counter = 0
var started = false
override def preStart: Unit = {
import context.dispatcher
this.context.system.scheduler.schedule(0 seconds, 5 seconds, self, TriggerStateChange)
this.context.system.scheduler.schedule(0 seconds, 1 seconds, self, SendMessage)
}
override def receive: Actor.Receive = {
case SendMessage =>
actorB ! "Message: " + counter
counter += 1
case TriggerStateChange =>
actorB ! (if (started) {
started = false
Stop
} else {
started = true
Start
})
}
}
object Akka {
def main(args: Array[String]) = {
val system = ActorSystem.create("TestActorSystem")
val actorB = system.actorOf(Props(classOf[ActorB]), "ActorB")
val actorA = system.actorOf(Props(classOf[ActorA], actorB), "ActorA")
system.awaitTermination()
}
}
Check the Become/Unbecome functionality. It lets you change the behavior of the actor.
If I understood correctly you want your ActorB to have two different states. In the first state it should cache the messages it receives. In the second state, it should print all the cached messages and start printing all the new ones.
Something like this:
case class Start
case class Stop
case class Message(val: String)
class ActorB extends Actor {
var cache = Queue()
def initial: Receive = {
case Message(msg) => cache += msg
case Start =>
for (m <- cache) self ! Message(m)
context.become(printing)
}
def printing: Receive = {
case Message(msg) => println(msg)
case Stop => context.become(initial) //or stop the actor
}
def receive = initial
}
Have Actor B alternate between two states (two different behaviours). In the initial 'pending' state, it waits for a 'start' message, while stashing any other messages.
On receipt of a 'start' message, unstash all the stored messages and become 'active', waiting on a 'stop' message and writing out the other messages received (which will include the unstashed ones). On receiveing a 'stop' message, revert to the 'pending' behaviour.
Some of my thoughts
Yes if the boolean flag is got from some system resource like db or a config file, but I don't think it should be dependent on any external message, given that the actor receive messages from multiple other actors. If ActorB is only used by ActorA, the two can be merged to one
Similar as 1, how to handle the messages from multiple actors? If there is only one actorA, the two actors can be merged. If there are multiple, the flag can be set in database, actorA change the flag in db to "Start" or "Stop". and Actor B will print or not based on the flag.
An actor should be doing something very independently on other actor's state. The start and stop is actually some state of ActorA instead of ActorB
You already have a lot of good answers, but somehow I feel compelled to offer something more brief, as what you need is not necessarily a state machine:
class ActorB extends Actor {
def receive: Receive = caching(Nil)
def caching(cached: List[String]): Receive = {
case msg: String =>
context.become(caching(msg :: cached))
case Start =>
cached.reverse.foreach(println)
context.become(caching(Nil))
}
}

Akka + WithinTimeRange

I've testing the fault tolerant system of akka and so far it's been good when talking about retrying to send a msg according the maxNrOfRetries specified.
However, it does not restart the actor within the given time range, it restarts all at once, ignoring the within time range.
I tried with AllForOneStrategy and OneForOneStrategy but does not change anything.
Trying to follow this blog post: http://letitcrash.com/post/23532935686/watch-the-routees, this is the code I've been working.
class Supervisor extends Actor with ActorLogging {
var replyTo: ActorRef = _
val child = context.actorOf(
Props(new Child)
.withRouter(
RoundRobinPool(
nrOfInstances = 5,
supervisorStrategy =
AllForOneStrategy(maxNrOfRetries = 3, withinTimeRange = 10.second) {
case _: NullPointerException => Restart
case _: Exception => Escalate
})), name = "child-router")
child ! GetRoutees
def receive = {
case RouterRoutees(routees) =>
routees foreach context.watch
case "start" =>
replyTo = sender()
child ! "error"
case Terminated(actor) =>
replyTo ! -1
context.stop(self)
}
}
class Child extends Actor with ActorLogging {
override def preRestart(reason: Throwable, message: Option[Any]): Unit = {
log.info("***** RESTARTING *****")
message foreach{ self forward }
}
def receive = LoggingReceive {
case "error" =>
log.info("***** GOT ERROR *****")
throw new NullPointerException
}
}
object Boot extends App {
val system = ActorSystem()
val supervisor = system.actorOf(Props[Supervisor], "supervisor")
supervisor ! "start"
}
Am I doing anything wrong to accomplish that?
EDIT
Actually, I misunderstood the purpose of the withinTimeRange.
To schedule my retries in a time range, I'm doing the following:
override def preRestart(reason: Throwable, message: Option[Any]): Unit = {
log.info("***** RESTARTING *****")
message foreach { msg =>
context.system.scheduler.scheduleOnce(30.seconds, self, msg)
}
}
It seems to work ok.
I think you have misunderstood the purpose of the withinTimeRange arg. That value is supposed to be used in conjunction with maxNrOfRetries to provide a window in which to support the limiting of the number of retries. For example, as you have specified, the implication is that the supervisor will no longer restart an individual child if that child needs to be restarted more than 3 times in 10 seconds.
From docs:
maxNrOfRetries - the number of times a child actor is allowed to be
restarted, negative value means no limit, if the limit is exceeded the
child actor is stopped
withinTimeRange - duration of the time window
for maxNrOfRetries, Duration.Inf means no window
Your code means that when any child fails with NullPointerException more than 3 times within 10 seconds it will not be restarted again. Because of AllForOneStrategy after first Routee fails all routees are restarted. And because you've overridden preRestart to resend failed message this situation repeats again until reaches 3 failures within 10 seconds(which is achieved in less than a second).