Akka Streams TCP socket client side termination - scala

I have the following flow:
val actorSource = Source.actorRef(10000, OverflowStrategy.dropHead)
val targetSink = Flow[ByteString]
.map(_.utf8String)
.via(new JsonStage())
.map { json =>
MqttMessages.jsonToObject(json)
}
.to(Sink.actorRef(self, "Done"))
sourceRef = Some(Flow[ByteString]
.via(conn.flow)
.to(targetSink)
.runWith(actorSource))
within an Actor (which is the Sink.actorRef one). The conn.flow is an incoming TCP Connection using Tcp().bind(address, port).
Currently the Sink.actorRef Actor keeps running when the tcp connection is closed from the client side. Is there a way to register the client side termination of the tcp connection to shutdown the Actor?
Edit:
I tried handling both cases as suggested:
case "Done" =>
context.stop(self)
case akka.actor.Status.Failure =>
context.stop(self)
But when I test with a socket client and cancel it, the actor is not being shutdown. So neither the "Done" message nor the Failure seem to be registered if the TCP connection is terminated.
Here is the whole code:
private var connection: Option[Tcp.IncomingConnection] = None
private var mqttpubsub: Option[ActorRef] = None
private var sourceRef: Option[ActorRef] = None
private val sdcTopic = "out"
private val actorSource = Source.actorRef(10000, OverflowStrategy.dropHead)
implicit private val system = context.system
implicit private val mat = ActorMaterializer.create(context.system)
override def receive: Receive = {
case conn: Tcp.IncomingConnection =>
connection = Some(conn)
mqttpubsub = Some(context.actorOf(Props(classOf[MqttPubSub], PSConfig(
brokerUrl = "tcp://127.0.0.1:1883", //all params is optional except brokerUrl
userName = null,
password = null,
//messages received when disconnected will be stash. Messages isOverdue after stashTimeToLive will be discard
stashTimeToLive = 1.minute,
stashCapacity = 100000, //stash messages will be drop first haft elems when reach this size
reconnectDelayMin = 10.millis, //for fine tuning re-connection logic
reconnectDelayMax = 30.seconds
))))
val targetSink = Flow[ByteString]
.alsoTo(Sink.foreach(println))
.map(_.utf8String)
.via(new JsonStage())
.map { json =>
MqttMessages.jsonToObject(json)
}
.to(Sink.actorRef(self, "Done"))
sourceRef = Some(Flow[ByteString]
.via(conn.flow)
.to(targetSink)
.runWith(actorSource))
case msg: MqttMessages.MqttMessage =>
processMessage(msg)
case msg: Message =>
val jsonMsg = JsonParser(msg.payload).asJsObject
val mqttMsg = MqttMessages.jsonToObject(jsonMsg)
try {
sourceRef.foreach(_ ! ByteString(msg.payload))
} catch {
case e: Throwable => e.printStackTrace()
}
case SubscribeAck(Subscribe(topic, self, qos), fail) =>
case "Done" =>
context.stop(self)
case akka.actor.Status.Failure =>
context.stop(self)
}

the Actor keeps running
Which actor do you mean, the one you've registered with Sink.actorRef? If yes, then to shut it down when the stream shuts down, you need to handle "Done" and akka.actor.Status.Failure messages in it and invoke context.stop(self) explicitly. "Done" message will be sent when the stream closes successfully, while Status.Failure will be sent if there is an error.
For more information see Sink.actorRef API docs, they explain the termination semantics.

I ended up creating another Stage, which only passes elements through but and emits an additional message to the next flow if the upstream closes:
class TcpStage extends GraphStage[FlowShape[ByteString, ByteString]] {
val in = Inlet[ByteString]("TCPStage.in")
val out = Outlet[ByteString]("TCPStage.out")
override val shape = FlowShape.of(in, out)
override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = new GraphStageLogic(shape) {
setHandler(out, new OutHandler {
override def onPull(): Unit = {
if (isClosed(in)) emitDone()
else pull(in)
}
})
setHandler(in, new InHandler {
override def onPush(): Unit = {
push(out, grab(in))
}
override def onUpstreamFinish(): Unit = {
emitDone()
completeStage()
}
})
private def emitDone(): Unit = {
push(out, ByteString("{ }".getBytes("utf-8")))
}
}
}
Which I then use in my flow:
val targetSink = Flow[ByteString]
.via(new TcpStage())
.map(_.utf8String)
.via(new JsonStage())
.map { json =>
MqttMessages.jsonToObject(json)
}
.to(Sink.actorRef(self, MqttDone))
sourceRef = Some(Flow[ByteString]
.via(conn.flow)
.to(targetSink)
.runWith(actorSource))

Related

How to add keepAlive event to SSE channel

I have a ListenerActor which is listening to messages from backend and pushing the messages through channel as SSE Events.
I want to keep my actor alive so that i can stream continuously. How do i add keepAlive to my actor.
P.S: I am not using Akka stream or Akka http.
def filter(inboxId:String): Enumeratee[SSEPublisher.ListenerEnvelope, SSEPublisher.ListenerEnvelope] = Enumeratee.filter[SSEPublisher.ListenerEnvelope] { envelope: SSEPublisher.ListenerEnvelope => envelope.inboxId == inboxId }
def convert: Enumeratee[SSEPublisher.ListenerEnvelope, String] = Enumeratee.map[SSEPublisher.ListenerEnvelope] {
envelope =>
Json.toJson(envelope).toString()
}
def connDeathWatch(addr: String): Enumeratee[SSEPublisher.ListenerEnvelope, SSEPublisher.ListenerEnvelope] =
Enumeratee.onIterateeDone { () => println(addr + " - SSE disconnected")
}
implicit def pair[E]: EventNameExtractor[E] = EventNameExtractor[E] { p =>
val parsedJson = scala.util.parsing.json.JSON.parseFull(s"$p").get
val topic = parsedJson.asInstanceOf[Map[String, String]].apply("topic")
Some(topic)
}
implicit def id[E]: EventIdExtractor[E] = EventIdExtractor[E](p => Some(UUID.randomUUID().toString))
def events(inboxId: String) = InboxResource(inboxId)(AuthScope.Basic)(authUser => Action { implicit request =>
Ok.feed(content = ncf.sseEnumerator
&> filter(inboxId)
&> convert
&> EventSource()
).as("text/event-stream")
})
override def receive: Receive = {
case Tick =>
log.info(s"sending re-register tick to event-publisher")
Topics.all.foreach { a: Topic =>
log.info(s"$a")
clusterClient ! ClusterClient.SendToAll(publisherPath, SSEPublisher.AddListener(a, self))
}
case ListenerEnvelope(topic, inboxId, itemId, sourceId, message) =>
log.info(s"Received message from event publisher for topic $topic, for inbox $inboxId, msg : $message")
channel.push(SSEPublisher.ListenerEnvelope(topic, inboxId, itemId, sourceId, message))
}
You can create a keepAlive protocol at the actor level and use the scheduler to send the keepAlive message to the actor.
def convert(t: SomeType): Enumeratee[SSEPublisher.ListenerEnvelope, String] =
// pattern match on type t
}

Akka Stream - Pausable GraphStage (Akka 2.5.7)

I would like to write a GraphStage which can be paused/unpaused by sending a message from another actor.
The code snipped below shows a simple GraphStage which generates random numbers. When the stage gets materialized the GraphStageLogic sends a message (within preStart()) containing the StageActor to a supervisor. The supervisor keeps the stage's ActorRef and can therefore be used to control the stage.
object RandomNumberSource {
case object Pause
case object UnPause
}
class RandomNumberSource(supervisor: ActorRef) extends GraphStage[SourceShape[Int]] {
val out: Outlet[Int] = Outlet("rnd.out")
override val shape: SourceShape[Int] = SourceShape(out)
override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = {
new RandomNumberSourceLogic(shape)
}
private class RandomNumberSourceLogic(shape: Shape) extends GraphStageLogic(shape) with StageLogging {
lazy val self: StageActor = getStageActor(onMessage)
val numberGenerator: Random = Random
var isPaused: Boolean = true
override def preStart(): Unit = {
supervisor ! AssignStageActor(self.ref)
}
setHandler(out, new OutHandler {
override def onPull(): Unit = {
if (!isPaused) {
push(out, numberGenerator.nextInt())
Thread.sleep(1000)
}
}
})
private def onMessage(x: (ActorRef, Any)): Unit =
{
x._2 match {
case Pause =>
isPaused = true
log.info("Stream paused")
case UnPause =>
isPaused = false
getHandler(out).onPull()
log.info("Stream unpaused!")
case _ =>
}
}
}
}
This is a very simple implementation of the supervisor actor:
object Supervisor {
case class AssignStageActor(ref: ActorRef)
}
class Supervisor extends Actor with ActorLogging {
var stageActor: Option[ActorRef] = None
override def receive: Receive = {
case AssignStageActor(ref) =>
log.info("Stage assigned!")
stageActor = Some(ref)
ref ! Done
case Pause =>
log.info("Pause stream!")
stageActor match {
case Some(ref) => ref ! Pause
case _ =>
}
case UnPause =>
log.info("UnPause stream!")
stageActor match {
case Some(ref) => ref ! UnPause
case _ =>
}
}
}
I'm using the following application to run the stream:
object Application extends App {
implicit val system = ActorSystem("my-actor-system")
implicit val materializer = ActorMaterializer()
val supervisor = system.actorOf(Props[Supervisor], "supervisor")
val sourceGraph: Graph[SourceShape[Int], NotUsed] = new RandomNumberSource(supervisor)
val randomNumberSource: Source[Int, NotUsed] = Source.fromGraph(sourceGraph)
randomNumberSource.take(100).runForeach(println)
println("Start stream by pressing any key")
StdIn.readLine()
supervisor ! UnPause
StdIn.readLine()
supervisor ! Pause
StdIn.readLine()
println("=== Terminating ===")
system.terminate()
}
When the application starts the stage ia in 'paused' state and does not produce any number. When i press a key my stage starts to emit numbers. But my problem is that all messages sent to the stage after it has been started are ignored. I can not pause the stage.
I'm interested in changing the behavior of a stage based on a message received from an actor, but all examples i found pass an actor's message into the stream.
Does somebody has a guess why my code does not work or has an idea how to build such a GraphStage?
Thank you very much!
The Akka Stream Contrib project has a Valve stage that materializes a value that can pause and resume a flow. From the Scaladoc for this class:
Materializes into a Future of ValveSwitch which provides a the method flip that stops or restarts the flow of elements passing through the stage. As long as the valve is closed it will backpressure.
For example:
val (switchFut, seqSink) = Source(1 to 10)
.viaMat(new Valve(SwitchMode.Close))(Keep.right)
.toMat(Sink.seq)(Keep.both)
.run()
switchFut is a Future[ValveSwitch], and since the switch is closed initially, the valve backpressures and nothing is emitted downstream. To open the valve:
switchFut.onComplete {
case Success(switch) =>
switch.flip(SwitchMode.Open) // Future[Boolean]
case _ =>
log.error("the valve failed")
}
More examples are in ValveSpec.

Apache Spark Receiver Scheduling

I have implemented a receiver that is supposed to connect to a WebSocket stream and get the messages for processing. Here is the implementation that I have done so far:
class WebSocketReader (wsConfig: WebSocketConfig, stringMessageHandler: String => Option[String],
storageLevel: StorageLevel) extends Receiver[String] (storageLevel) {
// TODO: avoid using a var
private var wsClient: WebSocketClient = _
def sendRequest(isRequest: Boolean, msgCount: Int) = {
while (isRequest) {
wsClient.send(msgCount.toString)
Thread.sleep(1000)
}
}
// TODO: avoid using Synchronization...
private def connect(): Unit = {
Try {
wsClient = createWsClient
} match {
case Success(_) =>
wsClient.connect().map {
case result if result.isSuccess =>
sendRequest(true, 10)
case _ =>
connect()
}
case Failure(ex) =>
// TODO: how to signal a failure so that it is tried the next time....
ex.printStackTrace()
}
}
def onStart(): Unit = {
new Thread(getClass.getSimpleName) {
override def run() { connect() }
}.start()
}
override def onStop(): Unit =
if (wsClient != null) wsClient.disconnect()
private def createWsClient = {
new DefaultHookupClient(new HookupClientConfig(new URI(wsConfig.wsUrl))) {
override def receive: Receive = {
case Disconnected(_) =>
// TODO: use Logging framework, try reconnecting....
println(s"the web socket is disconnected")
case TextMessage(message) =>
stringMessageHandler(message).foreach(store)
case JsonMessage(jsValue) =>
stringMessageHandler(jsValue.toString).foreach(store)
}
}
}
}
How is this Receiver being run? Does this Receiver run on the worker nodes or on the driver node? Is this way of sleeping a thread a correct approach?
The reason why I want to do this is that the server that is exposing the WebSocket end point would need a count on the messages that I want to receive. Say if I ask the server for 100 messages, it would give me 100 messages and so on. So I need a way to periodically schedule this request to the server. Currently, I'm using the Thread.sleep mechanism. Is this advisable? What could be the alternative?

postRestart and preRestart methods are not getting invoke in akka actots

I am following this tutorial here is my code
case class ArtGroupDeleteFromES (uuidList:List[String])
class ArtGroupDeleteESActor extends Actor{
val log = LoggerFactory.getLogger(this.getClass)
override def preStart() {
log.debug("preStart Starting ArtGroupDeleteESActor instance hashcode # {}",
this.hashCode())
}
override def postStop() {
log.debug("postStop Stopping ArtGroupDeleteESActor instance hashcode # {}",
this.hashCode())
}
override def preRestart(reason: Throwable, message: Option[Any]) {
log.debug("I am restarting")
log.debug("ArtGroupDeleteESActor: preRestart")
log.debug(s" MESSAGE: ${message.getOrElse("")}")
log.debug(s" REASON: ${reason.getMessage}")
super.preRestart(reason, message)
}
override def postRestart(reason: Throwable) {
log.debug("restart completed!")
log.debug("ArtGroupDeleteESActor: postRestart")
log.debug(s" REASON: ${reason.getMessage}")
super.postRestart(reason)
}
def receive = {
case ArtGroupDeleteFromES(uuidList) =>
throw new Exception("Booom")
sender ! true
}
case message =>
log.warn("Received unknown message: {}", message)
unhandled(message)
}
}
and here is the how i am sending this actor a message
class ArtGroupDeletionActor extends Actor{
val log = LoggerFactory.getLogger(this.getClass)
override val supervisorStrategy = OneForOneStrategy(
maxNrOfRetries = 10, withinTimeRange = 10 seconds) {
case _:Exception => Restart
}
val artGroupDeleteESActor=context.actorOf(Props[ArtGroupDeleteESActor]
.withDispatcher("akka.actor.ArtGroupDeleteESActor-dispatcher")
,name = "ArtGroupDeleteESActor")
def receive = {
case DeleteArtGroup(uuidList) =>
val future1 = ask(artGroupDeleteESActor, ArtGroupDeleteFromES(uuidList)).mapTo[Boolean]
var isDeletedfromES = Await.result(future1, timeout.duration)
case message =>
log.warn("Unhandled message received : {}", message)
unhandled(message)
}
}
object test extends App{
val artGroupDeletionActor=system.actorOf(Props[ArtGroupDeletionActor]
.withDispatcher("akka.actor.ArtGroupDeletionActor-dispatcher")
,name = "ArtGroupDeletionActor")
artGroupDeletionActor ! DeleteArtGroup(List("123"))
}
the PostRestart() and preRestart() methods are not invoking,but preStart() and postStop() gets called, please guide me where i am doing wrong
(for simplicity I'll call your actors Parent and Child from now on)
What happens here is that when an exception occurs inside Child.receive, it doesn't send a response to Parent, instead, the actor system sends some control instruction for the supervision strategy. However, Parent is blocked on Await waiting for completion of future1, which only happens after the timeout exceeds, and then, in turn, a TimeoutException is thrown inside Parent.receive, killing (restarting) the Parent actor itself, and thus the supervising message of an exception in Child is then passed to deadLetters, never restarting the Child.
You should never, ever, ever block inside an actor, so this is incorrect:
val future1 = ask(artGroupDeleteESActor, ArtGroupDeleteFromES(uuidList)).mapTo[Boolean]
var isDeletedfromES = Await.result(future1, timeout.duration)
Instead, you have to either utilize some kind of message identification to distinguish one reply from another in concurrent environment, or add an onComplete to the Future and send a message to self in the closure (beware: no logic other than sending a message should be executed inside the closure to the Future!).
So, option A:
case class ArtGroupDeleteFromES(id: Long, uuidList: List[String])
case class ArtGroupDeleteFromESResult(id: Long, success: Boolean)
class Parent extends Actor {
override val supervisionStrategy = ...
var msgId = 0L
var pendingRequesters = Map.empty[Long, ActorRef]
val child = context.actorOf(Props[Child])
def nextId = {
msgId += 1
msgId
}
def receive = {
case DeleteArtGroup(uuidList) =>
val id = nextId
pendingRequesters += id -> sender() // store a reference to the sender so that you can send it a message when everything completes
child ! DeleteArtGroupFromES(nextId, uuidList)
case ArtGroupDeleteFromESResult(id, success) =>
// process result...
pendingRequesters(id) ! "done"
pendingRequesters -= id
}
}
And option B:
case class ArtGroupDeleteFromES(uuidList: List[String])
case class ArtGroupDeleteFromESResult(replyTo: ActorRef, success: Boolean)
class Parent extends Actor {
override val supervisionStrategy = ...
val child = context.actorOf(Props[Child])
def receive = {
case DeleteArtGroup(uuidList) =>
val requester = sender() // when the future completes, sender may have already changed, so you need to remember it
(child ? DeleteArtGroupFromES(uuidList)).onComplete {
case Success(success) => self ! ArtGroupDeleteFromESResult(requester, success)
case Failure(e) =>
log.warn("Could not delete...", e)
self ! ArtGroupDeleteFromESResult(requester, success = false)
}
}

spray.can.Http$ConnectionException: Premature connection close

In my below test, I tried to simulate a timeout and then send a normal request. however, I got spray.can.Http$ConnectionException: Premature connection close (the server doesn't appear to support request pipelining)
class SprayCanTest extends ModuleTestKit("/SprayCanTest.conf") with FlatSpecLike with Matchers {
import system.dispatcher
var app = Actor.noSender
protected override def beforeAll(): Unit = {
super.beforeAll()
app = system.actorOf(Props(new MockServer))
}
override protected def afterAll(): Unit = {
system.stop(app)
super.afterAll()
}
"response time out" should "work" in {
val setup = Http.HostConnectorSetup("localhost", 9101, false)
connect(setup).onComplete {
case Success(conn) => {
conn ! HttpRequest(HttpMethods.GET, "/timeout")
}
}
expectMsgPF() {
case Status.Failure(t) =>
t shouldBe a[RequestTimeoutException]
}
}
"normal http response" should "work" in {
//Thread.sleep(5000)
val setup = Http.HostConnectorSetup("localhost", 9101, false)
connect(setup).onComplete {
case Success(conn) => {
conn ! HttpRequest(HttpMethods.GET, "/hello")
}
}
expectMsgPF() {
case HttpResponse(status, entity, _, _) =>
status should be(StatusCodes.OK)
entity should be(HttpEntity("Helloworld"))
}
}
def connect(setup: HostConnectorSetup)(implicit system: ActorSystem) = {
// for the actor 'asks'
import system.dispatcher
implicit val timeout: Timeout = Timeout(1 second)
(IO(Http) ? setup) map {
case Http.HostConnectorInfo(connector, _) => connector
}
}
class MockServer extends Actor {
//implicit val timeout: Timeout = 1.second
implicit val system = context.system
// Register connection service
IO(Http) ! Http.Bind(self, interface = "localhost", port = 9101)
def receive: Actor.Receive = {
case _: Http.Connected => sender ! Http.Register(self)
case HttpRequest(GET, Uri.Path("/timeout"), _, _, _) => {
Thread.sleep(3000)
sender ! HttpResponse(entity = HttpEntity("ok"))
}
case HttpRequest(GET, Uri.Path("/hello"), _, _, _) => {
sender ! HttpResponse(entity = HttpEntity("Helloworld"))
}
}
}
}
and My config for test:
spray {
can {
client {
response-chunk-aggregation-limit = 0
connecting-timeout = 1s
request-timeout = 1s
}
host-connector {
max-retries = 0
}
}
}
I found that in both cases, the "conn" object is the same.
So I guess when RequestTimeoutException happens, spray put back the conn to the pool (by default 4?) and the next case will use the same conn but at this time, this conn is keep alive, so the server will treat it as chunked request.
If I put some sleep in the second case, it will just passed.
So I guess I must close the conn when got RequestTimeoutException and make sure the second case use a fresh new connection, right?
How should I do? Any configurations?
Thanks
Leon
You should not block inside an Actor (your MockServer). When it is blocked, it is unable to respond to any messages. You can wrap the Thread.sleep and response inside a Future. Or even better: use the Akka Scheduler. Be sure to assign the sender to a val because it may change when you respond to the request asynchronously. This should do the trick:
val savedSender = sender()
context.system.scheduler.scheduleOnce(3 seconds){
savedSender ! HttpResponse(entity = HttpEntity("ok"))
}