Scala & Play Websockets: Storing messages exchanged - scala

I started playing around scala and came to this particular boilerplate of web socket chatroom in scala.
They use MessageHub.source() and BroadcastHub.sink() as their Source and Sink for sending the messages to all connected clients.
The example is working fine for exchanging messages as it is.
private val (chatSink, chatSource) = {
// Don't log MergeHub$ProducerFailed as error if the client disconnects.
// recoverWithRetries -1 is essentially "recoverWith"
val source = MergeHub.source[WSMessage]
.log("source")
.recoverWithRetries(-1, { case _: Exception ⇒ Source.empty })
val sink = BroadcastHub.sink[WSMessage]
source.toMat(sink)(Keep.both).run()
}
private val userFlow: Flow[WSMessage, WSMessage, _] = {
Flow.fromSinkAndSource(chatSink, chatSource)
}
def chat(): WebSocket = {
WebSocket.acceptOrResult[WSMessage, WSMessage] {
case rh if sameOriginCheck(rh) =>
Future.successful(userFlow).map { flow =>
Right(flow)
}.recover {
case e: Exception =>
val msg = "Cannot create websocket"
logger.error(msg, e)
val result = InternalServerError(msg)
Left(result)
}
case rejected =>
logger.error(s"Request ${rejected} failed same origin check")
Future.successful {
Left(Forbidden("forbidden"))
}
}
}
I want to store the messages that are exchanged in the chatroom in a DB.
I tried adding map and fold functions to source and sink to get hold of the messages that are sent but I wasn't able to.
I tried adding a Flow stage between MergeHub and BroadcastHub like below
val flow = Flow[WSMessage].map(element => println(s"Message: $element"))
source.via(flow).toMat(sink)(Keep.both).run()
But it throws a compilation error that cannot reference toMat with such signature.
Can someone help or point me how can I get hold of messages that are sent and store them in DB.
Link for full template:
https://github.com/playframework/play-scala-chatroom-example

Let's look at your flow:
val flow = Flow[WSMessage].map(element => println(s"Message: $element"))
It takes elements of type WSMessage, and returns nothing (Unit). Here it is again with the correct type:
val flow: Flow[Unit] = Flow[WSMessage].map(element => println(s"Message: $element"))
This will clearly not work as the sink expects WSMessage and not Unit.
Here's how you can fix the above problem:
val flow = Flow[WSMessage].map { element =>
println(s"Message: $element")
element
}
Not that for persisting messages in the database, you will most likely want to use an async stage, roughly:
val flow = Flow[WSMessage].mapAsync(parallelism) { element =>
println(s"Message: $element")
// assuming DB.write() returns a Future[Unit]
DB.write(element).map(_ => element)
}

Related

Avoid syncronous calls in Akka Actors due of Await result

Because I do some "complex" operations, I think my Actor became asyncronous. The main problem I think is that I use Await.result inside the method which return responses.
actor:
def process(subscribers: Set[ActorRef]): Receive = {
case Join(ref) => context become process(subscribers + ref)
case Leave(ref) => context become process(subscribers - ref)
case Push(request) =>
val filteredSubscribers = (subscribers - sender())
.filter(s => exists(s, request)) // just some actor filters
filteredSubscribers.foreach { subscriber =>
// here I have a Map with each actor requests
val actorOptions = getActorOptions(subscriber)
subscriber ? getResponse(actorOptions, request)
}
}
The problem is inside getResponse (I think).
getResponse(actorOptions: JsValue, request: SocketRequest): JsValue = {
(actorOptions \ "dashboardId").asOpt[Int] match {
case Some(id) => {
val response = widgetsService.getByDashboadId(id) map { widgets =>
val widgetsResponse: List[Future[String]] = widgets.map(w => {
widgetsService.getDataById(w.id) map {
data => s"""{ "widgetId": ${w.id}, "data": $data }"""
}
})
var responses: List[String] = List.empty
widgetsResponse.foreach(f => {
f.onComplete {
case Success(value) => responses = value :: responses
case Failure(e) => println(s"Something happened: ${e.getMessage}")
}
})
// first time when I use Await.result
// used to populate the responses list with data from all futures
Await.result(Future.sequence(widgetsResponse), Duration.Inf)
Json.parse(s"""{
"dashboardId": $id,
"widgets": [${response.mkString(", ")}]
}""".stripMargin)
}
// second time when I use Await.result
// used to return a JsValue instead of a Future[JsValue]
Await.result(response, Duration.Inf)
}
case None => buildDefaultJson // return default json value, unimportant for this example
}
}
Due of that, In frontend, if I have 2 sockets clients, the response for the second will be send only after first.
I found that I can obtain a "fake" increase of performance if I embrance the getResponse in a future inside of my Actor.
filteredSubscribers.foreach { subscriber =>
val actorOptions = getActorOptions(subscriber)
Future(subscriber ? getResponse(actorOptions, request))
}
So, for both subscribers the action will be started in same time, but when the first will reach the Await.result, the second will be locked until first is done.
I need to avoid using Await.result there, but I don't know how to get the results of a list of futures, without using for-comprehension (because is a dynamically list) for first time where I use it.
Because Akka ask operator (?) return a Future[Any], I tried that my getResponse method to return directly a JsValue to be mapped then in Future[JsValue]. If I remove the second Await.result and my method will return Future[JsValue], then the actor will return a Future[Future[JsValue]] which I don't think is too right.
After some more researches and solutions found on so, my code become:
Future.sequence(widgetsResponse) map { responses =>
Json.parse(
s"""
|{
|"dashboardId": $id,
|"tableSourceId": $tableSourceId,
|"widgets": [ ${responses.mkString(", ")}]
|}""".stripMargin
)
}
getResponse returns a Future[JsValue] now, removing both Await.result cases, and actor case become:
filteredSubscribers.foreach { subscriber =>
val actorOptions = getActorOptions(subscriber)
getResponse(actorOptions, request) map { data =>
subscriber ? data
}
}
I don't know why, still have a synchronous behavior. Damn, this can be due of my subscribers type: Set[ActorRef]? I tried to use parallel foreach and this looks like solving my problem:
filteredSubscribers.par.foreach { subscriber =>
val actorOptions = getActorOptions(subscriber)
getResponse(actorOptions, request) map { data =>
subscriber ? data
}
}

Response from Server Scala with Play

I'm doing my first application with Scala and Play! Where I use a WebSocket exactly like they do in the chatroom example, so far I can send messages to my server but I don't seem to understand where I can "handle" this messages that I get from my client, also I wanted to know if I can send Json Arrays from my server to my client:
#Singleton
class HomeController #Inject()(cc: ControllerComponents)
(implicit actorSystem: ActorSystem,
mat: Materializer,
executionContext: ExecutionContext)
extends AbstractController(cc) {
private type WSMessage = String
private val logger = Logger(getClass)
private implicit val logging = Logging(actorSystem.eventStream, logger.underlyingLogger.getName)
// chat room many clients -> merge hub -> broadcasthub -> many clients
private val (chatSink, chatSource) = {
// Don't log MergeHub$ProducerFailed as error if the client disconnects.
// recoverWithRetries -1 is essentially "recoverWith"
val source = MergeHub.source[WSMessage]
.log("source")
.recoverWithRetries(-1, { case _: Exception ⇒ Source.empty })
val sink = BroadcastHub.sink[WSMessage]
source.toMat(sink)(Keep.both).run()
}
private val userFlow: Flow[WSMessage, WSMessage, _] = {
Flow.fromSinkAndSource(chatSink, chatSource)
}
def index: Action[AnyContent] = Action { implicit request: RequestHeader =>
val webSocketUrl = routes.HomeController.chat().webSocketURL()
logger.info(s"index: ")
Ok(views.html.index(webSocketUrl))
}
def chat(): WebSocket = {
WebSocket.acceptOrResult[WSMessage, WSMessage] {
case rh if sameOriginCheck(rh) =>
Future.successful(userFlow).map { flow =>
Right(flow)
}.recover {
case e: Exception =>
val msg = "Cannot create websocket"
logger.error(msg, e)
val result = InternalServerError(msg)
Left(result)
}
case rejected =>
logger.error(s"Request ${rejected} failed same origin check")
Future.successful {
Left(Forbidden("forbidden"))
}
}
}
}
Btw I'm sending the messages from my client through a Jquery function.
Edit The way I want to handle this messages is by passing them as parameters to a function, the function will return an Array of strings or integers which I want to return to the client
All you need to do is add a flow stage to your between your source and sink which can store in DB and return some other message to clients.
Here's how you can fix the above problem:
Do whatever you want with the received message here.
val flow = Flow[WSMessage].map { element =>
println(s"Message: $element")
element
}
Note that for persisting messages in the database, you will most likely want to use an async stage, roughly:
val flow = Flow[WSMessage].mapAsync(parallelism) { element =>
println(s"Message: $element")
// assuming DB.write() returns a Future[Unit]
DB.write(element).map(_ => element)
}
And finally, you need to add the flow stage to the stream pipeline.
source.via(flow).toMat(sink)(Keep.both).run()

How to add an error flow for Akka http websockets

I've been banging my head against the wall for quite some time as I can't figure out how to add an error flow for an akka http websocket flow. What I'm trying to achieve is:
Message comes in from WS client
It's parsed with circe from json
If the message was the right format send the parsed message to an actor
If the message was the wrong format return an error message to the client
The actor can additionally send messages to the client
Without the error handling this was quite easy, but I can't figure out how to add the errors. Here's what I have:
type GameDecodeResult =
Either[(String, io.circe.Error), GameLobby.LobbyRequest]
val errorFlow =
Flow[GameDecodeResult]
.mapConcat {
case Left(err) => err :: Nil
case Right(_) => Nil
}
.map { case (message, error) =>
logger.info(s"failed to parse message $message", error)
TextMessage(Error(error.toString).asJson.spaces2)
}
val normalFlow = {
val normalFlowSink =
Flow[GameDecodeResult]
.mapConcat {
case Right(msg) => msg :: Nil
case Left(_) => Nil
}
.map(req => GameLobby.IncomingMessage(userId, req))
.to(Sink.actorRef[GameLobby.IncomingMessage](gameLobby, PoisonPill))
val normalFlowSource: Source[Message, NotUsed] =
Source.actorRef[GameLobby.OutgoingMessage](10, OverflowStrategy.fail)
.mapMaterializedValue { outActor =>
gameLobby ! GameLobby.UserConnected(userId, outActor)
NotUsed
}
.map(outMessage => TextMessage(Ok(outMessage.message).asJson.spaces2))
Flow.fromSinkAndSource(normalFlowSink, normalFlowSource)
}
val incomingMessageParser =
Flow[Message]
.flatMapConcat {
case tm: TextMessage =>
tm.textStream
case bm: BinaryMessage =>
bm.dataStream.runWith(Sink.ignore)
Source.empty }
.map { message =>
decode[GameLobby.LobbyRequest](message).left.map(err => message -> err)
}
These are my flows defined and I think this should bee good enough, but I have no idea how to assemble them and the complexity of the akka streaming API doesn't help. Here's what I tried:
val x: Flow[Message, Message, NotUsed] =
GraphDSL.create(incomingMessageParser, normalFlow, errorFlow)((_, _, _)) { implicit builder =>
(incoming, normal, error) =>
import GraphDSL.Implicits._
val partitioner = builder.add(Partition[GameDecodeResult](2, {
case Right(_) => 0
case Left(_) => 1
}))
val merge = builder.add(Merge[Message](2))
incoming.in ~> partitioner ~> normal ~> merge
partitioner ~> error ~> merge
}
but admittedly I have absolutely no idea how GraphDSL.create works, where I can use the ~> arrow or what I'm doing in genreal at the last part. It just won't type check and the error messages are not helping me one bit.
A few things needing to be fixed in the Flow you're building using the GraphDSL:
There is no need to pass the 3 subflows to the GraphDSL.create method, as this is only needed to customize the materialized value of your graph. You have already decided the materialized value of your graph is going to be NotUsed.
When connecting incoming using the ~> operator, you need to connect its outlet (.out) to the partition stage.
Every GraphDSL definition block needs to return the shape of your graph - i.e. its external ports. You do that by returning a FlowShape that has incoming.in as input, as merge.out as output. These will define the blueprint of your custom flow.
Because in the end you want to obtain a Flow, you're missing a last call to create is from the graph you defined. This call is Flow.fromGraph(...).
Code example below:
val x: Flow[Message, Message, NotUsed] =
Flow.fromGraph(GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
val partitioner = builder.add(Partition[GameDecodeResult](2, {
case Right(_) => 0
case Left(_) => 1
}))
val merge = builder.add(Merge[Message](2))
val incoming = builder.add(incomingMessageParser)
incoming.out ~> partitioner
partitioner ~> normalFlow ~> merge
partitioner ~> errorFlow ~> merge
FlowShape(incoming.in, merge.out)
})

How to keep connection open for all the time in websockets

Server code :
object EchoService {
def route: Route = path("ws-echo") {
get {
handleWebSocketMessages(flow)
}
} ~ path("send-client") {
get {
sourceQueue.map(q => {
println(s"Offering message from server")
q.offer(BinaryMessage(ByteString("ta ta")))
} )
complete("Sent from server successfully")
}
}
val (source, sourceQueue) = {
val p = Promise[SourceQueue[Message]]
val s = Source.queue[Message](100, OverflowStrategy.backpressure).mapMaterializedValue(m => {
p.trySuccess(m)
m
})
(s, p.future)
}
val flow =
Flow.fromSinkAndSourceMat(Sink.ignore, source)(Keep.right)
}
Client Code :
object Client extends App {
implicit val actorSystem = ActorSystem("akka-system")
implicit val flowMaterializer = ActorMaterializer()
val config = actorSystem.settings.config
val interface = config.getString("app.interface")
val port = config.getInt("app.port")
// print each incoming strict text message
val printSink: Sink[Message, Future[Done]] =
Sink.foreach {
case message: TextMessage.Strict =>
println(message.text)
case _ => println(s"received unknown message format")
}
val (source, sourceQueue) = {
val p = Promise[SourceQueue[Message]]
val s = Source.queue[Message](100, OverflowStrategy.backpressure).mapMaterializedValue(m => {
p.trySuccess(m)
m
})
(s, p.future)
}
val flow =
Flow.fromSinkAndSourceMat(printSink, source)(Keep.right)
val (upgradeResponse, sourceClosed) =
Http().singleWebSocketRequest(WebSocketRequest("ws://localhost:8080/ws-echo"), flow)
val connected = upgradeResponse.map { upgrade =>
// just like a regular http request we can get 404 NotFound,
// with a response body, that will be available from upgrade.response
if (upgrade.response.status == StatusCodes.SwitchingProtocols || upgrade.response.status == StatusCodes.OK ) {
Done
} else {
throw new RuntimeException(s"Connection failed: ${upgrade.response.status}")
}
}
connected.onComplete(println)
}
when i hit http://localhost:8080/send-client i see messages coming to client but after a while if try to send to client again i don't see any messages on client side :s . I also tried source.concatMat(Source.maybe)(Keep.right) but no luck :(
Edit : I tested with js client, somehow connection/flow closed on server end , is there anyway to prevent this ? and how can i listen to this event while using akka-http websocket client :s
Hi,
The reason why it does not keep connected is because by default all
HTTP connections have idle-timeout on by default to keep the system
from leaking connections if clients disappear without any signal.
One way to overcome this limitation (and actually my recommended
approach) is to inject keep-alive messages on the client side
(messages that the server otherwise ignore, but informs the underlying
HTTP server that the connection is still live).
You can override the idle-timeouts in the HTTP server configuration to
a larger value but I don't recommend that.
If you are using stream based clients, injecting heartbeats when
necessary is as simple as calling keepAlive and providing it a time
interval and a factory for the message you want to inject:
http://doc.akka.io/api/akka/2.4.7/index.html#akka.stream.scaladsl.Flow#keepAliveU>:Out:FlowOps.this.Repr[U]
That combinator will make sure that no periods more than T will be
silent as it will inject elements to keep this contract if necessary
(and will not inject anything if there is enough background traffic)
-Endre
thank you Endre :) , working snippet ..
// on client side
val (source, sourceQueue) = {
val p = Promise[SourceQueue[Message]]
val s = Source.queue[Message](Int.MaxValue, OverflowStrategy.backpressure).mapMaterializedValue(m => {
p.trySuccess(m)
m
}).keepAlive(FiniteDuration(1, TimeUnit.SECONDS), () => TextMessage.Strict("Heart Beat"))
(s, p.future)
}

Akka-Streams ActorPublisher does not receive any Request messages

I am trying to continuously read the wikipedia IRC channel using this lib: https://github.com/implydata/wikiticker
I created a custom Akka Publisher, which will be used in my system as a Source.
Here are some of my classes:
class IrcPublisher() extends ActorPublisher[String] {
import scala.collection._
var queue: mutable.Queue[String] = mutable.Queue()
override def receive: Actor.Receive = {
case Publish(s) =>
println(s"->MSG, isActive = $isActive, totalDemand = $totalDemand")
queue.enqueue(s)
publishIfNeeded()
case Request(cnt) =>
println("Request: " + cnt)
publishIfNeeded()
case Cancel =>
println("Cancel")
context.stop(self)
case _ =>
println("Hm...")
}
def publishIfNeeded(): Unit = {
while (queue.nonEmpty && isActive && totalDemand > 0) {
println("onNext")
onNext(queue.dequeue())
}
}
}
object IrcPublisher {
case class Publish(data: String)
}
I am creating all this objects like so:
def createSource(wikipedias: Seq[String]) {
val dataPublisherRef = system.actorOf(Props[IrcPublisher])
val dataPublisher = ActorPublisher[String](dataPublisherRef)
val listener = new MessageListener {
override def process(message: Message) = {
dataPublisherRef ! Publish(Jackson.generate(message.toMap))
}
}
val ticker = new IrcTicker(
"irc.wikimedia.org",
"imply",
wikipedias map (x => s"#$x.wikipedia"),
Seq(listener)
)
ticker.start() // if I comment this...
Thread.currentThread().join() //... and this I get Request(...)
Source.fromPublisher(dataPublisher)
}
So the problem I am facing is this Source object. Although this implementation works well with other sources (for example from local file), the ActorPublisher don't receive Request() messages.
If I comment the two marked lines I can see, that my actor has received the Request(count) message from my flow. Otherwise all messages will be pushed into the queue, but not in my flow (so I can see the MSG messages printed).
I think it's something with multithreading/synchronization here.
I am not familiar enough with wikiticker to solve your problem as given. One question I would have is: why is it necessary to join to the current thread?
However, I think you have overcomplicated the usage of Source. It would be easier for you to work with the stream as a whole rather than create a custom ActorPublisher.
You can use Source.actorRef to materialize a stream into an ActorRef and work with that ActorRef. This allows you to utilize akka code to do the enqueing/dequeing onto the buffer while you can focus on the "business logic".
Say, for example, your entire stream is only to filter lines above a certain length and print them to the console. This could be accomplished with:
def dispatchIRCMessages(actorRef : ActorRef) = {
val ticker =
new IrcTicker("irc.wikimedia.org",
"imply",
wikipedias map (x => s"#$x.wikipedia"),
Seq(new MessageListener {
override def process(message: Message) =
actorRef ! Publish(Jackson.generate(message.toMap))
}))
ticker.start()
Thread.currentThread().join()
}
//these variables control the buffer behavior
val bufferSize = 1024
val overFlowStrategy = akka.stream.OverflowStrategy.dropHead
val minMessageSize = 32
//no need for a custom Publisher/Queue
val streamRef =
Source.actorRef[String](bufferSize, overFlowStrategy)
.via(Flow[String].filter(_.size > minMessageSize))
.to(Sink.foreach[String](println))
.run()
dispatchIRCMessages(streamRef)
The dispatchIRCMessages has the added benefit that it will work with any ActorRef so you aren't required to only work with streams/publishers.
Hopefully this solves your underlying problem...
I think the main problem is Thread.currentThread().join(). This line will 'hang' current thread because this thread is waiting for himself to die. Please read https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.html#join-long- .