I have a REST service (Play Framework 2.0 w/Scala) that receives messages via a POST request.
I want to allow a user to see the queue of messages received in a webpage. I wanted to create a SSE channel between browser and server, so the server pushes new messages to the browser.
To create that SSE stream, as per documentation, I'm using a chain of Enumerator/Enumeratee/Iteratee.
My problem is: how do I inject the messages received from the POST request to the enumerator. So given a code like follows:
def receive(msg: String) = Action {
sendToEnumerator()
Ok
}
val enumerator = Enumerator.fromCallback( ??? )
def sseStream() = Action {
Ok.stream(enumerator &> anotherEnumeratee ><> EventStrem()).as("text/evetn-stream")
}
What should I put in both sendToEnumerator and enumerator (where the ??? are). Or should I just use WebSockets and Actors instead? (I favour SEE due to broader compatibility, so would like to use SSE if possible)
Ok, found a way:
// The enum for pushing data to spread to all connected users
val hubEnum = Enumerator.imperative[String]()
// The hub used to get multiple output of a common input (the hubEnum)
val hub = Concurrent.hub[String](hubEnum)
// Converts message to Json for the web version
private val asJson: Enumeratee[String, JsValue] = Enumeratee.map[String] {
text => JsObject(
List(
"eventName" -> JsString("eventName"),
"text" -> JsString(text)
)
)
}
// loads data into hubEnum
def receiveData(msg: String) = Action { implicit request =>
hubEnum push msg
}
// read the Hub iterator and pushes back to clients
def stream = Action { implicit request =>
Ok.stream(hub.getPatchCord &> asJson ><> EventSource()).as("text/event-stream")
}
The trick is to create an imperative Enumerator. This enumerator allows you to push data into it when it becomes available. With this then you can follow the standard procedure: create a Hub based on the enumerator, convert it with some Enumeratee and send it back to browsers via SSE.
Thanks to this website for giving me the solution :)
Related
The aim is to stream data from a database, perform some computation on this chunk of data(this computation returns a Future of some case class) and send this data as chunked response to the user. Currently I am able to stream data and send the response without performing any computation. However, I am unable to perform this computation and then stream the result.
This is the route I have implemented.
def streamingDB1 =
path("streaming-db1") {
get {
val src = Source.fromPublisher(db.stream(getRds))
complete(src)
}
}
The function getRds returns the rows of a table mapped into a case class(Using slick). Now consider the function compute which takes each row as an input and returns a Future of another case class. Something like
def compute(x: Tweet) : Future[TweetNew] = ?
How can I implement this function on variable src and send the chunked response(as a stream) of this computation to the user.
You could transform the source using mapAsync:
val src =
Source.fromPublisher(db.stream(getRds))
.mapAsync(parallelism = 3)(compute)
complete(src)
Adjust the level of parallelism as needed.
Note that you might need to configure a few settings as mentioned in the Slick documentation:
Note: Some database systems may require session parameters to be set in a certain way to support streaming without caching all data at once in memory on the client side. For example, PostgreSQL requires both .withStatementParameters(rsType = ResultSetType.ForwardOnly, rsConcurrency = ResultSetConcurrency.ReadOnly, fetchSize = n) (with the desired page size n) and .transactionally for proper streaming.
So if you're using PostgreSQL, for example, then your Source might look something like the following:
val src =
Source.fromPublisher(
db.stream(
getRds.withStatementParameters(
rsType = ResultSetType.ForwardOnly,
rsConcurrency = ResultSetConcurrency.ReadOnly,
fetchSize = 10
).transactionally
)
).mapAsync(parallelism = 3)(compute)
You need to have a way to marshall TweetNew and also if you send a chunk with length 0 client may close connection.
This code works with curl:
case class TweetNew(str: String)
def compute(string: String) : Future[TweetNew] = Future {
TweetNew(string)
}
val route = path("hello") {
get {
val byteString: Source[ByteString, NotUsed] = Source.apply(List("t1", "t2", "t3"))
.mapAsync(2)(compute)
.map(tweet => ByteString(tweet.str + "\n"))
complete(HttpEntity(ContentTypes.`text/plain(UTF-8)`, byteString))
}
}
How to keep the client (web) connection in a memory variable and then send outgoing messages to the client (web) when needed?
I already have some simple code for pushing back message to the client once the server receives messages from the client. How to modify the code below for the outgoing messaging part?
implicit val actorSystem = ActorSystem("akka-system")
implicit val flowMaterializer = ActorMaterializer()
implicit val executionContext = actorSystem.dispatcher
val ip = "127.0.0.1"
val port = 32000
val route = get {
pathEndOrSingleSlash {
complete("Welcome to websocket server")
}
} ~
path("hello") {
get {
handleWebSocketMessages(echoService)
}
}
def sendMessageToClient(msg : String) {
// *** How to implement this?
// *** How to save the client connection when it is first connected?
// Then how to send message to this connection?
}
val echoService = Flow[Message].collect {
// *** Here the server push back messages when receiving msg from client
case tm : TextMessage => TextMessage(Source.single("Hello ") ++ tm.textStream)
case _ => TextMessage("Message type unsupported")
}
val binding = Http().bindAndHandle(route, ip, port)
You can look into pipelining the sink flow via .map call. Inside the .map call you can capture the value and then return the same message. For example:
Flow[Message].collect {
case tm : TextMessage =>
TextMessage(Source.single("Hello ") ++ tm.textStream.via(
Flow[String].map((message) => {println(message) /* capture value here*/; message})))
case _ => TextMessage("Message type unsupported")
}
Now, if your intention is to process those values and send out values later, what you want is not a single source-to-sink flow, but two separate streams for sink and source, for which you can use Flow.fromSinkAndSource e.g.
Flow.fromSinkAndSource[Message, Message](
Flow[Message].collect { /* capture values */},
// Or send stream to other sink for more processing
source
)
In all likelihood, this source will be either constructed out of graph DSL, a hand-rolled actor, or you can look into utilizing reusable helpers such as MergeHub.
I'm building a micro-service using Play Framework 2.3.x using Scala (I'm a beginner in both) but I can't figure out a way to stream my request body.
Here is the problem:
I need an endpoint /transform where I can receive a huge TSV file that I will parse and render in another format: simple transformation. The problem is that every single command in my controller is ran "too late". It waits to receive the full file before starting the code.
Example:
def transform = Action.async {
Future {
Logger.info("Too late")
Ok("A response")
}
}
I want to be able to read line-by-line the request body during its upload and process already the request without having to wait for the file to be received completely.
Any hint would be welcome.
This answer applies to Play 2.5.x and higher since it uses the Akka streams API that replaced Play's Iteratee-based streaming in that version.
Basically, you can create a body parser that returns a Source[T] that you can pass to Ok.chunked(...). One way to do this is to use Accumulator.source[T] in the body parser. For example, an action that just returned data sent to it verbatim might look like this:
def verbatimBodyParser: BodyParser[Source[ByteString, _]] = BodyParser { _ =>
// Return the source directly. We need to return
// an Accumulator[Either[Result, T]], so if we were
// handling any errors we could map to something like
// a Left(BadRequest("error")). Since we're not
// we just wrap the source in a Right(...)
Accumulator.source[ByteString]
.map(Right.apply)
}
def stream = Action(verbatimBodyParser) { implicit request =>
Ok.chunked(request.body)
}
If you want to do something like transform a TSV file you can use a Flow to transform the source, e.g:
val tsvToCsv: BodyParser[Source[ByteString, _]] = BodyParser { req =>
val transformFlow: Flow[ByteString, ByteString, NotUsed] = Flow[ByteString]
// Chunk incoming bytes by newlines, truncating them if the lines
// are longer than 1000 bytes...
.via(Framing.delimiter(ByteString("\n"), 1000, allowTruncation = true))
// Replace tabs by commas. This is just a silly example and
// you could obviously do something more clever here...
.map(s => ByteString(s.utf8String.split('\t').mkString(",") + "\n"))
Accumulator.source[ByteString]
.map(_.via(transformFlow))
.map(Right.apply)
}
def convert = Action(tsvToCsv) { implicit request =>
Ok.chunked(request.body).as("text/csv")
}
There may be more inspiration in the Directing the Body Elsewhere section of the Play docs.
I'm trying to write an Actor which connects to an Amazon Kinesis stream and then relays any messages received via Comet to a Web UI. I'm using Source.actorPublisher for this and using the json method with Comet in Play described here. I got the events working just fine using Source.tick(), but when I tried using an ActorPublisher, the Actor never seems to be sent any Request messages as expected. How are requests for data usually sent down an Akka flow? I'm using v2.5 of the Play Framework.
My controller code:
def subDeviceSeen(id: Id): Action[AnyContent] = Action {
val deviceSeenSource: Source[DeviceSeenMessage, ActorRef] = Source.actorPublisher(DeviceSeenEventActor.props)
Ok.chunked(deviceSeenSource
.filter(m => m.id == id)
.map(Json.toJson(_))
via Comet.json("parent.deviceSeen")).as(ContentTypes.JSON)
}
Am I doing anything obviously wrong in the above? Here is my Actor code:
object DeviceSeenEventActor {
def props: Props = Props[DeviceSeenEventActor]
}
class DeviceSeenEventActor extends ActorPublisher[DeviceSeenMessage] {
implicit val mat = ActorMaterializer()(context)
val log = Logging(context.system, this)
def receive: Receive = {
case Request => log.debug("Received request message")
initKinesis()
context.become(run)
case Cancel => context.stop(self)
}
def run: Receive = {
case vsm:DeviceSeenMessage => onNext(vsm)
log.debug("Received request message")
onCompleteThenStop() //we are currently only interested in one message
case _:Any => log.warning("Unknown message received by event Actor")
}
private def initKinesis() = {
//init kinesis, a worker is created and given a reference to this Actor.
//The reference is used to send messages to the actor.
}
}
The 'Received request message' log line is never displayed. Am I missing some implicit? There are no warnings or anything else obvious displayed in the play console.
The issue was that I was pattern matching on case Request => ... instead of case Request() => .... Since I didn't have a default case in my receive() method, the message was simply dropped by the Actor.
I would like to use a SourceQueue to push elements dynamically into an Akka Stream source.
Play controller needs a Source to be able to stream a result using the chuncked method.
As Play uses its own Akka Stream Sink under the hood, I can't materialize the source queue myself using a Sink because the source would be consumed before it's used by the chunked method (except if I use the following hack).
I'm able to make it work if I pre-materialize the source queue using a reactive-streams publisher, but it's a kind of 'dirty hack' :
def sourceQueueAction = Action{
val (queue, pub) = Source.queue[String](10, OverflowStrategy.fail).toMat(Sink.asPublisher(false))(Keep.both).run()
//stupid example to push elements dynamically
val tick = Source.tick(0 second, 1 second, "tick")
tick.runForeach(t => queue.offer(t))
Ok.chunked(Source.fromPublisher(pub))
}
Is there a simpler way to use an Akka Streams SourceQueue with PlayFramework?
Thanks
The solution is to use mapMaterializedValue on the source to get a future of its queue materialization :
def sourceQueueAction = Action {
val (queueSource, futureQueue) = peekMatValue(Source.queue[String](10, OverflowStrategy.fail))
futureQueue.map { queue =>
Source.tick(0.second, 1.second, "tick")
.runForeach (t => queue.offer(t))
}
Ok.chunked(queueSource)
}
//T is the source type, here String
//M is the materialization type, here a SourceQueue[String]
def peekMatValue[T, M](src: Source[T, M]): (Source[T, M], Future[M]) = {
val p = Promise[M]
val s = src.mapMaterializedValue { m =>
p.trySuccess(m)
m
}
(s, p.future)
}
Would like to share an insight I got today, though it may not be appropriate to your case with Play.
Instead of thinking of a Source to trigger, one can often turn the problem upside down and provide a Sink to the function that does the sourcing.
In such a case, the Sink would be the "recipe" (non-materialized) stage and we can now use Source.queue and materialize it right away. Got queue. Got the flow that it runs.