Play Framework: How to modify the response body without blocking? - scala

I'm cutting my teeth on Play using the book Reactive Web Applications: Covers Play, Akka, and Reactive Streams. Chapter 4, among other things, teaches how to write a filter, but the code shown in the book doesn't compile, because in Play 2.4.x, Result.body used to be Enumerator[Array[Byte]] and in 2.5.x it is play.api.http.HttpEntity.
My version of the filter is as below:
class ScoreFilter #Inject()(implicit val mat: Materializer, ec: ExecutionContext) extends Filter {
override def apply(nextFilter: (RequestHeader) => Future[Result])(rh: RequestHeader) =
nextFilter(rh).map { result =>
if (result.header.status == OK || result.header.status == NOT_ACCEPTABLE) {
val correct = result.session(rh).get("correct").getOrElse(0)
val wrong = result.session(rh).get("wrong").getOrElse(0)
val score = s"\nYour current score is: $correct correct " +
s"answers and $wrong wrong answers"
val contentType = result.body.contentType
val scoreByteString = ByteString(score.getBytes(UTF_8))
val maybeNewBody = result.body.consumeData.map(_.concat(scoreByteString))
import scala.concurrent.duration._
val newBody = Await.result(maybeNewBody, 10 seconds)
result.copy(body = Strict(newBody, contentType))
} else {
result
}
}
}
In the book:
// result.body returns a play.api.http.HttpEntity which doesn't have an andThen method
val newBody = result.body andThen Enumerator(score.getBytes(UTF_8))
result.copy(body = newBody)
As you can see, my version of the filter works but it blocks on the future. I'm wondering if there's a better way to do this without blocking?
P.S.: Before dismissing my question as duplicate, please be aware that I've read all of the following threads and they convert the response body into a string, which isn't what I want.
Scala play http filters: how to find the request body
Play framework filter that modifies json request and response
Play 2.4: intercept and modify response body

if you want to avoid Await.result, you can do:
nextFilter(rh).flatMap { result =>
...
maybeNewBody map { newBody =>
result.copy(body = Strict(newBody, contentType))
}
} else Future.successful(result)
(note the change of map to flatMap)

Related

Implement simple architecture using Akka Graphs

I’m attempting to setup a simple graph structure that process data via invoking rest services, forwards the result of each service to an intermediary processing unit before forwarding the result. Here is a high level architecture :
Can this be defined using Akka graph streams ? Reading https://doc.akka.io/docs/akka/current/stream/stream-graphs.html I don't understand how to even implement this simple architecture.
I've tried to implement custom code to execute functions within a graph :
package com.graph
class RestG {
def flow (in : String) : String = {
return in + "extra"
}
}
object RestG {
case class Flow(in: String) {
def out : String = in+"out"
}
def main(args: Array[String]): Unit = {
List(new RestG().flow("test") , new RestG().flow("test2")).foreach(println)
}
}
I'm unsure how to send data between the functions. So I think I should be using Akka Graphs but how to implement the architecture above ?
Here's how I would approach the problem. First some types:
type Data = Int
type RestService1Response = String
type RestService2Response = String
type DisplayedResult = Boolean
Then stub functions to asynchronously call the external services:
def callRestService1(data: Data): Future[RestService1Response] = ???
def callRestService2(data: Data): Future[RestService2Response] = ???
def resultCombiner(resp1: RestService1Response, resp2: RestService2Response): DisplayedResult = ???
Now for the Akka Streams (I'm leaving out setting up an ActorSystem etc.)
import akka.Done
import akka.stream.scaladsl._
type SourceMatVal = Any
val dataSource: Source[Data, SourceMatVal] = ???
def restServiceFlow[Response](callF: Data => Future[Data, Response], maxInflight: Int) = Flow[Data].mapAsync(maxInflight)(callF)
// NB: since we're fanning out, there's no reason to have different maxInflights here...
val service1 = restServiceFlow(callRestService1, 4)
val service2 = restServiceFlow(callRestService2, 4)
val downstream = Flow[(RestService1Response, RestService2Response)]
.map((resultCombiner _).tupled)
val splitAndCombine = GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._
val fanOut = b.add(Broadcast[Data](2))
val fanIn = b.add(Zip[RestService1Response, RestService2Response])
fanOut.out(0).via(service1) ~> fanIn.in0
fanOut.out(1).via(service2) ~> fanIn.in1
FlowShape(fanOut.in, fanIn.out)
}
// This future will complete with a `Done` if/when the stream completes
val future: Future[Done] = dataSource
.via(splitAndCombine)
.via(downstream)
.runForeach { displayableData =>
??? // Display the data
}
It's possible to do all the wiring within the Graph DSL, but I generally prefer to keep my graph stages as simple as possible and only use them to the extent that the standard methods on Source/Flow/Sink can't do what I want.

How to bind akka http with akka streams?

I'm trying to use streams instead of pure actors to handle http requests and I came with the following code:
trait ImagesRoute {
val log = LoggerFactory.getLogger(this.getClass)
implicit def actorRefFactory: ActorRefFactory
implicit def materializer: ActorMaterializer
val source =
Source
.actorRef[Image](Int.MaxValue, OverflowStrategy.fail)
.via(Flow[Image].mapAsync(1)(ImageRepository.add))
.toMat(Sink.asPublisher(true))(Keep.both)
val route = {
pathPrefix("images") {
pathEnd {
post {
entity(as[Image]) { image =>
val (ref, publisher) = source.run()
val addFuture = Source.fromPublisher(publisher)
val future = addFuture.runWith(Sink.head[Option[Image]])
ref ! image
onComplete(future.mapTo[Option[Image]]) {
case Success(img) =>
complete(Created, img)
case Failure(e) =>
log.error("Error adding image resource", e)
complete(InternalServerError, e.getMessage)
}
}
}
}
}
}
}
I'm not sure if this is the correct way to do that, or even if this is a good approach or if I should use an actor to interact with the route, using the ask pattern and then inside the actor, stream everything.
Any ideas?
If you're only expecting 1 image from the entity then you don't need to create a Source from an ActorRef and you don't need Sink.asPublisher, you can simply use Source.single:
def imageToComplete(img : Option[Image]) : StandardRoute =
img.map(i => complete(Created, i))
.getOrElse {
log error ("Error adding image resource", e)
complete(InternalServerError, e.getMessage
}
...
entity(as[Image]) { image =>
val future : Future[StandardRoute] =
Source.single(image)
.via(Flow[Image].mapAsync(1)(ImageRepository.add))
.runWith(Sink.head[Option[Image]])
.map(imageToComplete)
onComplete(future)
}
Simplifying your code further, the fact that you are only processing 1 image means that Streams are unnecessary since there is no need for backpressure with just 1 element:
val future : Future[StandardRoute] = ImageRepository.add(image)
.map(imageToComplete)
onComplete(future)
In the comments you indicated
"this is just a simple example, but the stream pipeline should be
bigger doing a lot of things like contacting external resources and
eventually back pressure things"
This would only apply if your entity was a stream of images. If you're only ever processing 1 image per HttpRequest then backpressure never applies, and any stream you create will be a slower version of a Future.
If your entity is in fact a stream of Images, then you could use it as part of stream:
val byteStrToImage : Flow[ByteString, Image, _] = ???
val imageToByteStr : Flow[Image, Source[ByteString], _] = ???
def imageOptToSource(img : Option[Image]) : Source[Image,_] =
Source fromIterator img.toIterator
val route = path("images") {
post {
extractRequestEntity { reqEntity =>
val stream = reqEntity.via(byteStrToImage)
.via(Flow[Image].mapAsync(1)(ImageRepository.add))
.via(Flow.flatMapConcat(imageOptToSource))
.via(Flow.flatMapConcat(imageToByteStr))
complete(HttpResponse(status=Created,entity = stream))
}
}
}

Play / Logging / Print Response Body / Run over enumerator / buffer the body

I'm looking for a way to print the response body in Play framework, I have a code like this:
object AccessLoggingAction extends ActionBuilder[Request] {
def invokeBlock[A](request: Request[A], block: (Request[A]) => Future[Result]) = {
Logger.info(s"""Request:
id=${request.id}
method=${request.method}
uri=${request.uri}
remote-address=${request.remoteAddress}
body=${request.body}
""")
val ret = block(request)
/*
ret.map {result =>
Logger.info(s"""Response:
id=${request.id}
body=${result.body}
""")
}
*/ //TODO: find out how to print result.body (be careful not to consume the enumerator)
ret
}
}
Currently the commented-out code is not working as I wanted, I mean, it would print:
Response:
id=1
body=play.api.libs.iteratee.Enumerator$$anon$18#39e6c1a2
So, I need to find a way to get a String out of Enumerator[Array[Byte]]. I tried to grasp the concept of Enumerator by reading this: http://mandubian.com/2012/08/27/understanding-play2-iteratees-for-normal-humans/
So..., if I understand it correctly:
I shouldn't dry-up the enumerator in the process of converting it to String. Otherwise, the client would receive nothing.
Let's suppose I figure out how to implement the T / filter mechanism. But then... wouldn't it defeat the purpose of Play framework as non-blocking streaming framework (because I would be building up the complete array of bytes in the memory, before calling toString on it, and finally log it)?
So, what's the correct way to log the response?
Thanks in advance,
Raka
This code works:
object AccessLoggingAction extends ActionBuilder[Request] {
def invokeBlock[A](request: Request[A], block: (Request[A]) => Future[Result]) = {
val start = System.currentTimeMillis
Logger.info(s"""Request:
id=${request.id}
method=${request.method}
uri=${request.uri}
remote-address=${request.remoteAddress}
body=${request.body}
""")
val resultFut = block(request)
resultFut.map {result =>
val time = System.currentTimeMillis - start
Result(result.header, result.body &> Enumeratee.map(arrOfBytes => {
val body = new String(arrOfBytes.map(_.toChar))
Logger.info(s"""Response:
id=${request.id}
method=${request.method}
uri=${request.uri}
delay=${time}ms
status=${result.header.status}
body=${body}""")
arrOfBytes
}), result.connection)
}
}
}
I partly learned it from here (on how to get the byte array out of enumerator): Scala Play 2.1: Accessing request and response bodies in a filter.
I'm using Play 2.3.7 while the link I gave uses 2.1 (and still uses PlainResult, which no longer exists in 2.3).
As it appears to me, if you do logging inside result.body &> Enumeratee.map (as suggested in https://stackoverflow.com/a/27630208/1781549) and the result body is presented in more than one chunk, then each chunk will be logged independently. You probably don't want this.
I'd implement it like this:
val ret = block(request).flatMap { result =>
val consume = Iteratee.consume[Array[Byte]]()
val bodyF = Iteratee.flatten(result.body(consume)).run
bodyF.map { bodyBytes: Array[Byte] =>
//
// Log the body
//
result.copy(body = Enumerator(bodyBytes))
}
}
But be warned: the whole idea of this is to consume all the data from the result.body Enumerator before logging (and return the new Enumerator). So, if the response is big, or you rely on streaming, then it's probably also the thing you don't want.
I used the above answer as a starting point, but noticed that it will only log responses if a body is present. We've adapted it to this:
var responseBody = None:Option[String]
val captureBody = Enumeratee.map[Array[Byte]](arrOfBytes => {
val body = new String(arrOfBytes.map(_.toChar))
responseBody = Some(body)
arrOfBytes
})
val withLogging = (result.body &> captureBody).onDoneEnumerating({
logger.debug(.. create message here ..)
})
result.copy(body=withLogging)

How to write tests for a Play Framework Filter?

I've written a Filter for my Play application:
object MyFilter {
def apply() = new MyFilter()
}
class MyFilter extends EssentialFilter {
def apply(next: EssentialAction) = new EssentialAction {
def apply(requestHeader: RequestHeader) = {
requestHeader.cookies.get("myCookie") match {
case Some(cookie) => {
requestHeader.session + ("importantValue", cookie.value)
next(requestHeader)
}
case None => {
val importantValue = ... // retrieve the value from somewhere
requestHeader.session + ("importantValue", importantValue)
next(requestHeader).map(_.withCookies(Cookie("importantValue", value)))
}
}
}
}
There're a lot of examples in the Play documentation about how to write test, but I've no idea how to unit test MyFilter. Can someone help me a little?
Thanks
Torben
The play framework source code shows test for filters.
for example:
val filter = SecurityHeadersFilter()
// Play.current is set at this point...
val rh = FakeRequest()
val action = Action(Ok("success"))
val result = filter(action)(rh).run()
header(X_FRAME_OPTIONS_HEADER, result) must beSome("DENY")
taken from here SecurityHeadersFilterSpec.scala
Not answering your question but requestHeader.session + ("importantValue", cookie.value) is never going to do anything. You're creating a new session object and don't do anything with it.

Concurrent.patchPanel not sending data from multiple enumerators - only sends from last enumerator added

I'm hoping someone else has used patchPanel to combine multiple enumerators together going down to a client over a websocket. The issue i'm running into is that the patchPanel is only sending the data feed from the last enumerator added into it.
I followed the example from; http://lambdaz.blogspot.ca/2012/12/play-21-multiplexing-enumerators-into.html which is the only reference I've been able to find regarding patchPanel.
Versions; play! 2.1.1 (using Java 1.7.0_11 and Scala 2.10.0)
The web socket method;
def monitorStream = WebSocket.async[JsValue] { request =>
val promiseIn = promise[Iteratee[JsValue, Unit]]
val out = Concurrent.patchPanel[JsValue] { patcher =>
val in = Iteratee.foreach[JsValue] { json =>
val event:Option[String] = (json \ "event").asOpt[String]
val systemId = (json \ "systemId").as[Long]
event.getOrElse("") match {
case "join" =>
val physicalSystem = SystemIdHandler.getById(systemId)
val monitorOut = (MonitorStreamActor.joinMonitor(physicalSystem))
monitorOut map { enum =>
val success = patcher.patchIn(enum)
}
}
}.mapDone { _ => Logger.info("Disconnected") }
promiseIn.success(in)
}
future(Iteratee.flatten(promiseIn.future),out)
}
The MonitorStreamActor call;
def joinMonitor(physicalSystem: PhysicalSystem):
scala.concurrent.Future[Enumerator[JsValue]]
= {
val monitorActor = ActorBase.akkaSystem.actorFor("/user/system-" + physicalSystem.name +"/stream")
(monitorActor ? MonitorJoin()).map {
case MonitorConnected(enumerator) =>
enumerator
}
}
The enumerator is returned fine, and the data fed into it is coming from a timer calling the actor. Actor definition, the timer hits the UpdatedTranStates case;
class MonitorStreamActor() extends Actor {
val (monitorEnumerator, monitorChannel) = Concurrent.broadcast[JsValue]
import play.api.Play.current
def receive = {
case MonitorJoin() => {
Logger.debug ("Actor monitor join")
sender ! MonitorConnected(monitorEnumerator)
}
case UpdatedTranStates(systemName,tranStates) => {
//println("Got updated Tran States")
val json = Json.toJson(tranStates.map(m => Map("State" -> m._1, "Count" -> m._2) ))
//println("Pushing updates to monitorChannel")
sendUpdateToClients(systemName, "states", json)
}
def sendUpdateToClients(systemName:String, updateType:String, json:JsValue) {
monitorChannel.push(Json.toJson(
Map(
"dataType"->Json.toJson(updateType),
"systemName" -> Json.toJson(systemName),
"data"->json)))
}
}
}
I've poked around for a while on this and haven't found a reason why only the last enumerator that is added into the patchPanel has the data sent. the API docs are not of much help, it really sounds like all you have to do is call patchIn and it should combine all enumerators to an iteratee, but that doesn't seem to be the case.
The PatchPanel by design replaces the current enumerator with the new one provided by the patchIn method.
In order to have multiple Enumerators combined together you need to use interleave or andThen methods to combine enumerators together. Interleave is preferred for this case as it will take events from each Enumerator as they are available, vs emptying one then moving to the next (as with andThen operator).
ie, in monitorStream;
val monitorOut = (MonitorStreamActor.joinMonitor(physicalSystem))
monitorOut map { enum =>
mappedEnums += ((physicalSystem.name, enum))
patcher.patchIn(Enumerator.interleave[JsValue]( mappedEnums.values.toSeq))
}
patcher is the patch panel, and mappedEnums is a HashMap[String,Enumerator[JsValue]] - re-add the patcher each time the Enumerators change (add or delete) - it works, not sure if it's the best way, but it'll do for now :)