Play / Logging / Print Response Body / Run over enumerator / buffer the body - scala

I'm looking for a way to print the response body in Play framework, I have a code like this:
object AccessLoggingAction extends ActionBuilder[Request] {
def invokeBlock[A](request: Request[A], block: (Request[A]) => Future[Result]) = {
Logger.info(s"""Request:
id=${request.id}
method=${request.method}
uri=${request.uri}
remote-address=${request.remoteAddress}
body=${request.body}
""")
val ret = block(request)
/*
ret.map {result =>
Logger.info(s"""Response:
id=${request.id}
body=${result.body}
""")
}
*/ //TODO: find out how to print result.body (be careful not to consume the enumerator)
ret
}
}
Currently the commented-out code is not working as I wanted, I mean, it would print:
Response:
id=1
body=play.api.libs.iteratee.Enumerator$$anon$18#39e6c1a2
So, I need to find a way to get a String out of Enumerator[Array[Byte]]. I tried to grasp the concept of Enumerator by reading this: http://mandubian.com/2012/08/27/understanding-play2-iteratees-for-normal-humans/
So..., if I understand it correctly:
I shouldn't dry-up the enumerator in the process of converting it to String. Otherwise, the client would receive nothing.
Let's suppose I figure out how to implement the T / filter mechanism. But then... wouldn't it defeat the purpose of Play framework as non-blocking streaming framework (because I would be building up the complete array of bytes in the memory, before calling toString on it, and finally log it)?
So, what's the correct way to log the response?
Thanks in advance,
Raka

This code works:
object AccessLoggingAction extends ActionBuilder[Request] {
def invokeBlock[A](request: Request[A], block: (Request[A]) => Future[Result]) = {
val start = System.currentTimeMillis
Logger.info(s"""Request:
id=${request.id}
method=${request.method}
uri=${request.uri}
remote-address=${request.remoteAddress}
body=${request.body}
""")
val resultFut = block(request)
resultFut.map {result =>
val time = System.currentTimeMillis - start
Result(result.header, result.body &> Enumeratee.map(arrOfBytes => {
val body = new String(arrOfBytes.map(_.toChar))
Logger.info(s"""Response:
id=${request.id}
method=${request.method}
uri=${request.uri}
delay=${time}ms
status=${result.header.status}
body=${body}""")
arrOfBytes
}), result.connection)
}
}
}
I partly learned it from here (on how to get the byte array out of enumerator): Scala Play 2.1: Accessing request and response bodies in a filter.
I'm using Play 2.3.7 while the link I gave uses 2.1 (and still uses PlainResult, which no longer exists in 2.3).

As it appears to me, if you do logging inside result.body &> Enumeratee.map (as suggested in https://stackoverflow.com/a/27630208/1781549) and the result body is presented in more than one chunk, then each chunk will be logged independently. You probably don't want this.
I'd implement it like this:
val ret = block(request).flatMap { result =>
val consume = Iteratee.consume[Array[Byte]]()
val bodyF = Iteratee.flatten(result.body(consume)).run
bodyF.map { bodyBytes: Array[Byte] =>
//
// Log the body
//
result.copy(body = Enumerator(bodyBytes))
}
}
But be warned: the whole idea of this is to consume all the data from the result.body Enumerator before logging (and return the new Enumerator). So, if the response is big, or you rely on streaming, then it's probably also the thing you don't want.

I used the above answer as a starting point, but noticed that it will only log responses if a body is present. We've adapted it to this:
var responseBody = None:Option[String]
val captureBody = Enumeratee.map[Array[Byte]](arrOfBytes => {
val body = new String(arrOfBytes.map(_.toChar))
responseBody = Some(body)
arrOfBytes
})
val withLogging = (result.body &> captureBody).onDoneEnumerating({
logger.debug(.. create message here ..)
})
result.copy(body=withLogging)

Related

Play Framework: How to modify the response body without blocking?

I'm cutting my teeth on Play using the book Reactive Web Applications: Covers Play, Akka, and Reactive Streams. Chapter 4, among other things, teaches how to write a filter, but the code shown in the book doesn't compile, because in Play 2.4.x, Result.body used to be Enumerator[Array[Byte]] and in 2.5.x it is play.api.http.HttpEntity.
My version of the filter is as below:
class ScoreFilter #Inject()(implicit val mat: Materializer, ec: ExecutionContext) extends Filter {
override def apply(nextFilter: (RequestHeader) => Future[Result])(rh: RequestHeader) =
nextFilter(rh).map { result =>
if (result.header.status == OK || result.header.status == NOT_ACCEPTABLE) {
val correct = result.session(rh).get("correct").getOrElse(0)
val wrong = result.session(rh).get("wrong").getOrElse(0)
val score = s"\nYour current score is: $correct correct " +
s"answers and $wrong wrong answers"
val contentType = result.body.contentType
val scoreByteString = ByteString(score.getBytes(UTF_8))
val maybeNewBody = result.body.consumeData.map(_.concat(scoreByteString))
import scala.concurrent.duration._
val newBody = Await.result(maybeNewBody, 10 seconds)
result.copy(body = Strict(newBody, contentType))
} else {
result
}
}
}
In the book:
// result.body returns a play.api.http.HttpEntity which doesn't have an andThen method
val newBody = result.body andThen Enumerator(score.getBytes(UTF_8))
result.copy(body = newBody)
As you can see, my version of the filter works but it blocks on the future. I'm wondering if there's a better way to do this without blocking?
P.S.: Before dismissing my question as duplicate, please be aware that I've read all of the following threads and they convert the response body into a string, which isn't what I want.
Scala play http filters: how to find the request body
Play framework filter that modifies json request and response
Play 2.4: intercept and modify response body
if you want to avoid Await.result, you can do:
nextFilter(rh).flatMap { result =>
...
maybeNewBody map { newBody =>
result.copy(body = Strict(newBody, contentType))
}
} else Future.successful(result)
(note the change of map to flatMap)

Download media file from twilio, using the media URI

I have been having issues with downloading media from the media uri provided on the mms messages.
val url = https://api.twilio.com/2010-04-01/Accounts/xx/Messages/xx/Media/xx
the media url provided is in the above structure,
new URL(url) #> new File("file.png") !! //this fails, due to multiple redirects
When I open the URI in browser the redirect ends up in
http://media.twiliocdn.com.s3-external-1.amazonaws.com/xx/xx
1st url -> 2nd url -> above url ;so,all in all 2 redirects
And if I try the snippet posted above with the new url, it works. I am sure its because of the multiple redirects, the snippet didnt work in the first place.
Been using play framework with scala, can I get any source example to download the file. Any help or pointers is appreciated. Tried various examples but still could not solve the issue.
Some findings =>
Accessing Twilio MMS images
anything similar for scala?
Update: #millhouse
def fileDownloader(urls: String, location: String) = {
import play.api.Play.current
import scala.concurrent.ExecutionContext.Implicits.global
// Make the request
val futureResponse: Future[(WSResponseHeaders, Enumerator[Array[Byte]])] =
WS.url(urls).withFollowRedirects(true).getStream()
futureResponse.flatMap {
case (headers, body) =>
val file = new File(location)
val outputStream = new FileOutputStream(file)
// The iteratee that writes to the output stream
val iteratee = Iteratee.foreach[Array[Byte]] { bytes =>
outputStream.write(bytes)
}
// Feed the body into the iteratee
(body |>>> iteratee).andThen {
case result =>
// Close the output stream whether there was an error or not
outputStream.close()
// Get the result or rethrow the error
result.get
}.map(_ => file)
}
}
This is the approach I had been using till now(works), as explained in the play docs. But I needed a sync approach, meaning I would need to carry out another step on successful file download. Sorry, for not clarifying out ahead.
Update 2 : Solved in this manner,
def fileDownloader(urls: String, location: String) = {
import play.api.Play.current
import scala.concurrent.ExecutionContext.Implicits.global
// Make the request
val futureResponse: Future[(WSResponseHeaders, Enumerator[Array[Byte]])] =
WS.url(urls).withFollowRedirects(true).getStream()
val downloadedFile: Future[File] = futureResponse.flatMap {
case (headers, body) =>
val file = new File(location)
val outputStream = new FileOutputStream(file)
// The iteratee that writes to the output stream
val iteratee = Iteratee.foreach[Array[Byte]] { bytes =>
outputStream.write(bytes)
}
// Feed the body into the iteratee
(body |>>> iteratee).andThen {
case result =>
// Close the output stream whether there was an error or not
outputStream.close()
// Get the result or rethrow the error
result.get
}.map(_ => file)
}
downloadedFile.map{ fileIn =>
//things needed to do
}
}
Thanks,
I haven't used the Twilio MMS API but it should be very straightforward to get the Play Framework HTTP client to follow redirects, using the documented option to the client:
val url = "https://api.twilio.com/2010-04-01/Accounts/xx/Messages/xx/Media/xx"
ws.url(url).withFollowRedirects(true).get().map { response =>
val theBytes:Array[Byte] = response.bodyAsBytes // Play 2.4 and lower
// ... save it
}
Note that the above code works for Play 2.4.x and lower; the bodyAsBytes method of WSResponse returns an Array[Byte]. If you're on the current cutting-edge and using Play 2.5.x, bodyAsBytes gives you an Akka ByteString with lots of nice functional methods, but you probably just want to call toArray on it if all you want is to store the data:
ws.url(url).withFollowRedirects(true).get().map { response =>
val theBytes:Array[Byte] = response.bodyAsBytes.toArray // Play 2.5
// ... save it
}

Play 2.4: intercept and modify response body

According to play documentation this is what a custom action should look like:
object CustomAction extends ActionBuilder[Request] {
def invokeBlock[A](request: Request[A], block: Request[A] => Future[Result]): Future[Result] = {
block(request)
}
}
But say if I wanted to append "foo" to every response body, how do I do that? Obviously below doesn't work:
block.andThen(result => result.map(r => r.body.toString + "foo")).apply(request)
Any ideas?
UPDATE: Something worth mentioning is that this action would be mostly used as asynchronous in the controller:
def test = CustomAction.async {
//...
}
You'll need to take the Enumerator[Array[Byte]] from the Result body and feed it to an iteratee to actually consume the result body before you can modify it. So a simple iteratee that consumes the result body and converts to a String might look like:
block.apply(request).flatMap { res =>
Iteratee.flatten(res.body |>> Iteratee.consume[Array[Byte]]()).run.map { byteArray =>
val bodyStr = new String(byteArray.map(_.toChar))
Ok(bodyStr + "foo")
}
}
I used flatMap as the result of running Iteratee.flatten is a Future[T]. Check out https://www.playframework.com/documentation/2.4.x/Enumerators for more details on how to work with Enumerators/Iteratees.

Scala play http filters: how to find the request body

I'm trying to write a filter similar to the simple one described in http://www.playframework.com/documentation/2.1.1/ScalaHttpFilters but I need to access the request body. The documentation below states that "when we invoke next, we get back an Iteratee. You could wrap this in an Enumeratee to do some transformations if you wished." I'm trying to figure out how to wrap the Iteratee so I can get the request body as a string within the filter so I can log that as well.
First thing you have to know is when the Filter is invoked, the request body is not parsed yet. That's why it's giving you a RequestHeader. You'll have to find out the type of body, and call the proper body parser accordingly.
You can find a example of body parsing in the CSRF filter (It can lookup for CSRF tokens in the first bytes of the request body).
See: https://github.com/playframework/playframework/blob/master/framework/src/play-filters-helpers/src/main/scala/csrf.scala#L221-L233.
Hope it helps.
I spent some time on this.
I am no means a Scala expert but this works pretty well! :)
object AccessLog extends EssentialFilter {
def apply(nextFilter: EssentialAction) = new EssentialAction {
def apply(requestHeader: RequestHeader) = {
val startTime = System.currentTimeMillis
nextFilter(requestHeader).map { result =>
val endTime = System.currentTimeMillis
val requestTime = endTime - startTime
val bytesToString: Enumeratee[ Array[Byte], String ] = Enumeratee.map[Array[Byte]]{ bytes => new String(bytes) }
val consume: Iteratee[String,String] = Iteratee.consume[String]()
val resultBody : Future[String] = result.body |>>> bytesToString &>> consume
resultBody.map {
body =>
Logger.info(s"${requestHeader.method} ${requestHeader.uri}" +
s" took ${requestTime}ms and returned ${result.header.status}")
val jsonBody = Json.parse(body)
Logger.debug(s"Response\nHeader:\n${result.header.headers.toString}\nBody:\n${Json.prettyPrint(jsonBody)}")
}
result.withHeaders("Request-Time" -> requestTime.toString)
}
}
}
The end result will print the body as a Json String (pretty printed).
In the controller method that routes to the action, simply call
Map<String, String[]> params = request().queryString();
This will get you a map of parameters, where you can then call
params.get("someParam")[0]
to get the param (if it is a single value). If the param is a list, ignore the indexing andthat will return an array.

Concurrent.patchPanel not sending data from multiple enumerators - only sends from last enumerator added

I'm hoping someone else has used patchPanel to combine multiple enumerators together going down to a client over a websocket. The issue i'm running into is that the patchPanel is only sending the data feed from the last enumerator added into it.
I followed the example from; http://lambdaz.blogspot.ca/2012/12/play-21-multiplexing-enumerators-into.html which is the only reference I've been able to find regarding patchPanel.
Versions; play! 2.1.1 (using Java 1.7.0_11 and Scala 2.10.0)
The web socket method;
def monitorStream = WebSocket.async[JsValue] { request =>
val promiseIn = promise[Iteratee[JsValue, Unit]]
val out = Concurrent.patchPanel[JsValue] { patcher =>
val in = Iteratee.foreach[JsValue] { json =>
val event:Option[String] = (json \ "event").asOpt[String]
val systemId = (json \ "systemId").as[Long]
event.getOrElse("") match {
case "join" =>
val physicalSystem = SystemIdHandler.getById(systemId)
val monitorOut = (MonitorStreamActor.joinMonitor(physicalSystem))
monitorOut map { enum =>
val success = patcher.patchIn(enum)
}
}
}.mapDone { _ => Logger.info("Disconnected") }
promiseIn.success(in)
}
future(Iteratee.flatten(promiseIn.future),out)
}
The MonitorStreamActor call;
def joinMonitor(physicalSystem: PhysicalSystem):
scala.concurrent.Future[Enumerator[JsValue]]
= {
val monitorActor = ActorBase.akkaSystem.actorFor("/user/system-" + physicalSystem.name +"/stream")
(monitorActor ? MonitorJoin()).map {
case MonitorConnected(enumerator) =>
enumerator
}
}
The enumerator is returned fine, and the data fed into it is coming from a timer calling the actor. Actor definition, the timer hits the UpdatedTranStates case;
class MonitorStreamActor() extends Actor {
val (monitorEnumerator, monitorChannel) = Concurrent.broadcast[JsValue]
import play.api.Play.current
def receive = {
case MonitorJoin() => {
Logger.debug ("Actor monitor join")
sender ! MonitorConnected(monitorEnumerator)
}
case UpdatedTranStates(systemName,tranStates) => {
//println("Got updated Tran States")
val json = Json.toJson(tranStates.map(m => Map("State" -> m._1, "Count" -> m._2) ))
//println("Pushing updates to monitorChannel")
sendUpdateToClients(systemName, "states", json)
}
def sendUpdateToClients(systemName:String, updateType:String, json:JsValue) {
monitorChannel.push(Json.toJson(
Map(
"dataType"->Json.toJson(updateType),
"systemName" -> Json.toJson(systemName),
"data"->json)))
}
}
}
I've poked around for a while on this and haven't found a reason why only the last enumerator that is added into the patchPanel has the data sent. the API docs are not of much help, it really sounds like all you have to do is call patchIn and it should combine all enumerators to an iteratee, but that doesn't seem to be the case.
The PatchPanel by design replaces the current enumerator with the new one provided by the patchIn method.
In order to have multiple Enumerators combined together you need to use interleave or andThen methods to combine enumerators together. Interleave is preferred for this case as it will take events from each Enumerator as they are available, vs emptying one then moving to the next (as with andThen operator).
ie, in monitorStream;
val monitorOut = (MonitorStreamActor.joinMonitor(physicalSystem))
monitorOut map { enum =>
mappedEnums += ((physicalSystem.name, enum))
patcher.patchIn(Enumerator.interleave[JsValue]( mappedEnums.values.toSeq))
}
patcher is the patch panel, and mappedEnums is a HashMap[String,Enumerator[JsValue]] - re-add the patcher each time the Enumerators change (add or delete) - it works, not sure if it's the best way, but it'll do for now :)