Log all requests in Play 2 - scala

How can I log all requests coming in to a Play 2 application, in a similar way to how Apache logs (i.e. including the URL and client IP)?

I wanted to do the same thing with with Play 2.5.4. It took me a little while to put the pieces together, so I thought I'd share my steps in the hope that it'll save time for someone else:
Make sure you have an access logger. There's an example of how to configure one at https://www.playframework.com/documentation/2.5.x/SettingsLogger
but you might want to play with the settings; I put configured my logger with <file>${application.home:-.}/logs/access.log</file>, and <immediateFlush>true</immediateFlush>.
Create a new RequestHandler in your root package, extending the default handler:
import javax.inject.Inject
import play.api.http._
import play.api.routing._
import play.api.mvc.RequestHeader
import play.api.Logger
/**
* Implemented to get custom, apache-style logging of requests without dumping the full netty wire.
*/
class RequestHandler #Inject() (router: Router, errorHandler: HttpErrorHandler,
configuration: HttpConfiguration, filters: HttpFilters) extends DefaultHttpRequestHandler(
router, errorHandler, configuration, filters) {
override def routeRequest(request: RequestHeader) = {
Logger("access").info(s"Request from ${request.remoteAddress}: ${request}")
super.routeRequest(request)
}
}
I was coming from Play 2.3, so I was originally planning to use GlobalSettings until I found this guide: https://www.playframework.com/documentation/2.5.x/GlobalSettings
Apologies to anyone who wants answers specific to Play 2.0, but seeing as my own 2.5-focused searches led me here, I figure this answer won't do much harm.

That's what http filters are for. Here are some detailed examples: http://www.playframework.com/documentation/2.1.1/ScalaHttpFilters

In Play 2.5.x, I used the following
import javax.inject.Inject
import akka.stream.Materializer
import play.api.Logger
import play.api.mvc._
import scala.concurrent.{ExecutionContext, Future}
import java.util.Calendar
import java.text.SimpleDateFormat
class ApacheLoggingFilter #Inject() (implicit val mat: Materializer, ec: ExecutionContext) extends Filter {
def apply(nextFilter: RequestHeader => Future[Result])
(requestHeader: RequestHeader): Future[Result] = {
nextFilter(requestHeader).map { result =>
val responseSize = result.body.contentLength.getOrElse("-")
Logger("access").info(s"""${requestHeader.remoteAddress} - - [${serverTime}] "${requestHeader}" ${result.header.status} ${responseSize}""")
result
}
}
private def serverTime = {
val calendar = Calendar.getInstance()
val dateFormat = new SimpleDateFormat(
"dd/MMM/yyyy:HH:mm:ss Z")
dateFormat.setTimeZone(calendar.getTimeZone)
dateFormat.format(calendar.getTime())
}
}
Make sure you configure this Filter correctly - https://www.playframework.com/documentation/2.5.x/ScalaHttpFilters#Using-filters

Create your own Action is far more powerful and flexible.
object MyAction {
def apply[A](bodyParser: BodyParser[A])(block: Request[A] => Result): Action[A] = Action(bodyParser) {
request =>
// TODO : authentication, cache logics here
// time it
val start = ...
// process
val r = block(request)
val end = ...
// log remote address, user agent, time, etc.
r
}
// simply override to use MyAction
def apply(block: Request[AnyContent] => Result): Action[AnyContent] = this.apply(BodyParsers.parse.anyContent)(block)
// simply override to use MyAction
def apply(block: => Result): Action[AnyContent] = this.apply(_ => block)
}
in order to use it in controller, simply replace Action with MyAction
def index = MyAction {
implicit request =>
// nothing to be changed here
}

Related

items fail to be processed in Akka streams app that uses Source.queues and Sink.queues in a flow

I am trying to create an (Akka HTTP) stream procsesing flow using the classes akka.stream.scaladsl.Source and Sink queues.
I am using a queue because I have a processing step in my flow that issues http requests and I want this step to take as many
items off the queue as there are max-open-requests, and stop taking off the queue once max-open-requests are in flight.
The result is that backpressure is applied when my connection pool is overloaded.
Below, I have a very simplified test that reflects the main logic of my app. In the test 'Stress Spec' (below)
I am simulating a number of simultaneous connections via which I will send a 'Source' of 'Requesto' objects
to the getResponses method of the class ServiceImpl.
In the processing step 'pullOffSinkQueue' you will note that I am incrementing a counter to see how many items
I have pulled off the queue.
The test will send Serviceimpl a set of requests whose cardinality is set to equal
streamedRequestsPerConnection * numSimultaneousConnections.
When I send 20 requests my test passes fine. In particular the count of requests pulled off the
Sink.queue will be equal to the number of requests I send out. However, if
I increase the number of requests I send to above 50 or so, I see consistent failures in the test.
I get a message such as the one below
180 was not equal to 200
ScalaTestFailureLocation: com.foo.StressSpec at (StressSpec.scala:116)
Expected :200
Actual :180
<Click to see difference>
This indicates that the number of items pulled off the queue does not equal the number of items put on the queue.
I have a feeling this might be due to the fact that my test is not properly waiting for all items put into the stream
to be processed. If anyone has any suggestions, I'd be all ears ! Code is below.
package com.foo
import java.util.concurrent.atomic.AtomicInteger
import akka.stream.ActorAttributes.supervisionStrategy
import akka.stream.{Attributes, Materializer, QueueOfferResult}
import akka.stream.Supervision.resumingDecider
import akka.stream.scaladsl.{Flow, Keep, Sink, Source}
import scala.concurrent.{ExecutionContext, Future}
import akka.NotUsed
import akka.actor.ActorSystem
import akka.event.{Logging, LoggingAdapter}
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{Sink, Source}
import org.scalatest.mockito.MockitoSugar
import org.scalatest.{FunSuite, Matchers}
import scala.collection.immutable
import scala.concurrent.duration._
import scala.concurrent.{Await, Future, _}
final case class Responso()
final case class Requesto()
object Handler {
val dbRequestCounter = new AtomicInteger(0)
}
class Handler(implicit ec: ExecutionContext, mat: Materializer) {
import Handler._
private val source =
Source.queue[(Requesto, String)](8, akka.stream.OverflowStrategy.backpressure)
private val sink =
Sink.queue[(Requesto, String)]().withAttributes(Attributes.inputBuffer(8, 8))
private val (sourceQueue, sinkQueue) = source.toMat(sink)(Keep.both).run()
def placeOnSourceQueue(ar: Requesto): Future[QueueOfferResult] = {
sourceQueue.offer((ar, "foo"))
}
def pullOffSinkQueue(qofr: QueueOfferResult): Future[Responso] = {
dbRequestCounter.incrementAndGet()
qofr match {
case QueueOfferResult.Enqueued =>
sinkQueue.pull().flatMap { maybeRequestPair: Option[(Requesto, String)] =>
Future.successful(Responso())
}
case error =>
println("enqueuing error: " + error)
Future.failed(new RuntimeException("enqueuing error: " + error))
}
}
}
class ServiceImpl(readHandler: Handler, writeHandler: Handler)
(implicit log: LoggingAdapter, mat: Materializer) {
private val readAttributeFlow: Flow[Requesto, Responso, NotUsed] = {
Flow[Requesto]
.mapAsyncUnordered(1)(readHandler.placeOnSourceQueue)
.mapAsyncUnordered(1)(readHandler.pullOffSinkQueue)
}
def getResponses(request: Source[Requesto, NotUsed]): Source[Responso, NotUsed] =
request
.via(readAttributeFlow)
.withAttributes(supervisionStrategy(resumingDecider))
}
class StressSpec
extends FunSuite
with MockitoSugar
with Matchers {
val streamedRequestsPerConnection = 10
val numSimultaneousConnections = 20
implicit val actorSystem: ActorSystem = ActorSystem()
implicit val materializer: ActorMaterializer = ActorMaterializer()
implicit val log: LoggingAdapter = Logging(actorSystem.eventStream, "test")
implicit val ec: ExecutionContext = actorSystem.dispatcher
import Handler._
lazy val requestHandler = new Handler()
lazy val svc: ServiceImpl =
new ServiceImpl(requestHandler, requestHandler)
test("can handle lots of simultaneous read requests") {
val totalExpected = streamedRequestsPerConnection * numSimultaneousConnections
def sendRequestAndAwaitResponse(): Unit = {
def getResponses(i: Integer) = {
val requestStream: Source[Requesto, NotUsed] =
Source(1 to streamedRequestsPerConnection)
.map { i =>
Requesto()
}
svc.getResponses(requestStream).runWith(Sink.seq)
}
val responses: immutable.Seq[Future[immutable.Seq[Responso]]] =
(1 to numSimultaneousConnections).map { getResponses(_) }
val flattenedResponses: Future[immutable.Seq[Responso]] =
Future.sequence(responses).map(_.flatten)
Await.ready(flattenedResponses, 1000.seconds).value.get
}
sendRequestAndAwaitResponse()
dbRequestCounter.get shouldBe(totalExpected)
}
}

how to mock external WS API calls in Scala Play framework

I have an existing Scala play application which has a REST API that calls another external REST API. I want to mock the external Web service returning fake JSON data for internal tests. Based on example from: https://www.playframework.com/documentation/2.6.x/ScalaTestingWebServiceClients
I followed example exactly as in Documentation and I'm getting compiler errors due to deprecated class Action.
import play.core.server.Server
import play.api.routing.sird._
import play.api.mvc._
import play.api.libs.json._
import play.api.test._
import scala.concurrent.Await
import scala.concurrent.duration._
import org.specs2.mutable.Specification
import product.services.market.common.GitHubClient
class GitHubClientSpec extends Specification {
import scala.concurrent.ExecutionContext.Implicits.global
"GitHubClient" should {
"get all repositories" in {
Server.withRouter() {
case GET(p"/repositories") => Action {
Results.Ok(Json.arr(Json.obj("full_name" -> "octocat/Hello-World")))
}
} { implicit port =>
WsTestClient.withClient { client =>
val result = Await.result(
new GitHubClient(client, "").repositories(), 10.seconds)
result must_== Seq("octocat/Hello-World")
}
}
}
}
}
object Action in package mvc is deprecated: Inject an ActionBuilder
(e.g. DefaultActionBuilder) or extend
BaseController/AbstractController/InjectedController
And this is the primary example from latest official docs which in fact contains a compile time error, given this example doesn't work how should be the proper way to easily mock an external API using Scala Play?
You may change your example to:
Server.withRouterFromComponents() { cs => {
case GET(p"/repositories") => cs.defaultActionBuilder {
Results.Ok(Json.arr(Json.obj("full_name" -> "octocat/Hello-World")))
}
}
} { implicit port =>
WsTestClient.withClient { client =>
val result = Await.result(
new GitHubClient(client, "").repositories(), 10.seconds)
result should be(Seq("octocat/Hello-World"))
}
}
To be honest, I'm not 100% sure if this is the nicest way. However I have submitted a PR to the play framework so you might watch that space for comments from the makers.
If you're using standalone version of play-ws you can use this library https://github.com/f100ded/play-fake-ws-standalone like this
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import org.f100ded.play.fakews._
import org.scalatest._
import play.api.libs.ws.JsonBodyWritables._
import scala.concurrent.duration.Duration
import scala.concurrent._
import scala.language.reflectiveCalls
/**
* Tests MyApi HTTP client implementation
*/
class MyApiClientSpec extends AsyncFlatSpec with BeforeAndAfterAll with Matchers {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
import system.dispatcher
behavior of "MyApiClient"
it should "put access token to Authorization header" in {
val accessToken = "fake_access_token"
val ws = StandaloneFakeWSClient {
case request # GET(url"http://host/v1/foo/$id") =>
// this is here just to demonstrate how you can use URL extractor
id shouldBe "1"
// verify access token
request.headers should contain ("Authorization" -> Seq(s"Bearer $accessToken"))
Ok(FakeAnswers.foo)
}
val api = new MyApiClient(ws, baseUrl = "http://host/", accessToken = accessToken)
api.getFoo(1).map(_ => succeed)
}
// ... more tests
override def afterAll(): Unit = {
Await.result(system.terminate(), Duration.Inf)
}
}

scala play framework how to unit test async controllers

Using Scala play version 2.5 and trying to follow the guidelines for unit testing controllers as per documentation at: https://www.playframework.com/documentation/2.5.x/ScalaTestingWithScalaTest
there's no example for unit testing an async controller.
I'm trying to create a unit test for my controller which has an async action method, I ended up mocking some objects
class ProductController #Inject()(
action: ProductAction,
handler: ProductResourceHandler)(implicit ec: ExecutionContext)
extends Controller {
/**
* Fetch a list of products
*/
def index: Action[AnyContent] = {
action.async { implicit request =>
handler.find.map { list =>
Ok(Json.toJson(list))
}
}
}
// ...
}
My unit test:
import scala.concurrent.Future
import org.scalatestplus.play._
import play.api.mvc._
import play.api.test._
import play.api.test.Helpers._
import org.scalatest.mockito.MockitoSugar
import product.ProductAction
import product.ProductController
import product.services.maps.GeolocationService
import product.ProductResourceHandler
import play.api.libs.concurrent.Execution.Implicits._
import scala.io.Source
import play.api.libs.json.Json
import product.model.OfferList
import product.model.OfferDetail
import org.mockito.Mockito._
class ProductControllerSpec extends PlaySpec with Results with MockitoSugar {
private val productList = Json.parse(Source.fromFile("conf/app/sample_products.json").getLines.mkString).as[ProductList]
"Example Page#index" should {
"should be valid" in {
val action = new ProductAction()
val handler = mock[ProductResourceHandler]
when(handler.find) thenReturn Future.successful(productList)
val controller = new ProductController(action, handler)
val result: Future[Result] = controller.index().apply(FakeRequest())
val bodyText: String = contentAsString(result)
bodyText != null mustBe true
}
}
}
up till now it's working but I'm wondering if this follows the best practices or guidelines for this type of test. Is this the right way to unit test an async controller in Scala play framework?
Some of my suggestions are
use contentAsJson instead of contentAsString and inspect the returned json.
use route to directly invoke the controller and test response (for eg route(app, FakeRequest..)
use status method to check if the returned status is HTTP OK (status code 200)
val Some(result) = route(app, FakeRequest(GET,
controllers.routes. ProductController.index.path()))
status(result) must be (OK)
val json = contentAsJson(result)
// inspect json fields like if you have to check if the json
// has string field called id you can do (json \ "id").as[String] must be ("<id value>")
According to Play documentation:
Play actions are asynchronous by default.
This means even if you are not using Action.async { Future { myAnonymousFunction } } but just Action { myAnonymousFunction }, internally the result of myAnonymousFunction will be enclosed in a Future.
For instance, say you have
class HelloWorld extends Controller {
def index = Action { request => Ok("") }
}
then
(new HelloWorld).index().apply(FakeRequest())
still has type
Future[Result]
This leads me to believe that your unit test is indeed appropriate way of testing controllers, that is, Play's documentation is implicitly covering also the case of Action.async.

Composing and outputting two future asynchronous actions on screen

I hate to ask - I really do but this one has got me for the moment..
I'm trying to compose some actions (in Play Framework & scala) with my main guide being this vid. However it was made a few years back so some of the functionality has since been deprecated and therefore I have had to find work-arounds as I go. Currently I am trying to output two asynchronous actions within some HTML markup.
I successfully outputted one action with this controller:
package controllers
import akka.actor.ActorSystem
import javax.inject._
import play.api.mvc._
import services.ServiceClient
import scala.concurrent.ExecutionContext
#Singleton
class AsyncController #Inject() (sc: ServiceClient)(actorSystem: ActorSystem)(implicit exec: ExecutionContext) extends Controller {
def index = Action.async { request =>
val asy1 = sc.makeServiceCall("async1")
for {
async1Message <- asy1
} yield {
Ok(views.html.async1.async1(async1Message))
}
}
}
In case you are wondering the sc.makeServiceCall refers to this file:
class ServiceClient #Inject() (ws: WSClient) {
def makeServiceCall(serviceName: String): Future[String] = {
ws.url(s"http://localhost:9000/mock/$serviceName").get().map(_.body)
}
}
So I followed the video in its' guidance to compose two asynchronous actions with some HTML. And this is where it gets difficult/interesting/upsetting:
package controllers
import javax.inject.Inject
import akka.actor.ActorSystem
import play.api.mvc._
import scala.concurrent.{ExecutionContext}
import Ui.Pagelet
class AsyncHomeController #Inject() (as1: AsyncController)(as2: Async2Controller)(actorSystem: ActorSystem)(implicit exec: ExecutionContext) extends Controller {
def index = Action.async { request =>
val asy1 = as1.index(request)
val asy2 = as2.index(request)
for {
async1Result <- asy1
async2Result <- asy2
async1Body <- Pagelet.readBody(async1Result)
async2Body <- Pagelet.readBody(async2Result)
} yield {
Ok(views.html.home2(async1Body, async2Body))
}
}
}
So Async2Controller is very similar to AsyncController and Pagelet.readBody refers to this:
package Ui
import play.api.libs.iteratee.Iteratee
import play.api.mvc.{Codec, Result}
import play.twirl.api.Html
import scala.concurrent._
object Pagelet {
def readBody(result: Result)(implicit codec: Codec): Future[Html] = {
result.body.run(Iteratee.consume()).map(bytes => Html(new String(bytes, codec.charset)))
}
}
And this is wherein the error lies - which is:
value run is not a member of play.api.http.HttpEntity
I cannot find documentation on whether it needs to be injected or any indication that it has since been deprecated. If someone has got an answer to this or a work-around please divulge. Many thanks
The Iteratee lib is deprecated and was replaced by akka-stream. You need to change the implementation of readBody:
def readBody(result: Result)(implicit mat: Materializer, ec: ExecutionContext, codec: Codec): Future[Html] = {
result.body.consumeData.map(byteString => Html(codec.decode(byteString))
}
You also need to change the dependencies of the controller to get a Materializer:
class AsyncHomeController #Inject() (as1: AsyncController, as2: Async2Controller)(actorSystem: ActorSystem)(implicit exec: ExecutionContext, mat: Materializer)
Edit: code updated

PlayFramework - Executing query in DBAction

I'm using Play 2.3.8 and Slick 2.1.0 and according to the docs, I should be able to execute queries in my controller actions like this:
val customers = TableQuery[Customers]
def index = DBAction { implicit request =>
val result = Json.toJson(customers.list)
Ok(result)
}
However, when I try this I get an error - no implicit session in scope. I can get around it by doing this:
val customers = TableQuery[Customers]
def index = DBAction { implicit request =>
implicit val session = request.dbSession
val result = Json.toJson(customers.list)
Ok(result)
}
Is this what's required or is there a simpler way? Thanks!
You can avoid that implicit using this imports (PoC):
import play.api.mvc._
import play.api.Play.current
import play.api.db.slick._
import play.api.db.slick.Config.driver.simple._
...
object Products extends Controller {
def all = DBAction { implicit rs =>
Ok(Json.toJson(products.list))
}
}