How to get response headers and body in dispatch request? - scala

I want to get both body and headers from dispatch request. How to do this?
val response = Http(request OK as.String)
for (r <- response) yield {
println(r.toString) //prints body
// println(r.getHeaders) // ???? how to print headers here ????
}

We needed the response body of failed requests to an API, so we came up with this solution:
Define your own ApiHttpError class with code and body (for the body text):
case class ApiHttpError(code: Int, body: String)
extends Exception("Unexpected response status: %d".format(code))
Define OkWithBodyHandler similar to what is used in the source of displatch:
class OkWithBodyHandler[T](f: Response => T) extends AsyncCompletionHandler[T] {
def onCompleted(response: Response) = {
if (response.getStatusCode / 100 == 2) {
f(response)
} else {
throw ApiHttpError(response.getStatusCode, response.getResponseBody)
}
}
}
Now, near your call to the code that might throw and exception (calling API), add implicit override to the ToupleBuilder (again similar to the source code) and call OkWithBody on request:
class MyApiService {
implicit class MyRequestHandlerTupleBuilder(req: Req) {
def OKWithBody[T](f: Response => T) =
(req.toRequest, new OkWithBodyHandler(f))
}
def callApi(request: Req) = {
Http(request OKWithBody as.String).either
}
}
From now on, fetching either will give you the [Throwable, String] (using as.String), and the Throwable is our ApiHttpError with code and body.
Hope it helped.

From the Dispatch docs:
import dispatch._, Defaults._
val svc = url("http://api.hostip.info/country.php")
val country = Http(svc OK as.String)
The above defines and initiates a request to the given host where 2xx
responses are handled as a string. Since Dispatch is fully
asynchronous, country represents a future of the string rather than
the string itself.
(source)
In your example, the type of response is Future[String], because 'OK as.String' converts 2xx responses into strings and non-2xx responses into failed futures. If you remove 'OK as.String', you'll get a Future[com.ning.http.client.Response]. You can then use getResponseBody, getHeaders, etc. to inspect the Request object.

Related

How to wrap an REST API client in an Actor

I'm trying to write an actor that calls an HTTP REST API. The rest API needs a query parameter that will be passed from the invoking Actor. The official documentation has an example to achieve the above using a preStart method that pipes the message to itself:
import akka.actor.{ Actor, ActorLogging }
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.stream.{ ActorMaterializer, ActorMaterializerSettings }
import akka.util.ByteString
class Myself extends Actor
with ActorLogging {
import akka.pattern.pipe
import context.dispatcher
final implicit val materializer: ActorMaterializer = ActorMaterializer(ActorMaterializerSettings(context.system))
val http = Http(context.system)
override def preStart() = {
http.singleRequest(HttpRequest(uri = "http://akka.io"))
.pipeTo(self)
}
def receive = {
case HttpResponse(StatusCodes.OK, headers, entity, _) =>
entity.dataBytes.runFold(ByteString(""))(_ ++ _).foreach { body =>
log.info("Got response, body: " + body.utf8String)
}
case resp # HttpResponse(code, _, _, _) =>
log.info("Request failed, response code: " + code)
resp.discardEntityBytes()
}
}
The above works but the URL is hardcoded. What I want to achieve is a REST client actor that I can send parameters as a message to and get back results of the call. I modified the code above to receive parameters as message (pseudo code):
def receive = {
case param: RESTAPIParameter => {
http.singleRequest(HttpRequest(URI("http://my-rest-url").withQuery("name", "value"))
.pipeTo(self)
}
case HttpResponse(StatusCodes.OK, headers, entity, _) =>
entity.dataBytes.runFold(ByteString(""))(_ ++ _).foreach { body =>
log.info("Got response, body: " + body.utf8String)
sender! body.utf8String //Will not work
}
case resp # HttpResponse(code, _, _, _) =>
log.info("Request failed, response code: " + code)
resp.discardEntityBytes()
}
The above should work, but can't really be used to send a response back to the client as the sender reference is lost when the result of REST call is piped back to self.
I guess I could try and store the sender locally in a variable and use it to pass the response back, but I don't think that is a great idea.
So, what is the right way to handle such a scenario?
Edit: The solution suggested below by #PH88 works, but I would like to keep the pattern matching on HttpResponse in the outer loop.
Edit 2: The reason I wanted to pipe the response back to self is because I wanted to implement a state machine ..kind of. The state changes based on the type of message received by the actor. As an example:
The first state could be receiving a query string from a calling actor. The actor invokes the REST api and becomes awaitingResult. Data piped to self for further processing.
When it receives an HTTPResponse with success code, the state becomes dataRecevied. Data is piped to self again for more processing.
The received data is then transformed into internal vendor neutral format and the result is finally sent back to the calling actor.
If the Response code is not successful in 1 above, the state could be changed to HttpError and handled accordingly.
Hope that clarifies the intent. Any other suggestions/designs to achieve a clean/simple design are welcome :-)
You can wrap the HttpResponse with your own case class and bundle the sender with it:
case class ServerResponse(requester: ActorRef, resp: HttpResponse)
then:
def receive = {
case param: RESTAPIParameter => {
val requester = sender
http.singleRequest(HttpRequest(URI("http://my-rest-url").withQuery("name", "value"))
.map(httpResp =>
// This will execute in some other thread, thus
// it's important to NOT use sender directly
ServerResponse(requester, httpResp)
)
.pipeTo(self)
}
case ServerResponse(requester, HttpResponse(...)) =>
val result = ...
requester ! result
...
}
The pipeTo method takes in the sender as an implicit argument:
def pipeTo(recipient: ActorRef)(implicit sender: ActorRef = Actor.noSender): Future[T]
Within an Actor self is defined as an implicit ActorRef:
implicit final val self: ActorRef
Therefore self is the sender that is being specified in pipeTo.
You can simply specify the sender explicitly to be the original sender:
def receive = {
case param: RESTAPIParameter =>
http.singleRequest(HttpRequest(URI("http://my-rest-url").withQuery("name", "value"))
.pipeTo(self)(sender) //specify original sender
case HttpResponse(StatusCodes.OK, headers, entity, _) =>
entity.dataBytes.runFold(ByteString(""))(_ ++ _).foreach { body =>
log.info("Got response, body: " + body.utf8String)
sender! body.utf8String //Works now!
}
case resp # HttpResponse(code, _, _, _) => {
log.info("Request failed, response code: " + code)
resp.discardEntityBytes()
}
}

Handling Future[WSResponse] to find success or error state

In Scala I have a call to service in controller which is returning me Future[WSResponse]. I want to make sure service is returning valid result so send Ok(..) otherwise send BadRequest(...). I don't think I can use map. Any other suggestion?
def someWork = Action.async(parse.xml) { request =>
val result:Future[WSResponse] = someService.processData(request.body.toString())
//Need to send back Ok or BadRequest Message
}
EDIT
Solution from #alextsc is working fine. Now moving to test my existing test is failing. It is getting 400 instead of 200.
test("should post something") {
val requestBody = <value>{UUID.randomUUID}</value>
val mockResponse = mock[WSResponse]
val expectedResponse: Future[WSResponse] = Future.successful(mockResponse)
val request = FakeRequest(Helpers.POST, "/posthere").withXmlBody(requestBody)
when(mockResponse.body).thenReturn("SOME_RESPONSE")
when(someService.processData(any[String])).thenReturn(expectedResponse)
val response: Future[Result] = call(controller.someWork , request)
whenReady(response) { response =>
assert(response.header.status == 200)
}
}
You're on the right track and yes, you can use map.
Since you're using Action.async already and your service returns a future as it stands all you need to do is map that future to a Future[Result] so Play can handle it:
def someWork = Action.async(parse.xml) { request =>
someService.processData(request.body.toString()).map {
// Assuming status 200 (OK) is a valid result for you.
case resp : WSResponse if resp.getStatus == 200 => Ok(...)
case _ => BadRequest(...)
}
}
(I note that your service returns WSResponse (from the play ws java library) and not play.api.libs.ws.Response (the scala version of it), hence getStatus and not just status)

Akka Stream + Akka Http - Get Request on Error

I have the following stream that works pretty well:
source
.map(x => HttpRequest(uri = x.rawRequest))
.via(Http().outgoingConnection(host, port))
.to(Sink.actorRef(myActor, IsDone))
.run()
and a simple actor to handle the response status and the final message when the stream completes:
/**
* A simple actor to count how many rows have been processed
* in the complete process given a http status
*
* It also finish the main thread upon a message of type [[IsDone]] is received
*/
class MyActor extends Actor with ActorLogging {
var totalProcessed = 0
def receive = LoggingReceive {
case response: HttpResponse =>
if(response.status.isSuccess()) {
totalProcessed = totalProcessed + 1
} else if(response.status.isFailure()) {
log.error(s"Http response error: ${response.status.intValue()} - ${response.status.reason()}")
} else {
log.error(s"Error: ${response.status.intValue()} - ${response.status.reason()}")
}
case IsDone =>
println(s"total processed: $totalProcessed")
sys.exit()
}
}
case object IsDone
I don't know if this is the best approach to count things and also to handle response status, but it's working so far.
The question is how to pass the original request to the actor in a way I could know what request caused a specific error.
My actor could expect the following instead:
case (request: String, response: HttpResponse) =>
But how to pass that information that I have at the beginning of my pipeline?
I was thinking of to map like this:
source
.map(x => (HttpRequest(uri = x.rawRequest), x.rawRequest))
But I have no idea on how to fire the Http flow.
Any suggestion?
With #cmbaxter help, I could solve my problem using the following piece of code:
val poolClientFlow = Http().cachedHostConnectionPool[String](host, port)
source
.map(url => HttpRequest(uri = url) -> url)
.via(poolClientFlow)
.to(Sink.actorRef(myActor, IsDone))
.run()
Now my actor is able to receive this:
case (respTry: Try[HttpResponse], request: String) =>

Asynchronous http requests using Netty and Scala actors

Asynchronous http requests using Netty and Scala actors
Hey hope someone can give me a hand with this.
I am trying to use the Scala Actors and Netty.io libraries to get make asynchronous http requests. (Yes I know Scala actors are being deprecated but this is a learning exercise for me)
I have written an actor HttpRequestActor that accepts a message in the form of a case class RequestPage(uri:URI).
When it receives the message it creates the necessary Netty objects need to make a http request, I have based most of the code from the [HttpSnoopClient] (http://static.netty.io/3.5/xref/org/jboss/netty/example/http/snoop/HttpSnoopClient.html) example.
I create a client and pass the current actor instance to my implementation of ChannelPipelineFactory which also passes the actor to my implementation of SimpleChannelUpstreamHandler, where I have overridden the messageReceived function.
The actor instance is passed as a listener, I create a request using the DefaultHttpRequest class and write to the channel to make the request.
There is a blocking call to an actor object using the ChannelFuture object returned from writing to the channel. When the messageRecieved function of my handler class is called I parse the response of the netty http request as a string, send a message back to actor with the content of the response and close the channel.
After the future is completed my code attempts to send a reply to the calling actor with the http content response received.
The code works, and I am able to get a reply, send it to my actor instance, print out the content and send a message to the actor instance release resources being used.
Problem is when I test it, the original call to the actor does not get a reply and the thread just stays open.
Code Sample - HttpRequestActor
my code for my HttpRequestActor class
import scala.actors.Actor
import java.net.{InetSocketAddress,URI}
import org.jboss.netty.handler.codec.http._
import org.jboss.netty.bootstrap.ClientBootstrap
import org.jboss.netty.channel.Channel
import org.jboss.netty.channel._
import org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory
import org.jboss.netty.channel.group.DefaultChannelGroup
import java.util.concurrent.{Executors,CancellationException}
import org.jboss.netty.util.CharsetUtil
import scala.concurrent.{ Promise, Future }
import scala.concurrent.ExecutionContext.Implicits.global
/**
* #author mebinum
*
*/
class HttpRequestActor extends Actor {
//initialize response with default uninitialized value
private var resp:Response = _
private val executor = Executors.newCachedThreadPool
private val executor2 = Executors.newCachedThreadPool
private val factory = new NioClientSocketChannelFactory(
executor,
executor2);
private val allChannels = new DefaultChannelGroup("httpRequester")
def act = loop {
react {
case RequestPage(uri) => requestUri(uri)
case Reply(msg) => setResponse(Reply(msg))
case NoReply => println("didnt get a reply");setResponse(NoReply)
case NotReadable => println("got a reply but its not readable");setResponse(NotReadable)
case ShutDown => shutDown()
}
}
private def requestUri(uri:URI) = {
makeChannel(uri) map {
channel => {
allChannels.add(channel)
val request = new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, uri.toString)
request.setHeader(HttpHeaders.Names.HOST, uri.getHost())
request.setHeader(HttpHeaders.Names.CONNECTION, HttpHeaders.Values.CLOSE)
request.setHeader(HttpHeaders.Names.ACCEPT_ENCODING, HttpHeaders.Values.GZIP)
val writeFuture = channel.write(request).awaitUninterruptibly()
FutureReactor !? writeFuture match {
case future : ChannelFuture => {
future.addListener(new ChannelFutureListener() {
def operationComplete(future:ChannelFuture) {
// Perform post-closure operation
println("current response is " + resp)
sendResponse("look ma I finished")
}
})
future.getChannel().close()
}
}
this ! ShutDown
}
}
//thread ends only if you send a reply from here
//println("this is final sender " + sender)
//reply("I am the true end")
}
private def makeChannel(uri:URI) = {
val scheme = Some(uri.getScheme()).getOrElse("http")
val host = Some(uri.getHost()).getOrElse("localhost")
val port = Utils.getPort(uri.getPort, uri.getScheme)
// Set up the event pipeline factory.
val client = new ClientBootstrap(factory)
client.setPipelineFactory(new PipelineFactory(this))
//get the promised channel
val channel = NettyFutureBridge(client.connect(new InetSocketAddress(host, port)))
channel
}
private def setResponse(aResponse:Response) = resp = aResponse
private def sendResponse(msg:String) = {
println("Sending the response " + msg)
reply(resp)
}
private def shutDown() = {
println("got a shutdown message")
val groupFuture = allChannels.close().awaitUninterruptibly()
factory.releaseExternalResources()
}
override def exceptionHandler = {
case e : CancellationException => println("The request was cancelled"); throw e
case tr: Throwable => println("An unknown exception happened " + tr.getCause()); throw tr
}
}
trait Response
case class RequestPage(url:URI)
case class Reply(content:String) extends Response
case object NoReply extends Response
case object NotReadable extends Response
case object ShutDown
object FutureReactor extends Actor{
def act = //loop {
react {
case future: ChannelFuture => {
if (future.isCancelled) {
throw new CancellationException()
}
if (!future.isSuccess()) {
future.getCause().printStackTrace()
throw future.getCause()
}
if(future.isSuccess() && future.isDone()){
future.getChannel().getCloseFuture().awaitUninterruptibly()
reply(future)
}
}
}
//}
this.start
}
class ClientHandler(listener:Actor) extends SimpleChannelUpstreamHandler {
override def exceptionCaught( ctx:ChannelHandlerContext, e:ExceptionEvent){
e.getCause().printStackTrace()
e.getChannel().close();
throw e.getCause()
}
override def messageReceived(ctx:ChannelHandlerContext, e:MessageEvent) = {
var contentString = ""
var httpResponse:Response = null.asInstanceOf[Response]
e.getMessage match {
case (response: HttpResponse) if !response.isChunked => {
println("STATUS: " + response.getStatus);
println("VERSION: " + response.getProtocolVersion);
println
val content = response.getContent();
if (content.readable()) {
contentString = content.toString(CharsetUtil.UTF_8)
httpResponse = Reply(contentString)
//notify actor
}else{
httpResponse = NotReadable
}
}
case chunk: HttpChunk if !chunk.isLast => {
//get chunked content
contentString = chunk.getContent().toString(CharsetUtil.UTF_8)
httpResponse = Reply(contentString)
}
case _ => httpResponse = NoReply
}
println("sending actor my response")
listener ! httpResponse
println("closing the channel")
e.getChannel().close()
//send the close event
}
}
class PipelineFactory(listener:Actor) extends ChannelPipelineFactory {
def getPipeline(): ChannelPipeline = {
// Create a default pipeline implementation.
val pipeline = org.jboss.netty.channel.Channels.pipeline()
pipeline.addLast("codec", new HttpClientCodec())
// Remove the following line if you don't want automatic content decompression.
pipeline.addLast("inflater", new HttpContentDecompressor())
// Uncomment the following line if you don't want to handle HttpChunks.
//pipeline.addLast("aggregator", new HttpChunkAggregator(1048576))
pipeline.addLast("decoder", new HttpRequestDecoder())
//assign the handler
pipeline.addLast("handler", new ClientHandler(listener))
pipeline;
}
}
object NettyFutureBridge {
import scala.concurrent.{ Promise, Future }
import scala.util.Try
import java.util.concurrent.CancellationException
import org.jboss.netty.channel.{ Channel, ChannelFuture, ChannelFutureListener }
def apply(nettyFuture: ChannelFuture): Future[Channel] = {
val p = Promise[Channel]()
nettyFuture.addListener(new ChannelFutureListener {
def operationComplete(future: ChannelFuture): Unit = p complete Try(
if (future.isSuccess) {
println("Success")
future.getChannel
}
else if (future.isCancelled) {
println("Was cancelled")
throw new CancellationException
}
else {
future.getCause.printStackTrace()
throw future.getCause
})
})
p.future
}
}
Code to test it
val url = "http://hiverides.com"
test("Http Request Actor can recieve and react to message"){
val actor = new HttpRequestActor()
actor.start
val response = actor !? new RequestPage(new URI(url))
match {
case Reply(msg) => {
println("this is the reply response in test")
assert(msg != "")
println(msg)
}
case NoReply => println("Got No Reply")
case NotReadable => println("Got a not Reachable")
case None => println("Got a timeout")
case s:Response => println("response string \n" + s)
case x => {println("Got a value not sure what it is"); println(x);}
}
}
Libraries used:
- Scala 2.9.2
- Netty.io 3.6.1.Final
- Junit 4.7
- scalatest 1.8
- I am also using #viktorklang NettyFutureBridge object gist to create a scala future for the Channel object returned
How can I send a reply back to the actor object with the content of response from Netty and end the thread?
Any help will be much appreciated
I don't know Scala, but I had a similar issue. Try specifying the content-length header of the response.
In plain java:
HttpRequest r = new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, uri);
ChannelBuffer buffer = ChannelBuffers.copiedBuffer(input);
r.setHeader(HttpHeaders.Names.HOST, "host");
r.setHeader(HttpHeaders.Names.CONTENT_TYPE, "application/octet-stream");
r.setHeader(HttpHeaders.Names.CONTENT_LENGTH, buffer.readableBytes());
r.setHeader(HttpHeaders.Names.CONNECTION, HttpHeaders.Values.CLOSE);
r.setContent(buffer);
Otherwise the server has no idea when the content is completed from the client, unless the client closes the connection.
You can also use chunked encoding, but you'll have to implement the chunk encoding yourself (At least I don't know of a library in Netty that does it).

Return exact response/header?

From the client-side of a webapp, I hit a server-side route which is just a wrapper for a third-party API. Using dispatch, I am trying to make that server-side request return the exact header and response of the third-party API to the client-side AJAX call.
When I do this:
val req = host("third-pary.api.com, 80)
val post = req.as("user", "pass") / "route" << Map("key" -> "akey", "val" -> "aval")
Http(post > as.String)
I always see a 200 response returned to the AJAX call (kind of expectedly). I have seen an Either syntax used, but I'm really more of an Any, as it's just the exact response and header. How would this be written?
I should mention I'm using Scalatra on the server-side, so the local route is:
post("/route") {
}
EDIT:
Here is the suggested Either matching example, which I'm playing with, but the match syntax doesn't make sense - I don't care if there is an error, I just want to return it. Also, I can't seem to get the BODY returned with this method.
val asHeaders = as.Response { response =>
println("BODY: " + response.getResponseBody())
scala.collection.JavaConverters.mapAsScalaMapConverter(
response.getHeaders).asScala.toMap.mapValues(_.asScala.toList)
}
val response: Either[Throwable, Map[String, List[String]]] =
Http(post > asHeaders).either()
response match {
case Left(wrong) =>
println("Left: " + wrong.getMessage())
// return Action with header + body
case Right(good) =>
println("Right: " + good)
// return Action with header + body
}
Ideally, the solutions returns the Scalatra ActionResult(responseStatus(status, reason), body, headers).
It's actually very easy to get response headers while using Dispatch. For example with 0.9.4:
import dispatch._
import scala.collection.JavaConverters._
val headers: java.util.Map[String, java.util.List[String]] = Http(
url("http://www.google.com")
)().getHeaders
And now, for example:
scala> headers.asScala.mapValues(_.asScala).foreach {
| case (k, v) => println(k + ": " + v)
| }
X-Frame-Options: Buffer(SAMEORIGIN)
Transfer-Encoding: Buffer(chunked)
Date: Buffer(Fri, 30 Nov 2012 20:42:45 GMT)
...
If you do this often it's better to encapsulate it, like this, for example:
val asHeaders = as.Response { response =>
scala.collection.JavaConverters.mapAsScalaMapConverter(
response.getHeaders
).asScala.toMap.mapValues(_.asScala.toList)
}
Now you can write the following:
val response: Either[Throwable, Map[String, List[String]]] =
Http(url("http://www.google.com") OK asHeaders).either()
And you've got error checking, nice immutable collections, etc.
We needed the response body of failed requests to an API, so we came up with this solution:
Define your own ApiHttpError class with code and body (for the body text):
case class ApiHttpError(code: Int, body: String)
extends Exception("Unexpected response status: %d".format(code))
Define OkWithBodyHandler similar to what is used in the source of displatch:
class OkWithBodyHandler[T](f: Response => T) extends AsyncCompletionHandler[T] {
def onCompleted(response: Response) = {
if (response.getStatusCode / 100 == 2) {
f(response)
} else {
throw ApiHttpError(response.getStatusCode, response.getResponseBody)
}
}
}
Now, near your call to the code that might throw and exception (calling API), add implicit override to the ToupleBuilder (again similar to the source code) and call OkWithBody on request:
class MyApiService {
implicit class MyRequestHandlerTupleBuilder(req: Req) {
def OKWithBody[T](f: Response => T) =
(req.toRequest, new OkWithBodyHandler(f))
}
def callApi(request: Req) = {
Http(request OKWithBody as.String).either
}
}
From now on, fetching either will give you the [Throwable, String] (using as.String), and the Throwable is our ApiHttpError with code and body.
Hope it helped.