Scala Dispatch library: how to handle connection failure or timeout? - scala

I've been using the Databinder Dispatch library in a client for a simple REST-ish API. I know how to detect if I get an HTTP response with an error status:
Http x (request) {
case (200, _, _, content) => successResult(content())
case (404, _, _, _) => notFoundErrorResult
case (_, _, _, _) => genericErrorResult
}
But how can I distinguish an error response from a failure to get any response at all, because of an invalid domain or failure to connect? And is there any way to implement a timeout while still using synchronous semantics? If there's anything relevant in the API, I've missed it.

There is also a more elegant way to configure client using Http.configure method which receives Builder => Builder function as an argument:
val http = Http.configure(_.setAllowPoolingConnection(true).setConnectionTimeoutInMs(5000))

The Periodic Table tells us that >! sets up an exception listener and a recent mailing list thread explains how to set a timeout.
All together, then, you might do something like:
val http = new dispatch.Http {
import org.apache.http.params.CoreConnectionPNames
client.getParams.setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 2000)
client.getParams.setParameter(CoreConnectionPNames.SO_TIMEOUT, 5000)
}
http(req >! {
case e => // ...
})
Note that I haven't tested this...

In case you are using Dispatch reboot (with AsyncHttpClient as the underlying library) this is how you'd set the client configuration:
val myHttp = new dispatch.Http {
import com.ning.http.client._
val builder = new AsyncHttpClientConfig.Builder()
builder.setCompressionEnabled(true)
.setAllowPoolingConnection(true)
.setRequestTimeoutInMs(5000)
override lazy val client = new AsyncHttpClient(builder.build())
}
and then just use this new object as you'd otherwise use http:
myHttp((url(baseUrl) <<? args) OK as.xml.Elem).either

Related

Why does Source.tick stop after one hundred HttpRequests?

Using akka stream and akka HTTP, I have created a stream which polls an api every 3 seconds, Unmarshalls the result to a JsValue object and sends this result to an actor. As can be seen in the following code:
// Source wich performs an http request every 3 seconds.
val source = Source.tick(0.seconds,
3.seconds,
HttpRequest(uri = Uri(path = Path("/posts/1"))))
// Processes the result of the http request
val flow = Http().outgoingConnectionHttps("jsonplaceholder.typicode.com").mapAsync(1) {
// Able to reach the API.
case HttpResponse(StatusCodes.OK, _, entity, _) =>
// Unmarshal the json response.
Unmarshal(entity).to[JsValue]
// Failed to reach the API.
case HttpResponse(code, _, entity, _) =>
entity.discardBytes()
Future.successful(code.toString())
}
// Run stream
source.via(flow).runWith(Sink.actorRef[Any](processJsonActor,akka.actor.Status.Success(("Completed stream"))))
This works, however the stream closes after 100 HttpRequests (ticks).
What is the cause of this behaviour?
Definitely something to do with outgoingConnectionHttps. This is a low level DSL and there could be some misconfigured setting somewhere which is causing this (although I couldn't figure out which one).
Usage of this DSL is actually discouraged by the docs.
Try using a higher level DSL like cached connection pool
val flow = Http().cachedHostConnectionPoolHttps[NotUsed]("akka.io").mapAsync(1) {
// Able to reach the API.
case (Success(HttpResponse(StatusCodes.OK, _, entity, _)), _) =>
// Unmarshal the json response.
Unmarshal(entity).to[String]
// Failed to reach the API.
case (Success(HttpResponse(code, _, entity, _)), _) =>
entity.discardBytes()
Future.successful(code.toString())
case (Failure(e), _) ⇒
throw e
}
// Run stream
source.map(_ → NotUsed).via(flow).runWith(...)
A potential issue is that there is no backpressure signal with Sink.actorRef, so the actor's mailbox could be getting full. If the actor, whenever it receives a JsValue object, is doing something that could take a long time, use Sink.actorRefWithAck instead. For example:
val initMessage = "start"
val completeMessage = "done"
val ackMessage = "ack"
source
.via(flow)
.runWith(Sink.actorRefWithAck[Any](
processJsonActor, initMessage, ackMessage, completeMessage))
You would need to change the actor to handle an initMessage and reply to the stream for every stream element with an ackMessage (with sender ! ackMessage). More information on Sink.actorRefWithAck is found here.

How to get error from akka-stream propagate to akka-http to both be logged and notify the client properly?

Right now I am using akka-stream and akka-HTTP to build a file streaming API. As such I am injecting a streaming source into an entity to have data streamed directly to the HTTP client like so:
complete(HttpEntity(ContentTypes.`application/octet-stream`, source))
However, if for some reason the stream fails, the connection gets closed by akka-http without further explanation or logging.
I would need 2 things:
How can I get the exception logs?
How can I notify my client with a message before closing the connection?
Thank you
As mentioned in comment HTTP protocol does not allow to signal error to the client side.
As to logging:
For me it boils down to missing proper access log directive in akka http.
In my current project we have decorator which register onComplete handler for http entity before giving it to akka http for rendering.
private def onResponseStreamEnd(response: HttpResponse)(action: StatusCode => Unit): HttpResponse =
if (!response.status.allowsEntity() || response.entity.isKnownEmpty()) {
action(response.status)
response
} else {
val dataBytes =
onStreamEnd(response.entity) { result =>
val overallStatusCode =
result match {
case Success(_) =>
response.status
case Failure(e) =>
logger.error(e, s"error streaming response [${e.getMessage}]")
StatusCodes.InternalServerError
}
action(overallStatusCode)
}
response.withEntity(response.entity.contentLengthOption match {
case Some(length) => HttpEntity(response.entity.contentType, length, dataBytes)
case None => HttpEntity(response.entity.contentType, dataBytes)
})
}
private def onStreamEnd(entity: HttpEntity)(onComplete: Try[Done] ⇒ Unit): Source[ByteString, _] =
entity.dataBytes.alsoTo { Sink.onComplete(onComplete) }
Usage:
complete(onResponseStreamEnd(HttpResponse(StatusCodes.OK, HttpEntity(ContentTypes.`application/octet-stream`, source))){ statusCode => .... })
Similar approach but using custom graph stage you can find here

Akka Streams with Akka HTTP Server and Client

I'm trying to create an endpoint on my Akka Http Server which tells the users it's IP address using an external service (I know this can be performed way easier but I'm doing this as a challenge).
The code that doesn't make use of streams on the upper most layer is this:
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
val requestHandler: HttpRequest => Future[HttpResponse] = {
case HttpRequest(GET, Uri.Path("/"), _, _, _) =>
Http().singleRequest(HttpRequest(GET, Uri("http://checkip.amazonaws.com/"))).flatMap { response =>
response.entity.dataBytes.runFold(ByteString(""))(_ ++ _) map { string =>
HttpResponse(entity = HttpEntity(MediaTypes.`text/html`,
"<html><body><h1>" + string.utf8String + "</h1></body></html>"))
}
}
case _: HttpRequest =>
Future(HttpResponse(404, entity = "Unknown resource!"))
}
Http().bindAndHandleAsync(requestHandler, "localhost", 8080)
and it is working fine. However, as a challenge, I wanted to limit myself to only using streams (no Future's).
This is the layout I thought I'd use for this kind of an approach:
Source[Request] -> Flow[Request, Request] -> Flow[Request, Response] ->Flow[Response, Response] and to accommodate the 404 route, also Source[Request] -> Flow[Request, Response]. Now, if my Akka Stream knowledge serves me well, I need to use a Flow.fromGraph for such a thing, however, this is where I'm stuck.
In a Future I can do an easy map and flatMap for the various endpoints but in streams that would mean dividing up the Flow into multiple Flow's and I'm not quite sure how I'd do that. I thought about using UnzipWith and Options or a generic Broadcast.
Any help on this subject would be much appreciated.
I don't if this would be necessary? -- http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0-M2/scala/stream-customize.html
You do not need to use Flow.fromGraph. Instead, a singular Flow that uses flatMapConcat will work:
//an outgoing connection flow
val checkIPFlow = Http().outgoingConnection("checkip.amazonaws.com")
//converts the final html String to an HttpResponse
def byteStrToResponse(byteStr : ByteString) =
HttpResponse(entity = new Default(MediaTypes.`text/html`,
byteStr.length,
Source.single(byteStr)))
val reqResponseFlow = Flow[HttpRequest].flatMapConcat[HttpResponse]( _ match {
case HttpRequest(GET, Uri.Path("/"), _, _, _) =>
Source.single(HttpRequest(GET, Uri("http://checkip.amazonaws.com/")))
.via(checkIPFlow)
.mapAsync(1)(_.entity.dataBytes.runFold(ByteString(""))(_ ++ _))
.map("<html><body><h1>" + _.utf8String + "</h1></body></html>")
.map(ByteString.apply)
.map(byteStrToResponse)
case _ =>
Source.single(HttpResponse(404, entity = "Unknown resource!"))
})
This Flow can then be used to bind to incoming requests:
Http().bindAndHandle(reqResponseFlow, "localhost", 8080)
And all without Futures...

How to handle response timeout?

In akka-http routing I can return Future as a response that implicitly converts to ToResponseMarshaller.
Is there some way to handle timeout of this future? Or timeout of connection in route level? Or one way is to use Await() function?
Right now client can wait response forever.
complete {
val future = for {
response <- someIOFunc()
entity <- someOtherFunc()
} yield entity
future.onComplete({
case Success(result) =>
HttpResponse(entity = HttpEntity(MediaTypes.`text/xml`, result))
case Failure(result) =>
HttpResponse(entity = utils.getFault("fault"))
})
future
}
Adding a timeout to an asynchronous operation means creating a new Future that is completed either by the operation itself or by the timeout:
import akka.pattern.after
val future = ...
val futureWithTimeout = Future.firstCompletedOf(
future ::
after(1.second, system.scheduler)(Future.failed(new TimeoutException)) ::
Nil
)
The second Future could also hold a successful result that replaces the error, depending on what exactly it is that you want to model.
As a side note: the presented code sample contains dead code, registering an onComplete handler on a Future only makes sense for side-effects but you seem to want to transform the Future’s value and create an HttpEntity from it. That should be done using map and recover:
future
.map(result => HttpResponse(entity = HttpEntity(MediaTypes.`text/xml`, result)))
.recover { case ex => HttpResponse(entity = utils.getFault("fault")) }
This would then be the overall return value that is passed to the complete directive.

How to chain Future's of client requests without nesting with onComplete?

I need to query a RESTful service that always returns a JSON response. I need to contact it a few times, always with some more information that I learned from the previous request. I'm using Akka2, Scala, Jerkson and Spray-Can.
My current approach seems to work, but it looks ugly and requires nesting everything. I read that there should be some techniques available regarding chaining and such, but I couldn't figure out how to apply them to my current use-case.
Here is the code I'm talking about:
def discoverInitialConfig(hostname: String, bucket: String) = {
val poolResponse: Future[HttpResponse] =
HttpDialog(httpClient, hostname, 8091)
.send(HttpRequest(uri = "/pools"))
.end
poolResponse onComplete {
case Right(response) =>
log.debug("Received the following global pools config: {}", response)
val pool = parse[PoolsConfig](response.bodyAsString)
.pools
.find(_("name") == defaultPoolname)
.get
val selectedPoolResponse: Future[HttpResponse] =
HttpDialog(httpClient, hostname, 8091)
.send(HttpRequest(uri = pool("uri")))
.end
selectedPoolResponse onComplete {
case Right(response) =>
log.debug("Received the following pool config: {}", response)
println(response.bodyAsString)
case Left(failure) =>
log.error("Could not load the pool config! {}", failure)
}
case Left(failure) =>
log.error("Could not load the global pools config! {}", failure)
}
I think you can see the pattern. Contact the REST service, read it, on success parse it into a JSON case class, extract information out and then do the next call.
My structure here is only two-levels deep but I need to add a third level as well.
Is there a technique available to improve this for better readability or can I only stick with this? If you need any further information I'm happy to provide it. You can see the full code here: https://github.com/daschl/cachakka/blob/f969d1f56a4c90a929de9c7ed4e4a0cccea5ba70/src/main/scala/com/cachakka/cluster/actors/InitialConfigLoader.scala
Thanks,
Michael
HttpDialog seems to cover your use case exactly.
I think I found a reasonable shortcut by using the provided reply method from spray-can.
def discoverInitialConfig(hostname: String, bucket: String) = {
val poolResponse: Future[HttpResponse] =
HttpDialog(httpClient, hostname, 8091)
.send(HttpRequest(uri = "/pools"))
.reply(response => {
log.debug("Received the following global pools config: {}", response)
val selectedPool = parse[PoolsConfig](response.bodyAsString)
.pools.find(_("name") == defaultPoolname).get
HttpRequest(uri = selectedPool("uri"))
})
.reply(response => {
log.debug("Received the following pool config: {}", response)
println(response.bodyAsString)
HttpRequest(uri = "/")
})
.end
}
If this is the best available approach, I'll mark it as "answered" but I'm eager to get actual replies from people who know this stuff much better than me.