I post some data to Server using the following code
def post(endpoint: String, entity: Strict) = {
Http().singleRequest(HttpRequest(uri = Notifier.notificationUrl + endpoint, method = HttpMethods.POST,
entity = entity)) onComplete {
case Success(response) => response match {
case HttpResponse(StatusCodes.OK, _, _, _) =>
log.info("communicated successfully with Server")
}
case Failure(response) =>
log.error("communicated failed with Server: {}", response)
}
}
This is called every 10 seconds when Notifier actor receives message as following
case ecMonitorInformation: ECMonitorInformation =>
post("monitor", httpEntityFromJson(ecMonitorInformation.toJson))
Problem?
I see that Initially (around 5 requests going to server) but then it hungs up, I do not see any logging, server does not receive any data. After a while on the client side, I see following
ERROR c.s.e.notification.Notifier - communicated failed with Server: java.lang.RuntimeException: Exceeded configured max-open-requests value of [32]
What is going on? How do I fix this issue?
I went through the docs and tried the following
val connectionFlow: Flow[HttpRequest, HttpResponse,
Future[Http.OutgoingConnection]] =
Http().outgoingConnection(host = "localhost", port = 8080)
and then
def httpPost(uri: String, httpEntity:Strict) {
val responseFuture: Future[HttpResponse] =
Source.single(HttpRequest(uri = "/monitor", method = HttpMethods.POST, entity=httpEntity))
.via(connectionFlow)
.runWith(Sink.head)
responseFuture onComplete {
case Success(response) => log.info("Communicated with Server: {}", response)
case Failure(failure) => log.error("Communication failed with Server: {}", failure)
}
and this worked for me
You can also overcome this error by upping the max-open-requests property of akka which is 32 by default.
The property to change will be:
akka.http.host-connection-pool.max-open-requests = 64
The only caveat is that this will fail when the client opens more concurrent connections than what the new value of that parameter is, in this example if the open connections exceed 64, you will get the same error.
If you are going to be repeatedly calling your method, you might want to consider using one of the connection pool based client methods as described here:
http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/scala/http/client-side/index.html
You can also set the connection pool settings in the akka-http client configuration:
http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/scala/http/configuration.html#akka-http-core
Search for host-connection-pool.
You could use Source.queue instead of Source.single to provide buffering and overflow strategy. See more details at https://stackoverflow.com/a/35115314/1699837
Related
I have an HTTP Connection Pool that hangs after a couple of hours of running:
private def createHttpPool(host: String): SourceQueue[(HttpRequest, Promise[HttpResponse])] = {
val pool = Http().cachedHostConnectionPoolHttps[Promise[HttpResponse]](host)
Source.queue[(HttpRequest, Promise[HttpResponse])](config.poolBuffer, OverflowStrategy.dropNew)
.via(pool).toMat(Sink.foreach {
case ((Success(res), p)) => p.success(res)
case ((Failure(e), p)) => p.failure(e)
})(Keep.left).run
}
I enqueue items with:
private def enqueue(uri: Uri): Future[HttpResponse] = {
val promise = Promise[HttpResponse]
val request = HttpRequest(uri = uri) -> promise
queue.offer(request).flatMap {
case Enqueued => promise.future
case _ => Future.failed(ConnectionPoolDroppedRequest)
}
}
And resolve the response like this:
private def request(uri: Uri): Future[HttpResponse] = {
def retry = {
Thread.sleep(config.dispatcherRetryInterval)
logger.info(s"retrying")
request(uri)
}
logger.info("req-start")
for {
response <- enqueue(uri)
_ = logger.info("req-end")
finalResponse <- response.status match {
case TooManyRequests => retry
case OK => Future.successful(response)
case _ => response.entity.toStrict(10.seconds).map(s => throw Error(s.toString, uri.toString))
}
} yield finalResponse
}
The result of this function is then always transformed if the Future is successful:
def get(uri: Uri): Future[Try[JValue]] = {
for {
response <- request(uri)
json <- Unmarshal(response.entity).to[Try[JValue]]
} yield json
}
Everything works fine for a while and then all I see in the logs are req-start and no req-end.
My akka configuration is like this:
akka {
actor.deployment.default {
dispatcher = "my-dispatcher"
}
}
my-dispatcher {
type = Dispatcher
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 256
parallelism-factor = 128.0
parallelism-max = 1024
}
}
akka.http {
host-connection-pool {
max-connections = 512
max-retries = 5
max-open-requests = 16384
pipelining-limit = 1
}
}
I'm not sure if this is a configuration problem or a code problem. I have my parallelism and connection numbers so high because without it I get very poor req/s rate (I want to request as fast possible - I have other rate limiting code to protect the server).
You are not consuming the entity of the responses you get back from the server. Citing the docs below:
Consuming (or discarding) the Entity of a request is mandatory! If
accidentally left neither consumed or discarded Akka HTTP will assume
the incoming data should remain back-pressured, and will stall the
incoming data via TCP back-pressure mechanisms. A client should
consume the Entity regardless of the status of the HttpResponse.
The entity comes in the form of a Source[ByteString, _] which needs to be run to avoid resource starvation.
If you don't need to read the entity, the simplest way to consume the entity bytes is to discard them, by using
res.discardEntityBytes()
(you can attach a callback by adding - e.g. - .future().map(...)).
This page in the docs describes all the alternatives to this, including how to read the bytes if needed.
--- EDIT
After more code/info was provided, it is clear that the resource consumption is not the problem. There is another big red flag in this implementation, namely the Thread.sleep in the retry method.
This is a blocking call that is very likely to starve the threading infrastructure of your underlying actor system.
A full blown explanation of why this is dangerous was provided in the docs.
Try changing that and using akka.pattern.after (docs). Example below:
def retry = akka.pattern.after(200 millis, using = system.scheduler)(request(uri))
Right now I am using akka-stream and akka-HTTP to build a file streaming API. As such I am injecting a streaming source into an entity to have data streamed directly to the HTTP client like so:
complete(HttpEntity(ContentTypes.`application/octet-stream`, source))
However, if for some reason the stream fails, the connection gets closed by akka-http without further explanation or logging.
I would need 2 things:
How can I get the exception logs?
How can I notify my client with a message before closing the connection?
Thank you
As mentioned in comment HTTP protocol does not allow to signal error to the client side.
As to logging:
For me it boils down to missing proper access log directive in akka http.
In my current project we have decorator which register onComplete handler for http entity before giving it to akka http for rendering.
private def onResponseStreamEnd(response: HttpResponse)(action: StatusCode => Unit): HttpResponse =
if (!response.status.allowsEntity() || response.entity.isKnownEmpty()) {
action(response.status)
response
} else {
val dataBytes =
onStreamEnd(response.entity) { result =>
val overallStatusCode =
result match {
case Success(_) =>
response.status
case Failure(e) =>
logger.error(e, s"error streaming response [${e.getMessage}]")
StatusCodes.InternalServerError
}
action(overallStatusCode)
}
response.withEntity(response.entity.contentLengthOption match {
case Some(length) => HttpEntity(response.entity.contentType, length, dataBytes)
case None => HttpEntity(response.entity.contentType, dataBytes)
})
}
private def onStreamEnd(entity: HttpEntity)(onComplete: Try[Done] ⇒ Unit): Source[ByteString, _] =
entity.dataBytes.alsoTo { Sink.onComplete(onComplete) }
Usage:
complete(onResponseStreamEnd(HttpResponse(StatusCodes.OK, HttpEntity(ContentTypes.`application/octet-stream`, source))){ statusCode => .... })
Similar approach but using custom graph stage you can find here
I am trying to force my redis client to timeout for testing purpose and I a fail to achieve so. I specify timeout to 2ms in my config and the set operation I perform takes > 2ms so why it does not timeout? Are these settings are kind of soft settings and not hard enforcement? I am using Jedis 2.6 and Scala 2.10 with Play 2.2.3
#Singleton
class RedisClient extends Cache {
// set timeout
val TIMEOUT = 2
private val pool = new JedisPool(new JedisPoolConfig(), getStringFromConfig("redis.url"), getIntFromConfig("redis.port"), TIMEOUT);
def isOpen = pool.getNumActive()
def set(key: String, value: String) = {
isOpen match {
case -1 => throw new Exception("Redis server is not running")
case _ => {
val jedis = pool.getResource()
val before = Platform.currentTime
jedis.set(key, value)
println("TIME TAKEN " + (Platform.currentTime - before))
pool.returnResource(jedis)
}
}
}
}
Actualy you do not need to test it but you may found answer in Jedis sources. TimeOut value used:
as java socket connection timeout
as SO_TIMEOUT value. More info about it here.
To achieve goal in your test:
Redis server should be heavy loaded during your test to not accept the connection. If you need connection timeout.
Try to use some proxy to drop down connection perfomance (timeout by SO_TIMEOUT value).
I need to query a RESTful service that always returns a JSON response. I need to contact it a few times, always with some more information that I learned from the previous request. I'm using Akka2, Scala, Jerkson and Spray-Can.
My current approach seems to work, but it looks ugly and requires nesting everything. I read that there should be some techniques available regarding chaining and such, but I couldn't figure out how to apply them to my current use-case.
Here is the code I'm talking about:
def discoverInitialConfig(hostname: String, bucket: String) = {
val poolResponse: Future[HttpResponse] =
HttpDialog(httpClient, hostname, 8091)
.send(HttpRequest(uri = "/pools"))
.end
poolResponse onComplete {
case Right(response) =>
log.debug("Received the following global pools config: {}", response)
val pool = parse[PoolsConfig](response.bodyAsString)
.pools
.find(_("name") == defaultPoolname)
.get
val selectedPoolResponse: Future[HttpResponse] =
HttpDialog(httpClient, hostname, 8091)
.send(HttpRequest(uri = pool("uri")))
.end
selectedPoolResponse onComplete {
case Right(response) =>
log.debug("Received the following pool config: {}", response)
println(response.bodyAsString)
case Left(failure) =>
log.error("Could not load the pool config! {}", failure)
}
case Left(failure) =>
log.error("Could not load the global pools config! {}", failure)
}
I think you can see the pattern. Contact the REST service, read it, on success parse it into a JSON case class, extract information out and then do the next call.
My structure here is only two-levels deep but I need to add a third level as well.
Is there a technique available to improve this for better readability or can I only stick with this? If you need any further information I'm happy to provide it. You can see the full code here: https://github.com/daschl/cachakka/blob/f969d1f56a4c90a929de9c7ed4e4a0cccea5ba70/src/main/scala/com/cachakka/cluster/actors/InitialConfigLoader.scala
Thanks,
Michael
HttpDialog seems to cover your use case exactly.
I think I found a reasonable shortcut by using the provided reply method from spray-can.
def discoverInitialConfig(hostname: String, bucket: String) = {
val poolResponse: Future[HttpResponse] =
HttpDialog(httpClient, hostname, 8091)
.send(HttpRequest(uri = "/pools"))
.reply(response => {
log.debug("Received the following global pools config: {}", response)
val selectedPool = parse[PoolsConfig](response.bodyAsString)
.pools.find(_("name") == defaultPoolname).get
HttpRequest(uri = selectedPool("uri"))
})
.reply(response => {
log.debug("Received the following pool config: {}", response)
println(response.bodyAsString)
HttpRequest(uri = "/")
})
.end
}
If this is the best available approach, I'll mark it as "answered" but I'm eager to get actual replies from people who know this stuff much better than me.
I've been using the Databinder Dispatch library in a client for a simple REST-ish API. I know how to detect if I get an HTTP response with an error status:
Http x (request) {
case (200, _, _, content) => successResult(content())
case (404, _, _, _) => notFoundErrorResult
case (_, _, _, _) => genericErrorResult
}
But how can I distinguish an error response from a failure to get any response at all, because of an invalid domain or failure to connect? And is there any way to implement a timeout while still using synchronous semantics? If there's anything relevant in the API, I've missed it.
There is also a more elegant way to configure client using Http.configure method which receives Builder => Builder function as an argument:
val http = Http.configure(_.setAllowPoolingConnection(true).setConnectionTimeoutInMs(5000))
The Periodic Table tells us that >! sets up an exception listener and a recent mailing list thread explains how to set a timeout.
All together, then, you might do something like:
val http = new dispatch.Http {
import org.apache.http.params.CoreConnectionPNames
client.getParams.setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 2000)
client.getParams.setParameter(CoreConnectionPNames.SO_TIMEOUT, 5000)
}
http(req >! {
case e => // ...
})
Note that I haven't tested this...
In case you are using Dispatch reboot (with AsyncHttpClient as the underlying library) this is how you'd set the client configuration:
val myHttp = new dispatch.Http {
import com.ning.http.client._
val builder = new AsyncHttpClientConfig.Builder()
builder.setCompressionEnabled(true)
.setAllowPoolingConnection(true)
.setRequestTimeoutInMs(5000)
override lazy val client = new AsyncHttpClient(builder.build())
}
and then just use this new object as you'd otherwise use http:
myHttp((url(baseUrl) <<? args) OK as.xml.Elem).either