I have a basic client which I use to test my server. For the configuration I am using application.json
"spray": {
"can": {
"client": {
"idle-timeout": "120 s",
"request-timeout": "180 s"
},
"host-connector": {
"max-retries": "1",
"max-connections": "64"
}
}
}
however in the sendrecieve method i see that the timeout is always 60 sec , as according to the documantation , if I use request-timeout it suppose to be the implicit value
def sendReceive(implicit refFactory: ActorRefFactory, executionContext: ExecutionContext,
futureTimeout: Timeout = 60.seconds): SendReceive =
sendReceive(IO(Http)(actorSystem))
Do I need explicitly to load the configuration?
This is a confusing aspect of spary's various timeout values, for a detailed explanation see: Understanding Spray Client Timeout Settings
A couple of points about the method definition above, the timeout is just used to satisfy the timeout required by the ask made to the transport actor, it does not relate to a request timeout for this connection. futureTimeout: Timeout = 60.seconds means that this default value of is used if none is provided, not that it is unconditionally used.
You can programmatically configure the requestTimeout by passing a HostConnectorSetup to either the host or request level API's, as you already have this in your spray.can.client configuration though you should not need to make further changes.
Related
I searched the other StackOverflow question/answers towards this error, but couldn't find a hint for solving this problem.
The Akka HTTP application runs for like 5 hours under high workload without problems, and than I start to get multiple:
Response entity was not subscribed after 1 second. Make sure to read the response `entity` body or call `entity.discardBytes()` on it -- in case you deal with `HttpResponse`, use the shortcut `response.discardEntityBytes()`. GET /api/name123 Empty -> 200 OK Default(142 bytes)
and later
The connection actor has terminated. Stopping now.
The actor is only sending out API requests and afterwards forwards those responses to another actor if successfully, in case of failure, that request is added back to the todo stack and retried later. This is the main code:
private def makeApiRequest(id: String): Unit = {
val url = UrlBuilder(id)
val request = HttpRequest(method = HttpMethods.GET, uri = url)
val f: Future[(StatusCode, String)] = Http(context.system)
.singleRequest(request)
.flatMap(_.toStrict(2.seconds))
.flatMap { resp =>
Unmarshal(resp.entity).to[String].map((resp.status, _))
}
context.pipeToSelf(f) {
case Success(response) =>
API_HandleResponseSuccess(id, response._1, response._2)
case Failure(e) =>
API_HandleResponseFailure(id, e.getMessage)
}
}
I don't really understand why I get the "Response entity was not subscribed..." error, as I do Unmarshal(resp.entity).to[String] and thereby would think, that no .DiscardEntityBytes() is needed, or does it needs to be still included somehow?
Side information: Also confusing to me, why the CPU performance doesn't stay constant.
Within the actor do I track the response times of each request and calculate the amount of max. parallel requests possible to handle with the given hardware conditions (restricted to a max max of 120 though) on a regular basis to account for API response time fluctuations, so there should be always enough room to make the requests without starving for that actor. In addition would that be the respective application.conf:
dispatcher-worker-io {
type = Dispatcher
executor = "thread-pool-executor"
thread-pool-executor {
fixed-pool-size = 120
keep-alive-time = 60s
allow-core-timeout = off
}
shutdown-timeout = 60s
throughput = 1
}
...
akka.http.client.host-connection-pool.max-connections = 180
akka.http.client.host-connection-pool.max-open-requests = 256
akka.http.client.host-connection-pool.max-retries = 0
Any ideas on why I after 5 hours without problems start to get those exceptions mentioned above?
or
Has an idea of which part of above shared code might leads to this non-linear CPU performance?
I also made multiple of those long lasting hour runs, and it always ends out like this, somehow it's starving after 5 to 6 hours.
val AkkaVersion = "2.6.15"
val AkkaHttpVersion = "10.2.6"
Directly from the docs (https://doc.akka.io/docs/akka-http/current/client-side/request-level.html):
Always make sure you consume the response entity streams (of type
Source[ByteString,Unit]). Connect the response entity Source to a
Sink, or call response.discardEntityBytes() if you don’t care about
the response entity.
Read the Implications of the streaming nature of Request/Response
Entities section for more details.
If the application doesn’t subscribe to the response entity within
akka.http.host-connection-pool.response-entity-subscription-timeout,
the stream will fail with a TimeoutException: Response entity was not
subscribed after ....
You need to .discardEntityBytes() in case of failure. Right now you only consume it on success.
Perhaps high CPU load is caused by all these unfreed resources on the JVM + retries of all the failures.
I am trying to do asynchronous http calls with akka streams.
This is what I tried.
Source(listEndpoints)
.mapAsync(20)(endpoint => Future(Await.result(request(HttpMethods.POST, endpoint, List(authHeader)), timeout)))
.runWith(Sink.seq[HttpResponse])
I am using akka-http within the request method and it returns Future[HttpResponse]
I think I am abusing Future here. The code above would give me a Future[List[HttpResponse]] and I have to use Await again to get a List[HttpResponse]. Is there a more elegant way to timeout functions within mapAsync?
Assuming your request method at some point does
Http().singleRequest
to get a Future[HttpResponse], you can pass a timeout for the request through:
// inside def request(...), will probably need to add a timeout argument here
val request = ??? // Build the HttpRequest
Http().singleRequest(
request = request,
settings = ConnectionPoolSettings.default.withMaxConnectionLifetime(timeout)
Then your stream would just be
Source(listEndpoints)
.mapAsync(request(...))
.runWith(Sink.seq[HttpResponse])
and you'd only need to Await at the "end of the world" for the Future[List[HttpResponse]] to complete.
You can also change the default max connection lifetime with akka.http.host-connection-pool.max-connection-lifetime in application.conf
Is it possible to set HTTP request-response timeout in a Finatra server?
The http controller callback typically returns a Future, that once resolved the response is transmitted. I would like to define, within Finatra, how long the server should wait before returning a 500 or 400 response.
You can extend the HttpServer and define your own timeout
trait CustomServer extends HttpServer with Tls {
then you overwrite the configureHttpServer method and you define timeout, requests sites and other attributes
override def configureHttpServer(server: Http.Server): Http.Server = {
server.withAdmissionControl.concurrencyLimit(maxConcurrentRequests = 2000, maxWaiters = 0)
.withResponseClassifier(HttpResponseClassifier.ServerErrorsAsFailures)
.withMaxRequestSize(StorageUnit.fromMegabytes(200))
.withRequestTimeout(50.seconds)
}
I think, you are looking for Future.within
In my Groovy script, I have following structure:
def sql = Sql.newInstance(connString, "user", "password",
"org.postgresql.Driver")
sql.withTransaction {
sql.withBatch(){}
sql.withBatch(){}
sql.withBatch(){}
.........
}
sql.close()
I want to take care of timeout issues here.
But Sql API doesn't have any method for it.
So how can I do it? I am using PostgreSQL driver.
I came across this. But I get error:
java.sql.SQLFeatureNotSupportedException: Method org.postgresql.jdbc4.Jdbc4Connection.setNetworkTimeout(Executor, int) is not yet implemented.
PS:
int[] modifyCount = sql.withBatch(batchSize, updateQuery) { ps ->
keyValue.each { k,v ->
ps.addBatch(keyvalue:k, newvalue:v)
}
}
In above code, when I try to add ps.setQueryTimeout(), error message says no such method defined.
Low-level timeouts could be defined through connection properties:
https://jdbc.postgresql.org/documentation/head/connect.html
loginTimeout Specify how long to wait for establishment of a database connection.
connectTimeout The timeout value used for socket connect operations.
socketTimeout The timeout value used for socket read operations.
These properties may be specified in either the connection URL or an additional Properties object parameter.
Query timeout. After connecting to the database you could define closure to be executed for each statement:
sql.withStatement{java.sql.Statement stmt->
stmt.setQueryTimeout( MY_SQL_TIMEOUT )
}
I'm using Spray client to consume a third-party API. Unfortunately, the API I'm consuming is not very secure and utilizes an authentication method using GET query parameters.
Sometimes we're getting timeouts or connection issues which we know to deal with applicatively. The problem is that Spray logs this at a WARN log-level, and the URL including the sensitive query parameters () are being written in our log files.
Here's an example of the log file.
2015-05-19 12:23:17,024 WARN HttpHostConnectionSlot - Connection attempt to 10.10.10.10:443 failed in response to GET request to /api/?type=keygen&user=test_user&password=S3kret! with 2 retries left, retrying...
2015-05-19 12:23:17,084 WARN HttpHostConnectionSlot - Connection attempt to 10.10.10.10:443 failed in response to GET request to /api/?type=keygen&user=test_user&password=S3kret! with 1 retries left, retrying...
Is there any way to filter this? (Maybe in Akka?)
Spray reuses akka-logging for doing all logging groundwork.
In akka you can redeclare a custom event logger in application config:
akka {
# event-handlers = ["akka.event.Logging$DefaultLogger"] // default one
event-handlers = ["com.example.PrivacyLogger"] // custom one
# Options: ERROR, WARNING, INFO, DEBUG
loglevel = "DEBUG"
}
It may look like this:
class PrivacyLogger extends DefaultLogger {
override def receive: Receive = {
case InitializeLogger(_) ⇒ sender() ! LoggerInitialized
case event: LogEvent ⇒ print(stripSecret(event))
}
private def stripSecret(event:LogEvent) = ...
}
But you always can implement your own message processing logic here instead of simple printing.
PS. If you use slf4j for logging, the solution will mostly look the same, but with some minor differences like overriding akka.event.slf4j.Slf4jEventHandler instead of DefaultLogger.