I'm using Spray client to consume a third-party API. Unfortunately, the API I'm consuming is not very secure and utilizes an authentication method using GET query parameters.
Sometimes we're getting timeouts or connection issues which we know to deal with applicatively. The problem is that Spray logs this at a WARN log-level, and the URL including the sensitive query parameters () are being written in our log files.
Here's an example of the log file.
2015-05-19 12:23:17,024 WARN HttpHostConnectionSlot - Connection attempt to 10.10.10.10:443 failed in response to GET request to /api/?type=keygen&user=test_user&password=S3kret! with 2 retries left, retrying...
2015-05-19 12:23:17,084 WARN HttpHostConnectionSlot - Connection attempt to 10.10.10.10:443 failed in response to GET request to /api/?type=keygen&user=test_user&password=S3kret! with 1 retries left, retrying...
Is there any way to filter this? (Maybe in Akka?)
Spray reuses akka-logging for doing all logging groundwork.
In akka you can redeclare a custom event logger in application config:
akka {
# event-handlers = ["akka.event.Logging$DefaultLogger"] // default one
event-handlers = ["com.example.PrivacyLogger"] // custom one
# Options: ERROR, WARNING, INFO, DEBUG
loglevel = "DEBUG"
}
It may look like this:
class PrivacyLogger extends DefaultLogger {
override def receive: Receive = {
case InitializeLogger(_) ⇒ sender() ! LoggerInitialized
case event: LogEvent ⇒ print(stripSecret(event))
}
private def stripSecret(event:LogEvent) = ...
}
But you always can implement your own message processing logic here instead of simple printing.
PS. If you use slf4j for logging, the solution will mostly look the same, but with some minor differences like overriding akka.event.slf4j.Slf4jEventHandler instead of DefaultLogger.
Related
I searched the other StackOverflow question/answers towards this error, but couldn't find a hint for solving this problem.
The Akka HTTP application runs for like 5 hours under high workload without problems, and than I start to get multiple:
Response entity was not subscribed after 1 second. Make sure to read the response `entity` body or call `entity.discardBytes()` on it -- in case you deal with `HttpResponse`, use the shortcut `response.discardEntityBytes()`. GET /api/name123 Empty -> 200 OK Default(142 bytes)
and later
The connection actor has terminated. Stopping now.
The actor is only sending out API requests and afterwards forwards those responses to another actor if successfully, in case of failure, that request is added back to the todo stack and retried later. This is the main code:
private def makeApiRequest(id: String): Unit = {
val url = UrlBuilder(id)
val request = HttpRequest(method = HttpMethods.GET, uri = url)
val f: Future[(StatusCode, String)] = Http(context.system)
.singleRequest(request)
.flatMap(_.toStrict(2.seconds))
.flatMap { resp =>
Unmarshal(resp.entity).to[String].map((resp.status, _))
}
context.pipeToSelf(f) {
case Success(response) =>
API_HandleResponseSuccess(id, response._1, response._2)
case Failure(e) =>
API_HandleResponseFailure(id, e.getMessage)
}
}
I don't really understand why I get the "Response entity was not subscribed..." error, as I do Unmarshal(resp.entity).to[String] and thereby would think, that no .DiscardEntityBytes() is needed, or does it needs to be still included somehow?
Side information: Also confusing to me, why the CPU performance doesn't stay constant.
Within the actor do I track the response times of each request and calculate the amount of max. parallel requests possible to handle with the given hardware conditions (restricted to a max max of 120 though) on a regular basis to account for API response time fluctuations, so there should be always enough room to make the requests without starving for that actor. In addition would that be the respective application.conf:
dispatcher-worker-io {
type = Dispatcher
executor = "thread-pool-executor"
thread-pool-executor {
fixed-pool-size = 120
keep-alive-time = 60s
allow-core-timeout = off
}
shutdown-timeout = 60s
throughput = 1
}
...
akka.http.client.host-connection-pool.max-connections = 180
akka.http.client.host-connection-pool.max-open-requests = 256
akka.http.client.host-connection-pool.max-retries = 0
Any ideas on why I after 5 hours without problems start to get those exceptions mentioned above?
or
Has an idea of which part of above shared code might leads to this non-linear CPU performance?
I also made multiple of those long lasting hour runs, and it always ends out like this, somehow it's starving after 5 to 6 hours.
val AkkaVersion = "2.6.15"
val AkkaHttpVersion = "10.2.6"
Directly from the docs (https://doc.akka.io/docs/akka-http/current/client-side/request-level.html):
Always make sure you consume the response entity streams (of type
Source[ByteString,Unit]). Connect the response entity Source to a
Sink, or call response.discardEntityBytes() if you don’t care about
the response entity.
Read the Implications of the streaming nature of Request/Response
Entities section for more details.
If the application doesn’t subscribe to the response entity within
akka.http.host-connection-pool.response-entity-subscription-timeout,
the stream will fail with a TimeoutException: Response entity was not
subscribed after ....
You need to .discardEntityBytes() in case of failure. Right now you only consume it on success.
Perhaps high CPU load is caused by all these unfreed resources on the JVM + retries of all the failures.
Most of our Lagom entrypoints don't use multipart form requests, but one does. Since Lagom doesn't currently support multipart requests natively, the general suggestion I have seen is to call the underlying Play API, using the PlayServiceCall mechanism.
We have done that, and it works--most of the time. But we experience intermittent errors, especially when submitting large files. These are always cases of java.util.zip.ZipException (of various kinds), looking as if not an entire file has been received for processing.
Here's how the entrypoint looks in the code; in particular, the Play wrapping mechanism:
def upload = PlayServiceCall[NotUsed, UUID] {
wrapCall => Action.async(multipartFormData) {
request => wrapCall(ServiceCall { _ =>
val upload = request.body.file("upload")
val input = new FileInputStream(upload.get.ref.file)
val filename = upload.get.filename
// ...
// other code to actually process the file
// ...
})(request).run
}
}
Here are just two examples of exceptions we're seeing:
Caused by: java.util.zip.ZipException: invalid code lengths set
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
at java.util.zip.ZipInputStream.read(ZipInputStream.java:194)
at org.apache.poi.openxml4j.util.ZipSecureFile$ThresholdInputStream.read(ZipSecureFile.java:214)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
etc.
Caused by: java.util.zip.ZipException: invalid distance too far back
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
at java.util.zip.ZipInputStream.read(ZipInputStream.java:194)
at org.apache.poi.openxml4j.util.ZipSecureFile$ThresholdInputStream.read(ZipSecureFile.java:214)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
etc.
We use Lagom 1.3.8, in Scala. Any suggestions, please?
Try using the new service gateway based on Akka HTTP.
You can enable this by adding the following to your build.sbt:
lagomServiceGatewayImpl in ThisBuild := "akka-http"
The new service gateway is still disabled by default in Lagom 1.3.8, but Lagom users that have experienced this problem have reported that it is resolved by enabling the akka-http gateway. This will become the default implementation in Lagom 1.4.0.
I have a basic client which I use to test my server. For the configuration I am using application.json
"spray": {
"can": {
"client": {
"idle-timeout": "120 s",
"request-timeout": "180 s"
},
"host-connector": {
"max-retries": "1",
"max-connections": "64"
}
}
}
however in the sendrecieve method i see that the timeout is always 60 sec , as according to the documantation , if I use request-timeout it suppose to be the implicit value
def sendReceive(implicit refFactory: ActorRefFactory, executionContext: ExecutionContext,
futureTimeout: Timeout = 60.seconds): SendReceive =
sendReceive(IO(Http)(actorSystem))
Do I need explicitly to load the configuration?
This is a confusing aspect of spary's various timeout values, for a detailed explanation see: Understanding Spray Client Timeout Settings
A couple of points about the method definition above, the timeout is just used to satisfy the timeout required by the ask made to the transport actor, it does not relate to a request timeout for this connection. futureTimeout: Timeout = 60.seconds means that this default value of is used if none is provided, not that it is unconditionally used.
You can programmatically configure the requestTimeout by passing a HostConnectorSetup to either the host or request level API's, as you already have this in your spray.can.client configuration though you should not need to make further changes.
recently I started to build some small web processing service using akka streams. It's quite simple, I'm pulling urls from redis, then I'm downloading those urls(they are images) later I'm processing images, and pushing them to s3 and some json to redis.
I'm downloading lot of different kinds of images from multiple sites, I'm getting whole bunch of errors like 404, Unexpected disconnect , Response Content-Length 17951202 exceeds the configured limit of 8388608, EntityStreamException: Entity stream truncation and redirects. With redirects I'm invoking requestWithRedirects with address founded in location header of response.
Part responsible for downloading is pretty much like this:
override lazy val http: HttpExt = Http()
def requestWithRedirects(request: HttpRequest, retries: Int = 10)(implicit akkaSystem: ActorSystem, materializer: FlowMaterializer): Future[HttpResponse] = {
TimeoutFuture(timeout, msg = "Download timed out!") {
http.singleRequest(request)
}.flatMap {
response => handleResponse(request, response, retries)
}.recoverWith {
case e: Exception if retries > 0 =>
requestWithRedirects(request, retries = retries - 1)
}
}
TimeoutFuture is quite simple it takes future and timeout. If future takes longer than timeout it returns other future with timeout exception.
The problem I'm having is: after some time I'm getting an error:
Message: RuntimeException: Exceeded configured max-open-requests value of [128] akka.http.impl.engine.client.PoolInterfaceActor$$anonfun$receive$1.applyOrElse in PoolInterfaceActor.scala::109
akka.actor.Actor$class.aroundReceive in Actor.scala::467
akka.http.impl.engine.client.PoolInterfaceActor.akka$stream$actor$ActorSubscriber$$super$aroundReceive in PoolInterfaceActor.scala::46
akka.stream.actor.ActorSubscriber$class.aroundReceive in ActorSubscriber.scala::208
akka.http.impl.engine.client.PoolInterfaceActor.akka$stream$actor$ActorPublisher$$super$aroundReceive in PoolInterfaceActor.scala::46
akka.stream.actor.ActorPublisher$class.aroundReceive in ActorPublisher.scala::317
akka.http.impl.engine.client.PoolInterfaceActor.aroundReceive in PoolInterfaceActor.scala::46
akka.actor.ActorCell.receiveMessage in ActorCell.scala::516
akka.actor.ActorCell.invoke in ActorCell.scala::487
akka.dispatch.Mailbox.processMailbox in Mailbox.scala::238
akka.dispatch.Mailbox.run in Mailbox.scala::220
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec in AbstractDispatcher.scala::397
scala.concurrent.forkjoin.ForkJoinTask.doExec in ForkJoinTask.java::260
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask in ForkJoinPool.java::1339
scala.concurrent.forkjoin.ForkJoinPool.runWorker in ForkJoinPool.java::1979
scala.concurrent.forkjoin.ForkJoinWorkerThread.run in ForkJoinWorkerThread.java::107
I'm not sure what could be the problem but I think I have some downloads that were not finished properly and they stay in some global pool of connections after a while causing mentioned error. Any ideas what could be causing the problem? Or how to try find root of the problem: I already tested 404 responses, and Response Content-Length exceeds... errors, and they doesn't seem to be my troublemakers.
EDIT:
Most likely the problem is with my TimeoutFuture. I'm filling it with error as described here https://stackoverflow.com/a/29330010/2963977 but in my opinion future that is actually downloading an image never completes and it's taking my connection pool resources.
I wonder why those settings doesn't have any impact in my case :
akka.http.client.connecting-timeout = 1 s
akka.http.client.idle-timeout = 1 s
akka.http.host-connection-pool.idle-timeout = 1 s
EDIT2:
Apparently timeouts are not supported yet. Here is my bug report
https://github.com/akka/akka/issues/17732#issuecomment-112315953
When I run my app using akka, it fails with the following exception:
Event Handler specified in config can't be loaded [com.despegar.hasp.impl.DummyLogEventHandler] due to [a06c8d75-0f07-40db-883a-16dc2914934bakka.event.Logging$LoggerInitializationException: Logger log1-DummyLogEventHandler did not respond with LoggerInitialized, sent instead [TIMEOUT]
DummyLogEventHandler is defined as:
class DummyLogEventHandler extends Actor {
def receive = {
case InitializeLogger(_) => sender ! LoggerInitialized
case Error(cause, logSource, logClass, message) =>
case Warning(logSource, logClass, message) =>
case Info(logSource, logClass, message) =>
case Debug(logSource, logClass, message) =>
}
}
My configuration has the following lines:
event-handlers = ["my.app.DummyLogEventHandler"]
event-handler-startup-timeout = 15s
But I've also tried with the default logger:
event-handlers = []
and with slf4j (my app is using the logback backend and logging works ok):
event-handlers = ["akka.event.slf4j.Slf4jEventHandler"]
Neither of those event-handlers nor incrementing the timeout to 60 seconds have worked so far. Moreover, the timeout is thrown sporadically. When I ran a test suite, the exception is thrown in different tests every time.
Can you help me find a solution?
Thanks,
Alex.
Try to set a larger timeout for the logger initialization :
For example in the conf file :
akka {logger-startup-timeout = 25s }
checkout the doc
http://doc.akka.io/docs/akka/snapshot/general/configuration.html
After discussion on akka-user the following was found.
This problem is a symptom of configuring the akka.actor.default-dispatcher to be of type = BalancingDispatcher, which cannot work, see the docs for that dispatcher type:
All the actors share a single Mailbox that they get their messages from.
It is assumed that all actors using the same instance of this dispatcher can process all messages that have been sent to one of the actors; i.e. the actors belong to a pool of actors, and to the client there is no guarantee about which actor instance actually processes a given message.
Sharability: Actors of the same type only
In my case, adding the timeout setting in conf does not work, but setting system property directly like below works.
System.setProperty("akka.logger-startup-timeout", "30s")