My Play router contains no logic but simply invokes this service class whose job is to invoke a URL, and process the response
class MyServiceImpl #Inject()(ws: WSClient)(implicit ec: ExecutionContext) extends MyService with LazyLogging {
val atomicInteger = new AtomicInteger(0)
override def process(request: Request): Response = {
val requestId = atomicInteger.incrementAndGet()
logger.info(s"Received request $requestId")
val futures: Seq[Future[String]] = request.records.map { record =>
val future = {
val response = ws.url("https://www.whatever.com").get()
response.map(r => {
r.status match {
case 200 => {
logger.info(s"Processed request ${requestId} ${atomicInteger.decrementAndGet()} remaining")
"successful"
}
case other => {
throw new RuntimeException(s"Returned response ${other}")
}
}
})
}
future
}
Response(Future.sequence(futures))
}
}
I am observing that under a high load the atomic integer will report 256 requests in flight. I am using the default Play dispatcher/execution contexts which I thought created a pool of one thread per core, and I have 8 cores on my machine.
How can I limit this to a much smaller number?
I have tried changing the Akka default-dispatcher and changing the Akka http akka.http.server.max-connections setting but it always continues to accept 256 requests.
Related
I am using Akka-http with scala. I've a class MainApp which is initialized at the startup and has a ActorRef.
abstract class MainApp{
def actorSystem: ActorSystem
def myService: ActorRef
def stop(): Unit = {
actorSystem.terminate()
}
}
object MyMain {
def main(args: Array[String]): Unit = {
implicit val system = ActorSystem("my-actor-system")
implicit val executionContext: ExecutionContextExecutor = system.dispatcher
val app = new MainApp {
override def actorSystem: ActorSystem = system
val dbPool: MyDbManagement = new MyDbManagement(someConfig)
override def preProcessorService: ActorRef =
actorSystem.actorOf(
Props(new MyService(actorSystem, dbPool))
.withDispatcher("processing-dispatcher")
)
}
val httpBindingFuture = Http().newServerAt("0.0.0.0",httpPort).bind(app.http.route)
....//start server and add shutDown hooks
The actor class receives a message and looks like the following
class MyService(actorSystem, dbPool){
override def receive: Receive = {
case SendMessage(request) => {
val actorSender = sender()
implicit val executionContext = actorSystem.dispatchers.lookup("db-blocking-dispatcher")
try {
val encryptedReq = dbPool.encrypt(request)
val responseFuture = dbPool.sendToDb(request)
responseFuture.onComplete { //handle error codes and return something like
actorSender ! MyResponse(202, "Accepted")
}
}
}
I've one http route, which asks the message
path("send") {
post {
entity(as[MyPojo]) { request =>
onSuccess(app.MyService ? SendMessage(request)) {
case response: MyResponse => {
complete(response.statusCode, response)
}
}
}
}
}
Problem-Statement: I've created only 1 actor in the application but there are multiple dispatchers(fork-join-executor) which performs multiple tasks and to free the default-dispatcher so that it can accept non-blocking incoming requests. Number of incoming requests are very high.
This application has a memory leak and degrades over time(heap size increases and full GC occurs which takes time and CPU)
I've taken a heap dump when it was in the degraded state.
During the investigation on MAT tool,
the major leak suspect was LocalActorRef
One instance of "akka.actor.LocalActorRef" loaded by "jdk.internal.loader.ClassLoaders$AppClassLoader # 0x609c0ee58" occupies ~7GB (95.80%) bytes.
Class Name | Shallow Heap | Retained Heap
akka.actor.LocalActorRef| 24 | ~ 7 GB
I would like to know what is causing this memory leak?
If there are any more design related questions or suggestions, let me know.
I've gone through the following but none helped:
https://discuss.lightbend.com/t/localactorref-memory-leak-and-full-gc-allocation-failure-causing-gc-to-take-too-long/7216/6
Akka Actor Memory Leak or Management
I've also thought of using Futures but with Futures the application has higher latencies
I have an HTTP Connection Pool that hangs after a couple of hours of running:
private def createHttpPool(host: String): SourceQueue[(HttpRequest, Promise[HttpResponse])] = {
val pool = Http().cachedHostConnectionPoolHttps[Promise[HttpResponse]](host)
Source.queue[(HttpRequest, Promise[HttpResponse])](config.poolBuffer, OverflowStrategy.dropNew)
.via(pool).toMat(Sink.foreach {
case ((Success(res), p)) => p.success(res)
case ((Failure(e), p)) => p.failure(e)
})(Keep.left).run
}
I enqueue items with:
private def enqueue(uri: Uri): Future[HttpResponse] = {
val promise = Promise[HttpResponse]
val request = HttpRequest(uri = uri) -> promise
queue.offer(request).flatMap {
case Enqueued => promise.future
case _ => Future.failed(ConnectionPoolDroppedRequest)
}
}
And resolve the response like this:
private def request(uri: Uri): Future[HttpResponse] = {
def retry = {
Thread.sleep(config.dispatcherRetryInterval)
logger.info(s"retrying")
request(uri)
}
logger.info("req-start")
for {
response <- enqueue(uri)
_ = logger.info("req-end")
finalResponse <- response.status match {
case TooManyRequests => retry
case OK => Future.successful(response)
case _ => response.entity.toStrict(10.seconds).map(s => throw Error(s.toString, uri.toString))
}
} yield finalResponse
}
The result of this function is then always transformed if the Future is successful:
def get(uri: Uri): Future[Try[JValue]] = {
for {
response <- request(uri)
json <- Unmarshal(response.entity).to[Try[JValue]]
} yield json
}
Everything works fine for a while and then all I see in the logs are req-start and no req-end.
My akka configuration is like this:
akka {
actor.deployment.default {
dispatcher = "my-dispatcher"
}
}
my-dispatcher {
type = Dispatcher
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 256
parallelism-factor = 128.0
parallelism-max = 1024
}
}
akka.http {
host-connection-pool {
max-connections = 512
max-retries = 5
max-open-requests = 16384
pipelining-limit = 1
}
}
I'm not sure if this is a configuration problem or a code problem. I have my parallelism and connection numbers so high because without it I get very poor req/s rate (I want to request as fast possible - I have other rate limiting code to protect the server).
You are not consuming the entity of the responses you get back from the server. Citing the docs below:
Consuming (or discarding) the Entity of a request is mandatory! If
accidentally left neither consumed or discarded Akka HTTP will assume
the incoming data should remain back-pressured, and will stall the
incoming data via TCP back-pressure mechanisms. A client should
consume the Entity regardless of the status of the HttpResponse.
The entity comes in the form of a Source[ByteString, _] which needs to be run to avoid resource starvation.
If you don't need to read the entity, the simplest way to consume the entity bytes is to discard them, by using
res.discardEntityBytes()
(you can attach a callback by adding - e.g. - .future().map(...)).
This page in the docs describes all the alternatives to this, including how to read the bytes if needed.
--- EDIT
After more code/info was provided, it is clear that the resource consumption is not the problem. There is another big red flag in this implementation, namely the Thread.sleep in the retry method.
This is a blocking call that is very likely to starve the threading infrastructure of your underlying actor system.
A full blown explanation of why this is dangerous was provided in the docs.
Try changing that and using akka.pattern.after (docs). Example below:
def retry = akka.pattern.after(200 millis, using = system.scheduler)(request(uri))
I'm trying to create an Akka system which would among other things respond to HTTP requests. I created a few actors who exchange messages fine. I can also use akka-http to respond to HTTP requests. The problem is in connecting those two parts.
TL;DR: How to talk to Akka actors during akka-http request processing?
I created a single actor responsible for bringing up the HTTP system:
class HttpActor extends Actor with ActorLogging {
/* implicits elided */
private def initHttp() = {
val route: Route = path("status") { get { complete { "OK" } } }
Http()(context.system).bindAndHandle(route, "localhost", 8080)
}
private var bound: Option[Future[Http.ServerBinding]] = None
override def receive = {
case HttpActor.Init =>
bound match {
case Some(x) => log.warning("Http already bootstrapping")
case None =>
bound = Some(initHttp(watcher))
}
}
}
object HttpActor {
case object Init
}
As you may see, the actor creates the akka-http service on the first message it receives (no reason, really, it could do it in the constructor as well).
Now, during the request processing I need to communicate with some other actors, and I can't get it working.
My approach:
private def initInteractiveHttp() = {
val route: Route = path("status") {
get { complete { "OK" } }
} ~ path("ask") {
get { complete {
// Here are the interesting two lines:
val otherActorResponse = someOtherActor ? SomeMessage
otherActorResponse.mapTo[String]
} }
Http()(context.system).bindAndHandle(route, "localhost", 8080)
}
This would send a SomeMessage to someOtherActor and wait for response prior to completing the Request-Response cycle. As I understand, however, that messages would be sent from the root HttpActor, which is bad and leads nowhere in terms of scalability. Ideally I would create a distinct instance of specialized actor for every request, but this fails due to akka-http typing. Consider the following example:
class DisposableActor(httpContext: HttpContext) {
override def preStart() = {
// ... send some questions to other actors
}
override def receive = {
case GotAllData(x) => httpContext.complete(x)
}
}
class HttpActorWithDisposables {
// there is a `context` value in scope - we're an actor, after all
private def initHttpWithDisposableActors() = {
val route: Route = path("status") {
get { complete { "OK" } }
} ~ path("ask") {
get { httpContext =>
val props = Props(new DisposableActor(httpContext))
val disposableActor = context.actorOf(props, "disposable-actor-name")
// I have nothing to return here
}
}
Http()(context.system).bindAndHandle(route, "localhost", 8080)
}
With this approach, I can force the DisposableActor to call httpContext.complete at some point of time. This should correctly end the Request-Response processing cycle. The Route DSL, however, requires to return a valid response (or Future) inside the get block, so this approach doesn't work.
Your first approach is quite alright, actually.
The ask pattern creates a lightweight, disposable actor for you that waits on the result (non-blockingly) to complete the future. It basically does all the things you want to replicate with the DisposableActor and your main HttpActor is not stressed by this.
If you still want to use a different actor, there is completeWith:
completeWith(instanceOf[String]) { complete =>
// complete is of type String => Unit
val props = Props(new DisposableActor(complete))
val disposableActor = context.actorOf(props, "disposable-actor-name")
// completeWith expects you to return unit
}
And in you actor, call the complete function when you have the result
class DisposableActor(complete: String => Unit) {
override def receive = {
case GotAllData(x) =>
complete(x)
context stop self // don't for get to stop the actor
}
}
I want to do something like the following:
object SprayTest extends App with SimpleRoutingApp {
implicit val system = ActorSystem("my-system")
import system.dispatcher
startServer(interface = "0.0.0.0", port = 8080) {
post {
path("configNetwork") {
entity(as[Config]) { config =>
complete {
// has a response indicating "OK"
// also, restarts the network interface
handleConfig(config)
}
}
}
}
}
}
The problem is that handleConfig reinitializes the network interface, so remote hosts accessing this endpoint never receive their response.
One way to solve this is to run handleConfig in a separate thread and complete the request immediately with some response like "OK". This isn't a good solution however because it introduces a race condition between the future and the request completion (also, it always fails if the future is executed in a "same thread" execution context).
Therefore, an ideal solution would be to attach a callback to a "write response" future and perform the network re-initialization there, after the response has been successfully sent. Is there a way to achieve this in the spray framework?
As a simple example of the race condition, consider the following two examples:
object SprayTest extends App with SimpleRoutingApp {
implicit val system = ActorSystem("my-system")
import system.dispatcher
startServer(interface = "0.0.0.0", port = 8080) {
post {
path("configNetwork") {
entity(as[Config]) { config =>
ctx =>
ctx.complete("OK")
System.exit(0) // empty response due to this executing before response is sent
}
}
}
}
}
object SprayTest extends App with SimpleRoutingApp {
implicit val system = ActorSystem("my-system")
import system.dispatcher
startServer(interface = "0.0.0.0", port = 8080) {
post {
path("configNetwork") {
entity(as[Config]) { config =>
ctx =>
ctx.complete("OK")
Thread.sleep(1000)
System.exit(0) // response is "OK" because of the sleep
}
}
}
}
}
You can use the withAck method on HttpResponse to receive notification when the response is sent on the wire. Here is a sketch of what that would look like in code, though I suspect if you're reconfiguring the low-level network interface then you will need to actually close the http listener and rebind.
case object NetworkReady
class ApiManager extends HttpServiceActor with Directives {
override def receive: Receive = networkReady
private def networkReady: Receive = runRoute(routes) orElse networkManagementEvents
private def networkManagementEvents: Receive = {
case Config =>
context.become(reconfiguringNetwork)
magicallyReconfigureNetwork pipeTo self
}
private def magicallyReconfigureNetwork: Future[NetworkReady] = ???
private def reconfiguringNetwork: Receive = {
case NetworkReady => context.become(networkReady)
case _: HttpRequest => sender() ! HttpResponse(ServiceUnavailable)
case _: Tcp.Connected => sender() ! Tcp.Close
}
private def routes: Route = {
(post & path("configNetwork") & entity(as[Config])) { config =>
complete(HttpResponse(OK).withAck(config))
}
}
}
I want to make a request to a server asynchronously involving actors. Say I have 2 actors:
class SessionRetriever extends Actor {
import SessionRetriever._
def receiver = {
Get =>
val s = getSessionIdFromServer() // 1
sender ! Result(s) // 2
}
def getSessionIdFromServer(): String = { ... } // 3
}
object SessionRetriever {
object Get
object Result(s: String)
}
And
class RequestSender extends Actor {
val sActor = context actorOf Props[SessionRetriever]
def receiver = {
// get session id
val sesId = sActor ! SessionRetriever.Get
val res = sendRequestToServer(sesId)
logToFile(res)
context shutdown sActor
}
def sendRequestToServer(sessionId: String): String = { .... }
}
My questions:
val s = getSessionIdFromServer() // 1
sender ! Result(s) // 2
1) getSessionIdFromServer() does a synchronous request to the server. I think it would be much better make a request asynchronous, correct? So it will return Future[String] instead of a plain String.
2) How do I make asynchronous: by using an AsyncHttpClient (if I recall correctly its name) or by wrapping its synchronous body into Future { } ?
3) Should I use blocking { } block ? If yes, then where exactly: inside its body or here val s = blocking { getSessionIdFromServer() } ?
P.S. I'd like not to use async { } and await { } at this point because they are quite high level functions and after all they are build on top on Futures.
you might try this non-blocking way
def receive = {
Get =>
//assume getSessionIdFromServer() run aysnchronize
val f: Future[String] = getSessionIdFromServer()
val client = sender //keep it local to use when future back
f onComplete {
case Success(rep) => client ! Result(rep)
case Failure(ex) => client ! Failed(ex)
}
}
1) If getSessionIdFromServer() is blocking then you should execute it asynchronously from your receive function, otherwise your actor will block each time it receives a new request and will always wait until it receives a new session before processing the next request.
2) Using a Future will "move" the blocking operation to a different thread. So, your actor will not be blocked and will be able to keep processing incoming requests - that's good -, however you are still blocking a thread - not so great. Using the AsyncHttpClient is a good idea. You can explore other non-blocking httpClient, like PlayWebService.
3) I am not quite familiar with blocking so not sure I should advise anything here. From what I understand, it will tell the thread pool that the operation is blocking and that it should spawn a temporary new thread to handle it - this avoids having all your workers being blocked. Again, if you do that your actor will not blocked, but you are still blocking a thread while getting the session from the server.
To summarize: just use an async http client in getSessionIdFromServer if it is possible. Otherwise, use either Future{} or blocking.
To do an asynchronous call with AsyncHttpClient you could deal with the java Future via a scala Promise.
import scala.concurrent.Future
import com.ning.http.client.AsyncHttpClient
import scala.concurrent.Promise
import java.util.concurrent.Executor
object WebClient {
private val client = new AsyncHttpClient
case class BadStatus(status: Int) extends RuntimeException
def get(url: String)(implicit exec: Executor): Future[String] = {
val f = client.prepareGet(url).execute();
val p = Promise[String]()
f.addListener(new Runnable {
def run = {
val response = f.get
if (response.getStatusCode / 100 < 4)
p.success(response.getResponseBodyExcerpt(131072))
else p.failure(BadStatus(response.getStatusCode))
}
}, exec)
p.future
}
def shutdown(): Unit = client.close()
}
object WebClientTest extends App {
import scala.concurrent.ExecutionContext.Implicits.global
WebClient get "http://www.google.com/" map println foreach (_ => WebClient.shutdown())
}
And then deal with the future completion via a callback.
Credit for the code to the awesome reactive programming course at Coursera.