I am making a web service request that produces a Future. Like so (a simplified reproduction):
import play.api.libs.ws.WSClient
import scala.concurrent.{ExecutionContext, Future}
import scala.concurrent.duration._
class Service(wsClient: WSClient)(implicit ec: ExecutionContext) {
def callWebService() = {
val req: WSRequest = wsClient.url(...).withRequestTimeout(180 seconds)...
val respFuture:Future[Response] = req.execute()
}
}
The web service being invoked gets 180 seconds to respond before WSClient gives up.
This Service class' client now calls it as service.callWebService(). When the third party web service takes > 120 seconds, instead of waiting for 180 seconds, the future timesout at 120 seconds (java.util.concurrent.TimeoutException: Read timeout to localhost/127.0.0.1:8081 after 120000 ms thrown in application-akka.actor.default-dispatcher-3 thread).
Appreciate any pointers on how to increase the 120 seconds used by the default dispatcher to 180 seconds.
Note:
The app uses Akka actors elsewhere. But there are no actors involved in this control flow.
This post comes close but doesn't help.
Play WS includes a global request timeout, which defaults to 2 minutes.
Putting the following in your application's config (by default application.conf) should rectify things.
play.ws.timeout.request = 3 minutes
play.ws.timeout.idle = 3 minutes
Related
I searched the other StackOverflow question/answers towards this error, but couldn't find a hint for solving this problem.
The Akka HTTP application runs for like 5 hours under high workload without problems, and than I start to get multiple:
Response entity was not subscribed after 1 second. Make sure to read the response `entity` body or call `entity.discardBytes()` on it -- in case you deal with `HttpResponse`, use the shortcut `response.discardEntityBytes()`. GET /api/name123 Empty -> 200 OK Default(142 bytes)
and later
The connection actor has terminated. Stopping now.
The actor is only sending out API requests and afterwards forwards those responses to another actor if successfully, in case of failure, that request is added back to the todo stack and retried later. This is the main code:
private def makeApiRequest(id: String): Unit = {
val url = UrlBuilder(id)
val request = HttpRequest(method = HttpMethods.GET, uri = url)
val f: Future[(StatusCode, String)] = Http(context.system)
.singleRequest(request)
.flatMap(_.toStrict(2.seconds))
.flatMap { resp =>
Unmarshal(resp.entity).to[String].map((resp.status, _))
}
context.pipeToSelf(f) {
case Success(response) =>
API_HandleResponseSuccess(id, response._1, response._2)
case Failure(e) =>
API_HandleResponseFailure(id, e.getMessage)
}
}
I don't really understand why I get the "Response entity was not subscribed..." error, as I do Unmarshal(resp.entity).to[String] and thereby would think, that no .DiscardEntityBytes() is needed, or does it needs to be still included somehow?
Side information: Also confusing to me, why the CPU performance doesn't stay constant.
Within the actor do I track the response times of each request and calculate the amount of max. parallel requests possible to handle with the given hardware conditions (restricted to a max max of 120 though) on a regular basis to account for API response time fluctuations, so there should be always enough room to make the requests without starving for that actor. In addition would that be the respective application.conf:
dispatcher-worker-io {
type = Dispatcher
executor = "thread-pool-executor"
thread-pool-executor {
fixed-pool-size = 120
keep-alive-time = 60s
allow-core-timeout = off
}
shutdown-timeout = 60s
throughput = 1
}
...
akka.http.client.host-connection-pool.max-connections = 180
akka.http.client.host-connection-pool.max-open-requests = 256
akka.http.client.host-connection-pool.max-retries = 0
Any ideas on why I after 5 hours without problems start to get those exceptions mentioned above?
or
Has an idea of which part of above shared code might leads to this non-linear CPU performance?
I also made multiple of those long lasting hour runs, and it always ends out like this, somehow it's starving after 5 to 6 hours.
val AkkaVersion = "2.6.15"
val AkkaHttpVersion = "10.2.6"
Directly from the docs (https://doc.akka.io/docs/akka-http/current/client-side/request-level.html):
Always make sure you consume the response entity streams (of type
Source[ByteString,Unit]). Connect the response entity Source to a
Sink, or call response.discardEntityBytes() if you don’t care about
the response entity.
Read the Implications of the streaming nature of Request/Response
Entities section for more details.
If the application doesn’t subscribe to the response entity within
akka.http.host-connection-pool.response-entity-subscription-timeout,
the stream will fail with a TimeoutException: Response entity was not
subscribed after ....
You need to .discardEntityBytes() in case of failure. Right now you only consume it on success.
Perhaps high CPU load is caused by all these unfreed resources on the JVM + retries of all the failures.
Concurrent requests are getting hung. Here's a sample code that I am using to test concurrent requests.
import aiohttp
import asyncio
async def fetch(session, url):
async with session.get(url) as response:
print(await response.text())
async def main(n):
url = "https://httpstat.us/200"
async with aiohttp.ClientSession() as session:
tasks = [asyncio.create_task(fetch(session, url)) for _ in range n]
await asyncio.gather(*tasks)
asyncio.run(main(10))
When I make 10 concurrent requests, the first 4-5 requests are made concurrently and then it gets hung for over 10 seconds and then starts the remaining tasks which get hung again after running 2-3 concurrent requests. If I make 100 concurrent requests, it makes about 25-30 concurrent requests and gets hung and then makes 5-6 requests and gets hung again, it does this until all tasks are complete.
It's taking over two minutes to make 100 requests to https://httpstat.us/200 with aiohttp.
If I don't use a persistent ClientSession and create new ClientSession for every request, then all hundred requests finish within 5 seconds without getting hung.
I am not sure what I am doing here. Any help will be highly appreciated.
I was able to run this code using:
python 3.7, asyncio 3.4.3, aiohttp 3.5.4
import aiohttp
import asyncio
async def fetch(session, url):
async with session.get(url) as response:
print(await response.text())
async def main(n):
url = "https://httpstat.us/200"
async with aiohttp.ClientSession() as session:
tasks = [asyncio.create_task(fetch(session, url)) for _ in range (0,n)]
await asyncio.gather(*tasks)
asyncio.run(main(10))
I have found an [example][1] where akka-http is used with Source.single to make a request. Now I'd like to use Source.tick to implement polling requests which are execute every X seconds like this:
import akka.http.scaladsl.model._
import scala.concurrent.duration._
val request: HttpRequest = RequestBuilding.Get(Uri("http://api.someSite.com"))
val source: Source[HttpRequest, Cancellable] = Source.tick(1.seconds, 1.seconds, request)
val sourceWithDest = source.via(Http().superPool())
However, I get a compile error in the last line which I cant resolve(Type mismatch). Any ideas on what I am doing wrong or suggestions for alternatives?
[1]: https://gist.github.com/steinybot/a1f79fe9a67693722164
As per the docs:
The Flow returned by Http().superPool(...) is very similar to the one
from the Host-Level Client-Side API, so the Using a Host Connection
Pool section also applies here.
And then
The “pool client flow” returned by
Http().cachedHostConnectionPool(...) has the following type:
Flow[(HttpRequest, T), (Try[HttpResponse], T), HostConnectionPool]
This is to give client-side code the possibility to implement some logic to match the original requests to the corresponding response. Assuming you don't need this kind of behaviour in your case, you can always proceed by appending NotUsed to your request before feeding it to the pool flow. E.g.
val sourceWithDest: Source[Try[HttpResponse], Cancellable] =
source.map(req ⇒ (req, NotUsed)).via(Http().superPool[NotUsed]()).map(_._1)
My colleague and I have been puzzled with the different behaviour the DistributedPubSubMediator has for subscribing/unsubscribing directly or via a proxy Actor. We put together a test to show the different result below.
From our understanding, the ActorRef.forward should pass in the original sender, hence whether the message is sent directly to the Mediator or via a proxy Actor should not matter. Ie. http://www.scala-lang.org/api/current/index.html#scala.actors.ActorRef.
To work around, we have to extend the DIstributedPubSubMediator class and include the logic DistributedPubSubMediator object already provides. Ideally, we'd prefer to use the object directly and revert our code.
This seems like a bug. Does anyone know the underlying reason for this unusual behaviour? Please help...
[22-Oct-2013] Test is updated based on Roland's answer (Thank you) and added expectMsgType on SubscriberAck and UnsubscribeAck. We now receive the SubscribeAck, but strangely not the UnSubscribeAck. It is not a major issue but we would like to know why.
Another question, if we may ask altogether, is whether it is good practice to Subscribe remote actors to the DistributedPubSubMediator via proxy Actor running in the same ActorSystem?
At the moment we have:
The subscribing App discovers the publishing App (in non-Akka-way) and gets Cluster address.
The remote subscriber uses this address and the known proxy actor's path to send Identity request.
The remote subscriber gets the ActorIdentity response and then Subscribes/Unsubscribes via this (remote) proxy.
On the publisher App Subscribe/Unsubscribe messages are forwarded to the DistributedPubSubMediator and it is used to publish subsequent business messages.
We are not joining the Cluster as per Akka Reactor pubsub chat client example (ie. only using the DistributedPubSubMediator to publish) because we need to handle Failover on the Publisher side.
[5-Nov-2013] Added a test on Send message. It does not seem to work and we haven't figured it out yet.
package star.common.pubsub
import org.scalatest.{BeforeAndAfterAll, FunSuite}
import org.junit.runner.RunWith
import akka.contrib.pattern.DistributedPubSubExtension
import akka.contrib.pattern.DistributedPubSubMediator._
import akka.testkit.TestKit
import akka.actor.{Actor, ActorSystem, ActorRef, Props}
import scala.concurrent.duration._
import com.typesafe.config.ConfigFactory
object MediatorTest {
val config = ConfigFactory.parseString(s"""
akka.actor.provider="akka.cluster.ClusterActorRefProvider"
akka.remote.netty.tcp.port=0
akka.extensions = ["akka.contrib.pattern.DistributedPubSubExtension"]
""")
}
#RunWith(classOf[org.scalatest.junit.JUnitRunner])
class MediatorTest extends TestKit(ActorSystem("test", MediatorTest.config)) with FunSuite {
val mediator = DistributedPubSubExtension(system).mediator
val topic = "example"
val message = "Published Message"
// val joinAddress = Cluster(system).selfAddress
// Cluster(system).join(joinAddress)
test("Direct subscribe to mediator") {
mediator.!(Subscribe(topic, testActor))(testActor)
expectMsgType[SubscribeAck](5 seconds)
mediator.!(Publish(topic, message))(testActor)
expectMsg(2 seconds, message)
mediator.!(Unsubscribe(topic, testActor))(testActor)
expectMsgType[UnsubscribeAck](5 seconds)
mediator ! Publish(topic, message)
expectNoMsg(2 seconds)
}
test("Subscribe to mediator via proxy") {
class Proxy extends Actor {
override def receive = {
case subscribe: Subscribe =>
mediator forward subscribe
case unsubscribe: Unsubscribe =>
mediator forward unsubscribe
case publish: Publish =>
mediator.!(publish)
}
}
val proxy = system.actorOf(Props(new Proxy), "proxy")
proxy.!(Subscribe(topic,testActor))(testActor)
expectMsgType[SubscribeAck](2 seconds)
proxy ! Publish(topic, message)
expectMsg(5 seconds, message)
proxy.!(Unsubscribe(topic,testActor))(testActor)
expectMsgType[UnsubscribeAck](5 seconds)
proxy ! Publish(topic, message)
expectNoMsg(5 seconds)
}
test("Send message to address") {
val testActorAddress = testActor.path.toString
// val system2 = ActorSystem("test", MediatorTest.config)
// Cluster(system2).join(joinAddress)
mediator.!(Subscribe(topic, testActor))(testActor)
expectMsgType[SubscribeAck](5 seconds)
println(testActorAddress) // akka://test/system/testActor1
mediator.!(Publish(topic, message))(testActor)
expectMsg(2 seconds, message)
mediator ! Send(testActorAddress, message, false)
expectMsg(5 seconds, message)
}
}
Two things:
whether or not you use forward does not matter much, since you do not have a useful sender in scope in your test procedure (you are not mixing in ImplicitSender); but this is not the problem
you are not forwarding the Publish message, which is why it does not publish the message
I need to write a client that continuously polls a web server for commands. A response from the server indicates that a command is available (in which case the response contains the command) or an instruction that no command is available, and you should fire off a new request for incoming commands.
I'm trying to figure out how to do it using spray-client and Akka, and I can think of ways to do it, but none of them look like they're the idiomatic way to get it done. So the question is:
what's the most sensible way to have a couple of threads poll the same web server for incoming commands and hand the commands off to an actor?
This example uses spray-client, scala futures, and Akka scheduler.
Implementation varies depending on desired behavior (execute many requests in parallel at the same time, execute in different intervals, send responses to one actor to process one response at a time, send responses to many actors to process in parallel... etc).
This particular example shows how execute many requests in parallel at the same time, and then do something with each result as it completes, without waiting for any other requests that were fired off at the same time to complete.
The code below will execute two HTTP requests every 5 seconds to 0.0.0.0:9000/helloWorld and 0.0.0.0:9000/goodbyeWorld in parallel.
Tested in Scala 2.10, Spray 1.1-M7, and Akka 2.1.2:
Actual scheduling code that handles periodic job execution:
// Schedule a periodic task to occur every 5 seconds, starting as soon
// as this schedule is registered
system.scheduler.schedule(initialDelay = 0 seconds, interval = 5 seconds) {
val paths = Seq("helloWorld", "goodbyeWorld")
// perform an HTTP request to 0.0.0.0:9000/helloWorld and
// 0.0.0.0:9000/goodbyeWorld
// in parallel (possibly, depending on available cpu and cores)
val retrievedData = Future.traverse(paths) { path =>
val response = fetch(path)
printResponse(response)
response
}
}
Helper methods / boilerplate setup:
// Helper method to fetch the body of an HTTP endpoint as a string
def fetch(path: String): Future[String] = {
pipeline(HttpRequest(method = GET, uri = s"/$path"))
}
// Helper method for printing a future'd string asynchronously
def printResponse(response: Future[String]) {
// Alternatively, do response.onComplete {...}
for (res <- response) {
println(res)
}
}
// Spray client boilerplate
val ioBridge = IOExtension(system).ioBridge()
val httpClient = system.actorOf(Props(new HttpClient(ioBridge)))
// Register a "gateway" to a particular host for HTTP requests
// (0.0.0.0:9000 in this case)
val conduit = system.actorOf(
props = Props(new HttpConduit(httpClient, "0.0.0.0", 9000)),
name = "http-conduit"
)
// Create a simple pipeline to deserialize the request body into a string
val pipeline: HttpRequest => Future[String] = {
sendReceive(conduit) ~> unmarshal[String]
}
Some notes:
Future.traverse is used for running futures in parallel (ignores order). Using a for comprehension on a list of futures will execute one future at a time, waiting for each to complete.
// Executes `oneThing`, executes `andThenAnother` when `oneThing` is complete,
// then executes `finally` when `andThenAnother` completes.
for {
oneThing <- future1
andThenAnother <- future2
finally <- future3
} yield (...)
system will need to be replaced with your actual Akka actor system.
system.scheduler.schedule in this case is executing an arbitrary block of code every 5 seconds -- there is also an overloaded version for scheduling messages to be sent to an actorRef.
system.scheduler.schedule(
initialDelay = 0 seconds,
frequency = 30 minutes,
receiver = rssPoller, // an actorRef
message = "doit" // the message to send to the actorRef
)
For your particular case, printResponse can be replaced with an actor send instead: anActorRef ! response.
The code sample doesn't take into account failures -- a good place to handle failures would be in the printResponse (or equivalent) method, by using a Future onComplete callback: response.onComplete {...}
Perhaps obvious, but spray-client can be replaced with another http client, just replace the fetch method and accompanying spray code.
Update: Full running code example is here:
git clone the repo, checkout the specified commit sha, $ sbt run, navigate to 0.0.0.0:9000, and watch the code in the console where sbt run was executed -- it should print Hello World!\n'Goodbye World! OR Goodbye World!\nHelloWorld! (order is potentially random because of parallel Future.traverse execution).
You can use HTML5 Server-Sent Events. It is implemented in many Scala frameworks. For example in xitrum code looks like:
class SSE extends Controller {
def sse = GET("/sse") {
addConnectionClosedListener {
// The connection has been closed
// Unsubscribe from events, release resources etc.
}
future {
respondEventSource("command1")
//...
respondEventSource("command2")
//...
}
}
SSE is pretty simple and can be used in any software not only in browser.
Akka integrated in xitrum and we use it in similar system. But it uses netty for async server it is also good for processing thousands of request in 10-15 threads.
So in this way your client will keep connection with server and reconnect when connection will be broken.