I am trying to do asynchronous http calls with akka streams.
This is what I tried.
Source(listEndpoints)
.mapAsync(20)(endpoint => Future(Await.result(request(HttpMethods.POST, endpoint, List(authHeader)), timeout)))
.runWith(Sink.seq[HttpResponse])
I am using akka-http within the request method and it returns Future[HttpResponse]
I think I am abusing Future here. The code above would give me a Future[List[HttpResponse]] and I have to use Await again to get a List[HttpResponse]. Is there a more elegant way to timeout functions within mapAsync?
Assuming your request method at some point does
Http().singleRequest
to get a Future[HttpResponse], you can pass a timeout for the request through:
// inside def request(...), will probably need to add a timeout argument here
val request = ??? // Build the HttpRequest
Http().singleRequest(
request = request,
settings = ConnectionPoolSettings.default.withMaxConnectionLifetime(timeout)
Then your stream would just be
Source(listEndpoints)
.mapAsync(request(...))
.runWith(Sink.seq[HttpResponse])
and you'd only need to Await at the "end of the world" for the Future[List[HttpResponse]] to complete.
You can also change the default max connection lifetime with akka.http.host-connection-pool.max-connection-lifetime in application.conf
Related
I am new in scala. Currently, to create rest API I'm using the spray. Now I want to consume API from another server. I'm calling this API on every key-stroke from UI. I'm aborting the request using AbortController if the user keeps typing and the previous request is in pending status. To hit other server request, I'm using spray-client. Which goes something like this:
def completeService(completeRequest: CompleteRequest): Future[HttpResponse] = {
val pipeline: HttpRequest => Future[HttpResponse] = sendReceive ~> unmarshal[HttpResponse]
val response: Future[HttpResponse] = pipeline(Post(someremoteUrl.concat("complete"), completeRequest)
~> addHeader("demo", "test"))
response
}
I'm able to access using this above code. And I'm getting the expected response. But the thing is time-consuming. It creates new TCP connection and communicates with the host, hits the API gives the response then terminates the connection. Here while terminating it sits idle for sometimes and didn't accept a new connection.
Is there any alternative way to do this?
You can create rest request using akka http client. you can see detailed example here
I have found an [example][1] where akka-http is used with Source.single to make a request. Now I'd like to use Source.tick to implement polling requests which are execute every X seconds like this:
import akka.http.scaladsl.model._
import scala.concurrent.duration._
val request: HttpRequest = RequestBuilding.Get(Uri("http://api.someSite.com"))
val source: Source[HttpRequest, Cancellable] = Source.tick(1.seconds, 1.seconds, request)
val sourceWithDest = source.via(Http().superPool())
However, I get a compile error in the last line which I cant resolve(Type mismatch). Any ideas on what I am doing wrong or suggestions for alternatives?
[1]: https://gist.github.com/steinybot/a1f79fe9a67693722164
As per the docs:
The Flow returned by Http().superPool(...) is very similar to the one
from the Host-Level Client-Side API, so the Using a Host Connection
Pool section also applies here.
And then
The “pool client flow” returned by
Http().cachedHostConnectionPool(...) has the following type:
Flow[(HttpRequest, T), (Try[HttpResponse], T), HostConnectionPool]
This is to give client-side code the possibility to implement some logic to match the original requests to the corresponding response. Assuming you don't need this kind of behaviour in your case, you can always proceed by appending NotUsed to your request before feeding it to the pool flow. E.g.
val sourceWithDest: Source[Try[HttpResponse], Cancellable] =
source.map(req ⇒ (req, NotUsed)).via(Http().superPool[NotUsed]()).map(_._1)
I'm new to Play and Scala. I'm trying to build an Application using Play and Scala. I need to make post call internally to get data from my server. But this should be synchronous. After getting the data from this post request, I need to send that data to front end. I've seen many resources but all are asynchronous. Please help me.
I'm fetching data from DB and then should return the data as response.
DB is at remote server not in the hosted server.
I think you should not block anyway.
def action = Action.async {
WS.url("some url")
.post(Json.toJson(Map("query"->query)))
.map { response =>
val jsonResponse = response.json
// in this place you have your response from your call
// now just do whatever you need to do with it,
// in this example I will return it as `Ok` result
Ok(jsonResponse)
}
}
Just map the result of your call and modify it staying in context of Future and use Action.async that takes a Future.
If you really want to block use Await.result(future, 5 seconds), importing
import scala.concurrent.duration._
import scala.concurrent.Await
See docs for Await here
All requests are asynchronous but nothing prevents you from waiting the response with await in your code.
val response = await(yourFutureRequest).body
The line written above will block until the future has finished.
I'm developing an application using Play framework in scala. I have to handle the below use case in my application.
For a particular request from the browser to the play server the Play server should make an http request to some external server (for Eg: somesite.com) and send the response from this request back to the web browser.
I have written the below code to send the request to external serever in the controller.
val holder = WS.url("http://somesite.com")
val futureResponse = holder.get
Now how do I send back the response recieved from "somesite.com" back to the browser?
There's an example in the Play documentation for WS, under Using in a controller; I've adapted it to your scenario:
def showSomeSiteContent = Action.async {
WS.url("http://somesite.com").get().map { response =>
Ok(response.body)
}
}
The key thing to note is the idiomatic use of map() on the Future that you get back from the get call - code inside this map block will be executed once the Future has completed successfully.
The Action.async "wrapper" tells the Play framework that you'll be returning a Future[Response] and that you want it to do the necessary waiting for things to happen, as explained in the Handling Asynchronous Results documentation.
You may also be interested in dynamically returning the status and content type:
def showSomeSiteContent = Action.async {
WS.url("http://somesite.com").get().map { response =>
Status(response.status)(response.body).as(response.ahcResponse.getContentType)
}
}
Dynamic status could help if the URL/service you call fails to answer correctly.
Dynamic content type can be handy if your URL/service can return different content HTML/XML... depending on some dynamic parameter.
I need to write a client that continuously polls a web server for commands. A response from the server indicates that a command is available (in which case the response contains the command) or an instruction that no command is available, and you should fire off a new request for incoming commands.
I'm trying to figure out how to do it using spray-client and Akka, and I can think of ways to do it, but none of them look like they're the idiomatic way to get it done. So the question is:
what's the most sensible way to have a couple of threads poll the same web server for incoming commands and hand the commands off to an actor?
This example uses spray-client, scala futures, and Akka scheduler.
Implementation varies depending on desired behavior (execute many requests in parallel at the same time, execute in different intervals, send responses to one actor to process one response at a time, send responses to many actors to process in parallel... etc).
This particular example shows how execute many requests in parallel at the same time, and then do something with each result as it completes, without waiting for any other requests that were fired off at the same time to complete.
The code below will execute two HTTP requests every 5 seconds to 0.0.0.0:9000/helloWorld and 0.0.0.0:9000/goodbyeWorld in parallel.
Tested in Scala 2.10, Spray 1.1-M7, and Akka 2.1.2:
Actual scheduling code that handles periodic job execution:
// Schedule a periodic task to occur every 5 seconds, starting as soon
// as this schedule is registered
system.scheduler.schedule(initialDelay = 0 seconds, interval = 5 seconds) {
val paths = Seq("helloWorld", "goodbyeWorld")
// perform an HTTP request to 0.0.0.0:9000/helloWorld and
// 0.0.0.0:9000/goodbyeWorld
// in parallel (possibly, depending on available cpu and cores)
val retrievedData = Future.traverse(paths) { path =>
val response = fetch(path)
printResponse(response)
response
}
}
Helper methods / boilerplate setup:
// Helper method to fetch the body of an HTTP endpoint as a string
def fetch(path: String): Future[String] = {
pipeline(HttpRequest(method = GET, uri = s"/$path"))
}
// Helper method for printing a future'd string asynchronously
def printResponse(response: Future[String]) {
// Alternatively, do response.onComplete {...}
for (res <- response) {
println(res)
}
}
// Spray client boilerplate
val ioBridge = IOExtension(system).ioBridge()
val httpClient = system.actorOf(Props(new HttpClient(ioBridge)))
// Register a "gateway" to a particular host for HTTP requests
// (0.0.0.0:9000 in this case)
val conduit = system.actorOf(
props = Props(new HttpConduit(httpClient, "0.0.0.0", 9000)),
name = "http-conduit"
)
// Create a simple pipeline to deserialize the request body into a string
val pipeline: HttpRequest => Future[String] = {
sendReceive(conduit) ~> unmarshal[String]
}
Some notes:
Future.traverse is used for running futures in parallel (ignores order). Using a for comprehension on a list of futures will execute one future at a time, waiting for each to complete.
// Executes `oneThing`, executes `andThenAnother` when `oneThing` is complete,
// then executes `finally` when `andThenAnother` completes.
for {
oneThing <- future1
andThenAnother <- future2
finally <- future3
} yield (...)
system will need to be replaced with your actual Akka actor system.
system.scheduler.schedule in this case is executing an arbitrary block of code every 5 seconds -- there is also an overloaded version for scheduling messages to be sent to an actorRef.
system.scheduler.schedule(
initialDelay = 0 seconds,
frequency = 30 minutes,
receiver = rssPoller, // an actorRef
message = "doit" // the message to send to the actorRef
)
For your particular case, printResponse can be replaced with an actor send instead: anActorRef ! response.
The code sample doesn't take into account failures -- a good place to handle failures would be in the printResponse (or equivalent) method, by using a Future onComplete callback: response.onComplete {...}
Perhaps obvious, but spray-client can be replaced with another http client, just replace the fetch method and accompanying spray code.
Update: Full running code example is here:
git clone the repo, checkout the specified commit sha, $ sbt run, navigate to 0.0.0.0:9000, and watch the code in the console where sbt run was executed -- it should print Hello World!\n'Goodbye World! OR Goodbye World!\nHelloWorld! (order is potentially random because of parallel Future.traverse execution).
You can use HTML5 Server-Sent Events. It is implemented in many Scala frameworks. For example in xitrum code looks like:
class SSE extends Controller {
def sse = GET("/sse") {
addConnectionClosedListener {
// The connection has been closed
// Unsubscribe from events, release resources etc.
}
future {
respondEventSource("command1")
//...
respondEventSource("command2")
//...
}
}
SSE is pretty simple and can be used in any software not only in browser.
Akka integrated in xitrum and we use it in similar system. But it uses netty for async server it is also good for processing thousands of request in 10-15 threads.
So in this way your client will keep connection with server and reconnect when connection will be broken.