In akka-http routing I can return Future as a response that implicitly converts to ToResponseMarshaller.
Is there some way to handle timeout of this future? Or timeout of connection in route level? Or one way is to use Await() function?
Right now client can wait response forever.
complete {
val future = for {
response <- someIOFunc()
entity <- someOtherFunc()
} yield entity
future.onComplete({
case Success(result) =>
HttpResponse(entity = HttpEntity(MediaTypes.`text/xml`, result))
case Failure(result) =>
HttpResponse(entity = utils.getFault("fault"))
})
future
}
Adding a timeout to an asynchronous operation means creating a new Future that is completed either by the operation itself or by the timeout:
import akka.pattern.after
val future = ...
val futureWithTimeout = Future.firstCompletedOf(
future ::
after(1.second, system.scheduler)(Future.failed(new TimeoutException)) ::
Nil
)
The second Future could also hold a successful result that replaces the error, depending on what exactly it is that you want to model.
As a side note: the presented code sample contains dead code, registering an onComplete handler on a Future only makes sense for side-effects but you seem to want to transform the Future’s value and create an HttpEntity from it. That should be done using map and recover:
future
.map(result => HttpResponse(entity = HttpEntity(MediaTypes.`text/xml`, result)))
.recover { case ex => HttpResponse(entity = utils.getFault("fault")) }
This would then be the overall return value that is passed to the complete directive.
Related
I am using Akka-hhtp (scala) to send multiple http batch requests asynchronously to an API and wondering what is the right way to handle exceptions when the response code is other than 200 OK.
Below is some pseudo-code to demonstrate my point.
/* Using For comprehension here because the server API has restriction on the amount of data we can send and the time it takes them to process each request. So they require us to send multiple mini requests instead. If one of those fails, then our entire job should fail.*/
val eventuallyResponses = for {
batches <- postBatch(payload)
} yield batches
val eventualResponses = Future.sequence(eventuallyResponses)
/* Do I need to recover here? If I don't, will the actor system terminate? */
eventualResponses.recover { case es =>
log.warn("some message")
List()
}
/* As I said I need to wait for all mini batch requests to complete. If one response is different than 200, then the entire job should fail. */
val result = Await.result(eventualResponses, 10.minutes)
actorSystem.terminate().oncomplete{
case Success(_) =>
if (result.isEmpty) =>
/* This doesn't seem to interrupt the program */
throw new RuntimeException("POST failed")
} else {
log.info("POST Successful")
}
case Failure(ex) =>
log.error("error message $ex")
throw ex
}
def postBatch(payload) = {
val responseFuture: Future[HttpResponse] = httpClient.post(payload)
responseFuture.flatMap{ res =>
res.status match {
case StatusCodes.OK => Future.successful(res)
case _ => Future.failed(new RuntimeException("error message"))
}
}
}
The above code throws exception when we receive StatusCodes different than OK. It does go through the branch of result.isEmpty true, but it doesn't seem to stop/interrupt the execution of the program. I need it to do that, as this is scheduled as an Autosys job, and I need to make the job fail if at least one of the batch requests returns different response than 200 OK.
If I don't recover and let the exception be thrown then (when we receive non 200 status code), will the Actor System be terminated properly?
Do you know of a good way to do the above?
Thanks :)
As far as I understand your question you need to throw an exception from main body if some responses haven't status 200.
def postBatch(payload: HttpRequest)(implicit system: ActorSystem, ec: ExecutionContext): Future[HttpResponse] = {
Http().singleRequest(payload).flatMap(response => response.status match {
case StatusCodes.OK => Future.successful(response)
case _ => Future.failed(new RuntimeException("error message"))
})
}
val reuests: List[HttpRequest] = List(...)
/*
You don't need for comprehension here because
val eventuallyResponses = for {
batches <- postBatch(payload)
} yield batches
is equal to
val eventuallyResponses = postBatch(payload)
For comprehension doesn't process recursive sending. If you need it you should write it yourself by flatMap on each request future.
*/
val eventualResponses: Future[List[HttpResponse]] =
Future.sequence(reuests.map(postBatch)) //also, its better to add some throttling logic here
//as far as i understand you need to wait for all responses and stop the actor system after that
Await.ready(eventualResponses, 10 minutes) //wait for all responses
Await.ready(actorSystem.terminate(), Duration.Inf) //wait for actor system termination
//because of Await.ready(eventualResponses, 10 minutes) we can match on the future value and expect that it should be completed
eventualResponses.value match {
case Some(Success(responses)) =>
log.info("All requests completed")
case Some(Failure(exception)) =>
log.error("Some request failed")
throw exception //rethrow this exception
case None =>
log.error("Requests timed out")
throw RuntimeException("Requests timed out")
}
I have the following code snippet that I use to read a record from the database and I'm using ReactiveMongo for this.
val futureList: Future[Option[BSONDocument]] = collection.find(query).cursor[BSONDocument].headOption
val os: Future[Option[Exam]] = futureList.map {
(list: Option[BSONDocument]) => list match {
case Some(examBSON) => {
val id = examBSON.getAs[Int]("id").get
val text = examBSON.getAs[String]("text").get
val description = examBSON.getAs[String]("description").get
val totalQuestions = examBSON.getAs[Int]("totalQuestions").get
val passingScore = examBSON.getAs[Int]("passingScore").get
Some(Exam(id, text, description, totalQuestions, passingScore))
}
case None => None
}
}.recover {
case t: Throwable => // Log exception
None
}
I do not want to change my method signature to return a Future. I want to get the value inside the Future and return it to the caller.
You need then to block using the awaitable object:
import scala.concurrent.duration._
val os: Future[Option[Exam]] = ???
val result = Await.result(os, 10 seconds)
result.getOrElse(/* some default */)
Note that blocking will block the thread until the future is completed or the timeout expires and an exception is thrown, note also that this kinda defeats the purpose of having async computation, but it may be ok depending on your use case.
If you don't need the result immediately you can attach a callback using onComplete
os onComplete {
case Success(someOption) => myMethod(someOption)
case Failure(t) => println("Error)
}
Note that onComplete will be fired only when the future is completed so the result is not immediately accessible, also the return type is Unit.
A piece of code using Play Framework 2.3
def signup = Action(parse.json) { implicit request =>
val email = (request.body \ "email").asOpt[String]
val holder =
WS.url("https://foobar.com/foo")
.withAuth("1234","5678",WSAuthScheme.BASIC)
val data = Json.obj(
"email" -> email,
)
holder.post(data)
Ok("OK")
}
This code will perform the post() when run locally on my machine, but not reliably on remote machines. From my understanding the post() call should create and return a future, signup() returns Ok(), the future is run on the default ExecutionContext and at some point completes and is cleaned up. Apparently this is not the case.
I have since changed the function to be an Action.async and am now waiting on the Future:
val res = holder.post(data)
res.map( x => Ok(Json.obj("status" -> "OK"))) recover {
case (e: Exception) =>
InternalServerError(Json.obj("status" -> "Not OK."))
}
This works reliably. Hence, my questions:
Why does the first version not work reliably?
How can I in general "fork off" a long running procedure, without intention of waiting on it, purely for its side-effects, if not like I did in version 1?
It's near impossible to say why the Future isn't completing "reliably", but I'd hazard a guess that there's something wrong with the web service you're trying to reach. SSL problems, timeouts not being reached,.. the possibilities are large without knowing what exception is being thrown (if any?).
If you don't want to wait on the WS call, you can use the onComplete callback:
val res = holder.post(data)
res.onComplete {
case Success(response) =>
// Create some side-effect to the response
case Failure(e) =>
// The `Future` failed.
}
I'm using the scala dispatch (0.11.0) library to send an HTTP GET request to a remote server. I want to wait for the response before executing the code which follows the request.
My request is of the form :
val req = :/("myurl.com") <:< myheaders OK as.Response(identity)
If I write :
val future = http(req)
future()
var res: String = null
future onComplete {
case Success(r) => res = r.getResponseBody
case _ => println("KO")
}
println(res)
I get null.
This is also the case if I write :
val future = http(req)
var res: String = null
while (!future.isCompleted) {
Thread.sleep(1000)
}
future onComplete {
case Success(r) => res = r.getResponseBody
case _ => println("KO")
}
println(res)
But with the following code :
val future = http(req)
var res: String = null
future onComplete {
case Success(r) => res = r.getResponseBody
case _ => println("KO")
}
while (!future.isCompleted) {
Thread.sleep(1000)
}
println(res)
I get the expected response.
Does someone understand this ?
It seems to me that calling Thread.sleep is not a good thing, could someone give me a hint on how I should handle this problem correctly ?
EDIT: #Randal Schulz thank you for your help, but as you posted in comments, I can't validate your answer.
As my problem was to wait (and do nothing else) until I get a valid response to an HTTP GET request, I think a satisfying way to do it is using Await.result. I removed the side-effect from my code. I used the option method to deal with Future failures (as I was interested only in successes), and I dealt with time out exception the classic way.
I think I could do it as wheaties mentionned, staying in Future, but I need more practice...
TL;DR
The best advice I can give you for working in an asynchronous work flow is that what goes into a Future stays in a Future.
Answer
The issue is that you have no idea when the Future will complete so if you want to use an asynchronous process, you're going to have to write in an asynchronous manner. The code as you have written never stops or blocks on the Future you create so the minute it creates the Future and hand it off to another thread, the current thread is free to then evaluate the res variable.
Hence, place most of what you're doing in a flow like follows:
myFuture map (func1) map (func2) map (func3) onComplete{
case Success(value) => println(value.getResponseBody)
case _ => println('KO')
}
Don't attempt to access something via side-effect like you are.
If you're really clever and you have several Future you can compose them:
val f1 = myFuture map(func1)
val f2 = myOtherFuture map(func2) map (func3)
val f3 = for{
v1 <- f1
v2 <- f2
} yield functionTakingBoth(v1, v2)
f3 onComplete{
//and do stuff here
}
I finally managed to write what I wanted using futures :
def loop(): Future[String] = {
val future = http(req).option
future flatMap ((x: Option[Response]) => x match {
case Some(rep) => rep.getResponseBody
case None => loop()
}
}
Now I can use the result of this function without explicitely waiting for the response to come.
I have an Akka actor responsible of handling http calls. I use scala dispatch to send multiple HTTP requests over an API:
urls.foreach { u
val service = url(u)
val promise = Http(service OK as.String).either
for(p <- promise)
{
p match
{
case Left(error) =>
faultHandler(error)
case Right(result) =>
resultHandler(result)
}
}
In the resultHandlerfunction, I increment an instance variable nbOfResults and compare to the number of calls I have done.
def resultHandler(result:String)
{
this.nbOfResults++
...
if(nbOfResults == nbOfCalls)
// Do something
}
Is it safe ? May the nbOfResultsvaraible be accessed at the same time if two calls return their results simultaneously ?
For now, I believed that the actor is more or less equivalent to a thread and therefore the callback functions are not executed concurrently. Is it correct ?
Here is a variant of Alexey Romanov response using only dispatch :
//Promises will be of type Array[Promise[Either[Throwable, String]]]
val promises = urls.map { u =>
val service = url(u)
Http(service OK as.String).either
}
//Http.promise.all transform an Iterable[Promise[A]] into Promise[Iterable[A]]
//So listPromise is now of type Promise[Array[Either[Throwable, String]]]
val listPromise = Http.promise.all(promises)
for (results <- listPromise) {
//Here results is of type Array[Either[Throwable, String]]
results foreach { result =>
result match {
Left(error) => //Handle error
Right(response) => //Handle response
}
}
}
There is a far better way:
val promises = urls.map {u =>
val service = url(u)
val promise = Http(service OK as.String).either
}
val listPromise = Future.sequence(promises)
listPromise.onComplete { whatever }
I agree with Alexey Romanov on his answer. Whatever way you choose to synchronize your http requests beware of the way your are processing the promises completion. Your intuition is correct in that concurrent access may appear on the state of the actor. The better way to handle this would be to do something like this:
def resultHandler(result: String) {
//on completion we are sending the result to the actor who triggered the call
//as a message
self ! HttpComplete(result)
}
and in the actor's receive function:
def receive = {
//PROCESS OTHER MESSAGES HERE
case HttpComplete(result) => //do something with the result
}
This way, you make sure that processing the http results won't violate the actor's state from the exterior, but from the actor's receive loop which is the proper way to do it
val nbOfResults = new java.util.concurrent.atomic.AtomicInteger(nbOfCalls)
// After particular call was ended
if (nbOfResults.decrementAndGet <= 0) {
// Do something
}
[EDIT] Removed old answer with AtomicReference CAS - while(true), compareAndSet, etc