I have a for comprehension of futures that all make remote service calls (REST, RPC, etc...). If one of those future fails, will the entire for comprehension fail?
For instance, let's say I have two futures that make service calls
val service1Future: Future[Response] = ...
val service2Future: Future[Response] = ...
Let's say service2Future also has some recoverWith that throws an exception PartialFunction[Throwable, Future[U]].
If service2Future fails, I don't really care. Can I enforce that in my for comprehension?
for {
service1Response <- service1Future
service2Response <- service2Future
} yield {
// do stuff with service1Response, but I want to get here even if service2Future fails
}
Yes, the entire for-comprehension will fail if the second future fails as written. Would something as simple as this work?
for {
service1Response <- service1Future
service2Response <- service2Future.recoverWith(...)
} yield {
...
}
One way would be to return Future[Try[Response]] instead of Future[Response]. In recoverWith you would return Future(Failure(e)). You can then pattern match in the yield to see if it was successful or not.
Related
I'm trying to use a for comprehension to both run some futures in order and merged results, but also kick off a separate thread after one of those futures completes and not care about the result (basically used to fire some logging info)
I've played around a bit with this with some thread sleeps and it looks like whatever i'm throwing inside the for block will end up blocking the thread.
private def testFunction(): EitherT[Future, Error, Response] =
for {
firstRes <- EitherT(client.getFirst())
secondRes <- EitherT(client.getSecond())
// Future i want to run on a separate async thread outside the comprehension
_ = runSomeLogging(secondRes)
thirdRes <- EitherT(client.getThird(secondRes.value))
} yield thirdRes
def runSomeLogging(): Future[Either[Error, Response]] =
Thread.sleep(10000)
Future.successful(Right(Response("123")))
}
So this above code will wait the 10 seconds before returning the thirdRes result. My hope was to kick off the runSomeLogging function on a separate thread after the secondRes runs. I thought the usage of = rather than <- would cause that, however it doesn't.
The way I am able to get this to work is below. Basically I run my second future outside of the comprehension and use .onComplete on the previous future to only run my logging if certain conditions were meant from the above comprehension. I only want to run this logging function if the secondRes function is successful in my example here.
private def runSomeLogging(response: SecondRes) =
Thread.sleep(10000)
response.value.onComplete {
case Success(either) =>
either.fold(
_ => { },
response => { logThing() }
)
case _ =>
}
private def testFunction(): EitherT[Future, Error, Response] =
val res = for {
firstRes <- EitherT(client.getFirst())
secondRes <- EitherT(client.getSecond())
thirdRes <- EitherT(client.getThird(secondRes.value))
} yield thirdRes
runSomeLogging(res)
return res
This second example works fine and does what I want, it doesn't block the for comprehension for 10 seconds from returning. However, because there are dependencies of this running for certain pieces of the comprehension, but not all of them, I was hoping there was a way to kick off the job from within the for comprehension itself but let it run on its own thread and not block the comprehension from completing.
You need a function that starts the Future but doesn't return it, so the for-comprehension can move on (since Future's map/flatMap functions won't continue to the next step until the current Future resolves). To accomplish a "start and forget", you need to use a function that returns immediately, or a Future that resolves immediately.
// this function will return immediately
def runSomeLogging(res: SomeResult): Unit = {
// since startLoggingFuture uses Future.apply, calling it will start the Future,
// but we ignore the actual Future by returning Unit instead
startLoggingFuture(res)
}
// this function returns a future that takes 10 seconds to resolve
private def startLoggingFuture(res: SomeResult): Future[Unit] = Future {
// note: please don't actually do Thread.sleep in your Future's thread pool
Thread.sleep(10000)
logger.info(s"Got result $res")
}
Then you could put e.g.
_ = runSomeLogging(res)
or
_ <- Future { runSomeLogging(res) }
in your for-comprehension.
Note, Cats-Effect and Monix have a nice abstraction for "start but ignore result", with io.start.void and task.startAndForget respectively. If you were using IO or Task instead of Future, you could use .start.void or .startAndForget on the logging task.
How would you construct a function in which you both want to do a side effect and return a value?
For example I would like the following function:
def futureFromHttpCall: Future[HttpResponse] =
doHttpCall.foreach(publishDomainEvent).returnOriginalFuture
(somehow I have a feeling that monads will come up so if that is the path Im somewhat familiar with cats if there is a solution for this problem there?)
The simplest thing I can think of is instead of "hiding" the side effect inside the Future[T] returning method, expose it as a continuation on the future:
def futureFromHttpCall: Future[HttpResponse] = doHttpCall
And then you could either onComplete on it as a side effect:
futureFromHttpCall.onComplete {
case Success(_) => publishDomainEvent
case Failure(e) => // Stuff
}
Making the effect explicit. Or if you're inside an actor system, you can can pipeTo the Future to your receive method and handle success / failure there.
I think your Future should only complete when all of your domain events are pushed. They should be a Future as well. Then you can use Future.sequence to wait for all of them to complete before returning.
Your question is a little unclear but i assume doHttpCall is a list of some type.
def doHttpCall(): Future[Seq[X]] = ???
def publishDomainEvent(x:X): Future[Unit] = ???
def futureFromHttpCall(): Future[Seq[X]] = {
val firstFuture = ???
firstFuture.flatMap { xs =>
val xxs: Seq[Future[Unit]]= xs.map(publishDomainEvent)
Future.sequence(xxs).map { _ => re }
}
}
All of this waiting can be pretty helpful when testing.
I have a function which provides a Context:
def buildContext(s:String)(request:RequestHeader):Future[Granite.Context] = {
.... // returns a Future[Granite.Context]
}
I then have another function which uses a Context to return an Option[Library.Document]:
def getDocument(tag: String):Option[Library.Document] = {
val fakeRequest = play.api.test.FakeRequest().withHeaders(CONTENT_TYPE -> "application/json")
val context = buildContext(tag)(fakeRequest)
val maybeDoc = context.getDocument //getDocument is defined on Granite.Context to return an Option[Library.Document]
}
How would this code take into account if the Future has returned or not? I have seen for/yield used to wait for the return but I always assumed that a for/yield just flatmaps things together and has nothing really to do with waiting for Futures to return. I'm kinda stuck here and don't really no the correct question to ask!
The other two answers are misleading. A for yield in Scala is a compiler primitive that gets transformed into map or flatMap chains. Do not use Await if you can avoid it, it's not a simple issue.
You are introducing blocking behaviour and you have yet to realise the systemic damage you are doing when blocking.
When it comes to Future, map and flatMap do different things:
map
is executed when the future completes. It's an asynchronous way to do a type safe mapping.
val f: Future[A] = someFutureProducer
def convertAToB(a: A): B = {..}
f map { a => convertAToB(a) }
flatMap
is what you use to chain things:
someFuture flatMap {
_ => {
someOtherFuture
}
}
The equivalent of the above is:
for {
result1 <- someFuture
result2 <- someOtherFuture
} yield result2
In Play you would use Async to handle the above:
Async {
someFuture.map(i => Ok("Got result: " + i))
}
Update
I misunderstood your usage of Play. Still, it doesn't change anything. You can still make your logic asynchronous.
someFuture onComplete {
case Success(result) => // doSomething
case Failure(err) => // log the error etc
}
The main difference when thinking asynchronously is that you always have to map and flatMap and do everything else inside Futures to get things done. The performance gain is massive.
The bigger your app, the bigger the gain.
When using a for-comprehension on a Future, you're not waiting for it to finish, you're just saying: when it is finished, use it like this, and For-comprehension returns another Future in this case.
If you want to wait for a future to finish, you should use the Await as follows:
val resultContext = Await.result(context , timeout.duration)
Then run the getDocument method on it as such:
val maybeDoc = resultContext.getDocument
EDIT
The usual way to work with Futures however is to wait until the last moment before you Await. As pointed out by another answer here, Play Framework does the same thing by allowing you to return Future[Result]. So, a good way to do things would be to only use for-comprehensions and make your methods return Futures, etc, until the last moment when you want to finally return your result.
You can use scala.concurrent.Await for that:
import scala.concurrent.duration._
import scala.concurrent.Await
def getDocument(tag: String):Option[Library.Document] = {
val fakeRequest = play.api.test.FakeRequest().withHeaders(CONTENT_TYPE -> "application/json")
val context = Await.result(buildContext(tag)(fakeRequest), 42.seconds)
val maybeDoc = context.getDocument
}
But Await will block thread while future is not completed, so would be better either make buildContext a synchronous operation returning Granite.Context, or make getDocument async too, returning Future[Option[Library.Document]].
Once you are in a future you must stay in the future or you must wait until the future arrives.
Waiting is usually a bad idea because it blocks your execution, so you should work in the future.
Basically you should change your getDocument method to return a Future to something like getDocument(tag: String):Future[Option[Library.Document]]
Then using map ro flatMap, you chain your future calls:
return buildContext(tag)(fakeRequest).map(_.getDocument)
If buildContext fails, map will wrap the Failure
Then call
getDocument("blah").onComplete {
case Success(optionalDoc) => ...
case Failure(e) =>...
}
I'm using the scala dispatch (0.11.0) library to send an HTTP GET request to a remote server. I want to wait for the response before executing the code which follows the request.
My request is of the form :
val req = :/("myurl.com") <:< myheaders OK as.Response(identity)
If I write :
val future = http(req)
future()
var res: String = null
future onComplete {
case Success(r) => res = r.getResponseBody
case _ => println("KO")
}
println(res)
I get null.
This is also the case if I write :
val future = http(req)
var res: String = null
while (!future.isCompleted) {
Thread.sleep(1000)
}
future onComplete {
case Success(r) => res = r.getResponseBody
case _ => println("KO")
}
println(res)
But with the following code :
val future = http(req)
var res: String = null
future onComplete {
case Success(r) => res = r.getResponseBody
case _ => println("KO")
}
while (!future.isCompleted) {
Thread.sleep(1000)
}
println(res)
I get the expected response.
Does someone understand this ?
It seems to me that calling Thread.sleep is not a good thing, could someone give me a hint on how I should handle this problem correctly ?
EDIT: #Randal Schulz thank you for your help, but as you posted in comments, I can't validate your answer.
As my problem was to wait (and do nothing else) until I get a valid response to an HTTP GET request, I think a satisfying way to do it is using Await.result. I removed the side-effect from my code. I used the option method to deal with Future failures (as I was interested only in successes), and I dealt with time out exception the classic way.
I think I could do it as wheaties mentionned, staying in Future, but I need more practice...
TL;DR
The best advice I can give you for working in an asynchronous work flow is that what goes into a Future stays in a Future.
Answer
The issue is that you have no idea when the Future will complete so if you want to use an asynchronous process, you're going to have to write in an asynchronous manner. The code as you have written never stops or blocks on the Future you create so the minute it creates the Future and hand it off to another thread, the current thread is free to then evaluate the res variable.
Hence, place most of what you're doing in a flow like follows:
myFuture map (func1) map (func2) map (func3) onComplete{
case Success(value) => println(value.getResponseBody)
case _ => println('KO')
}
Don't attempt to access something via side-effect like you are.
If you're really clever and you have several Future you can compose them:
val f1 = myFuture map(func1)
val f2 = myOtherFuture map(func2) map (func3)
val f3 = for{
v1 <- f1
v2 <- f2
} yield functionTakingBoth(v1, v2)
f3 onComplete{
//and do stuff here
}
I finally managed to write what I wanted using futures :
def loop(): Future[String] = {
val future = http(req).option
future flatMap ((x: Option[Response]) => x match {
case Some(rep) => rep.getResponseBody
case None => loop()
}
}
Now I can use the result of this function without explicitely waiting for the response to come.
Like the author of this question I'm trying to understand the reasoning for user-visible promises in Scala 2.10's futures and promises.
Particularly, going again to the example from the SIP, isn't it completely flawed:
import scala.concurrent.{ future, promise }
val p = promise[T]
val f = p.future
val producer = future {
val r = produceSomething()
p success r
continueDoingSomethingUnrelated()
}
val consumer = future {
startDoingSomething()
f onSuccess {
case r => doSomethingWithResult()
}
}
I am imagining the case where the call to produceSomething results in a runtime exception. Because promise and producer-future are completely detached, this means the system hangs and the consumer will never complete with either success or failure.
So the only safe way to use promises requires something like
val producer = future {
try {
val r.produceSomething()
p success r
} catch {
case e: Throwable =>
p failure e
throw e // ouch
}
continueDoingSomethingUnrelated()
}
Which obviously error-prone and verbose.
The only case I can see for a visible promise type—where future {} is insufficient—is the one of the callback hook in M. A. D.'s answer. But the example of the SIP doesn't make sense to me.
This is why you rarely use success and failure unless you already know something is bulletproof. If you want bulletproof, this is what Try is for:
val producer = future {
p complete Try( produceSomething )
continueDoingSomethingUnrelated()
}
It doesn't seem necessary to throw the error again; you've already dealt with it by packing it into the answer to the promise, no? (Also, note that if produceSomething itself returns a future, you can use completeWith instead.)
Combinators
You can use Promise to build additional Future combinators that aren't already in the library.
"Select" off the first future to be satisfied. Return as a result, with the remainder of the Futures as a sequence: https://gist.github.com/viktorklang/4488970.
An after method that returns a Future that is completed after a certain period of time, to "time out" a Future: https://gist.github.com/3804710.
You need Promises to be able to create other combinators like this.
Adapt Callbacks
Use Promise to adapt callback-based APIs to Future-based APIs. For example:
def retrieveThing(key: String): Future[Thing] = {
val p = Promise[Thing]()
val callback = new Callback() {
def receive(message: ThingMessage) {
message.getPayload match {
case t: Thing =>
p success t
case err: SystemErrorPayload =>
p failure new Exception(err.getMessage)
}
}
}
thingLoader.load(key, callback, timeout)
p.future
}
Synchronizers
Build synchronizers using Promise. For example, return a cached value for an expensive operation, or compute it, but don't compute twice for the same key:
private val cache = new ConcurrentHashMap[String, Promise[T]]
def getEntry(key: String): Future[T] = {
val newPromise = Promise[T]()
val foundPromise = cache putIfAbsent (key, newPromise)
if (foundPromise == null) {
newPromise completeWith getExpensive(key)
newPromise.future
} else {
foundPromise.future
}
}
Promise has a completeWith(f: Future) method that would solve this problem by automatically handling the success/failure scenarios.
promise.completeWith( future {
r.produceSomething
})