I have three methods which return future, how do I properly chain them so that output from first one is passed to the second and second's return to third and finally send them all of the above data to final future. I am not sure this is the correct way to approach this problem.
Source
.fromFuture(someFuture)
.mapAsync(1)(modelData => queryModelData(modelQuery))
.mapAsync(1)(modelId => findModelId(modelData))
.mapAsync(1)(jobData => queryJobData(jobQuery))
.mapAsync(1)(status => setModelStatus(modelData,modelId,jobData))
You can chain multiple Futures sequentially using .flatMap (or a for-comprehension which is syntactic sugar for the same).
e.g.
Source
.fromFuture(someFuture)
.mapAsync(1){ data =>
for {
modelData <- queryModelData(data)
modelId <- findModelId(modelData)
jobData <- queryJobData(modelId)
status <- setModelStatus(modelData, modelId, jobData)
} yield status
}
Related
I'm writing an authentication client that takes an Option[Credentials] as a parameter. This Credentials object has a .token method on it which I will then use to construct an HTTP request to post to an endpoint. This returns a Future[HttpResponse], which I then need to validate, unmarshal, and then convert back to my return type, which is an Option[String].
My first thought was to use a for comprehension like this:
val resp = for {
c <- creds
req <- buildRequest(c.token)
resp <- Http().singleRequest(req)
} yield resp
but then I found out that monads cannot be composed like that. My next thought is to do something like this:
val respFut = Http().singleRequest(buildRequest(token))
respFut.onComplete {
case Success(resp) => Some("john.doe")//do stuff
case Failure(_) => None
}
Unfortunately onComplete returns a unit, and map leaves me with a Future[Option[String]], and the only way I currently know to strip off the future wrapper is using the pipeTo methods in the akka framework. How can I convert this back to just an option string?
Once you've got a Future[T], it's usually good practice to not try to unbox it until you absolutely have to. Can you change your method to return a Future[Option[String]]? How far up the call stack can you deal with futures? Ideally it's all the way.
Something like this will give you a Future[Option[String]] as a result:
val futureResult = creds map {
case Some(c) => {
val req = buildRequest(c.token)
val futureResponse = Http().singleRequest(req)
futureResponse.map(res => Some(convertResponseToString(res)))
}
case None => Future(None)
}
If you really need to block and wait on the result, you can do Await.result as described here.
And if you want to do it in a more monadic style (in a for-comprehension, like you tried), cats has an OptionT type that will help with that, and I think scalaz does as well. But whether you want to get into either of those libraries is up to you.
It's easy to "upgrade" an Option to a Future[Option[...]], so use Future as your main monad. And deal with the simpler case first:
val f: Future[Option[String]] =
// no credential? just wrap a `None` in a successful future
credsOpt.fold(Future.successful(Option.empty[String])) {creds =>
Http()
.singleRequest(buildRequest(creds.token))
.map(convertResponseToString)
.recover {case _ => Option.empty[String]}
}
The only way to turn that future into Option[String] is to wait for it with Await.result(...)... but it's better if that future can be passed along to the next caller (no blocking).
I'm not 100% certain about what all your types are, but it seems like you want a for comprehension that mixes option and futures. I've often been in that situation and I find I can just chain my for comprehensions as a way to make the code look a bit better.
val resp = for {
c <- creds
req <- buildRequest(c.token)
} yield for {
resp <- Http().singleRequest(req)
} yield resp
resp becomes an Option[Future[HttpResponse]] which you can match / partial func around with None meaning the code never got to execute because it failed its conditions. This is a dumb little trick I use to make comprehensions look better and I hope it gives you a hint towards your solution.
I would like to utilize a simple Flow to gather some extra data from a http service and enhance my data object with the results. The following illustrates the Idea:
val httpClient = Http().superPool[User]()
val cityRequest = Flow[User].map { user=>
(HttpRequest(uri=Uri(config.getString("cityRequestEndpoint"))), User)
}
val cityResponse = Flow[(Try[HttpResponse], User)].map {
case (Failure(ex), user) => user
case (Success(resp), user) => {
// << What to do here to get the value >> //
val responseData = processResponseSomehowToGetAValue?
val enhancedUser = new EnhancedUser(user.data, responseData)
enhancedUser
}
}
val processEnhancedUser = Flow[EnhancedUser].map {
// e.g.: Asynchronously save user to a database
}
val useEnhancementGraph = userSource
.via(getRequest)
.via(httpClient)
.via(getResponse)
.via(processEnhancedUser)
.to(Sink.foreach(println))
I have a problem to understand the mechanics and difference between
the streaming nature and materialization / Futures inside the Flow.
Following ideas did not explain it to me:
http://doc.akka.io/docs/akka-http/current/scala/http/implications-of-streaming-http-entity.html
akka HttpResponse read body as String scala
How do i get the value from the response into the new user object,
so i can handle that object in the following steps.
Thanks for help.
Update:
I was evaluating the code with a remote akka http server answering to requests between immediately and 10 seconds using the code below for parsing.
This led to the effect that some "EnhancedUser" Instances showed up at the end, but the ones who took too long to answer were missing their values.
I added .async to the end of the cityResponse parser at some time and the result output took longer, but was correct.
What is the reason for that behaviour and how does it fit together with the accepted Answer?
val cityResponse = Flow[(Try[HttpResponse], User)].map {
case (Failure(ex), member) => member
case (Success(response), member) => {
Unmarshal(response.entity).to[String] onComplete {
case Success(s) => member.city = Some(s)
case Failure(ex) => member.city = None
}
}
member
}.async // <<-- This changed the behavior to be correct, why?
There are two different strategies you could use depending on the nature of the entity you are getting from "cityRequestEndpoint":
Stream Based
The typical way to handle this situation is to always assume that the entity coming from the source endpoint can contain N pieces of data, where N is not known in advance. This is usually the pattern to follow because it is the most generic and therefore "safest" in the real world.
The first step is to convert the HttpResponse coming from the endpoint into a Source of data:
val convertResponseToByteStrSource : (Try[HttpResponse], User) => Source[(Option[ByteString], User), _] =
(response, user) => response match {
case Failure(_) => Source single (None -> user)
case Success(r) => r.entity.dataBytes map (byteStr => Some(byteStr) -> user)
}
The above code is where we don't assume the size of N, r.entity.dataBytes could be a Source of 0 ByteString values, or potentially an infinite number values. But our logic doesn't care!
Now we need to combine the data coming from the Source. This is a good use case for Flow.flatMapConcat which takes a Flow of Sources and converts it into a Flow of values (similar to flatMap for Iterables):
val cityByteStrFlow : Flow[(Try[HttpResponse], User), (Option[ByteString], User), _] =
Flow[(Try[HttpResponse], User)] flatMapConcat convertResponseToByteStrSource
All that is left to do is convert the tuples of (ByteString, User) into EnhancedUser. Note: I am assuming below that User is a subclass of EnhancedUser which is inferred from the question logic:
val convertByteStringToUser : (Option[ByteString], User) => EnhancedUser =
(byteStr, user) =>
byteStr
.map(s => EnhancedUser(user.data, s))
.getOrElse(user)
val cityUserFlow : Flow[(Option[ByteString], User), EnhancedUser, _] =
Flow[(ByteString, User)] map convertByteStringToUser
These components can now be combined:
val useEnhancementGraph =
userSource
.via(cityRequest)
.via(httpClient)
.via(cityByteStrFlow)
.via(cityUserFlow)
.via(processEnhancedUser)
.to(Sink foreach println)
Future Based
We can use Futures to solve the problem, similar to the stack question you referenced in your original question. I don't recommend this approach for 2 reasons:
It assumes only 1 ByteString is coming from the endpoint. If the endpoint sends multiple values as ByteStrings then they all get concatenated together and you could get an error when creating EnhancedUser.
It places an artificial timeout on the materialization of the ByteString data, similar to Async.await (which should almost always be avoided).
To use the Future based approach the only big change to your original code is to use Flow.mapAsync instead of Flow.map to handle the fact that a Future is being created in the function:
val parallelism = 10
val timeout : FiniteDuration = ??? //you need to specify the timeout limit
val convertResponseToFutureByteStr : (Try[HttpResponse], User) => Future[EnhancedUser] =
_ match {
case (Failure(ex), user) =>
Future successful user
case (Success(resp), user) =>
resp
.entity
.toStrict(timeout)
.map(byteStr => new EnhancedUser(user.data, byteStr))
}
val cityResponse : Flow[(Try[HttpResponse], User), EnhancedUser, _] =
Flow[(Try[HttpResponse], User)].mapAsync(parallelism)(convertResponseToFutureByteStr)
I'm slowly wrapping my brain around Futures in Scala, and have a bit of a layer cake going on that I'm trying to unravel.
The specific use case is a DeferredResolver in sangria-graphql + akka. I've stolen their demo code, which looks like this
Future.fromTry(Try(
friendIds map (id => CharacterRepo.humans.find(_.id == id) orElse CharacterRepo.droids.find(_.id == id))))
and added my own modification to it. Theirs does an in-memory lookup, whereas mine asks something of another actor:
Future.fromTry(Try(
accountIds match {
case h :: _ =>
val f = sender ? TargetedMessage(h)
val resp = Await.result(f, timeout.duration).asInstanceOf[TargetedMessage]
marshallAccount(resp.body)
case _ => throw new Exception("Not found")
}
))
The pertinent piece here is that I pick the first element in the list, send it to an ActorRef that I got elsewhere and wait for the result. This works. What I'd like to do, however, is not have to wait for the result here, but return the whole thing as a Future
Future.fromTry(Try(
accountIds match {
case h :: _ =>
sender ? TargetedMessage(h) map {
case resp:TargetedMessage => marshallAccount(resp.body)
}
case _ => throw new Exception("Not found")
}
))
This doesn't work. When this is consumed, instead of being of type Account (the return type of function marshallAccount, it's of type Promise. If I understand correctly, it's because instead of having a return type of Future[Account], this has a type of Future[Future[Account]]
How do I flatten this?
You are looking at the wrong API method. Future.fromTry is used to create an immediately resolved Future, meaning the call is not actually asynchronous. Dive into the implementation of Future.fromTry which will take you to:
def fromTry[T](result: Try[T]): Promise[T] = new impl.Promise.KeptPromise[T](result)
A promise kept is basically something that has already happened, so just like Future.successful this is just used to ensure the right return type or similar, it's not actually a way to make something async.
The reason why the return type is Future[Future[Something]] is because you are trying to wrap something that already returns a future into another future.
The ask pattern, namely sender ? TargetMessage(h) is a way to ask something of an actor and await for a result, which will return a future.
The correct way to approach this:
val future: Future[Account] = accountIds match {
case h :: _ => sender ? TargetedMessage(h) map (marshallAccount(_.body)
case _ => Future.failed(throw new Exception("Not found"))
}
Basically you need to use Future.failed to return a failed future from an exception if you want to keep the return type consistent. It's worth reviewing this tutorial to learn a bit more about Futures and how to write application logic with them.
Starting playing with Scala futures, I get stuck with dependent futures.
Let's get a example. I search for places and get a Future[Seq[Place]]. For each of theses places, I search for the closest subway stations (the service resurns a Future[List[Station]]).
I would write this:
Place.get()
.map { places =>
places.map { place =>
Station.closestFrom(place).map { stations =>
SearchResult(place, stations)
}
}
}
That thing will make me get a Future[Seq[Future[SearchResult]]]... which is... not what I would have expected.
What did I miss to get a Future[Seq[SearchResult]] ?
Thanks for all,
Alban
You are missing two Future concepts in your solution: flatMap and Future.sequence
To explain each:
flatMap is like map except instead of giving it a function from future.map(A => B) you give it a function from future.flatMap(A => Future[B]). This way you can chain Futures together.
Future.sequence is a helper function that combines a list of futures to a future of a list: Seq[Future[A]] => Future[Seq[A]]
Using these two features of the Future API we can change your answer to be:
Place.get().flatMap { places =>
Future.sequence(places.map { place =>
Station.closestFrom(place).map { stations =>
SearchResult(place, stations)
}
})
}
Short version
Working with futures is generaly easier using for-comprehension rather than directly map/flatMap. In your situation it should look like this:
for {places <- Place.get()
searchResults <- Future.traverse(places)(place => for (stations <- Station.closestFrom(place))
yield SearchResult(place,stations)
)
} yield searchResults
Detailed Version
Future being a monad, it offers you several ways to chain your operations.
If you want to apply an 'regular' function f : A => B to what's inside the box myfuture : Future[A], indeed map is the way to get a Future[B]. But in the present situation Station.closestFrom a does not give you a List[Stattion] but a Future[List[Station]].
If you want to apply a monadic operation h : A => Future[B] or chain several of them (here Places.get and Station.closestFrom), flatMap is the way to go. Apply h to a Future[A] gives you a Future[B].
If you want to apply a monadic operation h : A => Future[B] to a collection like a places : Seq[A], you should use Future.traverse : Seq[A] => (A => Future[B]) => Future[Seq[B]].
Furthermore, Scala's for-compresention is just syntactic sugar for flatMap/map so instead of writing complex code using those directly you can use a clean and clear for loop. The loop:
for { variable1 <- f1
variable2 <- f2
} yield expression
is equivalent to (without optimisations) :
f1.flatMap( variable1 => f2.map(variable2 => expression))
Don't hesitate to use for-comprehension, it really helps.
If I have some computation that takes a while I might place it in a scala.concurrent.Future:
val f = Future { someLongRunningFunction() }
and let's say I want to do something else asynchronously once that computation is completed:
f.flatMap{ _ => anotherLongRunningFunction() }
In the event that f's initial block fails, how do I "idiomatically" handle this when using flatMap or other combinators? Is it merely a case of using recover or onFailure before the flatMap?
I like the elegance and simplicity of using a flatMap but it seems failure scenarios get in the way of it.
Edit: the second future is reliant on the first, hence the flatMap. I'm looking for a solution that'll elegantly let me chain like I would with flatMap but also handle failures of the first.
To quote the scaladoc for flatMap:
Creates a new future by applying a function to the successful result
of this future, and returns the result of the function as the new
future. If this future is completed with an exception then the new
future will also contain this exception.
Notice the bold, meaning that whatever you pass to flatMap will only be executed if the initial future completes successfully.
So you should only handle the result of the entire execution:
val result = future1.flatMap {
result => functionReturningFuture2(result)
}
and then:
result.onFailure // or
result.onSuccess // or
result.recover
If you have several futures you can put them in a for comprehension.
val futureResult = for {
result1 <- future1
result2 <- future2
...
} yield {
//computation with results
}
You can add a recover at the end in case you want to process any exception you may find:
futureResult.recover{
case exceptionResult: Throwable => // Process exception
}
I think this is more clean that using flatMap.
I'm only starting with Future, but here are some ideas.
If you really want to use flatMap, you have to turn the failure into a success.
for{ a <- f recover r
b <- another(a)
} yield b
This works if the return type of r is :> the result type of f.
Or you can pass the problem of what to do with the failure on to the next process
for{ a <- f map (x => Success(x)) recover (ex => Failure(ex))
b <- another(a)
} yield b
Here the argument type of another would be Try[T] where the type of f is Future[T].