Scala Synchronising Asynchronous calls with Future - scala

I have a method that does a couple of database look up and performs some logic.
The MyType object that I return from the method is as follows:
case class MyResultType(typeId: Long, type1: Seq[Type1], type2: Seq[Type2])
The method definition is like this:
def myMethod(typeId: Long, timeInterval: Interval) = async {
// 1. check if I can find an entity in the database for typeId
val myTypeOption = await(db.run(findMyTypeById(typeId))) // I'm getting the headOption on this result
if (myTypeOption.isDefined) {
val anotherDbLookUp = await(doSomeDBStuff) // Line A
// the interval gets split and assume that I get a List of thse intervals
val intervalList = splitInterval(interval)
// for each of the interval in the intervalList, I do database look up
val results: Seq[(Future[Seq[Type1], Future[Seq[Type2])] = for {
interval <- intervalList
} yield {
(getType1Entries(interval), getType2Entries(interval))
}
// best way to work with the results so that I can return MyResultType
}
else {
None
}
}
Now the getType1Entries(interval) and getType2Entries(interval) each returns a Future of Seq(Type1) and Seq(Type2) entries!
My problem now is to get the Seq(Type1) and Seq(Type2) out of the Future and stuff that into the MyResultType case class?

You could refer to this question you asked
Scala transforming a Seq with Future
so you get the
val results2: Future[Seq([Iterable[Type1], [Iterable[Type2])] = ???
and then call await on it and you have no Futures at all, you can do what you want.
I hope I understood the question correctly.
Oh and by the way you should map myTypeOption instead of checking if it's defined and returning None if it's not
if (myTypeOption.isDefined) {
Some(x)
} else {
None
}
can be simply replaced with
myTypeOption.map { _ => // ignoring what actually was inside option
x // return whatever you want, without wrapping it in Some
}

If I understood your question correctly, then this should do the trick.
def myMethod(typeId: Long, timeInterval: Interval): Option[Seq[MyResultType]] = async {
// 1. check if I can find an entity in the database for typeId
val myTypeOption = await(db.run(findMyTypeById(typeId))) // I'm getting the headOption on this result
if (myTypeOption.isDefined) {
// the interval gets split and assume that I get a List of thse intervals
val intervalList = splitInterval(interval)
// for each of the interval in the intervalList, I do database look up
val results: Seq[(Future[Seq[Type1]], Future[Seq[Type2]])] = for {
interval <- intervalList
} yield {
(getType1Entries(interval), getType2Entries(interval))
}
// best way to work with the results so that I can return MyResultType
Some(
await(
Future.sequence(
results.map{
case (l, r) =>
l.zip(r).map{
case (vl, vr) => MyResultType(typeId, vl, vr)
}
})))
}
else {
None
}
}

There are two parts to your problem, 1) how to deal with two dependent futures, and 2) how to extract the resulting values.
When dealing with dependent futures, I normally compose them together:
val future1 = Future { 10 }
val future2 = Future { 20 }
// results in a new future with type (Int, Int)
val combined = for {
a <- future1
b <- future2
} yield (a, b)
// then you can use foreach/map, Await, or onComplete to do
// something when your results are ready..
combined.foreach { ((a, b)) =>
// do something with the result here
}
To extract the results I generally use Await if I need to make a synchronous response, use _.onComplete() if I need to deal with potential failure, and use _.foreach()/_.map() for most other circumstances.

Related

Scala Futures for-comprehension with a list of values

I need to execute a Future method on some elements I have in a list simultaneously. My current implementation works sequentially, which is not optimal for saving time. I did this by mapping my list and calling the method on each element and processing the data this way.
My manager shared a link with me showing how to execute Futures simultaneously using for-comprehension but I cannot see/understand how I can implement this with my List.
The link he shared with me is https://alvinalexander.com/scala/how-use-multiple-scala-futures-in-for-comprehension-loop/
Here is my current code:
private def method1(id: String): Tuple2[Boolean, List[MyObject]] = {
val workers = List.concat(idleWorkers, activeWorkers.keys.toList)
var ready = true;
val workerStatus = workers.map{ worker =>
val option = Await.result(method2(worker), 1 seconds)
var status = if (option.isDefined) {
if (option.get._2 == id) {
option.get._1.toString
} else {
"INVALID"
}
} else "FAILED"
val status = s"$worker: $status"
if (option.get._1) {
ready = false
}
MyObject(worker.toString, status)
}.toList.filterNot(s => s. status.contains("INVALID"))
(ready, workerStatus)
}
private def method2(worker: ActorRef): Future[Option[(Boolean, String)]] = Future{
implicit val timeout: Timeout = 1 seconds;
Try(Await.result(worker ? GetStatus, 1 seconds)) match {
case Success(extractedVal) => extractedVal match {
case res: (Boolean, String) => Some(res)
case _ => None
}
case Failure(_) => { None }
case _ => { None }
}
}
If someone could suggest how to implement for-comprehension in this scenario, I would be grateful. Thanks
For method2 there is no need for the Future/Await mix. Just map the Future:
def method2(worker: ActorRef): Future[Option[(Boolean, String)]] =
(worker ? GetStatus).map{
case res: (Boolean, String) => Some(res)
case _ => None
}
For method1 you likewise need to map the result of method2 and do the processing inside the map. This will make workerStatus a List[Future[MyObject]] and means that everything runs in parallel.
Then use Future.sequence(workerStatus) to turn the List[Future[MyObject]] into a Future[List[MyObject]]. You can then use map again to do the filtering/ checking on that List[MyObject]. This will happen when all the individual Futures have completed.
Ideally you would then return a Future from method1 to keep everything asynchronous. You could, if absolutely necessary, use Await.result at this point which would wait for all the asynchronous operations to complete (or fail).

Evaluating a Future Boolean asynchronously using Scala and Play

I have a method that returns a Future[Boolean] in a Play controller and i want to evaluate that using async but i can't seem to get it to compile.
The following will work:
def health = Action {
logger.info("Endpoint method: health")
val isHealthy = healthCheckService.checkDynamo()
val b: Boolean = Await.result(isHealthy, scala.concurrent.duration.Duration(5, "seconds"))
Ok(Json.toJson(HealthCheckResponse(b.toString)))
}
But i don't think i want that Await in there. So i'm trying things like this with no success:
def health =
Action.async {
Future {
logger.info("Endpoint method: health")
healthCheckService.checkDynamo() match {
case Future.successful(true) => Ok(Json.toJson("false"))
case false => Ok(Json.toJson("true"))
}
val r = healthCheckService.checkDynamo() match {
case true => Ok(Json.toJson("false"))
case false => Ok(Json.toJson("true"))
}
}
}
I can't even get those to compile to test them out.
Any suggestions?
Try this:
def health = Action.async {
healthCheckService.checkDynamo().map {
case true => Ok(Json.toJson("false"))
case false => Ok(Json.toJson("true"))
}
}
Let Play handle the awaiting for you under the hood. That is, Action.async accepts a Future, which checkDynamo() already returns. All you have to do is map it to the appropriate result.
With Futures you have to use combinators like map and flatMap to express the final value. For example:
Action.async {
healthCheckService.checkDynamo()
.map { result => // boolean
HealthCheckResponse(result.toString)
}
.map(Json.toJson(_))
.map(Ok(_))
}
(You can merge maps above to one map and construct the final Ok value there; it is more or less a matter of taste)
If you have, say, two async calls which you want to execute and return a result based on their results, you can use flatMap, which could be easily expressed using a for comprehension:
Action.async {
for {
result1 <- someService.someCall()
result2 <- anotherService.anotherCall(result1.someProperty)
finalResult = SomeFinalResultType(result1, result2)
} yield Ok(Json.toJson(finalResult))
}
If you are not familiar with futures, you might want to read some tutorial which explains their nature, how to combine them and how to get useful results from them, like this one: http://hello-scala.com/920-scala-futures.html

Return value from method definition that has multiple case statements

val value = authenticateUser
private def authenticateUser = {
val holder = WS.url(platformUrl + "/userApi/auth/login?username=testuser&password=testPass")
val res = holder.post(Results.EmptyContent()).onComplete {
case Success(response) => response.cookies.map{cookie => println(cookie.value.get)}
case Failure(errors) => println("")
// The `Future` failed.
}
}
How to return cookie.value.get from authenticateUser method?
First of all, you seem to have many cookies, so it's not clear, the value of which one you want to return.
More importantly, you should not actually return the value itself. The idea is to return the Future, that can be further transformed downstream:
def authenticateUser = WS.url(..)
.post(Results.EmptyContent)
.map { _.cookies }
.map { _.find(_.name == "Auth").flatMap(_.value).getOrElse("") }
Now, somewhere downstream, you can have something like:
def serve(request: Request): Future[Response] = authenticateUser
.map { user =>
serveRequest(user, request)
}.map { result => makeResponse(result) }
I don't know the specifics of Play, so, consider the Request/Response stuff "pseudocode", but the basic idea is that your entire processing should be a chain of Future transformations. The general rule is to never block the service thread.

Scala - Batched Stream from Futures

I have instances of a case class Thing, and I have a bunch of queries to run that return a collection of Things like so:
def queries: Seq[Future[Seq[Thing]]]
I need to collect all Things from all futures (like above) and group them into equally sized collections of 10,000 so they can be serialized to files of 10,000 Things.
def serializeThings(Seq[Thing]): Future[Unit]
I want it to be implemented in such a way that I don't wait for all queries to run before serializing. As soon as there are 10,000 Things returned after the futures of the first queries complete, I want to start serializing.
If I do something like:
Future.sequence(queries)
It will collect the results of all the queries, but my understanding is that operations like map won't be invoked until all queries complete and all the Things must fit into memory at once.
What's the best way to implement a batched stream pipeline using Scala collections and concurrent libraries?
I think that I managed to make something. The solution is based on my previous answer. It collects results from Future[List[Thing]] results until it reaches a treshold of BatchSize. Then it calls serializeThings future, when it finishes, the loop continues with the rest.
object BatchFutures extends App {
case class Thing(id: Int)
def getFuture(id: Int): Future[List[Thing]] = {
Future.successful {
List.fill(3)(Thing(id))
}
}
def serializeThings(things: Seq[Thing]): Future[Unit] = Future.successful {
//Thread.sleep(2000)
println("processing: " + things)
}
val ids = (1 to 4).toList
val BatchSize = 5
val future = ids.foldLeft(Future.successful[List[Thing]](Nil)) {
case (acc, id) =>
acc flatMap { processed =>
getFuture(id) flatMap { res =>
val all = processed ++ res
val (batch, rest) = all.splitAt(5)
if (batch.length == BatchSize) { // if futures filled the batch with needed amount
serializeThings(batch) map { _ =>
rest // process the rest
}
} else {
Future.successful(all) //if we need more Things for a batch
}
}
}
}.flatMap { rest =>
serializeThings(rest)
}
Await.result(future, Duration.Inf)
}
The result prints:
processing: List(Thing(1), Thing(1), Thing(1), Thing(2), Thing(2))
processing: List(Thing(2), Thing(3), Thing(3), Thing(3), Thing(4))
processing: List(Thing(4), Thing(4))
When the number of Things isn't divisible by BatchSize we have to call serializeThings once more(last flatMap). I hope it helps! :)
Before you do Future.sequence do what you want to do with individual future and then use Future.sequence.
//this can be used for serializing
def doSomething(): Unit = ???
//do something with the failed future
def doSomethingElse(): Unit = ???
def doSomething(list: List[_]) = ???
val list: List[Future[_]] = List.fill(10000)(Future(doSomething()))
val newList =
list.par.map { f =>
f.map { result =>
doSomething()
}.recover { case throwable =>
doSomethingElse()
}
}
Future.sequence(newList).map ( list => doSomething(list)) //wait till all are complete
instead of newList generation you could use Future.traverse
Future.traverse(list)(f => f.map( x => doSomething()).recover {case th => doSomethingElse() }).map ( completeListOfValues => doSomething(completeListOfValues))

Scala wait of list of future to execute

Hi I am trying to process data in a file.
This the code I am using below.
I have a list of Futures and trying to get the output from these futures.
Everything is fine but the last line of return is executing before OnSuccess.
How can I change that behaviour without having a blocking operation.
def processRow(rowNumber: Int, row: String, delimiter: String, rules: List[Rule]): RowMessage = {
var cells = row.split(delimiter)
var passedRules = new ListBuffer[RuleResult]()
val failedRules = new ListBuffer[RuleResult]()
val rulesFuture = rules.map {
i => Future {
val cells = row.split(delimiter);
//some processing....
}
}
val f1 = Future.sequence(rulesFuture)
f1 onComplete {
case Success(results) => for (result <- results) (result.map(x => {
if (x.isPassFailed) {
passedRules += x
}
else {
failedRules += x
}
}))
case Failure(t) => println("An error has occured: " + t.getMessage)
}
return new RowMessage(passedRules.toList, failedRules.toList)
}
You can't avoid blocking and return a plain RowMessage. You need to return a Futureas well.
def processRow(rowNumber: Int, row: String, delimiter: String, rules: List[Rule]): Future[RowMessage] = {
val cells = row.split(delimiter)
Future.traverse(rules) { i =>
Future {
//some processing....
}
} map { results =>
val (passed, failed) = results.partition(_.isPassFailed)
new RowMessage(passed, failed)
}
}
Also think about your algorithm to avoid mutable state, especially when you change it from different Futures.
Future.traverse is equivalent of your map + Future.sequence. Then instead of onComplete, just map your Future to modify the list. You can split it easly using partition instead of what you've been doing.
You don't need to use return, in fact you shouldn't unless you know what you are doing.
Btw isPassFailed doesn't sound like a reasonable method name to me, especially considering that when it's true you are adding it to passed rules.