I have this sample code below. I don't want to block inside combinator resource.map, so I put Await.result inside Future. But when I test this async recursion, I get concurrent.TimeoutException from myRes. What is the problem with the code? How do I fix it? Thank you.
// Code start here
val responses = Future.sequence(resource.map(params => asyncRecursionFunc(params)))
Await.result(responses, 5.minutes)
// My async function
def asyncRecursionFunc(params: sampleParameters, maxTryTimes: Int = 2, awaitTime: Int = 1): Future[(Seq[sample], String)] = {
val res: Future[GraphQLCluent.GraphQLResponse[SearchQueryResponse]] = client.query("gql", params).result
Future {
val myRes: GraphQLCluent.GraphQLResponse[SearchQueryResponse] = Await.result(res, 2.minutes)
val myValue: Seq[sample] = myRes.right.toSeq.flatmap(res => res)
if( !myValue.isEmpty || maxTryTimes <= 0 ){
(myValue, "stringMessage")
}
else{
Await.result(asyncRecursionFunc(params, maxTryTimes - 1, awaitTime + 1), (2*(awaitTime + 1)).minutes)
}
}
}
Yeah, you should use flatMap.
Await is bad even inside a Future (in some way, it is actually worse).
It is likely what is causing your immediate problem too (first few queries exhaust all the threads in the pool, and then Await blocks forever).
Try doing it in a truly asynchronous way:
def doQuery(params: Foo, maxTries: Int) = client.query("gql", params).result.flatMap {
case Right(r) if r.nonEmpty => Future.successful(r -> "stringmessage")
case _ if maxTries == 1 => Future.successful(Nil -> "stringmessage")
case _ => doQuery(params, maxTries-1)
}
Related
I need to execute a Future method on some elements I have in a list simultaneously. My current implementation works sequentially, which is not optimal for saving time. I did this by mapping my list and calling the method on each element and processing the data this way.
My manager shared a link with me showing how to execute Futures simultaneously using for-comprehension but I cannot see/understand how I can implement this with my List.
The link he shared with me is https://alvinalexander.com/scala/how-use-multiple-scala-futures-in-for-comprehension-loop/
Here is my current code:
private def method1(id: String): Tuple2[Boolean, List[MyObject]] = {
val workers = List.concat(idleWorkers, activeWorkers.keys.toList)
var ready = true;
val workerStatus = workers.map{ worker =>
val option = Await.result(method2(worker), 1 seconds)
var status = if (option.isDefined) {
if (option.get._2 == id) {
option.get._1.toString
} else {
"INVALID"
}
} else "FAILED"
val status = s"$worker: $status"
if (option.get._1) {
ready = false
}
MyObject(worker.toString, status)
}.toList.filterNot(s => s. status.contains("INVALID"))
(ready, workerStatus)
}
private def method2(worker: ActorRef): Future[Option[(Boolean, String)]] = Future{
implicit val timeout: Timeout = 1 seconds;
Try(Await.result(worker ? GetStatus, 1 seconds)) match {
case Success(extractedVal) => extractedVal match {
case res: (Boolean, String) => Some(res)
case _ => None
}
case Failure(_) => { None }
case _ => { None }
}
}
If someone could suggest how to implement for-comprehension in this scenario, I would be grateful. Thanks
For method2 there is no need for the Future/Await mix. Just map the Future:
def method2(worker: ActorRef): Future[Option[(Boolean, String)]] =
(worker ? GetStatus).map{
case res: (Boolean, String) => Some(res)
case _ => None
}
For method1 you likewise need to map the result of method2 and do the processing inside the map. This will make workerStatus a List[Future[MyObject]] and means that everything runs in parallel.
Then use Future.sequence(workerStatus) to turn the List[Future[MyObject]] into a Future[List[MyObject]]. You can then use map again to do the filtering/ checking on that List[MyObject]. This will happen when all the individual Futures have completed.
Ideally you would then return a Future from method1 to keep everything asynchronous. You could, if absolutely necessary, use Await.result at this point which would wait for all the asynchronous operations to complete (or fail).
Trying to execute a function in a given time frame, but if computation fails by TimeOut get a partial result instead of an empty exception.
The attached code solves it.
The timedRun function is from Computation with time limit
Any better approach?.
package ga
object Ga extends App {
//this is the ugly...
var bestResult = "best result";
try {
val result = timedRun(150)(bestEffort())
} catch {
case e: Exception =>
print ("timed at = ")
}
println(bestResult)
//dummy function
def bestEffort(): String = {
var res = 0
for (i <- 0 until 100000) {
res = i
bestResult = s" $res"
}
" " + res
}
//This is the elegant part from stackoverflow gruenewa
#throws(classOf[java.util.concurrent.TimeoutException])
def timedRun[F](timeout: Long)(f: => F): F = {
import java.util.concurrent.{ Callable, FutureTask, TimeUnit }
val task = new FutureTask(new Callable[F]() {
def call() = f
})
new Thread(task).start()
task.get(timeout, TimeUnit.MILLISECONDS)
}
}
I would introduce a small intermediate class for more explicitly communicating the partial results between threads. That way you don't have to modify non-local state in any surprising ways. Then you can also just catch the exception within the timedRun method:
class Result[A](var result: A)
val result = timedRun(150)("best result")(bestEffort)
println(result)
//dummy function
def bestEffort(r: Result[String]): Unit = {
var res = 0
for (i <- 0 until 100000) {
res = i
r.result = s" $res"
}
r.result = " " + res
}
def timedRun[A](timeout: Long)(initial: A)(f: Result[A] => _): A = {
import java.util.concurrent.{ Callable, FutureTask, TimeUnit }
val result = new Result(initial)
val task = new FutureTask(new Callable[A]() {
def call() = { f(result); result.result }
})
new Thread(task).start()
try {
task.get(timeout, TimeUnit.MILLISECONDS)
} catch {
case e: java.util.concurrent.TimeoutException => result.result
}
}
It's admittedly a bit awkward since you don't usually have the "return value" of a function passed in as a parameter. But I think it's the least-radical modification of your code that makes sense. You could also consider modeling your computation as something that returns a Stream or Iterator of partial results, and then essentially do .takeWhile(notTimedOut).last. But how feasible that is really depends on the actual computation.
First, you need to use one of the solution to recover after the future timed out which are unfortunately not built-in in Scala:
See: Scala Futures - built in timeout?
For example:
def withTimeout[T](fut:Future[T])(implicit ec:ExecutionContext, after:Duration) = {
val prom = Promise[T]()
val timeout = TimeoutScheduler.scheduleTimeout(prom, after)
val combinedFut = Future.firstCompletedOf(List(fut, prom.future))
fut onComplete{case result => timeout.cancel()}
combinedFut
}
Then it is easy:
var bestResult = "best result"
val expensiveFunction = Future {
var res = 0
for (i <- 0 until 10000) {
Thread.sleep(10)
res = i
bestResult = s" $res"
}
" " + res
}
val timeoutFuture = withTimeout(expensiveFunction) recover {
case _: TimeoutException => bestResult
}
println(Await.result(timeoutFuture, 1 seconds))
Hi I am trying to process data in a file.
This the code I am using below.
I have a list of Futures and trying to get the output from these futures.
Everything is fine but the last line of return is executing before OnSuccess.
How can I change that behaviour without having a blocking operation.
def processRow(rowNumber: Int, row: String, delimiter: String, rules: List[Rule]): RowMessage = {
var cells = row.split(delimiter)
var passedRules = new ListBuffer[RuleResult]()
val failedRules = new ListBuffer[RuleResult]()
val rulesFuture = rules.map {
i => Future {
val cells = row.split(delimiter);
//some processing....
}
}
val f1 = Future.sequence(rulesFuture)
f1 onComplete {
case Success(results) => for (result <- results) (result.map(x => {
if (x.isPassFailed) {
passedRules += x
}
else {
failedRules += x
}
}))
case Failure(t) => println("An error has occured: " + t.getMessage)
}
return new RowMessage(passedRules.toList, failedRules.toList)
}
You can't avoid blocking and return a plain RowMessage. You need to return a Futureas well.
def processRow(rowNumber: Int, row: String, delimiter: String, rules: List[Rule]): Future[RowMessage] = {
val cells = row.split(delimiter)
Future.traverse(rules) { i =>
Future {
//some processing....
}
} map { results =>
val (passed, failed) = results.partition(_.isPassFailed)
new RowMessage(passed, failed)
}
}
Also think about your algorithm to avoid mutable state, especially when you change it from different Futures.
Future.traverse is equivalent of your map + Future.sequence. Then instead of onComplete, just map your Future to modify the list. You can split it easly using partition instead of what you've been doing.
You don't need to use return, in fact you shouldn't unless you know what you are doing.
Btw isPassFailed doesn't sound like a reasonable method name to me, especially considering that when it's true you are adding it to passed rules.
I have a method that does a couple of database look up and performs some logic.
The MyType object that I return from the method is as follows:
case class MyResultType(typeId: Long, type1: Seq[Type1], type2: Seq[Type2])
The method definition is like this:
def myMethod(typeId: Long, timeInterval: Interval) = async {
// 1. check if I can find an entity in the database for typeId
val myTypeOption = await(db.run(findMyTypeById(typeId))) // I'm getting the headOption on this result
if (myTypeOption.isDefined) {
val anotherDbLookUp = await(doSomeDBStuff) // Line A
// the interval gets split and assume that I get a List of thse intervals
val intervalList = splitInterval(interval)
// for each of the interval in the intervalList, I do database look up
val results: Seq[(Future[Seq[Type1], Future[Seq[Type2])] = for {
interval <- intervalList
} yield {
(getType1Entries(interval), getType2Entries(interval))
}
// best way to work with the results so that I can return MyResultType
}
else {
None
}
}
Now the getType1Entries(interval) and getType2Entries(interval) each returns a Future of Seq(Type1) and Seq(Type2) entries!
My problem now is to get the Seq(Type1) and Seq(Type2) out of the Future and stuff that into the MyResultType case class?
You could refer to this question you asked
Scala transforming a Seq with Future
so you get the
val results2: Future[Seq([Iterable[Type1], [Iterable[Type2])] = ???
and then call await on it and you have no Futures at all, you can do what you want.
I hope I understood the question correctly.
Oh and by the way you should map myTypeOption instead of checking if it's defined and returning None if it's not
if (myTypeOption.isDefined) {
Some(x)
} else {
None
}
can be simply replaced with
myTypeOption.map { _ => // ignoring what actually was inside option
x // return whatever you want, without wrapping it in Some
}
If I understood your question correctly, then this should do the trick.
def myMethod(typeId: Long, timeInterval: Interval): Option[Seq[MyResultType]] = async {
// 1. check if I can find an entity in the database for typeId
val myTypeOption = await(db.run(findMyTypeById(typeId))) // I'm getting the headOption on this result
if (myTypeOption.isDefined) {
// the interval gets split and assume that I get a List of thse intervals
val intervalList = splitInterval(interval)
// for each of the interval in the intervalList, I do database look up
val results: Seq[(Future[Seq[Type1]], Future[Seq[Type2]])] = for {
interval <- intervalList
} yield {
(getType1Entries(interval), getType2Entries(interval))
}
// best way to work with the results so that I can return MyResultType
Some(
await(
Future.sequence(
results.map{
case (l, r) =>
l.zip(r).map{
case (vl, vr) => MyResultType(typeId, vl, vr)
}
})))
}
else {
None
}
}
There are two parts to your problem, 1) how to deal with two dependent futures, and 2) how to extract the resulting values.
When dealing with dependent futures, I normally compose them together:
val future1 = Future { 10 }
val future2 = Future { 20 }
// results in a new future with type (Int, Int)
val combined = for {
a <- future1
b <- future2
} yield (a, b)
// then you can use foreach/map, Await, or onComplete to do
// something when your results are ready..
combined.foreach { ((a, b)) =>
// do something with the result here
}
To extract the results I generally use Await if I need to make a synchronous response, use _.onComplete() if I need to deal with potential failure, and use _.foreach()/_.map() for most other circumstances.
I'd like to run my future call n times, for example 5. Future "execution" will take some time and I want to call new one only when previous was completed. Something like:
def fun(times: Int): Future[AnyRef] = {
def _fun(times: Int) = {
createFuture()
}
(1 to times).foldLeft(_fun)((a,b) => {
println(s"Fun: $b of $times")
a.flatMap(_ => _fun)
})
}
So I want to call "_fun" function n times one by one. "createFuture()" will take some time, so "_fun" shouldn't be called again before previous future was completed. Also, I want to create a non-blocking solution. Currently, this code executes without waiting for previous future to end.
Any ideas how to make it work?
Thanks for your answers!
Without understanding what exactly you want the final future to return (I'm going to just return the result of the last completed future), you could try something like this:
def fun(times: Int): Future[AnyRef] = {
val prom = Promise[AnyRef]()
def _fun(t: Int) {
val fut = createFuture()
fut onComplete {
case Failure(ex) => prom.failure(ex)
case Success(result) if t >= times => prom.success(result)
case Success(result) => _fun(t + 1)
}
}
_fun(1)
prom.future
}
This is a sort of recursive solution that will chain the futures together on completion, stopping the chaining when the max number of times has been reached. This code is not perfect, but certainly conveys one possible solution for making sure the successive futures do not fire until the previous future completed successfully.
I think it will be nicer if you make it recursive using flatMap.
Let's imagine you have your createFuture defined as:
def createFuture() = Future( println("Create future"))
We can create a function to compose the result of createFuture with:
def compose(f: () => Future[Unit])(b: Future[Unit]) = b.flatMap(_ => f())
And then you can define fun as:
def fun(n : Int) = {
def nTimes(n : Int, f : Future[Unit] => Future[Unit], acc : Future[Unit]) = if (n == 0) acc else nTimes(n-1,f,f(acc))
nTimes(n,compose(createFuture),Future())
}