Problems with afterAll() in ScalaTest in case of Future usage - scala

At the end of my ScalaTest suite I need to do some DB clean up.
The cleanup itself is a Future. The suite does not call the super.afterAll() which leaves some resources used by the suite (like a web browser and db connections) pending.
Here is relevant pieces of code:
override def afterAll():Unit={
var cleanUpsInProgress = true
DB.cleanUpDeletedSegments(db).onComplete{case _ =>
cleanUpsInProgress = false
}
while(cleanUpsInProgress){}
db.close()
aggregatesDB.close()
super.afterAll()
}
and
def cleanUpDeletedSegments(implicit db:ADMPDB):Future[Int]={
db.run{
segments.filter(_.deleted === 1).delete
}
}
I've debugged and scratched my head for a while and got to conclusion it is not even processing the code in the future's onComplete callback. Even when I substitute Slick's db action with stub Future.successfull(1) I still have everything pending and super.afterAll() gets NOT invoked.
Probably I'm doing something stupidly wrong? Could you help?
Note: Also I do think I need to use this ugly var and while loop here because otherwise the main thread gets completed and the framework which initiates the suite running just closes JVM. Maybe I am wrong here so would be great to hear some comments.
--------------------------UPDATE----------------------
The solution by Tyler works. But when I flatMap one more asynch cleanup (which I actually need to do) then the problem is the same again. The code below freezes and does not call super.afterAll:
override def afterAll():Unit={
val cleanUp = DB.cleanUpDeletedSegments(db).flatMap(_ => DB.cleanUpDeletedSegmentGroups(db))
Await.result(cleanUp, 6 seconds)
db.close()
aggregatesDB.close()
super.afterAll()
}
Await.result also does not throw TimeoutException and from what I see neither completed normally. Any ideas?
It works only if I use Await.result sequentially for each future like below:
override def afterAll():Unit={
val cleanUpSegments = DB.cleanUpDeletedSegments(db)
Await.result(cleanUpSegments, 3 seconds)
val cleanUpSegmentGroups = DB.cleanUpDeletedSegmentGroups(db)
Await.result(cleanUpSegmentGroups, 3 seconds)
db.close()
aggregatesDB.close()
super.afterAll()
}

Its probably just easier to await for your Future cleanup to finish:
import scala.concurrent.Await
import scala.concurrent.duration._
override def afterAll() ={
val future = DB.cleanUpDeletedSegments(db)
Await.result(future, 2 minutes)
aggregatesDB.close()
super.afterAll()
}
You can set the timeout to whatever is reasonable

Use solution by #Tyler. Your solution didn't work because you used non-volatile variable cleanupInProgress from multiple threads.

Related

Scala App doesn't exit even though Future is completed

I wait for a Future to complete and print the content on the console. Even when everything is finished, the main application doesn't exit and I have to kill it manually.
def main(args: Array[String]): Unit {
val req = HttpRequest(GET, myURL)
val res = Http().singleRequest(req)
val resultsFutures = Future {
val resultString = Await.result(HttpRequests.unpackResponse(res), Duration.Inf)
JsonMethods.parse(resultString).extract[List[Results]]
}
val results = Await.result(resultsFutures, Duration.Inf)
println(results)
}
So results gets printed on the console with the expected contend, but the application still doesn't end.
Is there something I can do to exit the application? Is there still something running, that the main is waiting for?
I'm using:
scala 2.12.10
akka 2.5.26
akkaHttp 10.1.11
As you are using Akka, you likely have an ActorSystem instantiated somehow under the hood that will keep the process running.
Either you are able to get a hand on it and call its actorSystem.terminate() method, or you can also use an explicit sys.exit(0) at the end of your main method (0 being the exit code you want).
Edit: you should also wrap the Awaits in Try and make sure to call sys.exit in case of failures as well.

Is synchronous HTTP request wrapped in a Future considered CPU or IO bound?

Consider the following two snippets where first wraps scalaj-http requests with Future, whilst second uses async-http-client
Sync client wrapped with Future using global EC
object SyncClientWithFuture {
def main(args: Array[String]): Unit = {
import scala.concurrent.ExecutionContext.Implicits.global
import scalaj.http.Http
val delay = "3000"
val slowApi = s"http://slowwly.robertomurray.co.uk/delay/${delay}/url/https://www.google.co.uk"
val nestedF = Future(Http(slowApi).asString).flatMap { _ =>
Future.sequence(List(
Future(Http(slowApi).asString),
Future(Http(slowApi).asString),
Future(Http(slowApi).asString)
))
}
time { Await.result(nestedF, Inf) }
}
}
Async client using global EC
object AsyncClient {
def main(args: Array[String]): Unit = {
import scala.concurrent.ExecutionContext.Implicits.global
import sttp.client._
import sttp.client.asynchttpclient.future.AsyncHttpClientFutureBackend
implicit val sttpBackend = AsyncHttpClientFutureBackend()
val delay = "3000"
val slowApi = uri"http://slowwly.robertomurray.co.uk/delay/${delay}/url/https://www.google.co.uk"
val nestedF = basicRequest.get(slowApi).send().flatMap { _ =>
Future.sequence(List(
basicRequest.get(slowApi).send(),
basicRequest.get(slowApi).send(),
basicRequest.get(slowApi).send()
))
}
time { Await.result(nestedF, Inf) }
}
}
The snippets are using
Slowwly to simulate slow API
scalaj-http
async-http-client sttp backend
time
The former takes 12 seconds whilst the latter takes 6 seconds. It seems the former behaves as if it is CPU bound however I do not see how that is the case since Future#sequence should executes the HTTP requests in parallel? Why does synchronous client wrapped in Future behave differently from proper async client? Is it not the case that async client does the same kind of thing where it wraps calls in Futures under the hood?
Future#sequence should execute the HTTP requests in parallel?
First of all, Future#sequence doesn't execute anything. It just produces a future that completes when all parameters complete.
Evaluation (execution) of constructed futures starts immediately If there is a free thread in the EC. Otherwise, it simply submits it for a sort of queue.
I am sure that in the first case you have single thread execution of futures.
println(scala.concurrent.ExecutionContext.Implicits.global) -> parallelism = 6
Don't know why it is like this, it might that other 5 thread is always busy for some reason. You can experiment with explicitly created new EC with 5-10 threads.
The difference with the Async case that you don't create a future by yourself, it is provided by the library, that internally don't block the thread. It starts the async process, "subscribes" for a result, and returns the future, which completes when the result will come.
Actually, async lib could have another EC internally, but I doubt.
Btw, Futures are not supposed to contain slow/io/blocking evaluations without blocking. Otherwise, you potentially will block the main thread pool (EC) and your app will be completely frozen.

Is map of Future lazy or not?

Basically I mean:
for(v <- Future(long time operation)) yield v*someOtherValue
This expression returns another Future, but the question is, is the v*someOhterValue operation lazy or not? Will this expression block on getting the value of Future(long time operation)?
Or it is like a chain of callbacks?
A short experiment can test this question.
import concurrent._;
import concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
object TheFuture {
def main(args: Array[String]): Unit = {
val fut = for (v <- Future { Thread.sleep(2000) ; 10 }) yield v * 10;
println("For loop is finished...")
println(Await.ready(fut, Duration.Inf).value.get);
}
}
If we run this, we see For loop is finished... almost immediately, and then two seconds later, we see the result. So the act of performing map or similar operations on a future is not blocking.
A map (or, equivalently, your for comprehension) on a Future is not lazy: it will be executed as soon as possible on another thread. However, since it runs on another thread, it isn't blocking, either.
If you want to do the definition and execution of the Future separately, then you have to use something like a Monix Task.
https://monix.io/api/3.0/monix/eval/Task.html

Reducing code repetition with recurring code in if-else

I'm wondering if the following short snippet, which does show repetition can be made more DRY. I seem to be hitting these kind of constructions quite often.
Say I want some computation to be done either synchronous or asynchronous, which is chosen at runtime.
for(i <- 1 to reps) {
Thread.sleep(expDistribution.sample().toInt)
if (async) {
Future {
sqlContext.sql(query).collect()
}
} else {
sqlContext.sql(query).collect()
}
}
It feels clumsy repeating the call to the sqlContext. Is there an idiom for this trivial recurring construct?
You can "store" your computation in a local def and then evaluate it either synchronously or asynchronously
def go = sqlContext.sql(query).collect()
if(async) Future(go) else Future.successful(go)
You can execture Future in your current thread using MoreExecutors.directExecutor() which is implemented in guava library.
(If you don't wan't to use guava library, see this question)
Using this method, you can switch the execution context according to async flag.
Here's the sample code.
You can see that setting async flag to false makes each Future executed in order.
import com.google.common.util.concurrent.MoreExecutors
import scala.concurrent.{Future,ExecutionContext}
import java.lang.Thread
object Main {
def main(args:Array[String]){
val async = false // change this to switch sync/async
implicit val ec = if(async){
ExecutionContext.Implicits.global // use thread pool
}else{
ExecutionContext.fromExecutor(MoreExecutors.directExecutor) // directy execute in current thread
}
println(Thread.currentThread.getId)
Future{
Thread.sleep(1000)
println(Thread.currentThread.getId)
}
Future{
Thread.sleep(2000)
println(Thread.currentThread.getId)
}
Thread.sleep(4000) // if you are doing asynchronously, you need this.
}
}

Can't seem to get Future to run callback in Scala

(I've not included the imports so as not to clutter this question)
(This is the simplest possible Scala App (created using scala-minimal template on Typesafe Activator))
I'm trying to run a query against an Elasticsearch Server.
I've run the same code on sbt console and I can see the results alright.
However, when I run the following code, I see "END" (code after the callbacks) being printed, but neither the Success callback nor the Failure callback get run.
I'm a Scala noob, so maybe I'm doing something wrong here? This code compiles. (Just to let you know all the imports are there)
object Hello{
def main(args: Array[String]): Unit = {
val client = ElasticClient.remote("vm-3bsa", 9300)
val res:Future[SearchResponse] = client.execute{ search in "vulnerabilities/3bsa" query "css" }
res onComplete{
case Success(s) => println(s)
case Failure(t) => println("An error has occured: " + t)
}
println("END")
//EDIT start
Await.result(res,10.seconds)
//EDIT end
}
}
FINAL EDIT
Instead of using onComplete, it works if I, instead, print result of the call to Await.result:
val await=Await.result(res,10.seconds)
println(await)
// results shown
The main thread will register your onComplete, println("END") and then exit, this makes the program terminate so therefore you never see your onComplete callback.
You can use Await.result(future, timeout) to block the main thread to keep it alive until the answer arrives. In a server context that would be a big no-no but in a small app like this it is not a problem blocking one thread.