I'm using a library (selenium) that do not implement asyncio and is blocking on I/O taks. When I'm performing several selenium I/O tasks, I am trying to yield back to the event loop in order to be able to do run other asyncio tasks, and the only way I found was to create a task and then await to yield back to the event loop. I wrote my own decorator to help in this process (see the code below). I was wondering if there were better ways to do so?
def task_await(func):
async def await_func(*args, **kwargs):
t = asyncio.create_task(func(*args, **kwargs))
await t
return t.result()
return await_func
#task_await
async def foo():
print('bar')
await foo()
Related
I'm trying to use scalatest Asynchronous Test Suites, but aside from some restrictions over setting timeout and what not, I don't see what does the test suite actually add.
I wonder if anyone versed into asynchronous testing with scalatest could quickly explain the differences between Asynchronous Test Suites and org.scalatest.concurrent. What do async test suites actually add over org.scalatest.concurrent? Is one approach better over the other?
We compare the following ScalaTest facilities for testing code that returns Futures:
Asynchronous style traits, for example, AsyncFlatSpec
ScalaFutures
Eventually
Asynchronous style traits
class AsyncSpec extends AsyncFlatSpec {
...
Future(3).map { v => assert(v == 3) }
...
}
non-blocking
we can assert before Future completes, i.e., return Future[Assertion] instead of Assertion
thread-safe
single-threaded serial execution context
Futures execute and complete in the order they are started and one after another
the same thread that is used to enqueue tasks in the test body is also used to execute them afterwards
Assertions can be mapped over Futures
no need to block inside the test body, i.e., use Await, whenReady
eliminates flakiness due to thread starvation
last expression in the test body must be Future[Assertion]
does not support multiple assertions in the test body
cannot use blocking constructs inside the test body as it will hang the test forever because of waiting on
enqueued but never started task
ScalaFutures
class ScalaFuturesSpec extends FlatSpec with ScalaFutures {
...
whenReady(Future(3) { v => assert(v == 3) }
...
}
blocking
we must wait to complete the Future before we can return Assertion
not thread-safe
Likely to be used with global execution context scala.concurrent.ExecutionContext.Implicits.global which is a
multi-threaded pool for parallel execution
supports multiple assertions within the same test body
last expression in the test body does not have to be Assertion
Eventually
class EventuallySpec extends FlatSpec with Eventually {
...
eventually { assert(Future(3).value.contains(Success(3))) }
...
}
more general facility intended not just for Futures
semantics here are that of retrying a block of code of any type passed in by-name until assertion is satisfied
when testing Futures it is likely global execution context will be used
intended primarily for integration testing where testing against real services with unpredictable response times
Single-threaded serial execution model vs. thread-pooled global execution model
scalatest-async-testing-comparison is an example
demonstrating the difference in two execution model.
Given the following test body
val f1 = Future {
val tmp = mutableSharedState
Thread.sleep(5000)
println(s"Start Future1 with mutableSharedState=$tmp in thread=${Thread.currentThread}")
mutableSharedState = tmp + 1
println(s"Complete Future1 with mutableSharedState=$mutableSharedState")
}
val f2 = Future {
val tmp = mutableSharedState
println(s"Start Future2 with mutableSharedState=$tmp in thread=${Thread.currentThread}")
mutableSharedState = tmp + 1
println(s"Complete Future2 with mutableSharedState=$mutableSharedState")
}
for {
_ <- f1
_ <- f2
} yield {
assert(mutableSharedState == 2)
}
let us consider the output of AsyncSpec against ScalaFuturesSpec
testOnly example.AsyncSpec:
Start Future1 with mutableSharedState=0 in thread=Thread[pool-11-thread-3-ScalaTest-running-AsyncSpec,5,main]
Complete Future1 with mutableSharedState=1
Start Future2 with mutableSharedState=1 in thread=Thread[pool-11-thread-3-ScalaTest-running-AsyncSpec,5,main]
Complete Future2 with mutableSharedState=2
testOnly example.ScalaFuturesSpec:
Start Future2 with mutableSharedState=0 in thread=Thread[scala-execution-context-global-119,5,main]
Complete Future2 with mutableSharedState=1
Start Future1 with mutableSharedState=0 in thread=Thread[scala-execution-context-global-120,5,main]
Complete Future1 with mutableSharedState=1
Note how in serial execution model same thread is used and Futures completed in order. On the other hand,
in global execution model different threads were used, and Future2 completed before Future1, which caused
race condition on the shared mutable state, which in turn made the test fail.
Which one should we use (IMO)?
In unit tests we should use mocked subsystems where returned Futures should be completing near-instantly, so there
is no need for Eventually in unit tests. Hence the choice is between async styles and ScalaFutures. The main difference
between the two is that former is non-blocking unlike the latter. If possible, we should never block, so we
should prefer async styles like AsyncFlatSpec. Further big difference is the execution model. Async styles
by default use custom serial execution model which provides thread-safety on shared mutable state, unlike global
thread-pool backed execution model often used with ScalaFutures. In conclusion, my suggestion is we use async style
traits unless we have a good reason not to.
I am using the following two code snippets to execute code in multiple threads. But I am getting different behaviour.
Snippet 1:
val futures = Future.sequence(Seq(f1, f2, f3, f4, f5))
futures.onComplete{
case Success(value) =>
case Failure(value) =>
}
Snippet 2:
Await.result(Future.sequence(Seq(f1, f2, f3, f4, f5)), Duration(500, TimeUnit.SECONDS))
In futures I am just setting some property and retrieving the result.
Note: knowing only the behaviour difference between above two snippets is sufficient.
onComplete runs on some arbitrary (unspecified) thread in the ExecutionContext, whereas Await.result runs on the current thread, and blocks it until it completes or the specified timeout is exceeded. The first is non-blocking, the second is blocking.
There's also a difference in how failures are handled in the two snippets, but this is kind of obvious from looking at the code.
Actually future.onComplete register a call back and wait for the result as soon as the future got completed control goes inside to the future, and see what the future has inside, it could be either success or failure.
On the other hand the Await blocks the thread on which the future is running until the future got completed for specific timeout.
Hence the onComplete is non blocking and Await is blocking in the nature.
If you want to use await then try collecting as much as future you can and then do Await once you should not use Await for each and every future you have In the code it will rather slow your code.
I have a service that makes use of the Scala Async library. I'm using this library primarily to time my database calls. The method that I want to test contains multiple calls to the database using the async await mechanism. A pseudo code of what I have is as below:
def myDbMethod() = async {
val firstCall = await(call the db and get the result)
val secondCall = await(all the db and get the result)
val thirdCall = await(all the db and get the result)
...
}
In my Scala test unit test, I have
Await.result(myDbMethod(), 10.seconds)
I was just trying to debug myMethod by running my unit test which would return with a test success even before getting into the secondCall. I mean I had breakpoints in all the 3 calls to the database, but the IntelliJ debugger would just exit out as soon as it finishes the first call to the database. Why is this? How can I test this behaviour using IntelliJ debugger?
I'm not sure that my answer would suffice your expectations, but it's a known issue. The problem is that async/await is quite a complicated macro, which does heavy transformations on the trees (you can check the output by enabling -Xprint:<phase_name_after_typer> flag). Unfortunately neither of existing IDEs (I'm working with Intellij and Ensime) can debug it, but I'm not familiar with their internals to explain why they can't in details.
From my experience I couldn't find any neat pros over the native for-comprehension, so you can stick with th native syntax or explicit flatmap calls, which is nicely debuggable.
This construct could be used for depdendant asynchronous calls.
async / await adds some sugar to make that easier, but to formalate that by hand you can do it like this:
def dbMethodDependant : Future[T] = for {
res1 <- firstCall
res2 <- secondCall(res2)
res3 <- thirdCall(res3)
} yield res3
Await.result(dbMethodDependant, forever)
What is in your view the best scala solution for the case that you have some plain chain of synchronous function calls, and you need to add an asynchronous action in the middle of it without blocking?
I think going for futures entails refactoring existing code to callback spaghetti, tethering futures all the way through the formerly synchronous chain of function calls, or polling on the promise every interval, not so optimal but maybe I am missing some trivial/suitable options here.
It may seem that refactoring for (akka) actors, entail a whole lot of boilerplate for such a simple feat of engineering.
How would you plug in an asynchronous function within an existing synchronous flow, without blocking, and without going into a lot of boilerplate?
Right now in my code I block with Await.result, which just means a thread is sleeping...
One simple dirty trick:
Let's say you have:
def f1Sync: T1
def f2Sync: T2
...
def fAsynchronous: Future[TAsync]
import scala.concurrent.{ Future, Promise }
object FutureHelper {
// A value class is much cheaper than an implicit conversion.
implicit class FutureConverter[T](val t: T) extends AnyVal {
def future: Future[T] = Promise.successful(t).future
}
}
Then you can for yield:
import FutureHelper._
def completeChain: Future[Whatever] = {
for {
r1 <- f1Sync.future
r2 <- f2Sync.future
.. all your syncs
rn <- fAsynchronous // this should also return a future
rnn <- f50Sync(rn).future// you can even pass the result of the async to the next function
} yield rn
}
There is minimal boilerplate of converting your sync calls to immediately resolved futures. Don't be tempted to do that with Future.apply[T](t) or simply Future(a) as that will put daemon threads onto the executor. You won't be able to convert without an implicit executor.
With promises you pay the price of 3, 4 promises which is negligible and you get what you want. for yield acts as a sequencer, it will wait for every result in turn, so you can even do something with your async result after it has been processed.
It will "wait" for every sync call to complete as well, which will happen immediately by design, except for the async call where normal async processing will be automatically employed.
You could also use the async/await library, with the caveat that you wind up with one big Future out of it that you still have to deal with:
http://docs.scala-lang.org/sips/pending/async.html
But, it results in code almost identical to the sychronous code; where you were previously blocking, you add an:
await { theAsyncFuture }
and then carry on with synchronous code.
I went to a very interesting lecture on Async (https://github.com/scala/async) a new library for Scala, what I am not sure about is how Akka and Async differ.
I am new to Scala so apologies if the answer is obvious.
Thanks.
Async simplifies asynchronous and concurrent programming. Async enables programming with non-blocking APIs in a familiar direct style. Direct-style code is as simple to write as blocking code, but it has all the advantages of efficient non-blocking code.
Out-of-the-box, Async makes programming with Scala's futures more convenient. In fact, by using Async with futures, your easy-to-read direct-style code is transformed to highly-efficient non-blocking code under the hood. Async can also be connected to other APIs (see below).
Akka provides a programming model and runtime to simplify concurrency, distribution, and fault tolerance. Async does not provide a runtime--it makes existing abstractions and their runtimes easier to use. However, Async and Akka can work together in several important ways:
Using Akka's "ask" pattern, sending a message using "?" returns a future. Async makes it easy to work with those futures.
Async can be connected to APIs other than Scala's Futures API. This opens up interesting ways to leverage Async to simplify programming with Akka actors. This is one area that we are going to explore in the near future at Typesafe.
Async is a replacement for Akka's dataflow API which is simpler and more robust. Async is simpler, because it does not introduce complex types stemming from the use of Scala's CPS plugin. Async is more robust, since it works well together with features like pattern matching and try-catch which are not fully supported by CPS/Akka's dataflow API. Moreover, it is clearly specified where Async cannot be used (await cannot occur inside closures, nested classes/traits/objects, or by-name arguments).
For more Async examples and documentation see my recent talk on Async and the documentation on the Async project website.
Async just adds helpfull API (and some other good internal things) for working with Future and has nothing to do with Actor model while Akka is a framework for creating massively-distributed with Actors. They have different use cases and have nothing in common. It more correctly to compare Async with Scala/Akka Futures API. I don't have much experience with Async, but the main point, is that you have two constructs async and await. Async marks a block of asynchronous code wich contains one or more await calls, which marks a point at which the computation will be suspended until the awaited Future is complete. Such API can be compared with a standart way of using map and flatmap .
Using standart API (for construct translates to combination of map and flatmap):
def slowCalcFuture: Future[Int] = ...
val future1 = slowCalcFuture
val future2 = slowCalcFuture
def combined: Future[Int] = for {
r1 <- future1
r2 <- future2
} yield r1 + r2
And Async:
def slowCalcFuture: Future[Int] = ...
def combined: Future[Int] = async {
val future1 = slowCalcFuture
val future2 = slowCalcFuture
await(future1) + await(future2)
}