Some of my code is async, and I want to test this code's execution has resulted in correct state. I do not have a reference to a Future or a JS Promise that I could map over – the async code in question lives inside a JS library that I'm using, and it just calls setTimeout(setSomeState, 0), which is why my only recourse is to test the state asynchronously, after a short delay (10 ms).
This is my best attempt:
import org.scalatest.{Assertion, AsyncFunSpec, Matchers}
import scala.concurrent.Promise
import scala.scalajs.js
import scala.scalajs.concurrent.JSExecutionContext
class FooSpec extends AsyncFunSpec with Matchers {
implicit override def executionContext = JSExecutionContext.queue
it("async works") {
val promise = Promise[Assertion]()
js.timers.setTimeout(10) {
promise.success {
println("FOO")
assert(true)
}
}
promise.future
}
}
This works when the assertion succeeds – with assert(true). However, when the assertion fails (e.g. if you replace it with assert(false)), the test suite freezes up. sbt just stops printing anything, and hangs indefinitely, the test suite never completes. In case of such failure FooSpec: line does get printed, but not the name of the test ("async works"), nor the "FOO" string.
If I comment out the executionContext line, I get the "Queue is empty while future is not completed, this means you're probably using a wrong ExecutionContext for your task, please double check your Future." error which is explained in detail in one of the links below.
I think these links are relevant to this problem:
https://github.com/scalatest/scalatest/issues/910
https://github.com/scalatest/scalatest/issues/1039
But I couldn't figure out a solution that would work.
Should I be building the Future[Assertion] in a different way, maybe?
I'm not tied to ScalaTest, but judging by the comments in one of the links above it seems that uTest has a similar problem except it tends to ignore the tests instead of stalling the test suite.
I just want to make assertions after a short delay, seems like it should definitely be possible. Any advice on how to accomplish that would be much appreciated.
As was explained to me in this scala.js gitter thread, I'm using Promise.success incorrectly. That method expects a value to complete the promise with, but assert(false) throws an exception, it does not return a value of type Assertion.
Since in my code assert(false) is evaluated before Promise.success is called, the exception is thrown before the promise has a chance to complete. However, the exception is thrown in an synchronous callback to setTimeout, so it is invisible to ScalaTest. ScalaTest is then left waiting for a promise.future that never completes (because the underlying promise never completes).
Basically, my code is equivalent to this:
val promise = Promise[Assertion]()
js.timers.setTimeout(10) {
println("FOO")
val successValue = assert(false) // exception thrown here
promise.success(successValue) // this line does not get executed
}
promise.future
Instead, I should have used Promise.complete which expects a Try. Try.apply accepts an argument in pass-by-name mode, meaning that it will be evaluated only after Try() is called.
So the working code looks like this:
it("async works") {
val promise = Promise[Assertion]()
js.timers.setTimeout(10) {
promise.complete(Try({
println("FOO")
assert(true)
})
})
promise.future
}
The real answer here is: you should try to get the "async" part out of your unit test setup.
All dealing with waits; sleeps; and so on adds a level of complexity that you do not want to have in your unit tests.
As you are not showing the production code you are using I can only make some suggestion how to approach this topic on a general level.
Example: when one builds his threading on top of Java's ExecutorService, you can go for a same-thread executor service; and your unit tests are using a single thread; and many things become much easier.
Long story short: consider looking into that "concept" that gives you "async" in your solution; and if there are ways to avoid the "really async" part; but of course without (!) making test-only changes to your production code.
Related
I have some async (ZIO) code, which I need to test. If I create a testing part using Thread.sleep() it works fine and I always get response:
for {
saved <- database.save(smth)
result <- eventually {
Thread.sleep(20000)
database.search(...)
}
} yield result
But if I made same logic using timeout and interval from eventually then it never works correctly ( I got timeouts):
for {
saved <- database.save(smth)
result <- eventually(timeout(Span(20, Seconds)), interval(Span(20, Seconds))) {
database.search(...)
}
} yield result
I do not understand why timeout and interval works different then Thread.sleep. It should be doing exactly same thing. Can someone explain it to me and tell how I should change this code to do not need to use Thread.sleep()?
Assuming database.search(...) returns ZIO[] object.
eventually{database.search(...)} most probably succeeds immediately after the first try.
It successfully created a task to query the database.
Then database is queried without any retry logic.
Regarding how to make it work:
val search: ZIO[Any, Throwable, String] = ???
val retried: ZIO[Any with Clock, Throwable, Option[String]] = search.retry(Schedule.spaced(Duration.fromMillis(1000))).timeout(Duration.fromMillis(20000))
Something like that should work. But I believe that more elegant solutions exist.
The other answer from #simpadjo addresses the "what" quite succinctly. I'll add some additional context as to why you might see this behavior.
for {
saved <- database.save(smth)
result <- eventually {
Thread.sleep(20000)
database.search(...)
}
} yield result
There are three different technologies being mixed here which is causing some confusion.
First is ZIO which is an asynchronous programming library that uses it's own custom runtime and execution model to perform tasks. The second is eventually which comes from ScalaTest and is useful for checking asynchronous computations by effectively polling the state of a value. And thirdly, there is Thread.sleep which is a Java api that literally suspends the current thread and prevents task progression until the timer expires.
eventually uses a simple retry mechanism that differs based on whether you are using a normal value or a Future from the scala standard library. Basically it runs the code in the block and if it throws then it sleeps the current thread and then retries it based on some interval configuration, eventually timing out. Notably in this case the behavior is entirely synchronous, meaning that as long as the value in the {} doesn't throw an exception it won't keep retrying.
Thread.sleep is a heavy weight operation and in this case it is effectively blocking the function being passed to eventually from progressing for 20 seconds. Meaning that by the time the database.search is called the operation has likely completed.
The second variant is different, it executes the code in the eventually block immediately, if it throws an exception then it will attempt it again based on the interval/timeout logic that your provide. In this scenario the save may not have completed (or propagated if it is eventually consistent). Because you are returning a ZIO which is designed not to throw, and eventually doesn't understand ZIO it will simply return the search attempt with no retry logic.
The accepted answer:
val retried: ZIO[Any with Clock, Throwable, Option[String]] = search.retry(Schedule.spaced(Duration.fromMillis(1000))).timeout(Duration.fromMillis(20000))
works because the retry and timeout are using the built-in ZIO operators which do understand how to actually retry and timeout a ZIO. Meaning that if search fails the retry will handle it until it succeeds.
Futures are executed in my code and are not being detected.
def f(): Future[String] = {
functionReturningFuture() // How to detect this?
Future("")
}
Ideally, a static analysis tool would help detect this.
The closer you can get is NonUnitStatements wart from WartRemover, but it cannot error only Future statements and skip all the others.
The fact that you have such issue could be used as an argument against using Future and replacing it with some IO: Cats' IO, Monix's Task or Scalaz ZIO. When it comes to them, you build your pipeline first, and the you run it. If you omitted IO value in return and you didn't compose it into the result in any other way (flatMap, map2, for comprehension etc) it would not get executed - it would still be there but it would cause no harm.
If you wanted to have greater control and error only on Future, you would probably have to write your own WartRemover's wart or ScalaFix rule.
I have a method. This method may return Future.failed(.....) or Future.successful(()).
def calculate(x: Int, y: Int): Future[Unit] = {
........
}
Now I need to test this method. What is the best way to assert for the test which verifies Future.successful(()) case .
Scalatest offers a few ways of working with Futures.
Option 1: isReadyWithin
import org.scalatest.concurrent.ScalaFutures._
import scala.concurrent.duration._
calculate(1, 3).isReadyWithin(1.second) should be(true)
If you wanted to do something with some return value here, you could use whenReady:
implicit val patienceConfig = PatienceConfig(1.second)
def calculateWithResult(i: Int, j: Int): Future[Int] = ???
whenReady(calculateWithResult(1,3)) { result =>
result should be(4)
}
You need an implicit PatienceConfig in scope that tells whenReady when to fail the test because of a timeout. I believe there's a default one in one of the scalatest libraries but the time period chosen is quite short - something on the order of 10 milliseconds - and can often cause flaky tests.
Option 2: AsyncXSpec
There are Async varieties of the FlatSpec, FreeSpec, FunSpec, etc. traits. They work much as their synchronous varieties except that any test must now return a value of type Future[Assertion]. For instance:
class Test extends AsyncFreeSpec {
"An asynchronous method" - {
"should succeed" in {
calculate(1,3).map { _ =>
// just want to make sure the future completes
succeed
}
}
}
}
Again, you can run a test against the result here. Note that this variant means that every test in your test class must return a Future, so it's not great if you want to mix synchronous and asynchronous tests. I'm also honestly not sure how AsyncXSpec chooses its timeout value.
Option Don't: Await.result
I would recommend against using Await.result as it blocks the thread for the duration. To my knowledge, the above two options are designed so asynchronous tests can be easily run in parallel.
Caveats:
You want to be super careful with your timeouts when doing asynchronous testing. Too long, and your tests can end up hanging for ages if something goes wrong. Too short, and your tests will be flaky. And the program may perform differently in different environments, so you might find that a timeout that's perfectly sufficient on your local machine has the tests on the build server failing 5% of the time. Be careful!
At least two options:
You can either Await.result and then make sure that the result is what you expected by asserting something about it.
You can make your assertions inside the future itself, but then use something like AsyncFlatSpec for your tests.
I assume you already extend ScalaFutures in the test class and you have configured the timeout(patienceConfig) as per your needs. If you have done that, then you could use one of the below approaches:
whenReady(calculate(1,3))(_ => succeed)
or
whenReady(calculate(1,3))(_ shouldBe ())
The first approach is preferable as there cannot be any other value for Unit, so we don't have to explicitly check for it, unless the code returns null for some reason.
This is a rather general question, but hopefully a reasonable one. When is ExecutionContext#reportFailure(Throwable) called?
It doesn't seem to be called in the Scala standard library. I suppose I perhaps should call it in some circumstances? What are these?
This method reports exceptions that cannot be reported otherwise.
It is called when an exception happens during the execution of a callback that has no other way of reporting failure. In particular, calls to Future.onComplete (e.g. via Future.foreach) return Unit and may be executed on any thread, so they have no way of reporting failure back to their callers. When an exception is thrown within onComplete, it is sent to this method.
It is called a couple of times deep within the implementation for Promise in the standard library. See the source.
try onComplete(value) catch { case NonFatal(e) => executor reportFailure e }
An ExecutionContext can implement reportFailure to execute arbitrary code when a future/promise fails. The only thing in the standard library that implements this method is ExecutionContextImpl, which is not in the public API, but found here. This class accepts a reporter: Throwable => Unit function. The default ExecutionContext.Implicits.global uses ExecutionContext.defaultReporter, which simply prints the stack trace to System.err.
If you wanted to customize the behavior, you could need to define your own ExecutionContext. It's difficult to imagine a scenario where you'd want to produce some other side-effect other than logging the stack trace, or some other kind of logging.
I am trying to understand Scala futures coming from Java background: I understand you can write:
val f = Future { ... }
then I have two questions:
How is this future scheduled? Automatically?
What scheduler will it use? In Java you would use an executor that could be a thread pool etc.
Furthermore, how can I achieve a scheduledFuture, the one that executes after a specific time delay? Thanks
The Future { ... } block is syntactic sugar for a call to Future.apply (as I'm sure you know Maciej), passing in the block of code as the first argument.
Looking at the docs for this method, you can see that it takes an implicit ExecutionContext - and it is this context which determines how it will be executed. Thus to answer your second question, the future will be executed by whichever ExecutionContext is in the implicit scope (and of course if this is ambiguous, it's a compile-time error).
In many case this will be the one from import ExecutionContext.Implicits.global, which can be tweaked by system properties but by default uses a ThreadPoolExecutor with one thread per processor core.
The scheduling however is a different matter. For some use-cases you could provide your own ExecutionContext which always applied the same delay before execution. But if you want the delay to be controllable from the call site, then of course you can't use Future.apply as there are no parameters to communicate how this should be scheduled. I would suggest submitting tasks directly to a scheduled executor in this case.
Andrzej's answer already covers most of the ground in your question. Worth mention is that Scala's "default" implicit execution context (import scala.concurrent.ExecutionContext.Implicits._) is literally a java.util.concurrent.Executor, and the whole ExecutionContext concept is a very thin wrapper, but is closely aligned with Java's executor framework.
For achieving something similar to scheduled futures, as Mauricio points out, you will have to use promises, and any third party scheduling mechanism.
Not having a common mechanism for this built into Scala 2.10 futures is a pity, but nothing fatal.
A promise is a handle for an asynchronous computation. You create one (assuming ExecutionContext in scope) by calling val p = Promise[Int](). We just promised an integer.
Clients can grab a future that depends on the promise being fulfilled, simply by calling p.future, which is just a Scala future.
Fulfilling a promise is simply a matter of calling p.successful(3), at which point the future will complete.
Play 2.x solves scheduling by using promises and a plain old Java 1.4 Timer.
Here is a linkrot-proof link to the source.
Let's also take a look at the source here:
object Promise {
private val timer = new java.util.Timer()
def timeout[A](message: => A, duration: Long, unit: TimeUnit = TimeUnit.MILLISECONDS)
(implicit ec: ExecutionContext): Future[A] = {
val p = Promise[A]()
timer.schedule(new java.util.TimerTask {
def run() {
p.completeWith(Future(message)(ec))
}
}, unit.toMillis(duration))
p.future
}
}
This can then be used like so:
val future3 = Promise.timeout(3, 10000) // will complete after 10 seconds
Notice this is much nicer than plugging a Thread.sleep(10000) into your code, which will block your thread and force a context switch.
Also worth noticing in this example is the val p = Promise... at the function's beginning, and the p.future at the end. This is a common pattern when working with promises. Take it to mean that this function makes some promise to the client, and kicks off an asynchronous computation in order to fulfill it.
Take a look here for more information about Scala promises. Notice they use a lowercase future method from the concurrent package object instead of Future.apply. The former simply delegates to the latter. Personally, I prefer the lowercase future.