How do I detect if a method is called by unit test in ScalaTest?
Edit: sorry, I was expressing wrongly the thing I wanted. I have a code block in a method which takes very long to finish (I cannot mock it) and it does not affect any logic. I want to skip that code block in the unit test. So I want to know whether it is called by unit test or normal running. If it is called by unit test, I skip it, otherwise, I let it runs normally.
I have a simple workaround by adding a trait like this:
trait AppConfig {
val isDebug:Boolean
}
use it in the places where I needs to check whether it is in debug mode:
class MyLogicClass {
_: AppConfig =>
def myMethod()={
if(isDebug){...}
}
}
Use a code coverage library such as scoverage for instance. It will generate reports that indicate you which parts of your code are used by unit tests.
Related
I have a method. This method may return Future.failed(.....) or Future.successful(()).
def calculate(x: Int, y: Int): Future[Unit] = {
........
}
Now I need to test this method. What is the best way to assert for the test which verifies Future.successful(()) case .
Scalatest offers a few ways of working with Futures.
Option 1: isReadyWithin
import org.scalatest.concurrent.ScalaFutures._
import scala.concurrent.duration._
calculate(1, 3).isReadyWithin(1.second) should be(true)
If you wanted to do something with some return value here, you could use whenReady:
implicit val patienceConfig = PatienceConfig(1.second)
def calculateWithResult(i: Int, j: Int): Future[Int] = ???
whenReady(calculateWithResult(1,3)) { result =>
result should be(4)
}
You need an implicit PatienceConfig in scope that tells whenReady when to fail the test because of a timeout. I believe there's a default one in one of the scalatest libraries but the time period chosen is quite short - something on the order of 10 milliseconds - and can often cause flaky tests.
Option 2: AsyncXSpec
There are Async varieties of the FlatSpec, FreeSpec, FunSpec, etc. traits. They work much as their synchronous varieties except that any test must now return a value of type Future[Assertion]. For instance:
class Test extends AsyncFreeSpec {
"An asynchronous method" - {
"should succeed" in {
calculate(1,3).map { _ =>
// just want to make sure the future completes
succeed
}
}
}
}
Again, you can run a test against the result here. Note that this variant means that every test in your test class must return a Future, so it's not great if you want to mix synchronous and asynchronous tests. I'm also honestly not sure how AsyncXSpec chooses its timeout value.
Option Don't: Await.result
I would recommend against using Await.result as it blocks the thread for the duration. To my knowledge, the above two options are designed so asynchronous tests can be easily run in parallel.
Caveats:
You want to be super careful with your timeouts when doing asynchronous testing. Too long, and your tests can end up hanging for ages if something goes wrong. Too short, and your tests will be flaky. And the program may perform differently in different environments, so you might find that a timeout that's perfectly sufficient on your local machine has the tests on the build server failing 5% of the time. Be careful!
At least two options:
You can either Await.result and then make sure that the result is what you expected by asserting something about it.
You can make your assertions inside the future itself, but then use something like AsyncFlatSpec for your tests.
I assume you already extend ScalaFutures in the test class and you have configured the timeout(patienceConfig) as per your needs. If you have done that, then you could use one of the below approaches:
whenReady(calculate(1,3))(_ => succeed)
or
whenReady(calculate(1,3))(_ shouldBe ())
The first approach is preferable as there cannot be any other value for Unit, so we don't have to explicitly check for it, unless the code returns null for some reason.
Some of my code is async, and I want to test this code's execution has resulted in correct state. I do not have a reference to a Future or a JS Promise that I could map over – the async code in question lives inside a JS library that I'm using, and it just calls setTimeout(setSomeState, 0), which is why my only recourse is to test the state asynchronously, after a short delay (10 ms).
This is my best attempt:
import org.scalatest.{Assertion, AsyncFunSpec, Matchers}
import scala.concurrent.Promise
import scala.scalajs.js
import scala.scalajs.concurrent.JSExecutionContext
class FooSpec extends AsyncFunSpec with Matchers {
implicit override def executionContext = JSExecutionContext.queue
it("async works") {
val promise = Promise[Assertion]()
js.timers.setTimeout(10) {
promise.success {
println("FOO")
assert(true)
}
}
promise.future
}
}
This works when the assertion succeeds – with assert(true). However, when the assertion fails (e.g. if you replace it with assert(false)), the test suite freezes up. sbt just stops printing anything, and hangs indefinitely, the test suite never completes. In case of such failure FooSpec: line does get printed, but not the name of the test ("async works"), nor the "FOO" string.
If I comment out the executionContext line, I get the "Queue is empty while future is not completed, this means you're probably using a wrong ExecutionContext for your task, please double check your Future." error which is explained in detail in one of the links below.
I think these links are relevant to this problem:
https://github.com/scalatest/scalatest/issues/910
https://github.com/scalatest/scalatest/issues/1039
But I couldn't figure out a solution that would work.
Should I be building the Future[Assertion] in a different way, maybe?
I'm not tied to ScalaTest, but judging by the comments in one of the links above it seems that uTest has a similar problem except it tends to ignore the tests instead of stalling the test suite.
I just want to make assertions after a short delay, seems like it should definitely be possible. Any advice on how to accomplish that would be much appreciated.
As was explained to me in this scala.js gitter thread, I'm using Promise.success incorrectly. That method expects a value to complete the promise with, but assert(false) throws an exception, it does not return a value of type Assertion.
Since in my code assert(false) is evaluated before Promise.success is called, the exception is thrown before the promise has a chance to complete. However, the exception is thrown in an synchronous callback to setTimeout, so it is invisible to ScalaTest. ScalaTest is then left waiting for a promise.future that never completes (because the underlying promise never completes).
Basically, my code is equivalent to this:
val promise = Promise[Assertion]()
js.timers.setTimeout(10) {
println("FOO")
val successValue = assert(false) // exception thrown here
promise.success(successValue) // this line does not get executed
}
promise.future
Instead, I should have used Promise.complete which expects a Try. Try.apply accepts an argument in pass-by-name mode, meaning that it will be evaluated only after Try() is called.
So the working code looks like this:
it("async works") {
val promise = Promise[Assertion]()
js.timers.setTimeout(10) {
promise.complete(Try({
println("FOO")
assert(true)
})
})
promise.future
}
The real answer here is: you should try to get the "async" part out of your unit test setup.
All dealing with waits; sleeps; and so on adds a level of complexity that you do not want to have in your unit tests.
As you are not showing the production code you are using I can only make some suggestion how to approach this topic on a general level.
Example: when one builds his threading on top of Java's ExecutorService, you can go for a same-thread executor service; and your unit tests are using a single thread; and many things become much easier.
Long story short: consider looking into that "concept" that gives you "async" in your solution; and if there are ways to avoid the "really async" part; but of course without (!) making test-only changes to your production code.
In scalatest using FunSpec I have some code that fires in the afterEach. I would like to execute some code to get a screenshot only when the test fails. Just about everything I have looked at tries to solve this by putting asserts in try blocks which seems awful. Is there anything like onTestFailure in TestNG or a context I can get like RSpec to determine if the test failed? Looking at the scala doc I see a implementation that takes a configMap, but that is empty when I run the test. Any help would be appreciated.
pretty sure I figured it out. coming from a TestNG background it seemed weird to have to mess with fixtures to accomplish it. I suspect others with a background like mine may also look in all the wrong places as well, so going to leave this here to help others:
override def withFixture(test: NoArgTest) = { // after
val outcome = super.withFixture(test)
outcome match {
case Failed(ex) =>
// log ex (the exception) and a screenshot
}
outcome
}
I want to ask a quick question.
Is there a way to run some code before and after a specific test?
Imagine for a moment that I want some environment-setup code to be executed before and after a specific test, but not in all tests.
I've tried to define a function to do that for me and just call that function ant the beginning and the end of my test. It works fine but if the test fails, the function I call at the end isn't called (since the test failed).
Is there a way to restrict the beforeAndAfter trait to be executed only at specific tests?
Thanks.
You can do this (or a variation of this):
def beforeAfter[T](before: =>Unit, after: =>Unit)(t: =>T) = {
before
try t
finally after
}
Then
beforeAfter(before = action1, after = action2) {
test
}
I'm trying to test-drive some Scala code using Specs2 and Mockito. I'm relatively new to all three, and having difficulty with the mocked methods returning null.
In the following (transcribed with some name changes)
"My Component's process(File)" should {
"pass file to Parser" in new modules {
val file = mock[File]
myComponent.process(file)
there was one(mockParser).parse(file)
}
"pass parse result to Translator" in new modules {
val file = mock[File]
val myType1 = mock[MyType1]
mockParser.parse(file) returns (Some(myType1))
myComponent.process(file)
there was one(mockTranslator).translate(myType1)
}
}
The "pass file to Parser" works until I add the translator call in the SUT, and then dies because the mockParser.parse method has returned a null, which the translator code can't take.
Similarly, the "pass parse result to Translator" passes until I try to use the translation result in the SUT.
The real code for both of these methods can never return null, but I don't know how to tell Mockito to make the expectations return usable results.
I can of course work around this by putting null checks in the SUT, but I'd rather not, as I'm making sure to never return nulls and instead using Option, None and Some.
Pointers to a good Scala/Specs2/Mockito tutorial would be wonderful, as would a simple example of how to change a line like
there was one(mockParser).parse(file)
to make it return something that allows continued execution in the SUT when it doesn't deal with nulls.
Flailing about trying to figure this out, I have tried changing that line to
there was one(mockParser).parse(file) returns myResult
with a value for myResult that is of the type I want returned. That gave me a compile error as it expects to find a MatchResult there rather than my return type.
If it matters, I'm using Scala 2.9.0.
If you don't have seen it, you can look the mock expectation page of the specs2 documentation.
In your code, the stub should be mockParser.parse(file) returns myResult
Edited after Don's edit:
There was a misunderstanding. The way you do it in your second example is the good one and you should do exactly the same in the first test:
val file = mock[File]
val myType1 = mock[MyType1]
mockParser.parse(file) returns (Some(myType1))
myComponent.process(file)
there was one(mockParser).parse(file)
The idea of unit testing with mock is always the same: explain how your mocks work (stubbing), execute, verify.
That should answer the question, now a personal advice:
Most of the time, except if you want to verify some algorithmic behavior (stop on first success, process a list in reverse order) you should not test expectation in your unit tests.
In your example, the process method should "translate things", thus your unit tests should focus on it: mock your parsers and translators, stub them and only check the result of the whole process. It's less fine grain but the goal of a unit test is not to check every step of a method. If you want to change the implementation, you should not have to modify a bunch of unit tests that verify each line of the method.
I have managed to solve this, though there may be a better solution, so I'm going to post my own answer, but not accept it immediately.
What I needed to do was supply a sensible default return value for the mock, in the form of an org.mockito.stubbing.Answer<T> with T being the return type.
I was able to do this with the following mock setup:
val defaultParseResult = new Answer[Option[MyType1]] {
def answer(p1: InvocationOnMock): Option[MyType1] = None
}
val mockParser = org.mockito.Mockito.mock(implicitly[ClassManifest[Parser]].erasure,
defaultParseResult).asInstanceOf[Parser]
after a bit of browsing of the source for the org.specs2.mock.Mockito trait and things it calls.
And now, instead of returning null, the parse returns None when not stubbed (including when it's expected as in the first test), which allows the test to pass with this value being used in the code under test.
I will likely make a test support method hiding the mess in the mockParser assignment, and letting me do the same for various return types, as I'm going to need the same capability with several return types just in this set of tests.
I couldn't locate support for a shorter way of doing this in org.specs2.mock.Mockito, but perhaps this will inspire Eric to add such. Nice to have the author in the conversation...
Edit
On further perusal of source, it occurred to me that I should be able to just call the method
def mock[T, A](implicit m: ClassManifest[T], a: org.mockito.stubbing.Answer[A]): T = org.mockito.Mockito.mock(implicitly[ClassManifest[T]].erasure, a).asInstanceOf[T]
defined in org.specs2.mock.MockitoMocker, which was in fact the inspiration for my solution above. But I can't figure out the call. mock is rather overloaded, and all my attempts seem to end up invoking a different version and not liking my parameters.
So it looks like Eric has already included support for this, but I don't understand how to get to it.
Update
I have defined a trait containing the following:
def mock[T, A](implicit m: ClassManifest[T], default: A): T = {
org.mockito.Mockito.mock(
implicitly[ClassManifest[T]].erasure,
new Answer[A] {
def answer(p1: InvocationOnMock): A = default
}).asInstanceOf[T]
}
and now by using that trait I can setup my mock as
implicit val defaultParseResult = None
val mockParser = mock[Parser,Option[MyType1]]
I don't after all need more usages of this in this particular test, as supplying a usable value for this makes all my tests work without null checks in the code under test. But it might be needed in other tests.
I'd still be interested in how to handle this issue without adding this trait.
Without the full it's difficult to say but can you please check that the method you're trying to mock is not a final method? Because in that case Mockito won't be able to mock it and will return null.
Another piece of advice, when something doesn't work, is to rewrite the code with Mockito in a standard JUnit test. Then, if it fails, your question might be best answered by someone on the Mockito mailing list.