I'm trying to run tests by sharing behaviour in traits. When I run such a test, the test always aborts early when there's a failure, not running the rest of the tests.
Here's my test:
trait TestBehaviour extends ShouldMatchers {
def failEarly = fail("this fails before the test is run")
def failEarlyAgain = fail("this also fails before the test is run")
}
class Test extends FlatSpec with TestBehaviour {
it must behave like failEarly
it must behave like failEarlyAgain
}
If I run this in the ScalaTest IDE for Eclipse, no tests are ever shown - the RUN ABORTED text is just shown in the console:
*** RUN ABORTED ***
An exception or error caused a run to abort: this fails before the test is run
So it looks like the first test is being run early.
What I would expect to see is both the tests are run, and fail (with different reason messages).
change
def failEarly = fail("this fails before the test is run")
def failEarlyAgain = fail("this also fails before the test is run")
to
def failEarly() = fail("this fails before the test is run")
def failEarlyAgain() = fail("this also fails before the test is run")
In your version you are invoking the methods defined (because they don't have a parameter list). In my version you are passing a function created from the method (without invoking it)
It seems that the templated test, should contain an actual test-case; like so
import org.scalatest.FlatSpec
trait TestTemplate { this: FlatSpec =>
def failEarly() = {
it should "fail at the right moment" in {
fail("this fails once for each test that should behave like me")
}
}
}
class MyTest extends FlatSpec with TestTemplate {
"This Test" should behave like failEarly()
"Some other test" should behave like failEarly()
}
The example above seems to work for me.
Related
Hi everybody I have these suites:
#DoNotDiscover
class Suite1 extends FunSuite with Suite{
test("Test1"){
println("Running test1")
println("Using data inserted by MasterDataSuite")
}
}
#DoNotDiscover
class Suite2 extends FunSuite with Suite{
test("Test2"){
println("Running test2")
println("Using data inserted by MasterDataSuite")
}
}
class MasterDataSuite extends Suites(new Suite1,new Suite2) with BeforeAndAfterAllConfigMap {
override def beforeAll(configMap: ConfigMap) {
//inserting in DB
}
override def afterAll(configMap: ConfigMap) {
//deleting in DB
}
}
If I run by command line, the data necessary in MasterData class is inserted, all good so far.
My big problem is if I want to run a single test in Suite1 or Suite2 the test will always fail because it can not find the data necessary to run successfully (this data should be inserted by beforeAll method). I am not sure if I need to set up something in Intellij, or I need to set up something from code or If it is possible.
Quick notes:
Why use that instead of before and after per Suite?: A: Because There are hundreds of tests that already exist in the project. Then is so long check test by test to guarantee correct data per Suite.
That works in the pipeline and by command line.
Thanks in advance
I want to know in what order the tests are running, because I want to run
my initial test setup tests only once before all the tests start running.
If I have one initial setup test class, and one essential test class it would be fine:
class EssentialTesting #Inject()(setupTests: SetupTest) extends ....{
setupTests.runInitialSetup()
.....
}
However if I have number of testings classes the setupTests.runInitialSetup will be repeated in each class. How to deal with this duplication?
Make your Setups class like:
class SetUpTest extends SpecificationLike with BeforeAndAfterAll{
override def beforeAll() = {
// runs before each test
}
override val afterAll() = {
// runs after each test
}
}
Now in your test file, you can extend your SetUpTest, like :
class Test extends PlaySpecification with SetupTest
Now you don't need to call the runinitialsetup every time, just put it into beforeAll method and it will get executed before each test. And SpecificationLike is required as BeforeAndAfterAll has self type reference for SpecificationLike.
I am using Scala 2.10 with ScalaMock 3.6.
I have a quite simple test case with 4 test scenarios. I have created a mock object for those tests to use (imitating file system):
class ProcessingOperatorTest extends FlatSpec with Matchers with BeforeAndAfterEach with MockFactory {
...
val fakeFS = mock[FileIO]
(fakeFS.createFile _).expects(*).returns(true).anyNumberOfTimes()
(fakeFS.exist _).expects(where { (p: String) => p.contains(existing) }).returns(true).anyNumberOfTimes()
(fakeFS.exist _).expects(where { (p: String) => p.contains(notExisting) }).returns(false).anyNumberOfTimes()
behavior of "Something"
it should "test 1" in {
...
}
it should "test 2" in {
...
}
it should "test 3" in {
...
}
it should "test 4" in {
...
}
Now:
1st test does not use any of mocked methods (but needs the mock object)
2nd test uses only existing mock method
3rd test uses both existing and not existing mock methods
4th test uses all methods, (also createFile)
Now, for some reason, when running all those tests together, 4th test fails giving me the following error. If running separately, it will pass.
Unexpected call: <mock-1> FileIO.exist(notExisting)
Expected:
inAnyOrder {
}
Actual:
<mock-1> FileIO.exist(notExisting)
ScalaTestFailureLocation: scala.Option at (Option.scala:120)
org.scalatest.exceptions.TestFailedException: Unexpected call: <mock-1> FileIO.exist(notExisting)
...
The other walkaround is to copy-paste the mock declaration and its behaviour inside 4th it should { ... } test scenario. Tests work then (separately, and all together).
Why is global mock instance failing?
I can try to prepare a similar test scenario as separate sbt project if needed.
Mix in org.scalatest.OneInstancePerTest as described here:
class ProcessingOperatorTest extends FlatSpec
with Matchers
with BeforeAndAfterEach
with MockFactory
with OneInstancePerTest {
...
}
I am running a multi-project SBT (v0.13) build and would like it to fail fast (stop) on the first error (compile) encountered in a sub-project.
The current behavior is that when something fails to compile in a sub-project, the build continues (to compile all the other sub-projects).
Is there a way/setting to make SBT stop and exit as soon as the first sub-project with a compile error is encountered?
In short, to my knowledge, no, SBT can not "fail fast" on a compiler or test error.
To my knowledge SBT doesn't control this. SBT is just going to invoke the appropriate test framework when it inspects your unit tests. The test framework can then decide what order to run the tests in, to run them concurrently, how to report issues, etc. Of course each test framework has its own features, configuration, and conventions. The two most widely used test frameworks for Scala are ScalaTest and Specs2.
Fortunately you can get the behavior you've requested in either Specs2 or ScalaTest. I've provided simple unit test examples below that fail early.
ScalaTest
You can get fail-fast behavior in ScalaTest for single test Suites by mixing in the CancelAfterFailure trait. For example, this test would execute the first test, fail the second, and show the third as cancelled.
class SomeSuite extends FunSuite with CancelAfterFailure with Assertions {
test("first") {
println("first")
}
test("second") {
assert(false)
}
test("third") {
println("third")
}
}
Specs2
Similar to ScalaTest you can control behavior in Specs2 on a per-Specification basis. To get fail-fast-like behavior you need to add two Arguments to your Specification: sequential and stopOnFail. See the docs for a full list of arguments you can set. (You do need both if you want an obvious linear ordering since Specs2 by default will execute your tests concurrently!)
class SomeSpec extends Specification {
sequential
stopOnFail
"specs2" should {
"first" in {
println("first")
ok
}
"second" in {
"foo" must equalTo ("bar")
}
"third" in {
println("third")
}
}
}
I've discovered that throwing a new java.lang.VirtualMachineError() {} halts the task immediately, while all other kinds of exceptions I have tried so far were swallowed.
I've tried ThreadDeath, InterruptedException, LinkageError, and ControlThrowable, and of course, the usual RuntimeException and such (the list came from scala.util.control.NonFatal)
<project root>/project/BuildUtils.scala
import sbt._
import sbt.Def._
object BuildUtils {
private def details(inc: Incomplete): Seq[Throwable] =
inc.directCause.toSeq ++ inc.causes.flatMap(details)
implicit class TaskSyntax[A](private val task: Initialize[Task[A]]) extends AnyVal {
def haltWhenFailed: Initialize[Task[A]] = task.result.map {
case Inc(cause) =>
throw new VirtualMachineError({
s"""Task has failed during execution.
|TaskNode: ${cause.node}
|DirectCause: ${details(cause).map(_.getMessage).distinct}
|""".stripMargin
}) {}
case Value(value) => value
}
}
}
<project root>/build.sbt
import BuildUtils._
lazy val ciBuild = taskKey[Unit]("Compile and run test")
ciBuild := {
val t1 # _ = (Test / compile).haltWhenFailed.value
val t2 # _ = (Test / test).haltWhenFailed.value
}
ciBuild will halt at first compilation error with rather verbose stacktrace and the specified message.
I have a suite of scalatest tests that test different endpoints of a RESTful API.
I really want them separated into different files for best organization.
My problem is how to start something (an HTTP server in my case, but it doesn't matter what it is) before all the tests and shut it down after all the tests are done.
I know about BeforeAndAfterAll, but that only accomplishes before/after inside one test file. I need something like that, but for all tests, for example:
-- start http server before tests
-- run all test suites
-- shut down http server
The intended way to do this is to use nested suites. Suite has a nestedSuites method that returns an IndexedSeq[Suite] (in 2.0, in 1.9.1 it was a List[Suite]). Suite also has a runNestedSuites method that is responsible for executing any nested suites. By default runNestedSuites calls nestedSuites, and on each returned Suite either invokes run directly, or if a Distributor is passed, puts the nested suites in the distributor so that they can be run in parallel.
So what you really probably want to do is make Foo and Bar into classes, and return instances of them from the nestedSuites method of EndpointTests. There's a class that makes that easy called Suites. Here's an example of its use:
import org.scalatest._
import matchers.MustMatchers
class Foo extends FunSpec with MustMatchers {
describe("Message here...") {
it("Must do something") { }
it("Must be ok") { }
}
}
class Bar extends FunSpec with MustMatchers {
describe("Hello you...") {
it("One more!") { }
}
}
class EndpointTests extends Suites(new Foo, new Bar) with BeforeAndAfterAll {
override def beforeAll(configMap: Map[String, Any]) {
println("Before!") // start up your web server or whatever
}
override def afterAll(configMap: Map[String, Any]) {
println("After!") // shut down the web server
}
}
One potential problem, though, is that if you are using discovery to find Suites to run, all three of EndpointTests, Foo, and Bar will be discovered. In ScalaTest 2.0 you can annotate Foo and Bar with #DoNotDiscover, and ScalaTest's Runner will not discover them. But sbt still will. We are currently enhancing sbt so that it passes over otherwise discoverable Suites that are annotated with DoNotDiscover, but this will be in sbt 0.13, which isn't out yet. In the meantime you can get sbt to ignore them by adding an unused constructor parameter to Foo and Bar.
Ok, found a way. It seems (unless someone here can correct me) that Scalatest does not have the facility of a "master" suite. But... you can kinda build one.
You can compose a suite from traits. So using my endpoint example:
class EndpointTests extends FunSpec with MustMatchers with BeforeAndAfterAll
with Foo with Bar {
override def beforeAll(configMap: Map[String, Any]) {
println("Before!") // start up your web server or whatever
}
override def afterAll(configMap: Map[String, Any]) {
println("After!") // shut down the web server
}
}
Ok, but what about the tests? Notice the with Foo with Bar. I'm bringing the dependent tests in as traits.
See here:
trait Foo extends FunSpec with MustMatchers {
describe("Message here...") {
it("Must do something") { }
it("Must be ok") { }
}
}
trait Bar extends FunSpec with MustMatchers {
describe("Hello you...") {
it("One more!") { }
}
}
Alternatively you can just use an object.
object TestServer {
startServer()
}
When you access the object it will be initialised, starting the server.
Just create a common trait in the body of which you access the object.
Then mixin that trait into all your tests. Done.
If your server runs in daemon mode (e.g. a Play! application in test mode) it will be automatically shut down after all tests are run.