I am trying to specify the order in which Spec2 Specifications are run, I know about the sequential keyword that makes sure they run one after another but this corresponds to the tests within a Specification (and doesn't actually guarantee any order)
I found this SO question: https://stackoverflow.com/a/15832297/1757402 which looked promising but again, seems to just sort the tests within a Specification
I am assuming SBT/Specs runs the Specifications in the order in which the JVM returns the classes, is there any way to change this? Or any way to guarantee an order?
So say I have the following Specifications:
CApplicationSpec.scala
#RunWith(classOf[JUnitRunner])
class CApplicationSpec extends Specification {
"CApplicationSpec" should {
"be OK" in new WithApplication{
OK must equalTo(OK)
}
}
}
BApplicationSpec
#RunWith(classOf[JUnitRunner])
class BApplicationSpec extends Specification {
"BApplicationSpec" should {
"be OK" in new WithApplication{
OK must equalTo(OK)
}
}
}
At the moment if I test these, the order of execution can change each time, I want a way to be able to guarantee that BApplication (or any other Spec) will always run first, maybe by sorting them alphabetically?
You can create a specification which will "link" other specifications and run them in a specified order:
object AllSpecifications extends Specification { def is = s2"""
${"run BSpecification" ~ BSpecification}
${"run CSpecification" ~ CSpecification}
"""
}
I ended up doing it via SBT with testGrouping
//Put all tests into a single Group so that SBT reports correct number of tests, but order them by test name
testGrouping in Test <<= definedTests in Test map { tests => {
val sortedTests = tests map {
test => new Tests.Group(test.name, Seq(test), Tests.InProcess)
} sortBy (_.name.toLowerCase) flatMap {
_.tests
}
Seq(new Tests.Group("Tests", sortedTests, Tests.InProcess))
}
}
Which orders all the tests alphabetically (including package name) I then have all specs I want to run first in a specific package
Related
I have a set of input cases stored in a file.
I would like each case to be a specific scalatest "test", i.e., reported in the console as an individual test and failed individually.
Unfortunately, experimentation and Google suggest that this capability might not be present?
E.g., this seems to be the common case (eliding for simplicity)
class MyTestingGoop extends FunSuite {
val input : Seq[SpecificTestCase] = ...
test("input data test") {
forAll(input) { case => ... }
}
//...
}
Ideally, each case presents as a separate test. How can this be done with ScalaTest?
You can do this:
class MyTestingGoop extends FunSuite {
val input : Seq[SpecificTestCase] = ...
forAll(input) {
test("testing input" + input) {
// do something with the test
}
}
}
The only limit is that input has a unique toString.
Basically calling test in Funsuite registers the test and later runs it so as long as your test creation is done as part of the class construction and each test has a unique string, you should be fine.
I am running a multi-project SBT (v0.13) build and would like it to fail fast (stop) on the first error (compile) encountered in a sub-project.
The current behavior is that when something fails to compile in a sub-project, the build continues (to compile all the other sub-projects).
Is there a way/setting to make SBT stop and exit as soon as the first sub-project with a compile error is encountered?
In short, to my knowledge, no, SBT can not "fail fast" on a compiler or test error.
To my knowledge SBT doesn't control this. SBT is just going to invoke the appropriate test framework when it inspects your unit tests. The test framework can then decide what order to run the tests in, to run them concurrently, how to report issues, etc. Of course each test framework has its own features, configuration, and conventions. The two most widely used test frameworks for Scala are ScalaTest and Specs2.
Fortunately you can get the behavior you've requested in either Specs2 or ScalaTest. I've provided simple unit test examples below that fail early.
ScalaTest
You can get fail-fast behavior in ScalaTest for single test Suites by mixing in the CancelAfterFailure trait. For example, this test would execute the first test, fail the second, and show the third as cancelled.
class SomeSuite extends FunSuite with CancelAfterFailure with Assertions {
test("first") {
println("first")
}
test("second") {
assert(false)
}
test("third") {
println("third")
}
}
Specs2
Similar to ScalaTest you can control behavior in Specs2 on a per-Specification basis. To get fail-fast-like behavior you need to add two Arguments to your Specification: sequential and stopOnFail. See the docs for a full list of arguments you can set. (You do need both if you want an obvious linear ordering since Specs2 by default will execute your tests concurrently!)
class SomeSpec extends Specification {
sequential
stopOnFail
"specs2" should {
"first" in {
println("first")
ok
}
"second" in {
"foo" must equalTo ("bar")
}
"third" in {
println("third")
}
}
}
I've discovered that throwing a new java.lang.VirtualMachineError() {} halts the task immediately, while all other kinds of exceptions I have tried so far were swallowed.
I've tried ThreadDeath, InterruptedException, LinkageError, and ControlThrowable, and of course, the usual RuntimeException and such (the list came from scala.util.control.NonFatal)
<project root>/project/BuildUtils.scala
import sbt._
import sbt.Def._
object BuildUtils {
private def details(inc: Incomplete): Seq[Throwable] =
inc.directCause.toSeq ++ inc.causes.flatMap(details)
implicit class TaskSyntax[A](private val task: Initialize[Task[A]]) extends AnyVal {
def haltWhenFailed: Initialize[Task[A]] = task.result.map {
case Inc(cause) =>
throw new VirtualMachineError({
s"""Task has failed during execution.
|TaskNode: ${cause.node}
|DirectCause: ${details(cause).map(_.getMessage).distinct}
|""".stripMargin
}) {}
case Value(value) => value
}
}
}
<project root>/build.sbt
import BuildUtils._
lazy val ciBuild = taskKey[Unit]("Compile and run test")
ciBuild := {
val t1 # _ = (Test / compile).haltWhenFailed.value
val t2 # _ = (Test / test).haltWhenFailed.value
}
ciBuild will halt at first compilation error with rather verbose stacktrace and the specified message.
I have a number of test data sets that run through the same ScalaTest unit tests. I'd love if each test data set was it's own set of named tests so if one data set fails one of the tests i know exactly which one it was rather than going to a single test and looking on what file it failed. I just can't seem to find a way for the test name to be generated at runtime. I've looked at property and table based testing and currently am using should behave like to share fixtures, but none of these seem to do what I want.
Have I not uncovered the right testing approach in ScalaTest or is this not possible?
You can write dynamic test cases with ScalaTest like Jonathan Chow wrote in his blog here: http://blog.echo.sh/2013/05/12/dynamically-creating-tests-with-scalatest.html
However, I always prefer the WordSpec testing definitions and this also works with dynamic test cases just like Jonathan mentions.
class MyTest extends WordSpec with Matchers {
"My test" should {
Seq(1,2,3) foreach { count =>
s"run test $count" in {
count should be(count)
}
}
}
}
When running this test it run 3 test cases
TestResults
MyTest
My test
run test 1
run test 2
run test 3
ps. You can even do multiple test cases in the same foreach function using the same count variable.
You could write a base test class, and extend it for each data set. Something like this:
case class Person(name: String, age: Int)
abstract class MyTestBase extends WordSpec with Matchers {
def name: String
def dataSet: List[Person]
s"Data set $name" should {
"have no zero-length names" in {
dataSet.foreach { s => s.name should not be empty }
}
}
}
class TheTest extends MyTestBase {
override lazy val name = "Family" // note lazy, otherwise initialization fails
override val dataSet = List(Person("Mom", 53), Person("Dad", 50))
}
Which produces output like this:
TheTests:
Data set Family
- should have no zero-length names
You can use scala string substitutions in your test names. Using behavior functions, something like this would work:
case class Person(name: String, age: Int)
trait PersonBehaviors { this: FlatSpec =>
// or add data set name as a parameter to this function
def personBehavior(person: => Person): Unit = {
behavior of person.name
it should s"have non-negative age: ${person.age}" in {
assert(person.age >= 0)
}
}
}
class TheTest extends FlatSpec with PersonBehaviors {
val person = Person("John", 32)
personBehavior(person)
}
This produces output like this:
TheTest:
John
- should have non-negative age: 32
What about using ScalaTest's clue mechanism so that any test failures can report as a clue which data set was being used?
You can use the withClue construct provided by Assertions,
which is extended by every style trait in ScalaTest, to add
extra information to reports of failed or canceled tests.
See also the documentation on AppendedClues
I have one superclass with WordSpecLike and several subclasses for each backend.
abstract class A extends TestKit(_system) with WordSpecLike {
"backend" must {
"do something useful" in {
//test useful stuff
}
"print something pretty" in {
//test pretty print
}
}
}
class B extends A {
}
class C extends A {
}
C and B tests different backends but the thing is I need to turn on/off each backend-test separately for integration tests (using exclude groups).
Obviously, I can't use taggedAs.
Using a separate traits like in a bellow example didn't work:
trait backendB { this: Tag =>
override val name = "backendB"
}
class B extends A with backendB {
//...
}
}
Pure java annotation didn't work ether:
#TagAnnotation
#Retention(RetentionPolicy.RUNTIME)
#Target({ElementType.METHOD, ElementType.TYPE})
public #interface backendB {
}
So, the question is: Have I any other options to run each backend test under a special group or I have to copy-paste code?
Scala version 2.10
ScalaTest version 2.2.2
Java version 1.8.0_20
Can't use sbt instead of maven, can't bump scala version.
If I understand correctly, you want sometimes to exclude class B and run just class C. Other times exclude class C and run just class B. If that's correct, then normally you would annotate B and C with a respective tag annotation (make using Java). For example, doing this would cause all tests of B to be tagged as backendB:
#backendB
class B extends A {
}
What you didn't show is where the actor system is coming from. One thing about tags is that it only excludes tests, an instance of B will still be created. That means any state it holds onto will be initialized, and that might in your case mean an actor system. This might be what you meant by "didn't work." If so, my usual advice is to make that state lazily initialized. The instance will still be created, but because no tests run, he state will never be accessed, and since it is lazy, will never be initialized.
Since there are a lot of unknowns in this reply, you might do better on scalatest-users, where it is easier to go back and forth.
Ok, my final solution is not so elegant as it could be but it's work at least (still kinda more java-way solution).
Several small changes:
trait A-suite extends WordSpecLike {
val tagToExclude = Tag("here we can write anything")
def createTestBackend: Backend
"backend" must {
"do something useful" taggedAs(tagToExclude) in {
val backend = createTestBackend
//test useful stuff
}
"print something pretty" taggedAs(tagToExclude) in {
val backend = createTestBackend
//test pretty print
}
}
}
And test classes for different backends:
`
class backendA(override val tagToExclude = "backendA") extends A-suite {
override def createTestBackend: Backend = new backendA
}
class backendB(override val tagToExclude = "backendB") extends A-suite {
override def createTestBackend: Backend = new backendB
}
Now exclude groups work properly with maven.
I want to know the list of running test cases and manipulate with those information. In TestNG, implementing the onFinish, onStart, etc., methods of ITestListener gives ITestContext to retrieve those information. Is there anything similar to that in scala-test. Suggestions are highly appreciated. Thanks in advance.
Sky's answer is actually looking in the right direction. Mixing in ScalaTest's BeforeAndAfterAll trait gives you access to some contextual information about the suite, such as:
The suite name
The number of tests in the suite
The names of tests in the suite
Tag information for the suite
The information you get is perhaps not as rich as the contextual information you get from TestNG (for example, this trait won't be able to tell you which tests passed/failed in afterAll). Maybe however the information it gives you is good enough for your purposes:
class MyTest extends FunSuite with BeforeAndAfterAll {
override def beforeAll() {
// suiteName will give you the name of the suite
info(suiteName)
// testNames will give you the names of your tests
testNames forEach info(_)
// tags will give you a mapping of test names to tags
tags.keys.forEach(t =>
info(t + " tagged with tags " + tags(t).mkString(",")))
}
...
}
Yes,
Scalatest has BeforeAndAfter trait which has:
before{
//write code here(run before each test cases in a test file )
}
after{
// write code here(run after each test cases in a test file )
}
and an other trait BeforeAndAfterAll which has:
override def afterAll: Unit = {
//write code here(run after all test cases in a test file )
}
override def beforeAll: Unit = {
//write code here(run before all test cases in a test file )
}