I have several tests distributed over several classes.
I would like to run only the first test from the first class:
The class is:
class Step1_PrimarySpec
The test is:
test("case1: Primary (in isolation) should properly register itself to the provided Arbiter")
I tried:
sbt test-only Step1_PrimarySpec
and:
sbt test-only Step1_PrimarySpec 1
and:
sbt test-only "*Step1_PrimarySpec 1"
and:
sbt test-only "*Step1_PrimarySpec"
However, all of these commands ran the entire test suite.
So how can I run only that specific test?
You must place the double quotes around the whole command like this:
sbt "test-only <test-name>"
And according to this answer you should camelCase it to testOnly and use the -z argument
sbt "testOnly *Step1_PrimarySpec -- -z mytest"
That will run the test named mytest from the class (not the file) named Step1_PrimarySpec.
By using the *, the test runner saves us from specifying its fully qualified class name. i.e. org.path.to.Step1_PrimarySpec
Related
As a scala beginner, I want to tag the integration tests in order to exclude them from running in certain scenarios (as they can be quite slow and might break due to external changes/problems).
I created the tag integration this way:
import org.scalatest.{FlatSpec, Matchers, Tag}
object integration extends Tag("com.dreamlines.tags.integration")
In my test I tag a test like so:
class SchemaValidation extends FlatSpec with Matchers {
it should "return valid json" taggedAs (integration) in {
...
assertSchema(response, endpointSchema)
}
}
Yet when reading up on how to filter certain tests based on the tag in the docs, I get highly confused as suddenly I read that I should be using org.scalatest.tools.Runner
scala [-cp scalatest-<version>.jar:...] org.scalatest.tools.Runner [arguments]
which has the flags I am looking for:
-l specifies a tag to exclude (Note: only one tag name allowed per -l) -l SlowTests -l PerfTests
Yet I am only used to run my tests via:
sbt test
I have no idea what the scala test runner here is referring to or what goes on under the hood when I run sbt test. I am truly lost.
I was expecting to execute the tests not entirely unlike this:
sbt test --ignore-tag integration
According to the sbt documentation on test options the following should work for you:
testOnly -- -l integration
Or in case ScalaTest uses the fully qualified tag id:
testOnly -- -l com.dreamlines.tags.integration
In case you want to do this by default, you can adjust the testOptions in the build.sbt file.
Edit:
You might consider a different approach for integration test. Instead of tagging them, you might want to put them in the src/it/{scala|resources} folders.
I have an sbt subproject that includes end to end tests. These are run as e2e:test. I have defined my config as
I have defined a tag in the same subproject.
object HealthCheckTest extends Tag("HealthCheckTest")
I am tagging some of my end to end tests with HealthCheckTest as follows:
it("should be able to verify the data", HealthCheckTest)
I want to run only the health check tests from command line. I am trying to do this via:
sbt 'project e2e' e2e:testOnly -- -n HealthCheckTest
but this leads to all of the e2e tests being run. I have tried giving the full path to the tag (com.s.p.e2etests.HealthCheckTest), but that does not work either.
Occasionally I get warnings about -- and - being deprecated; however, all documentation online says to use this syntax including scalatest docs.
How can I run just my tagged e2e tests?
I have also tried to create a separate task for health check tests but could only figure out how to filter based on test class name, not by tag.
The commandline you posted was missing quotes. Try it like this:
sbt 'project e2e' 'testOnly -- -n HealthCheckTest'
The sbt executable treats each argument as a full line to run in the sbt console, so you must place quotes around each full line.
Note that while this won't run the test cases, it will still instantiate test class and print out test case names.
I usually do below command in my CI:
clean update compile test publish
However, I'd like to exclude 1 (or a few) test class from the sbt command line.
How can I do this? (I don't want to change my code to use ignore, etc)
Two possible options
test-only See http://www.scalatest.org/user_guide/using_scalatest_with_sbt
Tags http://www.scalatest.org/user_guide/tagging_your_tests
Just to elaborate on the 2 correct options #Gonfva suggested above:
To use testOnly you should run:
sbt "testOnly org.fully.qualified.domainn.name.ASpec"
When the argument is the FQDN of the class. You can use multiple classes separate them by space. This can be used with glob as well. For example:
sbt "testOnly *ASpec"
Using tags. First, define a tag:
import org.scalatest.Tag
object CustomTag extends Tag("tagName")
Then, define a test with this tag:
it should "test1" taggedAs CustomTag in { println("test1") }
Now, in order to include tests using this tag, run:
sbt "testOnly * -- -n tagName"
Note: * is a wild card. It can be any glob as described in section 1.
In order to exclude this tag, you need to run:
sbt "testOnly * -- -l aaa"
Including or excluding tests is dependent on the test framework you are using. The command you will be employing in SBT is not completely parsed by SBT, but partially parsed by SBT, partially parsed by the test framework you are using.
So, if you are suing scalameta/munit, you may enter a command like this:
sbt> myproject/Test/testOnly MyProjectTest -- --exclude-tags=tag1,tag2,tag3
If you are using another test framework, the specific syntax after -- will be probably different.
When you are writing your test cases, you have to add tags somewhere, obviously. More details can be found below:
scalameta/munit: https://scalameta.org/munit/docs/filtering.html
scalatest: https://www.scalatest.org/user_guide/tagging_your_tests (stolen from another answer)
I have been using ScalaTest for a while and I find pretty useful the ability to Tag your tests and run just those with a specific Tag from the command line.
Is there anything similar in Specs2?
I know that you can run a specific test class with testOnly but I would like to just run a test with a specific Tag within a Specification.
Here is how to do it:
import org.specs2.mutable._
class MySpec extends Specification {
tag("fast")
"example1" >> ok
tag("slow")
"example2" >> ok
}
Then in sbt
sbt> test-only *MySpec* -- include fast
You will find more information here.
In my specs2 tests, I frequently use helper functions to test groups of conditions at once. Unfortunately, that makes the line number output of failed tests useless, since all failures are on the same line.
Google turned up that there's a "failtrace" option that will output the stack trace of failure. However, I can't find an example of how to actually use that. Is it in build.sbt? Is it used on the SBT command line? Is it set somehow in the constructor of the Specification class?
You can set the failtrace option at the specification level:
class MySpec extends org.specs2.mutable.Specification {
args.report(failtrace = true)
...
}
You can also pass failtrace on the sbt command line:
sbt> test-only *MySpec* -- failtrace
Eric's solution can also be applied to all tests run by SBT by including the following in your build.sbt file:
testOptions += Tests.Argument(TestFrameworks.Specs2, "failtrace")