I have an sbt subproject that includes end to end tests. These are run as e2e:test. I have defined my config as
I have defined a tag in the same subproject.
object HealthCheckTest extends Tag("HealthCheckTest")
I am tagging some of my end to end tests with HealthCheckTest as follows:
it("should be able to verify the data", HealthCheckTest)
I want to run only the health check tests from command line. I am trying to do this via:
sbt 'project e2e' e2e:testOnly -- -n HealthCheckTest
but this leads to all of the e2e tests being run. I have tried giving the full path to the tag (com.s.p.e2etests.HealthCheckTest), but that does not work either.
Occasionally I get warnings about -- and - being deprecated; however, all documentation online says to use this syntax including scalatest docs.
How can I run just my tagged e2e tests?
I have also tried to create a separate task for health check tests but could only figure out how to filter based on test class name, not by tag.
The commandline you posted was missing quotes. Try it like this:
sbt 'project e2e' 'testOnly -- -n HealthCheckTest'
The sbt executable treats each argument as a full line to run in the sbt console, so you must place quotes around each full line.
Note that while this won't run the test cases, it will still instantiate test class and print out test case names.
Related
As a scala beginner, I want to tag the integration tests in order to exclude them from running in certain scenarios (as they can be quite slow and might break due to external changes/problems).
I created the tag integration this way:
import org.scalatest.{FlatSpec, Matchers, Tag}
object integration extends Tag("com.dreamlines.tags.integration")
In my test I tag a test like so:
class SchemaValidation extends FlatSpec with Matchers {
it should "return valid json" taggedAs (integration) in {
...
assertSchema(response, endpointSchema)
}
}
Yet when reading up on how to filter certain tests based on the tag in the docs, I get highly confused as suddenly I read that I should be using org.scalatest.tools.Runner
scala [-cp scalatest-<version>.jar:...] org.scalatest.tools.Runner [arguments]
which has the flags I am looking for:
-l specifies a tag to exclude (Note: only one tag name allowed per -l) -l SlowTests -l PerfTests
Yet I am only used to run my tests via:
sbt test
I have no idea what the scala test runner here is referring to or what goes on under the hood when I run sbt test. I am truly lost.
I was expecting to execute the tests not entirely unlike this:
sbt test --ignore-tag integration
According to the sbt documentation on test options the following should work for you:
testOnly -- -l integration
Or in case ScalaTest uses the fully qualified tag id:
testOnly -- -l com.dreamlines.tags.integration
In case you want to do this by default, you can adjust the testOptions in the build.sbt file.
Edit:
You might consider a different approach for integration test. Instead of tagging them, you might want to put them in the src/it/{scala|resources} folders.
When using the cobertura-maven-plugin for Java, I get a nice <instrumentation> block in the config where I can put the incredibly useful <ignoreMethodAnnotation> block.
Best thing to happen to coverage since the gin martini.
Now I'm using scoverage-sbt, but I can't seem to find where I can configure it! The available keys in build.scala are limited. I can do package exclusion and file exclusion but there's nowhere to tell cobertura anything else.
Is there a -D I can supply on the SBT command line, maybe?
There is no similar configuration in Scoverage ATM.
update:
You can use special comments to exclude a block of code from instrumentation:
// $COVERAGE-OFF
...
// $COVERAGE-ON$
One way to pass parameters and commands into SBT from the command line is:
$ sbt 'set coverageEnabled := true' clean coverage test coverageReport coverageAggregate codacyCoverage
Where you call SBT once and then separate each parameter or command by a space.
In this example, I first set the property coverageEnabled := true and then run several commands in sequence: clean, coverage, test, coveraReport, coverageAggregate and finally codacyCoverage
Please note that setting properties like this requires that you enclose your statements in single quotes, for example:
$ sbt 'set coverageEnabled := true'...
I have several tests distributed over several classes.
I would like to run only the first test from the first class:
The class is:
class Step1_PrimarySpec
The test is:
test("case1: Primary (in isolation) should properly register itself to the provided Arbiter")
I tried:
sbt test-only Step1_PrimarySpec
and:
sbt test-only Step1_PrimarySpec 1
and:
sbt test-only "*Step1_PrimarySpec 1"
and:
sbt test-only "*Step1_PrimarySpec"
However, all of these commands ran the entire test suite.
So how can I run only that specific test?
You must place the double quotes around the whole command like this:
sbt "test-only <test-name>"
And according to this answer you should camelCase it to testOnly and use the -z argument
sbt "testOnly *Step1_PrimarySpec -- -z mytest"
That will run the test named mytest from the class (not the file) named Step1_PrimarySpec.
By using the *, the test runner saves us from specifying its fully qualified class name. i.e. org.path.to.Step1_PrimarySpec
I usually do below command in my CI:
clean update compile test publish
However, I'd like to exclude 1 (or a few) test class from the sbt command line.
How can I do this? (I don't want to change my code to use ignore, etc)
Two possible options
test-only See http://www.scalatest.org/user_guide/using_scalatest_with_sbt
Tags http://www.scalatest.org/user_guide/tagging_your_tests
Just to elaborate on the 2 correct options #Gonfva suggested above:
To use testOnly you should run:
sbt "testOnly org.fully.qualified.domainn.name.ASpec"
When the argument is the FQDN of the class. You can use multiple classes separate them by space. This can be used with glob as well. For example:
sbt "testOnly *ASpec"
Using tags. First, define a tag:
import org.scalatest.Tag
object CustomTag extends Tag("tagName")
Then, define a test with this tag:
it should "test1" taggedAs CustomTag in { println("test1") }
Now, in order to include tests using this tag, run:
sbt "testOnly * -- -n tagName"
Note: * is a wild card. It can be any glob as described in section 1.
In order to exclude this tag, you need to run:
sbt "testOnly * -- -l aaa"
Including or excluding tests is dependent on the test framework you are using. The command you will be employing in SBT is not completely parsed by SBT, but partially parsed by SBT, partially parsed by the test framework you are using.
So, if you are suing scalameta/munit, you may enter a command like this:
sbt> myproject/Test/testOnly MyProjectTest -- --exclude-tags=tag1,tag2,tag3
If you are using another test framework, the specific syntax after -- will be probably different.
When you are writing your test cases, you have to add tags somewhere, obviously. More details can be found below:
scalameta/munit: https://scalameta.org/munit/docs/filtering.html
scalatest: https://www.scalatest.org/user_guide/tagging_your_tests (stolen from another answer)
I'm trying to get sbt to compile and build some benchmarks. I've told it to add the benchmarks to the test path so they're recompiled along with tests, but I can't figure out how to write an action to let me actually run them. Is it possible to invoke classes from the Project definition class, or even just from the command line?
Yes, it is.
If you'd like to run them in the same VM the SBT is run in, then write a custom task similar to the following in your project definition file:
lazy val benchmark = task {
// code to run benchmarks
None // Some("will return an error message")
}
Typing benchmark in SBT console will run the task above. To actually run the benchmarks, or, for that matter, any other class you've compiled, you can reuse some of the existing infrastructure of SBT, namely the method runTask which will create a task that runs something for you. It has the following signature:
def runTask(mainClass: => Option[String], classpath: PathFinder, options: String*): Task
Simply add the following to your file:
lazy val benchmark = task { args =>
runTask(Some("whatever.your.mainclass.is"), testClasspath, args)
}
When running benchmarks, it is sometimes recommended that you run them in a separate jvm invocation, to get more reliable results. SBT allows you to run separate processes by invoking a method ! on a string command. Say you have a command java -jar path-to-artifact.jar you want to run. Then:
"java -jar path-to-artifact.jar" !
runs the command in SBT. You want to put the snippet above in a separate task, same as earlier.
And don't forget to reload when you change your project definition.
Couldn't you simply write the benchmarks as tests, so they will be run when you call 'test' in SBT?
You could also run a specific test with 'test-only', or run a main with 'run' or 'exec' (see http://code.google.com/p/simple-build-tool/wiki/RunningSbt for details).