I have a test that is non-functional, so I do not want it to affect the coverage report. Other than putting it in a separate project, is there a way to exclude it from the report when running
sbt clean coverage test coverageReport
As it stands, it is obscuring missing coverage by other tests.
To explain better I've made a sample project in GitHub. There are two test classes - MainSpec and DoubleSpec. I've commented out the test in DoubleSpec to show that the test in MainSpec gives 100% coverage. However, it's obscuring that DoubleSpec is "broken". MainSpec is not testing the functionality, only checking the return type.
The example is contrived. I have a much more complex example which I cannot share.
I've tried using coverageExcludedFiles but this only excludes source from coverage, not tests.
Consider creating a ScalaTest tag
import org.scalatest.Tag
object ExcludeFromCoverage extends Tag("ExcludeFromCoverage")
based on which to exclude particular tests only when running coverage
"double" should "return an Int" taggedAs ExcludeFromCoverage in {
...
}
so now executing using testOnly like so
sbt clean coverage "testOnly * -- -l ExcludeFromCoverage" coverageReport
will exclude "double" should "return an Int" from the coverage report. Note you can still execute all tests with sbt test.
Given above, then to still run tests tagged with ExcludeFromCoverage but not include them in coverage consider defining a custom command like so
commands += Command.command("testWithSmartCoverage") { state =>
"clean" ::
"set coverageEnabled := true" ::
"testOnly * -- -l ExcludeFromCoverage" ::
"set coverageEnabled := false" ::
"testOnly * -- -n ExcludeFromCoverage" ::
"coverageReport" ::
state
}
Related
As a scala beginner, I want to tag the integration tests in order to exclude them from running in certain scenarios (as they can be quite slow and might break due to external changes/problems).
I created the tag integration this way:
import org.scalatest.{FlatSpec, Matchers, Tag}
object integration extends Tag("com.dreamlines.tags.integration")
In my test I tag a test like so:
class SchemaValidation extends FlatSpec with Matchers {
it should "return valid json" taggedAs (integration) in {
...
assertSchema(response, endpointSchema)
}
}
Yet when reading up on how to filter certain tests based on the tag in the docs, I get highly confused as suddenly I read that I should be using org.scalatest.tools.Runner
scala [-cp scalatest-<version>.jar:...] org.scalatest.tools.Runner [arguments]
which has the flags I am looking for:
-l specifies a tag to exclude (Note: only one tag name allowed per -l) -l SlowTests -l PerfTests
Yet I am only used to run my tests via:
sbt test
I have no idea what the scala test runner here is referring to or what goes on under the hood when I run sbt test. I am truly lost.
I was expecting to execute the tests not entirely unlike this:
sbt test --ignore-tag integration
According to the sbt documentation on test options the following should work for you:
testOnly -- -l integration
Or in case ScalaTest uses the fully qualified tag id:
testOnly -- -l com.dreamlines.tags.integration
In case you want to do this by default, you can adjust the testOptions in the build.sbt file.
Edit:
You might consider a different approach for integration test. Instead of tagging them, you might want to put them in the src/it/{scala|resources} folders.
I have an sbt subproject that includes end to end tests. These are run as e2e:test. I have defined my config as
I have defined a tag in the same subproject.
object HealthCheckTest extends Tag("HealthCheckTest")
I am tagging some of my end to end tests with HealthCheckTest as follows:
it("should be able to verify the data", HealthCheckTest)
I want to run only the health check tests from command line. I am trying to do this via:
sbt 'project e2e' e2e:testOnly -- -n HealthCheckTest
but this leads to all of the e2e tests being run. I have tried giving the full path to the tag (com.s.p.e2etests.HealthCheckTest), but that does not work either.
Occasionally I get warnings about -- and - being deprecated; however, all documentation online says to use this syntax including scalatest docs.
How can I run just my tagged e2e tests?
I have also tried to create a separate task for health check tests but could only figure out how to filter based on test class name, not by tag.
The commandline you posted was missing quotes. Try it like this:
sbt 'project e2e' 'testOnly -- -n HealthCheckTest'
The sbt executable treats each argument as a full line to run in the sbt console, so you must place quotes around each full line.
Note that while this won't run the test cases, it will still instantiate test class and print out test case names.
When using the cobertura-maven-plugin for Java, I get a nice <instrumentation> block in the config where I can put the incredibly useful <ignoreMethodAnnotation> block.
Best thing to happen to coverage since the gin martini.
Now I'm using scoverage-sbt, but I can't seem to find where I can configure it! The available keys in build.scala are limited. I can do package exclusion and file exclusion but there's nowhere to tell cobertura anything else.
Is there a -D I can supply on the SBT command line, maybe?
There is no similar configuration in Scoverage ATM.
update:
You can use special comments to exclude a block of code from instrumentation:
// $COVERAGE-OFF
...
// $COVERAGE-ON$
One way to pass parameters and commands into SBT from the command line is:
$ sbt 'set coverageEnabled := true' clean coverage test coverageReport coverageAggregate codacyCoverage
Where you call SBT once and then separate each parameter or command by a space.
In this example, I first set the property coverageEnabled := true and then run several commands in sequence: clean, coverage, test, coveraReport, coverageAggregate and finally codacyCoverage
Please note that setting properties like this requires that you enclose your statements in single quotes, for example:
$ sbt 'set coverageEnabled := true'...
I have several tests distributed over several classes.
I would like to run only the first test from the first class:
The class is:
class Step1_PrimarySpec
The test is:
test("case1: Primary (in isolation) should properly register itself to the provided Arbiter")
I tried:
sbt test-only Step1_PrimarySpec
and:
sbt test-only Step1_PrimarySpec 1
and:
sbt test-only "*Step1_PrimarySpec 1"
and:
sbt test-only "*Step1_PrimarySpec"
However, all of these commands ran the entire test suite.
So how can I run only that specific test?
You must place the double quotes around the whole command like this:
sbt "test-only <test-name>"
And according to this answer you should camelCase it to testOnly and use the -z argument
sbt "testOnly *Step1_PrimarySpec -- -z mytest"
That will run the test named mytest from the class (not the file) named Step1_PrimarySpec.
By using the *, the test runner saves us from specifying its fully qualified class name. i.e. org.path.to.Step1_PrimarySpec
I usually do below command in my CI:
clean update compile test publish
However, I'd like to exclude 1 (or a few) test class from the sbt command line.
How can I do this? (I don't want to change my code to use ignore, etc)
Two possible options
test-only See http://www.scalatest.org/user_guide/using_scalatest_with_sbt
Tags http://www.scalatest.org/user_guide/tagging_your_tests
Just to elaborate on the 2 correct options #Gonfva suggested above:
To use testOnly you should run:
sbt "testOnly org.fully.qualified.domainn.name.ASpec"
When the argument is the FQDN of the class. You can use multiple classes separate them by space. This can be used with glob as well. For example:
sbt "testOnly *ASpec"
Using tags. First, define a tag:
import org.scalatest.Tag
object CustomTag extends Tag("tagName")
Then, define a test with this tag:
it should "test1" taggedAs CustomTag in { println("test1") }
Now, in order to include tests using this tag, run:
sbt "testOnly * -- -n tagName"
Note: * is a wild card. It can be any glob as described in section 1.
In order to exclude this tag, you need to run:
sbt "testOnly * -- -l aaa"
Including or excluding tests is dependent on the test framework you are using. The command you will be employing in SBT is not completely parsed by SBT, but partially parsed by SBT, partially parsed by the test framework you are using.
So, if you are suing scalameta/munit, you may enter a command like this:
sbt> myproject/Test/testOnly MyProjectTest -- --exclude-tags=tag1,tag2,tag3
If you are using another test framework, the specific syntax after -- will be probably different.
When you are writing your test cases, you have to add tags somewhere, obviously. More details can be found below:
scalameta/munit: https://scalameta.org/munit/docs/filtering.html
scalatest: https://www.scalatest.org/user_guide/tagging_your_tests (stolen from another answer)