How can I mark some ScalaTests so that they only execute when invoked explicitly - scala

Question

I'm new to SBT and scalatest but am wondering how I get some of my org.scalatest._ tests to only execute "on demand".
In SBT I can invoke all of the Unit Tests like sbt:Sprout> test, or all of the IntegrationTests like sbt:Sprout> it:test. I need a way to annotate the tests that allows an sbt:Sprout test invocation to skip them but with some other invocation executes ONLY these tests. Scalatest docs speak of some sbt:Sprout> test-only *RedSuite invocation to allow me to "Categorize" my tests, but its not clear how to leverage that so they DON'T run as Unit Tests. org.scalatest.Tags alone do not get it off the "default" of getting executed when sbt:Sprout> test. I need these to be ignored unless invoked explicitly.
Is this use case possible in ScalaTest through SBT?

Answer 1

You can specify tag names of tests to include or exclude from a run. To specify tags to include, use -n followed by a list of tag names to include. Similarly, to specify tags to exclude, use -l followed by a list of tag names to exclude
(please look here for more info in the official documents).
For example:
package com.test
import org.scalatest.FlatSpec
import org.scalatest.Tag
object IncludeTest extends Tag("com.tags.Include")
object ExcludeTest extends Tag("com.tags.Exclude")
class TestSuite extends FlatSpec {
"Test1" taggedAs(IncludeTest) in {
val sum = 1 + 1
assert(sum === 2)
}
"Test2" taggedAs(ExcludeTest) in {
val minus = 2 - 1
assert(minus === 1)
}
}
To include IncludeTest and exclude ExcludeTest tag, you should do:
test-only org.* -- -n com.tags.Include -l com.tags.Exclude

Answer 2

assume assertion specified inside a fixture-context object could be used to implement conditional-ignore semantics dependant on environmental flag. For example, consider the following IfIgnored fixture
trait IfIgnored extends Assertions {
assume(System.getenv("runIgnoredTest").toBoolean, "!!! TEST IGNORED !!!")
}
which can be instantiated like so
it should "not say goodbye" in new IfIgnored {
Hello.greeting shouldNot be ("goodbye")
}
Now if we define the following settings in build.sbt
Test / fork := true,
Test / envVars := Map("runIgnoredTest" -> "false")
and the following tests
class HelloSpec extends FlatSpec with Matchers {
"The Hello object" should "say hello" in {
Hello.greeting shouldEqual "hello"
}
it should "not say goodbye" in new IfIgnored {
Hello.greeting shouldNot be ("goodbye")
}
it should "not say live long and prosper" in new IfIgnored {
Hello.greeting shouldNot be ("live long and prosper")
}
}
then executing sbt test should output
[info] HelloSpec:
[info] The Hello object
[info] - should say hello
[info] - should not say goodbye !!! CANCELED !!!
[info] scala.Predef.augmentString(java.lang.System.getenv("runIgnoredTest")).toBoolean was false !!! TEST IGNORED !!! (HelloSpec.scala:6)
[info] - should not say live long and prosper !!! CANCELED !!!
[info] scala.Predef.augmentString(java.lang.System.getenv("runIgnoredTest")).toBoolean was false !!! TEST IGNORED !!! (HelloSpec.scala:6)
[info] Run completed in 2 seconds, 389 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 2, ignored 0, pending 0
[info] All tests passed.
where we see only should say hello ran whilst the rest were ignored.
To execute only ignored tests we could define the following custom command testOnlyIgnored:
commands += Command.command("testOnlyIgnored") { state =>
val ignoredTests = List(
""""should not say goodbye"""",
""""should not say live long and prosper""""
).mkString("-z ", " -z ", "")
"""set Test / envVars := Map("runIgnoredTest" -> "true")""" ::
s"""testOnly -- $ignoredTests""" :: state
}
Note how we are making use of -z runner argument to run particular tests, for example,
testOnly -- -z "should not say goodbye" -z "should not say live long and prosper"
Also note how we are manually adding test names to ignoredTests. Now executing sbt testOnlyIgnored should output
[info] HelloSpec:
[info] The Hello object
[info] - should not say goodbye
[info] - should not say live long and prosper
[info] Run completed in 2 seconds, 298 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
where we see should say hello was not run whilst all the ignored tests ran.
If we drop the requirement of having to run ignored tests separately, then we can use provided ignore annotation like so
ignore should "not say goodbye" in {
Hello.greeting shouldNot be ("goodbye")
}
which on sbt test outputs
[info] HelloSpec:
[info] The Hello object
[info] - should say hello
[info] - should not say goodbye !!! IGNORED !!!
[info] - should not say live long and prosper !!! IGNORED !!!
[info] Run completed in 2 seconds, 750 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 2, pending 0