How to run Test Suites sequentially in ScalaTest / SBT?
For example if I have this test suites A, B and C I want to make sure that the tests of A will be run 1st then the ones of B then the ones of C.
Is there in configuration that I can make in Scalatest or SBT?
Thank you.
try using parallelExecution in Test := false
According to the documentation http://doc.scalatest.org/1.7/org/scalatest/Suite.html
You need to create your own Test Suite like the following:
FirstTest.scala
import org.scalatest.{DoNotDiscover, FunSuite}
#DoNotDiscover
class FirstTest extends FunSuite {
test("first test"){
assert(1 == 1)
}
}
SecondTest.scala
import org.scalatest.{DoNotDiscover, FunSuite}
#DoNotDiscover
class SecondTest extends FunSuite{
test("second test"){
assert(2 == 2)
}
}
MainTest.scala
import org.scalatest.Suites
class MainTest extends Suites (new FirstTest,new SecondTest)
Now, if you run sbt test it's work properly.
Notes: the property #DoNotDiscover is mandatory. this avoid unexpected behavior like execution of FirstTest and SecondTest after the execution of the MainSuite that are already executed the two test suites.
I hope it was helpful
As raj mehra said, the solution is to configure to run the tests not in parallel.
Test / parallelExecution := false
Former parallelExecution in Test := false is deprecated.
Here is the Documentation that explains it: SBT Parallel Execution
From it:
As before, parallelExecution in Test controls whether tests are mapped to separate tasks.
Related
I have a bunch of integration tests running by sbt, given test N suites with 1..M tests per each suite.
I have set fork in IntegrationTest := true, but test suites are always executed sequentially. According to the docs, this must not be the case: test suites should be executed concurrently.
the test suites are a class as following:
class MyTestSuite1 extends FlatSpec with Matchers
...
it should "do A" {}
it should "do B" {}
class MyTestSuite2 extends FlatSpec with Matchers
...
it should "do C" {}
it should "do D" {}
the problem
MyTestSuite1 and MyTestSuiteN are executed sequentially (by the alphabet order to be exact)
expectation
MyTestSuite1 and MyTestSuiteM are executed concurrently
env
.sbopts:
-J-Xms1G
-J-Xmx4G
-J-XX:MaxMetaspaceSize=512m
-J-Xss4M
note
I noticed that all test are running using the same pool and thread, for example, pool-1-thread-1 for all tests.
sbt version: 1.2.8
Scala: 2.12.8
os: MacOS 10.15, Ubuntu 19.04
Scalatest ver: 3.2.0-SNAP10
Tried sbt v. 1.3.2 - same result.
Adding
testOptions in IntegrationTest += Tests.Argument(TestFrameworks.ScalaTest, "-P4"),
does not help.
============
Update
fork in(IntegrationTest, test) := true works on a global level, but I have 2 projects and I want to make it work to preserve relative path to the proj.
e.g.
lazy val `p1` = Project(id = "p1", base = file("./p1"))
.configs(IntegrationTest)
.settings(Defaults.itSettings: _*)
.settings(
fork in(IntegrationTest, test) := true,
...)
lazy val `p2` = Project(id = "p2", base = file("./p2"))
.configs(IntegrationTest)
.settings(Defaults.itSettings: _*)
.settings(
fork in(IntegrationTest, test) := true,
...)
does not run tests in parallel
instead, this runs in parallel, but, obviously, the home dir is set to be "." rather than to be "./p1" or "./p2" respectively:
fork in(IntegrationTest, test) := true
lazy val `p1` = Project(id = "p1", base = file("./p1"))
.configs(IntegrationTest)
.settings(Defaults.itSettings: _*)
SOLUTION:
It appears there's testForkedParallel in IntegrationTest := true option which does exactly what I needed - it spawns new JVM per test suite.
==============
Remark:
So, the only problem is that now it spawns JVMs as many as the count of all available CPUs and I can't funnel only test concurrency:
OPTION 1 - funnels all sbt processes to be only 4 in parallel
concurrentRestrictions in Global := Seq(Tags.limitAll(4))
OPTION 2 - just does nothing (test are in the subproject)
concurrentRestrictions in Global += Tags.limit(Tags.Test, 4),
By default tests executed in forked JVM are executed sequentially. Refer the following para from sbt testing docs:
The setting:
Test / fork := true
specifies that all tests will be executed in a single external JVM.
See Forking for configuring standard options for forking. By default,
tests executed in a forked JVM are executed sequentially. More control
over how tests are assigned to JVMs and what options to pass to those
is available with testGrouping key.
So, you have two options:
Do not fork JVM and your tests will run parallelly by default
If you want to fork and still rut tests parallelly go through this docs:
https://www.scala-sbt.org/1.x/docs/Testing.html#Forking+tests
Are these 2 settings alternatives or are both necessary to run the tests sequentially ?
Global / concurrentRestrictions += Tags.limit(Tags.Test, 1)
And :
parallelExecution := false
Another question :
Does the first configuration guarantee to have only one task per all the tests whatever it is their Test Suite?
I know, I saw Run custom task automatically before/after standard task but it seems outdated. I also found SBT before/after hooks for a task but it does not have any code example.
I am on SBT 0.13.17.
So I want to run my task MyBeforeTask and MyAfterTask automatically after an other tasks, says Compile.
So when you do sbt compile I would like to see:
...log...
This is my before test text
...compile log...
This is my after test text
So I would need to have:
object MyPlugin extends AutoPlugin {
object autoImport {
val MyBeforeTask = taskKey[Unit]("desc...")
val MyAfterTask = taskKey[Unit]("desc...")
}
import autoImport._
override def projectSettings: Seq[Def.Setting[_]] = {
MyBeforeTask := {
println("This is my before test text")
},
MyAfterTask := {
println("This is my after test text")
}
}
}
So I think I need things like dependsOn and in but I am not sure how to set them up.
It is not possible to configure for a particular task to run after the given task, because that's not how the task dependencies model works - when you specify the task, its dependencies and itself will be executed, but there is no way to define an "after" dependency. However, you can simulate that with dynamic tasks.
To run some task before another, you can use dependsOn:
compile in Compile := (compile in Compile).dependsOn(myBeforeTask).value
This establishes a dependency between two tasks, which ensures that myBeforeTask will be run before compile in Compile.
Note that there is a more generic way to make multiple tasks run one after another:
aggregateTask := Def.sequential(task1, task2, task3, task4).value
Def.sequential relies on the dynamic tasks machinery, which sets up dependencies between tasks at runtime. However, there are some limitations to this mechanism, in particular, you cannot reference the task being defined in the list of tasks to execute, so you can't use Def.sequential to augment existing tasks:
compile in Compile := Def.sequential(myBeforeTask, compile in Compile).value
This definition will fail at runtime with a strange error message which basically means that you have a loop in your task dependencies graph. However, for some use cases it is extremely useful.
To run some task after another, however, you have to resort to defining a dynamic task dependency using Def.taskDyn:
compile in Compile := Def.taskDyn {
val result = (compile in Compile).value
Def.task {
val _ = myAfterTask.value
result
}
}.value
Def.taskDyn accepts a block which must return a Def.Initialize[Task[T]], which will be used to instantiate a task to be run later, after the main body of Def.taskDyn completes. This allows one to compute tasks dynamically, and establish dependencies between tasks at runtime. As I said above, however, this can result in very strange errors happening at runtime, which are usually caused by loops in the dependency graph.
Therefore, the full example, with both "before" and "after" tasks, would look like this:
compile in Compile := Def.taskDyn {
val result = (compile in Compile).value
Def.task {
val _ = myAfterTask.value
result
}
}.dependsOn(myBeforeTask).value
I want to disable certain automated tests tagged as "Slow" by default but allow the user to enable their execution with a simple command line. I imagine this is a very common use case.
Given this test suite:
import org.scalatest.FunSuite
import org.scalatest.tagobjects.Slow
class DemoTestSuite extends FunSuite {
test("demo test tagged as slow", Slow) {
assert(1 + 1 === 2)
}
test("demo untagged test") {
assert(1 + 1 === 2)
}
}
By default, sbt test will run both tagged and untagged tests.
If I add the following to my build.sbt:
testOptions in Test += Tests.Argument("-l", "org.scalatest.tags.Slow")
Then I get my desired default behavior where untagged tests run and the Slow tagged test will not run.
However, I can't figure out a command line option that will run the slow tests when I want to run them. I've done several searches and tried several examples. I'm somewhat surprised as this seems like a very common scenario.
I had a similar issue: I wanted to have tests that are disabled by default, but run in the release process. I solved it by creating a custom test configuration and setting testOptions in different scopes. So adapting this solution to your case, it should be something along these lines (in your build.sbt):
lazy val Slow = config("slow").extend(Test)
configs(Slow)
inConfig(Slow)(Defaults.testTasks)
Now by default exclude slow tests:
testOptions in Test += Tests.Argument("-l", "org.scalatest.tags.Slow")
But in the Slow scope don't exclude them and run only them:
testOptions in Slow -= Tests.Argument("-l", "org.scalatest.tags.Slow")
testOptions in Slow += Tests.Argument("-n", "org.scalatest.tags.Slow")
Now when you run test in sbt, it will run everything except slow test and when you run slow:test it will run only slow tests.
How can an application tell if it's running under an 'sbt test' context? Is there a system property that can be checked?
There are probably different ways. I found the following works:
testOptions += Tests.Setup(_ => sys.props("testing") = "true")
And then you can just test for sys.props.get("testing") in your class.