How to run subprojects tests (including setup methods) sequentially when testing - scala

Question

EDIT4 Most of the text below the horizontal rule doesn't have anything to do with the real problem. At the beginning I thought forking is the problem, but that is not true.
I'm trying to run all the tests of an aggregates subprojects. Before all the tests in a subproject a setup method should run, and after the tests in a subproject a cleanup method should run.
When running the test task for the aggregate project, I expect the following sequence
setup method of project A is called
tests for project A are executed
cleanup method for project A is called
the same for project B
But the sequence is:
setup methods of project A and B are called
tests of project A are executed
tests of project B are executed
cleanup methods of project A and B are called
A build script having this behaviour can be found here.
How can I fix this problem to get my expected sequence?
I'm using forking to run the tests in my subprojects.
For every subproject a mongo db is started before the tests and stopped
after the tests.
The tests within one project run sequentially; this works well if I run the tests for a single project.
But if I run the task test for project root (the aggregate containing the sub projects),
I want the forked jvms to be started sequentially, i.e.
the jvm for project A is forked and its tests are executed
the jvm for project B is forked and its tests are executed
...
But it looks like the jvms are started in parallel; this is not what I want.
I tried the following (which should be set to 1 already, according to the documentation):
concurrentRestrictions in Test := Seq(
Tags.limit(Tags.ForkedTestGroup, 1)
)
But it didn't work. Directly after starting the test task, the following is
printed (before any test log is printed) from my setup method:
startupDb, thread name = pool-4-thread-5
startupDb, thread name = pool-4-thread-7
startupDb, thread name = pool-4-thread-2
startupDb, thread name = pool-4-thread-6
startupDb, thread name = pool-4-thread-8
startupDb, thread name = pool-4-thread-3
startupDb, thread name = pool-4-thread-9
These are my test related settings:
parallelExecution in Test := false,
testOptions in Test += Tests.Setup( () => MongoTest.startupDb() ),
testOptions in Test += Tests.Cleanup( () => MongoTest.shutdownDb() ),
fork in Test := true,
concurrentRestrictions in Test := Seq(
Tags.limit(Tags.ForkedTestGroup, 1)
)
It's important for me to use forking, but in the way as described above.
I'm using sbt 0.13.0 on Windows 7.
EDIT I created a gist with a sample build.
EDIT2 In the documentation it says:
Control the number of forked JVMs allowed to run at the same time by setting the limit on Tags.ForkedTestGroup tag, which is 1 by default
So this should work theoretically.
Is this a bug? If not, how can I manage to do that?
EDIT3 It looks like forking is not the problem. The problem is that the setup methods are called immediately for all subprojects.

Answer 1

Hmm. Trivial answer convert to comment. It sounds like you are asking the same question that got solved in the other question (see comments). The answer seems to be
parallelExecution in ThisBuild := false
and I found another blog post that suggested
parallelExecution in Global := false
to be the answer. The "Global" answer also suggested that would turn off parallel execution for things like compilation but that's probably less important than running tests.

Answer 2

As of sbt 0.12, you can make all test tasks in all subprojects run serially with this setting:
concurrentRestrictions in Global += Tags.limit(Tags.Test, 1)
(Tags.Test is a built in semantic tag for all test tasks).
See: Parallel-Execution in the SBT manual.

Answer 3

There is another way to prevent parallel execution. You can make the test tasks of the different projects depend on each other:
test in Project2 := (test in Project2).dependsOn(test in Project1).value
parallelExecution in Test in Project2 := false