Why does scalatest mix up the output? - scala

Question

I run my scalatest from sbt, and the output gets mixed up - scalatest prints all the test run, and comments to them, and somewhere in the middle it prints the statistics:
> test
[info] Compiling 1 Scala source to /home/platon/Tor/scala-dojo-02/target/scala-2.9.1/classes...
[info] FunsWithListsTests:
[info] - should return list of labels
[info] - should return the average rating of games belonging to Zenga
[info] - should return the total ratings of all games
[info] - should return the total ratings of EA games *** FAILED ***
[info] 0 did not equal 170 (FunsWithListsTests.scala:35)
[error] Failed: : Total 8, Failed 5, Errors 0, Passed 3, Skipped 0
[info] - should increase all games rating by 10 *** FAILED ***
[error] Failed tests:
[error] dojo.FunsWithListsTests
[info] List() did not equal List(Game(Activision,40), Game(Zenga,70), Game(Zenga,20), Game(EA,70), Game(EA,120)) (FunsWithListsTests.scala:40)
[info] - should decrease all Zenga games rating by 10 *** FAILED ***
[info] List() did not equal List(Game(Activision,30), Game(Zenga,50), Game(Zenga,0), Game(EA,60), Game(EA,110)) (FunsWithListsTests.scala:45)
[info] - should create function to find Activision games *** FAILED ***
[info] List(Game(Activision,30), Game(Zenga,60), Game(Zenga,10), Game(EA,60), Game(EA,110)) did not equal List(Game(Activision,30)) (FunsWithListsTests.scala:50)
[info] - should return a List of tuples consisting of game label and game *** FAILED ***
[info] List() did not equal List((ACTIVISION,Game(Activision,30)), (ZENGA,Game(Zenga,60)), (ZENGA,Game(Zenga,10)), (EA,Game(EA,60)), (EA,Game(EA,110))) (FunsWithListsTests.scala:56)
[error] {file:/home/platon/Tor/scala-dojo-02/}default-940f03/test:test: Tests unsuccessful
[error] Total time: 1 s, completed Mar 20, 2012 9:27:13 AM
It seems that if I would accumulate a great number of tests, searching for those stats and failed tests would become a pain.
Is there a way to fix this?

Answer 1

It looks to me the reason for that is that SBT by default executes the tests in parallel, the same way maven-surefire-plugin does.
As it is explained in ScalaTest wiki, this can be solved by:
Disable Parallel Execution of Tests
By default, sbt runs all tasks in parallel. Because each test is mapped to a task, tests are also run in parallel by default. To disable parallel execution of tests:
parallelExecution in Test := false

Answer 2

Have the same issue and solved it by saving logs into separate files. Not ideal, but good for CI, especially in case of long-running integration tests. How I did it:
Create a special directory for logs (for example, at the end of build.sbt):
lazy val logDirectory = taskKey[File]("A directory for logs")
logDirectory := {
val r = target.value / "logs"
IO.createDirectory(r)
r
}
[Optionally] Specify a file for a summary information in a project:
testOptions in test += Tests.Argument(
TestFrameworks.ScalaTest, "-fW", (logDirectory.value / "summary.log").toString
)
W disables colors
Create a group for each test. Each group must run a test in a forked JVM with special arguments:
testGrouping in test := testGrouping.value.flatMap { group =>
group.tests.map { suite =>
val fileName = {
// foo.bar.baz.TestSuite -> f.b.b.TestSuite
val parts = suite.name.split('.')
(parts.init.map(_.substring(0, 1)) :+ parts.last).mkString(".")
}
val forkOptions = ForkOptions(
bootJars = Nil,
javaHome = javaHome.value,
connectInput = connectInput.value,
outputStrategy = outputStrategy.value,
runJVMOptions = javaOptions.value ++ Seq(
"-Dour.logging.appender=FILE",
s"-Dour.logging.dir=${logDirectory.value / fileName}"
),
workingDirectory = Some(baseDirectory.value),
envVars = envVars.value
)
group.copy(
name = suite.name,
runPolicy = Tests.SubProcess(forkOptions),
tests = Seq(suite)
)
}
}
Note, we can tell the logback, where we save our logs through our.logging.appender and our.logging.dir
Create a configuration for logging (test/resources/logback-test.xml):
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<target>System.out</target>
<encoder>
<pattern>%date{yyyy-MM-dd HH:mm:ss} %-5level [%thread] %logger{26} - %msg%n</pattern>
</encoder>
</appender>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>${our.logging.dir}/test.log</file>
<append>false</append>
<encoder>
<pattern>%date{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%thread] %logger{26} - %msg%n</pattern>
</encoder>
</appender>
<root level="TRACE">
<appender-ref ref="${our.logging.appender}"/>
</root>
</configuration>
That's all.