How can I get a return value from ScalaTest indicating test suite failure? - scala

Question

I'm running a ScalaTest (FlatSpec) suite programmatically, like so:
new MyAwesomeSpec().execute()
Is there some way I can figure out if all tests passed? Suite#execute() returns Unit here, so does not help. Ideally, I'd like to run the whole suite and then get a return value indicating whether any tests failed; an alternative would be to fail/return immediately on any failed test.
I can probably achieve this by writing a new FlatSpec subclass that overrides the Scalatest Suite#execute() method to return a value, but is there a better way to do what I want here?

Answer 1

org.scalatest.Suite also has run function, which returns the status of a single executed test.
With a few tweaking, we can access the execution results of each test. To run a test, we need to provide a Reporter instance. An ad-hoc empty reporter will be enough in our simple case:
val reporter = new Reporter() {
override def apply(e: Event) = {}
}
So, let's execute them:
import org.scalatest.events.Event
import org.scalatest.{Args, Reporter}
val testSuite = new MyAwesomeSpec()
val testNames = testSuite.testNames
testNames.foreach(test => {
val result = testSuite.run(Some(test), Args(reporter))
val status = if (result.succeeds()) "OK" else "FAILURE!"
println(s"Test: '$test'\n\tStatus=$status")
})
This will produce output similar to following:
Test: 'This test should pass'
Status=OK
Test: 'Another test should fail'
Status=FAILURE!
Having access to each test case name and its respective execution result, you should have enough data to achieve your goal.