How to use scalacheck prop generators in scalatest FlatSpec - scala

Question

I'm trying to use the scalacheck property generators in a scalatest.FlatSpec test file.
The test should fail and be reported by junit framework (and eclipse in my case) but the test pass and error is just displayed in console.
import scala.collection.immutable.TreeSet
import org.junit.runner.RunWith
import org.raisercostin.namek.UnitSpec
import org.scalatest.junit.JUnitRunner
import org.scalatest.FlatSpec
import org.scalatest._
#RunWith(classOf[JUnitRunner])
class SetsTest2 extends FlatSpec with Matchers
with OptionValues with Inside with Inspectors {
import org.scalacheck.Prop
"set intersection" should "be commutative" in {
Prop.forAll { (l1: TreeSet[Int], l2: TreeSet[Int]) =>
l1.intersect(l2) == l1.intersect(l1)
}.check
}
}
The output is the following
Run starting. Expected test count is: 1
SetsTest2:
set intersection
! Falsified after 1 passed tests.
> ARG_0: TreeSet(0)
> ARG_0_ORIGINAL: TreeSet(1288089760)
> ARG_1: TreeSet()
> ARG_1_ORIGINAL: TreeSet(0)
- should be commutative
Run completed in 505 milliseconds.
Total number of tests run: 1
Suites: completed 1, aborted 0
Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
I was expecting that the error is bubbled up to the junit framework.
I'm having the following dependencies:
scalaVersion = "2.10.4"
"junit" % "junit" % "4.10" % "test"
"org.scalatest" %% "scalatest" % "2.2.4" % "test"
"org.scalacheck" %% "scalacheck" % "1.12.2" % "test"

Answer 1

You should use scalatest.prop.Checkers that is different than the scalacheck.Prop.check
import scala.collection.immutable.TreeSet
import org.junit.runner.RunWith
import org.raisercostin.namek.UnitSpec
import org.scalatest.junit.JUnitRunner
import org.scalatest.FlatSpec
import org.scalatest._
import org.scalatest.prop.Checkers
#RunWith(classOf[JUnitRunner])
class SetsTest2 extends FlatSpec with Matchers
with OptionValues with Inside with Inspectors with Checkers {
import org.scalacheck.Prop
"set intersection" should "be commutative" in {
check(Prop.forAll { (l1: TreeSet[Int], l2: TreeSet[Int]) =>
l1.intersect(l2) == l1.intersect(l1)
})
}
}
Now the output is the following
Run starting. Expected test count is: 1
SetsTest2:
set intersection
- should be commutative *** FAILED ***
GeneratorDrivenPropertyCheckFailedException was thrown during property evaluation.
(SetsTest.scala:17)
Falsified after 1 successful property evaluations.
Location: (SetsTest.scala:17)
Occurred when passed generated values (
arg0 = TreeSet(0), // 1 shrink
arg1 = TreeSet() // 1 shrink
)
Run completed in 452 milliseconds.
Total number of tests run: 1
Suites: completed 1, aborted 0
Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***

Answer 2

For many people, raisercostin 's answer should be good enough. However, I have seen some issues where the most recent versions of ScalaCheck and ScalaTest are not fully integrated, and maybe you want some new feature.
However, one of the nice things about using a tool like sbt is that you can run both side-by-side. This may not be the best way to do it, but you can have your FlatSpec tests in one file and your ScalaCheck Props in another one, say like SetsTest2 and SetsProps2.
Then, when you run sbt test, it should just run all your tests and return correctly! To verify, I ran an intentionally false ScalaCheck Prop in a small application with 33 FlatSpec tests and 2 ScalaCheck Props and got
[info] ScalaTest
[info] Run completed in 2 seconds, 211 milliseconds.
[info] Total number of tests run: 33
[info] Suites: completed 8, aborted 0
[info] Tests: succeeded 33, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[error] Failed: Total 35, Failed 1, Errors 0, Passed 34
[error] Failed tests:
[error] com.xxx.xxx.TestProps
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful