ScalaCheck specificy minimum successful tests for property - scala

I'm trying to make sure that my ScalaCheck property runs 500 times instead of the default 100 times. I'm having trouble configuring this though.
class BlockSpec extends Properties("BlockSpec") with BitcoinSLogger {
val myParams = Parameters.default.withMinSuccessfulTests(500)
override def overrideParameters(p: Test.Parameters) = myParams
property("Serialization symmetry") =
Prop.forAll(BlockchainElementsGenerator.block) { block =>
logger.warn("Hex:" + block.hex)
Block(block.hex) == block
}
}
However when I actually run this test it only says 100 tests passed successfully
EDIT:
$ sbt
[info] Loading project definition from /home/chris/dev/bitcoins-core/project
[info] Set current project to bitcoin-s-core (in build file:/home/chris/dev/bitcoins-core/)
> test-only *BlockSpec*
[info] + BlockSpec.Serialization symmetry: OK, passed 100 tests.
[info] Elapsed time: 1 min 59.775 sec
[info] ScalaCheck
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[info] ScalaTest
[info] Run completed in 2 minutes.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 123 s, completed Aug 1, 2016 11:36:17 AM
>
How do I actually pass this to my property?

As far as I understand you can specify the test parameters at two levels and they don't seem to communicate.
The first option is within the property as you're trying to do:
import org.scalacheck.Properties
import org.scalacheck.Test.{ TestCallback, Parameters }
import org.scalacheck.Prop.{ forAll, BooleanOperators }
import org.scalacheck.Test
class TestFoo extends Properties("BlockSpec") {
override def overrideParameters(p: Parameters) =
p.withMinSuccessfulTests(1000000)
property("Serialization symmetry") = forAll { n: Int =>
(n > 0) ==> (math.abs(n) == n)
}
}
This will have no impact as long as you don't call .check on the property.
Can be from the sbt shell or directly within the class.
Now if you want to impact the number of tests run when calling the sbt:test target, it seems you have to play with options build.sbt (taken from here):
name := "scalacheck-demo"
scalaVersion := "2.11.5"
libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.12.2" % "test"
testOptions in Test += Tests.Argument(TestFrameworks.ScalaCheck, "-maxSize", "5", "-minSuccessfulTests", "33", "-workers", "1", "-verbosity", "1")

There's definitely an easier way to achieve that than overriding any kind of global test config:
class SampleTest extends FlatSpec
with Matchers with GeneratorDrivenPropertyChecks {
it should "work for a basic scenario" in {
// This will require 500 successful tests to succeed
forAll(minSuccessful(500)) { (d: String) =>
whenever (d.nonEmpty) {
d.length shouldBe > 0
}
}
}
}

Related

ScalaTest: no tests are run for GeneratorDrivenPropertyChecks

I am trying to set-up a property based testing with ScalaTest and ScalaCheck ... and based on the ouput it seems that I am succeeding, but it takes too fast and from what I understand normally ScalaCheck should inform you about how may tests were run, in my case this information is absent:
[IJ]sbt:algorithms2_1> testOnly *MedianOf3PartitioningProps
[info] Compiling 1 Scala source to /Users/vasile.gorcinschi/gitPerso/Algorithms/Chapter 2 Sorting/algorithms2_1/target/scala-2.12/test-classes ...
[warn] there was one deprecation warning; re-run with -deprecation for details
[warn] one warning found
[info] Done compiling.
[info] MedianOf3PartitioningProps:
[info] sort
[info] - should sort array of ints from 0 to 100
[info] +
[info] ScalaTest
[info] Run completed in 412 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
Here is the test class:
class MedianOf3PartitioningProps extends FlatSpec with Matchers with GeneratorDrivenPropertyChecks with Gens {
private val medianOf3Partitioning = new MedianOf3Partitioning[Int]
implicit override val generatorDrivenConfig: PropertyCheckConfiguration = PropertyCheckConfig(minSuccessful = 1, maxDiscarded = 500, workers = 1)
behavior of "sort"
it should "sort array of ints from 0 to 100" in {
forAll(arraysGen){ a: Array[Int] =>
info(s"${a.mkString(",")}")
medianOf3Partitioning.sort(a) shouldEqual a.sorted }
}
}
The Gens trait is mine - it only comprises definition for the Gen[Array[Int]]:
trait Gens {
val arraysGen: Gen[Array[Int]] = containerOf[Array, Int](
chooseNum(Int.MinValue, Int.MaxValue) suchThat { _ < 100 }
).suchThat(_.length < 50)
}
I used this source for the test set-up. Just in case, I'm providing versions of scalacheck and scalatest (from Dependencies.scala and build.sbt):
lazy val scalaTest = "org.scalatest" %% "scalatest" % "3.0.5"
...
libraryDependencies ++= {
val scalaTestVersion = "3.0.5"
val scalaCheckVersion = "1.14.0"
Seq(scalaTest % Test,
"org.scalatest" %% "scalatest" % scalaTestVersion % "test",
"org.scalacheck" %% "scalacheck" % scalaCheckVersion % "test",
"com.storm-enroute" %% "scalameter" % "0.9"
)
}
Based on the small example from M. Odersky's "Programming in Scala", I switched from GeneratorDrivenPropertyChecks to more general PropertyChecks. I've also discovered issues with my Gen[Array[Int]] so I had to tweek that too. Posting a solution that worked (discovered cases that fail) in case this will help anyone else:
Gens trait:
trait Gens {
val minIntArraysGen: Gen[Array[Int]] = containerOf[Array, Int](Gen.chooseNum(0, 100))
}
The property based test:
import ca.vgorcinschi.Gens
import org.scalatest.MustMatchers._
import org.scalatest.WordSpec
import org.scalatest.prop.PropertyChecks
class MedianOf3PartitioningProps extends WordSpec with PropertyChecks with Gens {
"sort method" must {
"sort any Int array" in {
forAll (minIntArraysGen){ (a: Array[Int]) =>
whenever(a.nonEmpty) {
val maybeSorted = new MedianOf3Partitioning[Int].sort(a)
maybeSorted must equal (a.sorted)
}
}
}
}
}

Property check should fail in Scalatest

I've created and am also using some external Scalacheck generators from Lists and appropriate types, using Gen.oneOf(List[T]) . I think it would be occasionally useful to return a placeholder for an empty value. Currently the lists are populated. How should I go about this? Do I try to append an empty type to the end of the list? If so how do I go about that? If not, how else can I get my generators to add an empty value. It seems straightforward, but I am having trouble figuring it out right now.
import org.scalatest.FlatSpec
import org.scalacheck.Gen
import org.scalacheck.Prop.exists
import org.scalatest.prop.PropertyChecks
class EventFieldGeneratorTest extends FlatSpec with PropertyChecks {
behavior of "Gen.option"
it should "occasionally return None" in {
val colors = Gen.oneOf("Blue", "Red", "Green", "Yellow")
val opt = Gen.option(colors)
val list = Gen.listOfN(20, opt)
val p1 = exists(list)(_ == None)
p1.check
}
}
Can anybody explain why my test is giving up?
Testing started at 10:31 AM ... ! Gave up after only 0 passed tests. 501 tests were discarded.
Process finished with exit code 0
How can I mark that as a failed result for ScalaTest? Is it a bad idea for me to use Flatspec?
Maybe I should be using something other than check...
Here's the documentation I used to sort it on. On the Scalatest page:
http://www.scalatest.org/user_guide/writing_scalacheck_style_properties
I don't think there's anything flawed with trying to use lists with optional values. There are just a few issues you're running in to that are giving you trouble.
It's confusing, but if you're using the Scalatest framework, you need to use the Scalatest infrastructure to use Scalacheck. So you'll need to use Scalatest matchers, and write Scalatest-flavored properties (using it's forAll), but you'll still use Scalacheck's generators directly.
For some reason the type inference between lists and Option type is giving you trouble. If you use the shouldBe matcher,
x shouldBe(None)
you'll get a relevant runtime error from Scalatest:
[info] - should occasionally return None *** FAILED ***
[info] TestFailedException was thrown during property evaluation.
[info] Message: List() was not equal to None
[info] Location: (GenTest.scala:13)
[info] Occurred when passed generated values (
[info] arg0 = List() // 5 shrinks
[info] )
[info] Run completed in 1 second, 621 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
You shouldn't be matching a list with an Option type. You need to be matching with the Scalatest "container" matcher should contain
import org.scalatest.FlatSpec
import org.scalatest.Matchers
import org.scalacheck.Gen
import org.scalatest.prop.PropertyChecks
class EventFieldGeneratorTest extends FlatSpec with Matchers with PropertyChecks {
behavior of "Gen.option"
it should "occasionally return None" in {
val colors = Gen.oneOf("Blue","Red","Green","Yellow")
val opt = Gen.option(colors)
val list = Gen.listOfN(20,opt)
forAll(list) { (xs: List[Option[String]]) =>
xs should contain (None)
}
}
}
This gives you a successful property check:
[info] EventFieldGeneratorTest:
[info] Gen.option
[info] - should occasionally return None
[info] ScalaTest
[info] Run completed in 1 second, 9 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
More on Scalatest matchers
http://www.scalatest.org/user_guide/using_matchers

Running setup/teardown tasks during stb test task

I try to configure sbt to run a setup task at the beginning and a tear down task at the end of myProject/test command.
My build.sbt is here:
name := "ch-2"
version := "1.0"
libraryDependencies += "org.specs2" % "specs2_2.10" % "1.14" % "test"
lazy val common = (
Project("common", file("common")).
settings()
)
lazy val subPro = (
Project("sub", file("subA")).settings(
).dependsOn(common)
settings(libraryDependencies += "org.specs2" % "specs2_2.10" % "1.14" % "test" )
)
val startS = taskKey[Unit]("Start")
val stopS = taskKey[Unit]("Stop")
startS := { println("Running start")}
stopS := { println("Running stop")}
testOptions in Test in subPro += Tests.Setup { () => startS.value }
testOptions in Test in subPro += Tests.Cleanup { () => stopS.value }
The actual dummy test class is here:
import org.specs2.mutable.Specification
/**
* Created by jk on 26.3.2017.
*/
object FooSpec extends Specification {
"The TEST method" should {
"blaa blaa 1" in {
println("test 1 running...")
true
}
"blaa blaa 2" in {
println("test 2 running...")
true
}
}
}
When I run the tests for project sub, I get following output:
> sub/test
Running stop
Running start
[info] Updating {file:/home/jk/workspace/sbt-in-action/ch2/}sub...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 1 Scala source to /home/jk/workspace/sbt-in-action/ch2/subA/target/scala-2.10/test-classes...
test 1 running...
test 2 running...
[info] FooSpec
[info]
[info] The TEST method should
[info] + blaa blaa 1
[info] + blaa blaa 2
[info]
[info]
[info] Total for specification FooSpec
[info] Finished in 18 ms
[info] 2 examples, 0 failure, 0 error
[info]
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2
[success] Total time: 3 s, completed Mar 26, 2017 7:09:34 PM
Why is stop task run even before compilation is finished and how to fix it so that it is run after all test cases are run (despite the result of the test cases)?
Also the start task should run after successfull compilation but before first test case. How to fix these?
How about doing it like this instead?
def startS(): Unit = { println("Running start")}
def stopS(): Unit = { println("Running stop")}
testOptions in Test in subPro += Tests.Setup { () => startS() }
testOptions in Test in subPro += Tests.Cleanup { () => stopS() }

How to use scalacheck prop generators in scalatest FlatSpec

I'm trying to use the scalacheck property generators in a scalatest.FlatSpec test file.
The test should fail and be reported by junit framework (and eclipse in my case) but the test pass and error is just displayed in console.
import scala.collection.immutable.TreeSet
import org.junit.runner.RunWith
import org.raisercostin.namek.UnitSpec
import org.scalatest.junit.JUnitRunner
import org.scalatest.FlatSpec
import org.scalatest._
#RunWith(classOf[JUnitRunner])
class SetsTest2 extends FlatSpec with Matchers
with OptionValues with Inside with Inspectors {
import org.scalacheck.Prop
"set intersection" should "be commutative" in {
Prop.forAll { (l1: TreeSet[Int], l2: TreeSet[Int]) =>
l1.intersect(l2) == l1.intersect(l1)
}.check
}
}
The output is the following
Run starting. Expected test count is: 1
SetsTest2:
set intersection
! Falsified after 1 passed tests.
> ARG_0: TreeSet(0)
> ARG_0_ORIGINAL: TreeSet(1288089760)
> ARG_1: TreeSet()
> ARG_1_ORIGINAL: TreeSet(0)
- should be commutative
Run completed in 505 milliseconds.
Total number of tests run: 1
Suites: completed 1, aborted 0
Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
I was expecting that the error is bubbled up to the junit framework.
I'm having the following dependencies:
scalaVersion = "2.10.4"
"junit" % "junit" % "4.10" % "test"
"org.scalatest" %% "scalatest" % "2.2.4" % "test"
"org.scalacheck" %% "scalacheck" % "1.12.2" % "test"
You should use scalatest.prop.Checkers that is different than the scalacheck.Prop.check
import scala.collection.immutable.TreeSet
import org.junit.runner.RunWith
import org.raisercostin.namek.UnitSpec
import org.scalatest.junit.JUnitRunner
import org.scalatest.FlatSpec
import org.scalatest._
import org.scalatest.prop.Checkers
#RunWith(classOf[JUnitRunner])
class SetsTest2 extends FlatSpec with Matchers
with OptionValues with Inside with Inspectors with Checkers {
import org.scalacheck.Prop
"set intersection" should "be commutative" in {
check(Prop.forAll { (l1: TreeSet[Int], l2: TreeSet[Int]) =>
l1.intersect(l2) == l1.intersect(l1)
})
}
}
Now the output is the following
Run starting. Expected test count is: 1
SetsTest2:
set intersection
- should be commutative *** FAILED ***
GeneratorDrivenPropertyCheckFailedException was thrown during property evaluation.
(SetsTest.scala:17)
Falsified after 1 successful property evaluations.
Location: (SetsTest.scala:17)
Occurred when passed generated values (
arg0 = TreeSet(0), // 1 shrink
arg1 = TreeSet() // 1 shrink
)
Run completed in 452 milliseconds.
Total number of tests run: 1
Suites: completed 1, aborted 0
Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***
For many people, raisercostin 's answer should be good enough. However, I have seen some issues where the most recent versions of ScalaCheck and ScalaTest are not fully integrated, and maybe you want some new feature.
However, one of the nice things about using a tool like sbt is that you can run both side-by-side. This may not be the best way to do it, but you can have your FlatSpec tests in one file and your ScalaCheck Props in another one, say like SetsTest2 and SetsProps2.
Then, when you run sbt test, it should just run all your tests and return correctly! To verify, I ran an intentionally false ScalaCheck Prop in a small application with 33 FlatSpec tests and 2 ScalaCheck Props and got
[info] ScalaTest
[info] Run completed in 2 seconds, 211 milliseconds.
[info] Total number of tests run: 33
[info] Suites: completed 8, aborted 0
[info] Tests: succeeded 33, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[error] Failed: Total 35, Failed 1, Errors 0, Passed 34
[error] Failed tests:
[error] com.xxx.xxx.TestProps
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful

Play test-only doesn't ignore tests

I have a play application and I need to ignore my functional tests when I compile and build, then later run only the integration tests.
This is my test code:
ApplicationSpec.scala
#RunWith(classOf[JUnitRunner])
class ApplicationSpec extends Specification {
"Application" should {
"send 404 on a bad request" in new WithApplication {
route(FakeRequest(GET, "/boum")) must beNone
}
}
IntegrationSpec.scala
#RunWith(classOf[JUnitRunner])
class IntegrationSpec extends Specification {
"Application" should {
"work from within a browser" in {
running(TestServer(9000), FIREFOX) { browser =>
browser.goTo("http://localhost:9000/app")
browser.pageSource must contain("Your new application is ready.")
}
}
} section "integration"
}
The docs tells me I can use something like this from the command line:
play "test-only -- exclude integration"
The only problem is that this doesn't actually exclude any tests and my integration tests invoke firefox and start running. What am I doing wrong? How can I exclude the integration tests and then later run them by themselves?
sbt's Testing documentation describes several ways to define separate test configurations.
For instance, the Additional test configurations with shared sources example could be used in this situation:
In this approach, the sources are compiled together using the same classpath and are packaged together. However, different tests are run depending on the configuration.
In a default Play Framework 2.2.2 application created with play new, add a file named project/Build.scala with this content:
import sbt._
import Keys._
object B extends Build {
lazy val root =
Project("root", file("."))
.configs(FunTest)
.settings(inConfig(FunTest)(Defaults.testTasks) : _*)
.settings(
libraryDependencies += specs,
testOptions in Test := Seq(Tests.Filter(unitFilter)),
testOptions in FunTest := Seq(Tests.Filter(itFilter))
)
def itFilter(name: String): Boolean = (name endsWith "IntegrationSpec")
def unitFilter(name: String): Boolean = (name endsWith "Spec") && !itFilter(name)
lazy val FunTest = config("fun") extend(Test)
lazy val specs = "org.specs2" %% "specs2" % "2.0" % "test"
}
To run standard unit tests:
$ sbt test
Which produces:
[info] Loading project definition from /home/fernando/work/scratch/so23160453/project
[info] Set current project to so23160453 (in build file:/home/fernando/work/scratch/so23160453/)
[info] ApplicationSpec
[info] Application should
[info] + send 404 on a bad request
[info] + render the index page
[info] Total for specification ApplicationSpec
[info] Finished in 974 ms
[info] 2 examples, 0 failure, 0 error
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2
[success] Total time: 3 s, completed Apr 22, 2014 12:42:37 PM
To run tests for the added configuration (here, "fun"), prefix it with the configuration name:
$ sbt fun:test
Which produces:
[info] Loading project definition from /home/fernando/work/scratch/so23160453/project
[info] Set current project to so23160453 (in build file:/home/fernando/work/scratch/so23160453/)
[info] IntegrationSpec
[info] Application should
[info] + work from within a browser
[info] Total for specification IntegrationSpec
[info] Finished in 0 ms
[info] 1 example, 0 failure, 0 error
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 6 s, completed Apr 22, 2014 12:43:17 PM
You can also test only a particular class, like this:
$ sbt "fun:testOnly IntegrationSpec"
See this example in GitHub.