ScalaTest: no tests are run for GeneratorDrivenPropertyChecks - scala

I am trying to set-up a property based testing with ScalaTest and ScalaCheck ... and based on the ouput it seems that I am succeeding, but it takes too fast and from what I understand normally ScalaCheck should inform you about how may tests were run, in my case this information is absent:
[IJ]sbt:algorithms2_1> testOnly *MedianOf3PartitioningProps
[info] Compiling 1 Scala source to /Users/vasile.gorcinschi/gitPerso/Algorithms/Chapter 2 Sorting/algorithms2_1/target/scala-2.12/test-classes ...
[warn] there was one deprecation warning; re-run with -deprecation for details
[warn] one warning found
[info] Done compiling.
[info] MedianOf3PartitioningProps:
[info] sort
[info] - should sort array of ints from 0 to 100
[info] +
[info] ScalaTest
[info] Run completed in 412 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
Here is the test class:
class MedianOf3PartitioningProps extends FlatSpec with Matchers with GeneratorDrivenPropertyChecks with Gens {
private val medianOf3Partitioning = new MedianOf3Partitioning[Int]
implicit override val generatorDrivenConfig: PropertyCheckConfiguration = PropertyCheckConfig(minSuccessful = 1, maxDiscarded = 500, workers = 1)
behavior of "sort"
it should "sort array of ints from 0 to 100" in {
forAll(arraysGen){ a: Array[Int] =>
info(s"${a.mkString(",")}")
medianOf3Partitioning.sort(a) shouldEqual a.sorted }
}
}
The Gens trait is mine - it only comprises definition for the Gen[Array[Int]]:
trait Gens {
val arraysGen: Gen[Array[Int]] = containerOf[Array, Int](
chooseNum(Int.MinValue, Int.MaxValue) suchThat { _ < 100 }
).suchThat(_.length < 50)
}
I used this source for the test set-up. Just in case, I'm providing versions of scalacheck and scalatest (from Dependencies.scala and build.sbt):
lazy val scalaTest = "org.scalatest" %% "scalatest" % "3.0.5"
...
libraryDependencies ++= {
val scalaTestVersion = "3.0.5"
val scalaCheckVersion = "1.14.0"
Seq(scalaTest % Test,
"org.scalatest" %% "scalatest" % scalaTestVersion % "test",
"org.scalacheck" %% "scalacheck" % scalaCheckVersion % "test",
"com.storm-enroute" %% "scalameter" % "0.9"
)
}

Based on the small example from M. Odersky's "Programming in Scala", I switched from GeneratorDrivenPropertyChecks to more general PropertyChecks. I've also discovered issues with my Gen[Array[Int]] so I had to tweek that too. Posting a solution that worked (discovered cases that fail) in case this will help anyone else:
Gens trait:
trait Gens {
val minIntArraysGen: Gen[Array[Int]] = containerOf[Array, Int](Gen.chooseNum(0, 100))
}
The property based test:
import ca.vgorcinschi.Gens
import org.scalatest.MustMatchers._
import org.scalatest.WordSpec
import org.scalatest.prop.PropertyChecks
class MedianOf3PartitioningProps extends WordSpec with PropertyChecks with Gens {
"sort method" must {
"sort any Int array" in {
forAll (minIntArraysGen){ (a: Array[Int]) =>
whenever(a.nonEmpty) {
val maybeSorted = new MedianOf3Partitioning[Int].sort(a)
maybeSorted must equal (a.sorted)
}
}
}
}
}

Related

How do I import a package defined in src/test directory from inside src/it [duplicate]

I would like to share a helper trait between my "test" and "it" configurations in SBT, but I have not figured out how.
Here is a minimal example:
project/Build.scala
import sbt._
import Keys._
object MyBuild extends Build {
val scalaTest = "org.scalatest" %% "scalatest" % "2.0" % "test,it"
lazy val myProject =
Project(id = "my-project", base = file("."))
.configs(IntegrationTest)
.settings(Defaults.itSettings: _*)
.settings(
scalaVersion := "2.10.3",
libraryDependencies ++= Seq(
scalaTest
)
)
}
src/test/scala/Helpers.scala
trait Helper {
def help() { println("helping.") }
}
src/test/scala/TestSuite.scala
import org.scalatest._
class TestSuite extends FlatSpec with Matchers with Helper {
"My code" should "work" in {
help()
true should be(true)
}
}
src/it/scala/ItSuite.scala
import org.scalatest._
class ItSuite extends FlatSpec with Matchers with Helper {
"My code" should "work" in {
help()
true should be(true)
}
}
then, in sbt, "test" works:
sbt> test
helping.
[info] TestSuite:
[info] My code
[info] - should work
[info] Run completed in 223 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 0 s, completed Dec 17, 2013 1:54:56 AM
but "it:test" doesn't compile:
sbt> it:test
[info] Compiling 1 Scala source to ./target/scala-2.10/it-classes...
[error] ./src/it/scala/ItSuite.scala:3: not found: type Helper
[error] class ItSuite extends FlatSpec with Matchers with Helper {
[error] ^
[error] ./src/it/scala/ItSuite.scala:5: not found: value help
[error] help()
[error] ^
[error] two errors found
[error] (it:compile) Compilation failed
[error] Total time: 1 s, completed Dec 17, 2013 1:55:00 AM
If you want to share code from Test configuration, it's probably better to create a custom test configuration from Test. See Custom test configuration.
Your project/Build.scala becomes:
import sbt._
import Keys._
object MyBuild extends Build {
lazy val FunTest = config("fun") extend(Test)
val scalaTest = "org.scalatest" %% "scalatest" % "2.0" % "test"
lazy val myProject =
Project(id = "my-project", base = file("."))
.configs(FunTest)
.settings(inConfig(FunTest)(Defaults.testSettings) : _*)
.settings(
scalaVersion := "2.10.3",
libraryDependencies ++= Seq(
scalaTest
)
)
}
Also rename src/it/ to src/fun/. Now fun:test works:
> fun:test
helping.
[info] ItSuite:
[info] My code
[info] - should work
[info] Run completed in 245 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 1 s, completed Dec 17, 2013 8:43:17 AM
You can redefine the IntegrationTest Configuration in your project to extend the Test configuration instead of the Runtime Configuration (the default). This will make everything in your test configuration available to your IntegrationTest configuration.
import sbt._
import Keys._
object MyBuild extends Build {
val scalaTest = "org.scalatest" %% "scalatest" % "2.0" % "test,it"
lazy val IntegrationTest = config("it") extend(Test)
lazy val myProject =
Project(id = "my-project", base = file("."))
.configs(IntegrationTest)
.settings(Defaults.itSettings: _*)
.settings(
scalaVersion := "2.10.3",
libraryDependencies ++= Seq(
scalaTest
)
)
}

Scala mock, MockFactory expects fails with 'Unexpected call'

I'm trying to follow this doc - scalatest with scala-mock to mock the function and check if it has been called
class AggregateSpec extends FlatSpec with Matchers with MockFactory {
val data = Seq("This", "is", "something", "I", "would", "like", "to", "know")
"combop function" should "BE called for par collection" in {
val mockCombop = mockFunction[Int, Int, Int]
val parData = data.par
val result: Int = parData.aggregate(0)(
seqop = (acc, next) => acc + next.length,
combop = mockCombop
)
result should === (31)
mockCombop.expects(*, *).atLeastOnce()
}
}
As result:
> [info] - should BE called for non-par collection *** FAILED *** [info]
> Unexpected call: MockFunction2-1(4, 2) [info] [info] Expected:
> [info] inAnyOrder { [info] [info] } [info] [info] Actual:
> [info] MockFunction2-1(9, 1) [info] MockFunction2-1(2, 4)
> [info] MockFunction2-1(4, 2) [info] MockFunction2-1(5, 4)
> (Option.scala:121)
Why? How to make it pass with scalatest + scala-mock ?
--
As deps I use:
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.1",
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.1" % "test",
libraryDependencies += "org.scalamock" %% "scalamock-scalatest-support" % "3.5.0"
You need to call mockCombop.expects before mockCombop gets called, not after:
"combop function" should "BE called for par collection" in {
val mockCombop = mockFunction[Int, Int, Int]
val parData = data.par
mockCombop.expects(*, *).atLeastOnce()
val result: Int = parData.aggregate(0)(
seqop = (acc, next) => acc + next.length,
combop = mockCombop
)
result should === (31)
}

Running setup/teardown tasks during stb test task

I try to configure sbt to run a setup task at the beginning and a tear down task at the end of myProject/test command.
My build.sbt is here:
name := "ch-2"
version := "1.0"
libraryDependencies += "org.specs2" % "specs2_2.10" % "1.14" % "test"
lazy val common = (
Project("common", file("common")).
settings()
)
lazy val subPro = (
Project("sub", file("subA")).settings(
).dependsOn(common)
settings(libraryDependencies += "org.specs2" % "specs2_2.10" % "1.14" % "test" )
)
val startS = taskKey[Unit]("Start")
val stopS = taskKey[Unit]("Stop")
startS := { println("Running start")}
stopS := { println("Running stop")}
testOptions in Test in subPro += Tests.Setup { () => startS.value }
testOptions in Test in subPro += Tests.Cleanup { () => stopS.value }
The actual dummy test class is here:
import org.specs2.mutable.Specification
/**
* Created by jk on 26.3.2017.
*/
object FooSpec extends Specification {
"The TEST method" should {
"blaa blaa 1" in {
println("test 1 running...")
true
}
"blaa blaa 2" in {
println("test 2 running...")
true
}
}
}
When I run the tests for project sub, I get following output:
> sub/test
Running stop
Running start
[info] Updating {file:/home/jk/workspace/sbt-in-action/ch2/}sub...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 1 Scala source to /home/jk/workspace/sbt-in-action/ch2/subA/target/scala-2.10/test-classes...
test 1 running...
test 2 running...
[info] FooSpec
[info]
[info] The TEST method should
[info] + blaa blaa 1
[info] + blaa blaa 2
[info]
[info]
[info] Total for specification FooSpec
[info] Finished in 18 ms
[info] 2 examples, 0 failure, 0 error
[info]
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2
[success] Total time: 3 s, completed Mar 26, 2017 7:09:34 PM
Why is stop task run even before compilation is finished and how to fix it so that it is run after all test cases are run (despite the result of the test cases)?
Also the start task should run after successfull compilation but before first test case. How to fix these?
How about doing it like this instead?
def startS(): Unit = { println("Running start")}
def stopS(): Unit = { println("Running stop")}
testOptions in Test in subPro += Tests.Setup { () => startS() }
testOptions in Test in subPro += Tests.Cleanup { () => stopS() }

ScalaCheck specificy minimum successful tests for property

I'm trying to make sure that my ScalaCheck property runs 500 times instead of the default 100 times. I'm having trouble configuring this though.
class BlockSpec extends Properties("BlockSpec") with BitcoinSLogger {
val myParams = Parameters.default.withMinSuccessfulTests(500)
override def overrideParameters(p: Test.Parameters) = myParams
property("Serialization symmetry") =
Prop.forAll(BlockchainElementsGenerator.block) { block =>
logger.warn("Hex:" + block.hex)
Block(block.hex) == block
}
}
However when I actually run this test it only says 100 tests passed successfully
EDIT:
$ sbt
[info] Loading project definition from /home/chris/dev/bitcoins-core/project
[info] Set current project to bitcoin-s-core (in build file:/home/chris/dev/bitcoins-core/)
> test-only *BlockSpec*
[info] + BlockSpec.Serialization symmetry: OK, passed 100 tests.
[info] Elapsed time: 1 min 59.775 sec
[info] ScalaCheck
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[info] ScalaTest
[info] Run completed in 2 minutes.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 123 s, completed Aug 1, 2016 11:36:17 AM
>
How do I actually pass this to my property?
As far as I understand you can specify the test parameters at two levels and they don't seem to communicate.
The first option is within the property as you're trying to do:
import org.scalacheck.Properties
import org.scalacheck.Test.{ TestCallback, Parameters }
import org.scalacheck.Prop.{ forAll, BooleanOperators }
import org.scalacheck.Test
class TestFoo extends Properties("BlockSpec") {
override def overrideParameters(p: Parameters) =
p.withMinSuccessfulTests(1000000)
property("Serialization symmetry") = forAll { n: Int =>
(n > 0) ==> (math.abs(n) == n)
}
}
This will have no impact as long as you don't call .check on the property.
Can be from the sbt shell or directly within the class.
Now if you want to impact the number of tests run when calling the sbt:test target, it seems you have to play with options build.sbt (taken from here):
name := "scalacheck-demo"
scalaVersion := "2.11.5"
libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.12.2" % "test"
testOptions in Test += Tests.Argument(TestFrameworks.ScalaCheck, "-maxSize", "5", "-minSuccessfulTests", "33", "-workers", "1", "-verbosity", "1")
There's definitely an easier way to achieve that than overriding any kind of global test config:
class SampleTest extends FlatSpec
with Matchers with GeneratorDrivenPropertyChecks {
it should "work for a basic scenario" in {
// This will require 500 successful tests to succeed
forAll(minSuccessful(500)) { (d: String) =>
whenever (d.nonEmpty) {
d.length shouldBe > 0
}
}
}
}

specs2 skip test based on condition without an exception

Is there a way to skip a test in specs2 without the SkippedException occurring? I'd like the test to be ignored under a certain condition so that it doesn't show as error but as ignored.
For example: I have the following in the test/scala/Skipped.scala file
package models
import org.specs2.mutable._
import org.specs2.runner._
class Skipped extends Specification{
"Application" should {
if( 3 < 2 ){
"do better with regex" in {
"axbcd" must find( "bc".r )
}
}
else skipped( "blah" )
}
}
and the following the my build.sbt
scalaVersion := "2.10.3"
libraryDependencies += "org.specs2" % "specs2_2.10" % "2.1.1" % "test"
When I run the test, I get the following:
org.specs2.execute.SkipException: blah
at org.specs2.matcher.ThrownExpectations$class.skipped(ThrownExpectations.scala:87)
Am I missing something?
Bonus question:
Why am I using such an old version? When I have the following in my build.sbt, I cannot compile the above code snippet.
scalaVersion := "2.11.1"
libraryDependencies += "org.specs2" %% "specs2" % "2.3.12" % "test"
The error is:
Skipped.scala:7: overloaded method value should with alternatives:
[error] (fs: => Unit)(implicit p1: Skipped.this.ImplicitParam1, implicit p2: Skipped.this.ImplicitParam2)org.specs2.specification.Fragments <and>
[error] (fs: => org.specs2.specification.Fragments)(implicit p: Skipped.this.ImplicitParam)org.specs2.specification.Fragments <and>
[error] (fs: => org.specs2.specification.Fragment)org.specs2.specification.Fragments
[error] cannot be applied to (Product with Serializable)
[error] "Application" should {
[error] ^
[error] one error found
So I'd appreciate if someone could help me with a way to skip a test without an exception, which preferably works in with the newer versions of specs2
This should work:
class Skipped extends Specification{
"Application" should {
if( 3 < 2 ){
"do better with regex" in {
"axbcd" must find( "bc".r )
}
}
else "3 is >= 2" in skipped( "blah" )
}
}
The difference is that skipped is declared in the body of an example which allows it to be reported to the console:
[info] o 3 is >= 2
[info] SKIPPED blah
(or something like that, I didn't run the code)