When I run tests in Junit, their ordering isn't guaranteed. By default, specs2 runs the examples in parallel so ordering isn't guaranteed here as well. However, if I add the sequential keyword, the tests are executed (at least from what it seems) in order. Is there a way to get the sequential behavior but have the examples run in random order?
You can use the latest specs2 2.3-SNAPSHOT version with the random command-line argument (or args.execute(random=true) inside the specification):
class TestSpec extends Specification { def is = s2"""
test1 $e1
test2 $e2
test3 $e3
"""
def e1 = { "starting e1".pp; Thread.sleep(30); "e1".pp; ok }
def e2 = { "starting e2".pp; Thread.sleep(20); "e2".pp; ok }
def e3 = { "starting e3".pp; Thread.sleep(40); "e3".pp; ok }
}
sbt> testOnly *TestSpec* -- random
starting e3
e3
starting e2
e2
starting e1
e1
[info] TestSpec
[info]
[info] + test1
[info] + test2
[info] + test3
[info]
How about decorating/wrapping the test cases with code that acquires a lock at the beginning of each test case? Hacky perhaps, but should work reliably and be easy to implement, until/unless you can find a more meant-to-be-used alternative.
Related
I have a case class that represents a phone number, and provides some helper methods that are used elsewhere in the code. One of these methods calls other methods within the same class:
case class PhoneNumber {
// class data and methods
def isPossiblyUsable(): Boolean = {
// both methods return a Boolean
this.isUsable() || this.usabilityUnknown()
}
}
I'd like to unit test this method, but I'm not sure what the best approach is. Coming from a Python background, I thought it'd be possible to simply mock PhoneNumber.isUsable and PhoneNumber.usabilityUnknown, but based off of some answers to this question it sounds like you cannot partially mock a class. I decided to try anways:
"PhoneNumber.isPossiblyUsable" should "return true when usable" in {
val m = mock[PhoneNumber]
(m.isUsable _).expects().returning(true)
(m.usabilityUnknown _).expects().returning(false)
assert(m.isPossiblyUsable() == true)
}
This fails with message:
[info] PhoneNumber.isPossiblyUsable
[info] - should return true when usable *** FAILED ***
[info] Unexpected call: <mock-1> PhoneNumber.isPossiblyUsable()
[info]
[info] Expected:
[info] inAnyOrder {
[info] <mock-1> PhoneNumber.isUsable() once (never called - UNSATISFIED)
[info] <mock-1> PhoneNumber.usabilityUnknown() once (never called - UNSATISFIED)
[info] }
[info]
[info] Actual:
[info] <mock-1> PhoneNumber.isPossiblyUsable() (Option.scala:201)
Sounds like isPossiblyUsable is not being called in the test. Is there a way to actually call it? Or is my approach here completely off?
You can "partially mock" a class (it's a called a "spy"):
val p = spy(PhoneNumber("5551212"))
when(p.isUsable).thenReturn(true)
when(p.usabilityUnknown).thenReturn(false)
p.possibleUseable shouldBe true
But you really should not do this. "Spying" is usually a sign of bad design, because it violates the single responsibility principle: the unit you are testing should only have one responsibility, so, you either mock it whole, or you test it in its entirety.
Additionally, mocking/spying case-classes is generally not a good idea too. Case-class is intended to be a simple data container, without external dependencies that normally need to be mocked.
In your case, if isUsable and usabilityUnknown are simple functions of the actual number, then you should not be mocking them at all, just supply the data that causes them to return the desired value.
For example:
def isUsable() = phone.length=10
def usabilityUnknown = phone.length=7
Then you can just do
PhoneNumber("9415551212").possiblyUsable shouldBe true
PhoneNumber("5551212").possiblyUsable shouldBe true
PhoneNumber("foo").possiblyUsable shouldBe false
without needing to mock anything at all.
Another possibility is that you have some external "blackbox" component that determines usability of the phone numbers.
In that case, those calls should probably not be a part of the PhoneNumber class to begin with. Instead, it should depend on that service, and then you can mock it. Like so:
case class PhoneNumber(phone: String, usability: PhoneUsabilityService = PhoneUsabilityService) {
def possiblyUsable = usability.isUsable(this) || usability.usabilityUnknown(this)
}
Now you can do things like
val u = mock[PhoneUsabilityService]
when(u.isUsable(any)).thenReturn(true)
PhoneNumber("foo", u).possiblyUsable() shouldBe true
Moreover, you can also make sure that conditions are checked in the correct order:
verify(u).isUsable(Phone("foo", u))
verifyNoMoreInteractions(u)
It may also make sense to supply the external component to just the method itself (perhaps, as an implicit parameter), rather than to the class. That seems a bit cleaner to me personally, as it provides a better indication of where this external service is used within the class:
case class PhoneService(phone: String) {
def possiblyUsable(implicit u: PhoneUsabilityService) =
u.isUsable(this) || u.usabilityUnknown(this)
}
PhoneNumber("foo").possiblyUsable(u) shouldBe true
It is easy to get all instances of TestDefinition in IntegrationTest:
val tests: Seq[TestDefinition] = (definedTests in IntegrationTest).value
But how do I get all instances of xsbti.api.Definition in IntegrationTest? It used to be possible to do this:
val defs: Seq[Definition] = {
val analysis = (compile in IntegrationTest).value
analysis.apis.internal.values.flatMap(_.source.api.definitions)
}
(for example, for filtering tests based on the suite annotations: say, #RequiresCassandra or #RequiresCluster and so on). But analysis.apis has been removed--not sure when, but it's absent in SBT 1.3.8. I haven't found any documentation on what to use instead.
One of SBT's chief strengths, compared to such XML-based tools like Maven, is the ability to define build settings programmatically. It should be possible to filter tests based on anything in the test code itself, not only on test names. I can't believe something so useful—something that really places SBT above competitors—could be just removed.
Any suggestions?
Tests.allDefs returns Definitions given CompileAnalysis. For example,
val analysis: CompileAnalysis = (compile in IntegrationTest).value
val definitions: Seq[Definition] = Tests.allDefs(analysis)
val annotations: Seq[Annotation] = definitions.flatMap(_.annotations())
There was a change back in 2016 from
val compile = TaskKey[Analysis]("compile", "Compiles sources.", APlusTask)
to
val compile = TaskKey[CompileAnalysis]("compile", "Compiles sources.", APlusTask)
where
trait Analysis extends CompileAnalysis {
...
val apis: APIs
...
}
so now we have to cast to Analysis
analysis match { case analysis: Analysis => ... }
which is what allDefs does.
I noticed that the type checker works by phases. Sometimes scalac returns only a few errors, which makes you think that are almost there, but once you fix them all – boom – next phases, you suddenly get a lot of errors that where not there before.
What are the different phases of the type checker?
Is there a way to know in which phase the type checker gave up on my code (other than recognizing the errors)?
As #Felix points out, this answer lists the compile phases:
$ scalac -version
Scala compiler version 2.11.6 -- Copyright 2002-2013, LAMP/EPFL
$ scalac -Xshow-phases
phase name id description
---------- -- -----------
parser 1 parse source into ASTs, perform simple desugaring
namer 2 resolve names, attach symbols to named trees
packageobjects 3 load package objects
typer 4 the meat and potatoes: type the trees
patmat 5 translate match expressions
superaccessors 6 add super accessors in traits and nested classes
extmethods 7 add extension methods for inline classes
pickler 8 serialize symbol tables
refchecks 9 reference/override checking, translate nested objects
uncurry 10 uncurry, translate function values to anonymous classes
tailcalls 11 replace tail calls by jumps
specialize 12 #specialized-driven class and method specialization
explicitouter 13 this refs to outer pointers
erasure 14 erase types, add interfaces for traits
posterasure 15 clean up erased inline classes
lazyvals 16 allocate bitmaps, translate lazy vals into lazified defs
lambdalift 17 move nested functions to top level
constructors 18 move field definitions into constructors
flatten 19 eliminate inner classes
mixin 20 mixin composition
cleanup 21 platform-specific cleanups, generate reflective calls
delambdafy 22 remove lambdas
icode 23 generate portable intermediate code
jvm 24 generate JVM bytecode
terminal 25 the last phase during a compilation run
Is there a way to know in which phase the type checker gave up on my code (other than recognizing the errors)?
If you add the -verbose flag to scalac, it will print the name of each phase and how long it took after that phase completes. You can then infer which phase failed
I don't think that scalac exposes different phases of typing, only the compile phases. The typer is listed as a single compile phase.
The compiler option for this is -Yissue-debug. It outputs a stack trace in 2.10 when an error is issued.
The code supporting it was removed in 2.11 during the refactor of reporting, but the option is still valid. (I restored it at some point because, in fact, it's the quickest way to see what is emitting an error; but apparently that PR died on the vine and disappeared. Probably a victim of push -f.)
In 2.12, you can supply a custom reporter that does just about the same thing. They claim that they will augment the reporter API with access to context, so you could perhaps directly query the current phase, inspect trees, and so on.
Here's an example of the situation you describe:
class C { def f: Int ; def g: Int = "" ; def h = "\000" }
There are three errors, but only one is reported at a time because they are emitted in different compiler phases.
To clarify the question, various phases besides "typer" can create and typecheck trees, and additionally can enforce well-typedness even after trees are typed. (There are also other kinds of errors, such as "unable to write the output file.")
For C, the parser emits an error (under -Xfuture) for the deprecated octal syntax, the typer reports the type mismatch in g, and the grab bag refchecks phase checks the declared but undefined (empty) member f. You would normally wind up fixing one error at a time. If the parser error is emitted as a warning, then the warning would be suppressed until the errors were fixed, so that would be the last to pop up instead of the first.
Here is a sample reporter that tries to do more than output huge stack traces.
package myrep
import scala.tools.nsc.Settings
import scala.tools.nsc.reporters.ConsoleReporter
import scala.reflect.internal.util._
class DebugReporter(ss: Settings) extends ConsoleReporter(ss) {
override def warning(pos: Position, msg: String) = debug {
super.warning(pos, msg)
}
override def error(pos: Position, msg: String) = debug {
super.error(pos, msg)
}
// let it ride
override def hasErrors = false
private def debug(body: => Unit): Unit = {
val pkgs = Set("nsc.ast.parser", "nsc.typechecker", "nsc.transform")
def compilerPackages(e: StackTraceElement): Boolean = pkgs exists (e.getClassName contains _)
def classname(e: StackTraceElement): String = (e.getClassName split """\.""").last
if (ss.Yissuedebug) echo {
((new Throwable).getStackTrace filter compilerPackages map classname).distinct mkString ("Issued from: ", ",", "\n")
}
body
}
}
It lies about having no errors so that the compiler won't abort early.
It would be invoked this way, with the reporter class on the "tool class path":
scalacm -toolcp repdir -Xreporter myrep.DebugReporter -Yissue-debug -deprecation errs.scala
where
$ scalacm -version
Scala compiler version 2.12.0-M2 -- Copyright 2002-2013, LAMP/EPFL
Sample output:
Issued from: Scanners$UnitScanner,Scanners$Scanner,Parsers$Parser,Parsers$Parser$$anonfun$templateStat$1,Parsers$Parser$$anonfun$topStat$1,Parsers$SourceFileParser,Parsers$UnitParser,SyntaxAnalyzer,SyntaxAnalyzer$ParserPhase
errs.scala:4: warning: Octal escape literals are deprecated, use \u0000 instead.
class C { def f: Int ; def g: Int = "" ; def h = "\000" }
^
Issued from: Contexts$ImmediateReporter,Contexts$ContextReporter,Contexts$Context,ContextErrors$ErrorUtils$,ContextErrors$TyperContextErrors$TyperErrorGen$,Typers$Typer,Analyzer$typerFactory$$anon$3
errs.scala:4: error: type mismatch;
found : String("")
required: Int
class C { def f: Int ; def g: Int = "" ; def h = "\000" }
^
Issued from: RefChecks$RefCheckTransformer,Transform$Phase
errs.scala:4: error: class C needs to be abstract, since method f is not defined
class C { def f: Int ; def g: Int = "" ; def h = "\000" }
^
one warning found
two errors found
I am trying to do property-based testing using ScalaTest. I have written a few test cases using 2 custom generators and they work fine. However, the moment I define a forAll with one custom generator, such as below:
it should "generate a given number of weights whose total is less than the given total when the given number is > 0, with default upperBound, lowerBound and total" in {
// TODO: Find out why single parameter does not work. maxSize here is a workaround
forAll(reasonableSizes) { (size: Int) =>
whenever(size > 0) {
val range = random.nextLong.abs
val values = DataGenerator.generatePartialWeights(size)
values should have length size
every(values) should (be >= BigDecimal(0) and be <= BigDecimal(1))
values.sum should be < BigDecimal(1)
}
}
}
I got compilation error in Eclipse as follows:
type mismatch; found : (org.scalacheck.Gen[A],
DataGeneratorTest.this.PropertyCheckConfigParam*) required: ?0C[?0E]
Note that implicit conversions are not applicable because they are
ambiguous: both method ArrowAssoc in object Predef of type [A](self:
A)ArrowAssoc[A] and method Ensuring in object Predef of type
[A](self: A)Ensuring[A] are possible conversion functions from
(org.scalacheck.Gen[A],
DataGeneratorTest.this.PropertyCheckConfigParam*) to ?0C[?0E]
I tried the example in ScalaTest documentaiton: http://www.scalatest.org/user_guide/generator_driven_property_checks
by doing
it should "testing" in {
val evenInts = for (n <- Gen.choose(-1000, 1000)) yield 2 * n
forAll(evenInts) { (n) => n % 2 should equal(0) }
}
and got the same error.
However, when I compile it in SBT, there is no error.
sbt compile
Java HotSpot(TM) 64-Bit Server
[info] Loading project definition from C:\xxx [info] Set current project to cree (in build file:/C:/xxx)
[info] Compiling 20 Scala sources to
C:\xxx\target\scala-2.11\classes...
[success] Total time: 37 s, completed 26-Mar-2015 20:04:15
I am not sure what is wrong. Can someone help? Thanks.
Environment:
OS: Windows 7 Enterprise SP1 64-bit
Scala IDE: 4.0.0
Scala Version: 2.11.6
ScalaTest Version: 2.2.4
ScalaCheck Version: 1.12.2
Can you perhaps post the full test class?
I ran into the exact same issue and found that there was a conflict with using the Inspectors trait along side the PropertyChecks trait. After removing the Inspectors trait from my test class, everything worked fine.
abstract class UnitSpec extends FunSpec
with Matchers
with OptionValues
with Inside
// with Inspectors <= this conflicts with PropertyChecks trait
with PropertyChecks
with ValidationMatchers
with MockFactory
My Test.scala has 2 tests that I'd like to run in parallel.
C:/users/Kevin/workspace/playApp> sbt test
class ConcurrentRequests extends FunSuite with BeforeAndAfterAll with ParallelTestExecution {
val start = System.nanoTime()
test("foo") {
println("foo")
foo
println("foo in " + (System.nanoTime - start) / 1000000000 + " seconds.")
}
test("bar") {
println("bar")
bar
println("bar in " + (System.nanoTime - start) / 1000000000 + " seconds.")
}
}
I tried this post's answers, namely adding testOptions to my $PLAY_APP/build.sbt, as well as using the -P option, but neither worked. I say it didn't work since "foo" printed out, executed its calls, and then bar printed out and executed after.
How can I run this Play test in parallel via sbt?
EDIT
As a work-around, I put the first test in ConcurrentTest.scala and the second in a separate ConcurrentTest2.scala file. And then ScalaTest ran the tests in parallel. From a maintenance point-of-view, I'd rather have a single test file and then run each of them in parallel.