Scalatest Suites do not have detailed test status output - scala

When I run the following simple test using sbt I get the output I would expect:
import org.scalatest.{FlatSpec, Matchers, Suites}
class TestSimple extends FlatSpec with Matchers {
"a" should "do" in {
Array(1,3) should equal (Array(1,2))
}
}
Output:
[info] TestSimple:
[info] a
[info] - should do *** FAILED ***
[info] Array(1, 3) did not equal Array(1, 2) (SimpleTest.scala:5)
[info] ScalaTest
[info] Run completed in 980 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed: Total 1, Failed 1, Errors 0, Passed 0
[error] Failed tests:
[error] TestSimple
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
When the Test is included in a Suite and annotated with DoNotDiscover like so:
import org.scalatest.{DoNotDiscover, FlatSpec, Matchers, Suites}
class FullTestSuite extends Suites(new TestSimple)
#DoNotDiscover
class TestSimple extends FlatSpec with Matchers {
"a" should "do" in {
Array(1,3) should equal (Array(1,2))
}
}
then the output does not include the per test success and failures but instead has just the overall results:
[info] ScalaTest
[info] Run completed in 975 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed: Total 1, Failed 1, Errors 0, Passed 0
[error] Failed tests:
[error] FullTestSuite
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
How can I get run tests inside a Suites instance to output where and how they have failed?
Thanks

I guess you are facing a bug #916. You should also try version >=3.0.0-M15 and provide your feedback to developers.

Related

sbt test fails due to "Forked test harness failed"

I've got a Scala project which I'm building with sbt. When running sbt test, the tests themselves pass, but then the command fails with "Forked test harness failed: java.io.EOFException".
The build.sbt file does not specify fork in Test.
Example of the error it fails with after running sbt test:
[info] Run completed in 5 seconds, 494 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[error] Error during tests:
[error] Forked test harness failed: java.io.EOFException
[error] at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2959)
[error] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1539)
[error] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
[error] at sbt.React.react(ForkTests.scala:177)
[error] at sbt.ForkTests$Acceptor$1$.run(ForkTests.scala:108)
[error] at java.lang.Thread.run(Thread.java:748)
[error] (serverTests / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 14 s, completed Mar 12, 2020 4:35:26 PM
Minimal example of test which fails:
package com.example
import akka.http.scaladsl.testkit.ScalatestRouteTest
import org.scalatest.FreeSpecLike
class ForkedTestHarnessFailedForNoReasonSpec extends FreeSpecLike with ScalatestRouteTest {
"This test" - {
"should not fail" in {
assert("Foo" == "Foo")
}
}
}
What does this error indicate and how should one resolve it?
The cause in my case was that AKKA is shutting down JVM on coordinated shutdown. Put this to your test config (src/test/resources/reference.conf in my case):
akka.coordinated-shutdown.exit-jvm = off

Scala test with sbt cross compile platforms (Native,JVM,JS)

I have sbt cross project and I'm trying to run test and testOnlycommands to run scala tests,currently I have jvm test and native one , always Jvm test running successfully and native fails I got this exception any idea ?
[IJ]> test
[info] Processing resources
[info] Linking (5527 ms)
[info] Discovered 2325 classes and 15319 methods
[info] JvmTest:
[info] name
[info] - should the name is set correctly in constructor
[info] JvmTest2:
[info] age
[info] - should be 28
[info] Run completed in 11 seconds, 488 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Optimizing (debug mode) (8718 ms)
[info] Generating intermediate code (1943 ms)
[info] Produced 68 files
[info] Compiling to native code (9007 ms)
[info] Linking native code (boehm gc) (538 ms)
[info] Starting process '/home/naseem/Documents/SFreely NEW/CrossCompilePlatforms3June/myProject/nativeUbuntu18/target/scala-2.11/myprojectnativeubuntu18-out' on port '30651'.
[error] Uncaught exception when running tests: java.io.InvalidClassException: sbt.testing.TaskDef; local class incompatible: stream classdesc serialVersionUID = -7417691495999416204, local class serialVersionUID = 2663134200025980977
[info] Run completed in 531 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
[error] Error during tests:
[error] Forked test harness failed: java.net.SocketException: Connection reset
[error] at java.net.SocketInputStream.read(SocketInputStream.java:210)
[error] at java.net.SocketInputStream.read(SocketInputStream.java:141)
[error] at java.net.SocketInputStream.read(SocketInputStream.java:224)
[error] at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2631)
[error] at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2825)
[error] at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2895)
[error] at java.io.ObjectInputStream$BlockDataInputStream.skipBlockData(ObjectInputStream.java:2797)
[error] at java.io.ObjectInputStream.skipCustomData(ObjectInputStream.java:2229)
[error] at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1871)
[error] at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1745)
[error] at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2033)
[error] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
[error] at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2278)
[error] at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:557)
[error] at java.lang.Throwable.readObject(Throwable.java:914)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.lang.reflect.Method.invoke(Method.java:498)
[error] at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
[error] at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
[error] at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
[error] at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
[error] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:427)
[error] at sbt.React.react(ForkTests.scala:122)
[error] at sbt.ForkTests$$anonfun$mainTestTask$1$Acceptor$2$.run(ForkTests.scala:76)
[error] at java.lang.Thread.run(Thread.java:748)
[error] (myProjectNativeUbuntu18/nativetest:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 35 s, completed Jul 10, 2019 11:57:19 AM

Property check should fail in Scalatest

I've created and am also using some external Scalacheck generators from Lists and appropriate types, using Gen.oneOf(List[T]) . I think it would be occasionally useful to return a placeholder for an empty value. Currently the lists are populated. How should I go about this? Do I try to append an empty type to the end of the list? If so how do I go about that? If not, how else can I get my generators to add an empty value. It seems straightforward, but I am having trouble figuring it out right now.
import org.scalatest.FlatSpec
import org.scalacheck.Gen
import org.scalacheck.Prop.exists
import org.scalatest.prop.PropertyChecks
class EventFieldGeneratorTest extends FlatSpec with PropertyChecks {
behavior of "Gen.option"
it should "occasionally return None" in {
val colors = Gen.oneOf("Blue", "Red", "Green", "Yellow")
val opt = Gen.option(colors)
val list = Gen.listOfN(20, opt)
val p1 = exists(list)(_ == None)
p1.check
}
}
Can anybody explain why my test is giving up?
Testing started at 10:31 AM ... ! Gave up after only 0 passed tests. 501 tests were discarded.
Process finished with exit code 0
How can I mark that as a failed result for ScalaTest? Is it a bad idea for me to use Flatspec?
Maybe I should be using something other than check...
Here's the documentation I used to sort it on. On the Scalatest page:
http://www.scalatest.org/user_guide/writing_scalacheck_style_properties
I don't think there's anything flawed with trying to use lists with optional values. There are just a few issues you're running in to that are giving you trouble.
It's confusing, but if you're using the Scalatest framework, you need to use the Scalatest infrastructure to use Scalacheck. So you'll need to use Scalatest matchers, and write Scalatest-flavored properties (using it's forAll), but you'll still use Scalacheck's generators directly.
For some reason the type inference between lists and Option type is giving you trouble. If you use the shouldBe matcher,
x shouldBe(None)
you'll get a relevant runtime error from Scalatest:
[info] - should occasionally return None *** FAILED ***
[info] TestFailedException was thrown during property evaluation.
[info] Message: List() was not equal to None
[info] Location: (GenTest.scala:13)
[info] Occurred when passed generated values (
[info] arg0 = List() // 5 shrinks
[info] )
[info] Run completed in 1 second, 621 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
You shouldn't be matching a list with an Option type. You need to be matching with the Scalatest "container" matcher should contain
import org.scalatest.FlatSpec
import org.scalatest.Matchers
import org.scalacheck.Gen
import org.scalatest.prop.PropertyChecks
class EventFieldGeneratorTest extends FlatSpec with Matchers with PropertyChecks {
behavior of "Gen.option"
it should "occasionally return None" in {
val colors = Gen.oneOf("Blue","Red","Green","Yellow")
val opt = Gen.option(colors)
val list = Gen.listOfN(20,opt)
forAll(list) { (xs: List[Option[String]]) =>
xs should contain (None)
}
}
}
This gives you a successful property check:
[info] EventFieldGeneratorTest:
[info] Gen.option
[info] - should occasionally return None
[info] ScalaTest
[info] Run completed in 1 second, 9 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
More on Scalatest matchers
http://www.scalatest.org/user_guide/using_matchers

Scalatest matching options

I have this simple test:
test("transform /home into Array(/home)") {
val path = "/home"
val expected: Option[Array[String]] = Some(Array("/home"))
val actual: Option[Array[String]] = luceneService.buildCategoryTree(path)
actual shouldEqual expected
}
And I get this failure:
Some(Array("/home")) did not equal Some(Array("/home"))
How can this be?
As I understand it the docs state that I should be able to use options in tests
If I change the test to
actual.get shouldEqual expected.get
it passes
In the sclatest docs
there is a section that says:
You can work with options using ScalaTest's equality, empty, defined,
and contain syntax. For example, if you wish to check whether an
option is None, you can write any of:
so with your test (sorry I do not have de lucerne object), I also think that is some thing wrong when using arrays
Unfortunately, the current implementation is not able to "properly"
understand Array equality if arrays are within another container such
as Set[Array[Int]]. For example, I would have expected the following
test to pass instead of throwing a TestFailedException:
import org.scalatest._
class SetSuite extends FunSuite with Matchers {
test("transform /home into Array(/home)") {
val path = "/home"
val expected: Option[Array[String]] = Some(Array("/home"))
val actual: Option[Array[String]] = Some(Array(path))
actual shouldEqual expected
}
}
[info] SetSuite:
[info] - transform /home into Array(/home) *** FAILED ***
[info] Some(Array("/home")) did not equal Some(Array("/home")) (TestScalaTest.scala:9)
[info] Run completed in 335 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed tests:
[error] SetSuite
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 11 s, completed Mar 17, 2016 12:26:25 AM
so for testing let's use
import org.scalatest._
class SetSuite extends FunSuite with Matchers {
test("transform /home into Array(/home)") {
val path = "/home"
val expected: Option[Array[String]] = Some(Array("/home"))
val actual: Option[Array[String]] = Some(Array(path))
actual should contain (Array("/home"))
}
}
[info] SetSuite:
[info] - transform /home into Array(/home)
[info] Run completed in 201 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 2 s, completed Mar 17, 2016 12:33:01 AM
Looks like there is a bug with the matchers.
Using Seq instead of Array works:
val expected = Some(Seq("/home"))
val actual = luceneService.buildCategoryTree(path).map(_.toSeq)
actual shouldEqual expected
This is scalatest's issue with arrays. Same test works just fine if you convert arrays to say vectors or lists

Spray-Json java.lang.OutOfMemoryError when calling parseJson

I'm using spray-json 1.3.0. I have a simple piece of Json that I am asking spray to parse, here it is
import org.scalatest.{FlatSpec, MustMatchers}
import spray.json._
class BlockCypherOutputMarshallerTest extends FlatSpec with MustMatchers {
val expectedOutput = """{"value":7454642,
|"script":"76a9148d5968ad26f9e277849ff9f8f39920f28944467388ac",
|"addresses":["mtQLgLiqmytKkgE9sVGwypAFsLvkxBQ6XX"],
|"script_type":"pay-to-pubkey-hash}""".stripMargin
val json = expectedOutput.parseJson
"BlockCypherOutputMarshaller" must "parse an output from the blockcypher api" in {
//test case
}
}
however I am getting an error message on the line that val json = expectedOutput.parseJson is called. Here is the error message
> last test:testOnly
[debug] Running TaskDef(com.blockcypher.api.marshallers.BlockCypherOutputMarshallerTest, org.scalatest.tools.Framework$$anon$1#4178c07b, false, [SuiteSelector])
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:622)
at java.lang.StringBuilder.append(StringBuilder.java:202)
at spray.json.JsonParser.appendSB(JsonParser.scala:179)
at spray.json.JsonParser.char(JsonParser.scala:138)
at spray.json.JsonParser.string(JsonParser.scala:129)
at spray.json.JsonParser.value(JsonParser.scala:62)
at spray.json.JsonParser.members$1(JsonParser.scala:80)
at spray.json.JsonParser.object(JsonParser.scala:84)
at spray.json.JsonParser.value(JsonParser.scala:59)
at spray.json.JsonParser.parseJsValue(JsonParser.scala:43)
at spray.json.JsonParser$.apply(JsonParser.scala:28)
at spray.json.PimpedString.parseJson(package.scala:45)
at com.blockcypher.api.marshallers.BlockCypherOutputMarshallerTest.<init>(BlockCypherOutputMarshallerTest.scala:16)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.lang.Class.newInstance(Class.java:442)
at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:641)
at sbt.TestRunner.runTest$1(TestFramework.scala:76)
at sbt.TestRunner.run(TestFramework.scala:85)
at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:185)
at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
at sbt.TestFunction.apply(TestFramework.scala:207)
at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
[error] Could not run test com.blockcypher.api.marshallers.BlockCypherOutputMarshallerTest: java.lang.OutOfMemoryError: Java heap space
[debug] Summary for ScalaCheck not available.
[debug] Summary for specs2 not available.
[info] ScalaTest
[info] Run completed in 1 second, 541 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
[error] Error: Total 0, Failed 0, Errors 0, Passed 0
[error] Error during tests:
[error] com.blockcypher.api.marshallers.BlockCypherOutputMarshallerTest
[error] (test:testOnly) sbt.TestsFailedException: Tests unsuccessful
I'm unsure of why I am running out of heap space, it seems that the piece of json is simple enough and I am not having any issues on any other similar size json test cases.
You need to close quote in your last line "pay-to-pubkey-hash
"script_type":"pay-to-pubkey-hash}"""
should be
"script_type":"pay-to-pubkey-hash"}"""