ScalaTest and SBT: Reporting progress of test suite? - scala

I am using ScalaTest and have a test suite that contains many tests, and each test can take about a minute to run.
class LargeSuite extends FunSuite {
test("Name of test 1") { ... }
...
test("Name of test n") { ... }
}
As is usual with running ScalaTest tests from the SBT console, nothing is reported to the screen until each test in the FunSuite has been run. The problem is that when n is large and each test is slow to run, you do not know what is happening. (Running top or Windows Task Manager and looking for the CPU usage of Java is not really satisfactory.)
But the real problem is when the build is run by Travis CI: Travis assumes that the build has gone wrong and kills it if 10 minutes pass and nothing is printed to the screen. In my case, Travis is killing my build, even though the tests are still running, and each individual test in a FunSuite does not require 10 minutes (although the entire suite does require more than 10 minutes because of the number of tests).
My first question is therefore this: how can I get ScalaTest to report on progress to the console after each test in FunSuite, in an idiomatic way?
My partial solution is to use the following trait as a mixin, which solves the problem with Travis:
trait ProgressConsolePrinter extends BeforeAndAfterEach with BeforeAndAfterAll {
self: Suite =>
override def beforeAll: Unit = {
Console.print(s"$suiteName running")
Console.flush
}
override def afterEach: Unit = {
Console.print(".")
Console.flush
}
override def afterAll: Unit = {
Console.println
Console.flush
}
}
But I understand that using Console to print to the SBT console is not entirely reliable (Googling this seems to somewhat confirm this from the experiences of others).
Also (i) anything you print from Console does not go via SBT's logger, and is therefore not prefixed with [info], and (ii) when trying the above, the messages printed from Console are jumbled up with other messages from SBT. If I was able to use the proper logger, then neither of these things would happen.
My second question is therefore this: How can I print to an SBT logger from within a test in ScalaTest?

To get ScalaTest to report to the console after each test within SBT, add
logBuffered in Test := false
to your build configuration. (See the SBT docs.)
For general logging purposes within SBT, you can use an instance of sbt.util.Logger obtained from streams.value.log within an SBT task as described here. Wilson's answer below can be used if logging is required from the test code.
(I'm answering my own question two years later, but this is currently the first hit on Google for "ScalaTest sbt progress".)

This may be helpful for your second question.
To log like that you must use scala logging. You must add scala logging as a test dependency, extend some of its logging classes (suggest LazyLogging) and then call logger.info("Your message").

Related

How to only run tests not having a certain tag in scala using flatspec through sbt?

As a scala beginner, I want to tag the integration tests in order to exclude them from running in certain scenarios (as they can be quite slow and might break due to external changes/problems).
I created the tag integration this way:
import org.scalatest.{FlatSpec, Matchers, Tag}
object integration extends Tag("com.dreamlines.tags.integration")
In my test I tag a test like so:
class SchemaValidation extends FlatSpec with Matchers {
it should "return valid json" taggedAs (integration) in {
...
assertSchema(response, endpointSchema)
}
}
Yet when reading up on how to filter certain tests based on the tag in the docs, I get highly confused as suddenly I read that I should be using org.scalatest.tools.Runner
scala [-cp scalatest-<version>.jar:...] org.scalatest.tools.Runner [arguments]
which has the flags I am looking for:
-l specifies a tag to exclude (Note: only one tag name allowed per -l) -l SlowTests -l PerfTests
Yet I am only used to run my tests via:
sbt test
I have no idea what the scala test runner here is referring to or what goes on under the hood when I run sbt test. I am truly lost.
I was expecting to execute the tests not entirely unlike this:
sbt test --ignore-tag integration
According to the sbt documentation on test options the following should work for you:
testOnly -- -l integration
Or in case ScalaTest uses the fully qualified tag id:
testOnly -- -l com.dreamlines.tags.integration
In case you want to do this by default, you can adjust the testOptions in the build.sbt file.
Edit:
You might consider a different approach for integration test. Instead of tagging them, you might want to put them in the src/it/{scala|resources} folders.

Is there a way to tag specs2 tests in Scala? [duplicate]

I have been using ScalaTest for a while and I find pretty useful the ability to Tag your tests and run just those with a specific Tag from the command line.
Is there anything similar in Specs2?
I know that you can run a specific test class with testOnly but I would like to just run a test with a specific Tag within a Specification.
Here is how to do it:
import org.specs2.mutable._
class MySpec extends Specification {
tag("fast")
"example1" >> ok
tag("slow")
"example2" >> ok
}
Then in sbt
sbt> test-only *MySpec* -- include fast
You will find more information here.

TypeSafe config in tests (Could not resolve substitution to a value)

All,
I am using the HOCON based typesafe config in a project.
This is a Scala project, and we are striving for 100% test coverage.
I have two test configs: test1.conf and test2.conf ... here is the code for loading
object TypeSafeTestConfigProvider {
System.setProperty("foo","value")
System.setProperty("bar","value")
val config1 = ConfigFactory.load("test1.conf")
val config2 = ConfigFactory.load("test2.conf")
}
neither foo nor bar can be resolved for some reason if I run a single test ... yet they resolve without an issue if I run all my tests together ... which is very inconvenient.
It appears the ConfigFactory class takes a static snapshot of System.properties and future changes to System properties are not reflected in the resolution process.
How would you recommend I setup my tests?
com.typesafe.config.ConfigFactory.invalidateCaches(); did the job for me. Playing with System Properties in unit tests is a Pain.

ScalaTest Plus not recognizing tests

I've been tasked to update and write a series of tests on an app in Scala Play, a language and framework I'm unfamiliar with. Part of what I'd like to do is integrate the ScalaTestPlus library. To get started I have been following the following tutorial:
https://www.playframework.com/documentation/2.2.x/ScalaTestingWithScalaTest
Unfortunately I am not getting very far. I have added a new unit test file to the tests folder:
import org.scalatestplus.play._
class StackSpec extends PlaySpec {
"A Test" must {
"pass" in {
assert(1 == 1)
}
"Fail" in {
assert(1 != 1)
}
}
}
and I have updated my build.sbt to include the scalatestplus library
"org.scalatestplus" % "play_2.37" % "1.2.0" % "test"//,
Using Activator, I am trying to run my test file with test-only. Everything compiles without errors, but activator is not finding any tests
[info] No tests were executed.
I don't believe the issue is with activator, since I can run old test files (from the previous engineer) using the test and test-only commands. A quick sample of one of the previous (working) test files:
import java.util.concurrent.TimeUnit
import com.sun.xml.internal.bind.v2.TODO
import scala.collection.JavaConverters._
import controllers.Application
import models.{Item, PriorityBucket}
import play.api.test._
class WebSpec extends PlaySpecification {
"Home page" should {
"do something" in new WithSeleniumDbData(TestUtil.testApp) {
Redacted.deleteAll()
val ObId = TestUtil.create(Some(PriorityBucket.Low),
Some(Application.ENGLISH))
val item = Item.find(ItemId).get
browser.goTo("/")
browser.await().atMost(2,
TimeUnit.SECONDS).until(Selectors.all_obs).isPresent
}
Any ideas where I've gone astray? Thanks in advance for the help!
I am using scala 2.11
I am using play 2.3.7
EDIT: Possibly relevant, I switched the extension from PlaySpec to FlatSpec and saw the following error when compiling:
SampleSpec.scala:10: value in is not a member of String
[error] "pass" in {
I made sure to import FlatSpec as well, which has me a bit confused--is FlatSpec a member of ScalaTest but not a member of ScalaTestPlus, I don't see why else the compilation would fail.
UPDATE: To further investigate the issue I spun up a brand new Play app and copied over my sample test. After some tooling around with versions I've been able to get my test to run on the activator test command with the rest of the suite. Unfortunately, any other commands like test-only are still returning no tests run.
For those following I ran across the issue...the class name in this case needed to be identical to the file name, otherwise test-only cannot locate it.

Using simple-build-tool for benchmarks

I'm trying to get sbt to compile and build some benchmarks. I've told it to add the benchmarks to the test path so they're recompiled along with tests, but I can't figure out how to write an action to let me actually run them. Is it possible to invoke classes from the Project definition class, or even just from the command line?
Yes, it is.
If you'd like to run them in the same VM the SBT is run in, then write a custom task similar to the following in your project definition file:
lazy val benchmark = task {
// code to run benchmarks
None // Some("will return an error message")
}
Typing benchmark in SBT console will run the task above. To actually run the benchmarks, or, for that matter, any other class you've compiled, you can reuse some of the existing infrastructure of SBT, namely the method runTask which will create a task that runs something for you. It has the following signature:
def runTask(mainClass: => Option[String], classpath: PathFinder, options: String*): Task
Simply add the following to your file:
lazy val benchmark = task { args =>
runTask(Some("whatever.your.mainclass.is"), testClasspath, args)
}
When running benchmarks, it is sometimes recommended that you run them in a separate jvm invocation, to get more reliable results. SBT allows you to run separate processes by invoking a method ! on a string command. Say you have a command java -jar path-to-artifact.jar you want to run. Then:
"java -jar path-to-artifact.jar" !
runs the command in SBT. You want to put the snippet above in a separate task, same as earlier.
And don't forget to reload when you change your project definition.
Couldn't you simply write the benchmarks as tests, so they will be run when you call 'test' in SBT?
You could also run a specific test with 'test-only', or run a main with 'run' or 'exec' (see http://code.google.com/p/simple-build-tool/wiki/RunningSbt for details).