I have a Gradle project with some tests. These tests are directly written in Spec files.
I want to run subset of these specs in parallel (Test inside specs shouldn't run in parallel). So I created a Suite like below:
class MainSuite extends Suites(
new TestSpec1,
new TestSpec2,
new TestSpec3) with BeforeAndAfterAll
{
//...
}
The reason for this is to be able to run something right before any of these TestSpecs. How can I run these Specs inside this Suite in parallel?
Related
I'm writing an integration test method in scala (play framework). The test class is SourceIntegrationTest. I've placed a file, source.json, in /test/resources. I'm aware that "sbt copies files from src/test/resources to target/scala-[scalaVersion]/test-classes" as described in this answer. However, using the answer referenced there only works for me when running my test in IntelliJ. When I run sbt it:testOnly SourceIntegrationTest in terminal, my test fails with a NullPointerException. sbt cannot find source.json. How can I get sbt to find my file when running my integration test in terminal?
My test method looks like:
#Test
def testGetSource(): Unit = {
val jsonSource: String = Source.fromInputStream(getClass.getClassLoader.getResourceAsStream("source.json")).mkString
val json: JsValue = Json.parse(jsonSource)
val source = controller.getSource(json)
assertEquals(source.sourceName = "Premier")
}
When you write an integration test, the test file is not placed in the test folder. You need to create another directory for integration test. Now the source folder will contain three directories main, test, it. And all integration test will be kept in it folder. You can read about it here
As a scala beginner, I want to tag the integration tests in order to exclude them from running in certain scenarios (as they can be quite slow and might break due to external changes/problems).
I created the tag integration this way:
import org.scalatest.{FlatSpec, Matchers, Tag}
object integration extends Tag("com.dreamlines.tags.integration")
In my test I tag a test like so:
class SchemaValidation extends FlatSpec with Matchers {
it should "return valid json" taggedAs (integration) in {
...
assertSchema(response, endpointSchema)
}
}
Yet when reading up on how to filter certain tests based on the tag in the docs, I get highly confused as suddenly I read that I should be using org.scalatest.tools.Runner
scala [-cp scalatest-<version>.jar:...] org.scalatest.tools.Runner [arguments]
which has the flags I am looking for:
-l specifies a tag to exclude (Note: only one tag name allowed per -l) -l SlowTests -l PerfTests
Yet I am only used to run my tests via:
sbt test
I have no idea what the scala test runner here is referring to or what goes on under the hood when I run sbt test. I am truly lost.
I was expecting to execute the tests not entirely unlike this:
sbt test --ignore-tag integration
According to the sbt documentation on test options the following should work for you:
testOnly -- -l integration
Or in case ScalaTest uses the fully qualified tag id:
testOnly -- -l com.dreamlines.tags.integration
In case you want to do this by default, you can adjust the testOptions in the build.sbt file.
Edit:
You might consider a different approach for integration test. Instead of tagging them, you might want to put them in the src/it/{scala|resources} folders.
I need some help regarding the run of the collected tests. So here is my scenario. I have the tests structured like this:
tests_folder {
feature01_tests_folder {
tests
}
feature02_tests_folder {
tests
}
}
I'm using Appium Python. My issue is that if I collect only one tests folder and then make the bundle zip with all the project, on AWS will run all the tests. So what is the point of collecting only some tests?
Notes:
Collection was ok. After running the py.test--collect-only feature_01_tests_folder command only the specified tests were collected.
Thanks
I am using ScalaTest and have a test suite that contains many tests, and each test can take about a minute to run.
class LargeSuite extends FunSuite {
test("Name of test 1") { ... }
...
test("Name of test n") { ... }
}
As is usual with running ScalaTest tests from the SBT console, nothing is reported to the screen until each test in the FunSuite has been run. The problem is that when n is large and each test is slow to run, you do not know what is happening. (Running top or Windows Task Manager and looking for the CPU usage of Java is not really satisfactory.)
But the real problem is when the build is run by Travis CI: Travis assumes that the build has gone wrong and kills it if 10 minutes pass and nothing is printed to the screen. In my case, Travis is killing my build, even though the tests are still running, and each individual test in a FunSuite does not require 10 minutes (although the entire suite does require more than 10 minutes because of the number of tests).
My first question is therefore this: how can I get ScalaTest to report on progress to the console after each test in FunSuite, in an idiomatic way?
My partial solution is to use the following trait as a mixin, which solves the problem with Travis:
trait ProgressConsolePrinter extends BeforeAndAfterEach with BeforeAndAfterAll {
self: Suite =>
override def beforeAll: Unit = {
Console.print(s"$suiteName running")
Console.flush
}
override def afterEach: Unit = {
Console.print(".")
Console.flush
}
override def afterAll: Unit = {
Console.println
Console.flush
}
}
But I understand that using Console to print to the SBT console is not entirely reliable (Googling this seems to somewhat confirm this from the experiences of others).
Also (i) anything you print from Console does not go via SBT's logger, and is therefore not prefixed with [info], and (ii) when trying the above, the messages printed from Console are jumbled up with other messages from SBT. If I was able to use the proper logger, then neither of these things would happen.
My second question is therefore this: How can I print to an SBT logger from within a test in ScalaTest?
To get ScalaTest to report to the console after each test within SBT, add
logBuffered in Test := false
to your build configuration. (See the SBT docs.)
For general logging purposes within SBT, you can use an instance of sbt.util.Logger obtained from streams.value.log within an SBT task as described here. Wilson's answer below can be used if logging is required from the test code.
(I'm answering my own question two years later, but this is currently the first hit on Google for "ScalaTest sbt progress".)
This may be helpful for your second question.
To log like that you must use scala logging. You must add scala logging as a test dependency, extend some of its logging classes (suggest LazyLogging) and then call logger.info("Your message").
I'm trying to get sbt to compile and build some benchmarks. I've told it to add the benchmarks to the test path so they're recompiled along with tests, but I can't figure out how to write an action to let me actually run them. Is it possible to invoke classes from the Project definition class, or even just from the command line?
Yes, it is.
If you'd like to run them in the same VM the SBT is run in, then write a custom task similar to the following in your project definition file:
lazy val benchmark = task {
// code to run benchmarks
None // Some("will return an error message")
}
Typing benchmark in SBT console will run the task above. To actually run the benchmarks, or, for that matter, any other class you've compiled, you can reuse some of the existing infrastructure of SBT, namely the method runTask which will create a task that runs something for you. It has the following signature:
def runTask(mainClass: => Option[String], classpath: PathFinder, options: String*): Task
Simply add the following to your file:
lazy val benchmark = task { args =>
runTask(Some("whatever.your.mainclass.is"), testClasspath, args)
}
When running benchmarks, it is sometimes recommended that you run them in a separate jvm invocation, to get more reliable results. SBT allows you to run separate processes by invoking a method ! on a string command. Say you have a command java -jar path-to-artifact.jar you want to run. Then:
"java -jar path-to-artifact.jar" !
runs the command in SBT. You want to put the snippet above in a separate task, same as earlier.
And don't forget to reload when you change your project definition.
Couldn't you simply write the benchmarks as tests, so they will be run when you call 'test' in SBT?
You could also run a specific test with 'test-only', or run a main with 'run' or 'exec' (see http://code.google.com/p/simple-build-tool/wiki/RunningSbt for details).