I've been working on a Scala application. To do it properly, I want the kernel of my code to be completely covered by tests. To do that, I'm using the SCCT plugin for SBT.
Unfortunately, my tests rely on an in-memory database (h2). Therefore, my tests cannot run in parallel. However, sbt runs all tasks in parallel by default.
To solve this, I've modified build.sbt file to disable parallel execution of the tests as:
parallelExecution in Test := false
The problem with the build configuration is that to generate code coverage, I need to run sbt scct:test rather than sbt test.
I've tried to disable the parallel execution of scct:test but the code below does not compile:
parallelExecution in scct:test := false
Can anyone help me by either disabling the parallel execution in scct:test by either setting the flag, or by making scct:test run in the test context?
I think the task name is actually ScctTest. Try:
parallelExecution in Test := false
parallelExecution in ScctTest := false
Have you considered creating a new database for each test and making the db name that is created be a random name for each one so they dont clash. Works well as long as you dont have absolutely massive numbers of tests which require a database.
Related
I have some long running tests in my project. These these are sitting in parallel to my integration and unit-tests in
/test/manual/*
Is there in Play 2.4 for Scala a way to disable/mark these test classes. So they are not run automaticly when
$ activator test
but only run when using the test-only command.
Problem is that I do not want to run these longer tests on my CI server.
Having similar problems for long-running integration tests, I created an It configuration derived from the standard test config (in <projectHome>/build.sbt):
lazy val It = config("it").extend(Test)
Then I add the sources and test sources to this config
scalaSource in It <<= (scalaSource in Test)
and you needd to enable to config and corresponding tasks available in the current project
lazy val root = (project in file(".")).configs(It)
.settings(inConfig(It)(Defaults.testTasks): _*)
I then disable long running tests in the Test config :
testOptions in Test := Seq(Tests.Argument("exclude", "LongRunning"))
And include only these long running tests in the It config:
testOptions in It := Seq(Tests.Argument("include", "LongRunning"))
These last 2 configs are kinda dependent on the test framework you use (specs2 in my case, scala test would probably use -n and -l in addition to tags to achieve the same)
Then sbt test will exclude all LongRunning tests and you can run it:test or it:testOnly your.long.running.TestCaseHere in an interactive sbt session if need be.
What sbt task does intellij IDEA 14 use to run scala tests with coverage? I'm test my spark code and need to prevent them running in parallel.
I've added the following to my build.sbt file and it prevents tests running in parallel when they aren't generating coverage reports:
parallelExecution in Test := false
However this has no effect when running with coverage. I tried using something similar, but with ScctTest instead of Test, but sbt couldn't resolve it.
So, what coverage plugin does intellij use, and how can I disable parallel test execution when running tests with coverage? Running sbt tasks doesn't show anything containing the word coverage. I haven't enabled the Emma plugin in intellij - only the default Coverage one is enabled and it has no information.
Try to add to settings:
fork in ThisBuild in Test:= false
If there is a test that fails only sometimes, can I ask sbt to run tests continually until failure?
Otherwise, I'm stuck hitting up-arrow while watching Arrow. (pun unintended but leavening)
https://stackoverflow.com/q/22366719/1296806
I had the same problem, and I've ended up implementing this as a command.
lazy val root = Project("test", file(".")).
settings(commands += Command.command("testUntilFailed") { state =>
"test" :: "testUntilFailed" :: state
})
The advantage is that it will skip the SBT loading. You can also add extra parameter or run testOnly to test a single test.
Old question, but having just had the same need myself, here is a solution: sbt will return non-zero exit code if you run it one off with the tests, so, one way to loop until it fails is to just look at the exit code in the shell:
while [ $? -eq 0 ]; do sbt test; done
I have created sbt plugin for flaky test detection: sbt-flaky. You can run test for:
specified duration sbt clean "flaky duration=30",
specified times sbt clean "flaky times=30"
until first failure sbt clean "flaky firstFail".
Advantage of this plugin is aggregation of failures, history trends and possibility of adding flaky test detection into pipeline.
In my opinion it doesn't particularly apply to SBT or any other build management tool. There's no built-in SBT feature for this. You'd have to build one, but it'd be far from what SBT is meant to offer - build configuration management. It can take quite a while to hit the case which causes the build/test fail. It's very unpredictable.
When your test fails, it means that there are cases the test doesn't pass. The error should tell you what they are that you use to improve the test.
ScalaCheck: Property-based testing for Scala might be of some help.
I have the following lines in my build.sbt
fork := true
javaOptions in run += "-Dmy.environment=local"
javaOptions in test += "-Dmy.environment=local"
This works totally fine when I use the "test" command in sbt and when my code checks the jvm system settings for my.environment, it finds the correct value (i.e., "local").
The problem that I am having is that when I run "test-only org.whatever.SomeTest" in this case the my.environment key isn't in the jvm system settings. Specifically, System.getProperty("my.environment") is null whereas it was "local" when I just ran "test".
Does anyone know how to fix this?
You are likely hitting #975: regression: fork in test doesn't work anymore, which is currently under review. Try:
javaOptions in Test += "-Dmy.environment=local"
I have a multi-project build with tests in sub-projects and in a parent project. The build is aggregated so that the parent project runs all tests in child projects.
I configured it so that there's no parallel execution of tests in both the sub-projects and the parent project, via
parallelExecution in Test := false
However, I have the nagging feeling that tests that span over multiple projects are ran in parallel. In the case of one of the sub-projects this is a problem because it mutates state in a test database concurrently, leading to the test to fail.
Any ideas as to how to globally switch of parallel execution of tests, between projects?
I think you can apply a setting across projects using scope ThisBuild, like
parallelExecution in ThisBuild := false
I don't know if you can combine that with scope Test, but it might not be necessary.
To restrict the number of concurrently executing tests in all projects, use:
concurrentRestrictions in Global += Tags.limit(Tags.Test, 1)
See sbt documentation
See discussion
This worked for me in 1.1.0:
Test / parallelExecution := false
See my answer here How to run subprojects tests (including setup methods) sequentially when testing
There is another way to prevent parallel execution. You can make the test tasks of the different projects depend on each other:
test in Project2 := (test in Project2).dependsOn(test in Project1).value
parallelExecution in Test in Project2 := false
Another possibility, based on https://stackoverflow.com/a/27068019/1922026, is to define a command alias in the root project:
.settings(addCommandAlias("test", ";s1/test;s2/test;s3/test"): _*)
where s1, s2 and s3 are the sub-projects. When you are in root project and run "test" the tests will be executed sequentially and in the order defined.
You can try also Global / parallelExecution := false
The "modern" (i.e. sbt 1.x) equivalent of disabling parallel execution in Test scope is to add the following to your build.sbt:
Global / concurrentRestrictions += Tags.limit(Tags.Test, 1)
For those not familiar with sbt syntax, in context you want to do something like:
lazy val main = project
.in(file("."))
.settings(
name := "foo",
// more settings
// ...
Global / concurrentRestrictions += Tags.limit(Tags.Test, 1)
)
From the tags and rules section of sbt docs.
p.s. quite a useful setting for ScalaTest, particularly around setup/teardown logic during database testing. Fixes some quite puzzling non-deterministic errors that you'll ineviitably hit with parallel execution enabled.