If there is a test that fails only sometimes, can I ask sbt to run tests continually until failure?
Otherwise, I'm stuck hitting up-arrow while watching Arrow. (pun unintended but leavening)
https://stackoverflow.com/q/22366719/1296806
I had the same problem, and I've ended up implementing this as a command.
lazy val root = Project("test", file(".")).
settings(commands += Command.command("testUntilFailed") { state =>
"test" :: "testUntilFailed" :: state
})
The advantage is that it will skip the SBT loading. You can also add extra parameter or run testOnly to test a single test.
Old question, but having just had the same need myself, here is a solution: sbt will return non-zero exit code if you run it one off with the tests, so, one way to loop until it fails is to just look at the exit code in the shell:
while [ $? -eq 0 ]; do sbt test; done
I have created sbt plugin for flaky test detection: sbt-flaky. You can run test for:
specified duration sbt clean "flaky duration=30",
specified times sbt clean "flaky times=30"
until first failure sbt clean "flaky firstFail".
Advantage of this plugin is aggregation of failures, history trends and possibility of adding flaky test detection into pipeline.
In my opinion it doesn't particularly apply to SBT or any other build management tool. There's no built-in SBT feature for this. You'd have to build one, but it'd be far from what SBT is meant to offer - build configuration management. It can take quite a while to hit the case which causes the build/test fail. It's very unpredictable.
When your test fails, it means that there are cases the test doesn't pass. The error should tell you what they are that you use to improve the test.
ScalaCheck: Property-based testing for Scala might be of some help.
Related
According to sbt documentation, when running testQuick, it executes
The tests that failed in the previous run
The tests that were not run before
The tests that have one or more transitive dependencies, maybe in a different project, recompiled.
How does sbt determine which tests qualify? If it is based on cached results, where is this cache?
By default its kept in target/streams/test/test/$global/streams/succeded_tests (path may vary depending on build settings) if You are interested how exactly its picking tests to run check testQuickFilter method in: https://github.com/sbt/sbt/blob/fd20d3039ad06cbee47c6386dc5839060417014b/main/src/main/scala/sbt/Defaults.scala#L758
Consider the scenario: I have a simple scala project managed by sbt, in a directory called foo.
cd foo
sbt compile
It now takes a few seconds and the project compiles correctly.
Now, thanks to sbt incremental compilation, if I run
sbt compile
it terminates the compile task in close to 0 seconds, since the source code hasn't changed.
But if I rename the foo directory into bar
cd ..
mv foo bar
and try to compile it again
cd bar
sbt compile
now the compile task takes again few seconds, hence not using the results from the previous compilation.
Is there a way to preserve the incremental compilation results of a project even when its absolute path on disk has changed?
You can also try to use Hoarder Plugin: https://github.com/romanowski/hoarder.
Ping me on Hoarder's gitter in case of any questions/problems.
As it turns out, there's some work in progress to make incremental compilation results cacheable. Here's the relevant PR: https://github.com/sbt/zinc/pull/216.
Also, there's an issue (being discussed at the time of this writing) about making zinc's analysis completely machine independent. See https://github.com/sbt/zinc/issues/218.
For SonarQube jobs in Jenkins we'd like to proceed even though some tests might fail. Currently the Sonar Runner is not kicked off, because a test fails.
In Maven you'd just add -DtestFailureIgnore = true, but I cannot find anything similar for SBT.
I did find a onFailure thing for sbt, but have not found any examples anywhere how to use this. Could this be used to ignore test failures so the build job continues so the Sonar Runner gets started afterwards?
Or is there a setting in Jenkins to ignore the result of the build?
We use 'sbt clean coverage test coverageReport' as build command and have Sonar Runner in a post-build step.
Finally found a solution myself.
In SBT you can define a new task A which captures the result of another task B. This dependency ensures that task B is run when the new task A is started. By capturing the result, the result of task B is not the result of task A so if B fails, A does not (have to) fail.
So in this case, I added created a new 'ciTests' tasks to the 'build.sbt'
// Define a special test task which does not fail when any test fails,
// so sequential tasks (like SonarQube analysis) will be performed no matter the test result.
lazy val ciTests = taskKey[Unit]("Run tests for CI")
ciTests := {
// Capture the test result
val testResult = (test in Test).result.value
}
Now in the Jenkins job it build the project using SBT with commands (using SCoverage SBT plugin):
update coverage ciTests coverageReport
This build will succeed ignoring any failing tests. Therefore a next build step to start SonarRunner will start the analysis of the Scala project and put the results in SonarQube.
Thanks to #hugo-zwaal for pointing me to this answer which helped me solving my issue.
I have a typical sbt (0.13) build and have added the jacoco4sbt plugin to my build.
addSbtPlugin("de.johoop" % "jacoco4sbt" % "2.1.1")
I use specs2 to run my tests (2.2.2).
If I run
~>sbt
>test
all my tests get run (120 of them). However, if I do
>jacoco:test
it runs 0 tests, as if the jacoco configuration cannot find them.
A quick search reveals that there is an issue with jacoco4sbt and Play because Play sets parallelExecution to false. However, I am not using Play, and parallelExecution is set to True for both configurations. I have tried to set them both to false to no avail.
Any idea what might be going wrong?
n.b. The project I am working on is open source, so I created a branch where I put my attempt at adding jacoco4sbt. Feel free to clone it and see what is happening for yourself.
https://github.com/jedesah/scala-codesheet-api/tree/jacoco
I had this issue, but upgraded to Specs2 2.2.3 and jacoco4sbt started producing output from that point.
For what it's worth, I had the same problem when using specs2. When I switched to ScalaTest, jacoco4sbt started detecting my tests.
I have a very basic configuration too, so I don't know we're missing something or if there's something wrong in the current jacoco4sbt version. I did try version 2.1.0 of jacoco4sbt but had the same results.
Is there a way to build tests with SBT without running them?
My own use case is to run static analysis on the test code by using a scalac plugin. Another possible use case is to run some or all of the test code using a separate runner than the one built into SBT.
Ideally there would be a solution to this problem that applies to any SBT project. For example, Maven has a test-compile command that can be used just to compile the tests without running them. It would be great if SBT had the same thing.
Less ideal, but still very helpful, would be solutions that involve modifying the project's build files.
Just use the Test / compile command.
Test/compile works for compiling your unit tests.
To compile integration tests you can use IntegrationTest/compile.
Another hint to continuously compile on every file change: ~Test/compile
We have a build.sbt file that is used for multiple projects. Doing sbt test:compile compiled the tests for every single project and took over 30 minutes.
I found out I can compile only the tests for a specific project named xyz by doing:
sbt xyz/test:compile
Using sbt version 1.5.0 and higher test:compile returns deprecation warning.
Use Test / compile.
(docs)