For SonarQube jobs in Jenkins we'd like to proceed even though some tests might fail. Currently the Sonar Runner is not kicked off, because a test fails.
In Maven you'd just add -DtestFailureIgnore = true, but I cannot find anything similar for SBT.
I did find a onFailure thing for sbt, but have not found any examples anywhere how to use this. Could this be used to ignore test failures so the build job continues so the Sonar Runner gets started afterwards?
Or is there a setting in Jenkins to ignore the result of the build?
We use 'sbt clean coverage test coverageReport' as build command and have Sonar Runner in a post-build step.
Finally found a solution myself.
In SBT you can define a new task A which captures the result of another task B. This dependency ensures that task B is run when the new task A is started. By capturing the result, the result of task B is not the result of task A so if B fails, A does not (have to) fail.
So in this case, I added created a new 'ciTests' tasks to the 'build.sbt'
// Define a special test task which does not fail when any test fails,
// so sequential tasks (like SonarQube analysis) will be performed no matter the test result.
lazy val ciTests = taskKey[Unit]("Run tests for CI")
ciTests := {
// Capture the test result
val testResult = (test in Test).result.value
}
Now in the Jenkins job it build the project using SBT with commands (using SCoverage SBT plugin):
update coverage ciTests coverageReport
This build will succeed ignoring any failing tests. Therefore a next build step to start SonarRunner will start the analysis of the Scala project and put the results in SonarQube.
Thanks to #hugo-zwaal for pointing me to this answer which helped me solving my issue.
Related
I have configured one SBT multi-module project with scoverage plugin, which is working fine.
To generate test coverage, I am using > SBT clean coverage test coverageReport but is there any way to create a new task which chains internally coverage test coverageReport.
I have tried
Run custom task automatically before/after standard task to create a custom task, but it seems not working with multimodule project.
And one more - http://eed3si9n.com/sequencing-tasks-with-sbt-sequential
Try addCommandAlias like so
addCommandAlias("coverageAll", ";clean;coverage;test;coverageReport")
Now executing sbt coverageAll should generate coverage report for all the sub-projects.
Scala's sbt has the option to run only tests that previously failed. Is there a gradle equivalent?
https://www.scala-sbt.org/1.x/docs/Testing.html#testQuick
gradle does this automatically with all update-to-date tests marked as such, based on code changes since latest test passing.
You can also force re-run of tests if need be: How to run Gradle test when all tests are UP-TO-DATE?
What sbt task does intellij IDEA 14 use to run scala tests with coverage? I'm test my spark code and need to prevent them running in parallel.
I've added the following to my build.sbt file and it prevents tests running in parallel when they aren't generating coverage reports:
parallelExecution in Test := false
However this has no effect when running with coverage. I tried using something similar, but with ScctTest instead of Test, but sbt couldn't resolve it.
So, what coverage plugin does intellij use, and how can I disable parallel test execution when running tests with coverage? Running sbt tasks doesn't show anything containing the word coverage. I haven't enabled the Emma plugin in intellij - only the default Coverage one is enabled and it has no information.
Try to add to settings:
fork in ThisBuild in Test:= false
If there is a test that fails only sometimes, can I ask sbt to run tests continually until failure?
Otherwise, I'm stuck hitting up-arrow while watching Arrow. (pun unintended but leavening)
https://stackoverflow.com/q/22366719/1296806
I had the same problem, and I've ended up implementing this as a command.
lazy val root = Project("test", file(".")).
settings(commands += Command.command("testUntilFailed") { state =>
"test" :: "testUntilFailed" :: state
})
The advantage is that it will skip the SBT loading. You can also add extra parameter or run testOnly to test a single test.
Old question, but having just had the same need myself, here is a solution: sbt will return non-zero exit code if you run it one off with the tests, so, one way to loop until it fails is to just look at the exit code in the shell:
while [ $? -eq 0 ]; do sbt test; done
I have created sbt plugin for flaky test detection: sbt-flaky. You can run test for:
specified duration sbt clean "flaky duration=30",
specified times sbt clean "flaky times=30"
until first failure sbt clean "flaky firstFail".
Advantage of this plugin is aggregation of failures, history trends and possibility of adding flaky test detection into pipeline.
In my opinion it doesn't particularly apply to SBT or any other build management tool. There's no built-in SBT feature for this. You'd have to build one, but it'd be far from what SBT is meant to offer - build configuration management. It can take quite a while to hit the case which causes the build/test fail. It's very unpredictable.
When your test fails, it means that there are cases the test doesn't pass. The error should tell you what they are that you use to improve the test.
ScalaCheck: Property-based testing for Scala might be of some help.
Hi I want to write a Spock test for my Gradle plugin to test if a report is being generated after the execution of a task from the plugin, so
private ProjectInternal project
...
public void 'check tasks'(){
given:
project.gradle.startParameter.taskNames = ["myTaskName"]
project.gradle.buildListenerBroadcaster.projectsLoaded(project.gradle)
when:
project.plugins.apply(MYPlugin.class)
project.?????
then:
...
But the "then:" section has to check the existance of a file but for this "myTaskName" has to be executed, how to make Ggradle to execute my task? There is no such method
afterExecution ??
This kind of test is suitable for testing plugins, but not for testing tasks. Applying a plugin only configures tasks, it doesn't execute them. In order to execute tasks, you'll have to kick off a "real" build from your test. The recommended way to do this is via the Gradle tooling API.
If you are prior Gradle 5 you can use project.your_task.execute()