Buildbot: how to skip a step if a file doesn't exist? - buildbot

I need to skip a build step when building some branches.
More exactly, I want to execute a ShellCommand step only if the script to be ran is present on the source tree.
I tried:
ShellCommand(command=["myscript"],
workdir="path/to",
doStepIf=(lambda step: os.path.isfile("path/to/myscript")))
but the step is never executed.

def doesMyCriticalFileExist(step):
if step.getProperty("myCriticalFileExists"):
return True
return False
<factory>.addStep(SetProperty(command='[ -f /path/to/myscript ] && ls -1 /path/to/myscript || exit 0', property='myCriticalFileExists')))
<factory>.addStep(ShellCommand(command=["myscript"], workdir="path/to", doStepIf=doesMyCriticalFileExist))

The better thing to do is to set a property in a previous step and then check the property in your doStepif method. the os.path.isfile you have there gets run at configure time, (buildbot startup) not run time.

I ended up solving this by letting the step run unconditionally but setting haltOnFailure=False. This way if the required file doesn't exist it fails but doesn't kill the rest of the build.

Just use simple python if statement before the step:
if(condition):
your buildbot step

Related

CalledProcessError: Command '\['git', 'submodule--helper', 'list'\]

all.
After I use devtool-modify to edit my recipe, when I bitbake my image, something wrong happend.
ExpansionError: Failure expanding variable do_compile\[file-checksums\], expression was ${#srctree_hash_files(d)} which triggered exception
CalledProcessError: Command '\['git', 'submodule--helper', 'list'\]' returned non-zero exit status 128.
The variable dependency chain for the failure is: do_compile\[file-checksums\]
11111111111111111111111111111111111
Git is no longer supports submodule--helper list , so fixed in https://git.yoctoproject.org/poky/commit/?id=0533edac277080e1bd130c14df0cbac61ba01a0c .
So you can either apply this commit or upgrade poky!

Argo stop workflow early, mark complete

Imagine I have a workflow with 5 steps.
Step 2 may or may not create a file as its output (which is then used as input to subsequent steps).
If the file is created, I want to run the subsequent steps.
If no file gets created in step 2, I want to mark the workflow as completed and not execute steps 3 through to 5.
I'm sure there must be a simple way to do this yet I'm failing to figure out how.
I tried by making step 2 return non-zero exit code when no file is created and then using
when: "{{steps.step2.outputs.exitCode}} == 0" on step 3, but that still executes step 4 and 5 (not to mention marks step 2 as "failed")
So I'm out of ideas, any suggestions are greatly appreciated.
By default, a step that exits with a non-zero exit code fails the workflow.
I would suggest writing an output parameter to determine whether the workflow should continue.
- name: yourstep
container:
command: [sh, -c]
args: ["yourcommand; if [ -f /tmp/yourfile ]; then echo continue > /tmp/continue; fi"]
outputs:
parameters:
- name: continue
valueFrom:
default: "stop"
path: /tmp/continue
Alternatively, you can override the fail-on-nonzero-exitcode behavior with continueOn.
continueOn:
failed: true
I'd caution against continueOn.failed: true. If your command throws a non-zero exit code for an unexpected reason, the workflow won't fail like it should, and the bug might go un-noticed.

Buildbot: How do I skip a build if got_revision is the same as the last run?

I have a nightly build that is triggered by another checkin build. The trigger from checkin build is predicated by the success of that build. In other words, got_revision for the nightly build will always point to the last passing checkin build.
I'd like to skip the nightly build if got_revision is the same as last build. What would the master config look like?
Thanks in advance.
See: http://docs.buildbot.net/current/manual/cfg-schedulers.html
Example:
c['schedulers'].append(
schedulers.Nightly(name='BeforeWork',
branch=`default`,
builderNames=['builder1'],
dayOfWeek=0,
hour=[6,8],
minute=23,
onlyIfChanged=True))
onlyIfChanged=True should do the trick => makes sure to only perform the build when someone has committed code in the interim.
Following code can be used for doStepIf callback. It will skip a run if the last build was same revision and passed. This could be extended to search for all previous builds instead of the last run. Also, in my original question I asked for got_revision. In doStepIf it's best not to use got_revision because not all builds will result in valid got_revision.
def do_step_if(step):
cur_status = step.build.build_status
prev_status = cur_status.getPreviousBuild()
# this is the first build
if prev_status==None:
return True
# never skip if this is a forced run
if cur_status.getProperty("revision")==None or cur_status.getProperty("revision")=="":
return True
# got_revision won't be set if update step wasn't run, so better to use revision
if prev_status.getResults()==SUCCESS and prev_status.getProperty("revision")==cur_status.getProperty("revision"):
return False
return True

Stop the pipeline when stage is unstable

I have a Jenkins build pipeline created using workflow plugin. At the beginning the pipeline runs a gulp build inside of the docker container and then archives test results using the following code
step([$class: 'JUnitResultArchiver', testResults: 'build/test-results/*.xml'])
In the following steps I package up the artifacts and ship them to the binary repository.
When unit tests are not passing Jenkins understands the build is unstable and marks it yellow. However it still continues with subsequent steps in the pipeline. Is there any way make the pipeline stop when unit tests are failing?
the JUnitResultArchiver will cause this condition to be true when the build is unstable:
currentBuild.result != null.
If I remember correctly it sets it to UNSTABLE, but it is enough to check that is different than null.
So you could do something like
step([$class: 'JUnitResultArchiver', testResults: 'build/test-results/*.xml'])
if (currentBuild.result == null) {
//contintue with your pipeline
} else {
//notify that the build is unstable. //or just do nothing
}
There is nothing to do at Jenkins side but at Gulp side. The call to gulp CLI needs to return a proper error value to have the sh step failing correctly.
Jenkins just interprets what the shell is returning, so you juts need to make Gulp to return a fail when tests fail (see this blog post, it seems to achieve exactly that).

How to fix "Test reports were found but none of them are new. Did tests run?" in Jenkins

I am getting the error "Test reports were found but none of them are new. Did tests run?" when trying to send unit test results by email. The reason is that I have a dedicated Jenkins job that imports the artifacts from a test job to itself, and sends the test results by email. The reason why I am doing this is because I don't want Jenkins to send all the developers email during the night :) so I am "post-poning" the email sending since Jenkins itself does not support delayed email notifications (sadly).
However, by the time the "send test results by email" job executes, the tests are hours old and I get the error as specified in the question title. Any ideas on how to get around this problem?
You could try updating the timestamps of the test reports as a build step ("Execute shell script"). E.g.
cd path/to/test/reports
touch *.xml
mvn clean test
via terminal or jenkins. This generates new tests reports.
The other answer that says cd path/to/test/reports touch *.xml didn't work for me, but mvn clean test yes.
Updating the last modified date can also be achieved in gradle itself is desired:
task jenkinsTest{
inputs.files test.outputs.files
doLast{
def timestamp = System.currentTimeMillis()
test.testResultsDir.eachFile { it.lastModified = timestamp }
}
}
build.dependsOn(jenkinsTest)
As mentioned here: http://www.practicalgradle.org/blog/2011/06/incremental-tests-with-jenkins/
Here's an updated version for Jenkinsfile (Declarative Pipeline):
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'make build'
}
}
stage('Test') {
steps {
sh 'make test'
script {
def testResults = findFiles(glob: 'build/reports/**/*.xml')
for(xml in testResults) {
touch xml.getPath()
}
}
}
}
}
post {
always {
archiveArtifacts artifacts: 'build/libs/**/*.jar', fingerprint: true
junit 'build/reports/**/*.xml'
}
}
}
Because gradle caches results from previous builds I ran into the same problem.
I fixed it by adding this line to my publish stage:
sh 'find . -name "TEST-*.xml" -exec touch {} \\;'
So my file is like this:
....
stage('Unit Tests') {
sh './gradlew test'
}
stage('Publish Results') {
// Fool Jenkins into thinking the tests results are new
sh 'find . -name "TEST-*.xml" -exec touch {} \\;'
junit '**/build/test-results/test/TEST-*.xml'
}
Had same issue for jobs running repeatedly (every 30 mins).
For the job, go to Configure, Build, Advanced and within the Switches section add:
--stacktrace
--continue
--rerun-tasks
This worked for me
Navigate to report directory cd /report_directory
Delete all older report rm *.xml
Add junit report_directory/*.xml in pipeline
Rerun the test script , navigate to Build Number → Test Result
Make sure you have one successful build without any failure, only after this you can able to see the reports
Make sure that you have mentioned the correct path against "Test report XMLs" under jenkins configuration, such as "target/surefire-reports/*.xml"
There is no need to touch *.xml as jenkins won't complain even though test results xml file does not change.
if you use Windows slave, you can 'touch' results using groovy pipeline stage with powershell:
powershell 'ls "junitreports\\*.*" | foreach-object { $_.LastWriteTime = Get-Date }'
It happens if you are using a test report which is not modified by that job in that run.
In case for test purpose if you are testing with already created file then, add below command inside jenkins job under Build > Execute Shell
chmod -R 775 /root/.jenkins/workspace/JmeterTest/output.xml
echo " " >> /root/.jenkins/workspace/JmeterTest/output.xml
Above command changes timestamp of file hence error wont display.
Note: To achieve same in Execute Shell instead of above, do not try renaming file using move mv command etc. it won't work , append and delete same for change file timestamp only works.
For me commands like chmod -R 775 test-results.xml or touch test-results.xml does not work due to permission error. As work around use is to set new file in test report settings and command to copy old xml report file to new file.
you can add following shell command to your "Pre Steps" section when configure your job on Jenkins
mvn clean test
this will clean the test
Here's an updated version of the gradle task that touch each test result files.
From Jenkins pipeline script, just call "testAndTouchTestResult" task instead of "test" task.
The code below is with Kotlin syntax:
tasks {
register("testAndTouchTestResult") {
setGroup("verification")
setDescription("touch Test Results for Jenkins")
inputs.files(test.get().outputs)
doLast {
val timestamp = System.currentTimeMillis()
fileTree(test.get().reports.junitXml.destination).forEach { f ->
f.setLastModified(timestamp)
}
}
}
}
The solution for me was delete node_modules and change node version (from 7.1 to 8.4) on jenkins. That's it.