jenkins plugin to fail build based on console output - plugins

Is there a plugin to fail a build if a pattern occurs on the console output?
For example:
Build Action - success
Build Action - success
Build Action - error_pattern
Build Action - success
Let's assume that the Jenkins build process does not fail the build on error_pattern, I need to have an external fail trigger of some sort.
EDIT
Looking for a solution to fail the build during, not a post build task.

You should try the Post Build Task plugin.
You can search for a pattern and then launch a script.
edit:
There is also Text finder plugin, looks better for your problem

As an ugly workaround I do the following: in the build script redirect all the output to some resulting .log file, then you can grep through this file in the background the way you like (personally I do the freezing check additionally - calculate the checksum and compare with previous, if the same - start counting for timeout until threshold), etc...
Disadvantage is the output goes to some file instead of Jenkins console, but I guess you can do both using tee (I don't care, because my goal is to archive the log anyways and send it via email, - so I just gzip my resulting .log file and attach it as an artifact to the build record + to the email).
Advantage is you have full control on what happens in the build output and can interrupt the build using your own return code / message.

I've used this answer as the basis of a pipeline script. In the "Build" stage I have two parallel sub-stages - one of them is doing the actual build and outputting to a log, whilst the other sub-stage is grepping the same log. I've negated the exit code (! grep) so that it'll error when the "ERROR:" string is found. Thanks to the failFast setting, this will cause the whole "Build" stage to fail as soon as the string is found. There's an extra grep at the end of the first sub-stage, in case an error was produced right at the end.
I'm tracking the build status with a variable (BUILD_COMPLETE).
pipeline {
agent {
label 'master'
}
environment {
BUILD_COMPLETE = false
}
stages {
stage('Build') {
failFast true
parallel {
stage('Building') {
steps {
sh 'echo foo | tee output.log'
sleep 5
sh 'echo bar | tee -a output.log'
sleep 5
sh 'echo ERROR: baz | tee -a output.log'
sleep 5
sh 'echo qux | tee -a output.log'
sleep 5
sh '! grep "^ERROR:" output.log'
script {
BUILD_COMPLETE = true
}
}
}
stage('Monitoring the logs') {
steps {
script {
while (BUILD_COMPLETE != true) {
sh '! grep "^ERROR:" output.log'
sleep 1
}
}
}
}
}
}
}
}

Related

Luigi does not send error codes to concourse ci

I have a test pipeline on concourse with one job that runs a set of luigi tasks. My problem is: failures in the luigi tasks do not rise up to the concourse job. In other words, if a luigi task fails, concourse will not register that failure and states that the concourse job completed successfully. I will first post the code I am running, then the solutions I have tried.
luigi-tasks.py
class Pipeline1(luigi.WrapperTask):
def requires(self):
yield Task1()
yield Task2()
yield Task3()
tasks.py
class Task1(luigi.Task):
def requires(self):
return None
def output(self):
return luigi.LocalTarget('stuff/task1.csv')
def run(self):
#uncomment line below to generate task failure
#assert(True==False)
print('task 1 complete...')
t = pd.DataFrame()
with self.output().open('w') as outtie:
outtie.write('complete')
# Tasks 2 and 3 are duplicates of this, but with 1s replaced with 2s or 3s.
config file
[retcode]
# codes are in increasing level of severity (for most applications)
already_running=10
missing_data=20
not_run=25
task_failed=30
scheduling_error=35
unhandled_exception=40
begin.sh
#!/bin/sh
set -e
export PYTHONPATH='.'
luigi --module luigi-tasks Pipeline1 --local-scheduler
echo $?
pipeline.yml
# <resources, resource types, and docker image build job defined here>
#job of interest
- name: run-docker-image
plan:
- get: timer
trigger: true
- get: docker-image-ecr
passed: [build-docker-image]
- get: run-git
- task: run-script
image: docker-image-ecr
config:
inputs:
- name: run-git
platform: linux
run:
dir: ./run-git
path: /bin/bash
args: ["begin.sh"]
I've introduced errors in a few ways: assertions/raising an exception (ValueError) within an individual task's run() method and within the wrapper, and sys.exit(luigi.retcodes.retcode().unhandled_exception). I also tried failing all tasks. I did this in case the error needed to be generated in a specific manner/location. Though they all produced a failed task, none of them produced an error in the concourse server.
At first, I thought concourse just gives a success if it can run the file or command tasked to it. I'm not sure it's that simple, though. Interestingly, when I run the pipeline on my local computer (luigi --modules luigi-tasks Pipeline1 --local-scheduler) I get an appropriate return code (e.g. 30), but when I run the pipeline within the concourse server, I get a return code of 0 after the luigi tasks complete (from echo $? in the bash script).
Would appreciate any insight into this problem.
My suspicion is that luigi doesn't see your config file with return codes. Its default behavior is to return 0, whether tasks fail or succeed.
This experiment should help to debug that:
Force a failed job: add an exit 1 at the end of begin.sh
Hijack the job: fly -t <target> i -j <pipeline>/<job> -> select run-script
cd ./run-git; /bin/bash begin.sh
Ensure the luigi config is present and named appropriately, e.g. luigi.cfg
Re-run the command: LUIGI_CONFIG_PATH=luigi.cfg bash ./begin.sh
Check output: echo $?

Gitlab CI pipeline failing: a tag issue

My gitlab CI pipeline is setup to run maven tests from a docker image created from my maven project.
I have tested the pipeline on my master branch and it worked fine and ran the test.
However I have created a new feature branch and now running the pipeline yet again, however I now get this error
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: getting tag for destination: repository can only contain the runes `abcdefghijklmnopqrstuvwxyz0123456789_-./`: it2901/cs344-maven:feature/produce-allocation-pdf
ERROR: Job failed: command terminated with exit code 1
I can't seem to pinpoint the problem at all. I have also pushed the tag: tut3 to the feature branch as well.
Here is my .gitlab-ci.yml: https://controlc.com/7a94a00f
Based on what you shared, you have this configured:
VERSIONLABELMETHOD: "tut3" # options: "","LastVersionTagInGit"
It should be either:
VERSIONLABELMETHOD: ""
or
VERSIONLABELMETHOD: "LastVersionTagInGit"
or
VERSIONLABELMETHOD: "OnlyIfThisCommitHasVersion"
When you specify "tut3", the script takes it as if it was "" (empty string). Assuming you didn't define $VERSIONLABEL anywhere $ADDITIONALTAGLIST will also be empty.
And later in the code you can see that this gets executed:
if [[ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]]; then ADDITIONALTAGLIST="$ADDITIONALTAGLIST latest"; fi
Assuming $CI_DEFAULT_BRANCH is set to master if you use a separate branch mybranch the code above won't get executed so it's likely that the Kaniko command line doesn't have any a neither a valid $FORMATTEDTAGLIST or $IMAGE_LABELS.
You can debug by seeing their output on the script which is happening at the end before calling Kaniko:
...
echo $FORMATTEDTAGLIST
echo $IMAGE_LABELS
mkdir -p /kaniko/.docker
...
A hack would be to override $CI_DEFAULT_BRANCH with your custom branch.
✌️

unable to trigger job in concourse

I was new to concourse, and set up the environment in my centos7.6 like below.
$ wget https://concourse-ci.org/docker-compose.yml
$ docker-compose up -d
Then login by `fly --target example login --team-name main --concourse-url http://192.168.77.140:8080/ -u test -p test`
I can see below.
[root#centostest ~]# fly targets
name url team expiry
example http://192.168.77.140:8080 main Sun, 16 Jun 2019 02:23:48 UTC
I used below yaml.xml named with 2.yaml
---
resources:
- name: my-git-repo
type: git
source:
uri: https://github.com/ruanbekker/concourse-test
branch: basic-helloworld
jobs:
- name: hello-world-job
public: true
plan:
- get: my-git-repo
- task: task_print-hello-world
file: my-git-repo/ci/task-hello-world.yml
Then I run below commands step by step.
fly -t example sp -c 2.yaml -p pipeline-01
fly -t example up -p pipeline-01
fly -t example tj -j pipeline-01/hello-world-job --watch
But i just hang on there , no useful response like below.
[root#centostest ~]# fly -t example tj -j pipeline-01/hello-world-job --watch
started pipeline-01/hello-world-job #3
Theoretically, it should print something like below.
Cloning into '/tmp/build/get'...
Fetching HEAD
292c84b change task name
initializing
running echo hello world
hello world
succeeded
Where I did wrong? thanks.
welcome to Concourse!
One thing that can be confusing when starting with Concourse is understanding when Concourse detects that the pipeline has changed and what happens if the pipeline is one file or multiple files.
Your pipeline (as the majority of real-world pipelines) is "nested": main pipeline file 2.yaml refers to a task file named my-git-repo/ci/task-hello-world.yml
What sets Concourse apart from other CI systems is that:
the main pipeline file (2.yaml) can reside everywhere, also in a different repository.
Due to 1, Concourse is unable to detect a change to the main pipeline file, you have to tell Concourse that the file has changed, either with fly set-pipeline or with automatic means such as the concourse-pipeline-resource.
So the following errors happen often:
Changing the main pipeline file, committing and pushing, and expecting Concourse to pick up the change. Missing: you have to do fly set-pipeline
Once doing fly set-pipeline becomes second nature, you can stumble upon the opposite error: Change both the main pipeline file and the nested task file, not pushing, doing set-pipeline. In this case, the only changes picked up by Concourse will be the ones to the main pipeline file, not to the task file. Missing: commit and push.
From the description of your problem, I have the feeling that it is a mixture of the gotchas I mentioned.

Code coverage for Swift 3.0 project in Gitlab CI

I have an iOS project written by Swift 3.0 with CocoaPods. I've configured Gitlab CI for this project and it works perfectly. This is my .gitlab-ci.yml file:
stages:
- build
build_project:
stage: build
script:
- rm -rf Pods;rm -rf MyProject.xcworkspace;rm -rf MyProject.xcodeproj/project.xcworkspace
- pod install
- xcodebuild clean -workspace MyProject.xcworkspace -scheme MyProject | xcpretty
- xcodebuild test -workspace MyProject.xcworkspace -scheme MyProject -destination 'platform=iOS Simulator,name=iPhone 7,OS=10.2' | xcpretty -s
tags:
- ios
- lunchdot
I can't see code coverage for this project in my Gitlab repo. At the moment a coverage column for all my builds is empty. I tried to set Test coverage parsing in CI/CD Pipelines Gitlab settings, but it hadn't any effect because I don't know regex for Swift. Is it possible to set up code coverage for Swift project in Gitlab CI? How can I do this?
So, I tried a million ways to do it and the issue turned out to be xcpretty. After I removed it I started getting consistent results with my gitlab ci coverage data. If you still do not want to get tons of data you do not care about, you can use -quiet in your gitlab yams file. I will post it all though, so keep reading.
One still needs an external tool for coverage analysis - xcov seems to not be available anymore so I used slather. Worked like a charm.
These are the steps:
1) Get slather.
2) Get your .slather.yml file straightened out. Mine looks like the following (YourApp is the name of your app obviously):
# .slather.yml
coverage_service: cobertura_xml
xcodeproj: ./YourApp.xcodeproj
workspace: ./YourApp.xcworkspace
scheme: YourApp
source_directory: ./YourApp
output_directory: path/to/xml_report
ignore:
- "**/Pods/*"
- "thirdparty/*"
- "./Scripts/*"
- "**/3rdParty/*"
- "../**/Developer/*"
- "**/libraries/*"
- "Module-Core/*"
- "Module-Components/*"
- "**/AppUnitTests/*"
You can get the test output as html, xml, in codecov.io, etc, totally up to you. Check out the slather GitHub page to see the possible ways of doing that. But for the current issue, all we need is slather reporting in the command line so gitlab can pick it up. That is where properly setting up the gitlab yaml file comes in.
3) Set up the .gitlab-ci.yml file. Mine looks like this:
stages:
- build
build_project_1:
stage: build
script:
- xcodebuild clean -workspace YourApp.xcworkspace -scheme YourApp -quiet
- xcodebuild test -workspace YourApp.xcworkspace -scheme YourApp -destination 'platform=iOS Simulator,name=iPhone 6,OS=10.3.1' -enableCodeCoverage YES -quiet
after_script:
- slather coverage -s
only:
- master
4) Next step is to:
Go to your gitlab page/profile or whatever you call it
Go to Settings and then Pipelines
Scroll down to Test coverage parsing and add this regex expression there: \d+\%\s*$
And that is it. All you need to do is invoke a build.
Since Xcode 9.3 Apple provides xccov, so here is a solution using this, that does not rely on third party dependencies (and does not care if you use xcpretty, -quiet etc.)
Overview of xccovreport and xccov
When your Scheme has test coverage enabled (or when you pass -enableCodeCoverage YES to xcodebuild), an .xccovreport file is created. It contains the coverage percentages that you can see in Xcode UI.
The file is located in:
(for Xcode 9)
/Users/somename/Library/Developer/Xcode/DerivedData/MyApp-airjvkmhmywlmehdusimolqklzri/Logs/Test/E387E6E7-0AE8-4424-AFBA-EF9FX71A7E46.xccovreport
(for Xcode 10)
/Users/somename/Library/Developer/Xcode/DerivedData/MyApp-airjvkmhmywlmehdusimolqklzri/Logs/Test/Test-MyApp-2018.10.12_20-13-43-+0100.xcresult/action.xccovreport
(unless you specify a different folder in xcodebuild via -derivedDataPath)
Note: In Xcode 10, the .xccovreport file location is printed out in console after test finishes, but Xcode 9 does not do this. In any case, relying on this is probably not a good idea, as it might be silenced (e.g. by xcpretty)
The file isn't a plain text and to view it you have to call:
xcrun xccov view <path_to_xccovreport_file>
Which will output the report.
(You can pass --json for JSON report)
What we need to do
We want to be able to parse the file and print out the total percentage (so then GitLab can pick this up and use it in the dashboard).
There are a couple of challenges:
- Firstly we need to find out the file path of the xccovreport which contains random strings (in two places)
- Then we need to parse the file (using some regular expressions) and extract the total percentage.
The script
Here's what I am using (any improvement suggestions welcome, as I am not a bash expert):
#!/bin/bash
#1
# Xcode 10
TEST_LOGS_DIR=`xcodebuild -project MyApp.xcodeproj -showBuildSettings | grep BUILD_DIR | head -1 | perl -pe 's/\s+BUILD_DIR = //' | perl -pe 's/\/Build\/Products/\/Logs\/Test/'`
TEST_RESULTS_DIR=`ls -t $TEST_LOGS_DIR | grep "xcresult" | head -1`
TEST_COV_REPORT_FILENAME=`ls "$TEST_LOGS_DIR/$TEST_RESULTS_DIR/1_Test" | grep "xccovreport"`
TEST_COV_REPORT_FULL_PATH="$TEST_LOGS_DIR/$TEST_RESULTS_DIR/1_Test/$TEST_COV_REPORT_FILENAME"
# Xcode 9
# TEST_LOGS_DIR=`xcodebuild -project MyApp.xcodeproj -showBuildSettings | grep BUILD_DIR | head -1 | perl -pe 's/\s+BUILD_DIR = //' | perl -pe 's/\/Build\/Products/\/Logs\/Test/'`
# TEST_COV_REPORT_FILENAME=`ls $TEST_LOGS_DIR | grep "xccovreport"`
# TEST_COV_REPORT_FULL_PATH="$TEST_LOGS_DIR/$TEST_COV_REPORT_FILENAME"
# More general recursive search. Perhaps less likely to fail on new Xcode versions. Relies on filepaths containing timestamps that sort alphabetically correctly in time
# TEST_LOGS_DIR=`xcodebuild -project MyApp.xcodeproj -showBuildSettings | grep BUILD_DIR | head -1 | perl -pe 's/\s+BUILD_DIR = //' | perl -pe 's/\/Build\/Products/\/Logs\/Test/'`
# TEST_COV_REPORT_FULL_PATH=`find $TEST_LOGS_DIR -name '*.xccovreport' | sort -r | head -1`
#2
TOTAL_XCTEST_COVERAGE=`xcrun xccov view $TEST_COV_REPORT_FULL_PATH | grep '.app' | head -1 | perl -pe 's/.+?(\d+\.\d+%).+/\1/'`
#3
echo "TOTAL_XCTEST_COVERAGE=$TOTAL_XCTEST_COVERAGE"
What it does
#1 - gets the BUILD_DIR and then manipulates the path to get to the xccovreport file. Comment / uncomment the block for your version of Xcode.
#2 - We start with the full report as text. grep '.app' takes only the lines that contain .app. This is guaranteed to exist, because there is a line that reports the total coverage and contains MyApp.app. There will be multiple matches, but the first match will always be the overall total codecov score. So we use head -1 to take that first line of the grep result.
Now we have a line that looks like this:
MyApp.app 12.34% (8/65)
We use a perl regex to take only the “12.34%” part.
#3 - We simply print out the result (together with the variable name to make it easier to locate later in GitLab CI)
How to use
Replace MyApp.xcodeproj with your correct value
Make sure the correct logic is applied in step #1 (Xcode 9 / Xcode 10 / Generalized recursive search)
Save to a printCodeCov.sh file in the root of your project.
Make the file executable chmod +x printCodeCov.sh
In your .gitlab-ci.yml file, add a line to the script that says - ./printCodeCov.sh
In your GitLab Pipeline Settings, set the Test coverage parsing to TOTAL_XCTEST_COVERAGE=(.+)%
Notes
This does not use the --json format of xccov. For that version, see below.
This solution might be fragile, because of multiple assumptions about folder locations and report format
I use perl instead of sed because the latter was too difficult (BSD/GNU differences, regex limitations etc).
JSON version
If you'd rather use the JSON report (from xccov), then in the script you need something like this:
# Xcode 10
COV_NUMBER_FRACTION=`xcrun xccov view --json $TEST_COV_REPORT_FULL_PATH | perl -pe 's/.+?\.app.+?"lineCoverage":([^,]+).+/\1/'`
# Xcode 9
# COV_NUMBER_FRACTION=`xcrun xccov view --json $TEST_COV_REPORT_FULL_PATH | perl -pe 's/.+"targets"[^l]+lineCoverage":([^,]+),.+/\1/'`
COV_NUMBER_PERCENTAGE=`bc <<< $COV_NUMBER_FRACTION*100`
TOTAL_XCTEST_COVERAGE=`printf "%0.2f%%\n" $COV_NUMBER_PERCENTAGE`

How to fix "Test reports were found but none of them are new. Did tests run?" in Jenkins

I am getting the error "Test reports were found but none of them are new. Did tests run?" when trying to send unit test results by email. The reason is that I have a dedicated Jenkins job that imports the artifacts from a test job to itself, and sends the test results by email. The reason why I am doing this is because I don't want Jenkins to send all the developers email during the night :) so I am "post-poning" the email sending since Jenkins itself does not support delayed email notifications (sadly).
However, by the time the "send test results by email" job executes, the tests are hours old and I get the error as specified in the question title. Any ideas on how to get around this problem?
You could try updating the timestamps of the test reports as a build step ("Execute shell script"). E.g.
cd path/to/test/reports
touch *.xml
mvn clean test
via terminal or jenkins. This generates new tests reports.
The other answer that says cd path/to/test/reports touch *.xml didn't work for me, but mvn clean test yes.
Updating the last modified date can also be achieved in gradle itself is desired:
task jenkinsTest{
inputs.files test.outputs.files
doLast{
def timestamp = System.currentTimeMillis()
test.testResultsDir.eachFile { it.lastModified = timestamp }
}
}
build.dependsOn(jenkinsTest)
As mentioned here: http://www.practicalgradle.org/blog/2011/06/incremental-tests-with-jenkins/
Here's an updated version for Jenkinsfile (Declarative Pipeline):
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'make build'
}
}
stage('Test') {
steps {
sh 'make test'
script {
def testResults = findFiles(glob: 'build/reports/**/*.xml')
for(xml in testResults) {
touch xml.getPath()
}
}
}
}
}
post {
always {
archiveArtifacts artifacts: 'build/libs/**/*.jar', fingerprint: true
junit 'build/reports/**/*.xml'
}
}
}
Because gradle caches results from previous builds I ran into the same problem.
I fixed it by adding this line to my publish stage:
sh 'find . -name "TEST-*.xml" -exec touch {} \\;'
So my file is like this:
....
stage('Unit Tests') {
sh './gradlew test'
}
stage('Publish Results') {
// Fool Jenkins into thinking the tests results are new
sh 'find . -name "TEST-*.xml" -exec touch {} \\;'
junit '**/build/test-results/test/TEST-*.xml'
}
Had same issue for jobs running repeatedly (every 30 mins).
For the job, go to Configure, Build, Advanced and within the Switches section add:
--stacktrace
--continue
--rerun-tasks
This worked for me
Navigate to report directory cd /report_directory
Delete all older report rm *.xml
Add junit report_directory/*.xml in pipeline
Rerun the test script , navigate to Build Number → Test Result
Make sure you have one successful build without any failure, only after this you can able to see the reports
Make sure that you have mentioned the correct path against "Test report XMLs" under jenkins configuration, such as "target/surefire-reports/*.xml"
There is no need to touch *.xml as jenkins won't complain even though test results xml file does not change.
if you use Windows slave, you can 'touch' results using groovy pipeline stage with powershell:
powershell 'ls "junitreports\\*.*" | foreach-object { $_.LastWriteTime = Get-Date }'
It happens if you are using a test report which is not modified by that job in that run.
In case for test purpose if you are testing with already created file then, add below command inside jenkins job under Build > Execute Shell
chmod -R 775 /root/.jenkins/workspace/JmeterTest/output.xml
echo " " >> /root/.jenkins/workspace/JmeterTest/output.xml
Above command changes timestamp of file hence error wont display.
Note: To achieve same in Execute Shell instead of above, do not try renaming file using move mv command etc. it won't work , append and delete same for change file timestamp only works.
For me commands like chmod -R 775 test-results.xml or touch test-results.xml does not work due to permission error. As work around use is to set new file in test report settings and command to copy old xml report file to new file.
you can add following shell command to your "Pre Steps" section when configure your job on Jenkins
mvn clean test
this will clean the test
Here's an updated version of the gradle task that touch each test result files.
From Jenkins pipeline script, just call "testAndTouchTestResult" task instead of "test" task.
The code below is with Kotlin syntax:
tasks {
register("testAndTouchTestResult") {
setGroup("verification")
setDescription("touch Test Results for Jenkins")
inputs.files(test.get().outputs)
doLast {
val timestamp = System.currentTimeMillis()
fileTree(test.get().reports.junitXml.destination).forEach { f ->
f.setLastModified(timestamp)
}
}
}
}
The solution for me was delete node_modules and change node version (from 7.1 to 8.4) on jenkins. That's it.