Basically I have inherited ShellCommand to overwrite evaluatecommand.
In evaluatecommand, I parse the log and find the actual maintainer of the package to send a mail notification.
Everything other than mailnotification does not work fine.
class CustomShellCommand(ShellCommand):
command = None
parser = None
haltOnFailure = True
buildername = ''
ci = None
def __init__(self,command, ci, buildername, **kwargs):
self.ci = ci
self.command = command
self.buildername = buildername
ShellCommand.__init__(self, **kwargs)
if len(self.command) > 0 and self.command[0] == 'make_isolated':
self.parser = ParseLog()
self.addLogObserver('stdio', self.parser)
self.setDefaultWorkdir("build")
def evaluateCommand(self, cmd):
if self.parser is not None:
self.parser.packages
for pkg in self.parser.packages:
emails = get_maintainer_emails()
if cmd.rc > 0:
mn = add_mail_notifiers([self.buildername], emails[-1])
self.ci.masterconfig['services'].append(mn)
return util.FAILURE
else:
return util.SUCCESS
But when I add mail notifiers in init it works, but does not work in evaluate command.
Any pointers would be appreciated.
I am no buildbot expert, I just started using it 2 months ago in my new job. But here I think the MailNotifier is something related to the master, and more precisely to the config. For your ShellCommand, I suppose the master executes the __init__ when it loads its config. But the evaluateCommand I think is only executed on runtime by the slave, and they cannot change the config of the master...
Here we have written an external script to send personalized mail for failed builds. It has a builder which triggers it once a day, early in the morning after the nightly builds have finished and before people arrive at the office. We will investigate how to do this more generally as only one of our project has this feature, the other projects' failures are summarized in a general mail sent to everyone. Maybe there is something to do with the SetProperty, but I cannot tell for now...
Related
I am trying to read a file from github using readFile,
sshagent (credentials: ["${github_cred}"]) {
script {
sh """
git checkout test_branch
"""
def file = readFile(file: "/myrepo/${params.value}.txt")
But in other case, this file will not be available for certain parameters passed. So I would like to check if the file exists in github and proceed with the next steps if it is available or else it should skip the stage.
First Try:
When I try to do with the above code, it is throwing NoSuchFileException when it is not available. I tried with fileExists function which actually works only on the controller node, not on the github. Is there any possible to achieve this?
Second Try:
I also tried with git show as below but I got illegal string body or character after dollar sign error, I don't know what is wrong here.
git show HEAD~1:/myrepo/"${params.value}".txt > /dev/null 2>&1
The fileExists should run in the current node the Pipeline is running on. So it should ideally work.
if (fileExists("/myrepo/${params.value}.txt") {
def file = readFile(file: "/myrepo/${params.value}.txt")
}
Another easy workaround is to wrap your readFile with a try-cath given you know readFile will fail if the file is not available.
def isSuccess = true
def file = null
try {
file = readFile(file: "/myrepo/${params.value}.txt")
}catch(e) {
isSuccess = false
}
The Jenkins machine is windows or macos or linux?
sh """
pwd
git checkout test_branch
"""
In case it's linux or macos add pwd to see the local full path of the repo
And then use this full path in the readfile
I worked with bugzilla and Eclipse, and I used Mylyn to manage issues though Eclipse.
Now I use Gitlab and gitlab issues, I wonder if there is a mylyn connector for Gitlab ?
I knwow that there is this one : gitlab connector , but it is no more usable and I did not found another one.
Did someone face with the same problem and did find a solution ?
After a while I can share my solution, maybe it will help others.
There is no Mylin connector for Gitlab that runs correctly. A solution could be to debug the buggy one but in fact Gitlab is not a powerfull tool to manage issues.
I chose to use Bugzilla at least for three points :
the bug workflow is easy to customize and this is an important feature to adapt the bug workflow to the company processes
Mylin connector for Bugzilla is avalaible since a long time and runs correctly
Bugzilla is still a reference tool
The first step is to define Bugzilla as the issues management tool, this is done through Gitlab UI and the documentation is here.
For me, if an external tool is used, the best is to desactivate Gitlab issues tracking. On your project, go to Settings->General->Visibility, project features and desactivate Issues.
Note: if Bugzilla and Gitlab are deployed on the same host, you have to accept request to localhost. On Gitlab administration, go to Settings->Network->Outbound requests, select the two options about local network.
After that, you can comment your commits with a message containing Ref #id where id is a bug id in Bugzilla. As with Gitlab issue, the commit will contain an hyperlink to the issue but the hyperlink will open Bugzilla bug page.
If you do not go further, you will lost a Gitlab feature : Gitlab issue references all commits related to it.
A solution to have a similar feature with Bugzilla is to add to bug a comment with an hyperlink to commits.
This could be achieve with a server hook, this is described here.
Note : each time you change the gitlab.rb file, do no forget to execute gitlab-ctl reconfigure.
The hook has to manage "standard" commit and merge commits.
The following python code could be seen as a starting point for a such hook.
It assumes that development are done on branches named feature/id nd that commits comments contains a string Ref #id. Id is a bug id.
It could be improve:
to manage exceptions better
to manage more push cases
to check more rules such as :
the bug has to be in progess
the bug assignee has to be the git user who performs the push
the bugzilla project has to be the one for the Gitlab project
the bug is open on a version that is still under development or debug
....
#!/usr/bin/env python3
import sys
import os
import fileinput
import glob
import subprocess
import requests
#
# Constants
#
G__GIT_CMD =["git","rev-list","--pretty"]
G__SEP ="commit "
G__NL ='\n'
G__AUTHOR ='Author'
G__AUTHOR_R ='Author: '
G__DATE ='Date'
G__DATE_R ='Date: '
G__C_MSG ='message'
G__URL_S ='https://<<gitlab server url>>/<<project>>/-/commit/'
G__MERGE_S ='Merge: '
G__MERGE ='Merge'
G__URL ='URL'
G__BUGZ_URL ='http://<<bugzilla url>>/rest/bug/{}/comment'
G__HEADERS = {'Content-type': 'application/json'}
G__JSON = {"Bugzilla_login":"<<bugzilla user>>","Bugzilla_password":"<<password>>","comment": "{}"}
G__JSON_MR = {"Bugzilla_login":"<<bugzilla user>>","Bugzilla_password":"<<password>>","comment": "Merge request {}"}
G__COMMENT_ELEM = 'comment'
G__MSG_REF ="Ref #"
G__MSG_REF_MR ="feature/"
G__WHITE =" "
G__APOS ="'"
#
# Filters some parts of message that are empty
#
def filter_message_elements(message_elements):
flag=False
for message_element in message_elements:
if len(message_element)!=0:
flag=True
return flag
#
# Add an element in commit dictionary.
#
# If this is a commit for a merge, an element is added.
#
def add_commit_in_dict(commits_dict, temp_list, flag_merge):
url = G__URL_S+temp_list[0]
commits_dict[temp_list[0]]={}
commits_dict[temp_list[0]][G__URL]=url
if False==flag_merge:
commits_dict[temp_list[0]][G__AUTHOR]=temp_list[1].replace(G__AUTHOR_R,'')
commits_dict[temp_list[0]][G__DATE]=temp_list[2].replace(G__DATE_R,'')
commits_dict[temp_list[0]][G__C_MSG]=temp_list[3]
else:
commits_dict[temp_list[0]][G__MERGE]=temp_list[1]
commits_dict[temp_list[0]][G__AUTHOR]=temp_list[2].replace(G__AUTHOR_R,'')
commits_dict[temp_list[0]][G__DATE]=temp_list[3].replace(G__DATE_R,'')
commits_dict[temp_list[0]][G__C_MSG]=temp_list[4]
#
# Fill commits data
#
def fills_commit_data(commits_dict, fileinput_line):
params=fileinput_line[:-1].split()
try:
# Git command to get commits list
cmd=G__GIT_CMD+[params[1],"^"+params[0]]
rev_message = subprocess.run(cmd,stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
# loop on commits
messages_list=rev_message.stdout.split(G__SEP)
for message in messages_list:
if len(message)==0:
continue
message_elements = message.split(G__NL)
# filters empty message
flag=filter_message_elements(message_elements)
if not flag:
continue
# Extracts commit data and detects merge commit
temp_list=[]
flag_merge=False
for message_element in message_elements:
text = message_element.strip()
if 0!=len(text):
temp_list.append(text)
if -1!=text.find(G__MERGE):
flag_merge=True
# adds the commit in commits dictionary
add_commit_in_dict(commits_dict, temp_list, flag_merge)
except Exception as inst:
sys.exit(1)
#
# Extract the bug id from the commit message
#
def find_bug_id(message):
issue_int=-1
pos=message.find(G__MSG_REF)
if pos==-1:
sys.exit(1)
issue_nb=message[pos+len(G__MSG_REF):]
pos2=issue_nb.find(G__WHITE)
issue_nb=issue_nb[:pos2]
try:
issue_int=int(issue_nb)
except ValueError:
sys.exit(1)
return(issue_int)
#
# Extract the bug id from the commit message
# in case of merge request
#
def find_bug_id_mr(message):
issue_int=-1
pos=message.find(G__MSG_REF_MR)
if pos==-1:
sys.exit(1)
issue_nb=message[pos+len(G__MSG_REF_MR):]
pos2=issue_nb.find(G__APOS)
issue_nb=issue_nb[:pos2]
try:
issue_int=int(issue_nb)
except ValueError:
sys.exit(1)
return(issue_int)
#
# Checks if the commit list contains a merge request commit
#
def is_merge_request(commits_dict):
flag=False
for key in commits_dict:
if G__MERGE in commits_dict[key]:
flag=True
break
return flag
#
# Add a comment to a bug
#
def add_comment_to_bug( commit_data):
bug_id = find_bug_id(commit_data[G__C_MSG])
url = G__BUGZ_URL.format(str(bug_id))
G__JSON[G__COMMENT_ELEM] = G__JSON[G__COMMENT_ELEM].format(commit_data[G__URL])
response = requests.post(url, json=G__JSON, headers=G__HEADERS)
#
# add a comment in case of merge request
#
def add_mr_comment_to_bug(commits_dict):
commit_data=None
for key in commits_dict:
if G__MERGE in commits_dict[key]:
commit_data=commits_dict[key]
break
bug_id = find_bug_id_mr(commit_data[G__C_MSG])
url = G__BUGZ_URL.format(str(bug_id))
G__JSON_MR[G__COMMENT_ELEM] = G__JSON_MR[G__COMMENT_ELEM].format(commit_data[G__URL])
response = requests.post(url, json=G__JSON_MR, headers=G__HEADERS)
#
# Main program
#
def main():
# dictionary containing all commits
commits_dict={}
# loop on inputs referencing data changes
for fileinput_line in sys.stdin:
fills_commit_data(commits_dict, fileinput_line)
# find if this is merge request or not
flag_merge_request = is_merge_request(commits_dict)
if False==flag_merge_request:
# loop on commit to add comments to bugs
for key in commits_dict.keys():
add_comment_to_bug(commits_dict[key])
else:
# in case of merge request, only the merge commit has to be added
# others commits have been processed before
add_mr_comment_to_bug(commits_dict)
if __name__ == "__main__":
main()
I have a jenkins organization pipeline job that executes on all repositories that have "Jenkinsfile" defined. The job clones the repository from github, then runs the powershell script that increments the version number in the file. I'm now trying to publish that updated file back to the original repository on github, so when developer pulls the changes he gets the latest version number.
I tried using the script (inside "jenkinsfile") as suggested in jenkins JIRA (https://issues.jenkins-ci.org/browse/JENKINS-28335), but to no avail. Any suggestions will be appreciated. Basically need to execute "git commit" and "git push" using the same parameters defined for a job.
Just as a reference, here is a previous solution used for free style (not pipeline job): How to push changes to github after jenkins build completes?.
Actually found couple solutions, first I modied script from Jenkins like this (some objects changed in workflow pipeline):
import hudson.FilePath
import org.eclipse.jgit.transport.URIish
node {
env.WORKSPACE = pwd()
stage 'Checkout'
checkout scm
def build = manager.build
def listener = manager.listener
def workspace = new FilePath(new File(env.WORKSPACE))
def environment = build.getEnvironment(listener)
final def project = build.getParent()
final def gitScm = project.getTypicalSCM()
final def gitClient = gitScm.createClient(listener, environment, build, workspace);
final def gitTagName = "TAG_NAME"
final def comment = "COMMENT"
final def remoteURI = new URIish("origin")
gitClient.tag(gitTagName, comment)
gitClient.push().tags(true).to(remoteURI).execute()
}
You need to run the script multiple times and then allow code execution in jenkins (manage jenkins->in process script approval).
Another solution, much simpler (using this one for now):
bat "\"${tool 'Git'}\" config user.email \"ci#virtocommerce.com\""
bat "\"${tool 'Git'}\" config user.name \"Virto Jenkins\""
bat "\"${tool 'Git'}\" commit -am \"Updated version number\""
bat "\"${tool 'Git'}\" push origin HEAD:master -f"
You have to have Git tool with a name "Git" configured in Jenkins.
I published a repository using the version control program mercurial. More specifically, I installed the mercurial-server program and followed the instructions as given at
http://ekkescorner.wordpress.com/blog-series/git-mercurial/step-by-step-install-mercurial-server-on-ubuntu/
and
http://dev.lshift.net/paul/mercurial-server/docbook.html
. The repository works great, but the problem is that there are no emails sent to the users when we do an hg push after we made some changes.
I found several pages on the internet describing how to supposedly solve this problem, but none of them used mercurial-server to publish the repository and the activation of sending emails with an hg push seems to differ per method of publication.
How do I activate the emailing with an hg push in case of the mercurial-server package? Do I need to setup a SMTP server for mercurial?
The hgrc file that is in the .hg folder of my repository directory is as follows.
[paths]
default = ssh://hg#<ip_address>/jays/project
[extensions]
hgext.notify =
[hooks]
# Enable either changegroup or incoming.
# changegroup will send one email for each push,
# whereas incoming sends one email per changeset.
# Note: Configuring both is possible, but probably not
# what you want, you'll get one email for the group
# and one for each changeset in the group.
changegroup.notify = python:hgext.notify.hook
#commit.notify = python:hgext.notify.hook
#incoming.notify = python:hgext.notify.hook
[email]
from = <from#email.address>
[smtp]
host = <smtp.server.address>
# Optional options:
username = <smtp.server.username>
password = <smtp.server.password>
port = 465
tls = true
# local_hostname = me.example.com
# presently it is necessary to specify the baseurl for the notify
# extension to work. It can be a dummy value if your repo isn't
# available via http
[web]
baseurl = file:///
[notify]
# multiple sources can be specified as a whitespace separated list
sources = serve push pull bundle
# set this to False when you're ready for mail to start sending
test = false
# While the subscription information can be included in this file,
# (in which case, set: config =)
# having it in a separate file allows for it to be version controlled
# and for the option of having subscribers maintain it themselves.
config =
# you can override the changeset template here, if you want.
# If it doesn't start with \n it may confuse the email parser.
# here's an example that makes the changeset template look more like hg log:
#template = \ndetails: {baseurl}{webroot}/rev/{node|short}\nchangeset: {rev}:{node|short}\nuser: {author}\ndate: {date|date}\ndescription:\n{desc}\n
template = \ndetails: "{baseurl}{webroot}"\nchangeset: {rev}:{node|short}\nuser: {author}\ndate: {date|date}\ndescription:\n{desc}\n
# max lines of diffs to include (0=none, -1=all)
maxdiff = 1000
[reposubs]
* = <first#recipient.address>
I am getting the error "Test reports were found but none of them are new. Did tests run?" when trying to send unit test results by email. The reason is that I have a dedicated Jenkins job that imports the artifacts from a test job to itself, and sends the test results by email. The reason why I am doing this is because I don't want Jenkins to send all the developers email during the night :) so I am "post-poning" the email sending since Jenkins itself does not support delayed email notifications (sadly).
However, by the time the "send test results by email" job executes, the tests are hours old and I get the error as specified in the question title. Any ideas on how to get around this problem?
You could try updating the timestamps of the test reports as a build step ("Execute shell script"). E.g.
cd path/to/test/reports
touch *.xml
mvn clean test
via terminal or jenkins. This generates new tests reports.
The other answer that says cd path/to/test/reports touch *.xml didn't work for me, but mvn clean test yes.
Updating the last modified date can also be achieved in gradle itself is desired:
task jenkinsTest{
inputs.files test.outputs.files
doLast{
def timestamp = System.currentTimeMillis()
test.testResultsDir.eachFile { it.lastModified = timestamp }
}
}
build.dependsOn(jenkinsTest)
As mentioned here: http://www.practicalgradle.org/blog/2011/06/incremental-tests-with-jenkins/
Here's an updated version for Jenkinsfile (Declarative Pipeline):
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'make build'
}
}
stage('Test') {
steps {
sh 'make test'
script {
def testResults = findFiles(glob: 'build/reports/**/*.xml')
for(xml in testResults) {
touch xml.getPath()
}
}
}
}
}
post {
always {
archiveArtifacts artifacts: 'build/libs/**/*.jar', fingerprint: true
junit 'build/reports/**/*.xml'
}
}
}
Because gradle caches results from previous builds I ran into the same problem.
I fixed it by adding this line to my publish stage:
sh 'find . -name "TEST-*.xml" -exec touch {} \\;'
So my file is like this:
....
stage('Unit Tests') {
sh './gradlew test'
}
stage('Publish Results') {
// Fool Jenkins into thinking the tests results are new
sh 'find . -name "TEST-*.xml" -exec touch {} \\;'
junit '**/build/test-results/test/TEST-*.xml'
}
Had same issue for jobs running repeatedly (every 30 mins).
For the job, go to Configure, Build, Advanced and within the Switches section add:
--stacktrace
--continue
--rerun-tasks
This worked for me
Navigate to report directory cd /report_directory
Delete all older report rm *.xml
Add junit report_directory/*.xml in pipeline
Rerun the test script , navigate to Build Number → Test Result
Make sure you have one successful build without any failure, only after this you can able to see the reports
Make sure that you have mentioned the correct path against "Test report XMLs" under jenkins configuration, such as "target/surefire-reports/*.xml"
There is no need to touch *.xml as jenkins won't complain even though test results xml file does not change.
if you use Windows slave, you can 'touch' results using groovy pipeline stage with powershell:
powershell 'ls "junitreports\\*.*" | foreach-object { $_.LastWriteTime = Get-Date }'
It happens if you are using a test report which is not modified by that job in that run.
In case for test purpose if you are testing with already created file then, add below command inside jenkins job under Build > Execute Shell
chmod -R 775 /root/.jenkins/workspace/JmeterTest/output.xml
echo " " >> /root/.jenkins/workspace/JmeterTest/output.xml
Above command changes timestamp of file hence error wont display.
Note: To achieve same in Execute Shell instead of above, do not try renaming file using move mv command etc. it won't work , append and delete same for change file timestamp only works.
For me commands like chmod -R 775 test-results.xml or touch test-results.xml does not work due to permission error. As work around use is to set new file in test report settings and command to copy old xml report file to new file.
you can add following shell command to your "Pre Steps" section when configure your job on Jenkins
mvn clean test
this will clean the test
Here's an updated version of the gradle task that touch each test result files.
From Jenkins pipeline script, just call "testAndTouchTestResult" task instead of "test" task.
The code below is with Kotlin syntax:
tasks {
register("testAndTouchTestResult") {
setGroup("verification")
setDescription("touch Test Results for Jenkins")
inputs.files(test.get().outputs)
doLast {
val timestamp = System.currentTimeMillis()
fileTree(test.get().reports.junitXml.destination).forEach { f ->
f.setLastModified(timestamp)
}
}
}
}
The solution for me was delete node_modules and change node version (from 7.1 to 8.4) on jenkins. That's it.