I'm receiving daily "Jenkins build is unstable" emails for a project I'm no longer working on.
I can see from the build result (build.xml) that I'm in the list of "culprits", because of a commit I made some time ago, and since the build has been unstable for weeks, the list of culprits hasn't been cleared.
Since I'm no longer working on this project, I want to stop receiving email notifications from this Jenkins job. How can I remove myself from the recipients? Should I edit the build.xml of the last build? (EDIT: apparently this approach doesn't work)
It is possible, but a little complicated. First, you need to stop Jenkins or send it to quiet mode to avoid further builds.
As the actual list of culprits is calculated at runtime from all recently failed builds, you need to remove your ID from the culprits list in the build.xml of all failed builds (since the last successful one) where it is listed.
Finally restart Jenkins so the internal data structure is updated from the modified build.xmls.
So, if build 10 was the last successful one, you need to remove the ID from builds/11/build.xml, builds/12/build.xml, etc.
Could you just spam filter/block the email by title in your email client? Most Jenkins build emails will have a common title (with different only in status, build number). Builds from different jobs will have different common titles. So block the "offending" email by title.
To get Jenkins to 'release' you from the culprits, the build has to succeed. If you have access to the job configuration, you can make a fake success build (a simple "exit 0" in the command line will suffice), but this will clear all other culprits too.
Related
We have an in-company system for creating production builds, and I need to create a Jenkins job to automate those builds.
There is a way to automate the launch of production build. But you can only know the results of the build by an email notification. Furthermore, you can only subscribe to notification using your personal domain credentials, i.e. the same login/pass that you use to log into your computer (and the password changes every month).
So now, Jenkins job is considered "pass" as soon as the build launches, and not when it actually finishes.
I wonder how to make Jenkins pass/fail the job with the given restrictions.
Additionally, I want to avoid:
hardcoding my personal credentials anywhere,
depending on my or anyone else's personal computer being turned on.
Ideas?
I'm not aware of any Jenkins plugin that is capable of receiving e-mails.
If you can manage to store the notifications' content in a file you could use the
Text-finder Plugin in the Post-build Actions of an additional monitor job that runs on a regular basis.
Have a look at Monitoring external jobs, too, whether this is useful for you.
UPDATE:
There is actually a plugin that receives e-mails: the poll-mailbox-trigger-plugin. Install it and do the following:
Create a new Freestyle project named e.g. Build Status Mail Monitor
Build Triggers → [✔] [Poll Mailbox Trigger] - Poll an email inbox
configure as desired
Build → Add build step → Execute Groovy script
◉ Groovy command: add code to change build launcher job's status
Background:
We have one Jenkins job (Production) to build a deliverable every night. We have another job (ProductionPush) that pushes out the deliverable over a proprietary protocol to production machines the next day. This is because some production machines are only available during certain hours during the day (It also gives us a chance to fix any last-minute build breaks). ProductionPush needs access to the deliverable built by the Production job (so it needs access to the same workspace). We have multiple nodes and concurrent builds (and thus unpredictable workspaces) and prefer not to tie the jobs to a fixed node/workspace since resources are somewhat limited.
Questions:
How to make sure both jobs share the same workspace and ensure that ProductionPush runs at a fixed time the next day only if Production succeeds -- without fixing both jobs to run out of the same node/workspace? I know the Parameterized Trigger Plugin might help with some of this but it does not seem to have time delay capability and 12 hours seems too long for a quiet period.
Is sharing the workspace a bad idea?
Answer 2: Yes, sharing workspace is a bad idea. There is possibility of file locks. There is the issue of workspace being wiped out. Just don't do it...
Answer 1: What you need is to Archive the artifacts of the build. This way, the artifacts for a particular build (by build number) will always be available, regardless of whether another build is running or not, or what state the workspaces are in
To Archive the artifacts
In your build job, under Post-build Actions, select Archive the artifacts
Specify what artifacts to archive (you can use a combination of below)
a) You can archive all: *.*
b) You can archive a particular file with wildcards: /path/to/file_version*.zip
c) You can ignore the intermediate directories like: **/file_version*.zip
To avoid storage problems with many artifacts, on the top of configuration you can select Discard Old Builds, Click Advanced button, and play around with Days to keep artifacts and Max # of builds to keep with artifacts. Note that these two settings do not control for how long the actual builds are kept (other settings control that)
To access artifacts from Jenkins
In the build history, select any previous build you want.
In addition to SCM changes and revisions data, you will now have a Build Artifacts link, under which you will find all the artifacts for that particular build.
You can also access them with Jenkins' permalinks, for example
http://JENKINS_URL/job/JOB_NAME/lastSuccessfulBuild/artifact/ and then the name of the artifact.
To access artifacts from another job
I've extensively explained how to access previous artifacts from another deploy job (in your example, ProductionPush) over here:
How to promote a specific build number from another job in Jenkins?
If your requirements are to always deploy latest build to Production, you can skip the configuration of promotion in the above link. Just follow the steps for configuration of the deploy job. Once you have your deploy job, if it is always run at the same time, just configure its Build periodically parameters. Alternatively, you can have yet another job that will trigger the deploy job based on whatever conditions you want.
In either case above, if your Default Selector is set to Latest successful build (as explained in the link above), the latest build will be pushed to Production
Not sure archiving artifacts is really a good idea. A staging repository might be better as it enables cross-functional teams to share artifacts across different builds when required by tweaking the Maven settings.xml file.
You really want a deployable (ear/war) as the thing that gets built, tested, then promoted to production once confidence is high with the build.
Use a build number on your deployable (major.minor.buildnumber). This is the thing you promote to production, providing your tests can be relied upon. Don't use a hyphen to separate minor with build number as that forces Maven to perform a lexical comparison... a decimal point will force a numeric comparison which will give you far less headaches.
Also, you didn't mention your target platform, but using the Maven APT/RPM plugin to push an APT/RPM to a APT/YUM repo that's available to a production box (AFTER successful testing!) would be a good fit, as per industry standards?
Does anyone know of a Jenkins plugin or script that would send out emails when someone does a manual build? We currently have a scheduled daily build for our dev env, but once in a while we need to do a manual build and I would like Jenkins to send an email out when this happens to warn our team that a build is in progress. But I don't want Jenkins to send out emails for the daily build because I think we all can agree that we get too many system emails.
Thanks
You can use the extended email plugin to send an email at the start of a build.
Obviously this means you will have to have separate manual and daily build jobs, but it's the best solution I've been able to come up with so far. If you use downstream job chains, you can still chain the same jobs to the manual and scheduled build so that the workflow remains the same and you don't need duplicates of all jobs in the chain.
I use rake to build my project and one of the steps is running the unit, integration and fitnesse tests. If too many of these fail, I fail the rake script.
That part is working fine.
Unfortunately, after the build is failed, jenkins doesn't publish the html reports I generated from the unit, integration and fitnesse tests I generated, making it tad difficult to track down the failure reason.
Am I missing a configuration step to get the reports published?
Is Jenkins supposed to skip the post-build steps when the build fails?
It seems like it some for most of the plugins I am using.
You have to tell Jenkins which artifacts to archive in a post-build step (there is a check box under general 'Post-build actions' heading which is called 'Archive the Artifacts'). Important: the artifact path is determined relative to the workspace directory. Make sure that the option Discard all but the last successful/stable artifact to save disk space is not checked.
Finally figured it out, one of those I could have had a V8 moments...
I'm using a rake file to build and one of it's tasks is failing just before some reporting tasks that need to run in order to have the HTML pushed into the correct area to be published.
In a CI environment, what exactly is considered a broken build?
There are several answers I can imagine (any combination of compiles, tests pass, metrics are in range, documentation exists etc.) , but I am not sure which of these are cannoncial.
For example, just today it happened to me that I actually checked in all code changes but forgot to commit the Visual Studio project file, thus breaking the unit tests. (even though I literally triple checked my commit, as it's a public OSS project on google code).
I was easily able to fix this in under a minute after my first commit, but should I consider myself a buildbreaker now?
How do you configure your CI environment: Is every revision built or only the newest version after each complete build, or do you use time based checking for new revs?
Ideally, you have
automated script that is scheduled to run each night to build the application from source code.
Scripts to copy the binaries to a directory/set of directories from which another script can be run to deploy the application if it is running in your environment, or used to create deliverables for a customer.
Automated suite of tests that run and verify all components pass all tests.
Automated script that verifies that the build has been built correctly.
Automated script/monitoring system that sends out an alert if the verification script fails.
When an alert is generated by the above process, then that is considered "breaking the build".
But since procedures/processes can vary from company to company, there may be alternate definitions. In some places it may be breaking the unit tests. Others it may be checking in source code that results in the code being unabl to compile.
Breaking the build is commiting any changes that make it impossible (or possible but not smart to do) to deploy the project.
Fixing a broken build does not repair the broken build, but makes it possible to create a new non-broken build.
I configure my CI-server to create a minimal build on every commit, and create a maximum build every period of time. The period depends on the number of people working on the project (more people is more commits) and the build duration (You can run your unit test suite every time, but the 30min. acceptance test suite once or twice a day).
Breaking a build is preventing any user that relies on the same standard tools and set of code than your CI environnment to get a compiling and running system.
If your coworkers cannot compile the system when they're up to date, because some config is missing, the build is broken, isn't it ?
If you coworkers cannot be confident that unit tests pass, because one of them is flaky, the build is broken, isn'it ?
If you have automated performance tests, and you're project have to be optimized, I would go as far as too say that if you code doesn't run fast enough, you've broken the build (but that is arguable)
I would not be so strongly minded about code coverage, or other metrics for example.
Breaking the build can happen. CI is just there to make sure it doesn't happen too much on the day you're supposed to ship ;)
for us, we use the term "breaking the build" any time the test suite fails after a new commit.
so, in your instance, yes you would have broken the build (according to our company at least)
As long as you broke the build because of a simple human error (like forgetting to commit something) and as long as this is an exception and not the rule, I would say it's OK. As long as you take care to fix this quickly :)
On the other hand, if somebody breaks builds on a regular basis by not executing the complete build locally on his/her machine before committing, then this is a sign of an ill-disciplined team member who doesn't really care for other team members and for the development process.
My experience is that one effective way to make people be more careful is to set up your CI server to send an email when (and only when) the status of the build changes, with the "culprit" on the To: list and the rest of the team on the Cc:. I guess you can could call it a "shame factor" ;)
A broken build is anything that doesn't pass the automated test suite.
You're a build breaker. ;) That's not really a big deal as long as you fix it fast. The whole point of a CI environment is to catch bugs, not make people afraid to commit.
My company builds the tip of each branch that is either in production or next up for production. We do this on every commit and each day at 4am. We're using Mercurial, so commit here means pushing changes to the integration repo.