I am modifying a concourse script I inherited.
We have a pipeline that looks like this:
[1]build - [2]test A - [3]test B - [4]publish - [5]deploy
Our team worked only with the master branch before but now we are using feature branches.
I made a modification to build step to run for every branch - not only master. But I want publish step to run only for the master branch builds.
How can I make step 5 to run only when the step 1 was run for the master branch? Is there a way of doing this on Concourse?
If the same pipeline builds all the branches, then either you are not using the git resource or you are using a modified git resource that supports multiple branches.
I assume that what you call "steps" 1...5 are Concourse jobs.
I can see two options.
Option 1. Simple but dirty/misleading.
At the beginning of steps 4 and 5, check for the git branch. If not master, return success immediately. The pipeline will stay green and the publish and deploy jobs will look like they have ran, while actually they did nothing.
Option 2. Slightly more complicated but the proper way.
Use two pipelines. One pipeline for all branches except master. This pipeline will contain only steps 1 ... 3. The second pipeline will track the master branch and have all the steps 1 ... 5.
The question now is how to manage these 2 pipelines in a DRY fashion (no code duplication, no possibility for the pipelines to drift away one from the other)?
The answer is to use a YAML merge tool like https://github.com/geofffranks/spruce. The README mentions BOSH but it is a completely generic YAML merge tool.
The last part is how to stay DRY and at the same time specify different branches. This can be obtained with https://concourse-ci.org/setting-pipelines.html#pipeline-vars.
Related
I have a ADO git repo with a YAML-based build pipeline that builds a docker image, runs some tests using Docker Compose, and then pushes the image and a helm chart to an ACR.
I'd like the have the build/test part run on PRs too. I have created a second pipeline that's just the first half of the normal build, and assigned it as a Build Validation pipeline for a branch.
However, I don't seem to be able to get the triggers right. If I have trigger: none in the test pipeline, it never triggers. If I have branch names, it is also run on merge alongside the normal build pipeline. How is this supposed to work? The docs define all the individual parts, but not really how they are expected to interact!
Am I supposed to have a multistage pipeline and use that somehow for the validation? (it's just 4 Steps in one Job in one Stage currently).
I am hoping to avoid building the same image too many times, or storing untested images anywhere outside the build agent.
I make it work with the following configuration:
In my YAML pipeline, I set the trigger: none.
In branch policies of a branch, I create a build validation with automatic trigger:
Then I create a pull request to that branch, and the pipeline runs automatically:
There are two possible mistakes:
The "Manual" trigger is selected in build validation, so that the pipeline needs to be run manually rather than triggered automatically.
The branch with branch policy set should be the same branch as the target branch of pull request.
I'm facing the following issue: I have one git repo with a Node.js application. The application is divided into several components, namely: server, client, microserviceA, microserviceB. There is also a directory named shared with some sharaed code used by all the other components.
I have a pipeline for each of the components that only runs tests, which run on pull-request to master. Each pipeline only runs when the PR contains changes relevant to him, e.g. server-ci will run only when there were changes in the server component, etc.
Now, on merge to master, I would like to build the components and deploy them on a staging server. Currently what I have is as follows: for each component (beside the shared) I have another build pipeline (<component>-build) which on merge to master builds the corresponding component (depending on the changes made, as above). I have one Release pipeline which takes as artifacts all these build pipelines and deploys them on the staging server. So the good thing about this is that merge to master which includes only changes in client will build only client and not all the rest of the components.
The problem is, that on merge to master that contains changes to several components, I'll have more than one build pipeline running, so they will both trigger the Release pipeline. This is not desirable.
A possible solution I thought about was, to have only one build pipeline which runs on merge to master, but then I'd have to build ALL the components on each merge, which is inefficient.
What is the best way to deal with such situation?
In the release stage settings you can configure that Number of parallel deployments will be 1:
TL;DR
How do I avoid rebuilding artifacts on master when a feature is merged without creating multiple pipelines per project? Where do I access the information about which branch was merged?
More Info
I run Jenkins to build many projects stored in two different VCSs (Gitlab, Bitbucket). Auto-discovery for both VCSs work and create multi-branch pipelines for every project/branch/PR containing a Jenkinsfile (Gitlab Branch Source Plugin, Bitbucket Branch Source Plugin).
Build artifacts get produced and stored on every build (e.g. docker images pushed to registry).
As I follow a feature branch workflow, these features get eventually merged into master, master will then be deployed in irregular intervals.
When doing the merge, there is an artifact already built and stored for this code(see appendix:1). It was built for the feature branch the code originated from (e.g. container mysuperapp:feat-add-better-things-3). I would like to take this artifact and promote it as the new master artifact (e.g. mysuperapp:master), avoiding a rebuild (and unit + integration testing everything).
But, merging a feature branch just kicks off a new build pipeline on branch master without any information about the merged branch (see appendix:2). This is correct behavior concerning master (new commit(s) where pushed) but prevents me from reacting to the merged branch (e.g. the aforementioned promoting or even just deleting unused artifacts). Is there any way to get the information, which branch was merged?
I am aware, that I can create a new pipeline listening for PR webhooks from my VCSs, running a pipeline to do the promotion and ignore builds on master completely. But this moves visibility of this process to a different pipeline and requires additional pipelines for projects, e.g. reducing the advantage of auto-discovery to 50% (have to create these merge pipelines for each project).
How can I keep the advantages of auto-discovery and visibility of executed steps while also executing something on a merge?
Ideas: Tag artifacts differently, but how (needs to be able to clean up correctly)? Parameterize pipelines and setup a single merge pipeline which re-triggers the pipeline 'push on master' with parameters of the merged branch. But can this be done without having to setup the webhooks for every project? Ask the VCSs via REST about which branch belonged to a commit?
Greets and thanks for the help you all! This may be a complicated one, but it would be so cool to get this to work. It's the last barrier for me to enable continuos delivery for a lot of projects!
Appendix:
1: I am also aware, that to have consistent builds, I have to enforce --ff-only merges. This question is not about the pitfalls of git but rather about the way to go with Jenkins.
2: Git provides me with the parent commits, I can easily find out, which commit was merged. But, especially using "Delete branch after merge", leaves me without the branch ref in git. Tagging my docker images with commits instead of branches leaves me with backtracking the last commit on each build to delete the old, obsolete build.
The final stage of our release pipeline is a manual stage used to confirm the deployed release got its final acceptance. Among the tasks we would like to run in this stage:
Tag the develop branch with the release label. Say "1.2.3".
Merge the develop branch into the master branch.
(We're using Azure Git repositories)
Although it looks like the right moment to make these changes in Git, I'm not quite certain this is the intended usage of Azure release pipelines. I confess being a bit new to Azure pipelines and there seem to be no evident pipeline task for doing such changes.
However, I believe this kind of post-release SCM changes is quite common.
My question is therefore: Where and how is the proper way to apply those SCM changes in Azure Devops ?
EDIT: I could make it work to use a command line task to run git commands. Config was passed by means of a variable group.
You can just do the regular git commands in a Command Line script (first , clone the repo (or add it as an artifact), then tag/merge).
Or install the Tag Git on Release & Git Merge extensions and use them.
I am using TravisCI's Build Stages to separate my job to two stages:
Build and test on multiple environments.
Build and deploy if stage 1 pass.
And I want to Travis to run the jobs on commits to two GitHub branches, master and dev, and pull requests to master and dev. However, I only want to run the stage two when a commit to master happens.
Is there a way to completely ignore stage 2 in commits to branches that are not master, and on pull requests?
This is what my .travis.yml looks like at the time of writing this:
https://github.com/SammyIsra/photostream-react/blob/c354a62c3cc963b345a5c2fb95658c90ddc39d21/.travis.yml
Update:
this seems to not be possible as of yet. However, the TravisCI team may be working on something like that, as of this comment on the Build Stages feedback board. Whenever I learn that it was added as a feature, or that it will definitely not be possible ever, I will change this question.
There doesn't seem to be a way to ignore specific branches when triggering jobs in the build stages section, however it looks like there's a way to only fire your deployments on pushes to master.
In your build stage script section, you could wrap the npm run build command in a quick Bash if statement to test the environment variable which shows what branch you're on:
if [ TRAVIS_BRANCH == "master" ]; then npm run build; fi
Then, in your Surge deploy section, you can restrict the deployment to a particular branch with:
deploy:
...
on: master
And basically the same for the NPM deployment:
deploy:
...
on:
branch: master
Note: I've not tried any of this myself, so it may or may not work in combination with the new build stages functionality.