I use Bamboo CI to manage my git repo for Salesforce development. It runs my tests after each commit and if successful, then deploys to an org. This all works wonderfully except for when I've deleted files, because Salesforce needs a special xml built-up in order to remove them from the org. So I have added a step to my build process that will diff the repo (from previous successful build to HEAD), extract a list of any files which have been removed from git since the last build, and parse that information info into the appropriate xml format.
The Question: What I need now is a way to set up the deployment job such that it will be able to tell me the revision number of the previous successful deployment. Is there a way to have a script task set a value on a Bamboo global variable? If so, maybe I could just have a task which runs at the end of deployment record the current revision number and have it available to the next job?
Note: The deployments only run after builds are stable, meaning that there might be many commits (and builds) between last run and current run. Bamboo does provide a system variable for current revision number ($Bamboo.planRepository.revision) but the analogous system variable for previous revision number ($Bamboo.planRepository.previousRevision) is available in build environments only, with no such luck for deployment environments.
Related
We have a release definition which delivers a bunch of asp.net core services along with an Angular app.
Most service are not updated very often so the question is how to compare an artifact version with already deployed into an environment and skip if the latest version had been deployed before?
We have multiple environments in the pipeline.
I dont think it is possible, at least natively, you can calculate file hashes and dont deploy if they match, another option would be using path triggers to filter when an app is build. for example, your directory structure looks like this:
root
|--app1
|--app2
etc
you can define path filters in your yaml build like this:
trigger:
paths:
include:
- app1/*
- sharedlibs/* (if you have them)
this way build will only trigger if there are any changes to files in those directories
You can add additional release environment to check current artifact version through PowerShell (e.g. Build.SourceVersion, check variables in release), then fail task if there was already successfully released.
For Staging environment, choose After environment option and select previous environment.
On the other hand, since you have mentioned most service are not updated very often, you could use 4c74356b41's suggestion to filter build, to only build and release the changes you want.
I have been researching and am trying to figure out the best branching and deployment strategy to accomplish the requirements below. Maybe I’m missing something but it is more complicated than it seems. Ideally, we’d just have one permanent branch, ‘master’, that could have specific commits tagged to mark releases to production.
Our current strategy is based on Git Flow and has permanent branches ‘master’ (only has releases to production) and ‘develop’. The primary thing that complicates using a multiple permanent-branches model is the concept of “promoting” the same build from the staging environment to production. Currently, this needs to be done in a separate source code branch (deployments to staging come from ‘develop’, deployments to prod come from ‘master’).
Tools: Git (VSTS), TeamCity, Octopus Deploy
Requirements (feature and hotfix lifecycles):
All code is reviewed via pull requests (enforced via branch policies)
All code gets deployed to a staging environment for testing
We can quickly go back to any snapshot of code that was deployed previously
If testing is successful, then the same build can be “promoted” from our staging environment to production (no need to build again)
Features accumulate over time before pushing out to production as a single release. Hotfixes have to be able to go through without getting caught up in the "all or nothing" next regular release.
I like the idea of having one permanent branch with tags (re: The master/develop split is redundant, http://endoflineblog.com/gitflow-considered-harmful), but having additional permanent branches may better facilitate deploying to different lifecycles/versions (feature and hotfix) to Octopus.
I have been wrestling with how best to pull this off and I may be over complicating things. Any feedback is appreciated.
It seems you have a number of questions and they are quite broad... I'll add some comments to each of your requirements as a conversation starter, but this whole thread might get blocked by moderators as it is definitely not the style of questions SO was made for.
All code is reviewed via pull requests (enforced via branch policies)
I haven't looked at VSTS for ages, but I'd expect they already support branch policies and pull-requests, so not sure if there's anything you need here other than configure settings in your repositories.
In case VSTS does not support that, you might consider moving to a tool that does e.g. BitBucket, GitHub, etc. Both of these have an on-premises version in case you can't (or don't want to) use the cloud hosted version.
All code gets deployed to a staging environment for testing
You achieve that with setting up lifecycles in Octopus Deploy, to make sure deployments/promotions follow the the sequence you want.
We can quickly go back to any snapshot of code that was deployed previously
You already have source control, so all you need now is traceability from the code that is deployed in an environment, to the deployment version in Octopus Deploy, the build job in TeamCity, the branch and exact commit in your source control.
There's a few things that you can do, to achieve that:
Define a versioning scheme that works for you. I like to use semantic versioning. "Major" and "Minor" versions are defined by the developers, and the "Patch" is the auto-incremented number from TeamCity (%build.number%). Every git push build the code and generates a unique build version (%major%.%minor%.%build.number%)
As part of the build steps in TeamCity, before you compile the code, make sure your source files are patched with the version number assigned by each build, the commit hash from your source control, and the branch name. e.g. if you are using .NET, make sure all the AssemblyInfo.cs files are updated with that version, so that the version is embedded in the binaries. This allows anyone to query the version looking at the properties of the binary files, and also allows you to display the app version on the app itself (e.g. status bar, footer, caption, about box, etc.)
Have TeamCity tag your source control with the version number of every build, so you can quickly see on your source control history. You probably only want to do that for the master branch, though which is what you care about.
Have Octopus tag your source control with the deployment version number and the environment name, so that you can quickly see (from your source control) what got deployed where.
Steps 1 and 2 are the most important ones, really. 3 and 4 are just nice-to-have. Most of the time you'll just open the app in the environment, check the commit hash in the "About", and do a git checkout to that commit hash...
If testing is successful, then the same build can be "promoted" from our staging environment to production (no need to build again)
Again, Octopus Deploy lifecycles, and make sure anything different in each environment is defined in the configuration file of the application, which is updated during the Octopus deployment, using environment-specific variables.
In terms of branch workflow, this last requirement makes it mandatory to merge changes into master (or whatever your "production" branch is) before the deployment lifecycle can begin.
The project that I am working on is a jQuery plugin. I have managed to get Travis CI to build a test project using Gulp/NodeJS successfully. Now I am trying to work out what workflow to use to bump the version number.
In TeamCity and MyGet there is a setting in the CI server to form a version number pattern that auto increments on each build, which can be used by the build script to update versions in the deployment files and to label the Git repo. However, in the free version of Travis CI, there doesn't seem to be an option for versioning at all.
I have read several articles on continuous deployment with Travis CI, here, here, and here, but none of them even broach the topic of versioning. Obviously, the version needs to be changed for the release. So what am I missing here?
Another problem I noted when going through the documentation is that it mentioned that Travis CI is not able to update the GitHub repository. Doesn't that basically mean it won't be able to create a Git tag?
If there is no way to version from Travis CI, then what is the typical workflow for the release process for such a plugin? Is the versioning always done manually? If so, how could there be "continuous deployment"?
Before it starts running the instructions in your .travis.yml file, Travis will set a bunch of environment variables (in the VM that is building your project) with various bits of information about your build, such as what branch is being built and so on.
You probably want one of these:
TRAVIS_BUILD_NUMBER: The number of the current build (for example, “4”).
TRAVIS_JOB_NUMBER: The number of the current job (for example, “4.1”).
But it's going to be very difficult to do anything sensible if you don't have control of the repository, because you'll need to upload a .travis.yml file into the root of your source code folder, otherwise Travis won't know what to do.
Use bumped for release versioning. When you're satisfied with the changes in master, run:
bumped release <major|minor|patch>
After you push the changes, either directly or through a release PR, you can check for the presence of new tags in Travis CI and publish the package to the registry automatically.
If you consider that every PR must end up to your enduser without thinking of the impact of such changes, then your version numbers have no meaning.
You don't give your user a way to know if it is a major change that break compatibility or a bug fix. You don't allow him to get update without worrying about backward compatibility.
Currently, the commit id is your version number.
If you want to give meaning to your version numbers then you have to think of the impact of your pull requests on the enduser (http://semver.org/). You have to choose a version number for a specific PR or a group of PR.
So basically, since you have to 'think' of a certain version number for a specific version that you want to deliver, you can't automate this process.
Release/tag creation is the way to go : )
You can accomplish this by setting up a script that would create a ~/.netrc file to access the repository. In this file you can specify something like:
machine https://github.com/xxx/yyy.git
login <blah>
And instead of putting in your credentials, you can pass an github access token. You can use the travis encrypt to register it in the .travis.yml file, and export the variable for your script's use. From there in your script, you can issue regular git commands such as:
git add <some file>
git commit -m "This is $TRAVIS_BUILD_NUMBER"
git push origin <branch>
Background:
We have one Jenkins job (Production) to build a deliverable every night. We have another job (ProductionPush) that pushes out the deliverable over a proprietary protocol to production machines the next day. This is because some production machines are only available during certain hours during the day (It also gives us a chance to fix any last-minute build breaks). ProductionPush needs access to the deliverable built by the Production job (so it needs access to the same workspace). We have multiple nodes and concurrent builds (and thus unpredictable workspaces) and prefer not to tie the jobs to a fixed node/workspace since resources are somewhat limited.
Questions:
How to make sure both jobs share the same workspace and ensure that ProductionPush runs at a fixed time the next day only if Production succeeds -- without fixing both jobs to run out of the same node/workspace? I know the Parameterized Trigger Plugin might help with some of this but it does not seem to have time delay capability and 12 hours seems too long for a quiet period.
Is sharing the workspace a bad idea?
Answer 2: Yes, sharing workspace is a bad idea. There is possibility of file locks. There is the issue of workspace being wiped out. Just don't do it...
Answer 1: What you need is to Archive the artifacts of the build. This way, the artifacts for a particular build (by build number) will always be available, regardless of whether another build is running or not, or what state the workspaces are in
To Archive the artifacts
In your build job, under Post-build Actions, select Archive the artifacts
Specify what artifacts to archive (you can use a combination of below)
a) You can archive all: *.*
b) You can archive a particular file with wildcards: /path/to/file_version*.zip
c) You can ignore the intermediate directories like: **/file_version*.zip
To avoid storage problems with many artifacts, on the top of configuration you can select Discard Old Builds, Click Advanced button, and play around with Days to keep artifacts and Max # of builds to keep with artifacts. Note that these two settings do not control for how long the actual builds are kept (other settings control that)
To access artifacts from Jenkins
In the build history, select any previous build you want.
In addition to SCM changes and revisions data, you will now have a Build Artifacts link, under which you will find all the artifacts for that particular build.
You can also access them with Jenkins' permalinks, for example
http://JENKINS_URL/job/JOB_NAME/lastSuccessfulBuild/artifact/ and then the name of the artifact.
To access artifacts from another job
I've extensively explained how to access previous artifacts from another deploy job (in your example, ProductionPush) over here:
How to promote a specific build number from another job in Jenkins?
If your requirements are to always deploy latest build to Production, you can skip the configuration of promotion in the above link. Just follow the steps for configuration of the deploy job. Once you have your deploy job, if it is always run at the same time, just configure its Build periodically parameters. Alternatively, you can have yet another job that will trigger the deploy job based on whatever conditions you want.
In either case above, if your Default Selector is set to Latest successful build (as explained in the link above), the latest build will be pushed to Production
Not sure archiving artifacts is really a good idea. A staging repository might be better as it enables cross-functional teams to share artifacts across different builds when required by tweaking the Maven settings.xml file.
You really want a deployable (ear/war) as the thing that gets built, tested, then promoted to production once confidence is high with the build.
Use a build number on your deployable (major.minor.buildnumber). This is the thing you promote to production, providing your tests can be relied upon. Don't use a hyphen to separate minor with build number as that forces Maven to perform a lexical comparison... a decimal point will force a numeric comparison which will give you far less headaches.
Also, you didn't mention your target platform, but using the Maven APT/RPM plugin to push an APT/RPM to a APT/YUM repo that's available to a production box (AFTER successful testing!) would be a good fit, as per industry standards?
I was hoping for some guidance on the best way to promote a build through its environments.
We have 3 environments, DEV, STAGING, PROD.
The DEV Jenkins build is running in a continuous integration set-up, as code is checked in to subversion, Jenkins will run a new build (clean, compile, test, deploy).
The tricky bit is when it comes to STAGING and PROD.
The idea was to be able to manually promote a successful DEV build to STAGING.
STAGING build would check out the DEV's SVN Revision number, build, test, deploy to staging and finally create a branch in SVN.
Lastly the release manager could manually promote the STAGING build to PROD.
PROD build would check out the branch from the previous STAGING build, deploy to PROD and tag the branch as a release.
I have tried to use a combination of the Promotion Builds Plugin and the Paramterized Trigger Plugin but with no luck. The Subversion Revision number doesn't seem to get passed between DEV build to STAGING build.
Does anyone have any guidance on their process to promote a build through multiple environments?
Another approach is to make use of the Artifact storage Jenkins provides coupled with the Copy Artifact Plugin.
When a build is completed, you could instruct Jenkins to persist your application, either as a compressed zip/tar.gz or as an application bundle (jar/war)
Trigger a downstream job and use the Copy Artifact to retrieve the recorded artifact from the upstream job (or use parameterised builds)
Deploy/Unzip artifact as necessary - Build shell script/maven deploy?
Retest application using the same sources/binaries as was created in step 1
Repeat for PROD as necessary
This approach would allow you to fingerprint the artifacts, and thus Jenkins would link builds together in the UI, as well as allow more formal sign off.
In this scenario, why do you need to go back and label the branch in svn? We don't use svn, but w/ TFS, when Hudson/Jenkins gets the code, the changeset number it has retrieved is in the build log. So we know what code the build came from, and could get back to it at any time.
Then we promote the build from environment to environment using Hudson, the source control system doesn't need to know where the code is deployed.
If it's absolutely necessary to store the SVN Revision ID, then add a build step to your DEV job that copies it to a file. Something like this:
echo %SVN_REVISION%>revision.ini
or something like this:
echo MY_SVN_REVISION=%SVN_REVISION%>revision.ini
Then artifact revision.ini. When doing a STAGING build, use the Copy Artifact plugin (as mentioned by a previous user) to retrieve the revision.ini file specific to the build and load it into a variable. Then use that variable in a command line call to "svn" to build the tag.