I am trying to find the most correct and simplest deployment process for each application release we have. The difficulties are the following:
It is Azure DevOps Releases.
Application has more than one artifact that should be delivered as a whole. Application distro, Its modules, and some configuration for servers that are not directly linked with the application, and of course automation scripts (which is versioned). All of them have their own build process and can be versioned separately.
All those components (App, separate server config, automation) are delivered as one product version. However, they could have different versions, for example: 12.2.1 (app), 12.2.3 (server), 12.2.1 (automation).
The question is how to build the process when after release (official), we can stick all of the final versions together (i mean not specifying them manually during pipeline release creation for each) taking into account that one of the component's version can be increased and we should be able to increase the version for release in terms of hotfixes for example.
Release Pipelines and 3 Artifacts: ok, there are 3 artifacts and a user has to specify all 3 versions manually during creation - quite a high risk to misclick. Unfortunately, there are 10 of them... 10 multiply 3 = 30 times to do a mistake.
Release Pipeline and 1 Artifact (app): Consider only one version of the application and automatically obtain automation scripts and configuration from feed by using app version. Could work, but no observability on what artifacts are going to be used, no way to downgrade, the only the latest version of artifacts (12.2.3.*).
Specify the version in the variable group connected to the stage (environments). Can easily make a mistake because release uses the baken version of the variable group. If you update VG, but not create a release - it will be epic fail. Moreover, there is no vision of what it is going to install/update, etc.
Please, share your ideas on how to manage multiple artifact versions within one product release to make the process more robust and clear with a little of flexibility.
That's how it works for me via this simple approach:-
A single build pipeline with many Agent Jobs where each one represents an artifact.
So here you will publish the different artifacts under separate folders and automatically increase versions "If needed", using a script per each and at the end all will be under the same BuildId.
Another variety if separation isn't needed that all be under the same Agent Job.
A single release pipeline that is linked to this big nested artifact that will perform the deploy.
Here also if separation is needed, could have multi stages or multi Agent Jobs as before.
Versions increasing either will happen manually at commits or scripts will automatically increase each build.
I would prefer the first as the 2nd doesn't indicate that there was any change.
The possibility of developer forgetting to update the version will be the same as developer introduces a bug so that can be fixed later.
Then at the end, you have a Single Release => Single Build => Many versions => A commit.
Related
Using TFS 2017. I have a build definition that creates artifacts to be deployed to WEB and SQL servers. Due to some limitations, each build has incremental package for SQL deployment and full deployment to WEB servers. I have new release created every time new version of artifact is created, but we don't want to deploy right away. Our deployment should start at specific time. So, the release definition has environment that is deployed to on schedule (let's say at 3 AM).
Now, sometimes we have multiple builds running during the day creating multiple releases to be deployed at 3 AM. Everything looks great but... at 3 AM all pending releases are deployed out of order (for example 5, 3, 1, 4, 2). This is not good as release 2 will "wipe out" changes that were added to release 3, 4 and 5.
How do we enforce order of releases to be deployed? I already set schedule for 3 AM, deploy only 1 release at a time and still not getting desired results.
I know there is an option to deploy latest release only, but this will complicate our process a lot (one of the reasons our builds produce incremental packages for database, and yes, release 5 may update same procedure or view that was updated by release 2, so order is important).
It is very strange to be unable to do this simple and very logical deployment type. Am I missing something?
Thanks.
According to Microsoft, this strange and potentially harmful problem of randomly deploying different versions to same environment is "by design".
I am following up with you to provide you with the review answers we
got from Product Group.
Answer: “This is one of the expected scenarios where if releases are
scheduled to run by some time (Scheduled trigger) and if multiple CI’s
triggering CD’s are generated before the next scheduled run then,
there are no. of releases queued to be picked by the agents. And when
the scheduled time hits the queued releases are randomly picked by the
agents and we see the difference in order of execution and creation.”
Recommendation: In YAML pipelines there is an option available to
allow these releases run in sequence in which they were generated but
in classic pipelines we only have the option deploy the most recent
release and cancelling the older ones. Though as the older releases
are also available but not deployed and if in case any revert to the
previous version is required, they can be deployed accordingly.
I did not try YAML version yet. And this is not a solution for us in nearest future a we are trying to upgrade to TFS 2017 (before we can convert most of our builds from XAML to vNext).
I need to deploy many similar apps to the same environments with Azure DevOps.
What are some ways to share and maintain the release process for these apps....
to avoid updating every app's Release process when it changes?
Background
I have dozens of similar apps with identical release processes.
Each app (in its own repo) will have its own Azure Pipeline.
I'm fine with a custom solution to this problem.
Options
Make a Release for every Pipeline -- not a fan!
CON: Azure seems to want a 1-to-1 relationship between Pipelines and Releases.
CON: I want to avoid many Releases at all costs, since changes would be nearly unmaintainable.
Use custom stage templates -- doesn't work for what I need
CON: Releases do not share custom stage templates.
Custom templates that you create are scoped to the project that you created them in. [Azure Documentation]
CON: Custom stage templates cannot be updated (as of this post).
To update an stage template, delete the existing template in a release pipeline and then save the stage as a template with the same name. [Azure Documentation]
Put the release process in the Pipeline .yml file -- seems possible, but....
PRO: The process would be stored in a shared "common" repo.
PRO: Each app will have a minimal Pipeline .yml file, setting some parameters before running the shared process.
CON: How would I track and manually deploy releases without the Release portal UI? Am I missing something?
Trigger the same Release with artifacts from different apps -- not sure this will work....
PRO: One release process, easy to track and deploy as needed via the Release portal UI.
CON: How would I track releases for different apps? Would I name Releases with the app name?
CON: I don't see how to set it up, since a Release is tied to a primary Pipeline.
At the time of linking an artifact source to a release pipeline.... [Azure Documentation]
CON: Though multiple artifact sources can be used, Releases need a default artifact Pipeline or source repo. Would I need to setup the artifact source programmatically?
When you link multiple artifact sources to a release pipeline, one of them is designated as the primary artifact source. The primary artifact source is used to set a number of pre-defined variables. It can also be used in naming releases. [Azure Documentation]
Use multiple artifact sources and artifact variables -- not sure this will work....
When there are multiple artifact sources linked to a release pipeline, you can access information about each of these. [Azure Documentation]
Bottom Line
I was able to solve many of these problems in Octopus Deploy. However, I'm having significant trouble seeing how to move my DevOps process into Azure DevOps. How would you handle this situation?
There is no perfect solution to meet your requirements.
Generally deploy multiple apps to multiple environments, we recommend that Make a Release for every Pipeline. But just as you mentioned it has 1-to-1 relationship between build pipelines and Releases and it has many releases.
We can deploy multiple apps to multiple environments through ONE release with customized Template as you mentioned, we can also use the Task Groups to combine a set of tasks together, then reuse them in different stages. But we still need to update the definitions accordingly (change the Task Group version) once the specific task group is updated.
Another way is Clone Stage, you mentioned they are the same environments, so you can config one stage, then clone stages from the existing one, just name the stages with the specific App names accordingly. But you still need to change the settings accordingly, for example the Artifact filters (see Release triggers) to determine the condition to trigger the specific stage. But in this way it will download all the artifacts first when you add multiple artifact sources, this will take long time to get sources...
I'm finalising my CI setup with TFS and I have one last part to overcome. Currently, every CI build is pushed to octopus with the nuget packages versioned as major.minor.patch.buildid, and these are used in releases in Octopus versioned as major.minor.patch-beta{buildid}. Therefore, every CI build will be tagged as a beta build for a given release, e.g. 1.3.0.194 / 1.3.0-beta194. There will no doubt be a number of iterative dev builds before one is chosen to go to the test team - let's assume the chosen build is 194 in this case. At this point I envisage the creation of a new build, 1.3.0-rc1, which uses the build 194 binaries. This is then pushed to the test environment and testing begins. There may be a number of testing cycles, so let's say the testers sign off the version 1.3.0-rc4. A new release could then be made, 1.3.0, based on the 1.3.0-rc4 binaries, which is the gold release for the product.
Firstly, is this a good idea? Some feedback would be much appreciated.
Seperately, is it possible to restrict deployments to certain environments based on a tag in the version? In my example, I would never want a release marked as -beta to be deployed to a test environment - only -rc builds should be. Likewise, only tag-less builds should be deployed to production environments.
Here is the reason why you definitely shouldn't do that.
When you create version 1.3.0.194, Octopus will guarantee that the artefacts, process, and variables are given a snap-shot that means the deployment will performed in the same way on every environment.
You can edit these things as much as you like, but 1.3.0.194 will not become a more risky deployment due to these changes, as it will be oblivious to them.
If you created version 1.3.0.194, then made changes to your deployment process, or variables, these changes would leak into the version 1.3.0.194-beta and it would no longer be the same deployment that you have tested. The same would occur when you change to version 1.3.0.194-rc - more changes would leak in that you haven't actually tested.
Your deployment process, just like your code, should be versioned and tested - and that is what you get by using the same version throughout your deployment lifecycle.
Tagging deployment packages with beta/rc is a bad idea because this introduces an extra step in your delivery process. You will have many built versions some will advance to upper environments, some won't. That's okay. Your trying to treat these like regular Nuget packages used for dependency versioning/management. Its different. Just increment your build numbers without the tags and promote the builds that get sign off.
You shouldn't restrict deployments to environment based on tags either.
In VS Team Services > Releases, I want to 'promote' a specific build/artifact version to a single environment. For example:
Dev QA Prod
v223
v123 v123
I want to promote #123 to Prod, but it looks like the only way to do that is to Create a release to all 3 environments, which will overwrite #223.
There isn't any way to do this for now. You can submit a feature request on VSTS User Voice: http://visualstudio.uservoice.com/forums/330519-team-services/
You have a release pipeline that goes Dev -> QA -> Prod. You create a release for version #123 and promote it progressively through the stages in your pipeline. Version #223 can be in Dev, and you can still promote the release for build #123 from QA->Production.
Skipping stages and not testing software in lower environments is a huge problem that results in pain and suffering. One of the cornerstones of a stable release process is consistency. There are tons of ways to make your release process both flexible enough to handle the scenario you're describing while still being consistent. Skipping stages isn't it.
If you need to re-release an older build of your software, it should still go through the same pipeline. Just because it's an old build doesn't mean that you don't need to at least do a cursory investigation to make sure it's being deployed correctly.
Will that overwrite the version currently on the lower environments? Depends on how you structure things! For web applications, maybe you make it multi-tenant, so you can have multiple versions running side-by-side on the same machine. Or deploy to IaaS VMs, so you can just provision a new one and leave the currently-running VM alone. Or deploy to Azure PaaS and use different staging slots for different versions.
Background:
We have one Jenkins job (Production) to build a deliverable every night. We have another job (ProductionPush) that pushes out the deliverable over a proprietary protocol to production machines the next day. This is because some production machines are only available during certain hours during the day (It also gives us a chance to fix any last-minute build breaks). ProductionPush needs access to the deliverable built by the Production job (so it needs access to the same workspace). We have multiple nodes and concurrent builds (and thus unpredictable workspaces) and prefer not to tie the jobs to a fixed node/workspace since resources are somewhat limited.
Questions:
How to make sure both jobs share the same workspace and ensure that ProductionPush runs at a fixed time the next day only if Production succeeds -- without fixing both jobs to run out of the same node/workspace? I know the Parameterized Trigger Plugin might help with some of this but it does not seem to have time delay capability and 12 hours seems too long for a quiet period.
Is sharing the workspace a bad idea?
Answer 2: Yes, sharing workspace is a bad idea. There is possibility of file locks. There is the issue of workspace being wiped out. Just don't do it...
Answer 1: What you need is to Archive the artifacts of the build. This way, the artifacts for a particular build (by build number) will always be available, regardless of whether another build is running or not, or what state the workspaces are in
To Archive the artifacts
In your build job, under Post-build Actions, select Archive the artifacts
Specify what artifacts to archive (you can use a combination of below)
a) You can archive all: *.*
b) You can archive a particular file with wildcards: /path/to/file_version*.zip
c) You can ignore the intermediate directories like: **/file_version*.zip
To avoid storage problems with many artifacts, on the top of configuration you can select Discard Old Builds, Click Advanced button, and play around with Days to keep artifacts and Max # of builds to keep with artifacts. Note that these two settings do not control for how long the actual builds are kept (other settings control that)
To access artifacts from Jenkins
In the build history, select any previous build you want.
In addition to SCM changes and revisions data, you will now have a Build Artifacts link, under which you will find all the artifacts for that particular build.
You can also access them with Jenkins' permalinks, for example
http://JENKINS_URL/job/JOB_NAME/lastSuccessfulBuild/artifact/ and then the name of the artifact.
To access artifacts from another job
I've extensively explained how to access previous artifacts from another deploy job (in your example, ProductionPush) over here:
How to promote a specific build number from another job in Jenkins?
If your requirements are to always deploy latest build to Production, you can skip the configuration of promotion in the above link. Just follow the steps for configuration of the deploy job. Once you have your deploy job, if it is always run at the same time, just configure its Build periodically parameters. Alternatively, you can have yet another job that will trigger the deploy job based on whatever conditions you want.
In either case above, if your Default Selector is set to Latest successful build (as explained in the link above), the latest build will be pushed to Production
Not sure archiving artifacts is really a good idea. A staging repository might be better as it enables cross-functional teams to share artifacts across different builds when required by tweaking the Maven settings.xml file.
You really want a deployable (ear/war) as the thing that gets built, tested, then promoted to production once confidence is high with the build.
Use a build number on your deployable (major.minor.buildnumber). This is the thing you promote to production, providing your tests can be relied upon. Don't use a hyphen to separate minor with build number as that forces Maven to perform a lexical comparison... a decimal point will force a numeric comparison which will give you far less headaches.
Also, you didn't mention your target platform, but using the Maven APT/RPM plugin to push an APT/RPM to a APT/YUM repo that's available to a production box (AFTER successful testing!) would be a good fit, as per industry standards?