Ivy is great for managing dependencies, but it isn't meant to handle the entire software lifecycle across many modules. That said, it does have several features that seem to support it (such as the status and branch attributes), and the ivy best practices blurb alludes to being able to promote integration revisions to milestone or release, "with some work".
Unfortunately I haven't found definitive guidance on how to manage the dev -> test -> deploy cycle. Here are some things I want to achieve:
(Given that devs typically work across many modules in a local workspace)
Dev can locally publish changes to a module, so that other modules in the workspace can get the updated artifacts.
Dev can designate a version as "ready to deploy to test" with one command.
Tester can designate a version as "ready for prod" with one command.
Dev can rebuild any version from source and the appropriate dependencies are picked up correctly (aka repeatable builds).
Some things I'm fairly clear about are:
The revision status should be used to denote whether that revision is meant only for development, is ready for testing, or is ready for production
The branch attribute should be sufficient to handle different project branches
Here is what I'm grappling with:
How to promote integration builds
Say I have these modules checked out in my workspace:
Now I'm happy with module a, and decide to publish a milestone using the checked out versions in my workspace. What needs to happen in the repo is:
e-1.0-RC1 gets published
d-1.1-RC2 gets published, referencing e-1.0-RC1 as a dependency
c-2.0-RC1 gets published, referencing d-1.1-RC2 as a dependency
b-3.3-RC1 gets published, referencing e-1.0-RC1 as a dependency
Finally, a-7.1-RC2 gets published, referencing c-2.0-RC1 and b-3.3-RC1 as dependencies.
If I try to roll my own for this, I'd probably end up doing some workspace management, ivy.xml find & replace, etc. Before I open that can of worms, I'd like to get some opinions. What's the best way to tackle this?
You can use recursive delivery to publish modules and their dependencies with a higher status.
Using your example:
e-1.0-RC1 gets published with an integration status
d-1.1-RC2 gets published with an integration status, referencing e-1.0-RC1 as a dependency
c-2.0-RC1 gets published with an integration status, referencing d-1.1-RC2 as a dependency
b-3.3-RC1 gets published with an integration status, referencing e-1.0-RC1 as a dependency
a-7.1-RC2 gets published with an integration status, referencing c-2.0-RC1 and b-3.3-RC1 as dependencies.
Finally, you decide to promote a-7.1-RC2 to a milestone status, so you do a recusive delivery (use the delivertarget attribute). This will recursively call the delivertarget for each dependency that has a status lower than milestone and publish it with a milestone status.
The nice thing about this, is that you don't need (or want) to have each project checked out in your workspace, just a. This also means that it's much easier to create a deployment pipeline and have your CI server:
run unit tests for a,
build a,
publish a as integration,
deploy a to a System Test environment,
run some System Tests
promote a from integration to milestone (which promotes it's dependencies)
deploy a to a Acceptance Test environment,
run some Acceptance Tests
promote a from milestone to release (which promotes it's dependencies)
deploy a to production (or upload it to a download site)
At no time does the pipeline need to access the dependant projects and, since the recursive delivery is generic, when you add or remove dependencies (via your ivy.xml files), you don't need to change anything in your pipeline.
I've marked this answer as a community wiki. Anyone else care to expand on it or correct anything I got wrong?
How do you do the line?:
promote a from milestone to release (which promotes it's dependencies)
I was planning on doing a retrieve and publish. Is there a better way?
Related
I'm having a hard time figuring out how to correctly deploy to different environments with TeamCity (in terms of cross BuildConfiguration dependencies) and hope to get some input as to how to configure my SubProjects/BuildConfigurations properly. Lets start based on a concrete example: I made this test "TeamcityConfigurationTests" to better learn how TeamCity handled dependencies, and the current state shows the result i am looking for:
I have 3 subProjects, Dev, Test and Prod - and all associated tasks for those "environments" as seperate build configurations within that subProject. This is to more clearly visualize what is going on, and if anything breaks, to be able to see immediately what is broken (separate Build, UnitTest and DeployToDev BuildConfigurations, rather than 3 different steps in one single Build Configuration).
Ideally, i only want to build my application once in the Dev.Build step, and let the Dev.UnitTest and Dev.DeployToDev steps grab that artifact and run tests and deploy. That i got going for me, by having snapshot and artifact dependencies. But i am having trouble getting the correct artifact when i want to deploy from Dev -> Test or Test -> Prod.
My issue is to correctly reference the latest successfully DEV deployed artifact when running Test.DeployToTest - and the same for getting the latest successfully TEST deployed artifact when running Prod.DeployToProd. (Essentially i want to promote the artifact to the next environment).
Now, my issue is, if I in the Test.DeployToTest have a SnapshotDependency to Dev.DeployToDev and an artifact dependency to Dev.Build, and the VCS source has changed since Deploy to Dev has run, it triggeres running all the DEV steps again. This is not the worst part, the same happens when i run Prod.DeployToProd if the VCS source changed since the initial build on dev (because of all the snapshot dependencies). Meaning, that rather than promoting Test -> Prod, I Build and deploy whatever is currently on VCS to Dev, Test AND Prod.
How am i supposed to set this up correctly?
The only other option i am aware of, is letting Dev.DeployToDev also publish the same artifact, and only have an (LatestSuccessful) ArtifactDependency in Test.DeployToTest. I would also have to publish the artifact again in Test.DeployToTest, for letting Prod.DeployToProd only have a (LatestSuccessfull) artifact dependency to Test.DeployToTest. (This would be to get rid of the SnapshotDependencies causing previous environments to run build/deploy again in case of VCS changes). But then i am publishing the artifact 3 times, rather than just the one time when the application is originally built in DEV - which i would like to avoid. Also, i have cases where no artifact is needed for deploying to Test and Prod, so there is no artifact to depend on (essentially i only need the BuildNumber from the "Dependent" environment i want to promote from).
I hope for some input. Thank you
Regards
Frederik
For anyone wondering, i made a JetBrains support ticket, and got the following response:
Basically, there are options to resolve your case:
Option 1: use "Promote" action form the build's Actions top-right menu
(or change the type of the Deploy* configurations to deployment and
use the action from the block on the build results. This is the
preferred way: before deploying you select the build to deploy and
"promote" it to the next environment. There is also an experimental
hidden feature to hide the "Run" button: add
"teamcity.ui.runButton.caption" configuration parameter in the build
configurations to empty value.
Option 2: do not use snapshot dependency, use only artifact dependency
on the latest successful build. However, when you run the build you
cannot be sure that the last successful build you see will be
deployed: while the build is standing int he queue, another
Dev.DeployToDev can finish and then be deployed as the last
successful.
We went with option 1
I have been researching and am trying to figure out the best branching and deployment strategy to accomplish the requirements below. Maybe I’m missing something but it is more complicated than it seems. Ideally, we’d just have one permanent branch, ‘master’, that could have specific commits tagged to mark releases to production.
Our current strategy is based on Git Flow and has permanent branches ‘master’ (only has releases to production) and ‘develop’. The primary thing that complicates using a multiple permanent-branches model is the concept of “promoting” the same build from the staging environment to production. Currently, this needs to be done in a separate source code branch (deployments to staging come from ‘develop’, deployments to prod come from ‘master’).
Tools: Git (VSTS), TeamCity, Octopus Deploy
Requirements (feature and hotfix lifecycles):
All code is reviewed via pull requests (enforced via branch policies)
All code gets deployed to a staging environment for testing
We can quickly go back to any snapshot of code that was deployed previously
If testing is successful, then the same build can be “promoted” from our staging environment to production (no need to build again)
Features accumulate over time before pushing out to production as a single release. Hotfixes have to be able to go through without getting caught up in the "all or nothing" next regular release.
I like the idea of having one permanent branch with tags (re: The master/develop split is redundant, http://endoflineblog.com/gitflow-considered-harmful), but having additional permanent branches may better facilitate deploying to different lifecycles/versions (feature and hotfix) to Octopus.
I have been wrestling with how best to pull this off and I may be over complicating things. Any feedback is appreciated.
It seems you have a number of questions and they are quite broad... I'll add some comments to each of your requirements as a conversation starter, but this whole thread might get blocked by moderators as it is definitely not the style of questions SO was made for.
All code is reviewed via pull requests (enforced via branch policies)
I haven't looked at VSTS for ages, but I'd expect they already support branch policies and pull-requests, so not sure if there's anything you need here other than configure settings in your repositories.
In case VSTS does not support that, you might consider moving to a tool that does e.g. BitBucket, GitHub, etc. Both of these have an on-premises version in case you can't (or don't want to) use the cloud hosted version.
All code gets deployed to a staging environment for testing
You achieve that with setting up lifecycles in Octopus Deploy, to make sure deployments/promotions follow the the sequence you want.
We can quickly go back to any snapshot of code that was deployed previously
You already have source control, so all you need now is traceability from the code that is deployed in an environment, to the deployment version in Octopus Deploy, the build job in TeamCity, the branch and exact commit in your source control.
There's a few things that you can do, to achieve that:
Define a versioning scheme that works for you. I like to use semantic versioning. "Major" and "Minor" versions are defined by the developers, and the "Patch" is the auto-incremented number from TeamCity (%build.number%). Every git push build the code and generates a unique build version (%major%.%minor%.%build.number%)
As part of the build steps in TeamCity, before you compile the code, make sure your source files are patched with the version number assigned by each build, the commit hash from your source control, and the branch name. e.g. if you are using .NET, make sure all the AssemblyInfo.cs files are updated with that version, so that the version is embedded in the binaries. This allows anyone to query the version looking at the properties of the binary files, and also allows you to display the app version on the app itself (e.g. status bar, footer, caption, about box, etc.)
Have TeamCity tag your source control with the version number of every build, so you can quickly see on your source control history. You probably only want to do that for the master branch, though which is what you care about.
Have Octopus tag your source control with the deployment version number and the environment name, so that you can quickly see (from your source control) what got deployed where.
Steps 1 and 2 are the most important ones, really. 3 and 4 are just nice-to-have. Most of the time you'll just open the app in the environment, check the commit hash in the "About", and do a git checkout to that commit hash...
If testing is successful, then the same build can be "promoted" from our staging environment to production (no need to build again)
Again, Octopus Deploy lifecycles, and make sure anything different in each environment is defined in the configuration file of the application, which is updated during the Octopus deployment, using environment-specific variables.
In terms of branch workflow, this last requirement makes it mandatory to merge changes into master (or whatever your "production" branch is) before the deployment lifecycle can begin.
Consider the following -
I have a solution that consists of multiple projects:
DAL (Class Library)
BusinessLogic (Class Library)
Website1 (Web Application)
Website2 (Web Application)
Both Website1 and Website2 share a reference to BusinessLogic, which in turn references the DAL.
Since these are just websites, I don't need to keep track of multiple versions, as such, but I do like to have the following branches:
Trunk
Production
Trunk is where I do all my development work, and after everything is tested and ready to go, I merge from Trunk to Production when a website is actually deployed to production servers. This allows me to shelve my current work, check out the Production branch and address any major bugs that were found after deployment and immediately deploy the fix.
My problem is that, using this approach, what lives in the Production branch isn't always correct. Let's say I make an update to BusinessLogic which is utilised by Website1. It passes testing and is deployed. If I merge all the projects to the Production branch, then it's wrong because Website2 wasn't deployed to production at that time.
Or, I could merge only the relevant projects to Production. So, in this case, I would merge Website1, BusinessLogic and DAL. This is still wrong, however. If I were to check out the Production branch to do work on Website2, it would have a newer version of BusinessLogic and DAL than actually exist on our production servers.
What is the correct approach here?
You should not use a code sharing or code promotion model. It reduces quality and forces rework. Instead look to create a release pipeline where you create a package for your Business and Dal layers and consume those packaged in the web apps.
The best approach for this is to use a build server and create a NuGet package for your DAL that is consumed by the Business Layer. This in turn is packaged as a NuGet package that your websites can consume.
Your workflow for getting a change to the business layer into your website is then:
Open Business Layer solution and make fix
Check in and trigger CI build
CI build creates and publishes NuGet package
Open Website solution and update NuGet package
Clean and simple. No branching is good branching.
There can be a lot of correct ways, and it always depends on what is correct for you. Each way would have pros and cons, of course.
If you'd like source-level dependencies, create several production branches per site, each includes external:
/Site1/Production
site content
BusinessLogic [external] -> /BusinessLogic/v1Branch
/Site2/Production
site content
BusinessLogic [external] -> /BusinessLogic/v1Branch
/BusinessLogic/v1Branch (from DevBranch)
/BusinessLogic/DevBranch
Here is how you perform version upgrade:
Make a change to BusinessLogic/DevBranch, test it.
Branch it as BusinessLogic/v2Branch
Update Site2/Production's external to point to BusinessLogic/v2Branch
Build site2, test and deploy.
So you'll have -
/Site1/Production
site content
BusinessLogic [external] -> /BusinessLogic/v1Branch
/Site2/Production
site content
BusinessLogic [external] -> /BusinessLogic/v2Branch
/BusinessLogic/v1Branch (from DevBranch)
/BusinessLogic/v2Branch (from DevBranch)
/BusinessLogic/DevBranch
This requires a certain level of development culture and some amount of svn management.
Also you can also put binaries to such svn branches, which is pretty much the same scheme. In general such approach is named vendor branches.
If you prefer binary dependencies outside the source control, you could utilize local nuget repository. It works the same as official one: you create a new version, publish to nuget, then refer it from site, the build and deploy. This requires additional setup and maintenance effort and is more appropriate for larger projects.
Background:
We have one Jenkins job (Production) to build a deliverable every night. We have another job (ProductionPush) that pushes out the deliverable over a proprietary protocol to production machines the next day. This is because some production machines are only available during certain hours during the day (It also gives us a chance to fix any last-minute build breaks). ProductionPush needs access to the deliverable built by the Production job (so it needs access to the same workspace). We have multiple nodes and concurrent builds (and thus unpredictable workspaces) and prefer not to tie the jobs to a fixed node/workspace since resources are somewhat limited.
Questions:
How to make sure both jobs share the same workspace and ensure that ProductionPush runs at a fixed time the next day only if Production succeeds -- without fixing both jobs to run out of the same node/workspace? I know the Parameterized Trigger Plugin might help with some of this but it does not seem to have time delay capability and 12 hours seems too long for a quiet period.
Is sharing the workspace a bad idea?
Answer 2: Yes, sharing workspace is a bad idea. There is possibility of file locks. There is the issue of workspace being wiped out. Just don't do it...
Answer 1: What you need is to Archive the artifacts of the build. This way, the artifacts for a particular build (by build number) will always be available, regardless of whether another build is running or not, or what state the workspaces are in
To Archive the artifacts
In your build job, under Post-build Actions, select Archive the artifacts
Specify what artifacts to archive (you can use a combination of below)
a) You can archive all: *.*
b) You can archive a particular file with wildcards: /path/to/file_version*.zip
c) You can ignore the intermediate directories like: **/file_version*.zip
To avoid storage problems with many artifacts, on the top of configuration you can select Discard Old Builds, Click Advanced button, and play around with Days to keep artifacts and Max # of builds to keep with artifacts. Note that these two settings do not control for how long the actual builds are kept (other settings control that)
To access artifacts from Jenkins
In the build history, select any previous build you want.
In addition to SCM changes and revisions data, you will now have a Build Artifacts link, under which you will find all the artifacts for that particular build.
You can also access them with Jenkins' permalinks, for example
http://JENKINS_URL/job/JOB_NAME/lastSuccessfulBuild/artifact/ and then the name of the artifact.
To access artifacts from another job
I've extensively explained how to access previous artifacts from another deploy job (in your example, ProductionPush) over here:
How to promote a specific build number from another job in Jenkins?
If your requirements are to always deploy latest build to Production, you can skip the configuration of promotion in the above link. Just follow the steps for configuration of the deploy job. Once you have your deploy job, if it is always run at the same time, just configure its Build periodically parameters. Alternatively, you can have yet another job that will trigger the deploy job based on whatever conditions you want.
In either case above, if your Default Selector is set to Latest successful build (as explained in the link above), the latest build will be pushed to Production
Not sure archiving artifacts is really a good idea. A staging repository might be better as it enables cross-functional teams to share artifacts across different builds when required by tweaking the Maven settings.xml file.
You really want a deployable (ear/war) as the thing that gets built, tested, then promoted to production once confidence is high with the build.
Use a build number on your deployable (major.minor.buildnumber). This is the thing you promote to production, providing your tests can be relied upon. Don't use a hyphen to separate minor with build number as that forces Maven to perform a lexical comparison... a decimal point will force a numeric comparison which will give you far less headaches.
Also, you didn't mention your target platform, but using the Maven APT/RPM plugin to push an APT/RPM to a APT/YUM repo that's available to a production box (AFTER successful testing!) would be a good fit, as per industry standards?
I have a couple of projects which are developed and released on different branches, namely development and release. The process works pretty well but unfortunately it has some drawbacks and I have been wondering if there is a better versioning scheme to apply in my situation.
The main development happens on a development branch (i.e. Subversion trunk but it doesn't matter much) where team of developers commit their changes. After building and packaging artifacts, Jenkins deploys them to maven repository and development integration application server. This is a DEVELOPMENT-SNAPSHOT and basically is just a feature branch containing all developed features on one common branch:
<groupId>pl.cyfrowypolsat.process-engine</groupId>
<artifactId>process-engine</artifactId>
<version>D.16-SNAPSHOT</version>
When one particular business change is done and requested by QA team, this single change is then being merged to the release branch (branches/release). Jenkins deploys the resulting artifact to QA application server:
<groupId>pl.cyfrowypolsat.process-engine</groupId>
<artifactId>process-engine</artifactId>
<version>R.16-SNAPSHOT</version>
Then there's a release which happens via maven-release-plugin on the release branch version of software (which creates a maintenance tag/branch for quick bug fixing). (R.16-SNAPSHOT => R.16)
Development and release branches are currently being versioned as D.16-SNAPSHOT and R.16-SNAPSHOT respectively. This allows to separate artifacts in maven repository but creates a problem with different maven mechanisms which rely on standard maven versioning style. And this breaks OSGI versioning as well.
Now, how would you name and version maven artifacts in such a scheme? Is there a better way? Maybe I could make some changes to maven structures other than simply changing the versioning and naming schemes? But I need to keep development and QA (release) SCM branches separate.
Would a maven classifier of 'development'/'production' be a reasonable alternative?
<groupId>pl.cyfrowypolsat.process-engine</groupId>
<artifactId>process-engine</artifactId>
<version>16-SNAPSHOT</version>
<classifier>D</classifier>
As far as I know, a common naming extension for a release artifact would be just the name of the artifact, without any stuff, only the version specified. A development branch would have the same artifact name but with snapshot.
For example, take twitter4j. The artifact name of the release version is
twitter4j-2.5.5
Snapshot of their(his) development version
twitter4j-2.6.5-SNAPSHOT
That is the naming convention almost everybody uses and is recognized by most tools. For example, my Nexus repository can specify a policy to ignore development releases which basically means it ignores the artifacts containing -SNAPSHOT in their name.
EDIT:
To your followup question:
Well, depending on your build tool, you can create your snapshots to have the timestamp or some other unique identifier. However, I have never heard of some branching logic being embedded in the artifact's name just so the continuous int server can distinguish it. From the artifact's perspective, it is either a release, or a SNAPSHOT, I don't see the benefit of embedding more logic into the name of the artifact just cause your Hudson allows yo to do so. To be honest, your release cycle seems OK to me, but it would require some fine tweaking of your maven tools. If you can't live with that I would suggest you to use a classifier instead of relying on the name as it is always easier to tweak the integration server than a lot of plugins that rely on standard naming convention. In conclusion, I believe you are on the right track.
I think you could simply the process by having only two types as far as maven is concerned
Snapshot (In perpetual development)
Releasable (with a version number that can be deployed to maven repository or production release)
I would handle your branching a little differently, If you look at the iterative/scrum development model your code should be releasable/shippable at end of a iteration/sprint
Main sub version trunk is where developers commit their code
At the end of the sprint/iteration branch the main trunk and called it release branch (there should not be a QA branch any code that is to be released is tested for quality)
Bug fixes should happen on the release branch and periodically merged back to main trunk
This way you can keep creating branches for a release and any bug fixes are committed to branch
Always make sure before creating a new branch from main trunk, It has all the merges from previous branches
The release plugin from Maven supports branching. It appears to work by assuming that the branch is created to support the next version of your code.
Personally, I'm more inclined to use the versions plug-in, and explicitly set my Maven project's version numbers.