I have a couple of projects which are developed and released on different branches, namely development and release. The process works pretty well but unfortunately it has some drawbacks and I have been wondering if there is a better versioning scheme to apply in my situation.
The main development happens on a development branch (i.e. Subversion trunk but it doesn't matter much) where team of developers commit their changes. After building and packaging artifacts, Jenkins deploys them to maven repository and development integration application server. This is a DEVELOPMENT-SNAPSHOT and basically is just a feature branch containing all developed features on one common branch:
<groupId>pl.cyfrowypolsat.process-engine</groupId>
<artifactId>process-engine</artifactId>
<version>D.16-SNAPSHOT</version>
When one particular business change is done and requested by QA team, this single change is then being merged to the release branch (branches/release). Jenkins deploys the resulting artifact to QA application server:
<groupId>pl.cyfrowypolsat.process-engine</groupId>
<artifactId>process-engine</artifactId>
<version>R.16-SNAPSHOT</version>
Then there's a release which happens via maven-release-plugin on the release branch version of software (which creates a maintenance tag/branch for quick bug fixing). (R.16-SNAPSHOT => R.16)
Development and release branches are currently being versioned as D.16-SNAPSHOT and R.16-SNAPSHOT respectively. This allows to separate artifacts in maven repository but creates a problem with different maven mechanisms which rely on standard maven versioning style. And this breaks OSGI versioning as well.
Now, how would you name and version maven artifacts in such a scheme? Is there a better way? Maybe I could make some changes to maven structures other than simply changing the versioning and naming schemes? But I need to keep development and QA (release) SCM branches separate.
Would a maven classifier of 'development'/'production' be a reasonable alternative?
<groupId>pl.cyfrowypolsat.process-engine</groupId>
<artifactId>process-engine</artifactId>
<version>16-SNAPSHOT</version>
<classifier>D</classifier>
As far as I know, a common naming extension for a release artifact would be just the name of the artifact, without any stuff, only the version specified. A development branch would have the same artifact name but with snapshot.
For example, take twitter4j. The artifact name of the release version is
twitter4j-2.5.5
Snapshot of their(his) development version
twitter4j-2.6.5-SNAPSHOT
That is the naming convention almost everybody uses and is recognized by most tools. For example, my Nexus repository can specify a policy to ignore development releases which basically means it ignores the artifacts containing -SNAPSHOT in their name.
EDIT:
To your followup question:
Well, depending on your build tool, you can create your snapshots to have the timestamp or some other unique identifier. However, I have never heard of some branching logic being embedded in the artifact's name just so the continuous int server can distinguish it. From the artifact's perspective, it is either a release, or a SNAPSHOT, I don't see the benefit of embedding more logic into the name of the artifact just cause your Hudson allows yo to do so. To be honest, your release cycle seems OK to me, but it would require some fine tweaking of your maven tools. If you can't live with that I would suggest you to use a classifier instead of relying on the name as it is always easier to tweak the integration server than a lot of plugins that rely on standard naming convention. In conclusion, I believe you are on the right track.
I think you could simply the process by having only two types as far as maven is concerned
Snapshot (In perpetual development)
Releasable (with a version number that can be deployed to maven repository or production release)
I would handle your branching a little differently, If you look at the iterative/scrum development model your code should be releasable/shippable at end of a iteration/sprint
Main sub version trunk is where developers commit their code
At the end of the sprint/iteration branch the main trunk and called it release branch (there should not be a QA branch any code that is to be released is tested for quality)
Bug fixes should happen on the release branch and periodically merged back to main trunk
This way you can keep creating branches for a release and any bug fixes are committed to branch
Always make sure before creating a new branch from main trunk, It has all the merges from previous branches
The release plugin from Maven supports branching. It appears to work by assuming that the branch is created to support the next version of your code.
Personally, I'm more inclined to use the versions plug-in, and explicitly set my Maven project's version numbers.
Related
I have been researching and am trying to figure out the best branching and deployment strategy to accomplish the requirements below. Maybe I’m missing something but it is more complicated than it seems. Ideally, we’d just have one permanent branch, ‘master’, that could have specific commits tagged to mark releases to production.
Our current strategy is based on Git Flow and has permanent branches ‘master’ (only has releases to production) and ‘develop’. The primary thing that complicates using a multiple permanent-branches model is the concept of “promoting” the same build from the staging environment to production. Currently, this needs to be done in a separate source code branch (deployments to staging come from ‘develop’, deployments to prod come from ‘master’).
Tools: Git (VSTS), TeamCity, Octopus Deploy
Requirements (feature and hotfix lifecycles):
All code is reviewed via pull requests (enforced via branch policies)
All code gets deployed to a staging environment for testing
We can quickly go back to any snapshot of code that was deployed previously
If testing is successful, then the same build can be “promoted” from our staging environment to production (no need to build again)
Features accumulate over time before pushing out to production as a single release. Hotfixes have to be able to go through without getting caught up in the "all or nothing" next regular release.
I like the idea of having one permanent branch with tags (re: The master/develop split is redundant, http://endoflineblog.com/gitflow-considered-harmful), but having additional permanent branches may better facilitate deploying to different lifecycles/versions (feature and hotfix) to Octopus.
I have been wrestling with how best to pull this off and I may be over complicating things. Any feedback is appreciated.
It seems you have a number of questions and they are quite broad... I'll add some comments to each of your requirements as a conversation starter, but this whole thread might get blocked by moderators as it is definitely not the style of questions SO was made for.
All code is reviewed via pull requests (enforced via branch policies)
I haven't looked at VSTS for ages, but I'd expect they already support branch policies and pull-requests, so not sure if there's anything you need here other than configure settings in your repositories.
In case VSTS does not support that, you might consider moving to a tool that does e.g. BitBucket, GitHub, etc. Both of these have an on-premises version in case you can't (or don't want to) use the cloud hosted version.
All code gets deployed to a staging environment for testing
You achieve that with setting up lifecycles in Octopus Deploy, to make sure deployments/promotions follow the the sequence you want.
We can quickly go back to any snapshot of code that was deployed previously
You already have source control, so all you need now is traceability from the code that is deployed in an environment, to the deployment version in Octopus Deploy, the build job in TeamCity, the branch and exact commit in your source control.
There's a few things that you can do, to achieve that:
Define a versioning scheme that works for you. I like to use semantic versioning. "Major" and "Minor" versions are defined by the developers, and the "Patch" is the auto-incremented number from TeamCity (%build.number%). Every git push build the code and generates a unique build version (%major%.%minor%.%build.number%)
As part of the build steps in TeamCity, before you compile the code, make sure your source files are patched with the version number assigned by each build, the commit hash from your source control, and the branch name. e.g. if you are using .NET, make sure all the AssemblyInfo.cs files are updated with that version, so that the version is embedded in the binaries. This allows anyone to query the version looking at the properties of the binary files, and also allows you to display the app version on the app itself (e.g. status bar, footer, caption, about box, etc.)
Have TeamCity tag your source control with the version number of every build, so you can quickly see on your source control history. You probably only want to do that for the master branch, though which is what you care about.
Have Octopus tag your source control with the deployment version number and the environment name, so that you can quickly see (from your source control) what got deployed where.
Steps 1 and 2 are the most important ones, really. 3 and 4 are just nice-to-have. Most of the time you'll just open the app in the environment, check the commit hash in the "About", and do a git checkout to that commit hash...
If testing is successful, then the same build can be "promoted" from our staging environment to production (no need to build again)
Again, Octopus Deploy lifecycles, and make sure anything different in each environment is defined in the configuration file of the application, which is updated during the Octopus deployment, using environment-specific variables.
In terms of branch workflow, this last requirement makes it mandatory to merge changes into master (or whatever your "production" branch is) before the deployment lifecycle can begin.
What is the best practice for managing dependencies within a Simulink Project when the project is worked on across a team and the project has dependencies on different models and libraries?
An parallel example would be when building an application using Gradle and declaring the dependencies of a project including the required version numbers. Grade will resolve and download the versions that are required to build the project.
e.g. the following declares a dependency on version 2.1 of library and version 1.0 upwards of some-library, so that the latest version 1.x (1.0, 1.1, 1.2...) that is available will be downloaded and used.
dependencies {
compile("com.example:library:2.1")
compile("com.example:some-library:1.+")
}
The documentation for Simulink (and also here covering manifests) seems to talk about models within a project having version numbers. It doesn't seem to mention libraries that are imported into the project. Models that are only used within a single project could all be contained in the overall project, but what happens if there are (for example) generic S-Functions defined within a separate project or library (or library defined within a project) that are applicable across multiple projects? This requirement is all with the aim of helping to support an automatic build process triggered by a Continuous Integration server, such as Jenkins.
I'm interested in a workflow that will easily support dependency management and automatic dependency resolution with a Github Flow git branching policy.
I've spent much time on this problem. Finally I didn't find an appropriate solution online, but I'd like to share the workflow we are using now and which fulfills our needs.
In short: We created our own dependency management by using git submodules.
Assumption: In fact, it is more a version management of persistent dependencies rather than offering the possibility to dynamically add new or remove old packages or libraries. This also works, but requires the git submodules to be added to or removed from the main git repository.
Objectives:
Consistent setup for everyone who works on the project.
Traceability of depdendencies.
Continous Integration with less effort.
How we do it (Example):
We have Project A and Project B which shall be used in Project C.
All three projects are under git version control and still under development.
We have set up additional release repositories for Project A and Project B, e.g. located on a network drive.
In Project C we add the release repositories of Project A and Project B as git submodules
We have set up some kind of auto-deployment to push only relevant files into these release repositories. For example if we want to make changes of Project B accessible to Project C, we only create a version tag in Project B's repository and it gets pushed to its release repository.
In Project C we update our git submodules and can checkout a new submodule version (if needed).
Advantages:
Since git stores the checked out version (commit) of git submodules in the main project, we can ensure that everyone works with the same files.
Changing the commit of a submodule is traceable in the main project.
The relation between the main project and the dependencies is always consistent.
Continuous Integration should work "out of the box". We are using GitLab and GitLab Runner and only had to setup our runner to fetch submodules recursively (in case of nested submodules).
I think this approach works as long as the repositories won't get too big, since you do not fetch only the version you need but also the whole version history.
Ivy is great for managing dependencies, but it isn't meant to handle the entire software lifecycle across many modules. That said, it does have several features that seem to support it (such as the status and branch attributes), and the ivy best practices blurb alludes to being able to promote integration revisions to milestone or release, "with some work".
Unfortunately I haven't found definitive guidance on how to manage the dev -> test -> deploy cycle. Here are some things I want to achieve:
(Given that devs typically work across many modules in a local workspace)
Dev can locally publish changes to a module, so that other modules in the workspace can get the updated artifacts.
Dev can designate a version as "ready to deploy to test" with one command.
Tester can designate a version as "ready for prod" with one command.
Dev can rebuild any version from source and the appropriate dependencies are picked up correctly (aka repeatable builds).
Some things I'm fairly clear about are:
The revision status should be used to denote whether that revision is meant only for development, is ready for testing, or is ready for production
The branch attribute should be sufficient to handle different project branches
Here is what I'm grappling with:
How to promote integration builds
Say I have these modules checked out in my workspace:
Now I'm happy with module a, and decide to publish a milestone using the checked out versions in my workspace. What needs to happen in the repo is:
e-1.0-RC1 gets published
d-1.1-RC2 gets published, referencing e-1.0-RC1 as a dependency
c-2.0-RC1 gets published, referencing d-1.1-RC2 as a dependency
b-3.3-RC1 gets published, referencing e-1.0-RC1 as a dependency
Finally, a-7.1-RC2 gets published, referencing c-2.0-RC1 and b-3.3-RC1 as dependencies.
If I try to roll my own for this, I'd probably end up doing some workspace management, ivy.xml find & replace, etc. Before I open that can of worms, I'd like to get some opinions. What's the best way to tackle this?
You can use recursive delivery to publish modules and their dependencies with a higher status.
Using your example:
e-1.0-RC1 gets published with an integration status
d-1.1-RC2 gets published with an integration status, referencing e-1.0-RC1 as a dependency
c-2.0-RC1 gets published with an integration status, referencing d-1.1-RC2 as a dependency
b-3.3-RC1 gets published with an integration status, referencing e-1.0-RC1 as a dependency
a-7.1-RC2 gets published with an integration status, referencing c-2.0-RC1 and b-3.3-RC1 as dependencies.
Finally, you decide to promote a-7.1-RC2 to a milestone status, so you do a recusive delivery (use the delivertarget attribute). This will recursively call the delivertarget for each dependency that has a status lower than milestone and publish it with a milestone status.
The nice thing about this, is that you don't need (or want) to have each project checked out in your workspace, just a. This also means that it's much easier to create a deployment pipeline and have your CI server:
run unit tests for a,
build a,
publish a as integration,
deploy a to a System Test environment,
run some System Tests
promote a from integration to milestone (which promotes it's dependencies)
deploy a to a Acceptance Test environment,
run some Acceptance Tests
promote a from milestone to release (which promotes it's dependencies)
deploy a to production (or upload it to a download site)
At no time does the pipeline need to access the dependant projects and, since the recursive delivery is generic, when you add or remove dependencies (via your ivy.xml files), you don't need to change anything in your pipeline.
I've marked this answer as a community wiki. Anyone else care to expand on it or correct anything I got wrong?
How do you do the line?:
promote a from milestone to release (which promotes it's dependencies)
I was planning on doing a retrieve and publish. Is there a better way?
We are reaching a point in our project where we need to make a production deployment but also need to have ongoing development for future features. Our source control currently has a single development branch. In my previous company a 3 branch system was set up with Development, Integration, Production. Feature development was done in Development, tesing in Integration and Production always was the code running in the current production environment (except for a breif time when a merge was done from Integaration into Production and the deployment was done).
Each time there was a Production change or a merge into Production that was deployed live, a new archive branch was taken off Production and given a version number. Any change to a higher branch was merged back so changes in Production would make it back to Development for example. It worked but I always didn't see the need for an ongoing Integration and Production branch.
Two aspects of this system I really didn't like: 1) the merge from Integration branch into Production branch: I would prefer to have a "clean" Production branch each time even though they should be in sync after a merge and 2) this system doesn't allow for multiple deployments of the system running different versions of the code at the same time, though this has not been a requirement in either team I have worked for (yet).
I have heard this model is common but in the system I am setting up now I am proposing the following:
Have a Development branch, create a new Release branch each time Development is ready for the next deployment to production. The Release branches are given a version number and then branched again to an archive branch. Testing is done on the Release branch. Once deployed any production fixes are done in the Release branch then a new archive branch is created with a minor version number increment once deployment is approved. When a new deployment from Development is ready the a new Release branch is created...
To me this is simpler and is actually better: There is NO Integaration branch (less merging) and there is a fresh Production (Release) branch for each deployment and caters for current Production versions. What am I missing? Why go for the Development, Integration, Production model?
Thanks
The advantage of the 3-branch system is that your developers, testers, and customers each have their own isolated version/branch of the code. This provides a "gated" development where features can only be promoted to the next branch when they pass a certain completeness/quality bar, giving high confidence that the release branch is always in a fit state to send to a customer.
Pros:
You (almost) always have a build that could be sent to a customer at a moments notice.
Your testers (almost) always have a working build to test
The release branch is stable so you don't often have to fix bugs in it and reverse-merge them into the test/dev branches.
Cons:
You have to merge frequently to promote completed features up to the release branch. This isn't too bad though, as this merge is simply a quick copy operation (as long as you don't make edits in the test/release branches, you never have a merge conflict to address)
If you take the approach of a single development branch from which you split off a release branch only when a release is required, then you are effectively working in an unbranched system most of the time. That is, you have an unstable work-in-progress build in your dev branch, which you then copy into a release branch, where you work on it as a work-in-progress until it is stabilised to release quality. If you continue developing features in the dev branch while you bugfix the release branch, you will then get a lot of merge conflicts to resolve. You spend a lot of time without a good build to test, and without a releasable build that you can ship if needed.
There also isn't really much need to branch the code into an archive branch when you release - all you need to do is label the code when you do a release to get a reliable "snapshot" of it that you can recover in future.
Your suggestion isn't too far off the recommendations.
The primary difference in the model is that you're suggesting that you develop directly on what was previously the 'integration' branch. A lot of people suggest taking a branch off this branch to do their work on, then merging back in. But it depends on the size of your team.
An invaluable resource for working with TFS branching is here:
http://tfsbranchingguideiii.codeplex.com/
Your proposal sounds exactly the way I am working:
I develop in the main branch until my developments are ready
Then a release branch is created, let's say release 1.5
After the release branch has been tested, it is branched again, calling it 1.5.1
Bug fixing is done in the main branch and all active release branches (e.g. 1.3, 1.4, 1.5)
Regularly new versions (branches) of the release branches are created and sent to customers (e.g. 1.5.2, 1.5.3, ...)
In my experience this works quite good, but probably it also depends on how your company is organized.
One place I used to work had a branch for each release. We spent more time branching than we did merging. It was a bit excessive.
If you think of a mental model with a graph of connected nodes, the Production branch is just another node connected to the main Integration branch.
In the spirit of keeping my SVN trunk clean and ready for deployment, I've been utilizing the following source control model. For the impatient, the basic concept is that you create development branches to do actual development, and leave the trunk clean and ready for deployment, at any time (no junk in the trunk).
In addition to this, I am configuring TeamCity for continuous integration. Within TeamCity, I'd like to ensure that all development branches, as well as the deployment-ready branch (the trunk, in my case) build correctly and pass all unit tests.
This might be a stupid question, but not being overly familiar with TeamCity, should I create a new TeamCity project for each branch? The deployment-ready branch, in particular, has a few additional rules than the development branch. For example, releases should be saved in versioned directories on the file system (e.g., C:\Projects\MyProject\1.0.187..., C:\Projects\MyProject\1.0.188...) to enable easy access to the binaries, at any point in time. On the other hands, saving versioned copies of the assemblies in the development branches is not necessary and would waste hard disk space.
Within TeamCity, I'd prefer to see only a single project for each software project. In other words, if my company is working on X number of development projects, I'd prefer to see that project listed only once, not X * 2 (assuming each project has only two branches).
You only need to create a single project, but you will need multiple build configurations - 1 for each branch. As far as I know, you can't customize the artifact folder name on disk (it's an auto-increment number), however you can download all artifacts as a zip file in TeamCity 4.5 from the UI. There is also a scheduler included with TeamCity that lets you cleanup artifacts so they don't consume too much disk space.
TeamCity 2018.1.5
TeamCity doesn't support multi branches for SVN as for GIT - so I solved such problem with Configuration parameter - where I set active branch from which I need to build and after can easily switch to another branch by running a custom build or change that configuration parameter.
After need just configures triggers to start building from a specific branch:
So on project side you can see different branches
And easily switch between branches by running Custom Build and change branch there: