My team and I are using GitFlow to versioning our code, but now we need to have a third branch for DEMO to be able to
publish from DEVELOP in DEMO for testing and then from DEMO to MASTER when all tests are passed
merge a fix from DEMO to DEVELOP
merge a hotfix from MASTER into DEMO and DEVELOP at the same time
here is a draft of the versioning flow (I hope it helps):
I can't find a solution with GitFlow, any advice? Should we choose another tool?
This problem sounds like it has arisen as a result of deployment processes, not Git merging strategy.
There is no concept of a demo branch in GitFlow, though it sounds like you're essentially trying to make use of a release branch. It's worth noting that there is nothing in GitFlow itself to regulate exactly how you deploy any of your branches, and the choice of whether or not to use GitFlow should be made independent of your server architecture.
What it sounds like you're looking for is to deploy out a release branch to an independent testing server, then test against that. Once the tests pass, you can deploy the same commit (on the release branch) out to the production server. This way you don't need to worry about a separate branch specific to testing. Note that master itself should not be deployed to production; it is merely considered a 'reference' to the last known stable version of the code. You should be merging back to master straight after a deployment (and tagging the commit) in case you need to revert back to the target version in the future.
This is illustrated in the standard Git Flow shown below:
To take this a step further, there is also a DevOps testing concept of shifting-left, where the develop branch itself is what is tested against, and considered to be shippable in itself. This simplifies the process even further, and there are pros and cons to choosing such an approach. It may be worth considering after evaluation.
Related
At work, we're now using GitHub, and with that GitHub flow. My understanding of GitHub flow is that there is a master branch and feature branches. Unlike git flow, there is no develop branch.
This works quite well on projects that we've done, and simplifies things.
However, for our products, we have a development and production environment. For the production environment, we use the master branch, whereas for the development environment we're not sure how to do it?
The only idea I can think of is:
When a branch is merged with master, redeploy master using GitHub actions.
When another branch is pushed, set up a GitHub action so that any other branch (other than master) is deployed to this environment.
Currently, for projects that require a development environment, we're essentially using git flow (features -> develop -> master).
Do you think my idea is sensible, and if not what would you recommend?
Edit:
Just to clarify, I'm asking the best way to implement development with GitHub Flow and not git flow.
In my experience, GitHub Flow with multiple environments works like this. Merging to master does not automatically deploy to production. Instead, merging to master creates a build artifact that is able to be promoted through environments using ChatOps tooling.
For example, pushing to master creates a build artifact named something like my-service-47cbd6c, which is a combination of the service name and the short commit hash. This is pushed to an artifact repository of some kind. The artifact can then be deployed to various environments using tooling such as ChatOps style slash commands to trigger the deloy. This tooling could also have checks to make sure test environments are not skipped, for example. Finally, the artifact is promoted to production.
So for your use case with GitHub Actions, what I would suggest is this:
Pushing to master creates the build artifact and automatically deploys it to the development environment.
Test in development
Promote the artifact by deploying to production using a slash command. The action slash-command-dispatch would help you with this.
You might also consider the notion of environments (as illustrated here)
Recently (Feb. 2021), you can:
##Limit which branches can deploy to an environment
You can now limit which branches can deploy to an environment using Environment protection rules.
When a job tries to deploy to an environment with Deployment branches configured Actions will check the value of github.ref against the configuration and if it does not match the job will fail and the run will stop.
The Deployment branches rule can be configured to allow:
All branches – Any branch in the repository can deploy
Protected branches – Only branches with protection rules
Selected branches – Branches matching a set of name patterns
That means you can define a job to deploy in dev environment, and that job, as a condition, will only run if triggered from a commit pushed from a given branch (master in your case)
For anyone facing the same question or wanting to simplify their process away from gitflow, I'd recommend taking a look at this article. Whilst it doesn't talk about Github flow explicitly it does effectively provide one solution to the OP.
Purests may consider this to be not strictly Gitflow but to my mind it's a simple tweak that makes the deployment & CI/CD strategy more explicit in git. I prefer to have this approach rather than add some magic to the tooling which can make a process harder for devs to follow and understand.
I think the Gitflow intro is written fairly pragmatically as well:
Different teams may have different deployment strategies. For some, it may be best to deploy to a specially provisioned testing environment. For others, deploying directly to production may be the better choice...
The diagram in the article sums it up well:
So here we have Master == Gitflow main and the useful addition is the temporary release branch from which you can deploy to other environments such as development. What is worth considering is what you choose to call this temporary branch, in the above it's considered a release, in your process it may be a test branch, etc.
You can take or leave the squashing and tagging and the tooling will change between teams. Equally you may or may not care about actual version numbers.
This isn't a million miles away from VonC's answer, the difference is the process is more tightly defined and it's more towards having multiple developers merge into a single branch & apply fixes in order to get a new version ready for production. It may well be that you configure the deployment of this temporary branch via a naming convention as in his answer.
The way I've implemented this flow is using PRs. I did it with Azure DevOps, but I'd say that the same can be achieved with GitHub Actions.
When you have a branch that you intent to test and eventually merge to master and release to production, you create a PR from that branch to master. The PR will trigger a pipeline, which will run your build, static analysis and tests. If that passes, the PR is deployed to a test environment where further automated and manual testing can happen. That PR can be reviewed and approved by other developers and, if you need to, by QA after manual testing. You can configure GitHub PR rules to enforce the approvals. Once approved, you can merge the PR to master.
What happens once in master is independent of the workflow above, but most likely a new pipeline will be triggered, which will build a release candidate and run the whole path to production (with or without manual intervention).
One of the tricks is how the PR pipeline decides which environment to deploy the PR too. I can think of three options:
Create an environment on the fly which will be killed once the PR is merged or closed. This is the most advanced and flexible option. This would require the system to publish the environment location to the PR.
Have a pool of environments and have the automation figure out which are free and automatically choose one. The environments could be stopped, so you find an environment which is stopped, start it up and deploy there. Once the PR is closed/merged, stop the environment again.You can publish the environment location to the PR.
Add a label to the PR indicating the environment (ie. env-1, env-2, etc.). This is the simplest option, but it requires that developers look at the open PRs to see which environments are already in use in other PRs to avoid overwriting other people's code.
With all these options, once the PR is created, you can just push new commits to the branch and the environment will be updated.
You also need to decide what you want to do when a new commit is pushed to master. You most likely want to trigger a new PR build to update the environments with the latest master, but you can do this automatically or manually, depending on how busy your master is.
Nathan, adding a development branch is good idea, you can work on development changes in new branch and test them in dev environment and after getting signoff to move to production environment you can merge your changes in master branch.
Don't forget to perform regression testing on merged master branch to test both old features and new features are working fine before releasing your code for installation in production
I have been researching and am trying to figure out the best branching and deployment strategy to accomplish the requirements below. Maybe I’m missing something but it is more complicated than it seems. Ideally, we’d just have one permanent branch, ‘master’, that could have specific commits tagged to mark releases to production.
Our current strategy is based on Git Flow and has permanent branches ‘master’ (only has releases to production) and ‘develop’. The primary thing that complicates using a multiple permanent-branches model is the concept of “promoting” the same build from the staging environment to production. Currently, this needs to be done in a separate source code branch (deployments to staging come from ‘develop’, deployments to prod come from ‘master’).
Tools: Git (VSTS), TeamCity, Octopus Deploy
Requirements (feature and hotfix lifecycles):
All code is reviewed via pull requests (enforced via branch policies)
All code gets deployed to a staging environment for testing
We can quickly go back to any snapshot of code that was deployed previously
If testing is successful, then the same build can be “promoted” from our staging environment to production (no need to build again)
Features accumulate over time before pushing out to production as a single release. Hotfixes have to be able to go through without getting caught up in the "all or nothing" next regular release.
I like the idea of having one permanent branch with tags (re: The master/develop split is redundant, http://endoflineblog.com/gitflow-considered-harmful), but having additional permanent branches may better facilitate deploying to different lifecycles/versions (feature and hotfix) to Octopus.
I have been wrestling with how best to pull this off and I may be over complicating things. Any feedback is appreciated.
It seems you have a number of questions and they are quite broad... I'll add some comments to each of your requirements as a conversation starter, but this whole thread might get blocked by moderators as it is definitely not the style of questions SO was made for.
All code is reviewed via pull requests (enforced via branch policies)
I haven't looked at VSTS for ages, but I'd expect they already support branch policies and pull-requests, so not sure if there's anything you need here other than configure settings in your repositories.
In case VSTS does not support that, you might consider moving to a tool that does e.g. BitBucket, GitHub, etc. Both of these have an on-premises version in case you can't (or don't want to) use the cloud hosted version.
All code gets deployed to a staging environment for testing
You achieve that with setting up lifecycles in Octopus Deploy, to make sure deployments/promotions follow the the sequence you want.
We can quickly go back to any snapshot of code that was deployed previously
You already have source control, so all you need now is traceability from the code that is deployed in an environment, to the deployment version in Octopus Deploy, the build job in TeamCity, the branch and exact commit in your source control.
There's a few things that you can do, to achieve that:
Define a versioning scheme that works for you. I like to use semantic versioning. "Major" and "Minor" versions are defined by the developers, and the "Patch" is the auto-incremented number from TeamCity (%build.number%). Every git push build the code and generates a unique build version (%major%.%minor%.%build.number%)
As part of the build steps in TeamCity, before you compile the code, make sure your source files are patched with the version number assigned by each build, the commit hash from your source control, and the branch name. e.g. if you are using .NET, make sure all the AssemblyInfo.cs files are updated with that version, so that the version is embedded in the binaries. This allows anyone to query the version looking at the properties of the binary files, and also allows you to display the app version on the app itself (e.g. status bar, footer, caption, about box, etc.)
Have TeamCity tag your source control with the version number of every build, so you can quickly see on your source control history. You probably only want to do that for the master branch, though which is what you care about.
Have Octopus tag your source control with the deployment version number and the environment name, so that you can quickly see (from your source control) what got deployed where.
Steps 1 and 2 are the most important ones, really. 3 and 4 are just nice-to-have. Most of the time you'll just open the app in the environment, check the commit hash in the "About", and do a git checkout to that commit hash...
If testing is successful, then the same build can be "promoted" from our staging environment to production (no need to build again)
Again, Octopus Deploy lifecycles, and make sure anything different in each environment is defined in the configuration file of the application, which is updated during the Octopus deployment, using environment-specific variables.
In terms of branch workflow, this last requirement makes it mandatory to merge changes into master (or whatever your "production" branch is) before the deployment lifecycle can begin.
I was recently reading the TFS Branching Guide and it suggests a branch for every release. For a web site, there is only one "version" released at a time. In that case is it appropriate to have a single "Production" branch? Then, during the process of preparing for a release, you merge changes from the Main branch into Production. (As opposed to the suggestion to branch each release.) If you need to do a hotfix, do it in the Production branch, then reverse integrate into Main. Doing it this way allows you to keep configuration files for Production intact in the Production branch.
p.s. I should mention that we are using a Code Promotion model.
p.p.s. Apparently what I'm talking about exists: it's called a "staging stream" in Practical Perforce
Usually, Production is for reflecting what is in production, that is:
what has been pushed in the live site
plus whatever hotfixes have been made directly in the live site
That is why, in this configuration, one Production branch is enough.
You will then need, following that logic:
release branches where you consolidate your developments and test them before merging the current release to the Production branch. They can serve as source for a staging environment, since, as mentioned in the document you reference in your question (Practical Perfoce):
it allows you do make extremely frequent releases without having to branch a new codeline for each release. (They’re commonly used to support web development)
A staging stream is essentially a reusable release codeline. Each staging stream is used for a particular stage of release stabilization.
development branches for daily devs (all of which cannot always be part of the next release), and for integrating back the hotfixes made in Prod (merge Prod to Dev).
I don't know if it's "appropriate" but I do something similar with Subversion...
Instead of branches/tags/trunk, I have development/test/production. New features/fixes are created in development. Once completed, they are merged to test for testing and client review (by going to the test website). Once passed QA the changes are merged to production. Hook scripts automatically update corresponding development, test, and production websites on checkin, and each "branch" has its own unique web.config file pointing to the appropriate development/test/production database.
What is your question?
This is currently how we use TFS (which btw is ace)
We are reaching a point in our project where we need to make a production deployment but also need to have ongoing development for future features. Our source control currently has a single development branch. In my previous company a 3 branch system was set up with Development, Integration, Production. Feature development was done in Development, tesing in Integration and Production always was the code running in the current production environment (except for a breif time when a merge was done from Integaration into Production and the deployment was done).
Each time there was a Production change or a merge into Production that was deployed live, a new archive branch was taken off Production and given a version number. Any change to a higher branch was merged back so changes in Production would make it back to Development for example. It worked but I always didn't see the need for an ongoing Integration and Production branch.
Two aspects of this system I really didn't like: 1) the merge from Integration branch into Production branch: I would prefer to have a "clean" Production branch each time even though they should be in sync after a merge and 2) this system doesn't allow for multiple deployments of the system running different versions of the code at the same time, though this has not been a requirement in either team I have worked for (yet).
I have heard this model is common but in the system I am setting up now I am proposing the following:
Have a Development branch, create a new Release branch each time Development is ready for the next deployment to production. The Release branches are given a version number and then branched again to an archive branch. Testing is done on the Release branch. Once deployed any production fixes are done in the Release branch then a new archive branch is created with a minor version number increment once deployment is approved. When a new deployment from Development is ready the a new Release branch is created...
To me this is simpler and is actually better: There is NO Integaration branch (less merging) and there is a fresh Production (Release) branch for each deployment and caters for current Production versions. What am I missing? Why go for the Development, Integration, Production model?
Thanks
The advantage of the 3-branch system is that your developers, testers, and customers each have their own isolated version/branch of the code. This provides a "gated" development where features can only be promoted to the next branch when they pass a certain completeness/quality bar, giving high confidence that the release branch is always in a fit state to send to a customer.
Pros:
You (almost) always have a build that could be sent to a customer at a moments notice.
Your testers (almost) always have a working build to test
The release branch is stable so you don't often have to fix bugs in it and reverse-merge them into the test/dev branches.
Cons:
You have to merge frequently to promote completed features up to the release branch. This isn't too bad though, as this merge is simply a quick copy operation (as long as you don't make edits in the test/release branches, you never have a merge conflict to address)
If you take the approach of a single development branch from which you split off a release branch only when a release is required, then you are effectively working in an unbranched system most of the time. That is, you have an unstable work-in-progress build in your dev branch, which you then copy into a release branch, where you work on it as a work-in-progress until it is stabilised to release quality. If you continue developing features in the dev branch while you bugfix the release branch, you will then get a lot of merge conflicts to resolve. You spend a lot of time without a good build to test, and without a releasable build that you can ship if needed.
There also isn't really much need to branch the code into an archive branch when you release - all you need to do is label the code when you do a release to get a reliable "snapshot" of it that you can recover in future.
Your suggestion isn't too far off the recommendations.
The primary difference in the model is that you're suggesting that you develop directly on what was previously the 'integration' branch. A lot of people suggest taking a branch off this branch to do their work on, then merging back in. But it depends on the size of your team.
An invaluable resource for working with TFS branching is here:
http://tfsbranchingguideiii.codeplex.com/
Your proposal sounds exactly the way I am working:
I develop in the main branch until my developments are ready
Then a release branch is created, let's say release 1.5
After the release branch has been tested, it is branched again, calling it 1.5.1
Bug fixing is done in the main branch and all active release branches (e.g. 1.3, 1.4, 1.5)
Regularly new versions (branches) of the release branches are created and sent to customers (e.g. 1.5.2, 1.5.3, ...)
In my experience this works quite good, but probably it also depends on how your company is organized.
One place I used to work had a branch for each release. We spent more time branching than we did merging. It was a bit excessive.
If you think of a mental model with a graph of connected nodes, the Production branch is just another node connected to the main Integration branch.
Describe the process you use to develop web applications at a not-so-high level, focusing on VC, bug tracking, QA, unit testing, deployment and anything else similar (minus the planning/client communication side of things).
I'm new in this area, so my rough example (read: haven't used this process) is no doubt abit off, so to speak - point out it's flaws so I can learn.
Eg.
Create project repository on local SVN server.
Create batch/shell scripts for DNS mappings.
Check out project, begin work on local working copy.
Develop features as branches.
Track bugs with Mantis (link commits to bugs through it's SVN integration (no idea if that exists)).
Document as you go.
Do QA on branch.
Merge to trunk when stable.
Unit Testing?
Commit to repository when feature is implemented and stable.
Copy releases to tags in repository. Eg. /project/tags/rel-123/
Use Phing to upload to staging server. (Could someone please clarify exactly what a staging server is used for beyond 'testing'?)
Use Phing to prep live site for update, set up DB/deploy, etc.
Create/checkout HEAD version ("main branch")
Develop code and sync with the main branch -at least- daily
After development is done, write and run unit tests
Go through code review and submit code/changes to the main branch
Let continuous builder run all unit tests and system/integration tests on main branch
When ready, cherry pick revisions and integrate them to the QA branch
Run system and integration tests, fix reported bugs, or rollback as necessary; this repeats steps 4-7
After QA signoff, integrate QA change to release branch
Run unit tests, system/integration tests on release branch
Deploy to production and run sanity tests.
A staging server is a copy of your production environment that is as up-to-date as possible. On my current project, we're able to keep each release independent from each other, so our "staging server" is our production server, just accessed from a different url.
Notes and discreprencies:
All of the steps have some variation depending on the size of your project. The larger your project, the better the benefit from cherry picking and environment separation. In smaller projects, these can just be time sinks and are often ignored or bypassed.
To clarify, there is a Development stack, QA stack, and Staging stack. Depending on your project size, QA might be staging, Production might be staging, or some combination thereof. The separation of the Dev and QA stacks is the most important one.
In the steps above, I'm assuming both code and relevant data is versioned or tracked. Having a release and build process that takes control data into account makes a release very, very easy.
In a small-medium sized project, there may or may not be a release branch; it depends on the frequency of code change. Also, depending on the frequency of code change and size of your project, you may integrate the full QA branch to the Release branch, or cherry pick specific revisions to integrate to the release branch.
FWIW, I've found "migration scripts" to be of little value. They're always a one-off script with little reuse and make rollbacks a pain in the ass. Its much easier, I would argue better, to have the application backwards-compatible. After a few releases (when a rollback is a laughable), data cleanup should be done, if necessary.
Very roughly:
Create repository in SVN
Checking local working copy to developer environment
Update/commit changes frequently
Deploy to stage from SVN trunk using custom deploy script
QA tests on stage, reports bugs in Mantis
Developers fix bugs, mark as resolved
Re-deploy to stage
QA tests bugs, closes if fixed
QA is finished, do regression testing
Deploy to production using custom deploy script
Do a little dance
We also create branches for future versions or features. These eventually get merged into the trunk.
We keep our db structures synchronized with a custom db comparison tool that is executed during the deploys.
Old post, but interesting question !
At my company now :
Create a new Github repo
Configure Jenkins
Clone locally
Start a branch
Develop and add tests (server, client and e2e)
Commit for every step, and fetch + rebase to keep the branch in sync
When ready, push the branch to the server : a pre-commit check lint and tests and block if not ok
Create a pull request for the branch
Here, jenkins automatically runs tests on the branch and flag it as "green" or "broken tests" directly in the pull request
Have at last 2 colleagues review the pull request and fix their findings (back to step 5)
When everything green and 2 colleagues have agreed, the last one merges the pull request
Delete the branch on server
When ready, push a new version
Latest version get immediately deployed on a testing platform
QA validate the corrections and features introduced (back to 5 if problem)
(TODO) deploy to a pre-prod with identical parameters than the production
Deploy to production
Go apologize to users for the bugs introduced ;) and report them in the issue manager
get feature requests and report them in the issue manager
restart cycle at step 2