When we deploy to our Testing environment with Sqitch, the migrations run and all is well. However, when we have to test a different code branch, the Sqitch migrations aren't rolled back. What I'm looking for is an idea of how others are solving this problem. For example, if I have a migration that creates a table my_table, and then deploy a migration from a different branch, Sqitch gets too confused to move forward.
Reason for branch change: we have features using our develop branch, and hotfixes being applied to master (I know; we're going to change that). Swapping over to test a hotfix caused all of this mess.
We considered having a Jenkins job to run a sqitch revert and have the user supply the migration to roll back to, but we want to eliminate the manual step.
How do you manage migrations when having to switch code branches?
I believe this is the purpose of sqitch-checkout
Related
Consider project with different language modules that use same database. Database is controlled by liquibase. Liquibase changeLog files placed in special github repository. So every developer could download it and run locally to keep local instance of database in actual state.
Often developers use inmemory database like H2 for testing and liquibase rebuilds database structure every test run. To keep testing database in actual state every developer need to download changeLogs and put in their modules manually. But sometimes he could miss that somebody else changed database structure, so his tests should fail, but will pass. This is more actual for testing in CI.
Could liquibase download changelogs automatically from internet source? Could it use private github repositories? Could it use some property like changeLogFile for this?
May I confirm some assumptions? It sounds you have a team of developers all commiting liquibase database changes and you want to be able to automatically deploy in a CI/CD process?
Thanks,
Ronak
Liquibase Developer Evangelist
Liquibase can't run changelogs placed into some remote git repo. But in your CI you can setup such thing: if there is a pull-request from branch 'feature' to branch 'master' automatically:
Before tests are executed checkout branch 'feature'
and merge 'master' there
Apply all changelogs
Run tests after it
So if another developer change DB structure and commit it into master tests in 'feature' branch will be executed with latest DB structure.
I want to add Build and Release pipelines in Azure Devops for my .Net Core application. Application uses EF Core migrations.
I used the solution provided for example here: https://blog.clear-measure.com/2019/01/07/run-ef-core-migrations-in-azure-devops/
It works fine. I generate script during build, publish it as another artifact and then use it to run against database during release step.
But how to deal with a situation that I want to revert the changes. Here I see two scenerios:
It would be great if somehow I could just click Deploy on the previous release and this way get back to the previous working version of the app (and revert the DB as well).
Revert the changes in the repo, deleting migrations that might have been created between releases and start build and deploy process again. How to revert already added migration to the DB?
Is even the first approach possible ?
Thanks for you help!
You would revert the migration via EF Core similar to the way you add a migration. Then commit and push and run the pipeline.
https://learn.microsoft.com/en-us/ef/core/managing-schemas/migrations/#revert-a-migration
First scenario is possible.
In your release pipeline, you can find the retained historical releases.
Select the previous release you want to revert back to, click redeploy and it will roll back to the previous version.
I have been researching and am trying to figure out the best branching and deployment strategy to accomplish the requirements below. Maybe I’m missing something but it is more complicated than it seems. Ideally, we’d just have one permanent branch, ‘master’, that could have specific commits tagged to mark releases to production.
Our current strategy is based on Git Flow and has permanent branches ‘master’ (only has releases to production) and ‘develop’. The primary thing that complicates using a multiple permanent-branches model is the concept of “promoting” the same build from the staging environment to production. Currently, this needs to be done in a separate source code branch (deployments to staging come from ‘develop’, deployments to prod come from ‘master’).
Tools: Git (VSTS), TeamCity, Octopus Deploy
Requirements (feature and hotfix lifecycles):
All code is reviewed via pull requests (enforced via branch policies)
All code gets deployed to a staging environment for testing
We can quickly go back to any snapshot of code that was deployed previously
If testing is successful, then the same build can be “promoted” from our staging environment to production (no need to build again)
Features accumulate over time before pushing out to production as a single release. Hotfixes have to be able to go through without getting caught up in the "all or nothing" next regular release.
I like the idea of having one permanent branch with tags (re: The master/develop split is redundant, http://endoflineblog.com/gitflow-considered-harmful), but having additional permanent branches may better facilitate deploying to different lifecycles/versions (feature and hotfix) to Octopus.
I have been wrestling with how best to pull this off and I may be over complicating things. Any feedback is appreciated.
It seems you have a number of questions and they are quite broad... I'll add some comments to each of your requirements as a conversation starter, but this whole thread might get blocked by moderators as it is definitely not the style of questions SO was made for.
All code is reviewed via pull requests (enforced via branch policies)
I haven't looked at VSTS for ages, but I'd expect they already support branch policies and pull-requests, so not sure if there's anything you need here other than configure settings in your repositories.
In case VSTS does not support that, you might consider moving to a tool that does e.g. BitBucket, GitHub, etc. Both of these have an on-premises version in case you can't (or don't want to) use the cloud hosted version.
All code gets deployed to a staging environment for testing
You achieve that with setting up lifecycles in Octopus Deploy, to make sure deployments/promotions follow the the sequence you want.
We can quickly go back to any snapshot of code that was deployed previously
You already have source control, so all you need now is traceability from the code that is deployed in an environment, to the deployment version in Octopus Deploy, the build job in TeamCity, the branch and exact commit in your source control.
There's a few things that you can do, to achieve that:
Define a versioning scheme that works for you. I like to use semantic versioning. "Major" and "Minor" versions are defined by the developers, and the "Patch" is the auto-incremented number from TeamCity (%build.number%). Every git push build the code and generates a unique build version (%major%.%minor%.%build.number%)
As part of the build steps in TeamCity, before you compile the code, make sure your source files are patched with the version number assigned by each build, the commit hash from your source control, and the branch name. e.g. if you are using .NET, make sure all the AssemblyInfo.cs files are updated with that version, so that the version is embedded in the binaries. This allows anyone to query the version looking at the properties of the binary files, and also allows you to display the app version on the app itself (e.g. status bar, footer, caption, about box, etc.)
Have TeamCity tag your source control with the version number of every build, so you can quickly see on your source control history. You probably only want to do that for the master branch, though which is what you care about.
Have Octopus tag your source control with the deployment version number and the environment name, so that you can quickly see (from your source control) what got deployed where.
Steps 1 and 2 are the most important ones, really. 3 and 4 are just nice-to-have. Most of the time you'll just open the app in the environment, check the commit hash in the "About", and do a git checkout to that commit hash...
If testing is successful, then the same build can be "promoted" from our staging environment to production (no need to build again)
Again, Octopus Deploy lifecycles, and make sure anything different in each environment is defined in the configuration file of the application, which is updated during the Octopus deployment, using environment-specific variables.
In terms of branch workflow, this last requirement makes it mandatory to merge changes into master (or whatever your "production" branch is) before the deployment lifecycle can begin.
We are developing a product and we have two branches - main and dev. Dev is work towards next release. Main contains last release and any urgent bug fixes.
Our database is defined in our web.config. We are using EF6 with migrations. We have strategies to deal with migrations in a current branch without that much issue. The problem is switching between branches. If we work on both branches the migrations aren't right for the main branch generally as dev has newer migrations. So we have to roll back our db from dev before starting work in main.
Is there a way in TFS to permanently prevent merging the database connection string between branches so that we can have one DB for production and one DB for dev on our local machines?
We have same setup as you and use deployment parameters to change connection strings and other settings. Web.config transforms are another option.
Describe the process you use to develop web applications at a not-so-high level, focusing on VC, bug tracking, QA, unit testing, deployment and anything else similar (minus the planning/client communication side of things).
I'm new in this area, so my rough example (read: haven't used this process) is no doubt abit off, so to speak - point out it's flaws so I can learn.
Eg.
Create project repository on local SVN server.
Create batch/shell scripts for DNS mappings.
Check out project, begin work on local working copy.
Develop features as branches.
Track bugs with Mantis (link commits to bugs through it's SVN integration (no idea if that exists)).
Document as you go.
Do QA on branch.
Merge to trunk when stable.
Unit Testing?
Commit to repository when feature is implemented and stable.
Copy releases to tags in repository. Eg. /project/tags/rel-123/
Use Phing to upload to staging server. (Could someone please clarify exactly what a staging server is used for beyond 'testing'?)
Use Phing to prep live site for update, set up DB/deploy, etc.
Create/checkout HEAD version ("main branch")
Develop code and sync with the main branch -at least- daily
After development is done, write and run unit tests
Go through code review and submit code/changes to the main branch
Let continuous builder run all unit tests and system/integration tests on main branch
When ready, cherry pick revisions and integrate them to the QA branch
Run system and integration tests, fix reported bugs, or rollback as necessary; this repeats steps 4-7
After QA signoff, integrate QA change to release branch
Run unit tests, system/integration tests on release branch
Deploy to production and run sanity tests.
A staging server is a copy of your production environment that is as up-to-date as possible. On my current project, we're able to keep each release independent from each other, so our "staging server" is our production server, just accessed from a different url.
Notes and discreprencies:
All of the steps have some variation depending on the size of your project. The larger your project, the better the benefit from cherry picking and environment separation. In smaller projects, these can just be time sinks and are often ignored or bypassed.
To clarify, there is a Development stack, QA stack, and Staging stack. Depending on your project size, QA might be staging, Production might be staging, or some combination thereof. The separation of the Dev and QA stacks is the most important one.
In the steps above, I'm assuming both code and relevant data is versioned or tracked. Having a release and build process that takes control data into account makes a release very, very easy.
In a small-medium sized project, there may or may not be a release branch; it depends on the frequency of code change. Also, depending on the frequency of code change and size of your project, you may integrate the full QA branch to the Release branch, or cherry pick specific revisions to integrate to the release branch.
FWIW, I've found "migration scripts" to be of little value. They're always a one-off script with little reuse and make rollbacks a pain in the ass. Its much easier, I would argue better, to have the application backwards-compatible. After a few releases (when a rollback is a laughable), data cleanup should be done, if necessary.
Very roughly:
Create repository in SVN
Checking local working copy to developer environment
Update/commit changes frequently
Deploy to stage from SVN trunk using custom deploy script
QA tests on stage, reports bugs in Mantis
Developers fix bugs, mark as resolved
Re-deploy to stage
QA tests bugs, closes if fixed
QA is finished, do regression testing
Deploy to production using custom deploy script
Do a little dance
We also create branches for future versions or features. These eventually get merged into the trunk.
We keep our db structures synchronized with a custom db comparison tool that is executed during the deploys.
Old post, but interesting question !
At my company now :
Create a new Github repo
Configure Jenkins
Clone locally
Start a branch
Develop and add tests (server, client and e2e)
Commit for every step, and fetch + rebase to keep the branch in sync
When ready, push the branch to the server : a pre-commit check lint and tests and block if not ok
Create a pull request for the branch
Here, jenkins automatically runs tests on the branch and flag it as "green" or "broken tests" directly in the pull request
Have at last 2 colleagues review the pull request and fix their findings (back to step 5)
When everything green and 2 colleagues have agreed, the last one merges the pull request
Delete the branch on server
When ready, push a new version
Latest version get immediately deployed on a testing platform
QA validate the corrections and features introduced (back to 5 if problem)
(TODO) deploy to a pre-prod with identical parameters than the production
Deploy to production
Go apologize to users for the bugs introduced ;) and report them in the issue manager
get feature requests and report them in the issue manager
restart cycle at step 2