I've spent the past few months developing a webApi solution that I'm ready to push up to Azure and hook into an Azure SQL Database. It was built with EF Code First.
I'm wondering what standard approaches there are to making changes to the database while in production. I've been using database initializers up to this point but they all blow away data and re-seed.
I have a feeling this question is too broad for a concise answer, so I'd like to ask: what terminology / processes / resources should a developer look into when designing a continuous integration workflow for a solution built with EF Code First and ASP.NET WebAPI, hosted as an Azure Service and hooked up to Azure SQL?
On the subject of database migration, there was an interesting article on ASP.NET about this subject: Strategies for Database Development and Deployment.
Also since you are using EF Code First you will be able to use Code First Migrations here for database changes. This will allow you to better manage the changes you make to the database.
I'm not sure how far you want to go with continuous integration but since you are using Azure it might be worth it to have a look at Continuous delivery to Windows Azure by using Team Foundation Service. Although it relies on TFS in the cloud it's of course also possible to configure it with for example Jenkins. However this does require a bit more work.
I use this technic:
1- Create a clone database for your development environment if it doesn't exist.
2- Make the necessary changes in your dev environment and dev
database.
3- Deploy to your staging environment.
4- If you added some static datas
that should also exist in your prod database, use a tool like
SQLDataExaminer to find the data differences and execute the
insert, update, deletes for according rows. Use Schema Compare in VS2012 to find differences between your dev
and prod environment by selecting source as dev and target as prod.
And execute the script in your prod.
5- Swap the environments
Related
I'm working on a web project(built with the .Net framework) on a remote windows server, and this project is connected to a database my SQL server management studio, now on multiple other remote windows servers exist the same web project linked to the same database, now I change a page's code in my project or add/remove a table or stored procedure in my database, is there a way(or an already existing software) which will my to deploy the changes that I made to all the others(or to choose multiple servers if I don't want to deploy the changes to all of them)?
If it were me, I would stand up a git server somewhere (cloud or local vm), make a branch called something like Prod or Stable, and create a script (powershell if the servers are windows, bash on anything else) on a nightly or hourly job to pull from that branch. Only push to that branch after testing thoroughly. If your code requires compilation, you have the choice to compile once before committing (in which case you're probably going to commit to releases), or on each endpoint after the pull. I would have the script that does the pull also compile and restart the service (only if there was something new in the pull).
You can probably achieve this by following two things :
Create a separate publishing profile for each server.
Use git/vsts branches to keep the code separate. (as suggested by #memtha).
Let's say you have total 6 servers and two branches A and B. So, you'll have to create 6 publishing profiles. Then, you can choose which branch to deploy where. e.g. you can deploy branch B on server 1,3 and 4.
For the codebase you could use Git Hooks.
https://gist.github.com/noelboss/3fe13927025b89757f8fb12e9066f2fa
And for the database, maybe you could use migrations or something similar. You will need to provide more info about your database, do you store your database across multiple servers etc.
If the same web project is connecting to the same database and the database changes, I suspect you would need to update all the web apps to ensure the database changes don't break any of the apps and to keep all the apps updated to prevent any being left behind.
You should look at using Azure Devops to build and deploy your apps and update the database.
If you use Entity Framework, you can run the migrations on startup and have the application update the database when deployed manually or automatically using devops.
To maintain the software updated in multiple server you could use Git with hooks, post-receive hook is what you need.
The idea is to use one server as your Remote Repository and here configure the post-receive hook to update the codebase in the same server and the others.
I want to create one click deployment on azure pipelines to move Postgres Sql changes from dev to QA environment,similar to what we implement using SQL Server Database project where a Powershell script deploy the changes to the remote server.
I have tried pg_dump and psql commands which will create dump file and restore it on the remote server. It does not perform diffing ie(comparing database changes on source and destination , and only replicating the missing changes)
You've stumbled upon one of the features lacking in the Postgres ecosystem. One of the more elegant ways to solve migrations using Postgres' own tooling is to package up your migrations as a Postgres Extension. This requires you to generate the deployment scripts yourself, but it is a neat way of applying and packaging up the deployments.
There are a number of commercial tools that will assist in this process, such as Datical, Liquibase, and Flyway. Note, some of these still require you to generate the change statements yourself, some attempt to create them for you.
Generating change statements is a whole different animal and I recommend you look at schema diffing tools for Postgres to find what best suites your needs.
My company is moving to microservices and as part of this shift devops is standing up CI/CD build release pipelines in Azure with VSTS and GIT.
The current model for managing migrations is a developer is given 2 projects in 2 separate git repositories.
Project 1 - API Services project - .NET Framework / .Net Core
Project 2 - Database project based on EF6 using the migration API
These projects have completely independent release pipelines based on the repositories. So when you create a pull request into master the pipeline builds and releases the project.
This new architecture also supports blue green deployments and our app services run on multiple nodes.
The issue we have is that with this set up we have to basically hand code our migrations and can't use any of the tooling provided in EF Core.
Most of the articles and documentation I have read shows running the migrations from app start up, but if you have multiple app service nodes how do you prevent 2 nodes from running the migrations?
Other articles I have looked at show moving migrations into a separate project, but that project needs a reference to the project with the dbcontext in it. In my company's setup this is not possible. Neither can we do the reverse since moving the dbcontext into the database project prevents us from referencing it in the api services project.
Is there any way to support this model with EF Core?
What is the preferred way to implement blue green deployments with EF Core Migrations on a multi node app service?
I will try to claim that there isn't and not because EF Core doesn't support it in some way, but because this sounds impossible from what I understood in your question.
Your company want the ability to do blue/green deployments, but it is only really possible on the service layer, but not on database. The idea sounds really cool, fast rollback, almost no downtime. But in reality databases complicate things a lot.
So imagine your Project 1 is running on machines A and B (representing blue and green deployments). A currently is a production environment and B is identical but not serving any requests ATM. Both of them are pointing to the exact same database (if not, it's not blue/green, it's just a separate environment). When you want to deploy your DB changes, you migrate your database, but now both of the machines A and B will be pointing to the updated database. You can keep switching from A to B, but they both might have stopped working if your database migration broke anything.
Therefore I don't really understand what you achieve with having DB migrations in a separate repository with a separate pipeline. This just complicates coordination of the releases as they are clearly dependent, but I don't see how it helps to solve anything. As you noted, you can't delegate creation of migration scripts to EF Core, at least without some manual work.
Would be happy to hear any advantages of such design.
We are using RedGate combined with SQL Test (tSQLt). In order to unit test, we install the framework on each database.
Is there a way to use the tSQLt framework in such a way where your unit tests and framework objects can reside in one central location which can then be used by multiple databases?
We are also using RedGate's SQL Source Control with TFS as our repository to track schema changes. These changes get promoted in the following environment order: Development --> Test --> Production.
Needless to say, the addition of the framework combined with the tests themselves represent large amount of new SQL objects (tables, stored procedures, etc) now in our databases. Ideally we would like these objects to reside only in Development and Test and avoid cluttering our production database. We could skip merging the tSQLt changes to Production, but then we would have unmerged changes sitting around in the Test environment's source control until the end of time.
Any thoughts on getting around this problem?
As you're using SQL Source Control to manage your database changes, checking in your tSQLt tests is the right thing to do. If you want to ensure that these don't get pushed to staging or production, you need to ensure that the tools you use to push the changes exclude the tSQLt tests. If you are using Redgate SQL Compare for this, use the option "Ignore tSQLt framework and tests". See the product documentation for a detailed explanation. If you are using a different tool or process, post a comment and I'll amend this answer.
There is currently no way to install tSQLt in a separate database. I have started the process of making tSQLt database agnostic, but that is basically a complete rewrite, so it will take a while.
In the meantime, you can exclude tSQLt from SQL Source Control: https://redgate.uservoice.com/forums/39019-sql-source-control/suggestions/4901910-faster-way-to-exclude-all-tsqlt-content
If you still want your tests in source control but don't want to promote them to the higher environments, that is the default behaviour in Redgate's DLM Automation Suite. You can either use one of the build server plugins (like TeamCity or TFS for build/test then Octopus Deploy for release) or do it all in PowerShell using SQL Release. https://documentation.red-gate.com/display/SR1/SQL+Release+documentation
If you have a license for Redgate's SQL Toolbelt, you might already be licensed for the Automation tools (this is a change to previous licensing); http://www.red-gate.com/products/sql-development/sql-toolbelt/#automation
Here is the story, I am using RedGate SqlCompare to generate update scripts for my Dev env, each package contains only changes from current Dev version to Latest in source control.
Here is an example:
I create a table (package-0.1) -> Deploy to DevDB
I add Columns (package-0.2) -> Deploy to DevDB
I renamed some Column (package-0.3) -> Deploy to DevDB
But once I want to promote it to QA it causes me problem because it promotes only latest package-0.3 that contains only part of the changes (renaming of the column)
So I am looking for a way to deploy all the packages prior to current on Promotion if it is possible.
By now I solved that by creating custom package that contains all the change scripts, but is it possible to solve that with Octopus?
Thanks
Ihor
each package contains only changes from current Dev version to Latest
The way you do it is going to be painful for you as SQL Compare takes a state based approach. What you want to apply is the migrations based approach. You can see Alex's post on the difference between two approaches.
SQL Source Control 5 will come with a better migrations approach which will work with SQL Compare command line tool and DLM Automation tools. However, beta is closed right now unfortunately but I suggest you to contact the team through the e-mail address provided there.
The other option you have is ReadyRoll which has the pure migrations based approach. You can see this post on its octopus deploy integration.