I am using Entity Framework code first. I am able to update my database during development using Update-Database from Nuget Package Manager. Now I want to the database to be updated when building from my Azure Pipeline. Is there any documentation from Microsoft of this? How are others handling this?
(Not sure how much information you are looking for, so ask a follow-up/add a comment if this isn't enough.)
Server-side (e.g. in CI pipelines) as well as client-side, this can be done with the CLI instead of the VS specific package manager console:
dotnet ef database update --verbose
See Entity Framework Core tools reference - .NET Core CLI: dotnet ef database update for further information.
Generally, I would discourage anybody from automatically applying database migrations to a production environment.
Migrations should always be scripted to a file via dotnet ef migrations script, thoroughly checked and tested and only then be applied directly from this script (after creating a backup) to a production database.
Related
Context
I'm working a pipeline to manage a dot net core project. In my pipeline, at the build stage I run dotnet ef migrations script --context "MyDbContext" --project "./MySolution/MyDataProject/" --output "./migrationScript.sql" --idempotent to generate scripts which can later be run against the various test, staging, and production environments databases to sync the schema (I use the same approach in non-production environments as in production to ensure those scripts are tested).
It's my understanding that these scripts aren't generated based on the database context, but rather use the migrations which are already defined in the project; with those migrations having been defined via the add command (e.g. dotnet ef migrations add vNext --context "MyDbContext" --project "./MySolution/MyDataProject/"). As such, it's possible that a developer may make changes to the data model, forget to run the migrations add command to add a new migration, and thus create a solution that will fail to update the schema.
Question
Is there a way to test if the migrations are in sync with the code / if a new migration is needed?
Thoughts around Solutions
I'd hoped that by running dotnet ef migrations add when there are no changes to the dbContext there would be no output; so I could test for the presence of a new file and have the pipeline terminate if this had been missed (checking nothing into git; so the new migration isn't persisted / it's left for the developer to manually rerun this).
Looking at the available options, there's only add, remove, list, and script; so nothing obvious for performing this check.
I could run the add command before the script command in my pipeline to ensure it's been run; but then there may be issues with future runs, since if the added migration isn't pushed to git, the next iteration won't be aware of this migration.
If I were to push this new migration to git that resolves that issue; but then creates a new issue that every build creates a new migration, so we'll have a lot of redundant clutter / this approach won't be sustainable.
The best solution I can think of is to run the add command, then inspect the generated scripts to see if the Up or Down methods have anything in the functions' bodies; but that feels hacky; and I'm not certain it's enough (e.g. do the methods in the .Designer.cs file ever change without producing anything in Up/Down in the .cs file?).
I'm sure MS would have created something in the toolset for this; but I'm struggling to find it. Thank-you in advance for any help.
Existing / Similar Answers
In EF Core, how to check whether a migration is needed or not? - This point answers the question about applying migrations, but not about generating them.
Something that appears to work with a few simple test cases I've tried is to add a test migration, and then check if the <ContextName>ModelSnapshot.cs file has changed in the migrations directory.
That file only appears to change when the core migration has been amended.
In Azure DevOps, the way I used to update the SQL Server Database was with Entity Framework Core, using two tasks:
In my Build Pipeline: This task that generated a sql script with my db
migrations.
In the Release Pipeline: This task
to update the database using this script.
The thing is, now that I'm using a PostgreSQL database, I can't find an easy and clean way to update the database in the same way. I've seen there's another task for MySQL that does exactly the same my release pipeline task did with SQL Server, but nothing for PostgreSQL.
So I thought I could basically execute dotnet ef update database (with its proper options set) in the pipeline, but I was wondering if there's actually a way to keep updating the database in a smooth way as I did before.
I finally got to fix it.
There are two solutions I found to fix the issue.
First, there's a common fix for all the databases that support Entity Framework Migrations:
Using a .NET Core Task, we'll have to install the dotnet ef tool:
The task would look like this:
And this would be the YAML (in case you want to use it in out of the release pipeline):
- task: DotNetCoreCLI#2
displayName: 'dotnet custom'
inputs:
command: custom
custom: tool
arguments: 'install --global dotnet-ef --version 3.1.4 --ignore-failed-sources'
And once we have the required tools installed, with a CMD or a Bash Task, we'll have to execute a script like this:
dotnet ef database update -c <DBCONTEXT> -p <PROJECT> -s <STARTUP_PROJECT> -v --no-build
You just have to add the flag -c in case you have more than one context in your project (sometimes the other DbContexts can come from some nugget packages).
Notice I added the flag --no-build since I already built the project in the build pipeline to follow good practices.
The other option (and the one I finally used), it's been to use this task that basically does the same process, with the difference that it does it by using your already compiled .dll files, so you won't have to copy the entire project to make the migrations work. The setup of the task, although you have to fill many inputs, it's pretty straightforward, and it's supposed to work with other Databases as well.
However, if I had to use SQL Server or MySQL I would use a migrations script, since the process it's much easier (you just need to generate a .sql script and then it's the only file required for deploying the migrations).
I use VSTS and Git as source controller and Azure as host to my ASP.NET Core 2.1 API application, I'm trying to have a CI in VSTS, So every things worked except automatically update-database, I used EF core Code first and I need to update DB (in Azure) Automatically in CI process.
But For now, each time I update the connection string in my project(to Azure DB) and run the update-database manually.
For automatic CI process I find:
Using Migration.exe in TFSBuild
How to UPDATE DATABASE in Code First approach
So I search in Debug Console to find package folder irst to set the command to use packages/migration.exe
Microsoft.AspNetCore.AzureAppServices.SiteExtension\store\x86\netcoreapp2.1
I can't find any package folder, or entifyframework libraries
So how could I find the migration.exe file?
Is there any other way to have automatic update-database in VSTS build for EF Code first?
UPDATE:
Here is my nuget reference:
I add the copy file step:
And config is:
Bu I got the Error:
It seems the source folder path is not correct, What should I add as a Source folder?
I recommend that you can do migration during deploy process:
Configure your web core project with EF migration: Migrations
Create a publish profile with Entity Framework Migrations (Settings > Entity Framework Migrations)
Include the publish profile to source control (include pubxml.user file to source control or copy ItemGroup section (<ItemGroup><EFMigrations …) to your publish profile (.pubxml))
Publish web app through Visual Studio Build task (MSBuild Arguments: /p:DeployOnBuild=true /p:PublishProfile=CustomProfile;DesktopBuildPackageLocation="$(build.artifactstagingdirectory)\myapp.zip")
Deploy package (step4) to your site (e.g. through WinRM-IIS Web App Deployment task or Azure App Service Deploy task)
With EF Migrations you can add migration scaffoldings and then keep running 'Update-Database' to apply changes. And in order to rollback the following works:
Update-Database -TargetMigration: <xyzMigration>
This is great for updating your dev. DB. However when I automate migrations using MigrateDatabaseToLatestVersion for deploying to other environments like test and prod. adding migration files for any changes and having them reflect works well.
But, in case I wanted to rollback to a specific migration how do I achieve that? 'Update-Database -TargetMigration:' only updates your local dev. DB or whatever your connection string is pointing to locally. And that has to be run via Package Manager Console. Is rollback not an option for actual deployments? Do you have to just 'Add-Migration' and specifiy a new migration that has all the changes you want reflected?
For a production database doing an Add-Migration with the changes you want would probably make the most sense, especially if you have multiple deployments.
Other options include:
Using DbMigrator.Update() with a parameter that indicates the target migration. (I'd probably use a separate build, just for this purpose...seems tricky to handle...especially when you add additional migrations in the future).
Update-Database also has optional arguments for -ConnectionString, but that would require setting up Visual Studio on a machine with access to the database you're targeting.
I've spent the past few months developing a webApi solution that I'm ready to push up to Azure and hook into an Azure SQL Database. It was built with EF Code First.
I'm wondering what standard approaches there are to making changes to the database while in production. I've been using database initializers up to this point but they all blow away data and re-seed.
I have a feeling this question is too broad for a concise answer, so I'd like to ask: what terminology / processes / resources should a developer look into when designing a continuous integration workflow for a solution built with EF Code First and ASP.NET WebAPI, hosted as an Azure Service and hooked up to Azure SQL?
On the subject of database migration, there was an interesting article on ASP.NET about this subject: Strategies for Database Development and Deployment.
Also since you are using EF Code First you will be able to use Code First Migrations here for database changes. This will allow you to better manage the changes you make to the database.
I'm not sure how far you want to go with continuous integration but since you are using Azure it might be worth it to have a look at Continuous delivery to Windows Azure by using Team Foundation Service. Although it relies on TFS in the cloud it's of course also possible to configure it with for example Jenkins. However this does require a bit more work.
I use this technic:
1- Create a clone database for your development environment if it doesn't exist.
2- Make the necessary changes in your dev environment and dev
database.
3- Deploy to your staging environment.
4- If you added some static datas
that should also exist in your prod database, use a tool like
SQLDataExaminer to find the data differences and execute the
insert, update, deletes for according rows. Use Schema Compare in VS2012 to find differences between your dev
and prod environment by selecting source as dev and target as prod.
And execute the script in your prod.
5- Swap the environments