I'm currently working on a project that we are using Prisma 2 and postgreSQL as database support. From my understanding, whenever I made changes to the schema.prisma file and I want to migrate the changes to the database, I run prisma migrate dev locally. Then I will push the auto created migration files under migration folder to Github, and then our repo will run prisma migrate deploy to the staging or production server.
So, my concern is, each time I run prisma migrate dev, a new migration file will be created in migrations folder. So, there will be a lot of migration files under the migration folder with the development of the project. Is that what supposed to happen? Or is there a better way?
Thank you for your help. I'm pretty new to Prisma 2 and still trying to find the correct way to work with it. BTW I think Prisma 1 was easier to use :)
I’d recommend using prisma db push while developing locally and generating the migration with prisma migrate dev only when you think the schema changes are good to go. More at https://www.prisma.io/docs/guides/database/prototyping-schema-db-push
So, there will be a lot of migration files under the migration folder
with the development of the project. Is that what supposed to happen?
Yes, this is exactly what's supposed to happen. The migration file contains the history of all changess to your Prisma schema (and underlying database tables/configuration), so they all need to be retained.
I'm not sure if you could go around this somehow, but it's certainly not the recommended way to be using migrate.
You can find a list of best practices regarding migration in the Prisma migrate article in the docs.
It's been a very long time since the question is asked, but i want to warn about something:
As you may notice a table named _prisma_migrations is created by prisma automaticly and every time you add a new migration you are able to see the new migration's data over there. So any manual change made in migrations folder can cause problems since it breaks the relations between folder names and _prisma_migrations table.
Also docs for "squashing migrations": https://www.prisma.io/docs/guides/database/developing-with-prisma-migrate/squashing-migrations
Related
I'm using efcore and I scaffolded from an existing database, but I forgot to make an initial migration when I did that so it picked up my new model. I decided that I wanted to keep them separate so I removed the migration, deleted the snapshot, and now every time I create another migration, it still has the new model in it.
On top of this, using dotnet ef migrations list will list pending migrations that no longer exist. If I do dotnet ef database update they will get 'migrated` and show up in __MigrationHistory, but again, they don't actually exist in the directory.
I've tried creating the files again, going as far as adding the class names to the files, doing a dotnet ef migrations remove, but I still have the same issue. It will delete the files, tell me everything is rolled back, but then the files will still be listed as pending, the database snapshot will still contain the new table (that doesn't exist) if I do any future migrations.
Not really sure how to proceed. Everything I've read says that if I delete the snapshot and migrations, it will create a fresh snapshot, but it doesn't seem to do that. Is this stuff cached somewhere locally that I can clear out?
Of course after posting this, I figured it out.
My issue was that I kept running dotnet ef commands with the --no-build flag because they would fail. I know why they were failing, it wasn't a big deal, but apparently this prevented the snapshot from updating as well.
So I fixed it by taking the offending files out of my folder for the moment, running dotnet ef migrations list, and with a successful build, it fixed all of the issues I was having.
Context
I'm working a pipeline to manage a dot net core project. In my pipeline, at the build stage I run dotnet ef migrations script --context "MyDbContext" --project "./MySolution/MyDataProject/" --output "./migrationScript.sql" --idempotent to generate scripts which can later be run against the various test, staging, and production environments databases to sync the schema (I use the same approach in non-production environments as in production to ensure those scripts are tested).
It's my understanding that these scripts aren't generated based on the database context, but rather use the migrations which are already defined in the project; with those migrations having been defined via the add command (e.g. dotnet ef migrations add vNext --context "MyDbContext" --project "./MySolution/MyDataProject/"). As such, it's possible that a developer may make changes to the data model, forget to run the migrations add command to add a new migration, and thus create a solution that will fail to update the schema.
Question
Is there a way to test if the migrations are in sync with the code / if a new migration is needed?
Thoughts around Solutions
I'd hoped that by running dotnet ef migrations add when there are no changes to the dbContext there would be no output; so I could test for the presence of a new file and have the pipeline terminate if this had been missed (checking nothing into git; so the new migration isn't persisted / it's left for the developer to manually rerun this).
Looking at the available options, there's only add, remove, list, and script; so nothing obvious for performing this check.
I could run the add command before the script command in my pipeline to ensure it's been run; but then there may be issues with future runs, since if the added migration isn't pushed to git, the next iteration won't be aware of this migration.
If I were to push this new migration to git that resolves that issue; but then creates a new issue that every build creates a new migration, so we'll have a lot of redundant clutter / this approach won't be sustainable.
The best solution I can think of is to run the add command, then inspect the generated scripts to see if the Up or Down methods have anything in the functions' bodies; but that feels hacky; and I'm not certain it's enough (e.g. do the methods in the .Designer.cs file ever change without producing anything in Up/Down in the .cs file?).
I'm sure MS would have created something in the toolset for this; but I'm struggling to find it. Thank-you in advance for any help.
Existing / Similar Answers
In EF Core, how to check whether a migration is needed or not? - This point answers the question about applying migrations, but not about generating them.
Something that appears to work with a few simple test cases I've tried is to add a test migration, and then check if the <ContextName>ModelSnapshot.cs file has changed in the migrations directory.
That file only appears to change when the core migration has been amended.
I have copy of prod DB and code on my DEV PC.
On production I saw in system tables that 8 migrations records exist in _MigrationHistory. (The first version of the code and deploy was written by another person).
I need to add few new tables and code for work with them - models and controller. I can add new models for new tables and code for controller.
On the DEV PC I can start command
add-migration myNewMigration
and
update-database with option "-script"
and without it to get tables in my DEV DB and script to execute on Production.
For deploy I need to copy content on bin folder and new views on Prod and to run SQL script to create new tables in DB.
I was wondering is that enough because at the moment I see records for all migrations in my DEV environment corresponding records in Production environment in system tables - _MigrationHistory.
If I only run the SQL script on PROD may be new record for this operation will not appear in _MigrationHistory. So could be that a reason to get error that model is different and to have any problem after deploy.
Before making changes to the production database, ensure you have a complete backup.
I was wondering is that enough because at the moment I see records for all migrations in my Dev environment corresponding records in Production environment in system tables - _MigrationHistory
If you ran the code in your development environment, it will apply your explicit migrations.
If I understand your comment correctly, you see the same migrations in DEV and in PROD. If that is accurate, then PROD may be up-to-date.
Look in your Migrations folder in your project. There you will see a complete list of migrations, each one in its own .cs file (plus related .Designer.cs and .resx files). Compare what you see there to the list of migrations in __MigrationHistory in DEV and in PROD to understand the full set of migrations, and where they have been applied.
If I only run sql script on Prod may be new record for this operation will not appear in _MigrationHistory
Have a look at your migration scripts. They insert the appropriate entry into __MigrationHistory where they are run (including in production). Here's an edited example from one of my projects:
INSERT [dbo].[__MigrationHistory]([MigrationId], [ContextKey], [Model], [ProductVersion])
VALUES (N'201409082209285_VDisposer', N'Survey.Model.Migrations.MembershipConfiguration.MembershipConfiguration', 0x1F8B0800000...1A0200 , N'6.1.1-30610')
So could be that a reason to get error that model is different and to have any problem after deploy
If you are having an error that the model does not match the database, there is another reason.
Here's what I suggest to narrow down the differences:
Backup your production database.
Restore it in your DEV environment (use a name different than your usual DEV database, or backup your DEV database first. If using a different name, remember to temporarily change web.config to the new database).
Create a new migration.
Look at the code generated for the new migration. It will point out differences between what is in PROD and the state that DEV is in.
Remember to undo changes to your web.config after you are done.
I've spent the past few months developing a webApi solution that I'm ready to push up to Azure and hook into an Azure SQL Database. It was built with EF Code First.
I'm wondering what standard approaches there are to making changes to the database while in production. I've been using database initializers up to this point but they all blow away data and re-seed.
I have a feeling this question is too broad for a concise answer, so I'd like to ask: what terminology / processes / resources should a developer look into when designing a continuous integration workflow for a solution built with EF Code First and ASP.NET WebAPI, hosted as an Azure Service and hooked up to Azure SQL?
On the subject of database migration, there was an interesting article on ASP.NET about this subject: Strategies for Database Development and Deployment.
Also since you are using EF Code First you will be able to use Code First Migrations here for database changes. This will allow you to better manage the changes you make to the database.
I'm not sure how far you want to go with continuous integration but since you are using Azure it might be worth it to have a look at Continuous delivery to Windows Azure by using Team Foundation Service. Although it relies on TFS in the cloud it's of course also possible to configure it with for example Jenkins. However this does require a bit more work.
I use this technic:
1- Create a clone database for your development environment if it doesn't exist.
2- Make the necessary changes in your dev environment and dev
database.
3- Deploy to your staging environment.
4- If you added some static datas
that should also exist in your prod database, use a tool like
SQLDataExaminer to find the data differences and execute the
insert, update, deletes for according rows. Use Schema Compare in VS2012 to find differences between your dev
and prod environment by selecting source as dev and target as prod.
And execute the script in your prod.
5- Swap the environments
I have been using EF migrations for some time now and have more than 100 migration files in my project. I would like to consolidate these into a single migration before moving forward - ie I want to replace the existing InitialCreate migration with a new version that takes all my subsequent changes into account so I can then delete all the other migration files.
I do this very easily if I am not concerned with losing all the data in the DB, but I am.
How can I achieve this whilst keeping all data intact and also retaining the ability to recreate the database from scratch (without data) by just running Update-Database (which I believe is not possible using the approach outlined by Julie Lerman)?
Consider reading this nice article from Rick Strahl :
https://weblog.west-wind.com/posts/2016/jan/13/resetting-entity-framework-migrations-to-a-clean-slate
Basically the solution is not trivial and needs more than just reseting all the migrations into one
because you have two scenarios that needs to fit in ONE migration class:
Create a new database => the migration class should contain every table creation
My database is already up to date => I need an empty migration class
Solution:
The idea of this process is basically this: The database and the EF schema are up to date and just the way you want it, so we are going to remove the existing migrations and create a new initial migration.
In summary, the steps to do this are:
Remove the _MigrationHistory table from the Database
Remove the individual migration files in your project's Migrations folder
Enable-Migrations in Package Manager Console
Add-migration Initial in PMC
Comment out the code inside of the Up method in the Initial Migration
Update-database in PMC (does nothing but creates Migration
Entry) Remove comments in the Initial method You've now essentially
reset the schema to the latest version.
once the commented out migration has been executed on the desired database, uncomment the migration code
If you're not concerned with keeping this migrations, what I've done is delete everything in your migrations folder, and then target a new database in the connection string (or pass in a new one). After that, you can just run the add-migration command:
add-migration InitialCreate
And it should create the migration for you.
Below procedure has the benefit of working without doing anything with the DBs, __MigrationHistory can stay as-is. Also it will work if you have multiple different environments with different versions of the structure - provided you have the branches to match.
I turn the last migration into an initial migration. The trick is to use the oldest version of the code and DB that is in use, replace its last migration with a new initial migration and delete all previous migrations. Newer branches keep the more recent migrations so those will still work after merging to older branches.
So start in the OLDEST branch - PROD, normally - and do:
Remove all but the last migration
Remove the migration code in both the "Up" and "Down" methods in the last migration
Change build action of the last migration to "None" to let EF ignore it
Change active connection to point to a local DB database.
Make sure this local DB database does not exist
add-migration Initial
Copy Up and Down code from the created "Initial" migration to the last migration
Delete Initial migration
Change build action of the last migration back to "Compile"
Check in
Merge changes up
Test in DEV branch on LocalDB DB - it should do the new initial migration as well as the subsequent ones with no issues
Test in main branch on the latest DB - it shouldn't do anything
Note above only works if you don't add stuff to the migrations that EF doesn't do itself. E.g. if you add DB views etc. than the newly created migration won't get those, it only gets the scripts EF generates based on your code.
Removing all migrations or regenerating them has drawbacks so we took an approach we merged all older migrations.
It require a bit of scripting. You read about the details here https://www.bokio.se/engineering-blog/how-to-squash-ef-core-migrations/ and see the scripts here https://github.com/bokio/EFCoreTools/tree/main/MigrationSquasher
The basic is the following steps though (copied from the blog post):
Overview of our approach
Create a new fresh database from the old migrations (We will use
this for comparison later)
Find a suitable target migration to be
the new initial (We picked one about 3 months old)
Write a script to
merge the Up() methods of all earlier migrations into the Up()
method of a new migration. We ignored Down() because we don't use it
for old migrations.
Generate this migration and add it to the
project. In our case we called it
20200730130157_SquashedMigrations1.cs. We used the snapshot from the
target migration we had picked.
Generate a 2nd prep migration that
inserts into the migrations history that
20200730130157_SquashedMigrations1.cs has already run. We called
this 20200730130156_SquashedMigrations1_prep.cs. Note the slightly
smaller timestamp on that one to make sure it runs before the real
migrations.
Delete the old migrations
Point our config to a new
database and run the migrations.
Compare that the schema we generate
is equal using the Sql Schema Compare in Visual Studio.
Work through
the issues until we have equal schemas. This part is a bit
complicated but I will get back to it.
Merge and 🤞 (Ok, we did run
more tests both on local and staging databases)
I hope this helps someone else. The EF team is looking at improving this story so if you have feedback on your requirements it probably help them to post that now https://github.com/dotnet/efcore/issues/2174.
We had the same problem.
The general solution we found was to
Archive the old migration / context on a Nuget Package (including dependancies and with a different namespace to avoid conflicts)
Delete all migrations and create a new Init (InitV2) Migration from scratch.
Change the startup sequence of our application:
If the database contains the first old Init migration then
Migrate the database using the Nuget Package to be sure it will be up to date
Then erase the content of the __EFMigrationHistory table and Insert the new migration in the table
After that use the standard Migrate method on the new context
That's it !
This solution is relatively simple. It solve all cases
New database (will be using the InitV2)
Old Database (will be upgrading to the last V1 Migration and after that to the last version of the v2 Database)
Be aware, if you used custom scripts/sql(...) in the v1 migrations, you have to check if the v2 Init migrations needs it.
To be sure it was OK, we created an empty database from v1 migrations and another one from v2 init migration and did a schema and data diff (with Visual Studio SQL Server Tools)