I have a NestJS app with TypeORM, dockerized. I have synchronize turned off, using migrations instead. In the container entry point, I do yarn typeorm migration:run. It works well the first time around, and according to the logs it inserts records into the migrations table.
I noticed that when I start the project the next time it often tries to re-run migrations and fails (as expected) due to "relation already exists". At this point I can verify that entries are indeed missing from the migrations table via docker-compose exec db psql -U postgres -c 'SELECT * FROM "migrations" "migrations". The DB schema is up to date. When I insert a new record manually it gets an incremental ID after the missing records. So the records were there at some point.
I can't figure out what might cause entries in the migrations table to disappear (be rolled back?). This happens on the project linked above. It's a straightforward example project. I don't have an entity accidentally named "migrations". :)
As a workaround I currently insert into the migrations table manually:
docker-compose exec db psql -U postgres -c "INSERT INTO migrations (timestamp, name) VALUES ('1619623728180', 'AddTable1619623728180');"
Running specs that synchronized the DB was the issue.
I had a .env.test to use a different DB, but as it turns out that is not supported by dotenv. There are a few ways to make it work. I chose dotenv-flow/config and added it to my test script:
jest --collect-coverage --setupFiles dotenv-flow/config
Related
I have a PostgreSQL db that is used by a Nest.Js / Prisma app.
We changed the name of a field in the Prisma schema and added a new field.
Now, when we want to update the PostreSQL structure, I'm running, as suggested by Prisma, the following commands:
npx prisma generate
and then
npx prisma migrate dev --name textSettings-added --create-only
The idea is to use the --create-only flag to review the migration before it is actually made.
However, when I run it I get a list of the changes to be made to the DB and the following message:
We need to reset the PostgreSQL database "my_database" at "my_db_name#cluster.us-east-1.rds.amazonaws.com:5432".
Do you want to continue? All data will be lost.
Of course I choose not to continue, because I don't want to lose the data. Upon inspection I see that the migration file actually contains DROP TABLE for the tables that I simply wanted to modify. How to avoid that?
So how do I run the update without affecting the data?
UPDATE:
I saw that running with --create-only creates a migration which can then be implemented on the DB level using prisma migrate dev, however, in that migration file there are still some commands that drop my previous tables because of some new parameters inside. How can I run prisma migration without deleting my PostgreSQL data?
UPDATE 2:
I don't want Prisma to drop my tables when I just updated them. The migration file generated, however, drops them and then alters them. Do you know what's the best procedure to avoid this drop? I saw somewhere I could first manually update the DB with the new options and then run the migration, so Prisma can find a way to update it, but that seems too manual to me... Maybe some other ideas?
For some cases like renaming tables or columns, Prisma's generated migration files need to be updated if they already contain data.
If that applies to your use case, Prisma's docs suggest to:
Make updates to the prisma schema
Create migration file without applying it (--create-only flag)
Update the migration script to remove the drops and instead write your custom query (e.g. RENAME <table_name> TO <new_name>)
Save and apply the migration (npx prisma migrate dev)
Note that those changes can lead to downtime (renaming a field or model), for which they have outlined the expand and contract pattern.
It might be a Prisma bug: https://github.com/prisma/prisma/issues/8053
I also recently had this situation. It probably should not try to run migration if you only want to create migration file.
But overall it is expected with Prisma to recreate your db sometimes. If you migration is breaking then it will be required to reset the data anyway when you apply it.
I suggest you to create some seeding script so you could consistently re-create the database state, it's very useful for your development environment.
More info
Ok, I screwed up.
I dumped one of my psql (9.6.18) staging database with the following command
pg_dump -U postgres -d <dbname> > db.out
And after doing some testing, I "restored" the data using the following command.
psql -f db.out postgres
Notice the absence of -d option? yup. And that was supposed to be the username.
Annnd as the database happend to have the same name as its user, it overwrote the 'default' database (postgres), which had data that other QAs are using.
I cancelled the operation quickly as soon as I realised my mistake, but the damage was still done. Around 1/3 ~ 1/2 of the database is roughly identical to the staging database - at least in terms of the schema.
Is there any way to revert this? I am still looking for any other dumps if any of these guys made one. But I don't think there is any past two to three months. Seems like I got no choice but to own up and apologise to them in the morning.
Without a recent dump or some sort of PITR replication setup, you can't un-revert this easily. The only option is to manually go through the log of what was restored and remove/alter it in the postgres database. This will work for the schema, the data is another matter. FYI, the postgres database should not really be used as a 'working' database. It is there to be a database to connect to for doing other operations, such as CREATE DATABASE or to bootstrap your way into a cluster. If left empty then the above would not have been a problem. You could have done, from another database, DROP DATABASE postgres; and then CREATE DATABASE postgres.
Do you have a capture of the output of the psql -f db.out postgres run?
Since the pg_dump didn't specify --clean or -c, it should not have overwritten anything, just appended. And if your tables have unique or primary keys, most of the data copy operations should have failed with unique key violations and rolled back. Even one overlapping row (per table) would roll back the entire dataset for that table.
Without having the output, it will be hard to figure out what damage has actually been done.
You should also immediately copy the pg_xlog data someplace safe. If it comes down to it, you might be able to use pg_xlogdump to figure out what changes committed and what did not.
How do you get EF back in Sync with code without losing data when Update-database returns the following message
Error Message: System.Data.SqlClient.SqlException (0x80131904): There
is already an object named '' in the database.
I originally wrote this as a self-answering question as I had struggled with the proble for some time, as had a few colleagues, but unfortunately, my answer was deleted and I can't recover it.
Since it's a situation that I suspect can happen several times as people try to "clean up" old migrations, I thought I'd document it with step by step instructions.
Description of the situation we found ourselves in:
We couldn't can't create a new local database because the init script was incomplete, and couldn't apply updates to the production database because the migration scripts create tables that already exist. And, we didn't want to delete production data.
Symptom: Can't run Update-Database because it's trying to run the creation script and the database already has tables with the same name.
Error Message: System.Data.SqlClient.SqlException (0x80131904): There
is already an object named '' in the database.
Problem Background:
To understand this in more detail, I'd recommend watching both videos referenced here:
https://msdn.microsoft.com/en-us/library/dn481501(v=vs.113).aspx
To summarise, EF understands where the current database is at compared to where the code is at based on a table in the database called dbo.__MigrationHistory. When it looks at the Migration Scripts, it tries to reconsile where it was last at with the scripts. If it can't, it just tries to apply them in order. This means, it goes back to the initial creation script and if you look at the very first part in the UP command, it'll be the CreeateTable for the table that the error was occurring on.
Solution: What we need to do is to trick EF into thinking that the current database is up to date, while "not" applying these CreateTable commands since the production database already exists. Once production DB is set, we still need to be able to create local databases as well.
Step 1: Production DB clean
First, make a backup of your production db. In SSMS, Right-Click on the database, Select "Tasks > Export Data-tier application..." and follow the prompts.
Open your production database and delete/drop the dbo.__MigrationHistory table.
Step 2: Local environment clean
Open your migrations folder and delete it. I'm assuming you can get this all back from git if necessary.
Step 3: Recreate Initial
In the Package Manager, run "Enable-Migrations" (EF will prompt you to use -ContextTypeName if you have multiple contexts).
Run "Add-Migration Initial -verbose". This will Create the initial script to create the database from scratch based on the current code.
If you had any seed operations in the previous Configuration.cs, then copy that across.
Step 4: Trick EF
At this point, if we ran Update-Database, we'd be getting the original error. So, we need to trick EF into thinking that it's up to date, without running these commands. So, go into the Up method in the Initial migration you just created and comment it all out.
Step 5: Update-Database
With no code to execute on the Up process, EF will create the dbo.__MigrationHistory table with the correct entry to say that it ran this script correctly. Go and check it out if you like.
Now, uncomment that code and save.
You can run Update-Database again if you want to check that EF thinks its up to date. It won't run the Up step with all of the CreateTable commands because it thinks it's already done this.
Step 6: Confirm EF is ACTUALLY up to date
If you had code that hadn't yet had migrations applied to it, this is what I did...
Run "Add-Migration MissingMigrations"
This will create practically an empty script. Because the code was there already, there was actually the correct commands to create these tables in the initial migration script, so I just cut the CreateTable and equivalent drop commands into the Up and Down methods.
Now, run Update-Database again and watch it execute your new migration script, creating the appropriate tables in the database.
Step 7: Re-confirm and commit.
Build, test, run. Ensure that everything is running then commit the changes.
Step 8: Let the rest of your team know how to proceed.
When the next person updates, EF won't know what hit it given that the scripts it had run before don't exist. But, assuming that local databases can be blown away and re-created, this is all good. They will need to drop their local database and add create it from EF again. If they had local changes and pending migrations, I'd recommend they create their DB again on master, switch to their feature branch and re-create those migration scripts from scratch.
I have a test database connected to a test server. I want to run set of selenium tests and I have to restore database after every test.
I made a backup with cli command "createdb" and I just drop the main table every time, but how can I restore database without turning the whole server off and on (can't use createdb with any open connections), as it would take hours or days to make a full set of tests?
I probably won't be given constant admin access to the server, unless it's necessary.
You can kill all connections vis SQL (see https://stackoverflow.com/a/5109190/2352344). Instead of dropping the whole database you can just remove the schema:
DROP SCHEMA public CASCADE;
CREATE SCHEMA public;
I think that instead of dropping the table, how about undoing or deleting the rows in the table. When you run the test, you know what entries will be made in the Table. With this information, just before the test terminates, invoke a script to delete the rows created due to running this test.
You can use a real tool for your backup/restore (Wal-E, barman or backrest). Particularly with backrest, you can do a diff restore where it restores only files that have changes.
I solved the problem by making a bash script that i run from java code.
String[] args = new String[]{"./script.sh"};
Process proc = new ProcessBuilder(args).start();
proc.waitFor();
script.sh:
#!/bin/bash
psql dbname -c "drop schema \"public\" cascade;"
psql dbname -c "create schema \"public\";"
psql dbname < "path/backupname"
I had to use script and not just make it arguments in args, probably becouse of the "<" sign. I found no flag replacement to it.
I have created entities in zend framework 2 using doctrine 2. After that I used this command to validate current schema.
./vendor/bin/doctrine-module orm:validate-schema
I got output like:
Mapping] OK - The mapping files are correct.
[Database] FAIL - The database schema is not in sync with the current mapping file.
Then I executed update command,
./vendor/bin/doctrine-module orm:schema-tool:update --force
The output for that is like:
Database schema updated successfully! "7" queries were executed
But, the problem is, There is no tables created in my database. What's wrong with this?
I used to run doctrine-module orm:validate-schema and then doctrine-module orm:schema-tool:create.
Here is good project to try:
Fmi-example on github