I have a production Postgresql database which consist of 70 tables. Some of them are very big and some are small. And I have my local Postgresql database on my local machine. I want to make some of my local database tables's content be the same as production ones. If I just backup some tables with pgAdmin on production database and then try to restore on my local machine I got constrain errors. Because for example table A has foreign key to table B and so on.
How could I copy some tables from production database and restore normally on my local machine which has already scheme and tables and without constrain errors?
P.s. I couldn't just dump all production database because some of tables are VERY BIG.
Dump complete production database, but without data in case of large tables:
$ pg_dump -t <VERY_BIG_TABLE_NAME> -s
If you want data also, avoid the -s option. Since you will have to repeat this 70 times, quicker solution is dividing tables into schemas:
$ pg_dump -n <SCHEMA_NAME_WITH_VERY_BIG_TABLES> -s
$ pg_dump -n <SCHEMA_NAME_WITH_SMALL_TABLES>
I'm not sure if I understood, but if you got constraint check errors you can disable the foreign key constraints, restore the tables and enable them again.
Re-create table structure in your local database but add DEFERRABLE INITIALLY DEFERRED options to problematic foreign key CONSTRAINTs
Use pg_dump to dump your selected table data from production DB to a file and write at the very beginning: BEGIN;. At the end of the file append:
UPDATE TABLE problem_no_1 SET fkey_column = NULL; for every FK column that causes problems and of course COMMIT; at the end
Execute this file on your local DB
Related
I have 2 Postgres tables (same schema) in different servers.
One table has the the latest data, and the other one has old data that must be updated with the latest data.
So I dump the table with the latest data:
pg_dump --table=table00 --data-only db00 > table00.sql
or
pg_dump --table=table00 --data-only --column-inserts db00 > table00.sql
But now when I want to read in the SQL dump, I get errors about duplicate keys.
psql postgresql://<username>:<password>#localhost:5432/db00 --file=table00.sql
The error goes away if I drop the table with the old data first, but not only is this undesirable, it is plain silly.
How can I update a Postgres table from a SQL dump then?
Eg. it'd be nice if pg_dump had a --column-updates option, where, instead of INSERT statements, you got INSERT ON CONFLICT (column) DO UPDATE SET statements...
I dumped the database with:
pg_dump dsmartgpp -f /data1/master/backup/db.sql
and I copied the database with
nohup psql -h gpnm1 -d dsmartgpp_1 -U gpadmin < /data1/master/backup/db.sql
the log information showed some errors about the function and datatype of postgis, such as
ALTER FUNCTION
ERROR: function "gidx_out" already exists with same argument types.
ALTER FUNCTION
ERROR: type "gidx" already exists
some tables between the two database have the same number or records, but about 1MB difference in size.
That is as expected. Restoring a pg_dump creates a logical copy of the database. In particular, the copy won't be bloated (don't worry – a bit of bloat is healthy).
If you want to avoid these errors, create the new database (dsmartgpp_1) with template0 as template (empty database). You seem to have PostGIS installed in template1, which gets copied into every new database. Or you installed PostGIS in your new database, before importing the dump.
Either way, create the empty database (dsmartgpp_1) and let the dump install the PostGIS functions.
Oh, one more thing, you can use "-f /data1/master/backup/db.sql" instead of the "<" redirect.
If you want to be super careful, also add "-v ON_ERROR_STOP=1". This will stop at the first error.
I have version A of my PostgreSQL database in AWS RDS and version B on my local machine. DB B was created from a dump of DB A and has been updated locally. I am trying to merge the data I have locally to RDS.
A has data that B doesn't have and B has data that A doesn't have, hence I cannot use the -c flag in pg_dump (as in this question).
I export my database with pg_dump:
pg_dump -f dump.sql mydb
I try to import my database to RDS using psql:
psql -h ABC.XYZ.eu-west-2.rds.amazonaws.com -U myself mydb < dump.sql
This updates schema, i.e. adds columns I had locally to RDS, but fails to insert any values into these columns. I get the following error for every table that exists in DB A:
ERROR: duplicate key value violates unique constraint "table_pkey"
As I understand from this question, my sequence may be out of sync, but this seems odd given I get it for every table in DB A, and DB B was created from a dump of DB A.
If I use the -c flag with pg_dump the merge works but all of the data in DB A that DB B did not have gets deleted.
How do I merge my local database into the remote one on AWS without losing data?
If you don't need to modify any of the existing rows, you can use pg_dump's option --on-conflict-do-nothing, new in v12.
This will not add any new columns. pg_dump is not a schema migration tool.
I have two postgres databases on the same server, a live one and another one which I use as a test database.
Periodically I need to copy my live database (both structure of the tables and their data) into the test one, but everything I've found online only copies table by table. Since I often create new tables in my live db I can't do this, otherwise I'd have to update the job every time.
Anybody knows how can I pull the whole live postgres db into the test postgres one?
Thanks!
You want to use pg_dump and pg_restore.
Example usage would be:
$ pg_dump -Fc <database_name> > dumpfile
$ pg_restore <another_database_name> < dumpfile
If you want to do this for all database you might want to consider pg_dumpall
A bit more can be found in https://www.postgresql.org/docs/current/backup-dump.html
Context: I'm new on a big project, with a huge number of PostgreSQL tables (46) and foreign keys, and no documentation. We work with Java, Spring, Hibernate and TestNG.
Goal: I'm looking for a script that's able to empty the database (to call it before every unit-test case). Unfortunately, I can't spend too much time to find foreign keys in order to empty tables in the correct order.
Question: is there a way to empty (I don't want to drop it) a table without checking constraints on foreign keys ?
As #a_horse_with_no_name commented, using truncate ... cascade for each table will work. The link also provides a function that automatically loops through all the tables in the database.
If you have large tables, with a lot of foreign keys, it will probably take a lot of time to travel through all the foreign key indexes. In that case it would probably be better to dump the schema, drop the database, recreate the database and reload the schema in a batch. There is an example in the link, but it doesn't tell you how to run them. This is a little more refined:
#!/bin/sh
# Run this as the postgres user because you need create database permissions
# I'm assuming you're using a Unix variant,
# the postgresql database is at localhost,
# and that psql trusts the Unix account.
# Otherwise add -h hostname to the command lines
# and set $PGPASSWORD variable before the first command.
pg_dump –s yourdb > /tmp/clean.sql
psql –c "drop database yourdb;"
psql –c "create database yourdb with owner you;"
psql yourdb < /tmp/clean.sql
rm –f /tmp/clean.sql