I'm working on the same databse schema on different machines (PostgreSQL). I would like to know, how to merge data from one machine to another. Schema has many tables (around 10). What I want to achieve?
Dump data from machine A to file A.dmp
Restore data from file A.dmp to machine B
When a record already exists in machine B, I would like to don't insert it into machine B.
I was trying dump data from machine A to simple SQL inserts commands, but when I'm trying to restore it, I am getting duplicate key errors. What is more, I would like to restore data from command line (I have to import 250 MB data), because now I'm trying to do it manually with pgAdmin.
What is the best way to do it?
I finally did it this way:
Export to dump with:
pg_dump -f dumpfile.sql --column-inserts -a -n <schema> -U <username> <dbname>
Set skip unique for all tables
CREATE OR REPLACE RULE skip_unique AS ON INSERT TO <table>
WHERE (EXISTS (SELECT 1 FROM <table> WHERE users.id = new.id))
DO INSTEAD NOTHING
Import with psql
\i <dumpfile.sql>
Related
I have 2 Postgres tables (same schema) in different servers.
One table has the the latest data, and the other one has old data that must be updated with the latest data.
So I dump the table with the latest data:
pg_dump --table=table00 --data-only db00 > table00.sql
or
pg_dump --table=table00 --data-only --column-inserts db00 > table00.sql
But now when I want to read in the SQL dump, I get errors about duplicate keys.
psql postgresql://<username>:<password>#localhost:5432/db00 --file=table00.sql
The error goes away if I drop the table with the old data first, but not only is this undesirable, it is plain silly.
How can I update a Postgres table from a SQL dump then?
Eg. it'd be nice if pg_dump had a --column-updates option, where, instead of INSERT statements, you got INSERT ON CONFLICT (column) DO UPDATE SET statements...
I have version A of my PostgreSQL database in AWS RDS and version B on my local machine. DB B was created from a dump of DB A and has been updated locally. I am trying to merge the data I have locally to RDS.
A has data that B doesn't have and B has data that A doesn't have, hence I cannot use the -c flag in pg_dump (as in this question).
I export my database with pg_dump:
pg_dump -f dump.sql mydb
I try to import my database to RDS using psql:
psql -h ABC.XYZ.eu-west-2.rds.amazonaws.com -U myself mydb < dump.sql
This updates schema, i.e. adds columns I had locally to RDS, but fails to insert any values into these columns. I get the following error for every table that exists in DB A:
ERROR: duplicate key value violates unique constraint "table_pkey"
As I understand from this question, my sequence may be out of sync, but this seems odd given I get it for every table in DB A, and DB B was created from a dump of DB A.
If I use the -c flag with pg_dump the merge works but all of the data in DB A that DB B did not have gets deleted.
How do I merge my local database into the remote one on AWS without losing data?
If you don't need to modify any of the existing rows, you can use pg_dump's option --on-conflict-do-nothing, new in v12.
This will not add any new columns. pg_dump is not a schema migration tool.
I would like to copy data from table A database A to table b database b in postgres.
I am able to use copy to write data out to a file from table A in DB A
but when I try copy into database b table B it says "ERROR: must be super user to copy to and from file"
Please help and let me know how I should data from a flatfile into a table in a database.... the db is postgres 9.x
According to the documentation:
COPY naming a file or command is only allowed to database superusers or users who are granted one of the default roles pg_read_server_files, pg_write_server_files, or pg_execute_server_program, since it allows reading or writing any file or running a program that the server has privileges to access.
However you can use pipeline to transmit data from one database to another one without using of intermediate file(s):
psql -d A -c "copy a to stdout" | psql -d B -c "copy b from stdin"
Read more about psql utility and its usage.
I have two postgres databases on the same server, a live one and another one which I use as a test database.
Periodically I need to copy my live database (both structure of the tables and their data) into the test one, but everything I've found online only copies table by table. Since I often create new tables in my live db I can't do this, otherwise I'd have to update the job every time.
Anybody knows how can I pull the whole live postgres db into the test postgres one?
Thanks!
You want to use pg_dump and pg_restore.
Example usage would be:
$ pg_dump -Fc <database_name> > dumpfile
$ pg_restore <another_database_name> < dumpfile
If you want to do this for all database you might want to consider pg_dumpall
A bit more can be found in https://www.postgresql.org/docs/current/backup-dump.html
What would be the best way to copy data from one table, one database, one server to the table in another database, another server in PostgreSQL?
pg_dump allows the dumping of only select tables:
pg_dump -Fc -f output.dump -t tablename databasename
(dump 'tablename' from database 'databasename' into file 'output.dump' in pg_dumps binary custom format)
You can restore that dump on your other server with pg_restore:
pg_restore -d databasename output.dump
If the table itself already exists in your target database, you can import only the rows by adding the --data-only flag.
I shared a shell to copy table from one server to another PostgreSQL server.
Please refer this another stack question.
Copying PostgreSQL database to another server