I have 2 Postgres tables (same schema) in different servers.
One table has the the latest data, and the other one has old data that must be updated with the latest data.
So I dump the table with the latest data:
pg_dump --table=table00 --data-only db00 > table00.sql
or
pg_dump --table=table00 --data-only --column-inserts db00 > table00.sql
But now when I want to read in the SQL dump, I get errors about duplicate keys.
psql postgresql://<username>:<password>#localhost:5432/db00 --file=table00.sql
The error goes away if I drop the table with the old data first, but not only is this undesirable, it is plain silly.
How can I update a Postgres table from a SQL dump then?
Eg. it'd be nice if pg_dump had a --column-updates option, where, instead of INSERT statements, you got INSERT ON CONFLICT (column) DO UPDATE SET statements...
Related
I try dumping tables from a production environment to a dev one. However, when dumping and restoring this table, using the following command:
pg_restore --no-owner --no-acl --clean --if-exists -d database dump_file.dump
I get an error stating that I can't drop that table unless I use something like CASCADE (i.e. dropping all other tables that depend on that one). Is there a way to determine the tables to be dropped? is there a way of maybe state in the pg_dump command to dump the table I'm looking to dump and all related tables ?
Here's the error raised:
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 4066; 2606 30526 CONSTRAINT table1 pkey user
pg_restore: error: could not execute query: ERROR: cannot drop constraint pkey on table public.table1 because other objects depend on it
DETAIL: constraint id_fkey on table public.dag depends on index public.pkey
constraint id_fkey on table public.dag depends on index public.pkey
HINT: Use DROP ... CASCADE to drop the dependent objects too...
You have a table on the dev database that has a pkey that is dependent and therefore can not be dropped before the restore. This is proper behavior.
I am not seeing dumping/restoring a particular table. You are dumping/restoring the entire database.
If you want recreate the production database as a dev one then do:
pg_restore -C --no-owner --no-acl --clean --if-exists -d postgres dump_file.dump
The -C with --clean will DROP DATABASE db_name and then rebuild it from scratch by connecting to the database postgres to do the DROP/CREATE db_name and then connect to db_name to load the rest of the objects.
This is the best way to clean out cruft and start at a consistent state.
UPDATE
Update your question with the pg_dump command so it is evident what you are doing.
If you want to see whether a particular table has dependencies, in the original database use psql and do \d the_table to see what the dependencies to and from the table are. If you tell pg_dump to dump a single table it will dump just that table. It will not follow dependencies and dump those also. That is up to you to do.
Look into using a schema management tool to do your changes/migrations. I use Sqitch for this.
I dumped the database with:
pg_dump dsmartgpp -f /data1/master/backup/db.sql
and I copied the database with
nohup psql -h gpnm1 -d dsmartgpp_1 -U gpadmin < /data1/master/backup/db.sql
the log information showed some errors about the function and datatype of postgis, such as
ALTER FUNCTION
ERROR: function "gidx_out" already exists with same argument types.
ALTER FUNCTION
ERROR: type "gidx" already exists
some tables between the two database have the same number or records, but about 1MB difference in size.
That is as expected. Restoring a pg_dump creates a logical copy of the database. In particular, the copy won't be bloated (don't worry – a bit of bloat is healthy).
If you want to avoid these errors, create the new database (dsmartgpp_1) with template0 as template (empty database). You seem to have PostGIS installed in template1, which gets copied into every new database. Or you installed PostGIS in your new database, before importing the dump.
Either way, create the empty database (dsmartgpp_1) and let the dump install the PostGIS functions.
Oh, one more thing, you can use "-f /data1/master/backup/db.sql" instead of the "<" redirect.
If you want to be super careful, also add "-v ON_ERROR_STOP=1". This will stop at the first error.
Just as the title states, I have a table that has well over 25 million records and I need all those records. it will only grow over time, so I need to export this one table from a psql database and import it into another psql database which is used for development.
Ideas?
I know you can dump a whole database, but can you dump a table? (sorry if thats a dumb question)
You can use:
$ pg_dump -d db -t big_table > big_table.sql
What would be the best way to copy data from one table, one database, one server to the table in another database, another server in PostgreSQL?
pg_dump allows the dumping of only select tables:
pg_dump -Fc -f output.dump -t tablename databasename
(dump 'tablename' from database 'databasename' into file 'output.dump' in pg_dumps binary custom format)
You can restore that dump on your other server with pg_restore:
pg_restore -d databasename output.dump
If the table itself already exists in your target database, you can import only the rows by adding the --data-only flag.
I shared a shell to copy table from one server to another PostgreSQL server.
Please refer this another stack question.
Copying PostgreSQL database to another server
I'm working on the same databse schema on different machines (PostgreSQL). I would like to know, how to merge data from one machine to another. Schema has many tables (around 10). What I want to achieve?
Dump data from machine A to file A.dmp
Restore data from file A.dmp to machine B
When a record already exists in machine B, I would like to don't insert it into machine B.
I was trying dump data from machine A to simple SQL inserts commands, but when I'm trying to restore it, I am getting duplicate key errors. What is more, I would like to restore data from command line (I have to import 250 MB data), because now I'm trying to do it manually with pgAdmin.
What is the best way to do it?
I finally did it this way:
Export to dump with:
pg_dump -f dumpfile.sql --column-inserts -a -n <schema> -U <username> <dbname>
Set skip unique for all tables
CREATE OR REPLACE RULE skip_unique AS ON INSERT TO <table>
WHERE (EXISTS (SELECT 1 FROM <table> WHERE users.id = new.id))
DO INSTEAD NOTHING
Import with psql
\i <dumpfile.sql>