I try dumping tables from a production environment to a dev one. However, when dumping and restoring this table, using the following command:
pg_restore --no-owner --no-acl --clean --if-exists -d database dump_file.dump
I get an error stating that I can't drop that table unless I use something like CASCADE (i.e. dropping all other tables that depend on that one). Is there a way to determine the tables to be dropped? is there a way of maybe state in the pg_dump command to dump the table I'm looking to dump and all related tables ?
Here's the error raised:
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 4066; 2606 30526 CONSTRAINT table1 pkey user
pg_restore: error: could not execute query: ERROR: cannot drop constraint pkey on table public.table1 because other objects depend on it
DETAIL: constraint id_fkey on table public.dag depends on index public.pkey
constraint id_fkey on table public.dag depends on index public.pkey
HINT: Use DROP ... CASCADE to drop the dependent objects too...
You have a table on the dev database that has a pkey that is dependent and therefore can not be dropped before the restore. This is proper behavior.
I am not seeing dumping/restoring a particular table. You are dumping/restoring the entire database.
If you want recreate the production database as a dev one then do:
pg_restore -C --no-owner --no-acl --clean --if-exists -d postgres dump_file.dump
The -C with --clean will DROP DATABASE db_name and then rebuild it from scratch by connecting to the database postgres to do the DROP/CREATE db_name and then connect to db_name to load the rest of the objects.
This is the best way to clean out cruft and start at a consistent state.
UPDATE
Update your question with the pg_dump command so it is evident what you are doing.
If you want to see whether a particular table has dependencies, in the original database use psql and do \d the_table to see what the dependencies to and from the table are. If you tell pg_dump to dump a single table it will dump just that table. It will not follow dependencies and dump those also. That is up to you to do.
Look into using a schema management tool to do your changes/migrations. I use Sqitch for this.
Related
I have 2 Postgres tables (same schema) in different servers.
One table has the the latest data, and the other one has old data that must be updated with the latest data.
So I dump the table with the latest data:
pg_dump --table=table00 --data-only db00 > table00.sql
or
pg_dump --table=table00 --data-only --column-inserts db00 > table00.sql
But now when I want to read in the SQL dump, I get errors about duplicate keys.
psql postgresql://<username>:<password>#localhost:5432/db00 --file=table00.sql
The error goes away if I drop the table with the old data first, but not only is this undesirable, it is plain silly.
How can I update a Postgres table from a SQL dump then?
Eg. it'd be nice if pg_dump had a --column-updates option, where, instead of INSERT statements, you got INSERT ON CONFLICT (column) DO UPDATE SET statements...
We are migrating some products, one of the steps is to migrate the product databases.
I have steps to
export the existing DB pg_dump --no-owner --clean --blobs --no-privileges -U dbuser old_dbname -f bkpfile.sql
import the dump to a different DB psql -U dbuser2 new_dbname -f bkpfile.sql
The problem is the old database contains statement COMMENT ON DATABASE old_dbname IS 'Rxxxxx';
The new DB user must not have permissions on the old database and imho it's not good to refer the old database name anyway in the dump.
Is there a way to create a complete DB dump without the COMMENT ON DATABASE statement?
Edit:
PostgreSQL 9.6
Steps to reproduce:
CREATE DATABASE testdb;
COMMENT ON DATABASE testdb IS 'some comment';
CREATE TABLE xx (id int);
and then dump the database, the dump contains reference to the database name COMMENT ON DATABASE testdb IS 'some comment'; which prevents importing the backup to a new database
pg_dump --no-owner --clean --blobs --no-privileges testdb
We could manually remove the comment statement or filter the comment using different tools (grep), but manual intervention or text-based filtering on top of the backup could cause data corruption.
This comment is only dumped in PostgreSQL versions below v11. See this entry in the release notes:
pg_dump and pg_restore, without --create, no longer dump/restore database-level comments and security labels; those are now treated as properties of the database.
9.6 will go out of support soon anyway, so this is a good opportunity to upgrade.
I am having an issue with pg_restore --clean not clearing the database.
Or do I misunderstand what the --clean does, I am expecting it to truncate the database tables and reinitialize the indexes/primary keys.
I am using 9.5 on rds
This is the full command we use
pg_restore --verbose --clean --no-acl --no-owner -h localhost -U superuser -d mydatabase backup.dump
Basically what is happening is this.
I do a nightly backup of my production db, and restore it to an analytics db for the analyst to churn and run their reports.
I found out recently that the rails application used to view the reports was complaining that the primary keys were missing from the restored analytics database.
So I started investigating the production db, the analytics db etc. Which was when I realized that multiple rows with the same primary key existed in the analytics database.
I ran a few short experiments and realized that every time the pg_restore script is run it inserts duplicate data into the tables, this leads me to think that the --clean is not dropping and restoring the data. Because if I were to drop the schema beforehand, I don't get duplicate data.
To remove all tables from a database (but keep the database itself), you have two options.
Option 1: Drop the entire schema
You will need to re-create the schema and its permissions. This is usually good enough for development machines only.
DROP SCHEMA public CASCADE;
CREATE SCHEMA public;
GRANT ALL ON SCHEMA public TO postgres;
GRANT ALL ON SCHEMA public TO public;
Applications usually use the "public" schema. You may encounter other schema names when working with a (legacy) application's database.
Note that for Rails applications, dropping and recreating the database itself is usually fine in development. You can use bin/rake db:drop db:create for that.
Option 2: Drop each table individually
Prefer this for production or staging servers. Permissions may be managed by your operations team, and you do not want to be the one who messed up permissions on a shared database cluster.
The following SQL code will find all table names and execute a DROP TABLE statement for each.
DO $$ DECLARE
r RECORD;
BEGIN
FOR r IN (SELECT tablename FROM pg_tables WHERE schemaname = current_schema()) LOOP
EXECUTE 'DROP TABLE IF EXISTS ' || quote_ident(r.tablename) || ' CASCADE'; -- DROP TABLE IF EXISTS instead DROP TABLE - thanks for the clarification Yaroslav Schekin
END LOOP;
END $$;
Original:
https://makandracards.com/makandra/62111-how-to-drop-all-tables-in-postgresql
I'm trying to backup up an entire postgres database and restore it properly, however I am seeing a list of errors when trying to restore the backup.
I am using pg_dump to create a backup sql file. (I have a .pgpass file for password)
sudo -u postgres pg_dump -d db-w > backup.sql
When I try to restore the database with:
sudo -u postgres psql db < backup.sql
I get a list of errors like:
ERROR: duplicate key value violates unique constraint
ERROR: multiple primary keys for table
ERROR: relation <relation> already exists
ERROR: trigger <trigger> for relation <relation> already exist
I haven't made any changes to the database. I simply performed a backup and restore the backup right after.
What am I doing wrong?
your restoring on an existing database, if you want and sure to replace database with the backup you can use option --clean and --create
-c, --clean
Clean (drop) database objects before recreating them. (This might
generate some harmless error messages, if any objects were not
present in the destination database.)
-C, --create
Create the database before restoring into it. If --clean is also
specified, drop and recreate the target database before connecting
to it.
I need to be able to somehow get a set of tables from my dev db into my production db. I've just been creating a dump file from the dev db and using pg_restore on the production db. The problem now is that I need to preserve one table(called users) on the production db while replacing the others
I think I have the dump properly from this command
pg_dump -Fc --no-acl --no-owner -h localhost -U <USER> --exclude-table=users* --data-only <DB NAME> > test.dump
But I can't get the restore part to work. I tried the following command
pg_restore -Fc --no-acl --no-owner -h <PROD HOST> -U <USER> -d <DB NAME> -p <PORT> <FILE LOCATION>
BUt I get the following errors
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 2009; 0 121384 TABLE DATA idx_descs Jason
pg_restore: [archiver (db)] COPY failed for table "idx_descs": ERROR: duplicate key value violates unique constraint "idx_descs_pkey"
DETAIL: Key (id)=(6) already exists.
CONTEXT: COPY idx_descs, line 1
It seems like for the tables I'm trying to overwrite, it is just trying to append the data and running into trouble because there are now duplicate primary keys. Any Ideas how to do this? Thanks
So you need to reassign primary keys?
You could try restoring to a temporary table (say for instance, in failing case: idx_desc_temp), then doing something like:
with t as ( select * from idx_descs_temp )
insert into idx_descs
select id + 100000 [or whatever], [other fields] from t;
Afterwards you need to reset sequences (if applicable -- fill in sequence name....):
select setval( 'idx_descs_id_seq'::regclass, 100000 + [suitable increment]);
If you have a large # of tables you could try to automate using the system catalog.
Note though that you also have to renumber foreign key refs. Possibly less pain would be to move data in production db first. If you are using an ORM, you could also automate via application APIs.