pg_restore not restoring sequences value - postgresql

I have 2 databases : dbprod and dbdev, with 2 users uprod and udev.
What I'd like to do is to copy the production database to my development database.
I succeeded to restore my database, but for some reason the value of my sequences are not set.
Here's what I'm doing :
// 1. Dumping
pg_dump --file '/home/punt/dump_db/prod.tar.gz' --format=t --schema 'public' 'dbprod'
// 2. Renaming the public schema to public_old
psql --dbname dbdev -c 'ALTER SCHEMA public RENAME TO public_old;'
// 3. Restoring
pg_restore --dbname='dbdev' --format=t --verbose /home/punt/dump_db/prod.tar.gz
// 4. And then I have to handle GRANTs...
Everything is working fine, except that the value of the sequences are not set.
I checked the content of my prod.tar.gz and I have the create and set, ex:
///...
CREATE SEQUENCE public.blockchain_id_seq
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER TABLE public.blockchain_id_seq OWNER TO uprod;
CREATE SEQUENCE public.cryptocurrency_id_seq
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER TABLE public.cryptocurrency_id_seq OWNER TO uprod;
//...
SELECT pg_catalog.setval('public.blockchain_id_seq', 34, true);
SELECT pg_catalog.setval('public.cryptocurrency_id_seq', 148, true);
And the --verbose does not show errors :
pg_restore: creating SEQUENCE "public.cryptocurrency_id_seq"
pg_restore: creating SEQUENCE "public.blockchain_id_seq"
// ...
pg_restore: executing SEQUENCE SET blockchain_id_seq
pg_restore: executing SEQUENCE SET cryptocurrency_id_seq
I'm thinking maybe some grants/privileges issue ?
Any idea ? Thanks.
psql --version
psql (PostgreSQL) 11.17 (Debian 11.17-0+deb10u1)

So... I don't know what was wrong...
But here's the solution, based on this : Postgresql - backup database and restore on different owner?
sudo -i -u postgres pg_dump --file /home/punt/dump_db/prod.tar.gz --format=t --clean --no-owner --no-privileges --schema public dbprod
sudo -i -u postgres psql --dbname dbdev -c 'ALTER SCHEMA public RENAME TO public_old;'
sudo -i -u postgres pg_restore --host localhost --username udev --password --dbname dbdev --no-owner --no-privileges --format t --verbose /home/punt/dump_db/prod.tar.gz
And then we can DROP public_old if we want. The solution with dropdb/createdb is also working, but I prefer to save the old schema and drop it when I need to.

Related

PostgreSQL: pg_dump for a single table

I'm beginner with PostgreSQL and doing backups using:
sudo -u postgres pg_dumpall > /~/postgreBackup.SQL
Works fine!
Now I want to backup a single table "TableName" in a scheme "SchemeName" and tried
sudo -u postgres pg_dump --table "SchemaName"."TableName" > /~/Dummy.SQL
pg_dump: no matching tables were found
How to get it working?
When you have case sensitive table and schema name you have to do the proper quoting of a table name. The below command should work fine as I have successfully executed it at my end.
Please make sure you are using the correct case sensitive name of database, schema and table in this command.
./pg_dump --dbname="myDatabase" --host=localhost --port=5432 --username=postgres --table='"MyScheme"."TableName 01"' --file=Dummy
OR
./pg_dump --dbname="myDatabase" --host=localhost --port=5432 --username=postgres --table='"MyScheme"."TableName 01"' > ~/Dummy.SQL

Unable to restore psql database from pg_dump with a different username

I need to dump a postgres database from computer1 with postgres username1 and then restore it on computer2 with postgres username2. I keep running into the error that looks like the backup file wants to use username1:
When I run this on computer2:
psql dbname < backupname.pgsql
I get this error:
ERROR: role "username1" does not exist
I have tried:
// Dumping from computer1:
pg_dump dbname > backupname.sql
pg_dump dbname > backupname.pgsql
pg_dump -U username1 dbname -N topology -T spacial_ref_sys > backupname.pgsql
// Restoring on computer2:
psql dbname < backupname.pgsql
Is it the dumping or the restoring that needs to be modified to get past this?
The problem is with the dumping. With insight from this post I was able to resolve this using:
// On Computer1
pg_dump dbname -O -x > backupname.sql
// On Computer2
psql dbname < backupname.sql
The option flags used with pg_dump are:
-O <-- No owner
Do not output commands to set ownership of objects to match the original database
-x <-- No privileges
Prevent dumping of access privileges (grant/revoke commands)
See the PostgreSQL docs for pg_dump for more info on the option flags.
You don't need to cripple your dump by discarding owner/privileges. You can do it at restore time.
Use pg_restore with the --no-acl (and probably --no-owner) options:
-x
--no-privileges
--no-acl
Prevent restoration of access privileges (grant/revoke commands).
--no-owner
Do not output commands to set ownership of objects to match the
original database. By default, pg_restore issues ALTER OWNER or SET
SESSION AUTHORIZATION statements to set ownership of created schema
elements. These statements will fail unless the initial connection
to the database is made by a superuser (or the same user that owns
all of the objects in the script). With -O, any user name can be
used for the initial connection, and this user will own all the
created objects.
So something like:
pg_restore --no-privileges --no-owner -U postgres --clean ... $Your_sql_backup
If you are using pgAdmin then you can either remove the checkbox in DumpOptions #2 with Owner and otherwise remove the privilege like
--no-privileges and remove ownership like --no-password in the dump query like
/usr/bin/pg_dump --host localhost --port 5432 --username "postgres" --no-password --format custom --no-privileges --no-tablespaces --verbose --file "as" "databasename".
Also if you have constraints on the table then disable triggers also while creating the dump.
If you cannot create another backup of the database then the alternate way is to replicate the owner and roles of the dumped database to the new database. If you don't do that then you will get an error saying
'ACL does not exist' (not sure as faced it long back)

pg_restore ERROR: “Relation does not exist” and creating new database

I have made a backup of my specific tables that I want to restore into a new database using:
call pg_dump -Fc -h server -d database -U user -p password -v -f dump.sql -t public.table1 -t public.table2
And I have no problems.
I then want to restore the dump by creating a new database with pg_restore using:
call pg_restore --clean --create -d temp -h server -p password -U user dump.sql
This gives me a "database temp does not exist" error. This should have created the database and restored it as far as I understand.
I However then create the "temp" database manually and run:
call pg_restore --clean --create -d temp -h server -p password -U user dump.sql
This follows through, but does not create the tables and gives me an error "relation table1" and "relation table2" does not exist and only creates the corresponding id_sequences for the two tables.
What I actually want is to not have to manually create the new database and that all tables in the backup is restored into a brand new database via pg_restore using:
call pg_restore --clean --create -d temp -h server -p password -U user dump.sql
as I understand it.
In order to create the database temp, pg_restore needs to be connected to a different database first.
So it won't do to use -d temp; you must specify an existing database, typically postgres.
pg_restore will connect to that database, issue CREATE DATABASE temp, connect to temp and proceed to restore the data.

How to restore postgres db from gzip file using psql? (arelle: XBRL SEC DB)

I downloaded the xbrldb_SEC_pg_2014-11-02.pg.gzip postgres pg_dump file from arelle.org. I then ran the schema ddl file in pgAdminIII and it recreated all of the databases, functions, etc.
When I try to restore the databases using the following:
desktop:~/Downloads$ sudo postgres zcat xbrldb_SEC_pg_2014-11-02.pg.gzip | psql -U postgres public
I get:
sudo: postgres: command not found psql: FATAL: Peer authentication failed for user "postgres"
I can zcat the file into a file to expand it. Looks like it is a pg_dump file.
postgres=> pg_restore -a /home/jeremy/Downloads/xbrldb_SEC_pg_2014-11-02.txt
postgres-> ;
ERROR: syntax error at or near "pg_restore"
LINE 1: pg_restore -a /home/jeremy/Downloads/xbrldb_SEC_pg_2014-11-0...
^
postgres=> pg_restore -a postgres /home/jeremy/Downloads/xbrldb_SEC_pg_2014-11-02.txt;
ERROR: syntax error at or near "pg_restore"
LINE 1: pg_restore -a postgres /home/jeremy/Downloads/xbrldb_SEC_pg_...
So then I tried to use PG Admin III, and my output:
/usr/bin/pg_restore --host localhost --port 5432 --username "postgres" --dbname "public" --role "postgres" --no-password --section data --data-only --exit-on-error --table accession --schema public --verbose "/home/jeremy/Downloads/xbrldb_SEC_pg_2014-11-02.backup"
pg_restore: [archiver] input file appears to be a text format dump. Please use psql.
Process returned exit code 1.
May I please ask what I need to do to get the databases restored?
Does anyone know what I need to do to get the database updated from 2014-11-02 to the current date?
You should run psql as postgres user, not zcat, so try to use following:
zcat xbrldb_SEC_pg_2014-11-02.pg.gzip | sudo -u postgres psql public
PS pg_restore is an utility, not a PostgreSQL command, that means you should run it from command line, not from psql.

How to restore postgres database into another database name

I use the postgres today
and got a problem
I dump the database that way
pg_dump zeus_development -U test > zeus_development.dump.out
what if I wnat to restore to another database zeus_production
How could I do?
Simple, first create your database using template0 as your template database:
createdb -U test -T template0 zeus_production
Then, restore your dump on this database:
psql -U test zeus_production -f /path/to/zeus_development.dump.out
When restoring, always use template0 explicit, as it is always an empty and unmodifiable database. If you don't use an explicit template, PostgreSQL will assume template1, and if it has some objects, like a table or function that your dumped database already has, you will get some errors while restoring.
Nonetheless, even if you were restoring on a database with the same name (zeus_development) you should create (or recreate) it the same way. Unless you used -C option while dumping (or -C of pg_restore if using a binary dump), which I don't recommend, because will give you less flexibility (like restoring on a different database name).
The PostgresSQL documentation has influenced me to use the custom format. I've been using it for years and it seems to have various advantages but your mileage may vary. That said, here is what worked for me:
pg_restore --no-owner --dbname postgres --create ~/Desktop/pg_dump
psql --dbname postgres -c 'ALTER DATABASE foodog_production RENAME TO foodog_development'
There was no foodog_development nor foodog_production databases existing before the sequence.
This restores the database from the dump (~/Desktop/pg_dump) which will create it with the name it was dumped as. The rename names the DB to whatever you want.
The --no-owner may not be needed if your user name is the same on both machines. In my case, the dump was done as user1 and the restore done as user2. The new objects need to be owned by user2 and --no-owner achieves this.
Isn't it easier to simply do the following?
createdb -U test -T zeus_development zeus_production
This has an answer on dba.stackexchange, which I reproduce here:
Let's define a few variables to make the rest easier to copy/paste
old_db=my_old_database
new_db=new_database_name
db_dump_file=backups/my_old_database.dump
user=postgres
The following assumes that your backup was created with the "custom" format like this:
pg_dump -U $user -F custom $old_db > "$db_dump_file"
To restore $db_dump_file to a new database name $new_db :
dropdb -U $user --if-exists $new_db
createdb -U $user -T template0 $new_db
pg_restore -U $user -d $new_db "$db_dump_file"
Here's a hacky way of doing it, that only works if you can afford the space and time to use regular .sql format, and if you can safely sed out your database name and user.
$ pg_dump -U my_production_user -h localhost my_production > my_prod_dump.sql
$ sed -i 's/my_production_user/my_staging_user/g' my_prod_dump.sql
$ sed -i 's/my_production/my_staging/g' my_prod_dump.sql
$ mv my_prod_dump.sql my_staging_dump.sql
$ sudo su postgres -c psql
psql> drop database my_staging;
psql> create database my_staging owner my_staging_user;
psql> \c my_staging;
psql> \i my_staging_dump.sql
If your dump does not include the name, the restore will use the DB defined in DESTINATION. Both SOURCE and DESTINATION are Connection URLs.
Dump without --create
pg_dump \
--clean --if-exists \
--file ${dump_path} \
--format=directory \
--jobs 5 \
--no-acl \
--no-owner \
${SOURCE}
Restore without --create
pg_restore \
--clean --if-exists \
--dbname=${DESTINATION} \
--format=directory \
--jobs=5 \
--no-acl \
--no-owner \
$dump_path