I am currently working on migrating a postgres database between two azure postgres instances. Since these are managed services, I am not able to use a superuser-account (those are not provided). I am first migrating my roles, then my database, with the following commands:
# https://learn.microsoft.com/en-us/azure/postgresql/single-server/how-to-upgrade-using-dump-and-restore#migrate-the-roles
# Migrate the roles
pg_dumpall -r --host=$OLD_HOST --username=$OLD_USERNAME --database=$DB_NAME > roles.sql
sed -i 's/NOSUPERUSER//g' roles.sql
sed -i 's/NOBYPASSRLS//g' roles.sql
psql -f roles.sql --host=$NEW_HOST --username=$NEW_USERNAME --dbname=postgres --clean --if-exists
pg_dump -Z0 -Fd -j 12 -h $OLD_HOST -U $OLD_USERNAME -d $DB_NAME -f $DIRECTORY
pg_restore -h $NEW_HOST -U $NEW_USERNAME -j 12 -Fd -C -d $DB_NAME $DIRECTORY
Unfortunately this results in a couple of errors like these:
pg_restore: from TOC entry 720; 1259 1421377 SEQUENCE id_publicatie_seq svp
pg_restore: error: could not execute query: ERROR: permission denied for schema svp
Command was: ALTER TABLE svp.id_publicatie_seq OWNER TO svp;
pg_restore: from TOC entry 718; 1259 1421365 TABLE publicatie svp
pg_restore: error: could not execute query: ERROR: permission denied for schema svp
Command was: ALTER TABLE svp.publicatie OWNER TO svp;
I'm able to get rid of these errors by adding the --no-owner flag in my pg_restore command, but this is a problem since I specifically want to keep all owners exactly the same as in my original database. Am I overseeing something?
Related
Here are my commands:
db_conn_str="${db_prefix}#${db_host}/${db_name}"
pg_dump -w --schema-only --clean --create \
-U dev -d "$db_conn_str" > "$tmp_folder/dev_dump.sql"
psql -h localhost -U postgres -1 -f "$tmp_folder/dev_dump.sql"
I am getting this error:
pg_dump: error: query failed: ERROR: permission denied for table
cp_users
pg_dump: error: query was: LOCK TABLE public.cp_users IN
ACCESS SHARE MODE
does anyone know how to mitigate that error? My goal is to copy a database from a remote db to my local db.
You need to give user dev the right to SELECT from all tables or use a different user.
I have a script where pg_dump is failing with a message like this:
pg_dump -h db1 --format plain --encoding UTF8 --schema=public --schema-only --no-owner me
pg_dump: [archiver (db)] query failed: ERROR: permission denied for relation notmytable
pg_dump: [archiver (db)] query was: LOCK TABLE public.notmytable IN ACCESS SHARE MODE
This is causing the whole dump to abort.
Is there a way to either:
Ignore tables that aren't owned by our user?
Ignore errors?
I really don't want these tables in the dump, so even if we could get access to them, that wouldn't exactly solve the problem.
(Postgres 9.6.3)
It doesn't appear there is a standard way to do this, but using the --exclude-table flag, we can use a workaround:
export EXCLUDETABLE=$(psql -t -h $HOST -d $DBNAME -c "select '--exclude-table=' || string_agg(tablename,' --exclude-table=') FROM pg_catalog.pg_tables WHERE tableowner NOT LIKE 'myuser';" )
This sets EXCLUDETABLE to look like --exclude-table=foo --exclude-table=blah
Now we pass that to pg_dump:
echo Excluding these tables from dump: $EXCLUDETABLE
pg_dump -h $HOST --format plain --encoding UTF8 --schema=public --schema-only --no-owner $EXCLUDETABLE $DBNAME > public-schema.sql
I downloaded the xbrldb_SEC_pg_2014-11-02.pg.gzip postgres pg_dump file from arelle.org. I then ran the schema ddl file in pgAdminIII and it recreated all of the databases, functions, etc.
When I try to restore the databases using the following:
desktop:~/Downloads$ sudo postgres zcat xbrldb_SEC_pg_2014-11-02.pg.gzip | psql -U postgres public
I get:
sudo: postgres: command not found psql: FATAL: Peer authentication failed for user "postgres"
I can zcat the file into a file to expand it. Looks like it is a pg_dump file.
postgres=> pg_restore -a /home/jeremy/Downloads/xbrldb_SEC_pg_2014-11-02.txt
postgres-> ;
ERROR: syntax error at or near "pg_restore"
LINE 1: pg_restore -a /home/jeremy/Downloads/xbrldb_SEC_pg_2014-11-0...
^
postgres=> pg_restore -a postgres /home/jeremy/Downloads/xbrldb_SEC_pg_2014-11-02.txt;
ERROR: syntax error at or near "pg_restore"
LINE 1: pg_restore -a postgres /home/jeremy/Downloads/xbrldb_SEC_pg_...
So then I tried to use PG Admin III, and my output:
/usr/bin/pg_restore --host localhost --port 5432 --username "postgres" --dbname "public" --role "postgres" --no-password --section data --data-only --exit-on-error --table accession --schema public --verbose "/home/jeremy/Downloads/xbrldb_SEC_pg_2014-11-02.backup"
pg_restore: [archiver] input file appears to be a text format dump. Please use psql.
Process returned exit code 1.
May I please ask what I need to do to get the databases restored?
Does anyone know what I need to do to get the database updated from 2014-11-02 to the current date?
You should run psql as postgres user, not zcat, so try to use following:
zcat xbrldb_SEC_pg_2014-11-02.pg.gzip | sudo -u postgres psql public
PS pg_restore is an utility, not a PostgreSQL command, that means you should run it from command line, not from psql.
I am trying to setup a script to take a copy of a database from one server to another.
Thanks to this post Copying PostgreSQL database to another server I have found a way to do that.
But what I need to do is change the name of the database during the copy.
I have thought about using sed and doing a simple text replace. But I am worried that this could corrupt the database.
Does any one know the proper way of doing this?
As requested here are the commands I am using
pg_dump -C -U remoteuser -h remoteServer dbname | psql -h localhost -U localadmin template1
Just restore to a different database. For pg_restore of -Fc dumps from pg_dump's custom format:
createdb newdbname
pg_restore --dbname newdbname database.dump
For SQL-format dumps not created with the -C option to pg_dump:
createdb newdbname
psql -f database_dump.sql newdbname
If you're streaming the dump from a remote host, just omit -f database_dump.sql as the dump data is coming from stdin.
You can't easily CREATE DATABASE in the same command as your restore, because you need to connect to a different DB like template1 in order to create the new DB. So in your example you might:
psql -h localhost -U localadmin template1 -c 'CREATE DATABASE newdb;'
pg_dump -U remoteuser -h remoteServer dbname | psql -h localhost -U localadmin newdb
Note the omission of the -C flag to pg_dump.
The first command is just the longhand way of writing createdb -h localhost -U localadmin newdb.
Update: If you're stuck with a pg_dump created with the -C flag you can indeed just sed the dump so long as you're extremely careful. There should only be four lines (one a comment) at the start of the file that refer to the database name. For the database name "regress" dumped with Pg 9.1's pg_dump -C:
--
-- Name: regress; Type: DATABASE; Schema: -; Owner: craig
--
CREATE DATABASE regress WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
ALTER DATABASE regress OWNER TO craig;
\connect regress
This can be transformed quite safely with three (or four if you want to rewrite the comment) very specific sed commands. Do not just do a global find and replace on the database name, though.
sed \
-e 's/^CREATE DATABASE regress/CREATE DATABASE newdbname/' \
-e 's/^ALTER DATABASE regress/ALTER DATABASE newdbname/' \
-e 's/^\\connect regress/\\connect newdbname/' \
-e 's/^--Name: regress/--Name: newdbname/'
This should be a last resort; it's much better to just dump without -C.
i was using the following syntax for pg_dump and restore
pg_dump eval --inserts -b -c --encoding UTF8 -Fc -f eval.sql.tar.gz -x -U postgres
createdb -T template0 test -U postgres
pg_restore -d test eval.sql.tar.gz -e -U postgres
the dump was successfull with no errors, but restore makes a some errors, i am dumping and restoring in same machine with same user and privilege all...
i have tried out with other formats also, plain, tar, compressed all gets the same error..
my version of pg is 8.4.11 and psql version is 8.4.11
i am not sure what makes these errors.. can anyone help me
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 4965; 0 138871 TABLE DATA ir_act_report_xml insigni
pg_restore: [archiver (db)] could not execute query: ERROR: invalid input syntax for integer: "purchase.order"
LINE 1: ...st for Quotation', 'ir.actions.report.xml', NULL, 'purchase....
^
Command was: INSERT INTO ir_act_report_xml VALUES (350, 'Request for Quotation', 'ir.actions.report.xml', NULL, 'purchase.order', 'purcha...
this did the trick
pg_dump database_name -c -Ft -f file_name.tar
pg_restore -d database_name -c file_name.tar
before this i was trying to restore with out including -c(clean)
even though -c is included in pg_dump it is not used in pg_restore unless we say to use...
The solution in my case:
pg_restore --verbose --clean --no-acl --no-owner -h localhost -U username -d database_name dump_name.dump
This worked for me:
Increase the max_wal_size postgresql setting (max_wal_size = 2GB) in postgresql.conf