Postgres DB Dump Carries Owner Information With It - postgresql

I am setting up a system of DB backup for a web project. At the moment, the DB is being backed up via Bash script like this
#!/bin/bash
# Set the DB configuration
export PGHOST=localhost
export PGPORT=5432
#export PGDATABASE=yoga
#export PGUSER=delme
export PGDATABASE=projectA
export PGUSER=userA
export PGPASSWORD=blah-blah
# Backs up mydatabase to a file and then uploads it to AWS S3.
# First, dump database backup to a file
TIME=$(date "+%s")
BACKUP_FILE="postgres_${PGDATABASE}_${TIME}.pgdump"
echo "Backing up $PGDATABASE to $BACKUP_FILE"
pg_dump --format=custom > $BACKUP_FILE
In the cluster, I have the following roles
Role name | Attributes | Member of
-----------+------------------------------------------------------------+-----------
delme | Superuser | {}
userA | | {}
postgres | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
The list of databases
postgres=# \list
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+---------+-----------------------
projectA | userA | UTF8 | C.UTF-8 | C.UTF-8 | =Tc/userA +
| | | | | userA=CTc/userA
postgres | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
yoga | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
(5 rows)
Whenever I bring this dump on a local machine and try to restore via pg_restore --host=localhost --port=5432 --username=postgres --verbose --create --dbname=new_db_name --format=custom --clean postgres_projectA_1660909386.pgdump, I get about 200 errors.
Most errors are of the following format:
pg_restore: creating SEQUENCE OWNED BY "public.plugins_mentorsplugin_mentors_id_seq"
pg_restore: creating TABLE "public.plugins_plugingeneral"
pg_restore: from TOC entry 374; 1259 19889 TABLE plugins_plugingeneral delme
pg_restore: error: could not execute query: ERROR: role "delme" does not exist
Command was: ALTER TABLE public.plugins_plugingeneral OWNER TO delme;
pg_restore: creating TABLE "public.plugins_retreatcommentplugin"
pg_restore: from TOC entry 375; 1259 19892 TABLE plugins_retreatcommentplugin delme
pg_restore: error: could not execute query: ERROR: role "delme" does not exist
Command was: ALTER TABLE public.plugins_retreatcommentplugin OWNER TO delme;
pg_restore: creating TABLE "public.plugins_retreatcommentplugin_comments"
pg_restore: from TOC entry 376; 1259 19895 TABLE plugins_retreatcommentplugin_comments delme
pg_restore: error: could not execute query: ERROR: role "delme" does not exist
Command was: ALTER TABLE public.plugins_retreatcommentplugin_comments OWNER TO delme;
pg_restore: creating SEQUENCE "public.plugins_retreatcommentplugin_comments_id_seq"
pg_restore: from TOC entry 377; 1259 19898 SEQUENCE plugins_retreatcommentplugin_comments_id_seq delme
pg_restore: error: could not execute query: ERROR: role "delme" does not exist
Command was: ALTER TABLE public.plugins_retreatcommentplugin_comments_id_seq OWNER TO delme;
pg_restore: creating SEQUENCE OWNED BY "public.plugins_retreatcommentplugin_comments_id_seq"
pg_restore: creating TABLE "public.plugins_retreatdayplan"
pg_restore: from TOC entry 378; 1259 19900 TABLE plugins_retreatdayplan delme
pg_restore: error: could not execute query: ERROR: role "delme" does not exist
Command was: ALTER TABLE public.plugins_retreatdayplan OWNER TO delme;
In short, the restoration does not work, so my thinking is to troubleshoot these errors. What is causing the Postgres to bring this information into the dump, despite my creating a new database with a new connection user? How can I make these errors go away?

The --no-owner flag prevents writing ownership when used with pg_dump.
Alternatively use --no-owner with pg_restore to avoid restoring ownership. All objects created will then belong to the user that was used to run pg_restore (in your example all of them would belong to postgres)

Related

Access privilege on enum type shows oid - can't drop

I have a PG 9.4 database that has an enum type. When I look at privs on that type I see
List of data types
Schema | Name | Internal name | Size | Elements | Owner | Access privileges | Description
-------------------+--------------------------+---------------+------+----------+----------+-------------------+-------------
myschema | myscchema.gender | gender | 4 | f +| postgres | =U/16524 |
| | | | m | | |
(1 row)
Note the =U/16524
I can't find the role in pg_authid or pg_auth_mmebers or anywhere. So can't revoke it.
And can't easily drop the enum since it's tied to a used production table.
And then when I use this db to test restores, it fails since that role does not exist.
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3286; 0 0 ACL TYPE gender postgres
pg_restore: [archiver (db)] could not execute query: ERROR: role "16524" does not exist
Command was: REVOKE ALL ON TYPE myschema.gender FROM PUBLIC;
Is this a bug, is there some fix (besides upgrading which we're planning to)?
Searched thru catalog views. Have tried revoking all from public.

Postgresql error database does not exist however it exists when listing the databases

When I drop the database I have an error that it doesn't exist however when I list the databases I can see it.
Here are the steps I am following:
sudo -u postgres psql
postgres=# \l
postgres=# DROP DATABASE IF EXISTS mydbname;
NOTICE: database "mydbname" does not exist, skipping
DROP DATABASE
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
--------------------+---------------+----------+---------+---------+----------------------------
mydbname | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =Tc/postgres +
| | | | | postgres=CTc/postgres +
| | | | | cryptouser=CTc/postgres

Postgres will list my database, but it doesn't exist when I try connecting to it

From terminal, I
sudo su postgres
psql
\l:
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
nwnx | nwnx | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
But when trying to connect to it:
\c nwnx:
FATAL: database "nwnx" does not exist
Previous connection kept
quote_ident:
postgres=# select quote_ident(datname) from pg_database;
quote_ident
-------------
postgres
template1
template0
nwnx
(4 rows)
dumpall:
pg_dumpall --schema-only | grep '\connect'
\connect template1
pg_dump: [archiver (db)] connection to database "nwnx" failed: FATAL: database "nwnx" does not exist
pg_dumpall: pg_dump failed on database "nwnx", exiting
Creation script:
pg_dumpall --schema-only | grep -i database
-- PostgreSQL database cluster dump
-- PostgreSQL database dump
-- Dumped from database version 11.5
-- PostgreSQL database dump complete
pg_dump: [archiver (db)] connection to database "nwnx" failed: FATAL: database "nwnx" does not exist
pg_dumpall: pg_dump failed on database "nwnx", exiting
Connecting as nwnx user
$: psql postgres -U nwnx
psql (11.5)
Type "help" for help.
postgres=> \conninfo
You are connected to database "postgres" as user "nwnx" via socket in "/run/postgresql" at port "5432".
postgres=> \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
nwnx | nwnx | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
postgres=> \c nwnx
FATAL: database "nwnx" does not exist
Previous connection kept
WORKING SOLUTION BY #laurenz-albe:
Showing all dbs
postgres=# select oid, datname, datname::bytea FROM pg_database;
oid | datname | datname
-------+-----------+----------------------
13121 | postgres | \x706f737467726573
1 | template1 | \x74656d706c61746531
13120 | template0 | \x74656d706c61746530
59515 | nwnx | \x6e776e78
(4 rows)
Checking if nwnx is omitted (had to use alias for datname)
postgres=# SELECT oid, datname dn, datname::bytea FROM pg_database ORDER BY dn;
oid | dn | datname
-------+-----------+----------------------
13121 | postgres | \x706f737467726573
13120 | template0 | \x74656d706c61746530
1 | template1 | \x74656d706c61746531
(3 rows)
I followed the instructions from the solution and it worked perfectly! Thank you very much!
Postgres version is 11.5
Any hints on what I'm doing wrong or whats going on?
That looks a lot like database corruption, in particular like the index pg_database_datname_index (which is used in GetDatabaseTuple()) got corrupted.
To be sure, try the following:
-- should show all databases
SELECT oid, datname, datname::bytea FROM pg_database;
SET enable_seqscan = off;
-- should omit database "nwnx"
SELECT oid, datname, datname::bytea FROM pg_database ORDER BY datname;
If that confirms my suspicion, do the following:
Stop the database with
pg_ctl stop -m immediate -D /path/to/data/directory
and take a cold backup of the database directory.
Start the database with
pg_ctl start -o -P -D /path/to/data/directory
Connect to the database postgres and run
REINDEX TABLE pg_database;
Stop and restart PostgreSQL.
Now take a pg_dumpall from the cluster and restore it to a new cluster you create with initdb.
DROP TABLE IF EXISTS nwnx CASCADE;
will work

psql saying database does not exist but it does exist in pgadmin

I have just installed Postgres v10.4 on Windows and created, using pgadmin, a new database called analysis. It is there, I can see it in pgadmin, and it has one table in it. However, I cannot connect to this database using psql.
C:\WINDOWS\system32>psql -d postgres -U postgres
psql (10.4)
WARNING: Console code page (850) differs from Windows code page (1252)
8-bit characters might not work correctly. See psql reference
page "Notes for Windows users" for details.
Type "help" for help.
postgres=# l
postgres-# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-----------------------------+-----------------------------+-----------------------
postgres | postgres | UTF8 | English_United Kingdom.1252 | English_United Kingdom.1252 |
template0 | postgres | UTF8 | English_United Kingdom.1252 | English_United Kingdom.1252 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | English_United Kingdom.1252 | English_United Kingdom.1252 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
postgres-# \q
C:\WINDOWS\system32>psql -d analysis -U postgres
psql: FATAL: database "analysis" does not exist
C:\WINDOWS\system32>
I don't know what I'm doing with psql, but all I've found is that I can connect to db postgres as user postgres. There's a \l command and it shows 3 databases (2 of which, template0 and template1) I can't even see in pgadmin.
How can I connect to the database (analysis) that I've just created in pgadmin? Can anybody explain what user I'm logged in under in pgadmin, if it's different to 'postgres' and if it isn't, how is it that I can't see in psql what I can see in pgadmin?
template0 and template1 are know as skeleton databases. When you use CREATE DATABASE command postgres copy the existing databae.
By default template1 database is used to create new database.
I think your pgadmin and psql are connected to different cluster. A cluster in postgresql is collection of one or more databases in a single instance of server
The image you posted does not show the complete view of pgadmin browser. But
if there are 2 cluster then you can see in pgadmin like "Servers(2)". To find running port of each cluster right click on corresponding cluster and select properties then click connection tab. Here port number can be seen.
Then connect to cluster in psql using that Port.

pg_restore on table failing because of hstore

Background:
I'm using PostgreSQL 9.3.5 on Ubuntu 14.04.
After a bad script, I have a table that I need to restore from a dump file created via pg_dump. On this table, I have an audit trigger that is based on
this wiki page. As you can see, the trigger function uses an hstore.
Error:
When I attempt the restore, I get:
$ pg_restore -a --dbname=a193 -Fc --host=localhost --port=5434 --username=postgres -W --table=foo ~/tmp/a193.dump
Password:
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 4600; 0 26146 TABLE DATA foo u2su8s81ul0a52
pg_restore: [archiver (db)] COPY failed for table "foo": ERROR: type "hstore" does not exist
LINE 6: h_old hstore;
The extension definitely does exist.
=> \dx
List of installed extensions
+--------------------+---------+------------+--------------------------------------------------------------+
| Name | Version | Schema | Description |
+--------------------+---------+------------+--------------------------------------------------------------+
| dblink | 1.1 | public | connect to other PostgreSQL databases from within a database |
| hstore | 1.2 | public | data type for storing sets of (key, value) pairs |
| isn | 1.0 | public | data types for international product numbering standards |
| pg_stat_statements | 1.1 | public | track execution statistics of all SQL statements executed |
| pgcrypto | 1.0 | public | cryptographic functions |
| plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language |
| plpythonu | 1.0 | pg_catalog | PL/PythonU untrusted procedural language |
| postgres_fdw | 1.0 | public | foreign-data wrapper for remote PostgreSQL servers |
| uuid-ossp | 1.0 | public | generate universally unique identifiers (UUIDs) |
+--------------------+---------+------------+--------------------------------------------------------------+
(9 rows)
And I can use it in a query (as the postgres user - the same role as I'm using above for the restore):
=> select current_user;
+--------------+
| current_user |
+--------------+
| postgres |
+--------------+
(1 row)
=> \du
List of roles
+----------------+------------------------------------------------+-----------+
| Role name | Attributes | Member of |
+----------------+------------------------------------------------+-----------+
| postgres | Superuser, Create role, Create DB, Replication | {} |
| u2su8s81ul0a52 | | {} |
+----------------+------------------------------------------------+-----------+
=> select 'a=>1'::hstore;
+----------+
| hstore |
+----------+
| "a"=>"1" |
+----------+
(1 row)
Questions:
Why am I getting this error when the database has this extension installed?
Other than dropping the trigger, how can I get around this issue? Dropping trigger is not the worse thing in the world, but it seems like this should be possible and in a production database, I'd like to be able to see the audit trail that someone did a restore of the data, etc.
It appears to be a bug in either pg_dump or pg_restore. Per Richard Huxton's suggestion above I restored to a file.
pg_restore --data-only --table=foo -f ~/tmp/foo.sql ~/tmp/a193.dump
When I looked at the contents, I found that it was doing the following at the top:
SET statement_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET client_min_messages = warning;
SET search_path = myschema, pg_catalog;
Running this line from inside of psql with \i still fails, but editing the last line to include the public schema (which is where hstore is installed) works.
SET search_path = myschema, pg_catalog, public;
I'm then able to run from inside psql with \i and import the lost data.
I had the same issue with two functions (checksum and is_valid) and a table (master_values) defined in public. Here checksum was calling is_valid and master_values had a check-constraint:
"master_values_master_id_check" CHECK(is_valid(master_id))"
Note that there was no search_path or schema-reference used in any of these.
When trying to restore the dump I got this during the restore:
pg_restore: [archiver (db)] COPY failed for table "master_values": ERROR: function checksum(integer) does not exist
The strange thing is that after the restore both functions and the table are there and works as expected. The only thing missing is the data in master_values that did not get restored.
This was solved by specifying the search_path for is_valid:
ALTER FUNCTION is_valid SET search_path = public;
For more info on this see:
The docs: https://www.postgresql.org/docs/current/sql-alterfunction.html
Article1: https://www.postgresonline.com/article_pfriendly/279.html
Article2: https://www.endpoint.com/blog/2012/11/15/postgresql-searchpath-behaviour