pg_restore on table failing because of hstore - postgresql

Background:
I'm using PostgreSQL 9.3.5 on Ubuntu 14.04.
After a bad script, I have a table that I need to restore from a dump file created via pg_dump. On this table, I have an audit trigger that is based on
this wiki page. As you can see, the trigger function uses an hstore.
Error:
When I attempt the restore, I get:
$ pg_restore -a --dbname=a193 -Fc --host=localhost --port=5434 --username=postgres -W --table=foo ~/tmp/a193.dump
Password:
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 4600; 0 26146 TABLE DATA foo u2su8s81ul0a52
pg_restore: [archiver (db)] COPY failed for table "foo": ERROR: type "hstore" does not exist
LINE 6: h_old hstore;
The extension definitely does exist.
=> \dx
List of installed extensions
+--------------------+---------+------------+--------------------------------------------------------------+
| Name | Version | Schema | Description |
+--------------------+---------+------------+--------------------------------------------------------------+
| dblink | 1.1 | public | connect to other PostgreSQL databases from within a database |
| hstore | 1.2 | public | data type for storing sets of (key, value) pairs |
| isn | 1.0 | public | data types for international product numbering standards |
| pg_stat_statements | 1.1 | public | track execution statistics of all SQL statements executed |
| pgcrypto | 1.0 | public | cryptographic functions |
| plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language |
| plpythonu | 1.0 | pg_catalog | PL/PythonU untrusted procedural language |
| postgres_fdw | 1.0 | public | foreign-data wrapper for remote PostgreSQL servers |
| uuid-ossp | 1.0 | public | generate universally unique identifiers (UUIDs) |
+--------------------+---------+------------+--------------------------------------------------------------+
(9 rows)
And I can use it in a query (as the postgres user - the same role as I'm using above for the restore):
=> select current_user;
+--------------+
| current_user |
+--------------+
| postgres |
+--------------+
(1 row)
=> \du
List of roles
+----------------+------------------------------------------------+-----------+
| Role name | Attributes | Member of |
+----------------+------------------------------------------------+-----------+
| postgres | Superuser, Create role, Create DB, Replication | {} |
| u2su8s81ul0a52 | | {} |
+----------------+------------------------------------------------+-----------+
=> select 'a=>1'::hstore;
+----------+
| hstore |
+----------+
| "a"=>"1" |
+----------+
(1 row)
Questions:
Why am I getting this error when the database has this extension installed?
Other than dropping the trigger, how can I get around this issue? Dropping trigger is not the worse thing in the world, but it seems like this should be possible and in a production database, I'd like to be able to see the audit trail that someone did a restore of the data, etc.

It appears to be a bug in either pg_dump or pg_restore. Per Richard Huxton's suggestion above I restored to a file.
pg_restore --data-only --table=foo -f ~/tmp/foo.sql ~/tmp/a193.dump
When I looked at the contents, I found that it was doing the following at the top:
SET statement_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET client_min_messages = warning;
SET search_path = myschema, pg_catalog;
Running this line from inside of psql with \i still fails, but editing the last line to include the public schema (which is where hstore is installed) works.
SET search_path = myschema, pg_catalog, public;
I'm then able to run from inside psql with \i and import the lost data.

I had the same issue with two functions (checksum and is_valid) and a table (master_values) defined in public. Here checksum was calling is_valid and master_values had a check-constraint:
"master_values_master_id_check" CHECK(is_valid(master_id))"
Note that there was no search_path or schema-reference used in any of these.
When trying to restore the dump I got this during the restore:
pg_restore: [archiver (db)] COPY failed for table "master_values": ERROR: function checksum(integer) does not exist
The strange thing is that after the restore both functions and the table are there and works as expected. The only thing missing is the data in master_values that did not get restored.
This was solved by specifying the search_path for is_valid:
ALTER FUNCTION is_valid SET search_path = public;
For more info on this see:
The docs: https://www.postgresql.org/docs/current/sql-alterfunction.html
Article1: https://www.postgresonline.com/article_pfriendly/279.html
Article2: https://www.endpoint.com/blog/2012/11/15/postgresql-searchpath-behaviour

Related

Access privilege on enum type shows oid - can't drop

I have a PG 9.4 database that has an enum type. When I look at privs on that type I see
List of data types
Schema | Name | Internal name | Size | Elements | Owner | Access privileges | Description
-------------------+--------------------------+---------------+------+----------+----------+-------------------+-------------
myschema | myscchema.gender | gender | 4 | f +| postgres | =U/16524 |
| | | | m | | |
(1 row)
Note the =U/16524
I can't find the role in pg_authid or pg_auth_mmebers or anywhere. So can't revoke it.
And can't easily drop the enum since it's tied to a used production table.
And then when I use this db to test restores, it fails since that role does not exist.
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3286; 0 0 ACL TYPE gender postgres
pg_restore: [archiver (db)] could not execute query: ERROR: role "16524" does not exist
Command was: REVOKE ALL ON TYPE myschema.gender FROM PUBLIC;
Is this a bug, is there some fix (besides upgrading which we're planning to)?
Searched thru catalog views. Have tried revoking all from public.

Postgres DB Dump Carries Owner Information With It

I am setting up a system of DB backup for a web project. At the moment, the DB is being backed up via Bash script like this
#!/bin/bash
# Set the DB configuration
export PGHOST=localhost
export PGPORT=5432
#export PGDATABASE=yoga
#export PGUSER=delme
export PGDATABASE=projectA
export PGUSER=userA
export PGPASSWORD=blah-blah
# Backs up mydatabase to a file and then uploads it to AWS S3.
# First, dump database backup to a file
TIME=$(date "+%s")
BACKUP_FILE="postgres_${PGDATABASE}_${TIME}.pgdump"
echo "Backing up $PGDATABASE to $BACKUP_FILE"
pg_dump --format=custom > $BACKUP_FILE
In the cluster, I have the following roles
Role name | Attributes | Member of
-----------+------------------------------------------------------------+-----------
delme | Superuser | {}
userA | | {}
postgres | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
The list of databases
postgres=# \list
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+---------+-----------------------
projectA | userA | UTF8 | C.UTF-8 | C.UTF-8 | =Tc/userA +
| | | | | userA=CTc/userA
postgres | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
yoga | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
(5 rows)
Whenever I bring this dump on a local machine and try to restore via pg_restore --host=localhost --port=5432 --username=postgres --verbose --create --dbname=new_db_name --format=custom --clean postgres_projectA_1660909386.pgdump, I get about 200 errors.
Most errors are of the following format:
pg_restore: creating SEQUENCE OWNED BY "public.plugins_mentorsplugin_mentors_id_seq"
pg_restore: creating TABLE "public.plugins_plugingeneral"
pg_restore: from TOC entry 374; 1259 19889 TABLE plugins_plugingeneral delme
pg_restore: error: could not execute query: ERROR: role "delme" does not exist
Command was: ALTER TABLE public.plugins_plugingeneral OWNER TO delme;
pg_restore: creating TABLE "public.plugins_retreatcommentplugin"
pg_restore: from TOC entry 375; 1259 19892 TABLE plugins_retreatcommentplugin delme
pg_restore: error: could not execute query: ERROR: role "delme" does not exist
Command was: ALTER TABLE public.plugins_retreatcommentplugin OWNER TO delme;
pg_restore: creating TABLE "public.plugins_retreatcommentplugin_comments"
pg_restore: from TOC entry 376; 1259 19895 TABLE plugins_retreatcommentplugin_comments delme
pg_restore: error: could not execute query: ERROR: role "delme" does not exist
Command was: ALTER TABLE public.plugins_retreatcommentplugin_comments OWNER TO delme;
pg_restore: creating SEQUENCE "public.plugins_retreatcommentplugin_comments_id_seq"
pg_restore: from TOC entry 377; 1259 19898 SEQUENCE plugins_retreatcommentplugin_comments_id_seq delme
pg_restore: error: could not execute query: ERROR: role "delme" does not exist
Command was: ALTER TABLE public.plugins_retreatcommentplugin_comments_id_seq OWNER TO delme;
pg_restore: creating SEQUENCE OWNED BY "public.plugins_retreatcommentplugin_comments_id_seq"
pg_restore: creating TABLE "public.plugins_retreatdayplan"
pg_restore: from TOC entry 378; 1259 19900 TABLE plugins_retreatdayplan delme
pg_restore: error: could not execute query: ERROR: role "delme" does not exist
Command was: ALTER TABLE public.plugins_retreatdayplan OWNER TO delme;
In short, the restoration does not work, so my thinking is to troubleshoot these errors. What is causing the Postgres to bring this information into the dump, despite my creating a new database with a new connection user? How can I make these errors go away?
The --no-owner flag prevents writing ownership when used with pg_dump.
Alternatively use --no-owner with pg_restore to avoid restoring ownership. All objects created will then belong to the user that was used to run pg_restore (in your example all of them would belong to postgres)

Cannot install any language with Postgres 9.2

I've been using Postgres 9.2 on several different servers for some time, but on one specific database on one specific instance of Postgres I can't install any languages.
$ sudo su - postgres
$ psql
postgres=# select * from pg_language;
This shows internal, sql, c, and plpgsql.
If I connect to myDb and try again:
postgres=# \connect myDb
You are now connected to database "myDb" as user "postgres".
postgres=# select * from pg_language;
I only see internal, sql, and c. I have tried installing the language using createlang as shown in their docs and receive an error:
$ createlang plpgsql myDb
createlang: language installation failed: ERROR: language validation function 2247 called for language 13 instead of 1
I can see that plpgsql.so is in the proper place — and it has to be for the postgres database to have it.
This is also not plpgsql specific as I get the same error message with plpythonu. I can install in the postgres database but not the myDb database.
The documentation and forums do not address this issue.
It seems that your pl_language catalog is broken.
You should see the following:
SELECT oid, * FROM pg_language ;
oid | lanname | lanowner | lanispl | lanpltrusted | lanplcallfoid | laninline | lanvalidator | lanacl
-------+----------+----------+---------+--------------+---------------+-----------+--------------+--------
12 | internal | 10 | f | f | 0 | 0 | 2246 |
13 | c | 10 | f | f | 0 | 0 | 2247 |
14 | sql | 10 | f | t | 0 | 0 | 2248 |
(3 rows)
Instead, your entry for the c language (OID 13) seems to have 1 instead of 2247 for lanvalidator. Any idea how that happened? Did you manipulate the catalogs?
See if there are any other differences to the above in your pg_language.
The safe way to proceed would be to pg_dump your database, drop and re-create it and load the dump. That should take care of all catalog manipulations.
If you prefer to live dangerous, you can try the following as superuser:
UPDATE pg_language SET lanvalidator = 2247 WHERE oid = 13;

Revoke privilege "create table" from public schema but not from custom made schema

I am trying to create an account that is only able to use "SELECT" within the mydb_schema.
I am able to revoke the creation of a tables in public schema in my database (mydb) from a specific role but I am not able to revoke the creation of tables within the schema (mydb_schema) the role (user_role) is assigned to.
Can someone assist? What am I missing?
Below are the commands I used as user postgres and as the owner of the schema. At the end I also altered the postgres account by using the noinherit just as a desperate test. No result.
\c mydb
mydb=# create role user_account
mydb=# alter role user_account login noinherit;
mydb=# alter role user_account set search_path = 'mydb_schema';
##Up until this point I don't have SELECT pivileges but I can create a table within mydb_schema only and not in public.
mydb=# grant select on all tables in schema mydb_schema to user_account;
mydb=# revoke CREATE on SCHEMA public from public;
These are my databases:
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges | Size | Tablespace | Description
--------------------------------------------------------------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 6820 kB | pg_default | default administrative connection database
mydb | seeker | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/seeker +| 3465 MB | pg_default |
seeker=CTc/seeker
These are my schemas:
List of schemas
Name | Owner | Access privileges | Description
----------------+----------+----------------------+------------------------
public | postgres | postgres=UC/postgres | standard public schema
mydb_schema | seeker | seeker=UC/seeker +|
| | =UC/seeker |
Found the solution here in dba.stackexchange
alter role servers_admin set default_transaction_read_only = 'ON';

Description of PostgreSQL database tables installed on a particular computer

A bit over a year ago I installed PostgreSQL on six computers. On one of those computers, I imported 2 gig of census data from CSV files.
Historically I have just worked with flat files, but in this case the files are so big they choke my analysis software. I am new to both PostgreSQL and relational databases in general, and I have a very basic beginner's question: What software (e.g. pgAdmin III) and what and commands would I use to quickly answer the following questions on each machine:
Is PostgreSQL in still installed and running on each machine?
(If 1 is yes) Does the machine in question have installed any non-bundled tables or data?
(If 2 is yes) How can I produce a summary description of the tables that are installed?
In terms of a summary description, I am hoping for the table name, a list of column names, the data type of each, and the number of lines or records in each table, and possibly any additional database-relevant facts like whether the column is a key or indexed.
I work mainly under Windows 7 & 8, though I have a virtual Ubuntu macine installed on one computer.
Is postgreSQL in still installed and running on each machine?
The method you would use to find that out will depend on the operating system on each machine.
On Linux hosts you could use this:
ps -ef | grep postgres
If you see a process named postgres then postgresql is installed and running.
If not, it may be installed but not running. You could check the package management system of your distro to check if it is installed, for example on RPM based systems:
rpm -qa "*postgres*"
On a Windows machine you may be able to see if it is running using the task manager. To check if it is installed go into the Control Panel "Programs and Features" option.
Does the machine in question have installed any non-bundled tables or data?
By non-bundled I assume you mean tables or data other than the system catalogs that are created when you install the system.
My preference is to interact via the command line psql interface. Once you get the psql prompt you can use various 'backslash commands' to inspect the database.
To open the psql command - well you will need appropriate credentials. The details are going to depend on how you configured things when you installed it. If you happen to be using Linux, and you have root access, then the easiest way is to su to the postgres Linux user first, which in most cases will be able to connect directly to the database:
$ sudo su - postgres
$ psql
To see what databases exist use the \l command:
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
harmic | harmic | UTF8 | en_AU.UTF-8 | en_AU.UTF-8 |
postgres | postgres | UTF8 | en_AU.UTF-8 | en_AU.UTF-8 |
template0 | postgres | UTF8 | en_AU.UTF-8 | en_AU.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_AU.UTF-8 | en_AU.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
The databases postgres, template0 and template1 are created by the system so the only database containing user data is 'harmic' in this case.
To connect to a database use \c:
postgres=# \c harmic
You are now connected to database "harmic" as user "postgres".
To list all the tables that exist in this database, including system catalogs use \dt:
harmic=# \dt+ *.*
List of relations
Schema | Name | Type | Owner | Size | Description
--------------------+-------------------------+-------+----------+------------+-------------
information_schema | sql_features | table | postgres | 96 kB |
information_schema | sql_implementation_info | table | postgres | 48 kB |
information_schema | sql_languages | table | postgres | 48 kB |
information_schema | sql_packages | table | postgres | 48 kB |
information_schema | sql_parts | table | postgres | 48 kB |
information_schema | sql_sizing | table | postgres | 48 kB |
information_schema | sql_sizing_profiles | table | postgres | 8192 bytes |
pg_catalog | pg_aggregate | table | postgres | 40 kB |
pg_catalog | pg_am | table | postgres | 40 kB |
... etc
public | aaa | table | harmic | 16 kB |
public | entry | table | harmic | 8192 bytes |
public | exams | table | harmic | 8192 bytes |
(60 rows)
The tables listed in the Schema's 'information_schema' and 'pg_catalog' are not user tables. pg_catalog contains the internal information used by the database to keep track of everything in the database, and information_schema contains information tables about the database which are standardized by the SQL standard. In my case there are a few tables in the 'public' schema that are actual user tables.
How can I produce a summary description of the tables that are installed?
To see a full description of one or more tables:
harmic=# \d public.aaa
Table "public.aaa"
Column | Type | Modifiers
--------+---------+-----------
a | integer |
b | text |
The above trivial table has two columns named a and b, of types integer and text respectively.
You can use wildcards to get this listing printed for matching tables, eg:
harmic=# \d public.*
You can get an exact number of rows that exists in each table by executing:
SELECT count(1) FROM aaa;
(where aaa would be the table name).
Doing this for each and every table could be slow and tedious. You can get an approximation of the number of rows in all tables by inspecting some tables in the pg_catalog like this:
harmic=# SELECT nspname as schemaname,
harmic-# relname as tablename,
harmic-# reltuples as approx_rows
harmic-# FROM pg_class LEFT JOIN pg_namespace ON pg_namespace.oid=pg_class.relnamespace WHERE nspname='public' and relkind='r';
schemaname | tablename | approx_rows
------------+-----------+-------------
public | exams | 3
public | entry | 2
public | aaa | 2
(3 rows)
Note that the number of rows shown is updated whenever the database is analysed. The autovacuum daemon does this automatically from time to time, but you can also manually trigger it using command ANALYZE;
Question 1: What OS are you running? If its a sensible one(*nix), try running psql in the terminal of any of them. That will definitely tell you if psql is running. Whether postgres is installed is a totally different question. Determining that is very situation specific (What OS do you have?).
Question 2: Im not familiar with bundling tables. But simple inspection of the GUI provided by pgAdmin would be your best bet. What is bundling?
Question 3: Best way to generate a summary of a series of tables would be to run raw sql :)
The following will answer most of your described needs. Connect to psql and run something like the following:
connect <database_name>
\d (will list all the tables in the database)
\d <tablename> ('describes' the table)
select count(*) from <tablename> (returns the total number of rows in the table.)
This is the best place for postgres stuff: http://www.postgresql.org/docs/
Bloody boring reading, but very simple and to the point.
Best of luck!