I run a daily backup of my database using pg_dump and pg_restore that recently stopped working after I pushed an update.
I have a function validate_id that's a Case/When statement just as a quick check for some the data that has integrity issues. Looks something like this:
CREATE OR REPLACE FUNCTION validate_id(
_string text,
_type type
) RETURNS boolean AS
$$
SELECT
CASE WHEN (stuff) THEN TRUE
WHEN (other stuff) THEN TRUE
When (more stuff) THEN raise_err('Not an accepted type, the accepted types are: x y z')
ELSE FALSE
$$
LANGUAGE SQL;
Since I added this function, when I dump using this command:
pg_dump -U postgres -h ipaddress -p 5432 -w -F t databaseName > backupsfolder/databaseName.tar
When I use this command:
pg_restore -U postgres -h localhost -p 5432 -d postgres -C "backupsfolder/databaseName.tar"
As of two days ago, this now throws an error:
pg_restore: error: could not execute query: ERROR: function raise_err(unknown) does not exist
I'm pretty lost on what to do. I think what might be going on is that it's trying to restore this function before it restores the raise_err function. Which I thought was built-in to postgres (I can SELECT raise_err('Hello, World');). Is this possible? Is it my CASE statement because I need to return only Booleans? All of the permissions seem correct and restoring with previous backups works fine.
The problem is that raise_err is not schema qualified in your function code.
This is potentially dangerous: a malicious user could create his own function raise_err and set search_path so that the wrong function is called.
Since pg_restore is typically run by a superuser, this can be a security problem. Imagine such a function being used in an index definition!
For these reasons pg_dump and pg_restore set an empty search_pathin current versions of PostgreSQL.
The solution to your problem is to explicitly use the function's schema in your SQL statement.
I ended up solving this issue by explicitly setting the search paths for both functions, raise_err() and validate_id() to public:
ALTER FUNCTION validate_id(text,text) SET search_path=public;
ALTER FUNCTION raise_err(text,text) SET search_path=public;
Related
Assuming I have the following table, functions and data
create table people (
id bigserial primary key,
age int,
height int,
weight int
);
create or replace function add_person (p_age int, p_height int, p_weight int);
create or replace function get_person_by_age (p_age int);
create or replace function get_person_by_height (p_height int);
create or replace function get_person_by_weight (p_weight int);
add_person (20,180,100);
add_person (20,181,101);
add_person (20,182,102);
add_person (20,183,103);
I am currently testing my database purely in bash so I would have these very long files like so (pseudocode only)
#!/bin/bash
# Insert data
sudo -u postgres psql -d $db_name -c "add_person(20,180,100)"
sudo -u postgres psql -d $db_name -c "add_person(20,181,101)"
sudo -u postgres psql -d $db_name -c "add_person(20,182,102)"
sudo -u postgres psql -d $db_name -c "add_person(20,183,103)"
# Retrieve data
persons=$(sudo -u postgres psql -d $db_name -c "get_person_by_age (20)")
# Count number of rows and make sure it matches the expected outcome
# (You have to manually strip header and footer lines but ignore for now)
if [ $(echo $persons | wc -l) -ne 4]
then
echo "Fail"
exit 1
fi
My test scripts have grown too large and there are so many things I am trying to catch (actions which should throw errors but which do not ie. false positives, actions which should not throw errors but which do ie false negatives, actions which throw errors other than that which they are supposed to, etc.). More importantly, the tests are incredibly slow as bash keeps trying to establish a connection to Postgre.
The reason I am not doing this in PGSQL is because the logic of queries can grow very complex as my db queries have many filters.
Is there a better existing solution to solve my problem? I looked at pgTAP but the documentation for that is horrendous
First of all, I think you could gain a lot of speed by running multiple commands in one client initialization. The "-c" flag only runs one command, and there is some small overhead in starting a connection that really adds up if you are running many commands, as you said.
For example, you could do something like this for your insert commands:
sudo -u postgres psql -d $db_name << EOF
add_person(20,180,100);
add_person(20,181,101);
add_person(20,182,102);
add_person(20,183,103);
Alternatively, you could list all the commands you want to run in a file and run them with "--file"/"-f":
sudo -u postgres psql -d $db_name -f "my_add_commands.psql"
I can't speak to why you are getting false errors, but generally I find that writing complicated test scripts in bash is a pain. If possible, I would consider using any language you are familiar with that has proper try/catch logic and a robust testing library with assertions. This is personally how I write tests for SQL functions. However, you would probably also need to use library that lets you make and process psql queries in the respective language.
There is a light psql testing library for python, but I haven't used it myself. Maybe worth a look though.
When I do dump/restore everything works fine. But when I add option --no-owner and dump/restore database the queries slows down.
For example one complex query run about 3 minues instead of 1sec before dump/restore
The only thing that is changed is --no-owner for pg_dump utility
I suppose when dump with --no-owner option and restore such database then user, which restore database, has no access to index info. But this is just assumption
DETAILS
make docker-dbdump && make docker-dbrestore slows down query execution:
.ONESHELL:
docker-dbdump:
file=${APP_ROOT}/db/${DB_NAME}-$$(date "+%Y-%m-%d_%H-%M-%S").sql.gz
docker exec ${DOCKER_CONTAINER} pg_dump --no-owner -U postgres ${DB_NAME} \
| gzip -f > $${file}
cp $${file} ${APP_ROOT}/db/${DB_NAME}.sql.gz
docker-dbrestore: dbclear
zcat ${APP_ROOT}/db/${DB_NAME}.sql.gz | \
docker exec -i ${DOCKER_CONTAINER} psql -U ${DB_USER} -d ${DB_NAME}
When I remove --no-owner options then everything is fine.
UPD
ERROR: permission denied to create extension "btree_gist"
HINT: Must be superuser to create this extension.
ERROR: extension "btree_gist" does not exist
...
ERROR: data type integer has no default operator class for access method "gist"
HINT: You must specify an operator class for the index or define a default operator class for the data type.
I want to compare how database is dumped with and w/o --no-owner option and show later today
Slows down because of error:
HINT: Must be superuser to create this extension.
ERROR: extension "btree_gist" does not exist
Extension is not created, so related indexes also are not created.
Without indexes database slows-down =(
When I fixed error with rights all also starts to work super fast!
I work with postgresql boosted with timescaledb fork (pretty impressed with its performance while it worked ;)
I got a script that downloads data, modifies it and puts into a csv file.
Then a psql script is invoked to create a temp table that inserts data into the database
psql -U postgres -d q1 -c "CREATE TABLE tmpp (time bigint NOT NULL, ask real NOT NULL, bid real NOT NULL)"
psql -U postgres -d q1 -c "\copy tmpp (time, ask, bid) from '/sth/sth.csv' delimiter ',' CSV"
psql -U postgres -d q1 -c "insert into realfun select * from tmpp"
psql -U postgres -d q1 -c "DROP TABLE tmpp"
Funny thing is, that it worked for me before, but now I got an error :
ERROR: Deprecated trigger function should not be invoked
I must have messed up sth, but cant figure out what it is [how original]
I will be happy to provide more details, if needed
I cannot find anything similar in google, please advise
It seems that the problem is that you have a newer shared library version than the extension version you have installed (Timescale is an extension, not a fork). You can fix this with ALTER EXTENSION timescaledb UPDATE.
The alter command is documented here.
all I am trying to restore my db from pg_dump that I got from my server. I know there are plenty similar question, I tried what was suggested but still can not resolve my issue.
I created my pg_dump via this command:
pg_dump name_of_database > name_of_backup_file
Then I create new db:
createdb -T template0 restored_database
Then I restore using this command:
psql -1 -f name_of_backup_file.sql restored_database
It runs and then I got different syntax errors, for example:
psql:nurate_pg_dump.sql:505: invalid command \nWatch
Query buffer reset (cleared).
psql:nurate_pg_dump.sql:512: invalid command \nThe
psql:nurate_pg_dump.sql:513: invalid command \N
psql:nurate_pg_dump.sql:4098: ERROR: syntax error at or near "9"
LINE 1: 9 the course is not difficult, but Zauresh Atakhanova is not...
I believe since I made pg_dump from the same server, and try to restore it there nothing changed in my server setup, or version of postgres, so I think my db should restore correctly. How can I resolve these syntax errors?
EDIT: Ok, the problem is not with the syntax:
CREATE EXTENSION
ERROR: must be owner of extension plpgsql
This error is thrown at this line:
CREATE EXTENSION IF NOT EXISTS plpgsql WITH SCHEMA pg_catalog;
try to restore your database with
psql restored_database < name_of_backup_file
I am having a problem upgrading my Postgresql 9.2 database to 9.4. The problem I have is that I have these tables that are incompatible for an upgrade:
public.pg_ts_dict.dict_init
public.pg_ts_dict.dict_lexize
public.pg_ts_parser.prs_start
public.pg_ts_parser.prs_nexttoken
public.pg_ts_parser.prs_end
public.pg_ts_parser.prs_headline
public.pg_ts_parser.prs_lextype
Postgresql says that I should delete these tables and the upgrade should work. I am currently trying to figure out how to do this.
Postgresql has pg_catalog.pg_ts_parser and pg_catalog.pg_ts_dict. These are system catalogs and can absolutely not be removed, nor do I want to. I want to remove the public.pg_ts_* tables.
More specifically I want to dump the tables, upgrade the database, and then restore both public.pg_ts_parser and public.pg_ts_dict. However every time I try to dump or drop the tables, it defaults to the system catalog. Which I don't want. How can I specify these exact tables? Thanks for any help in advance.
-------EDIT------
Here are the commands I am running to dump the tables.
pg_dump -Fc -t public.pg_ts_dict -t public.pg_ts_parser > file.dump
pg_dump: No matching tables were found
Here is a variation
pg_dump -Fc -t pg_ts_dict -t pg_ts_parser > file.dump
The second variation contains the dump of the system catalog pg_ts_dict and parser not the public version. However, it is very confusing because the contents of the file.dump contains these lines of code among # signs and ^ signs.
DROP TABLE pg_catalog.pg_ts_dict;
^#^#^#pg_catalog^#^#^#^#^#^#^H^#^#^#postgres^#^D^#^#^#true^A^A^#^#^#^C^#^#^#^#^#^#^#^#^# ^G^#^#^#^#^#^#^#^#^A^#^#^#0^#^A^#^#^#0^#
^#^#^#pg_ts_dict^#^C^#^#^#ACL^#^A^#^#^#^#<86>^#^#^#REVOKE ALL ON TABLE pg_ts_dict FROM PUBLIC;
REVOKE ALL ON TABLE pg_ts_dict FROM postgres;
GRANT SELECT ON TABLE pg_ts_dict TO PUBLIC;
^#^#^#^#^#^A^A^#^#^#^#
^#^#^#pg_catalog^A^A^#^#^#^#^H^#^#^#postgres^#^E^#^#^#false^#^B^#^#^#54^A^A^#^#^#^C^#^#^#^#^#^#^#^#^#7^#^#^#^#^#^#^#^#^#^D^#^#^#1259^#^D^#^#^#3601^#^L^#^#^#pg_ts_parser^#^E^#^#^#TABLE^#^B^#^#^#^#ö^#^#^#CREATE TABLE pg_ts_parser (
prsname name NOT NULL,
prsnamespace oid NOT NULL,
prsstart regproc NOT NULL,
prstoken regproc NOT NULL,
prsend regproc NOT NULL,
prsheadline regproc NOT NULL,
prslextype regproc NOT NULL
Not sure what to make of this.
Your call should actually work as is:
pg_dump -Fc -t public.pg_ts_dict -t public.pg_ts_parser > file.dump
You can use a wildcard to include all tables starting with pg_ts_.
pg_dump -Fc -t 'public.pg_ts_*' > file.dump
On the Linux shell, you may need the extra quotes. (Related question on dba.SE.) Remove the quotes in Windows.
To make it abundantly clear you could exclude the same tables from pg_catalog explicitly. Normally, this is not necessary, but something seems to be abnormal in your case.
pg_dump -Fc -t 'public.pg_ts_*' -T 'pg_catalog.pg_ts_*' > file.dump
The documentation:
Also, you must write something like -t sch.tab to select a table in a
particular schema, rather than the old locution of -n sch -t tab.