I've been running into an issue dumping and restoring one of my databases I think because of some extensions in the public schema. The extension that's throwing the error seems to be the Cube extension or the EarthDistance extension. This is the error I'm getting:
pg_restore: [archiver (db)] Error from TOC entry 2983;
pg_restore: [archiver (db)] could not execute query: ERROR: type "earth" does not exist
LINE 1: ...ians($1))*sin(radians($2))),earth()*sin(radians($1)))::earth
QUERY: SELECT cube(cube(cube(earth()*cos(radians($1))*cos(radians($2))),earth()*cos(radians($1))*sin(radians($2))),earth()*sin(radians($1)))::earth
CONTEXT: SQL function "ll_to_earth" during inlining
Command was: REFRESH MATERIALIZED VIEW public.locationsearch
I was having a similarly different issue with some functions that I had written myself and the issue ended up being the search path, so explicitly setting the search path for those functions to public solved my issue. I tried the same with ll_to_earth but it seems the entire extension is the problem. I don't really want to try and install an extension to pg_catalog because that seems like poor practice.
This is my typical dump command:
pg_dump -U postgres -h ipAddress -p 5432 -w -F t database > database.tar
Followed by:
pg_restore -U postgres -h localhost -p 5432 -w -d postgres -C "database.tar"
The full dump with data is about 4gb, but I tried dumping just the schema with -s and -F p and interestingly this is the beginning:
--
-- PostgreSQL database dump
--
-- Dumped from database version 12.2
-- Dumped by pg_dump version 12.2
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
--
-- Name: cube; Type: EXTENSION; Schema: -; Owner: -
--
CREATE EXTENSION IF NOT EXISTS cube WITH SCHEMA public;
--
-- Name: EXTENSION cube; Type: COMMENT; Schema: -; Owner:
--
COMMENT ON EXTENSION cube IS 'data type for multidimensional cubes';
--
-- Name: earthdistance; Type: EXTENSION; Schema: -; Owner: -
--
CREATE EXTENSION IF NOT EXISTS earthdistance WITH SCHEMA public;
--
-- Name: EXTENSION earthdistance; Type: COMMENT; Schema: -; Owner:
--
COMMENT ON EXTENSION earthdistance IS 'calculate great-circle distances on the surface of the Earth';
I guess I'm confused by...logically, isn't this the same as it would be if I'd used the tar format? I know the issue is that when pg_restore gets to that materialized view and tries to use the function ll_to_earth(float8, float8) it fails because that function either isn't in its search path or hasn't been restored yet, but wouldn't this indicate that the extensions are the first thing to be restored? Can this be fixed?
This is part of a script I wrote that will dump the databases on my production environment and restore the databases on my test environment daily so that they match. It worked for months until I started using this extension, and I'm lost on how to rectify it.
For security reasons, pg_dump sets the search_path empty, so only objects in the system schema pg_catalog will be found if they are referenced without schema qualification.
Now your SQL functions uses the data type earth without the schema (probably public), so you get the error message.
You will have to change the function to use qualified names like public.earth for extension objects. Alternatively, and probably better, is to fix the search_path for the function:
ALTER FUNCTION myfun SET search_path = public;
This is a good idea anyway, because otherwise your function will cease to work if a user changes search_path. Depending on the way your function is defined or used, this can even constitute a security problem (which is why pg_dump does it this way).
I found a way around in this answer. I create extensions in pg_catalog scheme, which is as documentation states always effectively part of the search path.
CREATE EXTENSION IF NOT EXISTS cube SCHEMA pg_catalog;
CREATE EXTENSION IF NOT EXISTS earthdistance SCHEMA pg_catalog;
People consider this a bad practice, because it might pollute system catalog etc. But it serves me well in my use case - now I can pg_dump/pg_restore without worries.
Note: I found this thread containing a patch for earthdistance extension. Unfortunately it hasn't been merged yet.
Had the same problem, however solutions above didn't work. During data insertion I got the following error
ERROR: type "earth" does not exist
LINE 1: ...ians($1))*sin(radians($2))),earth()*sin(radians($1)))::earth
QUERY: SELECT cube(cube(cube(earth()*cos(radians($1))*cos(radians($2))),earth()*cos(radians($1))*sin(radians($2))),earth()*sin(radians($1)))::earth
CONTEXT: SQL function "ll_to_earth" during inlining
In pgdump file I had to replace the following line (at the top of the file)
SELECT pg_catalog.set_config('search_path', '', false);
With
SELECT pg_catalog.set_config('search_path', 'public', false);
Related
When running restore from Production to my Local Machine using :
pg_restore server_employees.tar -d server_employees
I get this:
tore: from TOC entry 2; 3079 16570 EXTENSION set_user (no owner)
pg_restore: error: could not execute query: ERROR: could not open extension control file "/usr/share/postgresql/14/extension/set_user.control": No such file or directory
Command was: CREATE EXTENSION IF NOT EXISTS set_user WITH SCHEMA public;
pg_restore: from TOC entry 3260; 0 0 COMMENT EXTENSION set_user
pg_restore: error: could not execute query: ERROR: extension "set_user" does not exist
Command was: COMMENT ON EXTENSION set_user IS 'similar to SET ROLE but with added logging';
pg_restore: from TOC entry 301; 1259 16592 VIEW pg_stat_statements postgres
pg_restore: error: could not execute query: ERROR: column reference "wal_bytes" is ambiguous
LINE 33: pg_stat_statements.wal_bytes
^
Command was: CREATE VIEW metric_helpers.pg_stat_statements AS
SELECT pg_stat_statements.userid,
pg_stat_statements.dbid,
pg_stat_statements.queryid,
pg_stat_statements.query,
pg_stat_statements.plans,
pg_stat_statements.total_plan_time,
pg_stat_statements.min_plan_time,
pg_stat_statements.max_plan_time,
pg_stat_statements.mean_plan_time,
pg_stat_statements.stddev_plan_time,
pg_stat_statements.calls,
pg_stat_statements.total_exec_time,
pg_stat_statements.min_exec_time,
pg_stat_statements.max_exec_time,
pg_stat_statements.mean_exec_time,
pg_stat_statements.stddev_exec_time,
pg_stat_statements.rows,
pg_stat_statements.shared_blks_hit,
pg_stat_statements.shared_blks_read,
pg_stat_statements.shared_blks_dirtied,
pg_stat_statements.shared_blks_written,
pg_stat_statements.local_blks_hit,
pg_stat_statements.local_blks_read,
pg_stat_statements.local_blks_dirtied,
pg_stat_statements.local_blks_written,
pg_stat_statements.temp_blks_read,
pg_stat_statements.temp_blks_written,
pg_stat_statements.blk_read_time,
pg_stat_statements.blk_write_time,
pg_stat_statements.wal_records,
pg_stat_statements.wal_fpi,
pg_stat_statements.wal_bytes
FROM metric_helpers.pg_stat_statements(true) pg_stat_statements(userid, dbid, queryid, query, plans, total_plan_time, min_plan_time, max_plan_time, mean_plan_time, stddev_plan_time, calls, total_exec_time, min_exec_time, max_exec_time, mean_exec_time, stddev_exec_time, rows, shared_blks_hit, shared_blks_read, shared_blks_dirtied, shared_blks_written, local_blks_hit, local_blks_read, local_blks_dirtied, local_blks_written, temp_blks_read, temp_blks_written, blk_read_time, blk_write_time, wal_records, wal_fpi, wal_bytes);
pg_restore: error: could not execute query: ERROR: relation "metric_helpers.pg_stat_statements" does not exist
Command was: ALTER TABLE metric_helpers.pg_stat_statements OWNER TO postgres;
Any idea how to fix this ?
--install extension
source: https://github.com/pgaudit/set_user
gh repo clone pgaudit/set_user
cd set_user
make PG_CONFIG=/usr/local/pgsql15/bin/pg_config
sudo make install PG_CONFIG=/usr/local/pgsql15/bin/pg_config
In here
make PG_CONFIG=/usr/local/pgsql15/bin/pg_config
replace "/usr/local/pgsql15/bin/pg_config" to the binary app pg_config path in your computer.
then create extension in your new database cluster via psql or other client.
CREATE EXTENSION set_user;
In here, your new database cluster at least need at two extensions: set_user,pg_stat_statements. So make these these two available in the cluster.
pg_dump example:
/usr/local/pgsql15/bin/pg_dump -d test15 --format=tar --extension=set_user --extension=pg_stat_statements --table=test >test.tar
if you not use --extension, then all the old cluster extensions should be in the new cluster.
Restore to a new database:
/usr/local/pgsql15/bin/createdb -T template0 newdb1
/usr/local/pgsql15/bin/pg_restore --verbose -d newdb1 test.tar
In: https://www.postgresql.org/docs/current/app-pgdump.html
search keyword: --extension=pattern
I'm running pg_dump to create a script to automate the creation of a system like this:
pg_dump --dbname=postgresql://postgres:ohdsi#127.0.0.1:5432/OHDSI -t webapi.* > webapi.sql
This creates a sql script, but it is not really a sql script as it has code in it like what is shown below.
When this script is run as a sql script, it fails giving the error shown below.
Is there a way to get pg_dump to create a sql script that is standard sql and can be executed as a sql script?
Code sample from sql generated by pg_dump:
COPY webapi.cohort_version (asset_id, comment, description, version, asset_json, archived, created_by_id, created_date) FROM stdin;
\.
--
-- Data for Name: concept_of_interest; Type: TABLE DATA; Schema: webapi; Owner: ohdsi_admin_user
--
COPY webapi.concept_of_interest (id, concept_id, concept_of_interest_id) FROM stdin;
1 4329847 4185932
2 4329847 77670
3 192671 4247120
4 192671 201340
Error seen when running the script generated by pg_dump:
--
-- Name: penelope_laertes_uni_pivot id; Type: DEFAULT; Schema: webapi; Owner: ohdsi_admin_user
--
ALTER TABLE ONLY webapi.penelope_laertes_uni_pivot ALTER COLUMN id SET DEFAULT nextval('webapi.penelope_laertes_uni_pivot_id_seq'::regclass)
--
-- Data for Name: achilles_cache; Type: TABLE DATA; Schema: webapi; Owner: ohdsi_admin_user
--
COPY webapi.achilles_cache (id, source_id, cache_name, cache) FROM stdin
Error executing: COPY webapi.achilles_cache (id, source_id, cache_name, cache) FROM stdin
. Cause: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: COPY commands are only supported using the CopyManager API.
Where: COPY achilles_cache, line 1
Exception in thread "main" java.lang.RuntimeException: org.apache.ibatis.jdbc.RuntimeSqlException: Error executing: COPY webapi.achilles_cache (id, source_id, cache_name, cache) FROM stdin
. Cause: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: COPY commands are only supported using the CopyManager API.
Where: COPY achilles_cache, line 1
at org.yaorma.database.Database.executeSqlScript(Database.java:344)
at org.yaorma.database.Database.executeSqlScript(Database.java:332)
at org.nachc.tools.fhirtoomop.tools.build.postgres.build.A04_CreateAtlasWebApiTables.exec(A04_CreateAtlasWebApiTables.java:29)
at org.nachc.tools.fhirtoomop.tools.build.postgres.build.A04_CreateAtlasWebApiTables.main(A04_CreateAtlasWebApiTables.java:19)
Caused by: org.apache.ibatis.jdbc.RuntimeSqlException: Error executing: COPY webapi.achilles_cache (id, source_id, cache_name, cache) FROM stdin
. Cause: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: COPY commands are only supported using the CopyManager API.
Where: COPY achilles_cache, line 1
at org.apache.ibatis.jdbc.ScriptRunner.executeLineByLine(ScriptRunner.java:109)
at org.apache.ibatis.jdbc.ScriptRunner.runScript(ScriptRunner.java:71)
at org.yaorma.database.Database.executeSqlScript(Database.java:342)
... 3 more
Caused by: org.postgresql.util.PSQLException: ERROR: COPY from stdin failed: COPY commands are only supported using the CopyManager API.
Where: COPY achilles_cache, line 1
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:329)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:315)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:291)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:286)
at org.apache.ibatis.jdbc.ScriptRunner.executeStatement(ScriptRunner.java:190)
at org.apache.ibatis.jdbc.ScriptRunner.handleLine(ScriptRunner.java:165)
at org.apache.ibatis.jdbc.ScriptRunner.executeLineByLine(ScriptRunner.java:102)
... 5 more
--- EDIT ------------------------------------
The --inserts method in the accepted answer gave me exactly what I needed.
I ended up doing this:
pg_dump --inserts --dbname=postgresql://postgres:ohdsi#127.0.0.1:5432/OHDSI -t webapi.* > webapi.sql
The client tool you are using to restore the dump cannot deal with the data from the (nonstandard) COPY command being mixed into the script. You need psql to restore such a dump.
You can use the --inserts option of pg_dump to create a dump that contains INSERT statements rather than COPY. That will be slower to restore, but will work with more client tools.
However, your wish to get a standard SQL script is hopeless. PostgreSQL extends the standard in many ways, so a database cannot be dumped with a standard SQL script. Note, for example, that indexes are not defined by the SQL standard. If you are looking to transfer a PostgreSQL dump to a different RDBMS, you will be disappointed. That is more difficult.
I've used pg_dump --no-privileges --format custom --compress=0 some_database > my-dump.pgdump to dump a database, but I'm running into issues when I try to restore it.
Specifically, it appears to be loading function definitions before table definitions:
$ pg_restore ./my-dump.pgdump
…
create function my_function() returns …
language sql $$
select …
from some_table
where …
$$;
… later in the dump …
create table some_table ( … );
…
Which causes an error when I try to restore the dump:
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 4863; 0 16735 TABLE DATA some_table some_database
pg_restore: [archiver (db)] COPY failed for table "some_table": ERROR: relation "some_table" does not exist
LINE 3: from some_table
^
QUERY:
select …
from some_table
where …
CONTEXT: SQL function "my_function" during inlining
What's going on here? How can I trick pg_dump / pg_restore into doing things in the correct order?
Check your dump file for commands which mess with search_path, for example:
SELECT pg_catalog.set_config('search_path', '', false);
I encountered the same kind error as you (relation xxx does not exist ... during inlining) in a legacy project I inherited, even though it's running PostgreSQL 9.4.x.
I traced it to the above command.
The solution for me was to remove this command from the dump file.
After I did this I was able to restore the database without errors.
Note: the OP is using the custom format. There is no editing the binary file that is emitted.
In my experience, pg_dump using the custom format (-Fc) doesn't set check_function_bodies = false. But since it adds random functions at the top of the dump file (instead of putting all routines at the end), this causes pg_restore to barf.
I was able to workaround this issue by setting PGOPTIONS:
export PGOPTIONS="-c check_function_bodies=false"
pg_restore ...
That is strange. Ever since commit ef88199f611e625b496ff92aa17a447d254b9796 in 2003, pg_dump and pg_restore have emitted
SET check_function_bodies = false;
This setting makes sure that an error like you describe won't happen, because PostgreSQL won't check the validity of the function bodies.
Are you using an ancient PostgreSQL version or are you doing anything else that could mess with that?
If you run pg_restore on your dump (without specifying a destination database), does it emit the line?
I am attempting to install the AWS "Approved" PostgreSql Extension on our on large RDS instance but every time I at the point I attempt to 'create extension postgis_tiger_geocoder' I get this:
SQL Error [42883]: ERROR: function soundex(character varying) does not exist
I have spent a good bit of time reading the AWS / postgis / postgresql forums but unfortunately haven't found the writing on the wall.
Steps Taken
Installed the POSTGIS extension
create EXTENSION postgis;
Installed the FuzzyStrMatch Extension which contains the soundex function (verified)
create EXTENSION fuzzystrmatch;
Finally when I run this create extension I get the error above
create extension postgis_tiger_geocoder;
SQL Error [42883]: ERROR: function soundex(character varying) does not exist
Hint: No function matches the given name and argument types. You might need to add explicit type casts.
Position: 57558
org.postgresql.util.PSQLException: ERROR: function soundex(character varying) does not exist
Hint: No function matches the given name and argument types. You might need to add explicit type casts.
Position: 57558
Things I have tried:
set search_path = <schema_name>, public
Followed here:
Installing PostgreSQL Extension to all schemas
Dug deeply into postgis installation documentation
Read through RDS documentation on adding Extensions...
If anyone has had to deal with this frustration on AWS I will happily swap a few of the remaining hairs left on my head as I have not been able to work around this.
Results of \dx+
Objects in extension "fuzzystrmatch"
Object Description
--------------------------------------------------------------------------------
function <schema>.difference(...)
function <schema>.dmetaphone_alt(...)
function <schema>.dmetaphone(...)
function <schema>.levenshtein_less_equal(...)
function <schema>.levenshtein_less_equal(...)
function <schema>.levenshtein(...)
function <schema>.levenshtein(...)
function <schema>.metaphone(...)
function <schema>.soundex(...)
function <schema>.text_soundex(...)
(10 rows)
Results of \dfS+ soundex
List of functions
Schema | Name | Result data type | Argument data types | Type | Volatility | Owner | Security | Access privileges | Language | Source code | Description
--------+------+------------------+---------------------+------+------------+-------+----------+-------------------+----------+-------------+-------------
(0 rows)
Had the same problem, resolved it by altering search_path for database and reconnect before creating extension postgis_tiger_geocoder. Look for the FIX part :
-- Postgis Installation
------------------------------------------------------------------------------------------------------------------------------------------------
-- PostGIS AWS Configuration --
-- https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.PostGIS --
------------------------------------------------------------------------------------------------------------------------------------------------
-- On postgis schema
SET SCHEMA '${POSTGIS_SCHEMA_NAME}';
-- Step 2: Load the PostGIS Extensions
create extension postgis;
create extension fuzzystrmatch;
-- FIX : To avoid "ERROR: function soundex(character varying) does not exist", change schema and reconnect
ALTER DATABASE ${DATABASE_NAME} SET search_path=${POSTGIS_SCHEMA_NAME};
\connect ${DATABASE_NAME};
-- End FIX
create extension postgis_tiger_geocoder;
create extension postgis_topology;
-- Step 3: Transfer Ownership of the Extensions to the rds_superuser Role
alter schema tiger owner to ${MASTER_USER};
alter schema tiger_data owner to ${MASTER_USER};
alter schema topology owner to ${MASTER_USER};
-- Step 4: Transfer Ownership of the Objects to the rds_superuser Role
CREATE FUNCTION exec(text) returns text language plpgsql volatile AS $f$ BEGIN EXECUTE $1; RETURN $1; END; $f$;
SELECT exec('ALTER TABLE ' || quote_ident(s.nspname) || '.' || quote_ident(s.relname) || ' OWNER TO ${MASTER_USER};')
FROM (
SELECT nspname, relname
FROM pg_class c JOIN pg_namespace n ON (c.relnamespace = n.oid)
WHERE nspname in ('tiger','topology') AND
relkind IN ('r','S','v') ORDER BY relkind = 'S')
s;
-- Adding postgis to default schema
ALTER DATABASE ${DATABASE_NAME} SET search_path=${SCHEMA_NAME},${POSTGIS_SCHEMA_NAME};
Had the same problem. Turns out that all the fuzzystrmatch's functions were created inside the wrong schema.
Connected with psql command line, I used the drop extension command to restart the process of creating the extensions:
drop extension postgis_topology;
drop extension postgis;
drop extension fuzzystrmatch;
Then, just to be sure, disconnected using \q.
Connected psql again.
Set the schema to public:
set schema 'public';
Then, follow the process described in AWS RDS Docs
create extension postgis;
create extension fuzzystrmatch;
create extension postgis_tiger_geocoder;
create extension postgis_topology;
The following works in PostgreSQL 8.4:
insert into credentials values('demo', pgp_sym_encrypt('password', 'longpassword'));
When I try it in version 9.1 I get this:
ERROR: function pgp_sym_encrypt(unknown, unknown) does not exist LINE
1: insert into credentials values('demo', pgp_sym_encrypt('pass...
^ HINT: No function matches the given name and argument types. You might need to add
explicit type casts.
*** Error ***
ERROR: function pgp_sym_encrypt(unknown, unknown) does not exist SQL
state: 42883 Hint: No function matches the given name and argument
types. You might need to add explicit type casts. Character: 40
If I try some explicit casts like this
insert into credentials values('demo', pgp_sym_encrypt(cast('password' as text), cast('longpassword' as text)))
I get a slightly different error message:
ERROR: function pgp_sym_encrypt(text, text) does not exist
I have pgcrypto installed. Does anyone have pgp_sym_encrypt() working in PostgreSQL 9.1?
On explanation could be that the module was installed into a schema that is not in your search path - or to the wrong database.
Diagnose your problem with this query and report back the output:
SELECT n.nspname, p.proname, pg_catalog.pg_get_function_arguments(p.oid) as params
FROM pg_catalog.pg_proc p
JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace
WHERE p.proname ~~* '%pgp_sym_encrypt%'
AND pg_catalog.pg_function_is_visible(p.oid);
Finds functions in all schemas in your database. Similar to the psql meta-command
\df *pgp_sym_encrypt*
Make sure you install the extension on the desired schema.
sudo -i -u postgres
psql $database
CREATE EXTENSION pgcrypto;
OK, problem solved.
I was creating the pgcrypto extension as the first operation in the script. Then I dropped and added the VGDB database. That's why pgcrypto was there immediately after creating it, but didn't exist when running the sql later in the script or when I opened pgadmin.
This script is meant for setting up new databases and if I had tried it on a new database the create extension would have failed right away.
My bad. Thanks for the help, Erwin.
Just mention de schema where is installed pgcrypto like this:
#ColumnTransformer(forColumn = "TEST",
read = "public.pgp_sym_decrypt(TEST, 'password')",
write = "public.pgp_sym_encrypt(?, 'password')")
#Column(name = "TEST", columnDefinition = "bytea", nullable = false)
private String test;
I ran my (python) script again and the CREATE EXTENSION ran without error. The script also executes this command
psql -d VGDB -U postgres -c "select * from pg_available_extensions order by name"
which includes the following in the result set:
pgcrypto | 1.0 | 1.0 | cryptographic functions
So psql believes that it has installed pgcrypto.
Later in the same script when I execute
psql -d VGDB -U postgres -f sql/Create.Credentials.table.sql
where sql/Create.Credentials.table.sql includes this
insert into credentials values('demo', pgp_sym_encrypt('password', 'longpassword'));
I get this
psql:sql/Create.Credentials.table.sql:31: ERROR: function pgp_sym_encrypt(unknown, unknown) does not exist
LINE 1: insert into credentials values('demo', pgp_sym_encrypt('pass...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
When I open pgadmin it does not show pgcrypto in either the VGDB or postgres databases even though the query above called by psql shows that pgcrypto is installed.
Could there be an issue with needing to commit after using psql to execute the "create extension ..." command? None of my other DDL or SQL statements require a commit when they get executed with psql.
It's starting to look like psql is just flakey. Is there another way to call "create extension pgcrypto" - e.g. with Python's database support classes - or does that have to be run through psql?