Related
I've been stuck with this problem for more than a week now, and I hope someone could help me out here.
I was running Postgresql 12 with PostGIS 3.1.1 extension. More than a week ago, I accidentally performed a PostGIS upgrade by running this query
SELECT PostGIS_Extensions_Upgrade(); which returned "Upgrade completed, run SELECT postgis_full_version(); for details".
Although it seemed the upgrade went on successfully, while I check with SELECT postgis_full_version(); it returned POSTGIS="3.2.0 c3e3cc0" [EXTENSION] PGSQL="120" GEOS="3.9.1-CAPI-1.14.2" PROJ="6.3.1" GDAL="GDAL 3.0.4, released 2020/01/28 GDAL_DATA not found" LIBXML="2.9.10" LIBJSON="0.13.1" LIBPROTOBUF="1.3.3" WAGYU="0.5.0 (Internal)" TOPOLOGY RASTER. Note that GDAL_DATA is no longer found.
This causes functions like ST_Transform breaks with error as following: SQL Error [XX000]: ERROR: could not form projection (LWPROJ) from 'srid=4326' to 'srid=4269'.
So far I have tried reinstalled postgresql-12, postgresql-client-12, postgis, postgresql-12-postgis-3 and postgresql-12-postgis-3-scripts on my Ubuntu 20.04 system, to no avail.
I tried downgrading to PostGIS to 3.1.4 (somehow apt repos only have 3.1.4 not 3.1.1), which returned POSTGIS="3.1.4 ded6c34" [EXTENSION] PGSQL="120" GEOS="3.9.1-CAPI-1.14.2" PROJ="7.2.1" GDAL="GDAL 3.3.2, released 2021/09/01" LIBXML="2.9.10" LIBJSON="0.13.1" LIBPROTOBUF="1.3.3" WAGYU="0.5.0 (Internal)" (core procs from "3.2.0 c3e3cc0" need upgrade) TOPOLOGY (topology procs from "3.2.0 c3e3cc0" need upgrade) RASTER (raster procs from "3.2.0 c3e3cc0" need upgrade) when querying SELECT postgis_full_version();. There didn't seem any error but the ST_Transform query still gave the same error above.
I am also tried to use the pg_upgrade function to upgrade the database to Postgresql 14. While running with the --check option, the results says "Clusters are compatible". But when running the actual upgrade, it breaks at the "Creating dump of database schemas" of the postgres database. Dumps of other databases went through just fine. The "pg_upgrade_dump_13427.log" log shows the following.
pg_dump: error: query was: SELECT classid, objid, refobjid FROM pg_depend WHERE refclassid = 'pg_extension'::regclass AND deptype = 'e' ORDER BY 3
command: "/usr/lib/postgresql/14/bin/pg_dump" --host /var/lib/postgresql --port 50432 --username postgres --schema-only --quote-all-identifiers --binary-upgrade --format=custom --file="pg_upgrade_dump_13427.custom" 'dbname=postgres' >> "pg_upgrade_dump_13427.log" 2>&1
pg_dump: WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
pg_dump: error: query failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
pg_dump: error: query was: SELECT c.tableoid, c.oid, c.relname, (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(c.relacl,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) WITH ORDINALITY AS perm(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) AS init(init_acl) WHERE acl = init_acl)) as foo) AS relacl, (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(c.relacl,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) AS permp(orig_acl) WHERE acl = orig_acl)) as foo) as rrelacl, CASE WHEN privtype = 'e' THEN (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(pip.initprivs) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner)) AS privm(orig_acl) WHERE acl = orig_acl)) as foo) END AS initrelacl, CASE WHEN privtype = 'e' THEN (SELECT pg_catalog.array_agg(acl) FROM (SELECT acl, row_n FROM pg_catalog.unnest(pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner)) WITH ORDINALITY AS privp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(pip.initprivs) AS initp(init_acl) WHERE acl = init_acl)) as foo) END as initrrelacl, c.relkind, c.relnamespace, (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = c.relowner) AS rolname, c.relchecks, c.relhastriggers, c.relhasindex, c.relhasrules, 'f'::bool AS relhasoids, c.relrowsecurity, c.relforcerowsecurity, c.relfrozenxid, c.relminmxid, tc.oid AS toid, tc.relfrozenxid AS tfrozenxid, tc.relminmxid AS tminmxid, c.relpersistence, c.relispopulated, c.relreplident, c.relpages, am.amname, CASE WHEN c.relkind = 'f' THEN (SELECT ftserver FROM pg_catalog.pg_foreign_table WHERE ftrelid = c.oid) ELSE 0 END AS foreignserver, CASE WHEN c.reloftype <> 0 THEN c.reloftype::pg_catalog.regtype ELSE NULL END AS reloftype, d.refobjid AS owning_tab, d.refobjsubid AS owning_col, (SELECT spcname FROM pg_tablespace t WHERE t.oid = c.reltablespace) AS reltablespace, array_remove(array_remove(c.reloptions,'check_option=local'),'check_option=cascaded') AS reloptions, CASE WHEN 'check_option=local' = ANY (c.reloptions) THEN 'LOCAL'::text WHEN 'check_option=cascaded' = ANY (c.reloptions) THEN 'CASCADED'::text ELSE NULL END AS checkoption, tc.reloptions AS toast_reloptions, c.relkind = 'S' AND EXISTS (SELECT 1 FROM pg_depend WHERE classid = 'pg_class'::regclass AND objid = c.oid AND objsubid = 0 AND refclassid = 'pg_class'::regclass AND deptype = 'i') AS is_identity_sequence, EXISTS (SELECT 1 FROM pg_attribute at LEFT JOIN pg_init_privs pip ON (c.oid = pip.objoid AND pip.classoid = 'pg_class'::regclass AND pip.objsubid = at.attnum)WHERE at.attrelid = c.oid AND ((SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(at.attacl,pg_catalog.acldefault('c',c.relowner))) WITH ORDINALITY AS perm(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault('c',c.relowner))) AS init(init_acl) WHERE acl = init_acl)) as foo) IS NOT NULL OR (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault('c',c.relowner))) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(at.attacl,pg_catalog.acldefault('c',c.relowner))) AS permp(orig_acl) WHERE acl = orig_acl)) as foo) IS NOT NULL OR CASE WHEN privtype = 'e' THEN (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(pip.initprivs) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(pg_catalog.acldefault('c',c.relowner)) AS privm(orig_acl) WHERE acl = orig_acl)) as foo) END IS NOT NULL OR CASE WHEN privtype = 'e' THEN (SELECT pg_catalog.array_agg(acl) FROM (SELECT acl, row_n FROM pg_catalog.unnest(pg_catalog.acldefault('c',c.relowner)) WITH ORDINALITY AS privp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(pip.initprivs) AS initp(init_acl) WHERE acl = init_acl)) as foo) END IS NOT NULL))AS changed_acl, pg_get_partkeydef(c.oid) AS partkeydef, c.relispartition AS ispartition, pg_get_expr(c.relpartbound, c.oid) AS partbound FROM pg_class c LEFT JOIN pg_depend d ON (c.relkind = 'S' AND d.classid = c.tableoid AND d.objid = c.oid AND d.objsubid = 0 AND d.refclassid = c.tableoid AND d.deptype IN ('a', 'i')) LEFT JOIN pg_class tc ON (c.reltoastrelid = tc.oid AND c.relkind <> 'p') LEFT JOIN pg_am am ON (c.relam = am.oid) LEFT JOIN pg_init_privs pip ON (c.oid = pip.objoid AND pip.classoid = 'pg_class'::regclass AND pip.objsubid = 0) WHERE c.relkind in ('r', 'S', 'v', 'c', 'm', 'f', 'p') ORDER BY c.oid
command: "/usr/lib/postgresql/13/bin/pg_dump" --host /var/lib/postgresql --port 50432 --username postgres --schema-only --quote-all-identifiers --binary-upgrade --format=custom --file="pg_upgrade_dump_13427.custom" 'dbname=postgres' >> "pg_upgrade_dump_13427.log" 2>&1
pg_dump: WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
pg_dump: error: query failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
pg_dump: error: query was: SELECT classid, objid, refobjid FROM pg_depend WHERE refclassid = 'pg_extension'::regclass AND deptype = 'e' ORDER BY 3
It seems the dependencies of postgres extensions are broken. How do I fix that? Can someone please help?
I have been using using DBeaver and I got a new laptop. With a fresh install adding the postgres remote database exactly as it is in the other computer.
I can see all the tables and data in the database. I just cannot take a backup.
When I right click tools->backup and I click start (same process works on old laptop) I am getting the error below. If I do this with my local dev database it works fine as expected. I can do a backup on my laptop both directly before and directly after trying on the new laptop.
C:\Program Files\PostgreSQL\10\bin\pg_dump.exe --format=c -n public --verbose --host=compute-1.amazonaws.com --port=5432 --username=elkjlkjkjyb d5kjhkjhjkht5
Backup started at Sun Sep 02 22:10:20 EDT 2018
pg_dump: last built-in OID is 16383
pg_dump: reading extensions
pg_dump: identifying extension members
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: [archiver (db)] query failed: SSL error: sslv3 alert bad record mac
pg_dump: [archiver (db)] query was: SELECT c.tableoid, c.oid, c.relname, (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(c.relacl,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) WITH ORDINALITY AS perm(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) AS init(init_acl) WHERE acl = init_acl)) as foo) AS relacl, (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(c.relacl,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) AS permp(orig_acl) WHERE acl = orig_acl)) as foo) as rrelacl, NULL AS initrelacl, NULL as initrrelacl, c.relkind, c.relnamespace, (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = c.relowner) AS rolname, c.relchecks, c.relhastriggers, c.relhasindex, c.relhasrules, c.relhasoids, c.relrowsecurity, c.relforcerowsecurity, c.relfrozenxid, c.relminmxid, tc.oid AS toid, tc.relfrozenxid AS tfrozenxid, tc.relminmxid AS tminmxid, c.relpersistence, c.relispopulated, c.relreplident, c.relpages, CASE WHEN c.reloftype <> 0 THEN c.reloftype::pg_catalog.regtype ELSE NULL END AS reloftype, d.refobjid AS owning_tab, d.refobjsubid AS owning_col, (SELECT spcname FROM pg_tablespace t WHERE t.oid = c.reltablespace) AS reltablespace, array_remove(array_remove(c.reloptions,'check_option=local'),'check_option=cascaded') AS reloptions, CASE WHEN 'check_option=local' = ANY (c.reloptions) THEN 'LOCAL'::text WHEN 'check_option=cascaded' = ANY (c.reloptions) THEN 'CASCADED'::text ELSE NULL END AS checkoption, tc.reloptions AS toast_reloptions, c.relkind = 'S' AND EXISTS (SELECT 1 FROM pg_depend WHERE classid = 'pg_class'::regclass AND objid = c.oid AND objsubid = 0 AND refclassid = 'pg_class'::regclass AND deptype = 'i') AS is_identity_sequence, EXISTS (SELECT 1 FROM pg_attribute at LEFT JOIN pg_init_privs pip ON (c.oid = pip.objoid AND pip.classoid = 'pg_class'::regclass AND pip.objsubid = at.attnum)WHERE at.attrelid = c.oid AND ((SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(at.attacl,pg_catalog.acldefault('c',c.relowner))) WITH ORDINALITY AS perm(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault('c',c.relowner))) AS init(init_acl) WHERE acl = init_acl)) as foo) IS NOT NULL OR (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault('c',c.relowner))) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(at.attacl,pg_catalog.acldefault('c',c.relowner))) AS permp(orig_acl) WHERE acl = orig_acl)) as foo) IS NOT NULL OR NULL IS NOT NULL OR NULL IS NOT NULL))AS changed_acl, pg_get_partkeydef(c.oid) AS partkeydef, c.relispartition AS ispartition, pg_get_expr(c.relpartbound, c.oid) AS partbound FROM pg_class c LEFT JOIN pg_depend d ON (c.relkind = 'S' AND d.classid = c.tableoid AND d.objid = c.oid AND d.objsubid = 0 AND d.refclassid = c.tableoid AND d.deptype IN ('a', 'i')) LEFT JOIN pg_class tc ON (c.reltoastrelid = tc.oid) LEFT JOIN pg_init_privs pip ON (c.oid = pip.objoid AND pip.classoid = 'pg_class'::regclass AND pip.objsubid = 0) WHERE c.relkind in ('r', 'S', 'v', 'c', 'm', 'f', 'p') ORDER BY c.oid
Backup finished at Sun Sep 02 22:10:21 EDT 2018
Process exit code: 1
UPDATE
I believe this has to do with a large query problem instead of a ssl problem. I did an insert with 50 rows and no problem. As soon as i go over a certain number of characters it seems to break connecting to the server. Also I believe the pg_dump is a problem because the query it tries to do is a very large query. This is a local problem because it doesnt happen on my other laptop.
Probably some max setting in Postgres 10.5-2 or JDBC. My old laptop has the same JDBC version as my new one. Old laptop has Postgres 10.4-1 though.
So this turns out not to be related directly with Postgres.
I have a MSI GS65. It has a hardware network driver called killer network.
The driver has settings you can modify. One of those setting is to optimize network speeds. It seems that this setting will drop larger connections. When I am doing a backup or a large query through dbeaver, it drops the connection and stops the backup/query.
Took me 2 years to find this issue. I hope this helps any devs with a MSI computer with network issues.
I am trying to audit all of the permissions for an application before release and I want to ensure no role has more access than it needs. I have looked at the different functions and system tables, but everything is very piecemeal.
Is there a good query or method to be able to dump out every grant a particular role has?
I am using pg 9.5.
The column relacl of the system catalog pg_class contains all informations on privileges.
Example data in schema public owned by postgres with grants to newuser:
create table test(id int);
create view test_view as select * from test;
grant select, insert, update on test to newuser;
grant select on test_view to newuser;
Querying the pg_class:
select
relname,
relkind,
coalesce(nullif(s[1], ''), 'public') as grantee,
s[2] as privileges
from
pg_class c
join pg_namespace n on n.oid = relnamespace
join pg_roles r on r.oid = relowner,
unnest(coalesce(relacl::text[], format('{%s=arwdDxt/%s}', rolname, rolname)::text[])) acl,
regexp_split_to_array(acl, '=|/') s
where nspname = 'public'
and relname like 'test%';
relname | relkind | grantee | privileges
-----------+---------+----------+------------
test | r | postgres | arwdDxt <- owner postgres has all privileges on the table
test | r | newuser | arw <- newuser has append/read/write privileges
test_view | v | postgres | arwdDxt <- owner postgres has all privileges on the view
test_view | v | newuser | r <- newuser has read privilege
(4 rows)
Comments:
coalesce(relacl::text[], format('{%s=arwdDxt/%s}', rolname, rolname)) - Null in relacl means that the owner has all privileges;
unnest(...) acl - relacl is an array of aclitem, one array element for a user;
regexp_split_to_array(acl, '=|/') s - split aclitem into: s[1] username, s[2] privileges;
coalesce(nullif(s[1], ''), 'public') as grantee - empty username means public.
Modify the query to select individual user or specific kind of relation or another schemas, etc...
Read in the documentation:
The catalog pg_class,
GRANT with the description of acl system.
In a similar way you can get information about privileges granted on schemas (the column nspacl in pg_namespace) and databases (datacl in pg_database)
The relacl column (and others of type aclitem) doesn't have to be parsed as text.
The function aclexplode unnests array, which makes it suitable for lateral join. Result is record with well named fields, just convert oid to human-readable name:
select c.*, n.nspname,
acl.grantor, acl.grantee,
pg_catalog.pg_get_userbyid(acl.grantor), pg_catalog.pg_get_userbyid(acl.grantee),
acl.privilege_type, acl.is_grantable
from pg_catalog.pg_class c
join pg_catalog.pg_namespace n on n.oid = c.relnamespace,
lateral aclexplode(c.relacl) acl;
I need to determine what privileges are currently granted for some materialized views in my database.
The query to do this for a table or standard view is pretty straight forward:
SELECT grantee, string_agg(privilege_type, ', ') AS privileges
FROM information_schema.table_privileges
WHERE table_schema = 'some_schema' AND table_name = 'some_table'
GROUP by grantee;
That said, there doesn't seem to be an analogous table for materialized views. Where does PostgreSQL store this information?
In Postgres system catalogs are the basic set of complete information about the installation and databases. System catalogs are the most reliable source of information.
Information schema as an auxiliary feature is based on system catalogs and is provided for compatibility with other RDBMs:
The information schema is defined in the SQL standard and can therefore be expected to be portable and remain stable — unlike the system catalogs, which are specific to PostgreSQL and are modeled after implementation concerns. The information schema views do not, however, contain information about PostgreSQL-specific features; to inquire about those you need to query the system catalogs or other PostgreSQL-specific views.
Materialized views are not SQL-standard objects hence the information schema does not contain information about them.
The system catalog pg_class contains all informations on privileges in the column relacl.
If the column is null then the owner has all privileges.
An empty string as a user name in acl string means public.
create materialized view test_view as select 1;
grant select on test_view to public;
grant delete on test_view to a_user;
select
coalesce(nullif(s[1], ''), 'public') as grantee,
s[2] as privileges
from
pg_class c
join pg_namespace n on n.oid = relnamespace
join pg_roles r on r.oid = relowner,
unnest(coalesce(relacl::text[], format('{%s=arwdDxt/%s}', rolname, rolname)::text[])) acl,
regexp_split_to_array(acl, '=|/') s
where nspname = 'public' and relname = 'test_view';
grantee | privileges
----------+------------
postgres | arwdDxt
public | r
a_user | d
(3 rows)
You need a function to show privileges in readable format:
create or replace function priviliges_from_acl(text)
returns text language sql as $$
select string_agg(privilege, ', ')
from (
select
case ch
when 'r' then 'SELECT'
when 'w' then 'UPDATE'
when 'a' then 'INSERT'
when 'd' then 'DELETE'
when 'D' then 'TRUNCATE'
when 'x' then 'REFERENCES'
when 't' then 'TRIGGER'
end privilege
from
regexp_split_to_table($1, '') ch
) s
$$;
Use:
select
coalesce(nullif(s[1], ''), 'public') as grantee,
priviliges_from_acl(s[2]) as privileges
from
pg_class c
join pg_namespace n on n.oid = relnamespace
join pg_roles r on r.oid = relowner,
unnest(coalesce(relacl::text[], format('{%s=arwdDxt/%s}', rolname, rolname)::text[])) acl,
regexp_split_to_array(acl, '=|/') s
where nspname = 'public' and relname = 'test_view';
grantee | privileges
----------+---------------------------------------------------------------
postgres | INSERT, SELECT, UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER
public | SELECT
a_user | DELETE
(3 rows)
Following klin's helpful answer, I've come up with a view that lists a summary of all privileges for all relations that appear in pg_class (tables, views, m. views, indexes, sequences, foreign tables, composite types) for all roles:
CREATE VIEW show_privileges AS (
SELECT
grantee,
string_agg(relname, ', ' ORDER BY relname) AS rel_names,
privileges
FROM (
SELECT
relname,
coalesce(nullif(s[1], ''), 'public') grantee,
(SELECT string_agg(privilege, ', ' ORDER BY privilege ASC)
FROM (SELECT
CASE ch
WHEN 'r' THEN 'SELECT'
WHEN 'w' THEN 'UPDATE'
WHEN 'a' THEN 'INSERT'
WHEN 'd' THEN 'DELETE'
WHEN 'D' THEN 'TRUNCATE'
WHEN 'x' THEN 'REFERENCES'
WHEN 't' THEN 'TRIGGER'
END AS privilege
FROM regexp_split_to_table(s[2], '') ch
) s
) AS privileges
FROM
pg_class
JOIN pg_namespace ON pg_namespace.oid = relnamespace
JOIN pg_roles ON pg_roles.oid = relowner,
unnest(coalesce(relacl::text[], format('{%s=arwdDxt/%s}', rolname, rolname)::text[])) AS acl,
regexp_split_to_array(acl, '=|/') AS s
WHERE nspname = 'public'
) AS t
GROUP BY grantee, privileges
ORDER BY grantee, privileges, rel_names
);
Hello I am trying to retrieve the schema of an existing table. I am mysql developer and am trying to work with amazon redshift. How can I export the schema of an existing table. In mysql we can use the show create table command.
SHOW CREATE TABLE tblName;
Recently I wrote a python script to clone table schemas between redshift clusters. If you only want the columns and column types of a table, you can do it via:
select column_name,
case
when data_type = 'integer' then 'integer'
when data_type = 'bigint' then 'bigint'
when data_type = 'smallint' then 'smallint'
when data_type = 'text' then 'text'
when data_type = 'date' then 'date'
when data_type = 'real' then 'real'
when data_type = 'boolean' then 'boolean'
when data_type = 'double precision' then 'float8'
when data_type = 'timestamp without time zone' then 'timestamp'
when data_type = 'character' then 'char('||character_maximum_length||')'
when data_type = 'character varying' then 'varchar('||character_maximum_length||')'
when data_type = 'numeric' then 'numeric('||numeric_precision||','||numeric_scale||')'
else 'unknown'
end as data_type,
is_nullable,
column_default
from information_schema.columns
where table_schema = 'xxx' and table_name = 'xxx' order by ordinal_position
;
But if you need the compression types and distkey/sortkeys, you need to query another table:
select * from pg_table_def where tablename = 'xxx' and schemaname='xxx';
This query will give you the complete schema definition including the Redshift specific attributes distribution type/key, sort key, primary key, and column encodings in the form of a create statement as well as providing an alter table statement that sets the owner to the current owner. The only thing it can't tell you are foreign keys. I'm working on the latter, but there's a current privilege issue in RS that prevents us from querying the right tables. This query could use some tuning, but I haven't had time or the need to work it further.
select pk.pkey, tm.schemaname||'.'||tm.tablename, 'create table '||tm.schemaname||'.'||tm.tablename
||' ('
||cp.coldef
-- primary key
||decode(pk.pkey,null,'',pk.pkey)
-- diststyle and dist key
||decode(d.distkey,null,') diststyle '||dist_style||' ',d.distkey)
--sort key
|| (select decode(skey,null,'',skey) from (select
' sortkey(' ||substr(array_to_string(
array( select ','||cast(column_name as varchar(100)) as str from
(select column_name from information_schema.columns col where col.table_schema= tm.schemaname and col.table_name=tm.tablename) c2
join
(-- gives sort cols
select attrelid as tableid, attname as colname, attsortkeyord as sort_col_order from pg_attribute pa where
pa.attnum > 0 AND NOT pa.attisdropped AND pa.attsortkeyord > 0
) st on tm.tableid=st.tableid and c2.column_name=st.colname order by sort_col_order
)
,'')
,2,10000) || ')' as skey
))
||';'
-- additional alter table queries here to set owner
|| 'alter table '||tm.schemaname||'.'||tm.tablename||' owner to "'||tm.owner||'";'
from
-- t master table list
(
SELECT substring(n.nspname,1,100) as schemaname, substring(c.relname,1,100) as tablename, c.oid as tableid ,use2.usename as owner, decode(c.reldiststyle,0,'EVEN',1,'KEY',8,'ALL') as dist_style
FROM pg_namespace n, pg_class c, pg_user use2
WHERE n.oid = c.relnamespace
AND nspname NOT IN ('pg_catalog', 'pg_toast', 'information_schema')
AND c.relname <> 'temp_staging_tables_1'
and c.relowner = use2.usesysid
) tm
-- cp creates the col params for the create string
join
(select
substr(str,(charindex('QQQ',str)+3),(charindex('ZZZ',str))-(charindex('QQQ',str)+3)) as tableid
,substr(replace(replace(str,'ZZZ',''),'QQQ'||substr(str,(charindex('QQQ',str)+3),(charindex('ZZZ',str))-(charindex('QQQ',str)+3)),''),2,10000) as coldef
from
( select array_to_string(array(
SELECT 'QQQ'||cast(t.tableid as varchar(10))||'ZZZ'|| ','||column_name||' '|| decode(udt_name,'bpchar','char',udt_name) || decode(character_maximum_length,null,'', '('||cast(character_maximum_length as varchar(9))||')' )
-- default
|| decode(substr(column_default,2,8),'identity','',null,'',' default '||column_default||' ')
-- nullable
|| decode(is_nullable,'YES',' NULL ','NO',' NOT NULL ')
-- identity
|| decode(substr(column_default,2,8),'identity',' identity('||substr(column_default,(charindex('''',column_default)+1), (length(column_default)-charindex('''',reverse(column_default))-charindex('''',column_default) ) ) ||') ', '')
-- encoding
|| decode(enc,'none','',' encode '||enc)
as str
from
-- ci all the col info
(
select cast(t.tableid as int), cast(table_schema as varchar(100)), cast(table_name as varchar(100)), cast(column_name as varchar(100)),
cast(ordinal_position as int), cast(column_default as varchar(100)), cast(is_nullable as varchar(20)) , cast(udt_name as varchar(50)) ,cast(character_maximum_length as int),
sort_col_order , decode(d.colname,null,0,1) dist_key , e.enc
from
(select * from information_schema.columns c where c.table_schema= t.schemaname and c.table_name=t.tablename) c
left join
(-- gives sort cols
select attrelid as tableid, attname as colname, attsortkeyord as sort_col_order from pg_attribute a where
a.attnum > 0 AND NOT a.attisdropped AND a.attsortkeyord > 0
) s on t.tableid=s.tableid and c.column_name=s.colname
left join
(-- gives encoding
select attrelid as tableid, attname as colname, format_encoding(a.attencodingtype::integer) AS enc from pg_attribute a where
a.attnum > 0 AND NOT a.attisdropped
) e on t.tableid=e.tableid and c.column_name=e.colname
left join
-- gives dist col
(select attrelid as tableid, attname as colname from pg_attribute a where
a.attnum > 0 AND NOT a.attisdropped AND a.attisdistkey = 't'
) d on t.tableid=d.tableid and c.column_name=d.colname
order by ordinal_position
) ci
-- for the working array funct
), '') as str
from
(-- need tableid
SELECT substring(n.nspname,1,100) as schemaname, substring(c.relname,1,100) as tablename, c.oid as tableid
FROM pg_namespace n, pg_class c
WHERE n.oid = c.relnamespace
AND nspname NOT IN ('pg_catalog', 'pg_toast', 'information_schema')
) t
)) cp on tm.tableid=cp.tableid
-- primary key query here
left join
(select c.oid as tableid, ', primary key '|| substring(pg_get_indexdef(indexrelid),charindex('(',pg_get_indexdef(indexrelid))-1 ,60) as pkey
from pg_index i , pg_namespace n, pg_class c
where i.indisprimary=true
and i.indrelid =c.oid
and n.oid = c.relnamespace
) pk on tm.tableid=pk.tableid
-- dist key
left join
( select
-- close off the col defs after the primary key
')' ||
' distkey('|| cast(column_name as varchar(100)) ||')' as distkey, t.tableid
from information_schema.columns c
join
(-- need tableid
SELECT substring(n.nspname,1,100) as schemaname, substring(c.relname,1,100) as tablename, c.oid as tableid
FROM pg_namespace n, pg_class c
WHERE n.oid = c.relnamespace
AND nspname NOT IN ('pg_catalog', 'pg_toast', 'information_schema')
) t on c.table_schema= t.schemaname and c.table_name=t.tablename
join
-- gives dist col
(select attrelid as tableid, attname as colname from pg_attribute a where
a.attnum > 0 AND NOT a.attisdropped AND a.attisdistkey = 't'
) d on t.tableid=d.tableid and c.column_name=d.colname
) d on tm.tableid=d.tableid
where tm.schemaname||'.'||tm.tablename='myschema.mytable'
If you want to get the table structure with create statement, constraints and triggers, you can use pg_dump utility
pg_dump -U user_name -s -t table_name -d db_name
Note: -s used for schema only dump
if you want to take the data only dump , you can use -a switch.
This will output the create syntax with all the constraints. Hope this will help you.
I did not find any complete solutions out there.
And wrote a python script:
https://github.com/cxmcc/redshift_show_create_table
It will work like pg_dump, plus dealing with basic redshift features, SORTKEY/DISTKEY/DISTSTYLES etc.
As show table doesn't work on Redshift:
show table <YOUR_TABLE>;
ERROR: syntax error at or near "<YOUR_TABLE>"
We can use pg_table_def table to get the schema out:
select "column", type, encoding, distkey, sortkey, "notnull"
from pg_table_def
where tablename = '<YOUR_TABLE>';
NOTE: If the schema is not on the search path, add it to search path using:
set search_path to '$user', 'public', '<YOUR_SCHEMA>';
For redshift please try
show table <**tablename**> ;
In Postgres, you'd query the catalog.
From with psql use the shorthands to a variety of commands whose list you'll get by using \? (for help). Therefor, either of:
\d yourtable
\d+ yourtable
For use in an app, you'll need to learn the relevant queries involved. It's relatively straightforward by running psql -E (for echo hidden queries) instead of plain psql.
If you need the precise create table statement, see #Anant answer.
One easy way to do this is to use the utility provided by AWS. All you need to do is to create the view in your database and then query that view to get any table ddl. The advantage to use this view is that it will give you the sortkey and distkey as well which was used in original create table command.
https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminViews/v_generate_tbl_ddl.sql
Once the view is created, to get the the ddl of any table. You need to query like this -
select ddl from table where tablename='table_name' and schemaname='schemaname';
Note: Admin schema might not be already there in your cluster. So you can create this view in public schema.
Below query will generate the DDL of the table for you:
SELECT ddl
FROM admin.v_generate_tbl_ddl
WHERE schemaname = '<schemaname>'
AND tablename in (
'<tablename>');
Are you needing to retrieve it programatically or from the psql prompt?
In psql use : \d+ tablename
Programatically, you can query the ANSI standard INFORMATION_SCHEMA views documented here:
http://www.postgresql.org/docs/9.1/static/information-schema.html
The INFORMATION_SCHEMA.TABLES and INFORMATION_SCHEMA.COLUMNS views should have what you need.
You can use admin view provided by AWS Redshift - https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminViews/v_generate_tbl_ddl.sql
once you have created the view you can get schema creation script by running:
select * from <db_schema>.v_generate_tbl_ddl where tablename = '<table_name>'
To get the column data and schema of a particular table:
select * from information_schema.columns where tablename='<<table_name>>'
To get the information of a table metadata fire the below query
select * from information_schema.tables where schema='<<schema_name>>'
In the new "query editor 2", you can right click on a table and select "show definition", this will place the DDL for the table in a query window.
The below command will work:
mysql > show create table test.users_info;
Redshift/postgress >pg_dump -U root-w --no-password -h 62.36.11.547 -p 5439 -s -t test.users_info ;