In a clean-up effort, I changed some schema names in Redshift. Then I nearly immediately switched the schema names back. All but a few of the tables disappeared.
Is this a known issue?
Should I be more careful about renaming tables to previous names?
sql> ALTER SCHEMA common_schema RENAME TO common_schema_v1
[2019-05-01 14:39:25] completed in 432 ms
sql> ALTER SCHEMA common_schema_v1 RENAME TO common_schema
[2019-05-01 14:48:41] completed in 371 ms
The tables would not normally be dropped by a rename operation.
It could be the rename changed your search path and you're just not seeing the tables now. Try re-adding the schema name to your search path.
SHOW search_path;
SET search_path TO public, common_schema;
You can also look for the tables in the catalog to confirm they're still there.
SELECT *
FROM information_schema.tables
WHERE table_schema = 'common_schema'
;
Or
SELECT nspname AS schema_name
, relname AS table_name
FROM pg_class c
, pg_namespace n
WHERE n.oid = c.relnamespace
AND c.reltype > 0
AND n.nspname = 'common_schema'
ORDER BY 1, 2
;
Related
In Redshift, I am checking Users' grants. How do I know if they can alter tables?
I can know if they can drop tables because "Only the owner of the table, the schema owner, or a superuser can drop a table." - https://docs.aws.amazon.com/redshift/latest/dg/r_DROP_TABLE.html No such qualification exists for alter table: https://docs.aws.amazon.com/redshift/latest/dg/r_ALTER_TABLE.html
HAS_TABLE_PRIVILEGE provides info about other privileges, but not alter table: https://docs.aws.amazon.com/redshift/latest/dg/r_HAS_TABLE_PRIVILEGE.html
I got a response from AWS Support. tl;dr: alter table can be run by those, and only those, who can drop table.
"""
This is because, determining whether or not a user can alter a table, works in a similar way to that were one determines whether or not a given user can drop a table. That is, only the owner of the table, the schema owner, or a superuser can Alter a table. According to our documentation, "The right to modify or destroy an object is always the privilege of the owner only." [1].
[1] Default database user privileges - https://docs.aws.amazon.com/redshift/latest/dg/r_Privileges.html
Therefore, to see the users with alter table permissions for a specific table, there is need to determine the owner of that specific table by running the following command:
Kindly note that in this example, the 'sales' table is used. You can edit this as you see fit. To see all the table owners, the AND section of the WHERE clause can be removed.
====Query to see table owners====
SELECT n.nspname AS schema_name
, pg_get_userbyid(c.relowner) AS table_owner
, c.relname AS table_name
, CASE WHEN c.relkind = 'v' THEN 'view' ELSE 'table' END
AS table_type
, d.description AS table_description
FROM pg_class As c
LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
LEFT JOIN pg_description As d
ON (d.objoid = c.oid AND d.objsubid = 0)
WHERE c.relkind IN('r', 'v')
AND c.relname = 'sales'
ORDER BY n.nspname, c.relname;
You can also see all the superusers who have permissions to Alter table by running the following query:
====Query to see superusers====
SELECT usename FROM pg_user WHERE usesuper = 'true';
The combination of both results will enable you to see all the users which have alter table permissions.
"""
I have to update all columns of type "uuid" to "varchar(38)". I created all the necessary queries with:
SELECT format(
'ALTER TABLE %I.%I.%I ALTER COLUMN %I SET DATA TYPE varchar(38);',
table_catalog,
table_schema,
table_name,
column_name
)
FROM information_schema.columns
WHERE data_type = 'uuid'
AND table_schema NOT LIKE 'pg_%'
AND lower(table_schema) <> 'information_schema'
AND is_updatable = 'YES';
Obviously, I can't execute the resulting queries because of all the existing PK and FK constraints involving the uuid columns.
Is there a way to temporarily disable the constraints, then executing all the queries and reactivating the constraints afterwards without dropping the constraints?
Or if I have to drop all the constraints first, is there a way to set them all up again after the updates? I am not the creator of the database so I don't have all necessary queries to create the constraints again.
I found a way to create all queries for dropping and creating all constraints of the database.
So first I have to save the output of the first query
SELECT 'ALTER TABLE "'||nspname||'"."'||relname||'" DROP CONSTRAINT "'||conname||'";'
FROM pg_constraint
INNER JOIN pg_class ON conrelid=pg_class.oid
INNER JOIN pg_namespace ON pg_namespace.oid=pg_class.relnamespace
ORDER BY CASE WHEN contype='f' THEN 0 ELSE 1 END,contype,nspname,relname,conname;
and of the second query
SELECT 'ALTER TABLE "'||nspname||'"."'||relname||'" ADD CONSTRAINT "'||conname||'" "'||
pg_get_constraintdef(pg_constraint.oid)||'";'
FROM pg_constraint
INNER JOIN pg_class ON conrelid=pg_class.oid
INNER JOIN pg_namespace ON pg_namespace.oid=pg_class.relnamespace
ORDER BY CASE WHEN contype='f' THEN 0 ELSE 1 END DESC,contype DESC,nspname DESC,relname DESC,conname DESC;
When I have all the queries, I first dropped every constrained, updated the tables and then executed the queries for adding the constraints again. Worked perfectly!
I just created an new database and it already takes up 7MB. Do you know what is taking up this much space? Is there a way to get the "real" size of the database used as in how much data is stored?
0f41ba72-a1ea-4516-a9f0-de8a3609bc4a=> select pg_size_pretty(pg_database_size(current_database()));
pg_size_pretty
----------------
7055 kB
(1 row)
0f41ba72-a1ea-4516-a9f0-de8a3609bc4a=> \dt
No relations found.
Well, even you don't created any relation yet the new database is not empty. When a CREATE DATABASE is issued, Postgres copy a TEMPLATE database - which comes with catalog tables - to a new database. In fact, "Nothing is created, everything is transformed". You can use commands below to inspect this:
--Size per table
SELECT pg_size_pretty(pg_total_relation_size(oid)), relname FROM pg_class WHERE relkind = 'r' AND NOT relisshared;
--Total size
SELECT pg_size_pretty(sum(pg_total_relation_size(oid))) FROM pg_class WHERE relkind = 'r' AND NOT relisshared;
--Total size of databases
SELECT pg_size_pretty(pg_database_size(oid)), datname FROM pg_database;
A quote from the docs:
By default, the new database will be created by cloning the standard
system database template1.
An empty database contains system catalogs and The Information Schema.
Execute this query to see them:
select nspname as schema, relname as table, pg_total_relation_size(c.oid)
from pg_class c
join pg_namespace n on n.oid = relnamespace
order by 3 desc;
schema | table | pg_total_relation_size
--------------------+-----------------------------+------------------------
pg_catalog | pg_depend | 1146880
pg_catalog | pg_proc | 950272
pg_catalog | pg_rewrite | 589824
pg_catalog | pg_attribute | 581632
... etc
You can get the total size of non-system relations with the query:
select sum(pg_total_relation_size(c.oid))
from pg_class c
join pg_namespace n on n.oid = relnamespace
where nspname not in ('information_schema', 'pg_catalog', 'pg_toast');
The query returns null on empty database.
Every PostgreSQL databases has own system catalogue .. 7MB. So your numbers are correct. PostgreSQL is designed for client-server architecture and 1GB and longer databases - so this cost is not significant.
If you need reduced space allocation, you can try embedded databases like SQLite or Firebird.
Introduction
I've been developing a wizard to create complex database Postgres queries for users without any programming/SQL background. Thanks to foreign key constraints stored in a view in information_schema, the user may select any number of tables and the tool will find the correct join settings (thus, the user does not have to add ON table_a.field_1 = table_b.field_2).
While developing, I have been using an administration database user and now wanted to change that to a read-only user to make it more secure. However, this read-only user seems not to be able to access the foreign key constraints.
Current situation
When more than one table has been selected, the tool tries to get the connections between the various tables in order to know how to join them. During that process, the following query is executed:
SELECT
tc.constraint_name,
tc.table_name,
kcu.column_name,
ccu.table_name AS foreign_table_name,
ccu.column_name AS foreign_column_name
FROM information_schema.table_constraints AS tc
JOIN information_schema.key_column_usage AS kcu
ON tc.constraint_name = kcu.constraint_name
JOIN information_schema.constraint_column_usage AS ccu
ON ccu.constraint_name = tc.constraint_name
WHERE constraint_type = 'FOREIGN KEY'
AND ccu.table_name = 'TableB'
AND tc.table_name IN ('TableA');
(Note: the last WHERE clause uses IN because there can be more than one base table available. TableA is the base table and each successfully connected/joined table will be available for additional joins, e.g. a third table could use AND ccu.table_name = 'TableC' AND tc.table_name IN ('TableA', 'TableB'); and so on.)
When using the admin db user (with most common privileges like GRANT, SELECT, INSERT, UPDATE, DELETE, TRUNCATE, ...) executes the query, the result looks something like this:
constraint_name | table_name | column_name | foreign_table_name | foreign_column_name
----------------+------------+-------------+--------------------+---------------------
constraint1 | TableA | field_1 | TableB | field_2
(1 row)
But when the read-only db user runs that query, it returns:
constraint_name | table_name | column_name | foreign_table_name | foreign_column_name
----------------+------------+-------------+--------------------+---------------------
(0 rows)
Due to the existing but not returned foreign key constraint entry, the joins can not be properly written as SQL and the user generated query (by using the wizard) fails.
What I tried
First of course, I thought the read-only user (ro_user) might not have the permissions to access tables and views in database information_schema. So I ran
GRANT SELECT ON ALL TABLES IN SCHEMA information_schema TO ro_user;
as admin but to no avail. Getting more into the depths of the documentation, I found that all tables and views in information_schema are available and accessible to any user by default in postgres anyways. So granting the select privilege shouldn't even change anything.
Just to make sure, I also ran
GRANT REFERENCES ON ALL TABLES IN SCHEMA actual_database TO ro_user;
but of course, this didn't change anything neither, since REFERENCESis only needed for creating new foreign key, I just need to read them.
Next, I thought, maybe the sql from the tool is failing due to some information not being available, so I queried the three views separately by running:
SELECT * FROM information_schema.table_constraints AS tc WHERE constraint_type = 'FOREIGN KEY';
SELECT * FROM information_schema.key_column_usage AS kcu;
SELECT * FROM information_schema.constraint_column_usage AS ccu;
And sure enough, the last one wouldn't return any single row for the ro_user:
psql=> SELECT * FROM information_schema.constraint_column_usage AS ccu;
table_catalog | table_schema | table_name | column_name | constraint_catalog | constraint_schema | constraint_name
---------------+--------------+------------+-------------+--------------------+-------------------+-----------------
(0 rows)
whereas the admin user got lots of results. So, it was coming down to that one view information_schema.constraint_column_usage.
As I was typing out that question over the course of an hour recollecting and boiling down all the ideas I tried during the last days, I finally found the cause.
The view constraint_column_usage identifies all columns in the current database that are used by some constraint. Only those columns are shown that are contained in a table owned by a currently enabled role.
From documentation via this SO answer
And through that I found a solution
SELECT
conrelid::regclass AS table_from,
conname,
pg_get_constraintdef(c.oid) AS cdef
FROM pg_constraint c
JOIN pg_namespace n
ON n.oid = c.connamespace
WHERE contype IN ('f')
AND n.nspname = 'public'
AND pg_get_constraintdef(c.oid) LIKE '%"TableB"%'
AND conrelid::regclass::text IN ('"TableA"')
ORDER BY conrelid::regclass::text, contype DESC;
It doesn't output the same format as the old query, but it contains the same information and is - most importantly - available to the ro_user.
Does anyone know how to find the OID of a table in Postgres 9.1?
I am writing an update script that needs to test for the existence of a column in a table before it tries to add the column. This is to prevent errors when running the script repeatedly.
To get a table OID, cast to the object identifier type regclass (while connected to the same DB):
SELECT 'mytbl'::regclass::oid;
This finds the first table (or view, etc.) with the given name along the search_path or raises an exception if not found.
Schema-qualify the table name to remove the dependency on the search path:
SELECT 'myschema.mytbl'::regclass::oid;
In Postgres 9.4 or later you can also use to_regclass('myschema.mytbl'), which doesn't raise an exception if the table is not found:
How to check if a table exists in a given schema
Then you only need to query the catalog table pg_attribute for the existence of the column:
SELECT TRUE AS col_exists
FROM pg_attribute
WHERE attrelid = 'myschema.mytbl'::regclass
AND attname = 'mycol'
AND NOT attisdropped -- no dropped (dead) columns
-- AND attnum > 0 -- no system columns (you may or may not want this)
;
The postgres catalog table pg_class is what you should look at. There should be one row per table, with the table name in the column relname, and the oid in the hidden column oid.
You may also be interested in the pg_attribute catalog table, which includes one row per table column.
See: http://www.postgresql.org/docs/current/static/catalog-pg-class.html and http://www.postgresql.org/docs/current/static/catalog-pg-attribute.html
SELECT oid FROM pg_class WHERE relname = 'tbl_name' AND relkind = 'r';
Just to complete the possibilities I'd like to add that there exists a syntax for dropping columns in order to no error out:
ALTER TABLE mytbl
DROP COLUMN IF EXISTS mycol
See http://www.postgresql.org/docs/9.0/static/sql-altertable.html
Then you can safely add your column.