Wrong row count using Catalog Queries - amazon-redshift

I got this query on the following page:
http://docs.aws.amazon.com/redshift/latest/dg/c_join_PG_examples.html
It lists the table names along with row count. But the row count seems to be wrong (10 times than actual)
select datname, nspname, relname, sum(rows) as rows
from pg_class, pg_namespace, pg_database, stv_tbl_perm
where pg_namespace.oid = relnamespace
and pg_class.oid = stv_tbl_perm.id
and pg_database.oid = stv_tbl_perm.db_id
and datname ='tickit'
group by datname, nspname, relname
order by datname, nspname, relname;
Is the sum(rows) correct?

Here, try mine. It includes the data size as well. :-D
/*
Ordered view of table sizes.
*/
CREATE VIEW my_schema.vw_table_summary
AS
SELECT db_name
,schema_name
,table_name
,TO_CHAR(rows_total,'999,999,999,999') rows_total
,TO_CHAR(rows_sorted,'999,999,999,999') rows_sorted
,CASE WHEN rows_total > 0
THEN ROUND((CONVERT(NUMERIC,rows_sorted)/CONVERT(NUMERIC,rows_total))*100,2)
ELSE 0.00 END percent_sorted
,COALESCE(size_in_gb,0) size_in_gb
FROM (SELECT id table_id
,datname db_name
,nspname schema_name
,relname table_name
,SUM(rows) rows_total
,SUM(sorted_rows) rows_sorted
FROM stv_tbl_perm
JOIN pg_class
ON pg_class.oid = stv_tbl_perm.id
JOIN pg_namespace
ON pg_namespace.oid = relnamespace
JOIN pg_database
ON pg_database.oid = stv_tbl_perm.db_id
WHERE name NOT LIKE 'pg_%'
AND name NOT LIKE 'stl_%'
AND name NOT LIKE 'stp_%'
AND name NOT LIKE 'padb_%'
AND nspname <> 'pg_catalog'
GROUP BY id, datname, nspname, relname
ORDER BY id, datname, nspname, relname) tbl_det
LEFT
JOIN (SELECT tbl table_id
,ROUND(CONVERT(REAL,COUNT(*))/1024,2) size_in_gb
FROM stv_blocklist bloc
GROUP BY tbl) tbl_size
ON tbl_size.table_id = tbl_det.table_id
ORDER BY db_name
,schema_name
,table_name
;

Related

Get information about table partitions

I have a partitioned table out of main table using range.
CREATE TABLE public.partition1 PARTITION OF public.maintable
FOR VALUES FROM ('2017-01-01 00:00:00') TO ('2050-01-01 00:00:00')
How can i get the Values range information using a query to postgres.
I have used a query that at least gives me information for the main and partitioned tables, but i cannot seem to find a way to access the value range
FROM ('2017-01-01 00:00:00') TO ('2050-01-01 00:00:00')
assigned to table partition1
Query used to get partition table information
WITH RECURSIVE partition_info
(relid,
relname,
relsize,
relispartition,
relkind) AS
(
(SELECT oid AS relid,
relname,
pg_relation_size(oid) AS relsize,
relispartition,
relkind
FROM pg_catalog.pg_class
WHERE relname = 'completedorders' AND
relkind = 'p')
UNION ALL
(SELECT
c.oid AS relid,
c.relname AS relname,
pg_relation_size(c.oid) AS relsize,
c.relispartition AS relispartition,
c.relkind AS relkind
FROM partition_info AS p,
pg_catalog.pg_inherits AS i,
pg_catalog.pg_class AS c
WHERE p.relid = i.inhparent AND
c.oid = i.inhrelid AND
c.relispartition = true)
)
SELECT * FROM partition_info;
The following query provides the information about partitions as well. From there on its just string manipulation in order to get further information.
Note: you will have to change the name of the table in the query.
with recursive inh as (
select i.inhrelid, null::text as parent
from pg_catalog.pg_inherits i
join pg_catalog.pg_class cl on i.inhparent = cl.oid
join pg_catalog.pg_namespace nsp on cl.relnamespace = nsp.oid
where nsp.nspname = 'public' ---<< change table schema here
and cl.relname = 'tablename' ---<< change table name here
union all
select i.inhrelid, (i.inhparent::regclass)::text
from inh
join pg_catalog.pg_inherits i on (inh.inhrelid = i.inhparent)
)
select c.relname as partition_name,
n.nspname as partition_schema,
pg_get_expr(c.relpartbound, c.oid, true) as partition_expression,
pg_get_expr(p.partexprs, c.oid, true) as sub_partition,
parent,
case p.partstrat
when 'l' then 'LIST'
when 'r' then 'RANGE'
end as sub_partition_strategy
from inh
join pg_catalog.pg_class c on inh.inhrelid = c.oid
join pg_catalog.pg_namespace n on c.relnamespace = n.oid
left join pg_partitioned_table p on p.partrelid = c.oid
order by n.nspname, c.relname

I need to know the list of tables on which there are locks currently

select
A.table_id,
last_update,
last_commit,
lock_owner_pid,
lock_status,
B.table
from stv_locks as A left outer join svv_table_info as B on A.table_id = B.table_id
order by last_update asc
I used the query above, but am getting null in a name for some table_ids. what are these?
select distinct(id) table_id
,trim(datname) db_name
,trim(nspname) schema_name
,trim(relname) table_name
from stv_locks
join stv_tbl_perm on stv_locks.table_id = stv_tbl_perm.id
join pg_class on pg_class.oid = stv_tbl_perm.id
join pg_namespace on pg_namespace.oid = relnamespace
join pg_database on pg_database.oid = stv_tbl_perm.db_id;
This should work. You can add other columns based on your use case!
If I’ve made a bad assumption please comment and I’ll refocus my answer.

Getting a column comment in PostgreSql [duplicate]

I'm running a project on a Postgres database and need to retrieve the comments on columns within the DB to be used as table headings and such. I have seen that there are a couple of built in functions (pg_description and col_description) but i haven't been able to find examples on how to use them and playing around with them has proved pretty futile.
So I was wondering if any has been able to do this before and if so, how?
select
c.table_schema,
c.table_name,
c.column_name,
pgd.description
from pg_catalog.pg_statio_all_tables as st
inner join pg_catalog.pg_description pgd on (
pgd.objoid = st.relid
)
inner join information_schema.columns c on (
pgd.objsubid = c.ordinal_position and
c.table_schema = st.schemaname and
c.table_name = st.relname
);
It all works by oid,
mat=> SELECT c.oid FROM pg_catalog.pg_class c WHERE c.relname = 'customers';
oid
-------
23208
(1 row)
Now, I have the oid for that table, so I can ask :
mat=> select pg_catalog.obj_description(23208);
obj_description
-------------------
Customers
(1 row)
Then, I can ask for the description of the fourth column :
mat=> select pg_catalog.col_description(23208,4);
col_description
-----------------------------------------
Customer codes, CHS, FACTPOST, POWER...
(1 row)
If you want to know which queries does psql run when you do \dt+ or \d+ customers, just run it with -E.
Just to be here if somebody will need it.
There are many answers here, but none of them was as simple as I would like it to be. So, based on previous answers and current postgres 9.4, I have created this query:
SELECT
obj_description(format('%s.%s',isc.table_schema,isc.table_name)::regclass::oid, 'pg_class') as table_description,
pg_catalog.col_description(format('%s.%s',isc.table_schema,isc.table_name)::regclass::oid,isc.ordinal_position) as column_description
FROM
information_schema.columns isc
It fetches table and column descriptions, without any confusing joins and ugly string concatenations.
Take care with schemas, this code considers them:
SELECT
cols.column_name, (
SELECT
pg_catalog.col_description(c.oid, cols.ordinal_position::int)
FROM
pg_catalog.pg_class c
WHERE
c.oid = (SELECT ('"' || cols.table_name || '"')::regclass::oid)
AND c.relname = cols.table_name
) AS column_comment
FROM
information_schema.columns cols
WHERE
cols.table_catalog = 'your_database'
AND cols.table_name = 'your_table'
AND cols.table_schema = 'your_schema';
References:
Postgresql Document Table and Column Description Comments on Table and Column
Determining the OID of a table in Postgres 9.1?
A slight change to one of the other answers which only gives you columns that have comments on them, this gives you all columns whether they have a comment or not.
select c.table_schema, st.relname as TableName, c.column_name,
pgd.description
from pg_catalog.pg_statio_all_tables as st
inner join information_schema.columns c
on c.table_schema = st.schemaname
and c.table_name = st.relname
left join pg_catalog.pg_description pgd
on pgd.objoid=st.relid
and pgd.objsubid=c.ordinal_position
where st.relname = 'YourTableName';
This works for me using the PostBooks 3.2.2 DB:
select cols.column_name,
(select pg_catalog.obj_description(oid) from pg_catalog.pg_class c where c.relname=cols.table_name) as table_comment
,(select pg_catalog.col_description(oid,cols.ordinal_position::int) from pg_catalog.pg_class c where c.relname=cols.table_name) as column_comment
from information_schema.columns cols
where cols.table_catalog='postbooks' and cols.table_name='apapply'
Regards,
Sylnsr
If you just need to show the comments for your columns among other data, you can also use:
\d+ my_table
Enhance for #Nick and #mat suggestions: use
SELECT obj_description('schemaName.tableName'::regclass, 'pg_class');
when you have string name (not oid).
To avoid to remember 'pg_class' parameter, and to avoid ugly concatenations at the function calls, as (tname||'.'||schema)::regclass, an useful overload for obj_description:
CREATE FUNCTION obj_description(
p_rname text, p_schema text DEFAULT NULL,
p_catalname text DEFAULT 'pg_class'
) RETURNS text AS $f$
SELECT obj_description((CASE
WHEN strpos($1, '.')>0 OR $2 IS NULL OR $2='' THEN $1
ELSE $2||'.'||$1
END)::regclass, $3);
$f$ LANGUAGE SQL IMMUTABLE;
-- USAGE: obj_description('mytable')
-- SELECT obj_description('s.t');
-- PS: obj_description('s.t', 'otherschema') is a syntax error,
-- but not generates exception: returns the same as ('s.t')
Now is easy to use, because the table name (rname parameter) is a varchar and can be expressed with a separated field for schema name, as in the main tables and queries.
See also "Getting list of table comments in PostgreSQL" or the new pg9.3 Guide
This answer is a little late, but it popped up on a google search I did to research this problem. We only needed Table descriptions, but the method would be the same for columns.
The column descriptions are in the pg_description table also, referenced by objoid.
Add this view:
CREATE OR REPLACE VIEW our_tables AS
SELECT c.oid, n.nspname AS schemaname, c.relname AS tablename, d.description,
pg_get_userbyid(c.relowner) AS tableowner, t.spcname AS "tablespace",
c.relhasindex AS hasindexes, c.relhasrules AS hasrules, c.reltriggers > 0 AS hastriggers
FROM pg_class c
LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
LEFT JOIN pg_description d ON c.oid = d.objoid
WHERE c.relkind = 'r'::"char";
ALTER TABLE our_tables OWNER TO postgres;
GRANT SELECT, UPDATE, INSERT, DELETE, REFERENCES, TRIGGER ON TABLE our_tables TO postgres;
GRANT SELECT ON TABLE our_tables TO public;
Then run:
SELECT tablename, description FROM our_tables WHERE schemaname = 'public'
The view is a modified version of the pg_tables view which adds in the description column.
You could also monkey around with the view definition to make it a single query.
I accessed table comments like this:
select c.relname table_name, pg_catalog.obj_description(c.oid) as comment from pg_catalog.pg_class c where c.relname = 'table_name';
and column comments thusly:
SELECT c.column_name, pgd.description FROM pg_catalog.pg_statio_all_tables as st inner join pg_catalog.pg_description pgd on (pgd.objoid=st.relid) inner join information_schema.columns c on (pgd.objsubid=c.ordinal_position and c.table_schema=st.schemaname and c.table_name=st.relname and c.table_name = 'table_name' and c.table_schema = 'public');
I asked a similar question about Postgresql comments last month. If you dig through that, you'll come across some Perl code over on my blog that automates the process of extracting a comment.
To pull out the column names of a table, you can use something like the following:
select
a.attname as "colname"
,a.attrelid as "tableoid"
,a.attnum as "columnoid"
from
pg_catalog.pg_attribute a
inner join pg_catalog.pg_class c on a.attrelid = c.oid
where
c.relname = 'mytable' -- better to use a placeholder
and a.attnum > 0
and a.attisdropped is false
and pg_catalog.pg_table_is_visible(c.oid)
order by a.attnum
You can then use the tableoid,columnoid tuple to extract the comment of each column (see my question).
I just found this here. It will provide you with all kind of metadata on one specific table (type, default value, not null flag, length, comment, foreign key name, primary key name). It seems to work well.
SELECT pg_tables.tablename, pg_attribute.attname AS field,
format_type(pg_attribute.atttypid, NULL) AS "type",
pg_attribute.atttypmod AS len,
(SELECT col_description(pg_attribute.attrelid,
pg_attribute.attnum)) AS comment,
CASE pg_attribute.attnotnull
WHEN false THEN 1 ELSE 0
END AS "notnull",
pg_constraint.conname AS "key", pc2.conname AS ckey,
(SELECT pg_attrdef.adsrc FROM pg_attrdef
WHERE pg_attrdef.adrelid = pg_class.oid
AND pg_attrdef.adnum = pg_attribute.attnum) AS def
FROM pg_tables, pg_class
JOIN pg_attribute ON pg_class.oid = pg_attribute.attrelid
AND pg_attribute.attnum > 0
LEFT JOIN pg_constraint ON pg_constraint.contype = 'p'::"char"
AND pg_constraint.conrelid = pg_class.oid AND
(pg_attribute.attnum = ANY (pg_constraint.conkey))
LEFT JOIN pg_constraint AS pc2 ON pc2.contype = 'f'::"char"
AND pc2.conrelid = pg_class.oid
AND (pg_attribute.attnum = ANY (pc2.conkey))
WHERE pg_class.relname = pg_tables.tablename
-- AND pg_tables.tableowner = "current_user"()
AND pg_attribute.atttypid <> 0::oid
AND tablename='your_table'
ORDER BY field ASC
Source: http://golden13.blogspot.de/2012/08/how-to-get-some-information-about_7.html
Ok, so i worked it out to degree...
select col_description(table id, column number)...
ie: select col_description(36698,2);
That worked, but is there an easier way to do this maybe bringing all the comments on all the columns and using the table name instead of the oid???
To display comments from all columns of all table :
SELECT
cols.table_name,
cols.column_name, (
SELECT
pg_catalog.col_description(c.oid, cols.ordinal_position::int)
FROM
pg_catalog.pg_class c
WHERE
c.oid = (SELECT ('"' || cols.table_name || '"')::regclass::oid)
AND c.relname = cols.table_name
) AS column_comment
FROM
information_schema.columns cols
WHERE
cols.table_name IN (SELECT cols.table_name FROM information_schema.columns)
AND cols.table_catalog = 'your_database_name'
AND cols.table_schema = 'your_schema_name';
You need to execute this query outside any schema/catalog/db
This query is based on another answer in this question which display comments from one table only
To extend on the response provided by #amxy; I found that adding a schema filter can help in some environments. As I found #amxy's solution didn't work until I added by schema filters
SELECT
pg_tables.schemaname,
pg_tables.TABLENAME,
pg_attribute.attname AS field,
format_type(pg_attribute.atttypid, NULL) AS "type",
pg_attribute.atttypmod AS len,
(
SELECT col_description(pg_attribute.attrelid, pg_attribute.attnum)) AS COMMENT,
CASE pg_attribute.attnotnull
WHEN FALSE THEN 1
ELSE 0
END AS "notnull",
pg_constraint.conname AS "key", pc2.conname AS ckey,
(
SELECT pg_attrdef.adsrc
FROM pg_attrdef
WHERE pg_attrdef.adrelid = pg_class.oid
AND pg_attrdef.adnum = pg_attribute.attnum) AS def
FROM pg_tables, pg_class
JOIN pg_attribute
ON pg_class.oid = pg_attribute.attrelid
AND pg_attribute.attnum > 0
LEFT JOIN pg_constraint
ON pg_constraint.contype = 'p'::"char"
AND pg_constraint.conrelid = pg_class.oid
AND
(pg_attribute.attnum = ANY (pg_constraint.conkey))
LEFT JOIN pg_constraint AS pc2
ON pc2.contype = 'f'::"char"
AND pc2.conrelid = pg_class.oid
AND (pg_attribute.attnum = ANY (pc2.conkey))
WHERE pg_class.relname = pg_tables.TABLENAME
AND pg_tables.schemaname IN ('op', 'im', 'cs','usr','li')
-- AND pg_tables.tableowner = "current_user"()
AND pg_attribute.atttypid <> 0::oid
---AND TABLENAME='your_table'
ORDER BY pg_tables.schemaname,
pg_tables.TABLENAME ASC;
RESULTS:
SELECT
relname table_name,
obj_description(oid) table_description,
column_name,
pgd.description column_description
FROM pg_class
INNER JOIN
information_schema.columns
ON table_name = pg_class.relname
LEFT JOIN
pg_catalog.pg_description pgd
ON pgd.objsubid = ordinal_position
WHERE
relname = 'your_table_name'
SELECT sc.table_schema , sc.table_name, sc.column_name, col_description(pc."oid" , sc.ordinal_position) col_description FROM pg_class pc
INNER JOIN pg_namespace ns ON ns."oid" =pc.relnamespace
INNER JOIN information_schema.COLUMNS sc ON sc.table_name=pc.relname AND sc.table_schema=ns.nspname
WHERE 1=1
AND upper(ns.nspname) = 'TABLE_SCHEMA'
AND upper(pc.relname) = 'TABLE_NAME'
Retrieving Comments from a PostgreSQL DB

Listing all relations with schema and columns in PostgreSQL

I am trying to list all the relations with their corresponding schema and columns in PostgreSQL using System Catalog constructs. But it also ends up showing some index names along with relation names, and shows attributes like cmax, cmin, ctid, etc. and so on which are not the actual attributes that I had create while constructing the table. Here is my query:
SELECT
ns.nspname AS schema_name,
idx.attrelid :: REGCLASS AS table_name,
idx.attname AS column_name
FROM pg_attribute AS idx
JOIN pg_class AS i
ON i.oid = idx.attrelid
JOIN pg_namespace AS NS ON i.relnamespace = NS.OID
WHERE nspname='public';
To filter out system attributes, add attnum > 0. To filter out non-tables, add relkind = 'r'. To it would look like
SELECT
ns.nspname AS schema_name,
idx.attrelid :: REGCLASS AS table_name,
idx.attname AS column_name
FROM pg_attribute AS idx
JOIN pg_class AS i
ON i.oid = idx.attrelid
JOIN pg_namespace AS NS ON i.relnamespace = NS.OID
WHERE nspname='public' AND attnum > 0 AND relkind = 'r';
You could also use the information schema, which is a bit more user-friendly in this aspect:
SELECT table_schema, table_name, column_name
FROM information_schema.columns
WHERE table_schema = 'public';

redshift - how to use listagg with information_schema.columns table

I'm using redshift and would like to create a comma separated list of columns. I'm trying to grab the column names from information schema using listagg:
SELECT
listagg(column_name,',') within group (order by ordinal_position)
FROM information_schema.columns
WHERE table_schema = 'my_schema'
AND table_name = 'my table';
I'm getting the following error:
[Amazon](500310) Invalid operation: Function (listagg(text,text)) must be applied on at least one user created tables;
Here is a work around I tested..
select listagg(column_name, ', ') within group (order by column_name)
from
(
select
a.attname::varchar as column_name, typname::varchar as data_type
from
pg_type t,
pg_attribute a,
pg_class c,
pg_namespace ns,
(select top 1 1 from my_schema.my_table)
where
t.oid=a.atttypid
and a.attrelid = c.oid
and c.relnamespace = ns.oid
and typname NOT IN ('oid','xid','tid','cid')
and attname not in ('deletexid', 'insertxid')
and trim(relname) = 'my_table'
and ns.nspname = 'my_schema'
)
Although this does not answer how to apply listagg on information_schema, I can recommend an alternative method of using listagg on the pg catalog tables instead.
Try this:
SELECT DISTINCT listagg(attname, ',') WITHIN GROUP (ORDER BY a.attsortkeyord) AS "columns"
FROM pg_attribute a, pg_namespace ns, pg_class c, pg_type t, stv_tbl_perm p, pg_database db
WHERE t.oid=a.atttypid AND a.attrelid=p.id AND ns.oid = c.relnamespace AND db.oid = p.db_id AND c.oid = a.attrelid
AND typname NOT IN ('oid','xid','tid','cid')
AND ns.nspname = 'my_schema'
AND RTRIM(name) = 'my table'