I am using pg_table_def to work out the primary keys associated to a given table, example below:
SELECT pg_table_def.tablename,
pg_table_def.column
FROM pg_table_def
WHERE schemaname = 'myschema'
AND tablename LIKE '%mytable%'
AND tablename LIKE '%_pkey%';
Example output from the above query:
tablename column
mytable_$mig_pkey id
This is correctly identifying the primary keys, but in some cases the format of the tablename returned has the text _$mig in it, for example: mytable_$mig_pkey.
I use the tablename in a later part of the code so currently have to remove this text. My questions are:
Why for some tables is the tablename formatted like this with _$mig?
Are there other strings that could appear between tablename and _pkey that should be taken into consideration?
Or is there a better way to identify the primary keys associated to a
table?
I've so far tried:
Checking the AWS documentation it recommends to check the search_path, so I've done this and made sure the schema is referenced in the where clause.
Checked that mytable_$mig doesn't exist / isn't available.
Other googling hasn't got me any further...
Update
Based on the comments below, running the below query:
select tablename, "column", type, encoding, distkey, sortkey, "notnull"
from pg_table_def
where schemaname = 'myschema' AND tablename LIKE '%mytable%';
Returns:
tablename column type encoding distkey sortkey notnull
mytable id integer none true 1 true
mytable desc character varying(50) lzo false 0 false
mytable_$mig_pkey id integer none true 1 false
Update:
So it looks like you have your actual table and one that was probably created accidentally. If you do select count(*) from mytable and select count(*) from mytable_$mig_pkey;, hopefully mytable has all your data and the other one is
empty. If so you can clean it up with drop table mytable_$mig_pkey.
I see you have the id column as the distkey which is Redshift's version of a primary key. You may also be trying to define constraints, which Redshift will allow you to do, but they are not enforced.
You can view the defined table constraints for a table like so:
select * from information_schema.table_constraints
where table_name='mytable';
On one of my tables gives a result like:
constraint_catalog | constraint_schema | constraint_name | table_catalog | table_schema | table_name | constraint_type | is_deferrable | initially_deferred
--------------------+-------------------+-----------------+---------------+--------------+------------+-----------------+---------------+--------------------
mydb | myschema | mytable_pkey | mydb | myschema | mytable | PRIMARY KEY | NO | NO
(1 row)
My current guess is that you attempted to define constraints on your table and accidentally created a second table at some point, which is why the other table has _pkey in its name.
I use a query like this myself:
select tablename, "column", type, encoding, distkey, sortkey, "notnull"
from pg_table_def
where schemaname = 'myschema' AND tablename LIKE '%mytable%';
Running something more general like that might help you figure out what's going on.
The \d I referenced in the comments is a way to list all the tables in the database that is a convenience function built into the psql command line app. The query it runs to do that is this:
SELECT n.nspname as "Schema",
c.relname as "Name",
CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' WHEN 'f' THEN 'foreign table' END as "Type",
pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
FROM pg_catalog.pg_class c
LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind IN ('r','v','m','S','f','')
AND n.nspname <> 'pg_catalog'
AND n.nspname <> 'information_schema'
AND n.nspname !~ '^pg_toast'
AND pg_catalog.pg_table_is_visible(c.oid)
ORDER BY 1,2;
Running that might help.
Hello I am trying to retrieve the schema of an existing table. I am mysql developer and am trying to work with amazon redshift. How can I export the schema of an existing table. In mysql we can use the show create table command.
SHOW CREATE TABLE tblName;
Recently I wrote a python script to clone table schemas between redshift clusters. If you only want the columns and column types of a table, you can do it via:
select column_name,
case
when data_type = 'integer' then 'integer'
when data_type = 'bigint' then 'bigint'
when data_type = 'smallint' then 'smallint'
when data_type = 'text' then 'text'
when data_type = 'date' then 'date'
when data_type = 'real' then 'real'
when data_type = 'boolean' then 'boolean'
when data_type = 'double precision' then 'float8'
when data_type = 'timestamp without time zone' then 'timestamp'
when data_type = 'character' then 'char('||character_maximum_length||')'
when data_type = 'character varying' then 'varchar('||character_maximum_length||')'
when data_type = 'numeric' then 'numeric('||numeric_precision||','||numeric_scale||')'
else 'unknown'
end as data_type,
is_nullable,
column_default
from information_schema.columns
where table_schema = 'xxx' and table_name = 'xxx' order by ordinal_position
;
But if you need the compression types and distkey/sortkeys, you need to query another table:
select * from pg_table_def where tablename = 'xxx' and schemaname='xxx';
This query will give you the complete schema definition including the Redshift specific attributes distribution type/key, sort key, primary key, and column encodings in the form of a create statement as well as providing an alter table statement that sets the owner to the current owner. The only thing it can't tell you are foreign keys. I'm working on the latter, but there's a current privilege issue in RS that prevents us from querying the right tables. This query could use some tuning, but I haven't had time or the need to work it further.
select pk.pkey, tm.schemaname||'.'||tm.tablename, 'create table '||tm.schemaname||'.'||tm.tablename
||' ('
||cp.coldef
-- primary key
||decode(pk.pkey,null,'',pk.pkey)
-- diststyle and dist key
||decode(d.distkey,null,') diststyle '||dist_style||' ',d.distkey)
--sort key
|| (select decode(skey,null,'',skey) from (select
' sortkey(' ||substr(array_to_string(
array( select ','||cast(column_name as varchar(100)) as str from
(select column_name from information_schema.columns col where col.table_schema= tm.schemaname and col.table_name=tm.tablename) c2
join
(-- gives sort cols
select attrelid as tableid, attname as colname, attsortkeyord as sort_col_order from pg_attribute pa where
pa.attnum > 0 AND NOT pa.attisdropped AND pa.attsortkeyord > 0
) st on tm.tableid=st.tableid and c2.column_name=st.colname order by sort_col_order
)
,'')
,2,10000) || ')' as skey
))
||';'
-- additional alter table queries here to set owner
|| 'alter table '||tm.schemaname||'.'||tm.tablename||' owner to "'||tm.owner||'";'
from
-- t master table list
(
SELECT substring(n.nspname,1,100) as schemaname, substring(c.relname,1,100) as tablename, c.oid as tableid ,use2.usename as owner, decode(c.reldiststyle,0,'EVEN',1,'KEY',8,'ALL') as dist_style
FROM pg_namespace n, pg_class c, pg_user use2
WHERE n.oid = c.relnamespace
AND nspname NOT IN ('pg_catalog', 'pg_toast', 'information_schema')
AND c.relname <> 'temp_staging_tables_1'
and c.relowner = use2.usesysid
) tm
-- cp creates the col params for the create string
join
(select
substr(str,(charindex('QQQ',str)+3),(charindex('ZZZ',str))-(charindex('QQQ',str)+3)) as tableid
,substr(replace(replace(str,'ZZZ',''),'QQQ'||substr(str,(charindex('QQQ',str)+3),(charindex('ZZZ',str))-(charindex('QQQ',str)+3)),''),2,10000) as coldef
from
( select array_to_string(array(
SELECT 'QQQ'||cast(t.tableid as varchar(10))||'ZZZ'|| ','||column_name||' '|| decode(udt_name,'bpchar','char',udt_name) || decode(character_maximum_length,null,'', '('||cast(character_maximum_length as varchar(9))||')' )
-- default
|| decode(substr(column_default,2,8),'identity','',null,'',' default '||column_default||' ')
-- nullable
|| decode(is_nullable,'YES',' NULL ','NO',' NOT NULL ')
-- identity
|| decode(substr(column_default,2,8),'identity',' identity('||substr(column_default,(charindex('''',column_default)+1), (length(column_default)-charindex('''',reverse(column_default))-charindex('''',column_default) ) ) ||') ', '')
-- encoding
|| decode(enc,'none','',' encode '||enc)
as str
from
-- ci all the col info
(
select cast(t.tableid as int), cast(table_schema as varchar(100)), cast(table_name as varchar(100)), cast(column_name as varchar(100)),
cast(ordinal_position as int), cast(column_default as varchar(100)), cast(is_nullable as varchar(20)) , cast(udt_name as varchar(50)) ,cast(character_maximum_length as int),
sort_col_order , decode(d.colname,null,0,1) dist_key , e.enc
from
(select * from information_schema.columns c where c.table_schema= t.schemaname and c.table_name=t.tablename) c
left join
(-- gives sort cols
select attrelid as tableid, attname as colname, attsortkeyord as sort_col_order from pg_attribute a where
a.attnum > 0 AND NOT a.attisdropped AND a.attsortkeyord > 0
) s on t.tableid=s.tableid and c.column_name=s.colname
left join
(-- gives encoding
select attrelid as tableid, attname as colname, format_encoding(a.attencodingtype::integer) AS enc from pg_attribute a where
a.attnum > 0 AND NOT a.attisdropped
) e on t.tableid=e.tableid and c.column_name=e.colname
left join
-- gives dist col
(select attrelid as tableid, attname as colname from pg_attribute a where
a.attnum > 0 AND NOT a.attisdropped AND a.attisdistkey = 't'
) d on t.tableid=d.tableid and c.column_name=d.colname
order by ordinal_position
) ci
-- for the working array funct
), '') as str
from
(-- need tableid
SELECT substring(n.nspname,1,100) as schemaname, substring(c.relname,1,100) as tablename, c.oid as tableid
FROM pg_namespace n, pg_class c
WHERE n.oid = c.relnamespace
AND nspname NOT IN ('pg_catalog', 'pg_toast', 'information_schema')
) t
)) cp on tm.tableid=cp.tableid
-- primary key query here
left join
(select c.oid as tableid, ', primary key '|| substring(pg_get_indexdef(indexrelid),charindex('(',pg_get_indexdef(indexrelid))-1 ,60) as pkey
from pg_index i , pg_namespace n, pg_class c
where i.indisprimary=true
and i.indrelid =c.oid
and n.oid = c.relnamespace
) pk on tm.tableid=pk.tableid
-- dist key
left join
( select
-- close off the col defs after the primary key
')' ||
' distkey('|| cast(column_name as varchar(100)) ||')' as distkey, t.tableid
from information_schema.columns c
join
(-- need tableid
SELECT substring(n.nspname,1,100) as schemaname, substring(c.relname,1,100) as tablename, c.oid as tableid
FROM pg_namespace n, pg_class c
WHERE n.oid = c.relnamespace
AND nspname NOT IN ('pg_catalog', 'pg_toast', 'information_schema')
) t on c.table_schema= t.schemaname and c.table_name=t.tablename
join
-- gives dist col
(select attrelid as tableid, attname as colname from pg_attribute a where
a.attnum > 0 AND NOT a.attisdropped AND a.attisdistkey = 't'
) d on t.tableid=d.tableid and c.column_name=d.colname
) d on tm.tableid=d.tableid
where tm.schemaname||'.'||tm.tablename='myschema.mytable'
If you want to get the table structure with create statement, constraints and triggers, you can use pg_dump utility
pg_dump -U user_name -s -t table_name -d db_name
Note: -s used for schema only dump
if you want to take the data only dump , you can use -a switch.
This will output the create syntax with all the constraints. Hope this will help you.
I did not find any complete solutions out there.
And wrote a python script:
https://github.com/cxmcc/redshift_show_create_table
It will work like pg_dump, plus dealing with basic redshift features, SORTKEY/DISTKEY/DISTSTYLES etc.
As show table doesn't work on Redshift:
show table <YOUR_TABLE>;
ERROR: syntax error at or near "<YOUR_TABLE>"
We can use pg_table_def table to get the schema out:
select "column", type, encoding, distkey, sortkey, "notnull"
from pg_table_def
where tablename = '<YOUR_TABLE>';
NOTE: If the schema is not on the search path, add it to search path using:
set search_path to '$user', 'public', '<YOUR_SCHEMA>';
For redshift please try
show table <**tablename**> ;
In Postgres, you'd query the catalog.
From with psql use the shorthands to a variety of commands whose list you'll get by using \? (for help). Therefor, either of:
\d yourtable
\d+ yourtable
For use in an app, you'll need to learn the relevant queries involved. It's relatively straightforward by running psql -E (for echo hidden queries) instead of plain psql.
If you need the precise create table statement, see #Anant answer.
One easy way to do this is to use the utility provided by AWS. All you need to do is to create the view in your database and then query that view to get any table ddl. The advantage to use this view is that it will give you the sortkey and distkey as well which was used in original create table command.
https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminViews/v_generate_tbl_ddl.sql
Once the view is created, to get the the ddl of any table. You need to query like this -
select ddl from table where tablename='table_name' and schemaname='schemaname';
Note: Admin schema might not be already there in your cluster. So you can create this view in public schema.
Below query will generate the DDL of the table for you:
SELECT ddl
FROM admin.v_generate_tbl_ddl
WHERE schemaname = '<schemaname>'
AND tablename in (
'<tablename>');
Are you needing to retrieve it programatically or from the psql prompt?
In psql use : \d+ tablename
Programatically, you can query the ANSI standard INFORMATION_SCHEMA views documented here:
http://www.postgresql.org/docs/9.1/static/information-schema.html
The INFORMATION_SCHEMA.TABLES and INFORMATION_SCHEMA.COLUMNS views should have what you need.
You can use admin view provided by AWS Redshift - https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminViews/v_generate_tbl_ddl.sql
once you have created the view you can get schema creation script by running:
select * from <db_schema>.v_generate_tbl_ddl where tablename = '<table_name>'
To get the column data and schema of a particular table:
select * from information_schema.columns where tablename='<<table_name>>'
To get the information of a table metadata fire the below query
select * from information_schema.tables where schema='<<schema_name>>'
In the new "query editor 2", you can right click on a table and select "show definition", this will place the DDL for the table in a query window.
The below command will work:
mysql > show create table test.users_info;
Redshift/postgress >pg_dump -U root-w --no-password -h 62.36.11.547 -p 5439 -s -t test.users_info ;
Is there a simple alternative in PostgreSQL to this statement produced in Oracle?
select table_name from user_tab_columns
where table_name = myTable and column_name = myColumn;
I am then testing whether the query returns anything so as to prove the column exists.
I am aware that using psql I can find these out individually but this is required to produce a result in a program I am writing to validate that a requested attribute field exists in my database table.
Try this :
SELECT column_name
FROM information_schema.columns
WHERE table_name='your_table' and column_name='your_column';
Accepted answer is correct, but is missing the schema and nicer output (True/False):
SELECT EXISTS (SELECT 1
FROM information_schema.columns
WHERE table_schema='my_schema' AND table_name='my_table' AND column_name='my_column');
Simpler and SQLi-safe using PostgreSQL's object identifier types:
SELECT true
FROM pg_attribute
WHERE attrelid = 'myTable'::regclass -- cast to a registered class (table)
AND attname = 'myColumn'
AND NOT attisdropped -- exclude dropped (dead) columns
-- AND attnum > 0 -- exclude system columns (you may or may not want this)
System catalogs are many times faster than querying the notoriously convoluted information_schema (but still just milliseconds for a single query). See:
Get column names and data types of a query, table or view
Read about the significance of the columns in the manual.
While building dynamic SQL with the column name supplied as parameter, use quote_ident() to defend against SQL injection:
...
AND attname = quote_ident('myColumn');
Works for tables outside the search_path, too:
...
WHERE attrelid = 'mySchema.myTable'::regclass
...
Unlike Oracle, PostgreSQL supports the ANSI standard INFORMATION_SCHEMA views.
The corresponding standard view to Oracle's user_tab_columns is information_schema.columns
http://www.postgresql.org/docs/current/static/infoschema-columns.html
SELECT attname
FROM pg_attribute
WHERE attrelid = (SELECT oid FROM pg_class WHERE relname = 'YOURTABLENAME')
AND attname = 'YOURCOLUMNNAME';
Of course, replace YOURTABLENAME and YOURCOLUMNNAME with the proper values. If a row is returned, a column with that name exists, otherwise it does not.
Here is a similar variant of Erwin Brandstetter answer.
Here we check schema too in case we have similar tables in different schema.
SELECT TRUE FROM pg_attribute
WHERE attrelid = (
SELECT c.oid
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE
n.nspname = CURRENT_SCHEMA()
AND c.relname = 'YOURTABLENAME'
)
AND attname = 'YOURCOLUMNNAME'
AND NOT attisdropped
AND attnum > 0
After the excellent answer by Alexandre GUIDET, I attempted to run the following query:
create table egg (id (SELECT
pg_catalog.format_type(a.atttypid, a.atttypmod) as Datatype
FROM
pg_catalog.pg_attribute a
WHERE
a.attnum > 0
AND NOT a.attisdropped
AND a.attrelid = (
SELECT c.oid
FROM pg_catalog.pg_class c
LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relname ~ '^(TABLENAME)$'
AND pg_catalog.pg_table_is_visible(c.oid)
)
and a.attname = 'COLUMNNAME'));
PostgreSQL, however, complains about incorrect syntax. Specifically it says that I cannot write: create table egg (id (SELECT.
Are there any workarounds? Can't I convert the result of a query to text and reuse it as a query?
There is a much simpler way to do that.
SELECT pg_typeof(col)::text FROM tbl LIMIT 1
Only precondition is that the template table holds at least one row. See the manual on pg_typeof()
As Milen wrote, you need to EXECUTE dynamic DDL statements like this.
A much simpler DO statement:
DO $$BEGIN
EXECUTE 'CREATE TABLE egg (id '
|| (SELECT pg_typeof(col)::text FROM tbl LIMIT 1) || ')';
END$$;
Or, if you are not sure the template table has any rows:
DO $$BEGIN
EXECUTE (
SELECT format('CREATE TABLE egg (id %s)'
, format_type(atttypid, atttypmod))
FROM pg_catalog.pg_attribute
WHERE attrelid = 'tbl'::regclass -- name of template table
AND attname = 'col' -- name of template column
AND attnum > 0 AND NOT attisdropped
);
END$$;
These conditions seem redundant, since you look for a specific column any
format() requires Postgres 9.1+.
Related:
How to check if a table exists in a given schema
You can either convert that query to a function or (if you have Postgres 9.0) to an anonymous code block:
DO $$DECLARE the_type text;
BEGIN
SELECT ... AS datatype INTO the_type FROM <the rest of your query>;
EXECUTE 'create table egg ( id ' || the_type || <the rest of your create table statement>;
END$$;
You can either have a table a definition or a query, but not both. Maybe your thinking of the select into command.