I am trying to alter table based on another table dynamically.
Below is the piece of code i wrote in postgresql stored procedure.
But running out into some syntax errors.
Please help me here.
I just started working in postgresql and i am from sql server background. Like how we do in sql server print stmts do debug dynamic queries inside procedures; do we have any link to refer please share that as well. It would help me.
DROP TABLE IF EXISTS temp_table;
CREATE TEMP TABLE temp_table AS
with cte as
(
select column_name,data_type,character_maximum_length
from information_schema."columns" c
where table_name = 'customer_new' and table_schema = 'public'
and column_default is null
)
,cte1 as
(
select column_name,data_type,character_maximum_length
from information_schema."columns" c
where table_name = 'customer_old' and table_schema = 'public'
and column_default is null
)
select cte.column_name,
case when cte.character_maximum_length is not null then cte.data_type||'('||cte.character_maximum_length||')' else cte.data_type end as data_type
from cte
left join cte1 on cte.column_name = cte1.column_name
where cte1.column_name is null;
for v_column_name,v_data_type in SELECT column_name,data_type FROM temp_table
loop
execute format ('alter table %s add column %s %s ;', v_dump_table_name, v_column_name, v_data_type);
end loop;
DROP TABLE IF EXISTS temp_table;
Thanks in advance.
DROP TABLE IF EXISTS temp_table;
CREATE TEMP TABLE temp_table AS
with cte as
(
select column_name,data_type,character_maximum_length
from information_schema."columns" c
where table_name = 'customer_new' and table_schema = 'public'
and column_default is null
)
,cte1 as
(
select column_name,data_type,character_maximum_length
from information_schema."columns" c
where table_name = 'customer_old' and table_schema = 'public'
and column_default is null
)
select 'alter table '||v_dump_table_name||' add column '||cte2.column_name||' '||data_type
as col
from cte2;
for v_column in SELECT col FROM temp_table
loop
execute format ('%s ;', v_column);
end loop;
DROP TABLE IF EXISTS temp_table;
Related
I have several tables in my PostgreSQL database's Public Schema. The tables are named "projects_2019", "projects_2020", "projects_2021", etc. and have the same columns. The idea is that a new table will be added every year.
I would like to select all records in all of the tables whose name includes "projects_", how could I do this without naming each and every table name (since I don't know how many there will be in the future)?
Here's what I have so far:
WITH t as
(SELECT * FROM information_schema.tables WHERE table_schema = 'public' and table_name ~ 'projects_')
SELECT * FROM t
You can do it using dynamic SQL and information_schema. For Example:
-- Sample Data
CREATE TABLE table1 (
id int4 NULL,
caption text NULL
);
CREATE TABLE table2 (
id int4 NULL,
caption text NULL
);
CREATE TABLE table3 (
id int4 NULL,
caption text NULL
);
CREATE TABLE table4 (
id int4 NULL,
caption text NULL
);
INSERT INTO table1 (id, caption) VALUES (1, 'text1');
INSERT INTO table2 (id, caption) VALUES (2, 'text2');
INSERT INTO table3 (id, caption) VALUES (3, 'text3');
INSERT INTO table4 (id, caption) VALUES (4, 'text4');
-- create function sample:
CREATE OR REPLACE FUNCTION select_tables()
RETURNS table(id integer, caption text)
LANGUAGE plpgsql
AS $function$
declare
v_sql text;
v_union text;
begin
SELECT string_agg('select * from ' || table_schema || '.' || table_name, ' union all ')
into v_sql
FROM information_schema.tables WHERE table_schema = 'public' and table_name ~ 'table';
return query
execute v_sql;
end ;
$function$
;
-- selecting data:
select * from select_tables()
-- Result:
id caption
1 text1
2 text2
3 text3
4 text4
You can try like this:
FOR i IN SELECT table_name
FROM information_schema.tables
WHERE table_schema = 'public'
and table_name ~ 'projects_'
LOOP
sqlstr := sqlstr || format($$
UNION
SELECT name FROM %I
$$,
i.table_name);
END LOOP;
EXECUTE sqlstr;
I'm looking for a way to get the list of all json attributes across all my PostgreSQL tables dynamically.
I have Query 1 which would generate a list of sql statements, and then run that sql statements to get the final output all in one go (like the dynamic SQL concept in SQL server).
Query 1 looks like this :
create temporary table test (ordr int, field varchar(1000));
-- Step 1 Create temp table to insert all table/col/json attrbute info
insert into test(ordr,field)
select 0 ordr,'create temporary table temp_table
( table_schema varchar(200)
,table_name varchar(200)
,column_name varchar(200)
,json_attribute varchar(200)
,data_type varchar(50)
);'
union
-- Non json type columns
select 1 ordr, 'insert into temp_table(table_name, column_name,data_type,json_attribute)'
union
-- Json columns with data like json object
select
3 ordr,
concat('select distinct ''', t.table_name, ''' tbl, ''', c.column_name, ''' col, ''' , c.data_type,''' data_type, '
,'jsonb_object_keys(', c.column_name, ') json_attribute', ' from ', t.table_name,
' where jsonb_typeof(' , c.column_name, ') = ''object'' union') AS field
from information_schema.tables t
join information_schema.columns c on c.table_name = t.table_name
where t.table_schema not in ('information_schema', 'pg_catalog')
--and table_type = 'BASE TABLE'
and c.data_type ='jsonb';
--final sql statements to build temp table
--copy all the column "txt" to a separate window and execute it, it will create a temp table "temp_table" which will have all tables/cols/json attributes
select ordr
,(case when t.ordr = (select max(t2.ordr) from test t2) then replace(field,'union','') else field end) txt
from test t
union
select 9999, ';select * from temp_table;'
order by 1 ;
Query 1 output : This is a list of sql statements
I'm looking for a way to run the Query 1 & Query 1 output which would get me the final output all in one go.
Any lead or guidance will be really appreciated.
We have a static database we constantly update with loader scripts. These loader scripts get current information from third party sources, clean it and upload it to database.
I have already made some SQL scripts to ensure schemas and tables required exists. Now I'd like to check that each table has the expected row count.
I did something like this:
select case when count(*) = <someNumber>
then 'someSchema.someTable OK'
else 'someSchema.someTable BAD row count' end
from someSchema.someTable;
But doing these kind of queries for ~300 tables is cumbersome.
Now I was thinking maybe there's a way to have a table like:
create table expected_row_count (
schema_name varchar,
table_name varchar,
row_count bigint
);
And somehow test all listed tables and only output the ones that fail the count check. But I'm kind of missing now... Should I try to write a function? Can a table like this be used to build queries and execute them?
Whole credit goes to #a-horse_with*_no_name , I'm posting a reply for completeness:
Check row count
First let's create some data to test the query:
create schema if not exists data;
create table if not exists data.test1 (nothing int);
create table if not exists data.test2 (nothing int);
insert into data.test1 (nothing)
(select random() from generate_series(1, 28));
insert into data.test2 (nothing)
(select random() from generate_series(1, 55));
create table if not exists public.expected_row_count (
table_schema varchar not null default '',
table_name varchar not null default '',
row_count bigint not null default 0
);
insert into public.expected_row_count (table_schema, table_name, row_count) values
('data', 'test1', (select count(*) from data.test1)),
('data', 'test2', (select count(*) from data.test2))
;
Now the query to check the data:
select * from (
select
table_schema,
table_name,
(xpath('/row/cnt/text()', xml_count))[1]::text::int as row_count
from (
select
table_schema,
table_name,
query_to_xml(format('select count(*) as cnt from %I.%I', table_schema, table_name), false, true, '') as xml_count
from information_schema.tables
where table_schema = 'data' --<< change here for the schema you want
) infs ) as r
inner join expected_row_count erc
on r.table_schema = erc.table_schema
and r.table_name = erc.table_name
and r.row_count != erc.row_count
;
Previous query should give an empty results if all counts are ok, and the
tables with missing data if not. To check it, update the count for some
table on expected_row_count and re-run the query. For example:
update expected_row_count set row_count = 666 where table_name = 'test1';
This is how we can check table existence in MSSQL:
IF OBJECT_ID(N'public."TABLE_NAME"', N'U') IS NOT NULL
select 1 as 'column'
else
select 0 as 'column';
which stores outcome in variable 'column'
How can I do same thing in PostgreSQL ? I want to return 1 or 0 for respective outcome.
Use a SELECT with an EXISTS operator checking e.g. information_schema.tables:
select exists (select *
from information_schema.tables
where table_name = 'table_name'
and table_schema = 'public') as table_exists;
If you can't (or won't) deal with proper boolean values, the simply cast the result to a number (but I have no idea why that should be better):
select exists (select *
from information_schema.tables
where table_name = 'table_name'
and table_schema = 'public')::int as "column";
Note that column is a reserved keyword and thus you need to quote it using double quotes.
Check for column in a table existence use view pg_tables
IF EXISTS ( SELECT attname
FROM pg_attribute
WHERE attrelid = (SELECT oid FROM pg_class WHERE relname = 'YOURTABLENAME')
AND attname = 'YOURCOLUMNNAME')
THEN
-- do something
END IF;
For my sql use INFORMATION_SCHEMA.COLUMNS
SELECT 1
FROM INFORMATION_SCHEMA.COLUMNS
WHERE table_name = 'tbl_name'
[AND table_schema = 'db_name']
[AND column_name LIKE 'wild']
I am trying to write sub-queries so that I search all tables for a column named id and since there are multiple tables with id column, I want to add the condition, so that id = 3119093.
My attempt was:
Select *
from information_schema.tables
where id = '3119093' and id IN (
Select table_name
from information_schema.columns
where column_name = 'id' );
This didn't work so I tried:
Select *
from information_schema.tables
where table_name IN (
Select table_name
from information_schema.columns
where column_name = 'id' and 'id' IN (
Select * from table_name where 'id' = 3119093));
This isn't the right way either. Any help would be appreciated. Thanks!
A harder attempt is:
CREATE OR REPLACE FUNCTION search_columns(
needle text,
haystack_tables name[] default '{}',
haystack_schema name[] default '{public}'
)
RETURNS table(schemaname text, tablename text, columnname text, rowctid text)
AS $$
begin
FOR schemaname,tablename,columnname IN
SELECT c.table_schema,c.table_name,c.column_name
FROM information_schema.columns c
JOIN information_schema.tables t ON
(t.table_name=c.table_name AND t.table_schema=c.table_schema)
WHERE (c.table_name=ANY(haystack_tables) OR haystack_tables='{}')
AND c.table_schema=ANY(haystack_schema)
AND t.table_type='BASE TABLE'
--AND c.column_name = "id"
LOOP
EXECUTE format('SELECT ctid FROM %I.%I WHERE cast(%I as text) like %L',
schemaname,
tablename,
columnname,
needle
) INTO rowctid;
IF rowctid is not null THEN
RETURN NEXT;
END IF;
END LOOP;
END;
$$ language plpgsql;
select * from search_columns('%3119093%'::varchar,'{}'::name[]) ;
The only problem is this code displays the table name and column name. I have to then manually enter
Select * from table_name where id = 3119093
where I got the table name from the code above.
I want to automatically implement returning rows from a table but I don't know how to get the table name automatically.
I took the time to make it work for you.
For starters, some information on what is going on inside the code.
Explanation
function takes two input arguments: column name and column value
it requires a created type that it will be returning a set of
first loop identifies tables that have a column name specified as the input argument
then it forms a query which aggregates all rows that match the input condition inside every table taken from step 3 with comparison based on ILIKE - as per your example
function goes into the second loop only if there is at least one row in currently visited table that matches specified condition (then the array is not null)
second loop unnests the array of rows that match the condition and for every element it puts it in the function output with RETURN NEXT rec clause
Notes
Searching with LIKE is inefficient - I suggest adding another input argument "column type" and restrict it in the lookup by adding a join to pg_catalog.pg_type table.
The second loop is there so that if more than 1 row is found for a particular table, then every row gets returned.
If you are looking for something else, like you need key-value pairs, not just the values, then you need to extend the function. You could for example build json format from rows.
Now, to the code.
Test case
CREATE TABLE tbl1 (col1 int, id int); -- does contain values
CREATE TABLE tbl2 (col1 int, col2 int); -- doesn't contain column "id"
CREATE TABLE tbl3 (id int, col5 int); -- doesn't contain values
INSERT INTO tbl1 (col1, id)
VALUES (1, 5), (1, 33), (1, 25);
Table stores data:
postgres=# select * From tbl1;
col1 | id
------+----
1 | 5
1 | 33
1 | 25
(3 rows)
Creating type
CREATE TYPE sometype AS ( schemaname text, tablename text, colname text, entirerow text );
Function code
CREATE OR REPLACE FUNCTION search_tables_for_column (
v_column_name text
, v_column_value text
)
RETURNS SETOF sometype
LANGUAGE plpgsql
STABLE
AS
$$
DECLARE
rec sometype%rowtype;
v_row_array text[];
rec2 record;
arr_el text;
BEGIN
FOR rec IN
SELECT
nam.nspname AS schemaname
, cls.relname AS tablename
, att.attname AS colname
, null::text AS entirerow
FROM
pg_attribute att
JOIN pg_class cls ON att.attrelid = cls.oid
JOIN pg_namespace nam ON cls.relnamespace = nam.oid
WHERE
cls.relkind = 'r'
AND att.attname = v_column_name
LOOP
EXECUTE format('SELECT ARRAY_AGG(row(tablename.*)::text) FROM %I.%I AS tablename WHERE %I::text ILIKE %s',
rec.schemaname, rec.tablename, rec.colname, quote_literal(concat('%',v_column_value,'%'))) INTO v_row_array;
IF v_row_array is not null THEN
FOR rec2 IN
SELECT unnest(v_row_array) AS one_row
LOOP
rec.entirerow := rec2.one_row;
RETURN NEXT rec;
END LOOP;
END IF;
END LOOP;
END
$$;
Exemplary call & output
postgres=# select * from search_tables_for_column('id','5');
schemaname | tablename | colname | entirerow
------------+-----------+---------+-----------
public | tbl1 | id | (1,5)
public | tbl1 | id | (1,25)
(2 rows)