How to dynamically copy tables from information schema inside a trigger function - postgresql

I have an insert trigger function in which NEW.schema_name references a schema. I want to dynamically copy the tables found inside that schema ('foobaz','barbaz') as 'foo' and 'bar'. I then can perform queries without dynamic sql.
How can I create a function or simply copy/paste the same block of code to achive that.
EDIT :
I cannot get that dynamic query to work.
The part inside the WITH statement is working.
Not the bottom 'execute' part. I do not know if it is a syntax problem, or bad cast or whatever constraint there is in pgsql that makes it not working.
WITH info_schema_subset_table as (SELECT table_schema, table_name,
array_to_string((regexp_split_to_array(table_name,'_'))[4:array_length(regexp_split_to_array(table_name,'_'),1)-1] as new_table
FROM information_schema.tables
where table_schema = "schema_searched"
ORDER BY new_table ASC)
EXECUTE 'CREATE TABLE $2 as (SELECT * FROM $1)'
USING info_schema_subset_table.table_schema || '.' ||info_schema_subset_table.table_name,info_schema_subset_table.new_table;
EDIT 2
... Broken code removed...
In the code below, in which I'm unsure if the syntax is right, I get the following from the trigger
Provider errors:
PostGIS error while adding features: ERREUR: l'opérateur n'existe pas : record ~~ unknown
LINE 1: SELECT old_table LIKE '%ens%'
^
HINT: Aucun opérateur ne correspond au nom donné et aux types d'arguments.
Vous devez ajouter des conversions explicites de type.
QUERY: SELECT old_table LIKE '%ens%'
CONTEXT: fonction PL/pgsql validation_sio.afi_validation_sio(), ligne 18 à CASE
EDIT 3 :
CREATE OR REPLACE FUNCTION foo.foo()
RETURNS TRIGGER AS
$BODY$
DECLARE
old_table record;
new_table record;
dynamic_query text;
BEGIN
IF TG_OP = 'INSERT'
THEN
FOR old_table IN SELECT table_schema|| '.' ||table_name
FROM information_schema.tables
where table_schema = NEW.nom_schema
LOOP
CASE
WHEN
old_table LIKE '%ens%' THEN
new_table := concat('SIT_',array_to_string((regexp_split_to_array(info_schema.old_table,'_'))[4:array_length(regexp_split_to_array(info_schema.old_table,'_'),1)-1],'_'));
ELSE
new_table := concat('SID_',array_to_string((regexp_split_to_array(info_schema.old_table,'_'))[4:array_length(regexp_split_to_array(info_schema.old_table,'_'),1)-1],'_'));
END CASE;
dynamic_query := format('SELECT * FROM' || old_table ||);
EXECUTE dynamic_query
INTO new_table;
END LOOP;
RETURN NEW;
END IF;
END;
$BODY$
LANGUAGE plpgsql VOLATILE;
CREATE TRIGGER foo
AFTER INSERT ON validation.validationfoo
FOR EACH ROW EXECUTE PROCEDURE foo.foo();

I've reformatted your trigger function a bit and changed a few things, see if this works.
CREATE OR REPLACE FUNCTION foo.foo()
RETURNS TRIGGER AS
$BODY$
DECLARE
old_table record;
new_table record;
dynamic_query text;
BEGIN
IF TG_OP = 'INSERT' THEN
FOR old_table IN
SELECT table_schema || '.' || table_name AS old_table_name
FROM information_schema.tables
WHERE table_schema = NEW.nom_schema
LOOP
new_table := concat(CASE WHEN old_table.old_table_name LIKE '%ens%' THEN 'SIT_' ELSE 'SID_' END,array_to_string((regexp_split_to_array(info_schema.old_table,'_'))[4:array_length(regexp_split_to_array(info_schema.old_table,'_'),1)-1],'_'));
dynamic_query := 'CREATE TABLE ' || new_table || ' AS SELECT * FROM ' || old_table.old_table_name;
EXECUTE dynamic_query;
END LOOP;
RETURN NEW;
END IF;
END;
$BODY$
LANGUAGE plpgsql VOLATILE;
So the main things:
old_table is a record, so your comparison of it to a string with LIKE was failing. You need to use the field name. So I gave your field a name, and used that field name in the LIKE comparison.
Changed the new_table assignment to put the CASE statement only on the one item that changes, to make the difference more obvious and the code more concise. Mind you, I don't know if the rest of that line is actually valid, I just left it as is.
Changed the creation dynamic_query. As I said in the comment, the format function was being used incorrectly, so I just went with standard string concatenation instead.
Changed dynamic_query's SQL to what I think you actually want it to do. You want it to copy the content of the table to a new table, right? So that will do it.

You cannot have EXECUTE inside an SQL statement, it is a PL/pgSQL statement.
Loop through the tables and issue one EXECUTE for each.
Mind that you cannot have a schema or table name as a parameter with USING, because these names need to be known at parse time.
Use the format function to construct your dynamic statement so you can avoid SQL injection by users who maliciously create tables with weird names.

Related

postgres function cannot see variable

I have function which supposed to create missing objects. I have a variable that I am using to set active schema tschem_name
However feels like proc cannot see this variable if it is trying to create this object, however, if which check if object exists can see this variable feels like it treats it as a column from the table. Any idea how to work around it? I was trying to use declare inside else statement but it doesn't work :(
ERROR: column "tschem_name" does not exist
CREATE OR REPLACE FUNCTION public.fn_upd_schema(
tschem_name character varying
)
RETURNS character varying
LANGUAGE 'plpgsql'
COST 100
VOLATILE PARALLEL UNSAFE
AS $BODY$
DECLARE
--tname text := t_shema;
--tschem_name text := tname||'_assets';
tsql_dyn text ;
nschema_check numeric := 0 ;
BEGIN
SELECT 1
INTO nschema_check
FROM PG_NAMESPACE
WHERE NSPNAME = tschem_name;
IF nschema_check = 0 THEN
RETURN 'Schema '||tschem_name ||' Does Not Exist';
ELSE
tsql_dyn := 'SET search_path TO '||tschem_name ||';';
raise notice 'EXECUTE %', tsql_dyn;
EXECUTE tsql_dyn;
IF EXISTS (SELECT FROM information_schema.tables
WHERE table_schema = tschem_name
AND table_type = 'VIEW'
AND table_name = 'vw_aaa')
THEN
null;-- raise notice 'vw_aaa EXISTS';
ELSE
--declare tschemaname text := tschem_name;
CREATE VIEW vw_aaa AS
SELECT
tschem_name::text AS database,
tale1.column1,
tale1.column2
From table1;
endif;
RETURN tschem_name ||' UPDATED TO STANDART';
END IF;
END;
$BODY$;
The problem is your CREATE VIEW statement. You need to use dynamic SQL with EXECUTE for that.
The underlying reason is that CREATE VIEW does not accept parameters.
By the way, your code is vulnerable to SQL injection. Never construct an SQL statement by concatenating strings with ||. Use format with the %I placeholder.

postgresql for loop script in text form can not be executed

I am trying to write function in postgresql, that creates temp_table with columns table_name text, table_rec jsonb and fill it through for loop with table names from my table containing names of tables and records in json. I have the for loop in string and I want to execute it. But it doesnt work.
I have variable rec record, sql_query text and tab_name text and I want to do this:
CREATE OR REPLACE FUNCTION public.test51(
)
RETURNS TABLE(tabel_name text, record_json jsonb)
LANGUAGE 'plpgsql'
COST 100
VOLATILE
ROWS 1000
AS $BODY$
declare
rec record;
tabel_name text;
tabel_names text[];
counter integer := 1;
sql_query text;
limit_for_sending integer;
rec_count integer;
begin
select into tabel_names array(select "TABLE_NAME" from public."TABLES");
create temp table temp_tab(tab_nam text, recik jsonb);
while array_length(tabel_names, 1) >= counter loop
tabel_name := '"' || tabel_names[counter] || '"';
select into limit_for_sending "TABLE_LIMIT_FOR_SENDING_DATA" from public."TABLES" where "TABLE_NAME" = tabel_name;
sql_query := 'select count(*) from public.' || tabel_name;
execute sql_query into rec_count;
if (rec_count >= limit_for_sending and limit_for_sending is not null) then
sql_query := 'for rec in select * from public.' || tabel_name || '
loop
insert into temp_tab
select ' || tabel_name || ', to_jsonb(rec);
end loop';
execute sql_query;
end if;
counter := counter + 1;
end loop;
return query
select * from temp_tabik;
drop table temp_tabik;
end;
$BODY$;
Thank you for response.
It seems you have some table that contains the information for which tables you want to return all rows as JSONB. And that meta-table also contains a column that sets a threshold under which the rows should not be returned.
You don't need the temp table or an array to store the table names. You can iterate through the query on the TABLES table and run the dynamic SQL directly in that loop.
return query in PL/pgSQL doesn't terminate the function, it just appends the result of the query to the result of the function.
Dynamic SQL is best created using the format() function because it is easier to read and using the %I placeholder will properly deal with quoted identifiers (which is really important as you are using those dreaded upper case table names)
As far as I can tell, your function can be simplified to:
CREATE OR REPLACE FUNCTION public.test51()
RETURNS TABLE(tabel_name text, record_json jsonb)
LANGUAGE plpgsql
AS
$BODY$
declare
rec record;
sql_query text;
rec_count bigint;
begin
for rec in
select "TABLE_NAME" as table_name, "TABLE_LIMIT_FOR_SENDING_DATA" as rec_limit
from public."TABLES"
loop
if rec.rec_limit is not null then
execute format('select count(*) from %I', rec.table_name)
into rec_count;
end if;
if (rec.rec_limit is not null and rec_count >= rec.rec_limit) then
sql_query := format('select %L, to_jsonb(t) from %I as t', rec.table_name, rec.table_name);
return query execute sql_query;
end if;
end loop;
end;
$BODY$;
Some notes
the language name is an identifier and should not be enclosed in single quotes. This syntax is deprecated and might be removed in a future version so don't get used to it.
you should really avoid those dreaded quoted identifiers. They are much more trouble than they are worth it. See the Postgres wiki for details.

SAS to PostgreSQL(PADB) code - summing field if they exists

I'm having a challenge with a piece of code from SAS that I need to convert to SQL.
Usually I'm very good at this but right not I'm facing a new challenge and so far all my ideas to resolve it are failing and I'm not finding the right way to do so.
I need to be able to pick up field dynamically for this request, like if a field has a certain pattern in it's name I need to sum those fields.
my version of PostgreSQL is 8.0.2, PADB 5.3.3.1 78560
So the table may or may not have a field like bas_txn_03cibc_vcl.
I wrote a function that should output ' ' as bas_txn_03cibc_vcl when the field is not found in the information_schema table and use bas_txn_03cibc_vcl if found.
But when I execute the command I get the error that UDF cannot be used on PADB tables.
"ERROR: XX000: User-defined SQL language function "check_if_field_exists(character varying,character varying,character varying)" cannot be used in a query that references PADB tables."
Right now I'm building a new approach using stored procedure but it will limit the use case. Any other idea on how I can select field dynamically?
Function:
CREATE OR REPLACE FUNCTION check_if_field_exists(_schm text, _tbl text, _field text)
RETURNS text AS
$BODY$
DECLARE
_output_ text:= '' as _field;
BEGIN
EXECUTE 'SELECT column_name into : _output_ FROM rdwaeprd.information_schema.columns
where table_schema='''|| _schm||'''
and table_name='''|| _tbl||'''
and column_name='''|| _field||'''
order by table_name,column_name;';
RETURN _output_;
END
$BODY$
LANGUAGE PLPGSQL;
and then I would use it like this
select indiv_id,ae_psamson.check_if_field_exists('ae_psamson','activ_cc', 'tot_txn_03AMX_AMXE') ,tot_txn_03AMX_AMXD
from activ_cc
group by indiv_id,tot_txn_03AMX_AMXD;
Where the function would either return '' as tot_txn_03AMX_AMXE or simply, tot_txn_03AMX_AMXE.... the idea is to make the query not return an error if the field does not exists.
Like I said I need a new function or approach as this one is not working...
I managed to make a function that make it work!
Basically one of the issue what that information schema was using unsupported function in UDF.
This solution works fine:
CREATE OR REPLACE FUNCTION check_if_field_exists(_schm text, _tbl text, _field text)
RETURNS varchar(55) AS
$BODY$
DECLARE
_output_ varchar(55) :=' 0 as '|| _field;
-- name := (SELECT t.name from test_table t where t.id = x);
BEGIN
EXECUTE 'drop table if exists col_name';
EXECUTE 'create table col_name as SELECT att.attname::character varying(128) AS colname
FROM pg_class cl, pg_namespace ns, pg_attribute att
WHERE cl.relnamespace = ns.oid AND cl.oid = att.attrelid AND ns.nspname='''|| _schm ||'''
and cl.relname='''|| _tbl ||'''
and colname like '''|| _field||''''; -- INTO _output_;
select colname from col_name into _output_ ;
if _output_ is null then
_output_ :=' 0 as '|| _field;
end if;
RETURN _output_ ;
END
$BODY$
LANGUAGE PLPGSQL;

SELECTing commands into a temp table to EXECUTE later in PostgreSQL

For some fancy database maintenance for my developer database I'd like to be able to use queries to generate commands to alter the database. The thing is: I'm a complete greenhorn to PostgreSQL. I've made my attempt but have failed colorfully.
So in the end, I would like to have a table with a single column and each row would be a command (or group of commands, depending on the case) that I would think would look something like this...
DO $$
DECLARE
command_entry RECORD;
BEGIN
FOR command_entry IN SELECT * FROM list_of_commands
LOOP
EXECUTE command_entry;
END LOOP;
END;
$$;
Where the table list_of_commands could be populated with something like the following (which in this example would remove all tables from the public schema)...
CREATE TEMP TABLE list_of_commands AS
SELECT 'drop table if exists "' || tablename || '" cascade;'
FROM pg_tables
WHERE schemaname = 'public';
However, with this I get the following error...
ERROR: syntax error at or near ""drop table if exists ""dummy_table"" cascade;""
LINE 1: ("drop table if exists ""dummy_table"" cascade;")
I assume this is a matter of escaping characters, but I'm not entirely sure how to fit that into either A) the population of the table or B) the execution of each row. Does anyone know what I could do to achieve the desired result?
The command_entry variable is of type record while the EXECUTE command expects a string. What is apparently happening is that PostgreSQL turns the record into a double-quoted string, but that messes up your command. Also, your temp table does not use a column name, making things a bit awkward to work with (the column name becomes ?column?), so change both as follows:
CREATE TEMP TABLE list_of_commands AS
SELECT 'drop table if exists public.' || quote_ident(tablename) || ' cascade' AS cmd
FROM pg_tables
WHERE schemaname = 'public';
DO $$
DECLARE
command_entry varchar;
BEGIN
FOR command_entry IN SELECT cmd FROM list_of_commands
LOOP
EXECUTE command_entry;
END LOOP;
END;
$$;
But seeing that you do all of this at session level (temp table, anonymous code block), why not write a stored procedure that performs all of this housekeeping when you are ready to do spring cleaning?
CREATE FUNCTION cleanup() RETURNS void AS $$
BEGIN
FOR tbl IN SELECT tablename FROM pg_tables WHERE schemaname = 'public'
LOOP
EXECUTE 'DROP TABLE IF EXISTS ' || quote_ident(tbl) || ' CASCADE';
END LOOP;
-- More housekeeping jobs
END;
$$ LANGUAGE plpgsql;
This saves a lot of typing: SELECT cleanup();. Any other housekeeping jobs you have you simply add to the stored procedure.
I had trouble with Patrick's answers, so here is an updated version for postgreSQL 10.
CREATE FUNCTION droptables(sn varchar) RETURNS void AS $$
DECLARE
tbl varchar;
BEGIN
FOR tbl IN SELECT tablename FROM pg_tables WHERE schemaname = sn
LOOP
EXECUTE 'DROP TABLE IF EXISTS ' || quote_ident(tbl) || ' CASCADE';
END LOOP;
END;
$$ LANGUAGE plpgsql;
And then "SELECT droptables('public');".

How to add column if not exists on PostgreSQL?

Question is simple. How to add column x to table y, but only when x column doesn't exist ? I found only solution here how to check if column exists.
SELECT column_name
FROM information_schema.columns
WHERE table_name='x' and column_name='y';
With Postgres 9.6 this can be done using the option if not exists
ALTER TABLE table_name ADD COLUMN IF NOT EXISTS column_name INTEGER;
Here's a short-and-sweet version using the "DO" statement:
DO $$
BEGIN
BEGIN
ALTER TABLE <table_name> ADD COLUMN <column_name> <column_type>;
EXCEPTION
WHEN duplicate_column THEN RAISE NOTICE 'column <column_name> already exists in <table_name>.';
END;
END;
$$
You can't pass these as parameters, you'll need to do variable substitution in the string on the client side, but this is a self contained query that only emits a message if the column already exists, adds if it doesn't and will continue to fail on other errors (like an invalid data type).
I don't recommend doing ANY of these methods if these are random strings coming from external sources. No matter what method you use (client-side or server-side dynamic strings executed as queries), it would be a recipe for disaster as it opens you to SQL injection attacks.
Postgres 9.6 added ALTER TABLE tbl ADD COLUMN IF NOT EXISTS column_name.
So this is mostly outdated now. You might use it in older versions, or a variation to check for more than just the column name.
CREATE OR REPLACE function f_add_col(_tbl regclass, _col text, _type regtype)
RETURNS bool
LANGUAGE plpgsql AS
$func$
BEGIN
IF EXISTS (SELECT FROM pg_attribute
WHERE attrelid = _tbl
AND attname = _col
AND NOT attisdropped) THEN
RETURN false;
ELSE
EXECUTE format('ALTER TABLE %s ADD COLUMN %I %s', _tbl, _col, _type);
RETURN true;
END IF;
END
$func$;
Call:
SELECT f_add_col('public.kat', 'pfad1', 'int');
Returns true on success, else false (column already exists).
Raises an exception for invalid table or type name.
Why another version?
This could be done with a DO statement, but DO statements cannot return anything. And if it's for repeated use, I would create a function.
I use the object identifier types regclass and regtype for _tbl and _type which a) prevents SQL injection and b) checks validity of both immediately (cheapest possible way). The column name _col has still to be sanitized for EXECUTE with quote_ident(). See:
Table name as a PostgreSQL function parameter
format() requires Postgres 9.1+. For older versions concatenate manually:
EXECUTE 'ALTER TABLE ' || _tbl || ' ADD COLUMN ' || quote_ident(_col) || ' ' || _type;
You can schema-qualify your table name, but you don't have to.
You can double-quote the identifiers in the function call to preserve camel-case and reserved words (but you shouldn't use any of this anyway).
I query pg_catalog instead of the information_schema. Detailed explanation:
How to check if a table exists in a given schema
Blocks containing an EXCEPTION clause are substantially slower.
This is simpler and faster. The manual:
Tip
A block containing an EXCEPTION clause is significantly more
expensive to enter and exit than a block without one.
Therefore, don't use EXCEPTION without need.
Following select query will return true/false, using EXISTS() function.
EXISTS(): The argument of EXISTS is an arbitrary SELECT statement, or
subquery. The subquery is evaluated to determine whether it returns
any rows. If it returns at least one row, the result of EXISTS is
"true"; if the subquery returns no rows, the result of EXISTS is
"false"
SELECT EXISTS(SELECT column_name
FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = 'x'
AND column_name = 'y');
and use the following dynamic SQL statement to alter your table
DO
$$
BEGIN
IF NOT EXISTS (SELECT column_name
FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = 'x'
AND column_name = 'y') THEN
ALTER TABLE x ADD COLUMN y int DEFAULT NULL;
ELSE
RAISE NOTICE 'Already exists';
END IF;
END
$$
For those who use Postgre 9.5+(I believe most of you do), there is a quite simple and clean solution
ALTER TABLE if exists <tablename> add if not exists <columnname> <columntype>
the below function will check the column if exist return appropriate message else it will add the column to the table.
create or replace function addcol(schemaname varchar, tablename varchar, colname varchar, coltype varchar)
returns varchar
language 'plpgsql'
as
$$
declare
col_name varchar ;
begin
execute 'select column_name from information_schema.columns where table_schema = ' ||
quote_literal(schemaname)||' and table_name='|| quote_literal(tablename) || ' and column_name= '|| quote_literal(colname)
into col_name ;
raise info ' the val : % ', col_name;
if(col_name is null ) then
col_name := colname;
execute 'alter table ' ||schemaname|| '.'|| tablename || ' add column '|| colname || ' ' || coltype;
else
col_name := colname ||' Already exist';
end if;
return col_name;
end;
$$
This is basically the solution from sola, but just cleaned up a bit. It's different enough that I didn't just want to "improve" his solution (plus, I sort of think that's rude).
Main difference is that it uses the EXECUTE format. Which I think is a bit cleaner, but I believe means that you must be on PostgresSQL 9.1 or newer.
This has been tested on 9.1 and works. Note: It will raise an error if the schema/table_name/or data_type are invalid. That could "fixed", but might be the correct behavior in many cases.
CREATE OR REPLACE FUNCTION add_column(schema_name TEXT, table_name TEXT,
column_name TEXT, data_type TEXT)
RETURNS BOOLEAN
AS
$BODY$
DECLARE
_tmp text;
BEGIN
EXECUTE format('SELECT COLUMN_NAME FROM information_schema.columns WHERE
table_schema=%L
AND table_name=%L
AND column_name=%L', schema_name, table_name, column_name)
INTO _tmp;
IF _tmp IS NOT NULL THEN
RAISE NOTICE 'Column % already exists in %.%', column_name, schema_name, table_name;
RETURN FALSE;
END IF;
EXECUTE format('ALTER TABLE %I.%I ADD COLUMN %I %s;', schema_name, table_name, column_name, data_type);
RAISE NOTICE 'Column % added to %.%', column_name, schema_name, table_name;
RETURN TRUE;
END;
$BODY$
LANGUAGE 'plpgsql';
usage:
select add_column('public', 'foo', 'bar', 'varchar(30)');
Can be added to migration scripts invoke function and drop when done.
create or replace function patch_column() returns void as
$$
begin
if exists (
select * from information_schema.columns
where table_name='my_table'
and column_name='missing_col'
)
then
raise notice 'missing_col already exists';
else
alter table my_table
add column missing_col varchar;
end if;
end;
$$ language plpgsql;
select patch_column();
drop function if exists patch_column();
In my case, for how it was created reason it is a bit difficult for our migration scripts to cut across different schemas.
To work around this we used an exception that just caught and ignored the error. This also had the nice side effect of being a lot easier to look at.
However, be wary that the other solutions have their own advantages that probably outweigh this solution:
DO $$
BEGIN
BEGIN
ALTER TABLE IF EXISTS bobby_tables RENAME COLUMN "dckx" TO "xkcd";
EXCEPTION
WHEN undefined_column THEN RAISE NOTICE 'Column was already renamed';
END;
END $$;
You can do it by following way.
ALTER TABLE tableName drop column if exists columnName;
ALTER TABLE tableName ADD COLUMN columnName character varying(8);
So it will drop the column if it is already exists. And then add the column to particular table.
Simply check if the query returned a column_name.
If not, execute something like this:
ALTER TABLE x ADD COLUMN y int;
Where you put something useful for 'x' and 'y' and of course a suitable datatype where I used int.