How to use variable as table name in plpgsql - postgresql

I'm new to plpgsql. I'm trying to run a simple query in plpgsql using a variable as table name in plpgsql. But the variable is being interpreted as the table name instead of the value of the variable being interpreted as variable name.
DECLARE
v_table text;
z_table text;
max_id bigint;
BEGIN
FOR v_table IN
SELECT table_name
FROM information_schema.tables
WHERE table_catalog = 'my_database'
AND table_schema = 'public'
AND table_name not like 'z_%'
LOOP
z_table := 'z_' || v_table;
SELECT max(id) from z_table INTO max_id;
DELETE FROM v_table where id > max_id;
END LOOP;
Some background information. For every table in my database, I have another table starting with "z_". E.g. for a table called "employee" I have identical table called "z_employee". z_employee contains the same set of data as employee. I use it to restore the employee table at the start of every test.
When I run this function I get the following error:
ERROR: relation "z_table" does not exist
LINE 1: SELECT max(id) from z_table
My guess is that I'm not allowed to use the variable z_table in the SQL query. At least not the way I'm using it here. But I don't know how it's supposed to be done.

Use dynamic SQL with EXECUTE, simplify, and escape identifiers properly:
CREATE OR REPLACE FUNCTION f_test()
RETURNS void AS
$func$
DECLARE
v_table text;
BEGIN
FOR v_table IN
SELECT table_name
FROM information_schema.tables
WHERE table_catalog = 'my_database'
AND table_schema = 'public'
AND table_name NOT LIKE 'z_%'
LOOP
EXECUTE format('DELETE FROM %I v WHERE v.id > (SELECT max(id) FROM %I)'
, v_table, 'z_' || v_table);
END LOOP;
END
$func$ LANGUAGE plpgsql;
Table names may need to be quoted to defend against syntax errors or even SQL injection! I use the convenient format() to concatenate the DELETE statement and escape identifiers properly.
A separate SELECT would be more expensive. You can do it all with a single DELETE statement.
Related:
Table name as a PostgreSQL function parameter
Aside:
You might use the (slightly faster) system catalog pg_tables instead:
SELECT tablename
FROM pg_catalog.pg_tables
WHERE schemaname = 'public'
AND tablename NOT LIKE 'z_%'
See:
How to check if a table exists in a given schema
table_catalog in information_schema.tables has no equivalent here. Only tables of the current database are visible anyway. So the above predicate WHERE table_catalog = 'my_database' produces an empty result set when connected to the wrong database.

Related

create missing sequence for database tables containing column called id

i'm trying to fix database issue related to creating missing sequence of the table after moving database from lower to higher version but i face 2 issues
first here is what i tried so far:
DO $$
DECLARE
i TEXT;
BEGIN
FOR i IN (select table_name from information_schema.tables where table_catalog='cst_sh' and table_schema='public') LOOP
IF EXISTS (SELECT count(*) FROM information_schema.columns WHERE table_name=table_name and column_name='id') THEN
EXECUTE 'CREATE SEQUENCE IF NOT EXISTS '''||i||'_id_seq''';
EXECUTE 'Select setval('''||i||'_id_seq'', (SELECT max(id) as a FROM ' || i ||')+1,true);';
end if;
END LOOP;
END$$;
1st problem is that the condition doesn't seem to work. First i check if the table has column called id then i start to create the sequence if exist and then set the value for it but some tables doesn't have id column so the second query fail.
2nd problem is with the 1st query that i use to create the sequence if it doesn't exist, it fails every time and i dont know why
the error is :
QUERY: CREATE SEQUENCE IF NOT EXISTS 'xxxx'
CONTEXT: PL/pgSQL function inline_code_block line 7 at EXECUTE
You should only loop over tables that actually have such a column. This can be achieved by using an EXISTS condition.
select t.table_name, t.table_schema
from information_schema.tables t
where t.table_schema = 'public'
and exists (select *
from information_schema.columns c
where t.table_name = c.table_name
and t.table_schema = c.table_schema
and c.column_name = 'id')
To create a safe SELECT statement you need to include the schema name of the table when getting the max() value.
Dynamic SQL is a lot easier to write if you use format() instead of string concatenation. You create a statement create sequence 'foo_id_seq' putting the sequence name in single quotes - but that's invalid for identifiers.
When you loop over a SELECT statement the loop variable should be a record, not a text value.
So putting that all together your code should look something like this:
DO $$
DECLARE
l_rec record;
l_seq_name text;
BEGIN
FOR l_rec IN select t.table_name, t.table_schema
from information_schema.tables t
where t.table_schema = 'stuff'
and exists (select *
from information_schema.columns c
where t.table_name = c.table_name
and t.table_schema = c.table_schema
and c.column_name = 'id')
LOOP
l_seq_name := l_rec.table_name||'_id_seq';
EXECUTE format('CREATE SEQUENCE IF NOT EXISTS %I', l_seq_name);
EXECUTE format('select setval(%L, max(id)) FROM %I.%I', l_seq_name, l_rec.table_schema, l_rec.table_name);
END LOOP;
END
$$;
You probably also want to make the sequence owned by the column, so you should add:
EXECUTE format('alter sequence %I owned by %I.%I.id', l_seq_name, l_rec.table_schema, l_rec.table_name);

How to dynamically copy tables from information schema inside a trigger function

I have an insert trigger function in which NEW.schema_name references a schema. I want to dynamically copy the tables found inside that schema ('foobaz','barbaz') as 'foo' and 'bar'. I then can perform queries without dynamic sql.
How can I create a function or simply copy/paste the same block of code to achive that.
EDIT :
I cannot get that dynamic query to work.
The part inside the WITH statement is working.
Not the bottom 'execute' part. I do not know if it is a syntax problem, or bad cast or whatever constraint there is in pgsql that makes it not working.
WITH info_schema_subset_table as (SELECT table_schema, table_name,
array_to_string((regexp_split_to_array(table_name,'_'))[4:array_length(regexp_split_to_array(table_name,'_'),1)-1] as new_table
FROM information_schema.tables
where table_schema = "schema_searched"
ORDER BY new_table ASC)
EXECUTE 'CREATE TABLE $2 as (SELECT * FROM $1)'
USING info_schema_subset_table.table_schema || '.' ||info_schema_subset_table.table_name,info_schema_subset_table.new_table;
EDIT 2
... Broken code removed...
In the code below, in which I'm unsure if the syntax is right, I get the following from the trigger
Provider errors:
PostGIS error while adding features: ERREUR: l'opérateur n'existe pas : record ~~ unknown
LINE 1: SELECT old_table LIKE '%ens%'
^
HINT: Aucun opérateur ne correspond au nom donné et aux types d'arguments.
Vous devez ajouter des conversions explicites de type.
QUERY: SELECT old_table LIKE '%ens%'
CONTEXT: fonction PL/pgsql validation_sio.afi_validation_sio(), ligne 18 à CASE
EDIT 3 :
CREATE OR REPLACE FUNCTION foo.foo()
RETURNS TRIGGER AS
$BODY$
DECLARE
old_table record;
new_table record;
dynamic_query text;
BEGIN
IF TG_OP = 'INSERT'
THEN
FOR old_table IN SELECT table_schema|| '.' ||table_name
FROM information_schema.tables
where table_schema = NEW.nom_schema
LOOP
CASE
WHEN
old_table LIKE '%ens%' THEN
new_table := concat('SIT_',array_to_string((regexp_split_to_array(info_schema.old_table,'_'))[4:array_length(regexp_split_to_array(info_schema.old_table,'_'),1)-1],'_'));
ELSE
new_table := concat('SID_',array_to_string((regexp_split_to_array(info_schema.old_table,'_'))[4:array_length(regexp_split_to_array(info_schema.old_table,'_'),1)-1],'_'));
END CASE;
dynamic_query := format('SELECT * FROM' || old_table ||);
EXECUTE dynamic_query
INTO new_table;
END LOOP;
RETURN NEW;
END IF;
END;
$BODY$
LANGUAGE plpgsql VOLATILE;
CREATE TRIGGER foo
AFTER INSERT ON validation.validationfoo
FOR EACH ROW EXECUTE PROCEDURE foo.foo();
I've reformatted your trigger function a bit and changed a few things, see if this works.
CREATE OR REPLACE FUNCTION foo.foo()
RETURNS TRIGGER AS
$BODY$
DECLARE
old_table record;
new_table record;
dynamic_query text;
BEGIN
IF TG_OP = 'INSERT' THEN
FOR old_table IN
SELECT table_schema || '.' || table_name AS old_table_name
FROM information_schema.tables
WHERE table_schema = NEW.nom_schema
LOOP
new_table := concat(CASE WHEN old_table.old_table_name LIKE '%ens%' THEN 'SIT_' ELSE 'SID_' END,array_to_string((regexp_split_to_array(info_schema.old_table,'_'))[4:array_length(regexp_split_to_array(info_schema.old_table,'_'),1)-1],'_'));
dynamic_query := 'CREATE TABLE ' || new_table || ' AS SELECT * FROM ' || old_table.old_table_name;
EXECUTE dynamic_query;
END LOOP;
RETURN NEW;
END IF;
END;
$BODY$
LANGUAGE plpgsql VOLATILE;
So the main things:
old_table is a record, so your comparison of it to a string with LIKE was failing. You need to use the field name. So I gave your field a name, and used that field name in the LIKE comparison.
Changed the new_table assignment to put the CASE statement only on the one item that changes, to make the difference more obvious and the code more concise. Mind you, I don't know if the rest of that line is actually valid, I just left it as is.
Changed the creation dynamic_query. As I said in the comment, the format function was being used incorrectly, so I just went with standard string concatenation instead.
Changed dynamic_query's SQL to what I think you actually want it to do. You want it to copy the content of the table to a new table, right? So that will do it.
You cannot have EXECUTE inside an SQL statement, it is a PL/pgSQL statement.
Loop through the tables and issue one EXECUTE for each.
Mind that you cannot have a schema or table name as a parameter with USING, because these names need to be known at parse time.
Use the format function to construct your dynamic statement so you can avoid SQL injection by users who maliciously create tables with weird names.

How to select from variable that is a table name n Postgre >=9.2

i have a variable that is a name of a table. How can i select or update from this using variable in query , for example:
create or replace function pg_temp.testtst ()
returns varchar(255) as
$$
declare
r record; t_name name;
begin
for r in SELECT tablename FROM pg_tables WHERE schemaname = 'public' limit 100 loop
t_name = r.tablename;
update t_name set id = 10 where id = 15;
end loop;
return seq_name;
end;
$$
language plpgsql;
it shows
ERROR: relation "t_name" does not exist
Correct reply is a comment from Anton Kovalenko
You cannot use variable as table or column name in embedded SQL ever.
UPDATE dynamic_table_name SET ....
PostgreSQL uses a prepared and saved plans for embedded SQL, and references to a target objects (tables) are deep and hard encoded in plans - a some characteristics has significant impact on plans - for one table can be used index, for other not. Query planning is relatively slow, so PostgreSQL doesn't try it transparently (without few exceptions).
You should to use a dynamic SQL - a one purpose is using for similar situations. You generate a new SQL string always and plans are not saved
DO $$
DECLARE r record;
BEGIN
FOR r IN SELECT table_name
FROM information_schema.tables
WHERE table_catalog = 'public'
LOOP
EXECUTE format('UPDATE %I SET id = 10 WHERE id = 15', r.table_name);
END LOOP;
END $$;
Attention: Dynamic SQL is unsafe (there is a SQL injection risks) without parameter sanitization. I used a function "format" for it. Other way is using "quote_ident" function.
EXECUTE 'UPDATE ' || quote_ident(r.table_name) || 'SET ...

Loop on tables with PL/pgSQL in Postgres 9.0+

I want to loop through all my tables to count rows in each of them. The following query gets me an error:
DO $$
DECLARE
tables CURSOR FOR
SELECT tablename FROM pg_tables
WHERE tablename NOT LIKE 'pg_%'
ORDER BY tablename;
tablename varchar(100);
nbRow int;
BEGIN
FOR tablename IN tables LOOP
EXECUTE 'SELECT count(*) FROM ' || tablename INTO nbRow;
-- Do something with nbRow
END LOOP;
END$$;
Errors:
ERROR: syntax error at or near ")"
LINE 1: SELECT count(*) FROM (sql_features)
^
QUERY: SELECT count(*) FROM (sql_features)
CONTEXT: PL/pgSQL function inline_code_block line 8 at EXECUTE statement
sql_features is a table's name in my DB. I already tried to use quote_ident() but to no avail.
I can't remember the last time I actually needed to use an explicit cursor for looping in PL/pgSQL.
Use the implicit cursor of a FOR loop, that's much cleaner:
DO
$$
DECLARE
rec record;
nbrow bigint;
BEGIN
FOR rec IN
SELECT *
FROM pg_tables
WHERE tablename NOT LIKE 'pg\_%'
ORDER BY tablename
LOOP
EXECUTE 'SELECT count(*) FROM '
|| quote_ident(rec.schemaname) || '.'
|| quote_ident(rec.tablename)
INTO nbrow;
-- Do something with nbrow
END LOOP;
END
$$;
You need to include the schema name to make this work for all schemas (including those not in your search_path).
Also, you actually need to use quote_ident() or format() with %I or a regclass variable to safeguard against SQL injection. A table name can be almost anything inside double quotes. See:
Table name as a PostgreSQL function parameter
Minor detail: escape the underscore (_) in the LIKE pattern to make it a literal underscore: tablename NOT LIKE 'pg\_%'
How I might do it:
DO
$$
DECLARE
tbl regclass;
nbrow bigint;
BEGIN
FOR tbl IN
SELECT c.oid
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind = 'r'
AND n.nspname NOT LIKE 'pg\_%' -- system schema(s)
AND n.nspname <> 'information_schema' -- information schema
ORDER BY n.nspname, c.relname
LOOP
EXECUTE 'SELECT count(*) FROM ' || tbl INTO nbrow;
-- raise notice '%: % rows', tbl, nbrow;
END LOOP;
END
$$;
Query pg_catalog.pg_class instead of tablename, it provides the OID of the table.
The object identifier type regclass is handy to simplify. n particular, table names are double-quoted and schema-qualified where necessary automatically (also prevents SQL injection).
This query also excludes temporary tables (temp schema is named pg_temp% internally).
To only include tables from a given schema:
AND n.nspname = 'public' -- schema name here, case-sensitive
The cursor returns a record, not a scalar value, so "tablename" is not a string variable.
The concatenation turns the record into a string that looks like this (sql_features). If you had selected e.g. the schemaname with the tablename, the text representation of the record would have been (public,sql_features).
So you need to access the column inside the record to create your SQL statement:
DO $$
DECLARE
tables CURSOR FOR
SELECT tablename
FROM pg_tables
WHERE tablename NOT LIKE 'pg_%'
ORDER BY tablename;
nbRow int;
BEGIN
FOR table_record IN tables LOOP
EXECUTE 'SELECT count(*) FROM ' || table_record.tablename INTO nbRow;
-- Do something with nbRow
END LOOP;
END$$;
You might want to use WHERE schemaname = 'public' instead of not like 'pg_%' to exclude the Postgres system tables.

How to add column if not exists on PostgreSQL?

Question is simple. How to add column x to table y, but only when x column doesn't exist ? I found only solution here how to check if column exists.
SELECT column_name
FROM information_schema.columns
WHERE table_name='x' and column_name='y';
With Postgres 9.6 this can be done using the option if not exists
ALTER TABLE table_name ADD COLUMN IF NOT EXISTS column_name INTEGER;
Here's a short-and-sweet version using the "DO" statement:
DO $$
BEGIN
BEGIN
ALTER TABLE <table_name> ADD COLUMN <column_name> <column_type>;
EXCEPTION
WHEN duplicate_column THEN RAISE NOTICE 'column <column_name> already exists in <table_name>.';
END;
END;
$$
You can't pass these as parameters, you'll need to do variable substitution in the string on the client side, but this is a self contained query that only emits a message if the column already exists, adds if it doesn't and will continue to fail on other errors (like an invalid data type).
I don't recommend doing ANY of these methods if these are random strings coming from external sources. No matter what method you use (client-side or server-side dynamic strings executed as queries), it would be a recipe for disaster as it opens you to SQL injection attacks.
Postgres 9.6 added ALTER TABLE tbl ADD COLUMN IF NOT EXISTS column_name.
So this is mostly outdated now. You might use it in older versions, or a variation to check for more than just the column name.
CREATE OR REPLACE function f_add_col(_tbl regclass, _col text, _type regtype)
RETURNS bool
LANGUAGE plpgsql AS
$func$
BEGIN
IF EXISTS (SELECT FROM pg_attribute
WHERE attrelid = _tbl
AND attname = _col
AND NOT attisdropped) THEN
RETURN false;
ELSE
EXECUTE format('ALTER TABLE %s ADD COLUMN %I %s', _tbl, _col, _type);
RETURN true;
END IF;
END
$func$;
Call:
SELECT f_add_col('public.kat', 'pfad1', 'int');
Returns true on success, else false (column already exists).
Raises an exception for invalid table or type name.
Why another version?
This could be done with a DO statement, but DO statements cannot return anything. And if it's for repeated use, I would create a function.
I use the object identifier types regclass and regtype for _tbl and _type which a) prevents SQL injection and b) checks validity of both immediately (cheapest possible way). The column name _col has still to be sanitized for EXECUTE with quote_ident(). See:
Table name as a PostgreSQL function parameter
format() requires Postgres 9.1+. For older versions concatenate manually:
EXECUTE 'ALTER TABLE ' || _tbl || ' ADD COLUMN ' || quote_ident(_col) || ' ' || _type;
You can schema-qualify your table name, but you don't have to.
You can double-quote the identifiers in the function call to preserve camel-case and reserved words (but you shouldn't use any of this anyway).
I query pg_catalog instead of the information_schema. Detailed explanation:
How to check if a table exists in a given schema
Blocks containing an EXCEPTION clause are substantially slower.
This is simpler and faster. The manual:
Tip
A block containing an EXCEPTION clause is significantly more
expensive to enter and exit than a block without one.
Therefore, don't use EXCEPTION without need.
Following select query will return true/false, using EXISTS() function.
EXISTS(): The argument of EXISTS is an arbitrary SELECT statement, or
subquery. The subquery is evaluated to determine whether it returns
any rows. If it returns at least one row, the result of EXISTS is
"true"; if the subquery returns no rows, the result of EXISTS is
"false"
SELECT EXISTS(SELECT column_name
FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = 'x'
AND column_name = 'y');
and use the following dynamic SQL statement to alter your table
DO
$$
BEGIN
IF NOT EXISTS (SELECT column_name
FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = 'x'
AND column_name = 'y') THEN
ALTER TABLE x ADD COLUMN y int DEFAULT NULL;
ELSE
RAISE NOTICE 'Already exists';
END IF;
END
$$
For those who use Postgre 9.5+(I believe most of you do), there is a quite simple and clean solution
ALTER TABLE if exists <tablename> add if not exists <columnname> <columntype>
the below function will check the column if exist return appropriate message else it will add the column to the table.
create or replace function addcol(schemaname varchar, tablename varchar, colname varchar, coltype varchar)
returns varchar
language 'plpgsql'
as
$$
declare
col_name varchar ;
begin
execute 'select column_name from information_schema.columns where table_schema = ' ||
quote_literal(schemaname)||' and table_name='|| quote_literal(tablename) || ' and column_name= '|| quote_literal(colname)
into col_name ;
raise info ' the val : % ', col_name;
if(col_name is null ) then
col_name := colname;
execute 'alter table ' ||schemaname|| '.'|| tablename || ' add column '|| colname || ' ' || coltype;
else
col_name := colname ||' Already exist';
end if;
return col_name;
end;
$$
This is basically the solution from sola, but just cleaned up a bit. It's different enough that I didn't just want to "improve" his solution (plus, I sort of think that's rude).
Main difference is that it uses the EXECUTE format. Which I think is a bit cleaner, but I believe means that you must be on PostgresSQL 9.1 or newer.
This has been tested on 9.1 and works. Note: It will raise an error if the schema/table_name/or data_type are invalid. That could "fixed", but might be the correct behavior in many cases.
CREATE OR REPLACE FUNCTION add_column(schema_name TEXT, table_name TEXT,
column_name TEXT, data_type TEXT)
RETURNS BOOLEAN
AS
$BODY$
DECLARE
_tmp text;
BEGIN
EXECUTE format('SELECT COLUMN_NAME FROM information_schema.columns WHERE
table_schema=%L
AND table_name=%L
AND column_name=%L', schema_name, table_name, column_name)
INTO _tmp;
IF _tmp IS NOT NULL THEN
RAISE NOTICE 'Column % already exists in %.%', column_name, schema_name, table_name;
RETURN FALSE;
END IF;
EXECUTE format('ALTER TABLE %I.%I ADD COLUMN %I %s;', schema_name, table_name, column_name, data_type);
RAISE NOTICE 'Column % added to %.%', column_name, schema_name, table_name;
RETURN TRUE;
END;
$BODY$
LANGUAGE 'plpgsql';
usage:
select add_column('public', 'foo', 'bar', 'varchar(30)');
Can be added to migration scripts invoke function and drop when done.
create or replace function patch_column() returns void as
$$
begin
if exists (
select * from information_schema.columns
where table_name='my_table'
and column_name='missing_col'
)
then
raise notice 'missing_col already exists';
else
alter table my_table
add column missing_col varchar;
end if;
end;
$$ language plpgsql;
select patch_column();
drop function if exists patch_column();
In my case, for how it was created reason it is a bit difficult for our migration scripts to cut across different schemas.
To work around this we used an exception that just caught and ignored the error. This also had the nice side effect of being a lot easier to look at.
However, be wary that the other solutions have their own advantages that probably outweigh this solution:
DO $$
BEGIN
BEGIN
ALTER TABLE IF EXISTS bobby_tables RENAME COLUMN "dckx" TO "xkcd";
EXCEPTION
WHEN undefined_column THEN RAISE NOTICE 'Column was already renamed';
END;
END $$;
You can do it by following way.
ALTER TABLE tableName drop column if exists columnName;
ALTER TABLE tableName ADD COLUMN columnName character varying(8);
So it will drop the column if it is already exists. And then add the column to particular table.
Simply check if the query returned a column_name.
If not, execute something like this:
ALTER TABLE x ADD COLUMN y int;
Where you put something useful for 'x' and 'y' and of course a suitable datatype where I used int.