PostgreSQL - Writing dynamic sql in stored procedure that returns a result set - postgresql

How can I write a stored procedure that contains a dynamically built SQL statement that returns a result set? Here is my sample code:
CREATE OR REPLACE FUNCTION reporting.report_get_countries_new (
starts_with varchar,
ends_with varchar
)
RETURNS TABLE (
country_id integer,
country_name varchar
) AS
$body$
DECLARE
starts_with ALIAS FOR $1;
ends_with ALIAS FOR $2;
sql VARCHAR;
BEGIN
sql = 'SELECT * FROM lookups.countries WHERE lookups.countries.country_name >= ' || starts_with ;
IF ends_with IS NOT NULL THEN
sql = sql || ' AND lookups.countries.country_name <= ' || ends_with ;
END IF;
RETURN QUERY EXECUTE sql;
END;
$body$
LANGUAGE 'plpgsql'
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER
COST 100 ROWS 1000;
This code returns an error:
ERROR: syntax error at or near "RETURN"
LINE 1: RETURN QUERY SELECT * FROM omnipay_lookups.countries WHERE o...
^
QUERY: RETURN QUERY SELECT * FROM omnipay_lookups.countries WHERE omnipay_lookups.countries.country_name >= r
CONTEXT: PL/pgSQL function "report_get_countries_new" line 14 at EXECUTE statement
I have tried other ways instead of this:
RETURN QUERY EXECUTE sql;
Way 1:
RETURN EXECUTE sql;
Way 2:
sql = 'RETURN QUERY SELECT * FROM....
/*later*/
EXECUTE sql;
In all cases without success.
Ultimately I want to write a stored procedure that contains a dynamic sql statement and that returns the result set from the dynamic sql statement.

There is room for improvements:
CREATE OR REPLACE FUNCTION report_get_countries_new (starts_with text
, ends_with text = NULL)
RETURNS SETOF lookups.countries AS
$func$
DECLARE
sql text := 'SELECT * FROM lookups.countries WHERE country_name >= $1';
BEGIN
IF ends_with IS NOT NULL THEN
sql := sql || ' AND country_name <= $2';
END IF;
RETURN QUERY EXECUTE sql
USING starts_with, ends_with;
END
$func$ LANGUAGE plpgsql;
-- the rest is default settings
Major points
PostgreSQL 8.4 introduced the USING clause for EXECUTE, which is useful for several reasons. Recap in the manual:
The command string can use parameter values, which are referenced in
the command as $1, $2, etc. These symbols refer to values supplied in
the USING clause. This method is often preferable to inserting data
values into the command string as text: it avoids run-time overhead of
converting the values to text and back, and it is much less prone to
SQL-injection attacks since there is no need for quoting or escaping.
IOW, it is safer and faster than building a query string with text representation of parameters, even when sanitized with quote_literal().
Note that $1, $2 in the query string refer to the supplied values in the USING clause, not to the function parameters.
While you return SELECT * FROM lookups.countries, you can simplify the RETURN declaration like demonstrated:
RETURNS SETOF lookups.countries
In PostgreSQL there is a composite type defined for every table automatically. Use it. The effect is that the function depends on the type and you get an error message if you try to alter the table. Drop & recreate the function in such a case.
This may or may not be desirable - generally it is! You want to be made aware of side effects if you alter tables. The way you have it, your function would break silently and raise an exception on it's next call.
If you provide an explicit default for the second parameter in the declaration like demonstrated, you can (but don't have to) simplify the call in case you don't want to set an upper bound with ends_with.
SELECT * FROM report_get_countries_new('Zaire');
instead of:
SELECT * FROM report_get_countries_new('Zaire', NULL);
Be aware of function overloading in this context.
Don't quote the language name 'plpgsql' even if that's tolerated (for now). It's an identifier.
You can assign a variable at declaration time. Saves an extra step.
Parameters are named in the header. Drop the nonsensical lines:
starts_with ALIAS FOR $1;
ends_with ALIAS FOR $2;

Use quote_literal() to avoid SQL injection (!!!) and fix your quoting problem:
CREATE OR REPLACE FUNCTION report_get_countries_new (
starts_with varchar,
ends_with varchar
)
RETURNS TABLE (
country_id integer,
country_name varchar
) AS
$body$
DECLARE
starts_with ALIAS FOR $1;
ends_with ALIAS FOR $2;
sql VARCHAR;
BEGIN
sql := 'SELECT * FROM lookups.countries WHERE lookups.countries.country_name ' || quote_literal(starts_with) ;
IF ends_with IS NOT NULL THEN
sql := sql || ' AND lookups.countries.country_name <= ' || quote_literal(ends_with) ;
END IF;
RETURN QUERY EXECUTE sql;
END;
$body$
LANGUAGE 'plpgsql'
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER
COST 100 ROWS 1000;
This is tested in version 9.1, works fine.

Related

liquibase sql migration issue

I am writing 1 PostgreSQL function for some operation.
Writing SQL migration for that function but facing formatting error as liquibase is not able to recognize some portion.
Function Liquibase Migration:
CREATE OR REPLACE FUNCTION schema.fncn(trId integer, sts integer, stIds character varying)
RETURNS double precision
LANGUAGE plpgsql
AS '
DECLARE
abc integer;
query CHAR(1500);
xyz integer;
BEGIN
query := ''select sum(t.a)
FROM schema.tbl t
where t.id in(1,2)
and t.status ='' || sts ||
'' and t.status <> 2
and t.tr_id ='' || trId ||
'' and t.sw in('''', ''N'')'';
IF stIds is not null then
query := query || '' AND t.st_id IN ('' || stIds || '')'';
ELSE
END IF;
EXECUTE query INTO abc;
SELECT abc INTO xyz;
RETURN xyz;
END;
'
;
Following error it throwing:
Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "N"
Reason: liquibase.exception.DatabaseException: ERROR: syntax error at or near "N"
Any suggestion what I am missing?
The immediate problem is the nesting of ' of single quotes. To make that easier, use dollar quoting for the function body. You can nest dollar quoted string by choosing different delimiters.
To avoid any problems with concatenation of parameters, use parameter place holders in the query and pass the values with the USING clause. That will however require two different execute calls.
I assume stIds is a comma separated string of values. To use that as a (single) placeholder, convert it to an array using string_to_array() - or even better: change the type of the input parameter to text[] and pass an array directly.
The query variable is better defined as text, don't use char. There is also no need to copy the result of the query into a different variable (which by the way would be more efficient using xyz := abc; rather than a select into)
CREATE OR REPLACE FUNCTION schema.fncn(trId integer, sts integer, stIds character varying)
RETURNS double precision
LANGUAGE plpgsql
AS
$body$
DECLARE
abc integer;
query text;
BEGIN
query := $q$ select sum(t.a)
FROM schema.tbl t
where t.id in (1,2)
and t.status = $1
and t.status <> 2
and t.tr_id = $2
and t.sw in ('''', 'N') $q$;
IF stIds is not null then
query := query || $sql$ AND t.st_id = ANY (string_to_array($4, ',') $sql$;
EXECUTE query INTO abc
using trid, sts, stids;
ELSE
EXECUTE query INTO abc
using trid, sts;
END IF;
RETURN abc;
END;
$body$
;
Note that in the Liquibase change, you must use splitStatements=false in order to run this without errors.

How to delimit PostgreSQL trigger procedure name?

For example, in Perl you delimit a variable like so:
${foo}_bar
I have a trigger in PostgreSQL borrowed from here that I am trying to make generic to work with multiple tables. Here's my code:
CREATE OR REPLACE FUNCTION update_parent_path() RETURNS TRIGGER AS $$
DECLARE
PATH ltree;
BEGIN
IF NEW.parent_id IS NULL THEN
NEW.parent_path = 'root'::ltree;
ELSEIF TG_OP = 'INSERT' OR OLD.parent_id IS NULL OR OLD.parent_id != NEW.parent_id THEN
SELECT parent_path || TG_TABLE_NAME_id::text FROM TG_TABLE_NAME WHERE TG_TABLE_NAME_id = NEW.parent_id INTO PATH;
IF PATH IS NULL THEN
RAISE EXCEPTION 'Invalid parent_id %', NEW.parent_id;
END IF;
NEW.parent_path = PATH;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
Each table that I am using this trigger against has a primary key like table_id (e.g., skill_id, level_id, etc). What I am trying to do is say WHERE skill_id = NEW.parent_id (for whatever table it's called against), thus the reason I am saying WHERE TG_TABLE_NAME_id = NEW.parent_id. What I'm wondering is how do I delimit TG_TABLE_NAME (trigger procedure) from _id?
Or, is there a better way to do this? Maybe I'm just going about this all wrong.
PLpgSQL has one fundamental rule - plpgsql variable cannot be used as table name or column name in embedded SQL. But there are a dynamic SQL - next way, how to execute SQL query. Dynamic SQL is a query generated in runtime from string (or string expression). There a PLpgSQL variable can be used everywhere. So your query fragment:
SELECT TG_TABLE_NAME_id::text FROM TG_TABLE_NAME ...
is wrong in more points, and should not work ever. But the dynamic query (PLpgSQL statement EXECUTE) should to work
EXECUTE format('SELECT %I FROM %I ...',
TG_TABLE_NAME || '_id', TG_TABLE_NAME) INTO path;
Related documentation: http://www.postgresql.org/docs/9.4/static/plpgsql-statements.html#PLPGSQL-STATEMENTS-EXECUTING-DYN

Function to DELETE rows taking the tablename as parameter

I want create a function:
CREATE OR REPLACE FUNCTION medibv.delAuto(tableName nvarchar(50), columnName nvarchar(100),value
nvarchar(100))
RETURNS void AS
$BODY$
begin
DELETE from tableName where columnName=value
end;
$BODY$
LANGUAGE plpgsql VOLATILE;
I have these parameters: tableName, columnName, value.
I want tableName as table in PostgreSQL.
CREATE OR REPLACE FUNCTION medibv.delauto(tbl regclass, col text, val text
,OUT success bool)
RETURNS bool AS
$func$
BEGIN
EXECUTE format('
DELETE FROM %s
WHERE %I = $1
RETURNING TRUE', tbl, col)
USING val
INTO success;
RETURN; -- optional in this case
END
$func$ LANGUAGE plpgsql;
Call:
SELECT medibv.delauto('myschema.mytbl', 'my_txt_col', 'foo');
Returns TRUE or NULL.
There is no nvarchar type in Postgres. You may be thinking of SQL Server. The equivalent would be varchar, but most of the time you can simply use text.
regclass is a specialized type for registered table names. It's perfect for the case an prevents SQL injection for the table name automatically and most effectively. More in the related answer below.
The column name is still prone to SQL injection. I sanitize the function with format(%I).
format() requires PostgreSQL 9.1+.
Your function did not report what happened. One or more rows may be found and deleted. Or none at all. As a bare minimum I added a boolean OUT column which will be TRUE if one or more rows were deleted. Because (quoting the manual here):
If multiple rows are returned, only the first will be assigned to the INTO variable.
Lastly, use USING with EXECUTE to pass in values. Don't cast back and forth. This is inefficient and prone to errors and to SQLi once more.
Find more explanation and links in this closely related answer:
Table name as a PostgreSQL function parameter
Use EXECUTE to run dynamic commands:
CREATE OR REPLACE FUNCTION medibv.delAuto(tableName nvarchar(50), columnName nvarchar(100),value
nvarchar(100))
RETURNS void AS
$BODY$
begin
EXECUTE 'DELETE FROM ' || tableName || ' WHERE ' || columnName || '=' || value;
end;
$BODY$
LANGUAGE plpgsql VOLATILE;

Using prepared statement in stored function

I have a table in the database:
create table store (
...
n_status integer not null,
t_tag varchar(4)
t_name varchar,
t_description varchar,
dt_modified timestamp not null,
...
);
In my stored function I need to execute the same select against this table multiple times:
select * from store
where n_place_id = [different values]
and t_tag is not null
and n_status > 0
and (t_name ~* t_search or t_description ~* t_search)
order by dt_modified desc
limit n_max;
Here, t_search and n_max are parameters into the stored function. I thought it would make sense to use a prepared statement for this, but I'm running into strange problems. Here's what I have:
create or replace function fn_get_data(t_search varchar, n_max integer)
returns setof store as
$body$
declare
resulter store%rowtype;
mid integer;
begin
prepare statement prep_stmt(integer) as
select *
from store
where n_place_id = $1
and (t_name ~* t_search or t_description ~* t_search)
order by dt_modified
limit n_max;
for mid in
(select n_place_id from ... where ...)
loop
for resulter in
execute prep_stmt(mid)
loop
return next resulter;
end loop;
end loop;
end;$body$
language 'plpgsql' volatile;
However when I actually run the function with
select * from fn_get_data('', 30)
I receive this error:
ERROR: column "t_search" does not exist
LINE 3: and (t_name ~* t_search or t_description ~* t_search)
^
QUERY: prepare prep_stmt(integer) as
select * from store where n_status > 0 and t_tag is not null and n_museum = $1
and (t_name ~* t_search or t_description ~* t_search)
order by dt_modified desc limit maxres_free
Ok, maybe it doesn't like external variables in the prepared statement, so I changed this to be
prepare prep_stmt(integer, varchar, integer) as
select * from store where n_status > 0 and t_tag is not null and n_museum = $1
and (t_name ~* $2 or t_description ~* $2)
order by dt_modified desc limit $3
...
for resulter in
execute prep_stmt(mid, t_search, n_max)
...
This time I get a different error:
ERROR: function prep_stmt(integer, character varying, integer) does not exist
LINE 1: SELECT prep_stmt(mid, t_search, n_max)
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
QUERY: SELECT prep_stmt(mid, t_search, n_max)
What am I missing here?
EDIT I added the relevant table structure at the top.
Looks to me like the PL/PgSQL EXECUTE for dynamic SQL trumps the regular SQL EXECUTE for prepared statements.
Code:
create or replace function prep_test() returns void as $$
begin
PREPARE do_something AS SELECT 1;
EXECUTE do_something;
end;
$$ LANGUAGE 'plpgsql';
Test:
regress=# select prep_test(1);
ERROR: column "do_something" does not exist
LINE 1: SELECT do_something
^
QUERY: SELECT do_something
CONTEXT: PL/pgSQL function "prep_test" line 4 at EXECUTE statement
outside PL/PgSQL it works fine:
regress=# EXECUTE do_something;
?column?
----------
1
(1 row)
I'm not sure how you'd execute a prepared statement within PL/PgSQL.
Out of interest, why are you trying to use prepared statements within PL/PgSQL? Plans are prepared and cached for PL/PgSQL anyway, it happens automatically.
There is a way to EXECUTE a prepared statement in a function, but like the accepted answer said, you typically don't wan't to do this in a function because the function already stores its plan.
That being said, there are still use cases where you do need to use a prepared statement in a function. My use case for this is when using multiple schemas for different users where the schemas contain tables that are similarly named and you want to use the same function to access one of these tables based off of what the search_path is set to. In this situation, because of the way the function stores its plan, using the same function after changing the search_path causes things to break. There are two solutions to this problem that I've stated. The first is to use EXECUTE '<Your query as a string here>'. But this can get very ugly for large queries, hence the reason to use the second method, which involves a PREPARE.
So with the background as to 'why' you'd want to do this out of the way, here is the how:
CREATE OR REPLACE FUNCTION prep_test()
RETURNS void AS $$
BEGIN
PREPARE do_something AS SELECT 1;
EXECUTE 'EXECUTE do_something;';
END;
$$ LANGUAGE plpgsql;
Though it will probably be in your best interests to add some protections to keep it from breaking. Something like:
CREATE OR REPLACE FUNCTION prep_test()
RETURNS void AS $$
BEGIN
IF (SELECT count(*) FROM pg_prepared_statements WHERE name ilike 'do_something') > 0 THEN
DEALLOCATE do_something;
END IF;
PREPARE do_something AS SELECT 1;
EXECUTE 'EXECUTE do_something;';
DEALLOCATE do_something;
END;
$$ LANGUAGE plpgsql;
Again, those who think that they want to do this, usually probably shouldn't, but for those cases where it is needed, this is how you do it.
You could use an EXECUTE statement like this in PLPGSQL:
select magicvalue into str_execute from magicvalues where magickey = ar_requestData[2];
EXECUTE str_execute into str_label USING ar_requestData[3], ar_requestData[4]::boolean, ar_requestData[5]::int, ar_requestData[6];
This is code I use in my application. ar_requestData is an array with text values.
In the table magicvalues do I store things like prepared statements.
The select statement is for example:
insert into classtypes(label, usenow, ranking, description) values($1,$2,$3,$4) returning label'
With kind regards,
Loek Bergman
PREPARE statement is not allowed within plpgsql. You can splice all the statements inside a function and use dynamic execute after then. Here is an example.
create or replace function sp_test(f_total int) returns void as $ytt$
declare v_sql text;
declare i int;
begin
v_sql:='prepare ytt_s1 (int,timestamp) as select * from tbl1 where id = $1 and log_time = $2;';
while i < f_total
loop
v_sql:=v_sql||'execute ytt_s1('||i||',now());';
i := i + 1;
end loop;
v_sql:=v_sql||'deallocate ytt_s1;';
execute v_sql;
end;
$ytt$ language plpgsql;

Postgres pl/pgsql ERROR: column "column_name" does not exist

i have a storerd procedure like below,
CREATE FUNCTION select_transactions3(text, text, int)
RETURNS SETOF transactions AS
$body$
DECLARE
rec transactions%ROWTYPE;
BEGIN
FOR rec IN (SELECT invoice_no, trans_date FROM transactions WHERE $1 = $2 limit $3 )
LOOP
RETURN NEXT rec;
END LOOP;
END;
$body$
LANGUAGE plpgsql VOLATILE SECURITY DEFINER;
when i execute query like this :
select * from select_transactions3("invoice_no", '1103300105472',10);
or
select * from select_transactions3(invoice_no, '1103300105472',10);
it getting error like this :
ERROR: column "invoice_no" does not exist
but when i try execute with one colon like this :
select * from select_transactions3('invoice_no', '1103300105472',10);
the result is no row.
how i can get the data like this :
invoice_no | trans_date
---------------+-------------------------
1103300105472 | 2011-03-30 12:25:35.694
thanks .
UPDATE : If we want a certain column of table that we want to show
CREATE FUNCTION select_to_transactions14(_col character varying, _val character varying, _limit int)
RETURNS SETOF RECORD AS
$$
DECLARE
rec record;
BEGIN
FOR rec IN EXECUTE 'SELECT invoice_no, amount FROM transactions
WHERE ' || _col || ' = $1 LIMIT $2' USING _val, _limit LOOP
RETURN NEXT rec;
END LOOP;
END;
$$ LANGUAGE plpgsql;
to get the result :
SELECT * FROM select_to_transactions14( 'invoice_no', '1103300105472',1)
as ("invoice_no" varchar(125), "amount" numeric(12,2));
Your function could look like this:
CREATE FUNCTION select_transactions3(_col text, _val text, _limit int)
RETURNS SETOF transactions AS
$BODY$
BEGIN
RETURN QUERY EXECUTE '
SELECT *
FROM transactions
WHERE ' || quote_ident(_col) || ' = $1
LIMIT $2'
USING _val, _limit;
END;
$BODY$
LANGUAGE plpgsql VOLATILE SECURITY DEFINER;
IN PostgreSQL 9.1 or later that's simpler with format()
...
RETURN QUERY EXECUTE format('
SELECT *
FROM transactions
WHERE %I = $1
LIMIT $2', _col)
USING _val, _limit;
...
%I escapes identifiers like quote_ident().
Major points:
You were bumping into the limitation of dynamic SQL that you cannot use parameters for identifiers. You have to build the query string with the column name and then execute it.
You can do that with values though. I demonstrate the use of the USING clause for EXECUTE. Also note the use of quote_ident(): prevents SQL injection and certain syntax errors.
I also largely simplified your function. [RETURN QUERY EXECUTE][3] makes your code shorter and faster. No need to loop if all you do is return the row.
I use named IN parameters, so you don't get confused with the $-notation in the query string. $1 and $2 inside the query string refer to the values provided in the USING clause, not to the input parameters.
I change to SELECT * as you have to return the whole row to match the declared return type anyway.
Last but not least: Be sure to consider what the manual has to say about functions declared SECURITY DEFINER.
RETURN TYPE
If you don't want to return the whole row, one convenient possibility is:
CREATE FUNCTION select_transactions3(_col text, _val text, _limit int)
RETURNS TABLE (invoice_no varchar(125), amount numeric(12,2) AS ...
Then you don't have to provide a column definition list with every call and can simplify to:
SELECT * FROM select_to_transactions3('invoice_no', '1103300105472', 1);
You can query all databases from the server and sort them according to your own database.
SELECT column_name
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'tableName';