liquibase sql migration issue - postgresql

I am writing 1 PostgreSQL function for some operation.
Writing SQL migration for that function but facing formatting error as liquibase is not able to recognize some portion.
Function Liquibase Migration:
CREATE OR REPLACE FUNCTION schema.fncn(trId integer, sts integer, stIds character varying)
RETURNS double precision
LANGUAGE plpgsql
AS '
DECLARE
abc integer;
query CHAR(1500);
xyz integer;
BEGIN
query := ''select sum(t.a)
FROM schema.tbl t
where t.id in(1,2)
and t.status ='' || sts ||
'' and t.status <> 2
and t.tr_id ='' || trId ||
'' and t.sw in('''', ''N'')'';
IF stIds is not null then
query := query || '' AND t.st_id IN ('' || stIds || '')'';
ELSE
END IF;
EXECUTE query INTO abc;
SELECT abc INTO xyz;
RETURN xyz;
END;
'
;
Following error it throwing:
Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "N"
Reason: liquibase.exception.DatabaseException: ERROR: syntax error at or near "N"
Any suggestion what I am missing?

The immediate problem is the nesting of ' of single quotes. To make that easier, use dollar quoting for the function body. You can nest dollar quoted string by choosing different delimiters.
To avoid any problems with concatenation of parameters, use parameter place holders in the query and pass the values with the USING clause. That will however require two different execute calls.
I assume stIds is a comma separated string of values. To use that as a (single) placeholder, convert it to an array using string_to_array() - or even better: change the type of the input parameter to text[] and pass an array directly.
The query variable is better defined as text, don't use char. There is also no need to copy the result of the query into a different variable (which by the way would be more efficient using xyz := abc; rather than a select into)
CREATE OR REPLACE FUNCTION schema.fncn(trId integer, sts integer, stIds character varying)
RETURNS double precision
LANGUAGE plpgsql
AS
$body$
DECLARE
abc integer;
query text;
BEGIN
query := $q$ select sum(t.a)
FROM schema.tbl t
where t.id in (1,2)
and t.status = $1
and t.status <> 2
and t.tr_id = $2
and t.sw in ('''', 'N') $q$;
IF stIds is not null then
query := query || $sql$ AND t.st_id = ANY (string_to_array($4, ',') $sql$;
EXECUTE query INTO abc
using trid, sts, stids;
ELSE
EXECUTE query INTO abc
using trid, sts;
END IF;
RETURN abc;
END;
$body$
;
Note that in the Liquibase change, you must use splitStatements=false in order to run this without errors.

Related

PL/pgSQL Looping through multiple schema, tables and rows

I have a database with multiple identical schemas. There is a number of tables all named 'tran_...' in each schema. I want to loop through all 'tran_' tables in all schemas and pull out records that fall within a specific date range. This is the code I have so far:
CREATE OR REPLACE FUNCTION public."configChanges"(starttime timestamp, endtime timestamp)
RETURNS SETOF character varying AS
$BODY$DECLARE
tbl_row RECORD;
tbl_name VARCHAR(50);
tran_row RECORD;
out_record VARCHAR(200);
BEGIN
FOR tbl_row IN
SELECT * FROM pg_tables WHERE schemaname LIKE 'ivr%' AND tablename LIKE 'tran_%'
LOOP
tbl_name := tbl_row.schemaname || '.' || tbl_row.tablename;
FOR tran_row IN
SELECT * FROM tbl_name
WHERE ch_edit_date >= starttime AND ch_edit_date <= endtime
LOOP
out_record := tbl_name || ' ' || tran_row.ch_field_name;
RETURN NEXT out_record;
END LOOP;
END LOOP;
RETURN;
END;
$BODY$
LANGUAGE plpgsql;
When I attempt to run this, I get:
ERROR: relation "tbl_name" does not exist
LINE 1: SELECT * FROM tbl_name WHERE ch_edit_date >= starttime AND c...
#Pavel already provided a fix for your basic error.
However, since your tbl_name is actually schema-qualified (two separate identifiers in : schema.table), it cannot be escaped as a whole with %I in format(). You have to escape each identifier individually.
Aside from that, I suggest a different approach. The outer loop is necessary, but the inner loop can be replaced with a simpler and more efficient set-based approach:
CREATE OR REPLACE FUNCTION public.config_changes(_start timestamp, _end timestamp)
RETURNS SETOF text AS
$func$
DECLARE
_tbl text;
BEGIN
FOR _tbl IN
SELECT quote_ident(schemaname) || '.' || quote_ident(tablename)
FROM pg_tables
WHERE schemaname LIKE 'ivr%'
AND tablename LIKE 'tran_%'
LOOP
RETURN QUERY EXECUTE format (
$$
SELECT %1$L || ' ' || ch_field_name
FROM %1$s
WHERE ch_edit_date BETWEEN $1 AND $2
$$, _tbl
)
USING _start, _end;
END LOOP;
RETURN;
END
$func$ LANGUAGE plpgsql;
You have to use dynamic SQL to parametrize identifiers (or code), like #Pavel already told you. With RETURN QUERY EXECUTE you can return the result of a dynamic query directly. Examples:
Return SETOF rows from PostgreSQL function
Refactor a PL/pgSQL function to return the output of various SELECT queries
Remember that identifiers have to be treated as unsafe user input in dynamic SQL and must always be sanitized to avoid syntax errors and SQL injection:
Table name as a PostgreSQL function parameter
Note how I escape table and schema separately:
quote_ident(schemaname) || '.' || quote_ident(tablename)
Consequently I just use %s to insert the already escaped table name in the later query. And %L to escape it a string literal for output.
I like to prepend parameter and variable names with _ to avoid naming conflicts with column names. No other special meaning.
There is a slight difference compared to your original function. This one returns an escaped identifier (double-quoted only where necessary) as table name, e.g.:
"WeIRD name"
instead of
WeIRD name
Much simpler yet
If possible, use inheritance to obviate the need for above function altogether. Complete example:
Select (retrieve) all records from multiple schemas using Postgres
You cannot use a plpgsql variable as SQL table name or SQL column name. In this case you have to use dynamic SQL:
FOR tran_row IN
EXECUTE format('SELECT * FROM %I
WHERE ch_edit_date >= starttime AND ch_edit_date <= endtime', tbl_name)
LOOP
out_record := tbl_name || ' ' || tran_row.ch_field_name;
RETURN NEXT out_record;
END LOOP;

plpgsql Select statement in For loop to create multiple CSV files

I would like to repeat the following query 8760 times, replacing ‘2’ with 1 to 8760 for every hour in the year. The idea is to create a separate CSV file for each hour for further processing.
COPY
(SELECT *
FROM
public.completedsolarirad2012
WHERE
completedsolarirad2012."UniquetmstmpID" = 2)
TO 'C:\temp\2012hour2.csv' WITH DELIMITER ','
CSV HEADER
I have put together the following function (testing with only a few hours):
CREATE OR REPLACE FUNCTION everyhour()
RETURNS void AS
$BODY$BEGIN
FOR i IN 0..5 LOOP
EXECUTE $x$
COPY (
SELECT *
FROM
public.completedsolarirad2012
WHERE
completedsolarirad2012."UniquetmstmpID" = i
)
TO $concat$ 'C:\temp.' || i::text
|| '.out.csv' WITH DELIMITER ',' CSV HEADER $concat$
$x$;
END LOOP;
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION everyhour()
OWNER TO postgres;
I seem to be having two separate problems:
Firstly, I’m getting:
Error: column "i" does not exist.
Secondly, when I test the concatenation statement only by replacing “i” with e.g. “2”, I get:
Error: relative path not allowed for COPY to file
I am the postgres superuser, so I do not understand why I am having this problem.
Note: Removing the $concat$ double-quoting around the concatenation statement gives the following error:
ERROR: syntax error at or near "||"
LINE 9: TO 'C:\temp.' || i::text
I would be very grateful for any help.
Assuming your server OS is Windows.
CREATE OR REPLACE FUNCTION everyhour()
RETURNS void AS
$func$
BEGIN
FOR i IN 0..5 LOOP
EXECUTE '
COPY (
SELECT *
FROM public.completedsolarirad2012 c
WHERE c."UniquetmstmpID" = ' || i || $x$)
TO 'C:/temp.'$x$ || i || $x$'.out.csv' WITH DELIMITER ',' CSV HEADER$x$;
END LOOP;
END
$func$ LANGUAGE plpgsql;
You had one layer of dollar-quoting too many.
You also accidentally concatenated the letter "i" instead of its value.
Use forward-slashes, even with windows. Or you may have to double up the backslashes, depending on your settings. More in this related answer:
PostgreSQL: export resulting data from SQL query to Excel/CSV
Simpler with format()
Since Postgres 9.1 you can use format() to simplify complex concatenations:
CREATE OR REPLACE FUNCTION everyhour()
RETURNS void AS
$func$
BEGIN
FOR i IN 0..5 LOOP
EXECUTE format($x$COPY (
SELECT *
FROM public.completedsolarirad2012 c
WHERE c."UniquetmstmpID" = %1$s)
TO 'C:/temp.%1$s.out.csv' WITH DELIMITER ',' CSV HEADER$x$, i);
END LOOP;
END
$func$ LANGUAGE plpgsql;

Return SETOF rows from PostgreSQL function

I have a situation where I want to return the join between two views. and that's a lot of columns. It was pretty easy in sql server. But in PostgreSQL when I do the join. I get the error "a column definition list is required".
Is there any way I can bypass this, I don't want to provide the definitions of returning columns.
CREATE OR REPLACE FUNCTION functionA(username character varying DEFAULT ''::character varying, databaseobject character varying DEFAULT ''::character varying)
RETURNS SETOF ???? AS
$BODY$
Declare
SqlString varchar(4000) = '';
BEGIN
IF(UserName = '*') THEN
Begin
SqlString := 'select * from view1 left join ' + databaseobject + ' as view2 on view1.id = view2.id';
End;
ELSE
Begin
SqlString := 'select * from view3 left join ' + databaseobject + ' as view2 on view3.id = view2.id';
End;
END IF;
execute (SqlString );
END;
$BODY$
Sanitize function
What you currently have can be simplified / sanitized to:
CREATE OR REPLACE FUNCTION func_a (username text = '', databaseobject text = '')
RETURNS ????
LANGUAGE plpgsql AS
$func$
BEGIN
RETURN QUERY EXECUTE
format ('SELECT * FROM %s v1 LEFT JOIN %I v2 USING (id)'
, CASE WHEN username = '*' THEN 'view1' ELSE 'view3' END
, databaseobject);
END
$func$;
You only need additional instances of BEGIN ... END in the function body to start separate code blocks with their own scope, which is rarely needed.
The standard SQL concatenation operator is ||. + is a "creative" addition of your former vendor.
Don't use CaMeL-case identifiers unless you double-quote them. Best don't use them at all See:
Are PostgreSQL column names case-sensitive?
varchar(4000) is also tailored to a specific limitation of SQL Server. It has no specific significance in Postgres. Only use varchar(4000) if you actually need a limit of 4000 characters. I would just use text - except that we don't need any variables at all here, after simplifying the function.
If you have not used format(), yet, consult the manual here.
Return type
Now, for your actual question: The return type for a dynamic query can be tricky since SQL requires that to be declared at call time at the latest. If you have a table or view or composite type in your database already matching the column definition list, you can just use that:
CREATE FUNCTION foo()
RETURNS SETOF my_view AS
...
Else, spell the column definition list with out with (simplest) RETURNS TABLE:
CREATE FUNCTION foo()
RETURNS TABLE (col1 int, col2 text, ...) AS
...
If you are making the row type up as you go, you can return anonymous records:
CREATE FUNCTION foo()
RETURNS SETOF record AS
...
But then you have to provide a column definition list with every call, so I hardly ever use that.
I wouldn't use SELECT * to begin with. Use a definitive list of columns to return and declare your return type accordingly:
CREATE OR REPLACE FUNCTION func_a(username text = '', databaseobject text = '')
RETURNS TABLE(col1 int, col2 text, col3 date)
LANGUAGE plpgsql AS
$func$
BEGIN
RETURN QUERY EXECUTE
format ($f$SELECT v1.col1, v1.col2, v2.col3
FROM %s v1 LEFT JOIN %I v2 USING (id)$f$
, CASE WHEN username = '*' THEN 'view1' ELSE 'view3' END
, databaseobject);
END
$func$;
For completely dynamic queries, consider building the query in your client to begin with, instead of using a function.
You need to understand basics first:
Refactor a PL/pgSQL function to return the output of various SELECT queries
PL/pgSQL in the Postgres manual
Then there are more advanced options with polymorphic types, which allow you to pass the return type at call time. More in the last chapter of:
Refactor a PL/pgSQL function to return the output of various SELECT queries

PostgreSQL - Writing dynamic sql in stored procedure that returns a result set

How can I write a stored procedure that contains a dynamically built SQL statement that returns a result set? Here is my sample code:
CREATE OR REPLACE FUNCTION reporting.report_get_countries_new (
starts_with varchar,
ends_with varchar
)
RETURNS TABLE (
country_id integer,
country_name varchar
) AS
$body$
DECLARE
starts_with ALIAS FOR $1;
ends_with ALIAS FOR $2;
sql VARCHAR;
BEGIN
sql = 'SELECT * FROM lookups.countries WHERE lookups.countries.country_name >= ' || starts_with ;
IF ends_with IS NOT NULL THEN
sql = sql || ' AND lookups.countries.country_name <= ' || ends_with ;
END IF;
RETURN QUERY EXECUTE sql;
END;
$body$
LANGUAGE 'plpgsql'
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER
COST 100 ROWS 1000;
This code returns an error:
ERROR: syntax error at or near "RETURN"
LINE 1: RETURN QUERY SELECT * FROM omnipay_lookups.countries WHERE o...
^
QUERY: RETURN QUERY SELECT * FROM omnipay_lookups.countries WHERE omnipay_lookups.countries.country_name >= r
CONTEXT: PL/pgSQL function "report_get_countries_new" line 14 at EXECUTE statement
I have tried other ways instead of this:
RETURN QUERY EXECUTE sql;
Way 1:
RETURN EXECUTE sql;
Way 2:
sql = 'RETURN QUERY SELECT * FROM....
/*later*/
EXECUTE sql;
In all cases without success.
Ultimately I want to write a stored procedure that contains a dynamic sql statement and that returns the result set from the dynamic sql statement.
There is room for improvements:
CREATE OR REPLACE FUNCTION report_get_countries_new (starts_with text
, ends_with text = NULL)
RETURNS SETOF lookups.countries AS
$func$
DECLARE
sql text := 'SELECT * FROM lookups.countries WHERE country_name >= $1';
BEGIN
IF ends_with IS NOT NULL THEN
sql := sql || ' AND country_name <= $2';
END IF;
RETURN QUERY EXECUTE sql
USING starts_with, ends_with;
END
$func$ LANGUAGE plpgsql;
-- the rest is default settings
Major points
PostgreSQL 8.4 introduced the USING clause for EXECUTE, which is useful for several reasons. Recap in the manual:
The command string can use parameter values, which are referenced in
the command as $1, $2, etc. These symbols refer to values supplied in
the USING clause. This method is often preferable to inserting data
values into the command string as text: it avoids run-time overhead of
converting the values to text and back, and it is much less prone to
SQL-injection attacks since there is no need for quoting or escaping.
IOW, it is safer and faster than building a query string with text representation of parameters, even when sanitized with quote_literal().
Note that $1, $2 in the query string refer to the supplied values in the USING clause, not to the function parameters.
While you return SELECT * FROM lookups.countries, you can simplify the RETURN declaration like demonstrated:
RETURNS SETOF lookups.countries
In PostgreSQL there is a composite type defined for every table automatically. Use it. The effect is that the function depends on the type and you get an error message if you try to alter the table. Drop & recreate the function in such a case.
This may or may not be desirable - generally it is! You want to be made aware of side effects if you alter tables. The way you have it, your function would break silently and raise an exception on it's next call.
If you provide an explicit default for the second parameter in the declaration like demonstrated, you can (but don't have to) simplify the call in case you don't want to set an upper bound with ends_with.
SELECT * FROM report_get_countries_new('Zaire');
instead of:
SELECT * FROM report_get_countries_new('Zaire', NULL);
Be aware of function overloading in this context.
Don't quote the language name 'plpgsql' even if that's tolerated (for now). It's an identifier.
You can assign a variable at declaration time. Saves an extra step.
Parameters are named in the header. Drop the nonsensical lines:
starts_with ALIAS FOR $1;
ends_with ALIAS FOR $2;
Use quote_literal() to avoid SQL injection (!!!) and fix your quoting problem:
CREATE OR REPLACE FUNCTION report_get_countries_new (
starts_with varchar,
ends_with varchar
)
RETURNS TABLE (
country_id integer,
country_name varchar
) AS
$body$
DECLARE
starts_with ALIAS FOR $1;
ends_with ALIAS FOR $2;
sql VARCHAR;
BEGIN
sql := 'SELECT * FROM lookups.countries WHERE lookups.countries.country_name ' || quote_literal(starts_with) ;
IF ends_with IS NOT NULL THEN
sql := sql || ' AND lookups.countries.country_name <= ' || quote_literal(ends_with) ;
END IF;
RETURN QUERY EXECUTE sql;
END;
$body$
LANGUAGE 'plpgsql'
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER
COST 100 ROWS 1000;
This is tested in version 9.1, works fine.

Postgres pl/pgsql ERROR: column "column_name" does not exist

i have a storerd procedure like below,
CREATE FUNCTION select_transactions3(text, text, int)
RETURNS SETOF transactions AS
$body$
DECLARE
rec transactions%ROWTYPE;
BEGIN
FOR rec IN (SELECT invoice_no, trans_date FROM transactions WHERE $1 = $2 limit $3 )
LOOP
RETURN NEXT rec;
END LOOP;
END;
$body$
LANGUAGE plpgsql VOLATILE SECURITY DEFINER;
when i execute query like this :
select * from select_transactions3("invoice_no", '1103300105472',10);
or
select * from select_transactions3(invoice_no, '1103300105472',10);
it getting error like this :
ERROR: column "invoice_no" does not exist
but when i try execute with one colon like this :
select * from select_transactions3('invoice_no', '1103300105472',10);
the result is no row.
how i can get the data like this :
invoice_no | trans_date
---------------+-------------------------
1103300105472 | 2011-03-30 12:25:35.694
thanks .
UPDATE : If we want a certain column of table that we want to show
CREATE FUNCTION select_to_transactions14(_col character varying, _val character varying, _limit int)
RETURNS SETOF RECORD AS
$$
DECLARE
rec record;
BEGIN
FOR rec IN EXECUTE 'SELECT invoice_no, amount FROM transactions
WHERE ' || _col || ' = $1 LIMIT $2' USING _val, _limit LOOP
RETURN NEXT rec;
END LOOP;
END;
$$ LANGUAGE plpgsql;
to get the result :
SELECT * FROM select_to_transactions14( 'invoice_no', '1103300105472',1)
as ("invoice_no" varchar(125), "amount" numeric(12,2));
Your function could look like this:
CREATE FUNCTION select_transactions3(_col text, _val text, _limit int)
RETURNS SETOF transactions AS
$BODY$
BEGIN
RETURN QUERY EXECUTE '
SELECT *
FROM transactions
WHERE ' || quote_ident(_col) || ' = $1
LIMIT $2'
USING _val, _limit;
END;
$BODY$
LANGUAGE plpgsql VOLATILE SECURITY DEFINER;
IN PostgreSQL 9.1 or later that's simpler with format()
...
RETURN QUERY EXECUTE format('
SELECT *
FROM transactions
WHERE %I = $1
LIMIT $2', _col)
USING _val, _limit;
...
%I escapes identifiers like quote_ident().
Major points:
You were bumping into the limitation of dynamic SQL that you cannot use parameters for identifiers. You have to build the query string with the column name and then execute it.
You can do that with values though. I demonstrate the use of the USING clause for EXECUTE. Also note the use of quote_ident(): prevents SQL injection and certain syntax errors.
I also largely simplified your function. [RETURN QUERY EXECUTE][3] makes your code shorter and faster. No need to loop if all you do is return the row.
I use named IN parameters, so you don't get confused with the $-notation in the query string. $1 and $2 inside the query string refer to the values provided in the USING clause, not to the input parameters.
I change to SELECT * as you have to return the whole row to match the declared return type anyway.
Last but not least: Be sure to consider what the manual has to say about functions declared SECURITY DEFINER.
RETURN TYPE
If you don't want to return the whole row, one convenient possibility is:
CREATE FUNCTION select_transactions3(_col text, _val text, _limit int)
RETURNS TABLE (invoice_no varchar(125), amount numeric(12,2) AS ...
Then you don't have to provide a column definition list with every call and can simplify to:
SELECT * FROM select_to_transactions3('invoice_no', '1103300105472', 1);
You can query all databases from the server and sort them according to your own database.
SELECT column_name
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'tableName';