Cannot Get Dynamic Exec of SP to Return INOUT Param - postgresql

Using PostgreSQL 13.2, wherein a stored procedure (the Requestor) is given a name of a list of stored procedures to run (the job group). All sp's executed this way are coded to write a log record as their last task. I have chosen to pull that 'append log' code from all of the sp's, and instead send back the log record (always a single record) using an INOUT rowtype param, but have run into trouble. In my example below, the requestor sp will load the records returned from the sp's it calls into a temp table shaped like the permanent log table.
That permanent table looks like this:
create table public.job_log (
log_id integer,
event_id integer,
job_id integer,
rows_affected integer);
Any one of the jobs that is executed by the requestor sp might look like this one:
CREATE OR REPLACE procedure public.get_log_rcd(
inout p_log_rcd public.job_log)
LANGUAGE 'plpgsql'
as
$BODY$
declare
v_log_id integer = 40;
v_event_id integer = 698;
v_job_id integer = 45;
v_rows_affected integer = 60;
begin
select
v_log_id
, v_event_id
, v_job_id
, v_rows_affected
into
p_log_rcd.log_id,
p_log_rcd.event_id,
p_log_rcd.job_id,
p_log_rcd.rows_affected;
end;
$BODY$
This sample sp doesn't do anything--it's purpose here is only to simulate initialize of the log parameters to return to caller.
Again, the requestor sp that's going to run jobs like the one above creates a temp table with the same structure as the permanent log:
drop table if exists tmp_log_cache;
create temp table tmp_log_cache as table public.job_log with no data;
If the requestor sp didn't have to do dynamic SQL, it would look something like this block here:
do
$$
declare
big_local public.job_log;
begin
call public.get_log_rcd( big_local );
insert into tmp_log_cache (
log_id
, event_id
, job_id
, rows_affected )
values (
big_local.log_id
, big_local.event_id
, big_local.job_id
, big_local.rows_affected);
end;
$$;
Doing a
select * from tmp_log_cache;
Returns a row containing the 4 column values expected, all is well. But, dynamic execution is required. And, as I'm sure most folks here know, the following dog don't hunt:
do
$$
declare
big_local public.job_log;
v_query_text varchar;
v_job_name varchar = 'public.get_log_rcd';
begin
select 'call ' || v_job_name || '( $1 );'
into v_query_text;
execute v_query_text using big_local::public.job_log;
insert into tmp_log_cache (
log_id
, event_id
, job_id
, rows_affected )
values (
big_local.log_id
, big_local.event_id
, big_local.job_id
, big_local.rows_affected);
end;
$$;
The above dynamic statement executes without error, but the insert statement only has NULL values to work with--a row is inserted, all nulls. Any suggestions warmly welcomed. The sp's that comprise the various job groups could probably have been implemented as functions, although in all cases their primary tasks are to massage, normalize, cleanse telemetry data, not to spit anything out, per se.

Hmm, the documentation states that "parameter symbols (...) only work in SELECT, INSERT, UPDATE, and DELETE commands.", so this probably isn't possible using parameters.
But as a workaround you can build a dynamic DO and include a variable to get the values and the INSERT in there.
DO
$o$
DECLARE
v_query_text varchar;
v_job_name varchar := format('%I.%I',
'public',
'get_log_rcd');
BEGIN
v_query_text := concat('DO ',
'$i$ ',
'DECLARE ',
' big_local public.job_log; ',
'BEGIN ',
' CALL ', v_job_name, '(big_local); ',
' INSERT INTO tmp_log_cache ',
' (log_id, ',
' event_id, ',
' job_id, ',
' rows_affected) ',
' VALUES (big_local.log_id, ',
' big_local.event_id, ',
' big_local.job_id, '
' big_local.rows_affected); ',
'END; ',
'$i$; ');
EXECUTE v_query_text;
END;
$o$;
db<>fiddle

Thanks--I would not have considered the ability to execute a 'do' using execute. It just would not have occurred to me. Well, here's my solution: flip to functions.
With the understanding that my 'Requestor' is only given sp's to run because that's what we had to do with SQL Server and it was reflex, I did the 1-line change needed to flip my example sp above to a function:
CREATE OR REPLACE function public.get_log_rcdf(
inout p_log_rcd public.job_log)
returns public.job_log
LANGUAGE 'plpgsql'
as
$BODY$
declare
v_log_id integer = 40;
v_event_id integer = 698;
v_job_id integer = 45;
v_rows_affected integer = 60;
begin
select
v_log_id
, v_event_id
, v_job_id
, v_rows_affected
into
p_log_rcd.log_id,
p_log_rcd.event_id,
p_log_rcd.job_id,
p_log_rcd.rows_affected;
end;
$BODY$
In fact, the change to a function required the addition of a RETURNS line. Done. Then, the dynamic call was tweaked to a SELECT and the execute modified with an INTO:
do
$$
declare
big_local public.job_log;
v_query_text varchar;
v_job_name varchar = 'public.get_log_rcdf';
begin
select 'select * from ' || v_job_name || '( $1 );'
into v_query_text;
raise info 'SQL text is: %', v_query_text;
execute v_query_text into big_local using big_local;
insert into tmp_log_cache (
log_id
, event_id
, job_id
, rows_affected )
values (
big_local.log_id
, big_local.event_id
, big_local.job_id
, big_local.rows_affected);
end;
$$;
and the process now works exactly as desired. I tidy up my handling of the dynamic function name as illustrated in the first answer, and I think we're done here.

Related

Declare a Table as a variable in a stored procedure?

I am currently working a stored procedure capable of detecting continuity on a specific set of entries..
The specific set of entries is extracted from a sql query
The function takes in two input parameter, first being the table that should be investigated, and the other being the list of ids which should be evaluated.
For every Id I need to investigate every row provided by the select statement.
DROP FUNCTION IF EXISTS GapAndOverlapDetection(table_name text, entity_ids bigint[]);
create or replace function GapAndOverlapDetection ( table_name text, enteity_ids bigint[] )
returns table ( entity_id bigint, valid tsrange, causes_overlap boolean, causes_gap boolean)
as $$
declare
x bigint;
var_r record;
begin
FOREACH x in array $2
loop
EXECUTE format('select entity_id, valid from' ||table_name|| '
where entity_id = '||x||'
and registration #> now()::timestamp
order by valid ASC') INTO result;
for var_r in result
loop
end loop;
end loop ;
end
$$ language plpgsql;
select * from GapAndOverlapDetection('temp_country_registration', '{1,2,3,4}')
I currently get an error in the for statement saying
ERROR: syntax error at or near "$1"
LINE 12: for var_r in select entity_id, valid from $1
You can iterate over the result of the dynamic query directly:
create or replace function gapandoverlapdetection ( table_name text, entity_ids bigint[])
returns table (entity_id bigint, valid tsrange, causes_overlap boolean, causes_gap boolean)
as $$
declare
var_r record;
begin
for var_r in EXECUTE format('select entity_id, valid
from %I
where entity_id = any($1)
and registration > now()::timestamp
order by valid ASC', table_name)
using entity_ids
loop
... do something with var_r
-- return a row for the result
-- this does not end the function
-- it just appends this row to the result
return query
select entity_id, true, false;
end loop;
end
$$ language plpgsql;
The %I injects an identifier into a string and the $1 inside the dynamic SQL is then populated through passing the argument with the using keyword
Firstly, decide whether you want to pass the table's name or oid. If you want to identify the table by name, then the parameter should be of text type and not regclass.
Secondly, if you want the table name to change between executions then you need to execute the SQL statement dynamically with the EXECUTE statement.

Dynamically add a column with multiple values to any table using a PL/pgSQL function

I would like to use a function/procedure to add to a 'template' table an additional column (e.g. period name) with multiple values, and do a cartesian product on the rows, so my 'template' is duplicated with the different values provided for the new column.
E.g. Add a period column with 2 values to my template_country_channel table:
SELECT *
FROM unnest(ARRAY['P1', 'P2']) AS prd(period)
, template_country_channel
ORDER BY period DESC
, sort_cnty
, sort_chan;
/*
-- this is equivalent to:
(
SELECT 'P2'::text AS period
, *
FROM template_country_channel
) UNION ALL (
SELECT 'P1'::text AS period
, *
FROM template_country_channel
)
--
*/
This query is working fine, but I was wondering if I could turn that into a PL/pgSQL function/procedure, providing the new column values to be added, the column to add the extra column to (and optionally specify the order by conditions).
What I would like to do is:
SELECT *
FROM template_with_periods(
'template_country_channel' -- table name
, ARRAY['P1', 'P2'] -- values for the new column to be added
, 'period DESC, sort_cnty, sort_chan' -- ORDER BY string (optional)
);
and have the same result as the 1st query.
So I created a function like:
CREATE OR REPLACE FUNCTION template_with_periods(template regclass, periods text[], order_by text)
RETURNS SETOF RECORD
AS $BODY$
BEGIN
RETURN QUERY EXECUTE 'SELECT * FROM unnest($2) AS prd(period), $1 ORDER BY $3' USING template, periods, order_by ;
END;
$BODY$
LANGUAGE 'plpgsql'
;
But when I run:
SELECT *
FROM template_with_periods('template_country_channel', ARRAY['P1', 'P2'], 'period DESC, sort_cnty, sort_chan');
I have the error ERROR: 42601: a column definition list is required for functions returning “record”
After some googling, it seems that I need to define the list of columns and types to perform the RETURN QUERY (as the error message precisely states).
Unfortunately, the whole idea is to use the function with many 'template' tables, so columns name & type lists is not fixed.
Is there any other approach to try?
Or is the only way to make it work is to have within the function, a way to get list of columns' names and types of the template table?
I did this with refcursor if You want output columns list completely dynamic:
CREATE OR REPLACE FUNCTION is_record_exists(tablename character varying, columns character varying[], keepcolumns character varying[] DEFAULT NULL::character varying[])
RETURNS SETOF refcursor AS
$BODY$
DECLARE
ref refcursor;
keepColumnsList text;
columnsList text;
valuesList text;
existQuery text;
keepQuery text;
BEGIN
IF keepcolumns IS NOT NULL AND array_length(keepColumns, 1) > 0 THEN
keepColumnsList := array_to_string(keepColumns, ', ');
ELSE
keepColumnsList := 'COUNT(*)';
END IF;
columnsList := (SELECT array_to_string(array_agg(name || ' = ' || value), ' OR ') FROM
(SELECT unnest(columns[1:1]) AS name, unnest(columns[2:2]) AS value) pair);
existQuery := 'SELECT ' || keepColumnsList || ' FROM ' || tableName || ' WHERE ' || columnsList;
RAISE NOTICE 'Exist query: %', existQuery;
OPEN ref FOR EXECUTE
existQuery;
RETURN next ref;
END;$BODY$
LANGUAGE plpgsql;
Then need to call FETCH ALL IN to get results. Detailed syntax here or there: https://stackoverflow.com/a/12483222/630169. Seems it is the only way for now. Hope something will be changed in PostgreSQL 11 with PROCEDURES.

Getting Results of Dynamic Query As A Table?

My company is going to start generating documents using data from our database and I am designing the function that will spit out the document text. These documents will need to contain data taken from multiple tables, with hundreds of columns and invariably some records will be missing data.
I am trying to make a function that will take null fields and replace them with a little error message that makes it clear to the end user that a piece of data is missing. Since the end user is totally unfamiliar with the backend, I want these messages to reference something intelligible to them.
My solution is pretty simple yet I for the life of me can't get it to work. The record identifier, table name are set as parameters in the function. The function then loops through names for each of the columns in the specified table, building a query that contains a bunch of case statements. Once the loop is complete, the identifier is appended and then the query is executed, returning the results to the calling function.
Despite reading around quite a bit, the best I can is a single column/row containing all the results - not useful to me at all, because I need to be able to easily reference specific pieces of data in the parent query. I am a beginner with Postgres and the documentation is too complex for me to understand, any help would be appreciated.
-- Function: data_handler(text, text)
-- DROP FUNCTION data_handler(text, text);
CREATE OR REPLACE FUNCTION data_handler(target_uri text, target_table TEXT)
RETURNS SETOF record AS
$BODY$
DECLARE
c text;
strSQL text;
site_only text;
result record;
BEGIN
--We need the schema for strSQL but the loop needs just the table name.
site_only = split_part(target_table, '.', 2);
FOR c IN
SELECT column_name
FROM information_schema.columns
WHERE table_name = site_only
LOOP
strSQL = concat(strSQL, chr(10), '(SELECT CASE WHEN ', c::text, '::text IS NULL THEN concat(', chr(39), '<Error:', chr(39), ', (SELECT lkp_value FROM alb_cr.lkp_field_values WHERE column_name = ', chr(39), c::text, chr(39), ')::text, ', chr(39), ' value not found>', chr(39), ')::text ELSE ',
c::text, '::text END AS ', c::text, '_convert) AS ', c::text, ',');
END LOOP;
strSQL = LEFT(strSQL, character_length(strSQL) - 1);
strSQL = concat('SELECT ', strSQL, ' FROM ', target_table, ' WHERE nm_site_id = ', chr(39), target_uri, chr(39));
RETURN QUERY EXECUTE strSQL;
RAISE NOTICE 'strSQL: %', strSQL;
--RETURN strSQL;
--RETURN QUERY EXECUTE format('SELECT ' || strSQL || 'FROM %s WHERE nm_site_id = $1', pg_typeof(target_table)) USING target_uri;
END
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION data_handler(text, text)
OWNER TO inti;
You could create views for that as well, in the following example on a schema nullsbegone:
-- create the schema to hold the views
create schema if not exists nullsbegone;
-- create a function to create the views (any and all that you might need)
create or replace function nullsbegone.f_make_view_of(p_tablename text) returns void as $f$
begin
execute ($$
create or replace view nullsbegone.$$||(select relname from pg_class where oid = $1::regclass)||$$
returns void as
select $$||array_to_string(array(
select case when not attnotnull then 'COALESCE('||quote_ident(attname)||$$::text, (SELECT '<Error:'''||lkp_value||''' value not found>' FROM alb_cr.lkp_field_values
WHERE column_name = $$||quote_literal(attname)||$$)) AS $$
else '' end || quote_ident(attname)
from pg_attribute
where attrelid = $1::regclass and attnum > 0 order by attnum
), E', \n')||$$
from $$||$1);
end;$f$ language plpgsql;
-- create the view based on a given table
select nullsbegone.f_make_view_of('yourschema.yourtable');
-- select from your view as if you were selecting from the actual table
select * from nullsbegone.yourtable
where nm_site_id = 'yoursite';

Print the output of execution of DO anonymous block

I have an anonymous DO block in a Postgres database like below:
DO LANGUAGE plpgsql $$
DECLARE
FUNC_ID smallint;
BEGIN
SELECT COUNT(1) INTO FUNC_ID FROM FUNCTION WHERE NAME = 'addition';
IF FUNC_ID = 0 THEN
INSERT INTO FUNCTION ( ID, NAME, DESCRIPTION, FUNCTION)
values ((select (max(id)+1) from FUNCTION), 'subtraction'
, 'calculate', 'catalog');
END IF;
END;
$$;
If execute this block of code it gives output as only DO.
How to output the whole block of code to console?
Use a RAISE NOTICE statement
postgres=# DO $$BEGIN RAISE NOTICE 'Hello %', SESSION_USER; END; $$;
NOTICE: Hello pavel
DO
See more in related documentation.
Your code is twisted in multiple ways. Use instead:
DO
$do$
BEGIN
IF EXISTS (SELECT 1 FROM function WHERE name = 'addition') THEN
INSERT INTO function(id, name, description, function)
SELECT max(id) + 1, 'subtraction', 'calculate', 'catalog'
FROM function;
RAISE NOTICE 'Whatever'; -- see Pavel's answer
END IF;
END
$do$;
You probably should have a serial primary key, drawing the next value from a SEQUENCE. What you have is prone to race conditions and a typical anti-pattern.
Also I wouldn't use function as identifier even if that's allowed in Postgres. It's a reserved word in the SQL standard.
CREATE TABLE func
func_id serial PRIMARY KEY
, func text NOT NULL
, description text
, find_proper_name text)
);
Then your whole statement can be:
INSERT INTO func(func, description, find_proper_name)
SELECT 'subtraction', 'calculate', 'catalog'
WHERE EXISTS (SELECT 1 FROM func WHERE func = 'addition');
You don't need a DO statement at all here.

How to clone a RECORD in PostgreSQL

I want to loop through a query, but also retain the actual record for the next loop, so I can compare two adjacent rows.
CREATE OR REPLACE FUNCTION public.test ()
RETURNS void AS
$body$
DECLARE
previous RECORD;
actual RECORD;
query TEXT;
isdistinct BOOLEAN;
tablename VARCHAR;
columnname VARCHAR;
firstrow BOOLEAN DEFAULT TRUE;
BEGIN
tablename = 'naplo.esemeny';
columnname = 'esemeny_id';
query = 'SELECT * FROM ' || tablename || ' LIMIT 2';
FOR actual IN EXECUTE query LOOP
--do stuff
--save previous record
IF NOT firstrow THEN
EXECUTE 'SELECT ($1).' || columnname || ' IS DISTINCT FROM ($2).' || columnname
INTO isdistinct USING previous, actual;
RAISE NOTICE 'previous: %', previous.esemeny_id;
RAISE NOTICE 'actual: %', actual.esemeny_id;
RAISE NOTICE 'isdistinct: %', isdistinct;
ELSE
firstrow = false;
END IF;
previous = actual;
END LOOP;
RETURN;
END;
$body$
LANGUAGE 'plpgsql'
VOLATILE
CALLED ON NULL INPUT
SECURITY INVOKER
COST 100;
The table:
CREATE TABLE naplo.esemeny (
esemeny_id SERIAL,
felhasznalo_id VARCHAR DEFAULT "current_user"() NOT NULL,
kotesszam VARCHAR(10),
idegen_azonosito INTEGER,
esemenytipus_id VARCHAR(10),
letrehozva TIMESTAMP WITHOUT TIME ZONE DEFAULT now() NOT NULL,
szoveg VARCHAR,
munkalap_id VARCHAR(13),
ajanlat_id INTEGER,
CONSTRAINT esemeny_pkey PRIMARY KEY(esemeny_id),
CONSTRAINT esemeny_fk_esemenytipus FOREIGN KEY (esemenytipus_id)
REFERENCES naplo.esemenytipus(esemenytipus_id)
ON DELETE RESTRICT
ON UPDATE RESTRICT
NOT DEFERRABLE
)
WITH (oids = true);
The code above doesn't work, the following error message is thrown:
ERROR: could not identify column "esemeny_id" in record data type
LINE 1: SELECT ($1).esemeny_id IS DISTINCT FROM ($2).esemeny_id
^
QUERY: SELECT ($1).esemeny_id IS DISTINCT FROM ($2).esemeny_id
CONTEXT: PL/pgSQL function "test" line 18 at EXECUTE statement
LOG: duration: 0.000 ms statement: SET DateStyle TO 'ISO'
What am I missing?
Disclaimer: I know the code doesn't make too much sense, I only created so I can demonstrate the problem.
This does not directly answer your question, and may be of no use at all, since you did not really describe your end goal.
If the end goal is to be able to compare the value of a column in the current row with the value of the same column in the previous row, then you might be much better off using a windowing query:
SELECT actual, previous
FROM (
SELECT mycolumn AS actual,
lag(mycolumn) OVER () AS previous
FROM mytable
ORDER BY somecriteria
) as q
WHERE previous IS NOT NULL
AND actual IS DISTINCT FROM previous
This example prints the rows where the current row is different from the previous row.
Note that I added an ORDER BY clause - it does not make sense to talk about "the previous row" without specifying ordering, otherwise you would get random results.
This is plain SQL, not PlPgSQL, but if you can wrap it in a function if you want to dynamically generate the query.
I am pretty sure, there is a better solution for your actual problem. But to answer the question asked, here is a solution with polymorphic types:
The main problem is that you need well known composite types to work with. the structure of anonymous records is undefined until assigned.
CREATE OR REPLACE FUNCTION public.test (actual anyelement, _col text
, OUT previous anyelement) AS
$func$
DECLARE
isdistinct bool;
BEGIN
FOR actual IN
EXECUTE format('SELECT * FROM %s LIMIT 3', pg_typeof(actual))
LOOP
EXECUTE format('SELECT ($1).%1$I IS DISTINCT FROM ($2).%1$I', _col)
INTO isdistinct
USING previous, actual;
RAISE NOTICE 'previous: %; actual: %; isdistinct: %'
, previous, actual, isdistinct;
previous := actual;
END LOOP;
previous := NULL; -- reset dummy output (optional)
END
$func$ LANGUAGE plpgsql;
Call:
SELECT public.test(NULL::naplo.esemeny, 'esemeny_id')
I am abusing an OUT parameter, since it's not possible to declare additional variables with a polymorphic composite type (at least I have failed repeatedly).
If your column name is stable you can replace the second EXECUTE with a simple expression.
I am running out of time, explanation in these related answers:
Declare variable of composite type in PostgreSQL using %TYPE
Refactor a PL/pgSQL function to return the output of various SELECT queries
Asides:
Don't quote the language name, it's an identifier, not a string.
Do you really need WITH (oids = true) in your table? This is still allowed, but largely deprecated in modern Postgres.