I'm having trouble referencing record variable type columns dynamically. I found loads of tricks online, but with regards to triggers mostly and I really hope the answer isn't "it can't be done"... I've got a very specific and simple need, see my example code below;
First I have an array containing a list of column names called "lCols". I loop through a record variable to traverse my data, replacing values in a paragraph which exactly match my column names.
DECLARE lTotalRec RECORD;
DECLARE lSQL text;
DECLARE lCols varchar[];
p_paragraph:= 'I am [names] and my surname is [surname]';
lSQL :=
'select
p.names,
p.surname
from
person p
';
FOR lTotalRec IN
execute lSQL
LOOP
-- Loop through the already created array of columns to replace the values in the paragraph
FOREACH lVal IN ARRAY lCols
LOOP
p_paragraph := replace(p_paragraph,'[' || lVal || ']',lTotalRec.lVal); -- This is where my problem is, because lVal is not a column of lTotalRec directly like this
END LOOP;
RETURN NEXT;
END LOOP;
My return value is the paragraph amended for each record in "lTotalRec"
You could convert your record to a json value using the row_to_json() function. Once in this format, you can extract columns by name, using the -> and ->> operators.
In Postgres 9.4 and up, you can also make use of the more efficient jsonb type.
DECLARE lJsonRec jsonb;
...
FOR lTotalRec IN
execute lSQL
LOOP
lJsonRec := row_to_json(lTotalRec)::jsonb;
FOREACH lVal IN ARRAY lCols
LOOP
p_paragraph := replace(p_paragraph, '[' || lVal || ']', lJsonRec->>lVal);
END LOOP;
RETURN NEXT;
END LOOP;
See the documentation for more details.
You can convert a row to JSON using row_to_json(), and then retrieve the column names using json_object_keys().
Here's an example:
drop table if exists TestTable;
create table TestTable (col1 text, col2 text);
insert into TestTable values ('a1', 'b1'), ('a2', 'b2');
do $$declare
sql text;
rec jsonb;
col text;
val text;
begin
sql := 'select row_to_json(row) from (select * from TestTable) row';
for rec in execute sql loop
for col in select * from jsonb_object_keys(rec) loop
val := rec->>col;
raise notice 'col=% val=%', col, val;
end loop;
end loop;
end$$;
This prints:
NOTICE: col=col1 val=a1
NOTICE: col=col2 val=b1
NOTICE: col=col1 val=a2
NOTICE: col=col2 val=b2
DO
Related
I am writing a function in POSTGRES v13.3 that when passed an array of column names returns an array of JSONB objects each with the distinct values of one of the columns. I have an existing script that I wish to refactor using FORMAT in the declaration portion of the function.
The existing and working function looks like below. It is passed an array of columns and a dbase name. The a loop presents each column name to an EXECUTE statement that uses JSONB_AGG on the distinct values in the column, creates a JSONB object, and appends that to an array. The array is returned on completion. This is the function:
CREATE OR REPLACE FUNCTION foo1(text[], text)
RETURNS text[] as $$
declare
col text;
interim jsonb;
temp jsonb;
y jsonb[];
begin
foreach col in array $1
loop
execute
'select jsonb_agg(distinct '|| col ||') from ' || $2 into interim;
temp := jsonb_build_object(col, interim);
y := array_append(y,temp);
end loop;
return y;
end;
$$ LANGUAGE plpgsql;
I have refactored the function to the following. The script is now in the DECLARE portion of the function.
CREATE OR REPLACE FUNCTION foo2(_cols text[], _db text)
RETURNS jsonb[]
LANGUAGE plpgsql as
$func$
DECLARE
_script text := format(
'select jsonb_agg( distinct $1) from %1$I', _db
);
col text;
interim jsonb;
temp jsonb;
y jsonb[];
BEGIN
foreach col in array _cols
loop
EXECUTE _script USING col INTO interim;
temp := jsonb_build_object(col, interim);
y := array_append(y,temp);
end loop;
return y;
END
$func$;
Unfortunately the two functions give different results on a toy data set (see bottom):
Original: {"{\"id\": [1, 2, 3]}","{\"val\": [1, 2]}"}
Refactored: {{"id": ["id"]},{"val": ["val"]}}
Here is a db<>fiddle of the preceding.
The challenge is in the EXECUTE. In the first instance the col argument is treated as a column identifier. In the refactored function it seems to be treated as just a text string. I think my approach is consistent with the docs and tutorials (example), and the answer from this forum here and the links included therein. I have tried playing around with combinations of ", ', and || but those were unsuccessful and don't make sense in a format statement.
Where should I be looking for the error in my use of FORMAT?
NOTE 1: From the docs I have so possibly the jsonagg() and distinct are what's preventing the behaviour I want:
Another restriction on parameter symbols is that they only work in SELECT, INSERT, UPDATE, and DELETE commands. In other statement types (generically called utility statements), you must insert values textually even if they are just data values.
TOY DATA SET:
drop table if exists example;
create temporary table example(id int, str text, val integer);
insert into example values
(1, 'a', 1),
(2, 'a', 2),
(3, 'b', 2);
https://www.postgresql.org/docs/14/plpgsql-statements.html#PLPGSQL-STATEMENTS-SQL-ONEROW
The command string can use parameter values, which are referenced in
the command as $1, $2, etc. These symbols refer to values supplied in
the USING clause.
What you want is paramterize sql identifier(column name).
You cannot do that. Access column using variable instead of explicit column name
Which means that select jsonb_agg( distinct $1) from %1$I In here "$1" must be %I type. USING Expression in the manual (EXECUTE command-string [ INTO [STRICT] target ] [ USING expression [, ... ] ];) will pass the literal value to it. But it's valid because select distinct 'hello world' from validtable is valid.
select jsonb_agg( distinct $1) from %1$I In here $1 must be same type as %1$I namely-> sql identifier.
--
Based on the following debug code, then you can solve your problem:
CREATE OR REPLACE FUNCTION foo2(_cols text[], _db text)
RETURNS void
LANGUAGE plpgsql as
$func$
DECLARE
col text;
interim jsonb;temp jsonb; y jsonb[];
BEGIN
foreach col in array _cols
loop
EXECUTE format( 'select jsonb_agg( distinct ( %1I ) ) from %2I', col,_db) INTO interim;
raise info 'interim: %', interim;
temp := jsonb_build_object(col, interim);
raise info 'temp: %', temp;
y := array_append(y,temp);
raise info 'y: %',y;
end loop;
END
$func$;
I am trying to write function in postgresql, that creates temp_table with columns table_name text, table_rec jsonb and fill it through for loop with table names from my table containing names of tables and records in json. I have the for loop in string and I want to execute it. But it doesnt work.
I have variable rec record, sql_query text and tab_name text and I want to do this:
CREATE OR REPLACE FUNCTION public.test51(
)
RETURNS TABLE(tabel_name text, record_json jsonb)
LANGUAGE 'plpgsql'
COST 100
VOLATILE
ROWS 1000
AS $BODY$
declare
rec record;
tabel_name text;
tabel_names text[];
counter integer := 1;
sql_query text;
limit_for_sending integer;
rec_count integer;
begin
select into tabel_names array(select "TABLE_NAME" from public."TABLES");
create temp table temp_tab(tab_nam text, recik jsonb);
while array_length(tabel_names, 1) >= counter loop
tabel_name := '"' || tabel_names[counter] || '"';
select into limit_for_sending "TABLE_LIMIT_FOR_SENDING_DATA" from public."TABLES" where "TABLE_NAME" = tabel_name;
sql_query := 'select count(*) from public.' || tabel_name;
execute sql_query into rec_count;
if (rec_count >= limit_for_sending and limit_for_sending is not null) then
sql_query := 'for rec in select * from public.' || tabel_name || '
loop
insert into temp_tab
select ' || tabel_name || ', to_jsonb(rec);
end loop';
execute sql_query;
end if;
counter := counter + 1;
end loop;
return query
select * from temp_tabik;
drop table temp_tabik;
end;
$BODY$;
Thank you for response.
It seems you have some table that contains the information for which tables you want to return all rows as JSONB. And that meta-table also contains a column that sets a threshold under which the rows should not be returned.
You don't need the temp table or an array to store the table names. You can iterate through the query on the TABLES table and run the dynamic SQL directly in that loop.
return query in PL/pgSQL doesn't terminate the function, it just appends the result of the query to the result of the function.
Dynamic SQL is best created using the format() function because it is easier to read and using the %I placeholder will properly deal with quoted identifiers (which is really important as you are using those dreaded upper case table names)
As far as I can tell, your function can be simplified to:
CREATE OR REPLACE FUNCTION public.test51()
RETURNS TABLE(tabel_name text, record_json jsonb)
LANGUAGE plpgsql
AS
$BODY$
declare
rec record;
sql_query text;
rec_count bigint;
begin
for rec in
select "TABLE_NAME" as table_name, "TABLE_LIMIT_FOR_SENDING_DATA" as rec_limit
from public."TABLES"
loop
if rec.rec_limit is not null then
execute format('select count(*) from %I', rec.table_name)
into rec_count;
end if;
if (rec.rec_limit is not null and rec_count >= rec.rec_limit) then
sql_query := format('select %L, to_jsonb(t) from %I as t', rec.table_name, rec.table_name);
return query execute sql_query;
end if;
end loop;
end;
$BODY$;
Some notes
the language name is an identifier and should not be enclosed in single quotes. This syntax is deprecated and might be removed in a future version so don't get used to it.
you should really avoid those dreaded quoted identifiers. They are much more trouble than they are worth it. See the Postgres wiki for details.
I have some function on PostgreSQL 9.6 returning a cursor (refcursor):
CREATE OR REPLACE FUNCTION public.test_returning_cursor()
RETURNS refcursor
IMMUTABLE
LANGUAGE plpgsql
AS $$
DECLARE
_ref refcursor = 'test_returning_cursor_ref1';
BEGIN
OPEN _ref FOR
SELECT 'a' :: text AS col1
UNION
SELECT 'b'
UNION
SELECT 'c';
RETURN _ref;
END
$$;
I need to write another function in which a temp table is created and all data from this refcursor are inserted to it. But INSERT INTO ... FETCH ALL FROM ... seems to be impossible. Such function can't be compiled:
CREATE OR REPLACE FUNCTION public.test_insert_from_cursor()
RETURNS table(col1 text)
IMMUTABLE
LANGUAGE plpgsql
AS $$
BEGIN
CREATE TEMP TABLE _temptable (
col1 text
) ON COMMIT DROP;
INSERT INTO _temptable (col1)
FETCH ALL FROM "test_returning_cursor_ref1";
RETURN QUERY
SELECT col1
FROM _temptable;
END
$$;
I know that I can use:
FOR _rec IN
FETCH ALL FROM "test_returning_cursor_ref1"
LOOP
INSERT INTO ...
END LOOP;
But is there better way?
Unfortunately, INSERT and SELECT don't have access to cursors as a whole.
To avoid expensive single-row INSERT, you could have intermediary functions with RETURNS TABLE and return the cursor as table with RETURN QUERY. See:
Return a query from a function?
CREATE OR REPLACE FUNCTION f_cursor1_to_tbl()
RETURNS TABLE (col1 text) AS
$func$
BEGIN
-- MOVE BACKWARD ALL FROM test_returning_cursor_ref1; -- optional, see below
RETURN QUERY
FETCH ALL FROM test_returning_cursor_ref1;
END
$func$ LANGUAGE plpgsql; -- not IMMUTABLE
Then create the temporary table(s) directly like:
CREATE TEMP TABLE t1 ON COMMIT DROP
AS SELECT * FROM f_cursor1_to_tbl();
See:
Creating temporary tables in SQL
Still not very elegant, but much faster than single-row INSERT.
Note: Since the source is a cursor only the first call succeeds. Executing the function a second time would return an empty set. You would need a cursor with the SCROLL option and move to the start for repeated calls.
This function does INSERT INTO from refcursor. It is universal for all the tables. The only requirement is that all columns of table corresponds to columns of refcursor by types and order (not necessary by names).
to_json() does the trick to convert any primitive data types to string with double-quotes "", which are later replaced with ''.
CREATE OR REPLACE FUNCTION public.insert_into_from_refcursor(_table_name text, _ref refcursor)
RETURNS void
LANGUAGE plpgsql
AS $$
DECLARE
_sql text;
_sql_val text = '';
_row record;
_hasvalues boolean = FALSE;
BEGIN
LOOP --for each row
FETCH _ref INTO _row;
EXIT WHEN NOT found; --there are no rows more
_hasvalues = TRUE;
SELECT _sql_val || '
(' ||
STRING_AGG(val.value :: text, ',') ||
'),'
INTO _sql_val
FROM JSON_EACH(TO_JSON(_row)) val;
END LOOP;
_sql_val = REPLACE(_sql_val, '"', '''');
_sql_val = TRIM(TRAILING ',' FROM _sql_val);
_sql = '
INSERT INTO ' || _table_name || '
VALUES ' || _sql_val;
--RAISE NOTICE 'insert_into_from_refcursor(): SQL is: %', _sql;
IF _hasvalues THEN --to avoid error when trying to insert 0 values
EXECUTE (_sql);
END IF;
END;
$$;
Usage:
CREATE TABLE public.table1 (...);
PERFORM my_func_opening_refcursor();
PERFORM public.insert_into_from_refcursor('public.table1', 'name_of_refcursor_portal'::refcursor);
where my_func_opening_refcursor() contains
DECLARE
_ref refcursor = 'name_of_refcursor_portal';
OPEN _ref FOR
SELECT ...;
I am creating a trigger, which uses dynamic names for columns
NEW.name:=2222; -- works fine !
but
dynamic_column:='name';
EXECUTE '$1.'||dynamic_column||':=2222 ' USING NEW; -- raises error
gives an error:
ERROR: syntax error at or near "$1" LINE 1: $1.name:=2222
I found info here: Assign to NEW by key in a Postgres trigger
If we enable the module hstore by:
CREATE EXTENSION hstore;
We can do this:
dynamic_column:='name';
temp_sql_string:='"'||dynamic_column||'"=>"2222"';
NEW := NEW #= temp_sql_string::hstore;
And the RECORD NEW.name now is set to the value 2222.
Thank you tough for making an effort to find a solution #Laurenz Albe
The problem is that this is not a valid SQL statement.
You can access the columns in new with dynamic SQL like this:
EXECUTE 'SELECT $1.id' INTO v_id USING NEW;
There is no comfortable way like that for changing individual columns in NEW.
You could use TG_RELID to get the OID of the table, query pg_attribute for the columns, compose a row literal string composed of the values in NEW and your new value, cast this to the table type and assign the result to NEW. Quite cumbersome.
Here is sample code that does that (I tested it, but there may be bugs left):
CREATE OR REPLACE FUNCTION dyntrig() RETURNS trigger
LANGUAGE plpgsql AS
$$DECLARE
colname text;
colval text;
newrow text := '';
fieldsep text := 'ROW(';
BEGIN
/* loop through the columns of the table */
FOR colname IN
SELECT attname
FROM pg_catalog.pg_attribute
WHERE attrelid = TG_RELID
AND attnum > 0
AND NOT attisdropped
ORDER BY attnum
LOOP
IF colname = 'name' THEN
colval = '2222';
ELSE
/* all other columns than 'name' retain their value */
EXECUTE 'SELECT CAST($1.' || quote_ident(colname) || ' AS text)'
INTO colval USING NEW;
END IF;
/* compose a string that represents the new table row */
IF colval IS NULL THEN
newrow := newrow || fieldsep || 'NULL';
ELSE
newrow := newrow || fieldsep || '''' || colval || '''';
END IF;
fieldsep := ',';
END LOOP;
newrow := newrow || ')';
/* assign the new table row to NEW */
EXECUTE 'SELECT (CAST(' || newrow || ' AS '
|| quote_ident(TG_TABLE_SCHEMA) || '.' || quote_ident(TG_TABLE_NAME)
|| ')).*'
INTO NEW;
RETURN NEW;
END;$$;
You already found my answer recommending the hstore operator #= on dba.SE. You may also be interested in the corresponding reference answer here on SO:
How to set value of composite variable field using dynamic SQL
Since you construct the auxiliary hstore value from variables I suggest the simple function hstore():
CREATE OR REPLACE FUNCTION dyn_trigger_func()
RETURNS TRIGGER AS
$func$
DECLARE
dyn_col_name text := 'name';
dyn_col_val text := '2222';
BEGIN
NEW := NEW #= hstore(dyn_col_name, dyn_col_val);
RETURN NEW;
END
$func$ LANGUAGE plpgsql;
Faster / simpler / clearer / more secure this way.
Or, since it's obviously a trigger function, you may want to pass column name and value in CREATE TRIGGER statements:
CREATE OR REPLACE FUNCTION dyn_trigger_func()
RETURNS TRIGGER AS
$func$
BEGIN
NEW := NEW #= hstore(TG_ARGV[0], TG_ARGV[1]);
RETURN NEW;
END
$func$ LANGUAGE plpgsql;
And:
CREATE TRIGGER ins_bef
BEFORE INSERT ON tbl
FOR EACH ROW EXECUTE PROCEDURE dyn_trigger_func('name', '2222');
Provide column name unquoted and case-sensitive.
Related:
Get values from varying columns in a generic trigger
Trigger with dynamic field name
I have got a cursor, it is pointing to a SELECT, but this select is generated dynamically. I want to assign the statement after the declarement.
I have done an example working and another example NOT working. This is a simple example to print some data only.
This is the table:
CREATE TABLE public.my_columns (
id serial NOT NULL,
"name" varchar(30) NOT NULL,
/* Keys */
CONSTRAINT my_columns_pkey
PRIMARY KEY (id)
) WITH (
OIDS = FALSE
);
CREATE INDEX my_columns_index01
ON public.my_columns
("name");
INSERT INTO public.my_columns
("name")
VALUES
('name1'),
('name2'),
('name3'),
('name4'),
('name5'),
('name6');
This is the function(I have put the working code and the code not working):
CREATE OR REPLACE FUNCTION public.dynamic_table
(
)
RETURNS text AS $$
DECLARE
v_sql_dynamic varchar;
--NOT WORKING:
--db_c CURSOR IS (v_sql_dynamic::varchar);
--WORKING:
db_c CURSOR IS (SELECT id, name from public.my_columns);
db_rec RECORD;
BEGIN
v_sql_dynamic := 'SELECT id, name from public.my_columns';
FOR db_rec IN db_c LOOP
RAISE NOTICE 'NAME: %', db_rec.name;
END LOOP;
RETURN 'OK';
EXCEPTION WHEN others THEN
RETURN 'Error: ' || SQLERRM::text || ' ' || SQLSTATE::text;
END;
$$ LANGUAGE plpgsql;
Any ideas?
Thank you.
Do you really need the explicit cursor? If you need iterate over dynamic SQL, then you can use FOR IN EXECUTE. It is loop over implicit (internal) cursor for dynamic SQL
FOR db_rec IN EXECUTE v_sql_dynamic
LOOP
..
END LOOP
Little bit more complex solution is described in documentation - OPEN FOR EXECUTE:
do $$
declare r refcursor; rec record;
begin
open r for execute 'select * from pg_class';
fetch next from r into rec;
while found
loop
raise notice '%', rec;
fetch next from r into rec;
end loop;
close r;
end $$;
With this kind of cursor, you cannot to use FOR IN