I have the following code, to insert data into my table res(ID bigserial,
Results jsonb not null). I want to insert data so that 'Display' column always has the 'i' appended to it, so that every row has a different value for 'Display' Column.
DO $$
declare cnt bigint;
BEGIN
FOR i IN 1..2 LOOP
INSERT INTO res (Results)
VALUES ('{"cid":"CID1","Display":"User One'|| i || '","FName":"Userfff","LName":"One"}');
INSERT INTO res (Results)
VALUES ('{"cid":"CID1","Display":"User One'|| i || '","FName":"Userfff","LName":"One"}');
INSERT INTO res (Results)
VALUES ('{"cid":"CID1","Display":"User One'|| i || '","FName":"Userfff","LName":"One"}');
END LOOP;
END;
$$
LANGUAGE plpgsql;
However, when I run this code, I get the following error:
ERROR: column "results" is of type jsonb but expression is of type text
LINE 2: VALUES ('{"cid":"CID1","Display":"User One'|| i ...
^
HINT: You will need to rewrite or cast the expression.
How should I modify my query so that the query runs successfully?
No need for a loop or even PL/pgSQL.
You can generate any number of rows using generate_series()
To put a value into a string I prefer to use format() as that makes dealing with strings a lot easier:
insert into res(results)
select format('{"cid":"CID1","Display":"User One %s","FName":"Userfff","LName":"One"}', i::text)::jsonb
from generate_series(1,5) g(i);
The second parameter to the format() function will be put where the %s in the first parameter.
The above inserts 5 rows each with a different value for display. This is what you stated in your question. Your sample code however inserts a total of 6 rows where 3 rows have the same value for display.
Related
The following code is not a full setup that you can run to check. It shall just make it a bit clearer what the question is about.
With an example function like this (the variadic example is taken from PostgreSQL inserting list of objects into a stored procedure or PostgreSQL - Passing Array to Stored Function):
CREATE OR REPLACE function get_v(variadic _v text[]) returns table (v varchar(50)) as
$F$
declare
begin
return query
select t.v
from test t
where t.v = any(_v)
end;
$F$
language plpgsql
;
If you copy the one-value output of a select string_agg... query, 'x','y','z', by hand and put it as the argument of the function, the function works:
SELECT v FROM get_v_from_v(
'x','y','z'
);
The 'x','y','z' gets read into the function as variadic _v text[] so that the function can check its values with where t.v = any(_v).
If you instead put the (select string_agg...) query that is behind that 'x','y','z' output in the same place, the function does not work:
select v from get_v_from_v(
(select string_agg(quote_literal(x.v), ',') from (select v from get_v_from_name('something')) as x)
);
That means: the "one-value output field" 'x','y','z' that comes from the (select string_agg...) query is not the same as the text[] list type: 'x','y','z'.
With get_v_from_name('something') as another function that returns a table of one column and the "v" values in the rows, and after running the string_agg() on its output, you get the 'x','y','z' output. I learnt this working function string_agg() at How to make a list of quoted strings from the string values of a column in postgresql?. The full list of such string functions is in the postgreSQL guide at 9.4. String Functions and Operators.
I guess that the format of the select query output is just a string, not a list, so that the input is not seen as a list of quoted strings, but rather like a string as a whole: ''x','y','z''. The get_v_from_v argument does not need just one string of all values, but a list of quoted strings, since the argument for the function is of type text[] - which is a list.
It seems as if this question does not depend on the query that is behind the output. It seems rather just a general thing that the output in a tuple of a table and taken as the argument of a function is not the same as the same output hand-copied as the same argument.
Therefore, the question. What needs to be done to make the output of a select query the same as the hand-copy of its output, so that the output is just the list 'x','y','z', as if it was just copied and pasted?
PS: I guess that this way of making lists of quoted strings from the one-column table output only to pass it to the function is not best practice. For example, in TSQL/SQL Server, you should pass "table valued parameters", so that you pass values as a table that you select from within the function to get the values, see How do I pass a list as a parameter in a stored procedure?. Not sure how this is done in postgreSQL, but it might be what is needed here.
CREATE OR REPLACE function get_v(_v text[]) returns table (v varchar(50)) as
$F$
declare
begin
return query
select t.v
from test t
where t.v = any((select * from unnest(_v)))
end;
$F$
language plpgsql
;
With get_v_from_name('something') as another function that returns a table of one column and the "v" values in the rows (this was said in the question), the following works:
select v from get_v_from_v(
(select array_agg(x.v) from (select v from get_v_from_name('something')) as x)
);
Side remark:
array_agg(quote_literal(x.v), ',') instead of array_agg(x.v) fails, the function does not allow a second argument.
I need to create a function that will generate a table.
This table will have columns brought with several left joins.
At some point i need to filter the data based on the value of a dynamic column (i need to have: WHERE table_name.dynamic_column_name = 1)(that column has 1s and 0s, i need to keep only the 1s)
So when I 'call' the function, user should type like this: SELECT * FROM function_name(dynamic_column_name)
What i did:
CREATE OR REPLACE FUNCTION check_gap (_col_name varchar)
RETRUNS TABLE ( ... here i have several columns with their type ...)
LANGUAGE plpsql
AS $function$
BEGIN
RETURN QUERY
SELECT ... a bunch of columns ...
FROM ... a bunch of tables left joined ...
WHERE _col_name = 1;
END;
$function$
;
I even tried with table_name._col_name .. though it wasn't necessary, the query (select from) works just as fine without
** I found some solutions for dynamic value but not for a dynamic column
*** I am using PostgreSQL (from DBeaver)
You need dynamic SQL for that. Pls. note that the code below is SQL injection prone.
CREATE OR REPLACE FUNCTION check_gap (col_name varchar)
RETURNS TABLE (... several columns with their type ...) LANGUAGE plpgsql AS
$$
BEGIN
RETURN QUERY EXECUTE format
(
'SELECT ... a bunch of columns ...
FROM ... a bunch of tables left joined ...
WHERE %I = 1;', col_name
);
END;
$$;
I have a table and some of the columns are "bytea" type. What I want to do is; Before any insert statement check if a column type is "bytea" then decode hex value of column value.
Is there a way to create a trigger function like below?
INSERT INTO USERS (ID, NAME, STATUS) VALUES ('0x3BEDDASTSFSFSDS', 'test', 'new')
to
INSERT INTO USERS (ID, NAME, STATUS) VALUES (decode('0x3BEDDASTSFSFSDS', 'hex'), 'test', 'new')
There is no way to intercept the input string before Postgres parses it as a bytea.
It'll be read as an "escape" format literal, i.e. the bytea will end up holding the hex digits' ASCII codepoints. You can undo this, though it's a little unpleasant:
create function trg() returns trigger language plpgsql as $$
begin
assert substring(new.id from 1 for 2) = '0x';
new.id = decode(convert_from(substring(new.id from 3), current_setting('server_encoding')), 'hex');
return new;
end
$$;
create trigger trg
before insert on users
for each row
execute procedure trg();
If you have any control over the input, just change the 0x to \x, and Postgres will convert it for you.
I implemented this function in my Postgres database: http://www.cureffi.org/2013/03/19/automatically-creating-pivot-table-column-names-in-postgresql/
Here's the function:
create or replace function xtab (tablename varchar, rowc varchar, colc varchar, cellc varchar, celldatatype varchar) returns varchar language plpgsql as $$
declare
dynsql1 varchar;
dynsql2 varchar;
columnlist varchar;
begin
-- 1. retrieve list of column names.
dynsql1 = 'select string_agg(distinct '||colc||'||'' '||celldatatype||''','','' order by '||colc||'||'' '||celldatatype||''') from '||tablename||';';
execute dynsql1 into columnlist;
-- 2. set up the crosstab query
dynsql2 = 'select * from crosstab (
''select '||rowc||','||colc||','||cellc||' from '||tablename||' group by 1,2 order by 1,2'',
''select distinct '||colc||' from '||tablename||' order by 1''
)
as ct (
'||rowc||' varchar,'||columnlist||'
);';
return dynsql2;
end
$$;
So now I can call the function:
select xtab('globalpayments','month','currency','(sum(total_fees)/sum(txn_amount)*100)::decimal(48,2)','text');
Which returns (because the return type of the function is varchar):
select * from crosstab (
'select month,currency,(sum(total_fees)/sum(txn_amount)*100)::decimal(48,2)
from globalpayments
group by 1,2
order by 1,2'
, 'select distinct currency
from globalpayments
order by 1'
) as ct ( month varchar,CAD text,EUR text,GBP text,USD text );
How can I get this function to not only generate the code for the dynamic crosstab, but also execute the result? I.e., the result when I manually copy/paste/execute is this. But I want it to execute without that extra step: the function shall assemble the dynamic query and execute it:
Edit 1
This function comes close, but I need it to return more than just the first column of the first record
Taken from: Are there any way to execute a query inside the string value (like eval) in PostgreSQL?
create or replace function eval( sql text ) returns text as $$
declare
as_txt text;
begin
if sql is null then return null ; end if ;
execute sql into as_txt ;
return as_txt ;
end;
$$ language plpgsql
usage: select * from eval($$select * from analytics limit 1$$)
However it just returns the first column of the first record :
eval
----
2015
when the actual result looks like this:
Year, Month, Date, TPV_USD
---- ----- ------ --------
2016, 3, 2016-03-31, 100000
What you ask for is impossible. SQL is a strictly typed language. PostgreSQL functions need to declare a return type (RETURNS ..) at the time of creation.
A limited way around this is with polymorphic functions. If you can provide the return type at the time of the function call. But that's not evident from your question.
Refactor a PL/pgSQL function to return the output of various SELECT queries
You can return a completely dynamic result with anonymous records. But then you are required to provide a column definition list with every call. And how do you know about the returned columns? Catch 22.
There are various workarounds, depending on what you need or can work with. Since all your data columns seem to share the same data type, I suggest to return an array: text[]. Or you could return a document type like hstore or json. Related:
Dynamic alternative to pivot with CASE and GROUP BY
Dynamically convert hstore keys into columns for an unknown set of keys
But it might be simpler to just use two calls: 1: Let Postgres build the query. 2: Execute and retrieve returned rows.
Selecting multiple max() values using a single SQL statement
I would not use the function from Eric Minikel as presented in your question at all. It is not safe against SQL injection by way of maliciously malformed identifiers. Use format() to build query strings unless you are running an outdated version older than Postgres 9.1.
A shorter and cleaner implementation could look like this:
CREATE OR REPLACE FUNCTION xtab(_tbl regclass, _row text, _cat text
, _expr text -- still vulnerable to SQL injection!
, _type regtype)
RETURNS text
LANGUAGE plpgsql AS
$func$
DECLARE
_cat_list text;
_col_list text;
BEGIN
-- generate categories for xtab param and col definition list
EXECUTE format(
$$SELECT string_agg(quote_literal(x.cat), '), (')
, string_agg(quote_ident (x.cat), %L)
FROM (SELECT DISTINCT %I AS cat FROM %s ORDER BY 1) x$$
, ' ' || _type || ', ', _cat, _tbl)
INTO _cat_list, _col_list;
-- generate query string
RETURN format(
'SELECT * FROM crosstab(
$q$SELECT %I, %I, %s
FROM %I
GROUP BY 1, 2 -- only works if the 3rd column is an aggregate expression
ORDER BY 1, 2$q$
, $c$VALUES (%5$s)$c$
) ct(%1$I text, %6$s %7$s)'
, _row, _cat, _expr -- expr must be an aggregate expression!
, _tbl, _cat_list, _col_list, _type);
END
$func$;
Same function call as your original version. The function crosstab() is provided by the additional module tablefunc which has to be installed. Basics:
PostgreSQL Crosstab Query
This handles column and table names safely. Note the use of object identifier types regclass and regtype. Also works for schema-qualified names.
Table name as a PostgreSQL function parameter
However, it is not completely safe while you pass a string to be executed as expression (_expr - cellc in your original query). This kind of input is inherently unsafe against SQL injection and should never be exposed to the general public.
SQL injection in Postgres functions vs prepared queries
Scans the table only once for both lists of categories and should be a bit faster.
Still can't return completely dynamic row types since that's strictly not possible.
Not quite impossible, you can still execute it (from a query execute the string and return SETOF RECORD.
Then you have to specify the return record format. The reason in this case is that the planner needs to know the return format before it can make certain decisions (materialization comes to mind).
So in this case you would EXECUTE the query, return the rows and return SETOF RECORD.
For example, we could do something like this with a wrapper function but the same logic could be folded into your function:
CREATE OR REPLACE FUNCTION crosstab_wrapper
(tablename varchar, rowc varchar, colc varchar,
cellc varchar, celldatatype varchar)
returns setof record language plpgsql as $$
DECLARE outrow record;
BEGIN
FOR outrow IN EXECUTE xtab($1, $2, $3, $4, $5)
LOOP
RETURN NEXT outrow
END LOOP;
END;
$$;
Then you supply the record structure on calling the function just like you do with crosstab.
Then when you all the query you would have to supply a record structure (as (col1 type, col2 type, etc) like you do with connectby.
I have created the following stored procedure, which basically receives a name of table, and a prefix. The function then finds all columns that share this prefix and returns as an output a 'select' query command ('myoneliner').
as follows:
CREATE OR REPLACE FUNCTION mytext (mytable text, myprefix text)
RETURNS text AS $myoneliner$
declare
myoneliner text;
BEGIN
SELECT 'SELECT ' || substr(cols,2,length(cols)-2) ||' FROM '||mytable
INTO myoneliner
FROM (
SELECT array(
SELECT DISTINCT quote_ident(column_name::text)
FROM information_schema.columns
WHERE table_name = mytable
AND column_name LIKE myprefix||'%'
order by quote_ident
)::text cols
) sub;
RETURN myoneliner;
END;
$myoneliner$ LANGUAGE plpgsql;
Call:
select mytext('dkj_p_k27ac','enri');
As a result of running this stored procedure and the 'select' that is following it, I get the following output at the Data Output window (all within one cell, named "mytext text"):
'SELECT enrich_d_dkj_p_k27ac,enrich_lr_dkj_p_k27ac,enrich_r_dkj_p_k27ac
FROM dkj_p_k27ac'
I would like to basically be able to take the output command line that I received as an output and execute it. In other words, I would like to be able and execute the output of my stored procedure.
How can I do so?
I tried the following:
CREATE OR REPLACE FUNCTION mytext (mytable text, myprefix text)
RETURNS SETOF RECORD AS $$
declare
smalltext text;
myoneliner text;
BEGIN
SELECT 'SELECT ' || substr(cols,2,length(cols)-2) ||' FROM '||mytable
INTO myoneliner
FROM (
SELECT array(
SELECT DISTINCT quote_ident(column_name::text)
FROM information_schema.columns
WHERE table_name = mytable
AND column_name LIKE myprefix||'%'
order by quote_ident
)::text cols
) sub;
smalltext=lower(myoneliner);
raise notice '%','my additional text '||smalltext;
RETURN QUERY EXECUTE smalltext;
END;
$$ LANGUAGE plpgsql;
Call function:
SELECT * from mytext('dkj_p_k27ac','enri');
But I'm getting the following error message, could you please advise what should I change in order for it to execute?:
ERROR: a column definition list is required for functions returning "record"
LINE 26: SELECT * from mytext('dkj_p_k27ac','enri');
********** Error **********
ERROR: a column definition list is required for functions returning "record"
SQL state: 42601
Character: 728
Your first problem was solved by using dynamic SQL with EXECUTE like Craig advised.
But the rabbit hole goes deeper:
CREATE OR REPLACE FUNCTION myresult(mytable text, myprefix text)
RETURNS SETOF RECORD AS
$func$
DECLARE
smalltext text;
myoneliner text;
BEGIN
SELECT INTO myoneliner
'SELECT '
|| string_agg(quote_ident(column_name::text), ',' ORDER BY column_name)
|| ' FROM ' || quote_ident(mytable)
FROM information_schema.columns
WHERE table_name = mytable
AND column_name LIKE myprefix||'%'
AND table_schema = 'public'; -- schema name; might be another param
smalltext := lower(myoneliner); -- nonsense
RAISE NOTICE 'My additional text: %', myoneliner;
RETURN QUERY EXECUTE myoneliner;
END
$func$ LANGUAGE plpgsql;
Major points
Don't cast the whole statement to lower case. Column names might be double-quoted with upper case letters, which are case-sensitive in this case (no pun intended).
You don't need DISTINCT in the query on information_schema.columns. Column names are unique per table.
You do need to specify the schema, though (or use another way to single out one schema), or you might be mixing column names from multiple tables of the same name in multiple schemas, resulting in nonsense.
You must sanitize all identifiers in dynamic code - including table names: quote_ident(mytable). Be aware that your text parameter to the function is case sensitive! The query on information_schema.columns requires that, too.
I untangled your whole construct to build the list of column names with string_agg() instead of the array constructor. Related answer:
Update multiple columns that start with a specific string
The assignment operator in plpgsql is :=.
Simplified syntax of RAISE NOTICE.
Core problem impossible to solve
All of this still doesn't solve your main problem: SQL demands a definition of the columns to be returned. You can circumvent this by returning anonymous records like you tried. But that's just postponing the inevitable. Now you have to provide a column definition list at call time, just like your error message tells you. But you just don't know which columns are going to be returned. Catch 22.
Your call would work like this:
SELECT *
FROM myresult('dkj_p_k27ac','enri') AS f (
enrich_d_dkj_p_k27ac text -- replace with actual column types
, enrich_lr_dkj_p_k27ac text
, enrich_r_dkj_p_k27ac text);
But you don't know number, names (optional) and data types of returned columns, not at creation time of the function and not even at call time. It's impossible to do exactly that in a single call. You need two separate queries to the database.
You could return all columns of any given table dynamically with a function using polymorphic types, because there is a well defined type for the whole table. Last chapter of this related answer:
Refactor a PL/pgSQL function to return the output of various SELECT queries