PostgreSQL: dealing with variables and apostrophes in dblink query - postgresql

I am trying to execute dblink, and into it's query I need to put variables, but the problem is that dblink executes string and before it's execution "converts" variables into that string.
p_int int := 1;
p_text text;
p_apostroph text := '''';
p_sql text;
p_text := (select columnXY from table2 where id =1);
p_sql: =
'insert into table (column1,column2,column3,column4)
select 1
' || p_int || ',
' || p_apostroph || p_text || p_apostroph || ',
''text''
';
dblink_exec('connection' , p_sql);
As seen in the code problem comes with text variables (and nightmare with arrays), because I already have to manually put my text variable p_text between apostrophes, and by nature I do not know what it contains (I fill it from another table or user interface or any other source I do not control). So every time this variable contains and apostrophe it ends up with error because string get's broken. So this means I would have to come up with some complicated string analyse to counter all possibilities.
Is there any other way how to put variables into dblink query without putting them into string?
PostgreSQL 11.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5
20150623 (Red Hat 4.8.5-39), 64-bit
as suggested by #a_horse_with_no_name I tried Dollar-Quoted String
create table table1 (column1 int, column2 int, column3 text, column4 text);
create table table2 (column1 text, id int);
insert into table2 values ('unsafe ''',1);
insert into table2 values ('safe ',2);
create or replace function test (p_id int) returns integer as
$body$
declare
p_int int := 1;
p_text text;
p_apostroph text := '''';
p_sql text;
begin
p_text := (select column1 from table2 where id = p_id);
p_sql:=
$$
insert into table1(column1,column2,column3,column4)
select
1,
$$ || p_int || $$,
$$ || p_apostroph || p_text || p_apostroph || $$,
'textz'
$$
;
raise notice '%', p_sql;
perform dblink_exec('connection' , p_sql );
return 1;
end;
$body$ LANGUAGE plpgsql
select * from test (2); -- no problem
select * from test (1); -- error
[Code: 0, SQL State: 42601] ERROR: unterminated quoted string at or
near "' " Kde: while executing command on unnamed dblink connection
SQL statement "SELECT dblink_exec('connection' , p_sql )" PL/pgSQL
function s5_text_apostrop(integer) line 22 at PERFORM

You forgot a comma in the INSERT statement.
Your code is also vulnerable to SQL injection. Use format to avoid that and make your code more readable:
p_sql: = format(
'insert into atable (column1, column2, column3, column4)'
'select 1, %s, %L, %L',
p_int, p_text, 'text'
);

Related

Postgresql : null values cannot be formatted as an SQL identifier

how to resolve this ERROR: null values cannot be formatted as an SQL identifier when trying to select my function:
select ws_sls_core.ars_pricing_test()
ERROR: null values cannot be formatted as an SQL identifier
CONTEXT: SQL statement "select string_agg(distinct format('(props ->> %L) as %I', w_order_line_d.matl_grp2_desc, w_order_line_d.matl_grp2_desc), ', ')
from ws_sls_core.w_support_pricing_d
left join ws_sls_core.w_order_line_d
on w_support_pricing_d.svc_pricing_type = w_order_line_d.matl_grp2_cd"
PL/pgSQL function ars_pricing_test() line 6 at SQL statement
I have checked and the query in use doesn't produce any NULL
select * from ws_sls_core.w_support_pricing_d where svc_pricing_type is
null
I have tried the below-mentioned code without JOIN and it works fine, I need an additional column and have to use join, and I am only seeing this error.
Here is my complete code
CREATE OR REPLACE FUNCTION ws_sls_core.ars_pricing_test(
)
RETURNS boolean
LANGUAGE 'plpgsql'
COST 100
VOLATILE
AS $BODY$
declare
l_sql text;
l_columns text;
begin
select string_agg(distinct format('(props ->> %L) as %I', w_order_line_d.matl_grp2_desc, w_order_line_d.matl_grp2_desc), ', ')
into l_columns
from ws_sls_core.w_support_pricing_d
left join ws_sls_core.w_order_line_d
on w_support_pricing_d.svc_pricing_type = w_order_line_d.matl_grp2_cd;
-- and A.svc_pricing_type is not null;
l_sql :=
'create or replace view ars_pricing_test as
select w_support_pricing_d.item_num, '||l_columns||'
from (
select w_support_pricing_d.item_num, json_object_agg(w_order_line_d.matl_grp2_desc,w_support_pricing_d.mnth_maint_price) as props
from ws_sls_core.w_support_pricing_d
left join ws_sls_core.w_order_line_d
on w_support_pricing_d.svc_pricing_type = w_order_line_d.matl_grp2_cd
group by w_support_pricing_d.item_num
) t';
execute l_sql;
return true;
end;
$BODY$;
ALTER FUNCTION ws_sls_core.ars_pricing_test()

postgresql for loop script in text form can not be executed

I am trying to write function in postgresql, that creates temp_table with columns table_name text, table_rec jsonb and fill it through for loop with table names from my table containing names of tables and records in json. I have the for loop in string and I want to execute it. But it doesnt work.
I have variable rec record, sql_query text and tab_name text and I want to do this:
CREATE OR REPLACE FUNCTION public.test51(
)
RETURNS TABLE(tabel_name text, record_json jsonb)
LANGUAGE 'plpgsql'
COST 100
VOLATILE
ROWS 1000
AS $BODY$
declare
rec record;
tabel_name text;
tabel_names text[];
counter integer := 1;
sql_query text;
limit_for_sending integer;
rec_count integer;
begin
select into tabel_names array(select "TABLE_NAME" from public."TABLES");
create temp table temp_tab(tab_nam text, recik jsonb);
while array_length(tabel_names, 1) >= counter loop
tabel_name := '"' || tabel_names[counter] || '"';
select into limit_for_sending "TABLE_LIMIT_FOR_SENDING_DATA" from public."TABLES" where "TABLE_NAME" = tabel_name;
sql_query := 'select count(*) from public.' || tabel_name;
execute sql_query into rec_count;
if (rec_count >= limit_for_sending and limit_for_sending is not null) then
sql_query := 'for rec in select * from public.' || tabel_name || '
loop
insert into temp_tab
select ' || tabel_name || ', to_jsonb(rec);
end loop';
execute sql_query;
end if;
counter := counter + 1;
end loop;
return query
select * from temp_tabik;
drop table temp_tabik;
end;
$BODY$;
Thank you for response.
It seems you have some table that contains the information for which tables you want to return all rows as JSONB. And that meta-table also contains a column that sets a threshold under which the rows should not be returned.
You don't need the temp table or an array to store the table names. You can iterate through the query on the TABLES table and run the dynamic SQL directly in that loop.
return query in PL/pgSQL doesn't terminate the function, it just appends the result of the query to the result of the function.
Dynamic SQL is best created using the format() function because it is easier to read and using the %I placeholder will properly deal with quoted identifiers (which is really important as you are using those dreaded upper case table names)
As far as I can tell, your function can be simplified to:
CREATE OR REPLACE FUNCTION public.test51()
RETURNS TABLE(tabel_name text, record_json jsonb)
LANGUAGE plpgsql
AS
$BODY$
declare
rec record;
sql_query text;
rec_count bigint;
begin
for rec in
select "TABLE_NAME" as table_name, "TABLE_LIMIT_FOR_SENDING_DATA" as rec_limit
from public."TABLES"
loop
if rec.rec_limit is not null then
execute format('select count(*) from %I', rec.table_name)
into rec_count;
end if;
if (rec.rec_limit is not null and rec_count >= rec.rec_limit) then
sql_query := format('select %L, to_jsonb(t) from %I as t', rec.table_name, rec.table_name);
return query execute sql_query;
end if;
end loop;
end;
$BODY$;
Some notes
the language name is an identifier and should not be enclosed in single quotes. This syntax is deprecated and might be removed in a future version so don't get used to it.
you should really avoid those dreaded quoted identifiers. They are much more trouble than they are worth it. See the Postgres wiki for details.

How to convert a PL/PgSQL procedure into a dynamic one?

I am trying to write a plpgsql procedure to perform spatial tiling of a postGIS table. I can perform the operation successfully using the following procedure in which the table names are hardcoded. The procedure loops through the tiles in tile_table and for each tile clips the area_table and inserts it into split_table.
CREATE OR REPLACE PROCEDURE splitbytile()
AS $$
DECLARE
tile RECORD;
BEGIN
FOR tile IN
SELECT tid, geom FROM test_tiles ORDER BY tid
LOOP
INSERT INTO split_table (id, areaname, ttid, geom)
SELECT id, areaname, tile.tid,
CASE WHEN st_within(base.geom, tile.geom) THEN st_multi(base.geom)
ELSE st_multi(st_intersection(base.geom, tile.geom)) END as geom
FROM area_table as base
WHERE st_intersects(base.geom, tile.geom);
COMMIT;
END LOOP;
END;
$$ LANGUAGE 'plpgsql';
Having tested this successfully, now I need to convert it to a dynamic procedure where I can provide the table names as parameters. I tried the following partial conversion, using format() for inside of loop:
CREATE OR REPLACE PROCEDURE splitbytile(in_table text, grid_table text, split_table text)
AS $$
DECLARE
tile RECORD;
BEGIN
FOR tile IN
EXECUTE format('SELECT tid, geom FROM %I ORDER BY tid', grid_table)
LOOP
EXECUTE
FORMAT(
'INSERT INTO %1$I (id, areaname, ttid, geom)
SELECT id, areaname, tile.tid,
CASE WHEN st_within(base.geom, tile.geom) THEN st_multi(base.geom)
ELSE st_multi(st_intersection(base.geom, tile.geom)) END as geom
FROM %2$I as base
WHERE st_intersects(base.geom, tile.geom)', split_table, in_table
);
COMMIT;
END LOOP;
END;
$$ LANGUAGE 'plpgsql';
But it throws an error
missing FROM-clause entry for table "tile"
So, how can I convert the procedure to a dynamic one? More specifically, how can I use the record data type (tile) returned by the for loop inside the loop? Note that it works when format is not used.
You can use EXECUTE ... USING to supply parameters to a dynamic query:
EXECUTE
format(
'SELECT r FROM %I WHERE c = $1.val',
table_name
)
INTO result_var
USING record_var;
The first argument to USING will be used for $1, the second for $2 and so on.
See the documentation for details.
Personally I use somehow different way to create dynamic functions. By concatination and execute function. You can also do like this.
CREATE OR REPLACE FUNCTION splitbytile()
RETURNS void AS $$
declare
result1 text;
table_name text := 'test_tiles';
msi text := '+7 9912 231';
msi text := 'Hello world';
code text := 'code_name';
_operator_id integer := 2;
begin
query1 := 'SELECT msisdn from ' || table_name || ' where msisdn = ''' || msi::text ||''';';
query2 := 'INSERT INTO ' || table_name || '(msisdn,usage,body,pr_code,status,sent_date,code_type,operator_id)
VALUES( ''' || msi::text || ''',' || true || ',''' || _body::text || ''',''' || code::text || ''',' || false || ',''' || time_now || ''',' || kod_type || ',' || _operator_id ||');';
execute query1 into result1;
execute query2;
END;
$function$
You just make your query as text then anywhere you want you can execute it. Maybe by checking result1 value inside If statement or smth like that.

Check to see if a record exists postgres function

I am attempting to create a function that will determine if a record exists for our applications developers to help simplify things.
CREATE FUNCTION records_exist(schema_name VARCHAR(255), table_name VARCHAR(255), field_name VARCHAR(255), field_value VARCHAR(255))
RETURNS BOOLEAN
LANGUAGE plpgsql
AS $$
DECLARE
_schema_name ALIAS FOR $1;
_table_name ALIAS FOR $2;
_field_name ALIAS FOR $3;
_field_value ALIAS FOR $4;
_sql_string VARCHAR(5000);
BEGIN
_sql_string= 'SELECT EXISTS(SELECT 1 FROM ' || _schema_name || '.' || _table_name || ' WHERE ' || _field_name || '=' || _field_value || ');';
RETURN BOOLEAN EXECUTE _sql_string;
--RETURN TABLE EXECUTE _sql_string; doesn't work
END
$$;
The following should work, but I keep getting ERROR: syntax error at or near "EXECUTE"
Please let me know the error of my ways.
also, your dynamic string is bad, better use:
select format('select exists from %I.%I where %I = %L',schema_name,table_name,field_name, field_value) into _sql_string;
also, you realize your _field_value has no check for data type?..
so in short, it could be smth similar to:
db=# CREATE or replace FUNCTION records_exist(schema_name VARCHAR(255), table_name VARCHAR(255), field_name VARCHAR(255), field_value VARCHAR(255))
RETURNS BOOLEAN
LANGUAGE plpgsql
AS $$
DECLARE
_sql text;
_b boolean;
BEGIN
_sql := format('select exists (select null from %I.%I where %I = %L)',schema_name,table_name,field_name, field_value);
execute _sql into _b;
return _b;
END
$$;
CREATE FUNCTION
Time: 10.680 ms
db=# select * from records_exist('pg_catalog','pg_database','datname','postgres'); records_exist
---------------
t
(1 row)
Time: 59.472 ms

Query string argument of EXECUTE is null

EDIT
It seems my issue is when this select statement returns null (which is the case I'm trying to handle - when it returns null, I want my new value to be -999). How can I go about doing this if it errors out whenever a null is found?
ORIGINAL
I have read every other SO post I could find regarding this error, but none of which seemed to address the root of my issue.
The error is pretty straightforward - one of my arguments within my EXECUTE statement is null. Great. However, I print out each of the values that make up my EXECUTE statement right before it gets called, and I can clearly see that none of the values are null.
Code:
CREATE FUNCTION inform_icrm_prob_flow_query(tablename text, location_id int,
product_date_str text, lead_time_start int,
lead_time_end int, first_member_id int,
last_member_id int, dest_file text)
RETURNS void AS $$
DECLARE
count int;
product_date TIMESTAMPTZ;
interval_lead_time_start text;
interval_lead_time_end text;
curr_value double precision;
query text;
BEGIN
product_date := product_date_str::TIMESTAMPTZ;
count := first_member_id;
curr_value := 0;
interval_lead_time_start := ''''|| product_date ||'''::timestamptz +
interval '''||lead_time_start||' hours''';
interval_lead_time_end := ''''|| product_date ||'''::timestamptz +
interval '''||lead_time_end||' hours'' -
interval ''6 hours''';
--create our temporary table and populate it's date column
EXECUTE 'CREATE TEMPORARY TABLE temp_table_icrm_prob_flow AS
SELECT * FROM generate_series('||interval_lead_time_start || ',' ||
interval_lead_time_end || ', ''6 hours'')
AS date_valid';
LOOP
EXIT WHEN count > last_member_id;
IF NOT EXISTS(
SELECT 'date_valid'
FROM information_schema.columns
WHERE table_name='temp_table_icrm_prob_flow'
and column_name='value'||count||'')
THEN
EXECUTE 'ALTER TABLE temp_table_icrm_prob_flow ADD COLUMN value' || count
|| ' double precision DEFAULT -999';
END IF;
raise notice 'tablename: %', tablename;
raise notice 'location_id: %', location_id;
raise notice 'product_date: %', product_date;
raise notice 'count: %', count;
query := 'SELECT value FROM '|| tablename ||'
INNER JOIN temp_table_icrm_prob_flow
ON (temp_table_icrm_prob_flow.date_valid = '|| tablename ||'.date_valid)
WHERE '|| tablename ||'.id_location = '|| location_id ||'
AND '|| tablename ||'.date_product = '''|| product_date ||'''
AND '|| tablename ||'.id_member = '|| count ||'';
EXECUTE query INTO curr_value;
EXECUTE 'UPDATE temp_table_icrm_prob_flow
SET value'|| count ||' = COALESCE('|| curr_value ||', -999)';
count := count + 1;
END LOOP;
EXECUTE 'ALTER TABLE temp_table_icrm_prob_flow DROP COLUMN date_valid';
EXECUTE 'COPY temp_table_icrm_prob_flow TO '''||dest_file||''' DELIMITER '','' CSV';
EXECUTE 'DROP TABLE temp_table_icrm_prob_flow';
END;
$$ LANGUAGE plpgsql;
Output:
NOTICE: tablename: inform_tseries_data_basin_proc_fcst_prob_flow
NOTICE: location_id: 38
NOTICE: product_date: 2015-02-05 12:00:00+00
NOTICE: count: 1
ERROR: query string argument of EXECUTE is null
CONTEXT: PL/pgSQL function inform_icrm_prob_flow_query(text,integer,text,integer,integer,integer,integer,text) line 38 at EXECUTE
If none of the variables I am passing in are null, and the only other thing referenced is a temp table that I know exists, what could be causing this error?
Note: when changing my query to:
query := 'SELECT value FROM '|| tablename ||' WHERE '|| tablename ||'.id_location = '|| location_id ||' AND '|| tablename ||'.date_product = '''|| product_date ||''' AND '|| tablename ||'.id_member = '|| count ||' AND temp_table_icrm_prob_flow.date_va lid = '|| tablename ||'.date_valid';
I get the following error:
NOTICE: tablename: inform_tseries_data_basin_proc_fcst_prob_flow
NOTICE: location_id: 38
NOTICE: product_date: 2015-02-05 12:00:00+00
NOTICE: count: 1
ERROR: missing FROM-clause entry for table "temp_table_icrm_prob_flow"
LINE 1: ..._data_basin_proc_fcst_prob_flow.id_member = 1 AND temp_table...
^
QUERY: SELECT value FROM inform_tseries_data_basin_proc_fcst_prob_flow WHERE inform_tseries_data_basin_proc_fcst_prob_flow.id_location = 38 AND inform_tseries_data_basin_proc_fcst_prob_flow.date_product = '2015-02-05 12:00:00+00' AND inform_tseries_data_basin_proc_fcst_prob_flow.id_member = 1 AND temp_table_icrm_prob_flow.date_valid = inform_tseries_data_basin_proc_fcst_prob_flow.date_valid
CONTEXT: PL/pgSQL function inform_icrm_prob_flow_query(text,integer,text,integer,integer,integer,integer,text) line 35 at EXECUTE
Sorry for small offtopic. Your code is pretty unreadable (and SQL injecttion vulnerable). There are some techniques, that you can use:
Use clause USING of EXECUTE statement for usual parameters.
DO $$
DECLARE
tablename text := 'mytab';
from_date date := CURRENT_DATE;
BEGIN
EXECUTE 'INSERT INTO ' || quote_ident(tablename) || ' VALUES($1)'
USING from_date;
END
$$;
This code will be safe (due using quote_ident function), little bit faster (due using binary value of from_date variable - removed multiple string<->date conversions and little bit more readable (because string expression is shorter).
Use function format. The building query string will be shorter and more readable (table aliases helps too):
query := format('
SELECT value
FROM %I _dtn
INNER JOIN temp_table_icrm_prob_flow t ON t.date_valid = _dtn.date_valid
WHERE _dtn.id_location = $1
AND _dtn.date_product = $2
AND _dtd.id_member = $3'
, tablename);
EXECUTE query INTO curr_value USING location_id, product_date, count;
Using variables named like important SQL keywords and identifier is wrong idea - names count, values are wrong.
The error message is clean - you are using the identifier temp_table_icrm_prob_flow.date_valid, but the table temp_table_icrm_prob_flow is not mentioned in query. The query missing JOIN part.