Query string argument of EXECUTE is null - postgresql

EDIT
It seems my issue is when this select statement returns null (which is the case I'm trying to handle - when it returns null, I want my new value to be -999). How can I go about doing this if it errors out whenever a null is found?
ORIGINAL
I have read every other SO post I could find regarding this error, but none of which seemed to address the root of my issue.
The error is pretty straightforward - one of my arguments within my EXECUTE statement is null. Great. However, I print out each of the values that make up my EXECUTE statement right before it gets called, and I can clearly see that none of the values are null.
Code:
CREATE FUNCTION inform_icrm_prob_flow_query(tablename text, location_id int,
product_date_str text, lead_time_start int,
lead_time_end int, first_member_id int,
last_member_id int, dest_file text)
RETURNS void AS $$
DECLARE
count int;
product_date TIMESTAMPTZ;
interval_lead_time_start text;
interval_lead_time_end text;
curr_value double precision;
query text;
BEGIN
product_date := product_date_str::TIMESTAMPTZ;
count := first_member_id;
curr_value := 0;
interval_lead_time_start := ''''|| product_date ||'''::timestamptz +
interval '''||lead_time_start||' hours''';
interval_lead_time_end := ''''|| product_date ||'''::timestamptz +
interval '''||lead_time_end||' hours'' -
interval ''6 hours''';
--create our temporary table and populate it's date column
EXECUTE 'CREATE TEMPORARY TABLE temp_table_icrm_prob_flow AS
SELECT * FROM generate_series('||interval_lead_time_start || ',' ||
interval_lead_time_end || ', ''6 hours'')
AS date_valid';
LOOP
EXIT WHEN count > last_member_id;
IF NOT EXISTS(
SELECT 'date_valid'
FROM information_schema.columns
WHERE table_name='temp_table_icrm_prob_flow'
and column_name='value'||count||'')
THEN
EXECUTE 'ALTER TABLE temp_table_icrm_prob_flow ADD COLUMN value' || count
|| ' double precision DEFAULT -999';
END IF;
raise notice 'tablename: %', tablename;
raise notice 'location_id: %', location_id;
raise notice 'product_date: %', product_date;
raise notice 'count: %', count;
query := 'SELECT value FROM '|| tablename ||'
INNER JOIN temp_table_icrm_prob_flow
ON (temp_table_icrm_prob_flow.date_valid = '|| tablename ||'.date_valid)
WHERE '|| tablename ||'.id_location = '|| location_id ||'
AND '|| tablename ||'.date_product = '''|| product_date ||'''
AND '|| tablename ||'.id_member = '|| count ||'';
EXECUTE query INTO curr_value;
EXECUTE 'UPDATE temp_table_icrm_prob_flow
SET value'|| count ||' = COALESCE('|| curr_value ||', -999)';
count := count + 1;
END LOOP;
EXECUTE 'ALTER TABLE temp_table_icrm_prob_flow DROP COLUMN date_valid';
EXECUTE 'COPY temp_table_icrm_prob_flow TO '''||dest_file||''' DELIMITER '','' CSV';
EXECUTE 'DROP TABLE temp_table_icrm_prob_flow';
END;
$$ LANGUAGE plpgsql;
Output:
NOTICE: tablename: inform_tseries_data_basin_proc_fcst_prob_flow
NOTICE: location_id: 38
NOTICE: product_date: 2015-02-05 12:00:00+00
NOTICE: count: 1
ERROR: query string argument of EXECUTE is null
CONTEXT: PL/pgSQL function inform_icrm_prob_flow_query(text,integer,text,integer,integer,integer,integer,text) line 38 at EXECUTE
If none of the variables I am passing in are null, and the only other thing referenced is a temp table that I know exists, what could be causing this error?
Note: when changing my query to:
query := 'SELECT value FROM '|| tablename ||' WHERE '|| tablename ||'.id_location = '|| location_id ||' AND '|| tablename ||'.date_product = '''|| product_date ||''' AND '|| tablename ||'.id_member = '|| count ||' AND temp_table_icrm_prob_flow.date_va lid = '|| tablename ||'.date_valid';
I get the following error:
NOTICE: tablename: inform_tseries_data_basin_proc_fcst_prob_flow
NOTICE: location_id: 38
NOTICE: product_date: 2015-02-05 12:00:00+00
NOTICE: count: 1
ERROR: missing FROM-clause entry for table "temp_table_icrm_prob_flow"
LINE 1: ..._data_basin_proc_fcst_prob_flow.id_member = 1 AND temp_table...
^
QUERY: SELECT value FROM inform_tseries_data_basin_proc_fcst_prob_flow WHERE inform_tseries_data_basin_proc_fcst_prob_flow.id_location = 38 AND inform_tseries_data_basin_proc_fcst_prob_flow.date_product = '2015-02-05 12:00:00+00' AND inform_tseries_data_basin_proc_fcst_prob_flow.id_member = 1 AND temp_table_icrm_prob_flow.date_valid = inform_tseries_data_basin_proc_fcst_prob_flow.date_valid
CONTEXT: PL/pgSQL function inform_icrm_prob_flow_query(text,integer,text,integer,integer,integer,integer,text) line 35 at EXECUTE

Sorry for small offtopic. Your code is pretty unreadable (and SQL injecttion vulnerable). There are some techniques, that you can use:
Use clause USING of EXECUTE statement for usual parameters.
DO $$
DECLARE
tablename text := 'mytab';
from_date date := CURRENT_DATE;
BEGIN
EXECUTE 'INSERT INTO ' || quote_ident(tablename) || ' VALUES($1)'
USING from_date;
END
$$;
This code will be safe (due using quote_ident function), little bit faster (due using binary value of from_date variable - removed multiple string<->date conversions and little bit more readable (because string expression is shorter).
Use function format. The building query string will be shorter and more readable (table aliases helps too):
query := format('
SELECT value
FROM %I _dtn
INNER JOIN temp_table_icrm_prob_flow t ON t.date_valid = _dtn.date_valid
WHERE _dtn.id_location = $1
AND _dtn.date_product = $2
AND _dtd.id_member = $3'
, tablename);
EXECUTE query INTO curr_value USING location_id, product_date, count;
Using variables named like important SQL keywords and identifier is wrong idea - names count, values are wrong.
The error message is clean - you are using the identifier temp_table_icrm_prob_flow.date_valid, but the table temp_table_icrm_prob_flow is not mentioned in query. The query missing JOIN part.

Related

Syntac Error when using EXECUTE command in postgresql Function

I'm trying to write a function which executes a dynamically prepared sql query and return the result as table.
I was refering to the SO answer, it mentioned to use language plpgsql, even after using it I'm getting the same syntax error.
Below is the function code provided.
CREATE OR REPLACE FUNCTION public.execute_test(
ddsmappingids text,
totalnumberofrecords bigint,
skiprecords bigint,
pagesize integer,
cid bigint)
RETURNS TABLE(sampletime timestamp without time zone, jsonstring text, rowscount bigint)
LANGUAGE 'plpgsql'
COST 100
VOLATILE PARALLEL UNSAFE
ROWS 1000
AS $BODY$
DECLARE
table_names text:='dynamictable';
sql_query text;
dsids_array int[];
sub_query text;
BEGIN
if cid>=2 then
select 'dynamictable_'||cid into table_names;
end if;
select string_to_array(ddsmappingids, ',')::int[] into dsids_array;
sub_query:='SELECT * , COUNT(*) over() AS row_count FROM (SELECT start_time, cast(jsonb_object_agg(vw.dataseries_name, ROUND(CAST(o.dbl_value as numeric), 4)) as text) AS DataSeriesValue
FROM ' || quote_ident(table_names) ||' o
join vwdds vw on o.ddmid=vw.id    
where ddmid in (' || array_to_string(dsids_array,',') || ')
and o.dbl_value is not null
GROUP BY start_time
ORDER BY start_time desc
limit ' || totalnumberofrecords || ') t offset '|| skiprecords || ' rows fetch next ' || pagesize || ' rows only;';
RAISE NOTICE 'Temporary table created';
RETURN QUERY Execute sub_query;
END
$BODY$;
Error Message when running the function
EXECUTE command, when i try with some simple query it is working fine, but the query mentioned in the code snippet it is giving syntax error.
Please help me where I'm doing wrong.

PostgreSQL: dealing with variables and apostrophes in dblink query

I am trying to execute dblink, and into it's query I need to put variables, but the problem is that dblink executes string and before it's execution "converts" variables into that string.
p_int int := 1;
p_text text;
p_apostroph text := '''';
p_sql text;
p_text := (select columnXY from table2 where id =1);
p_sql: =
'insert into table (column1,column2,column3,column4)
select 1
' || p_int || ',
' || p_apostroph || p_text || p_apostroph || ',
''text''
';
dblink_exec('connection' , p_sql);
As seen in the code problem comes with text variables (and nightmare with arrays), because I already have to manually put my text variable p_text between apostrophes, and by nature I do not know what it contains (I fill it from another table or user interface or any other source I do not control). So every time this variable contains and apostrophe it ends up with error because string get's broken. So this means I would have to come up with some complicated string analyse to counter all possibilities.
Is there any other way how to put variables into dblink query without putting them into string?
PostgreSQL 11.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5
20150623 (Red Hat 4.8.5-39), 64-bit
as suggested by #a_horse_with_no_name I tried Dollar-Quoted String
create table table1 (column1 int, column2 int, column3 text, column4 text);
create table table2 (column1 text, id int);
insert into table2 values ('unsafe ''',1);
insert into table2 values ('safe ',2);
create or replace function test (p_id int) returns integer as
$body$
declare
p_int int := 1;
p_text text;
p_apostroph text := '''';
p_sql text;
begin
p_text := (select column1 from table2 where id = p_id);
p_sql:=
$$
insert into table1(column1,column2,column3,column4)
select
1,
$$ || p_int || $$,
$$ || p_apostroph || p_text || p_apostroph || $$,
'textz'
$$
;
raise notice '%', p_sql;
perform dblink_exec('connection' , p_sql );
return 1;
end;
$body$ LANGUAGE plpgsql
select * from test (2); -- no problem
select * from test (1); -- error
[Code: 0, SQL State: 42601] ERROR: unterminated quoted string at or
near "' " Kde: while executing command on unnamed dblink connection
SQL statement "SELECT dblink_exec('connection' , p_sql )" PL/pgSQL
function s5_text_apostrop(integer) line 22 at PERFORM
You forgot a comma in the INSERT statement.
Your code is also vulnerable to SQL injection. Use format to avoid that and make your code more readable:
p_sql: = format(
'insert into atable (column1, column2, column3, column4)'
'select 1, %s, %L, %L',
p_int, p_text, 'text'
);

How to use a Function Parameter in a Cursor that's incorporated with Dynamic SQL in Postgres Functions?

Created this Postgres Function which is working fine, but the actual requirement is to pass the input parameter in the function to the Cursor which uses the dynamic SQL as follows,
The below is the Function
CREATE OR REPLACE FUNCTION ssp2_pcat.find_shift_dates (date_to_find date)
RETURNS void
LANGUAGE 'plpgsql'
COST 100
VOLATILE
AS $BODY$
DECLARE
C1 CURSOR FOR
SELECT TABLE_NAME, 'SELECT COUNT(*) FROM ' || TABLE_NAME || ' WHERE ' ||
COLUMN_NAME || ' = '||
'CASE WHEN ' || COLUMN_NAME || ' LIKE ' || '''%START%'''||' THEN
date_to_find ELSE date_to_find-1 END;' SQL_TEXT
FROM (
SELECT TABLE_NAME, COLUMN_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME IN (SELECT TABLE_NAME FROM RESET_DATES WHERE RESET_IT =
'Y') AND
UPPER(DATA_TYPE) = 'DATE'
AND (COLUMN_NAME LIKE '%START%' OR COLUMN_NAME LIKE '%END%')
AND (COLUMN_NAME NOT LIKE '%TEST%'
AND COLUMN_NAME NOT LIKE '%PCAT%'
AND COLUMN_NAME NOT LIKE '%ORDER%'
AND COLUMN_NAME NOT LIKE '%SEASON%'
AND COLUMN_NAME NOT LIKE '%_AT')
ORDER BY 1, 2) A;
END_COUNT INTEGER := 0;
START_COUNT INTEGER := 0;
TABLENAME VARCHAR(32) := 'ALFU';
l_start TIMESTAMP;
l_end TIMESTAMP;
Time_Taken VARCHAR(20);
BEGIN
l_start := clock_timestamp();
DELETE FROM SHIFT_DATES_COUNT;
FOR I IN C1 LOOP
IF I.TABLE_NAME <> TABLENAME THEN
INSERT INTO SHIFT_DATES_COUNT VALUES (TABLENAME, START_COUNT,
END_COUNT, current_timestamp::timestamp(0));
TABLENAME := I.TABLE_NAME;
END_COUNT := 0;
START_COUNT := 0;
END IF;
IF STRPOS(I.SQL_TEXT, 'END') > 0 THEN
EXECUTE I.SQL_TEXT INTO END_COUNT;
RAISE NOTICE '% ', ('END: ' || I.SQL_TEXT);
ELSE
EXECUTE I.SQL_TEXT INTO START_COUNT;
RAISE NOTICE '% ', ('START: ' || I.SQL_TEXT);
END IF;
END LOOP;
INSERT INTO SHIFT_DATES_COUNT VALUES (TABLENAME, START_COUNT, END_COUNT,
current_timestamp::timestamp(0));
RAISE NOTICE '% ', ('INSERT INTO SHIFT_DATES_COUNT Done...');
l_end := clock_timestamp();
Time_Taken := (l_end-l_start);
RAISE NOTICE '% ', ('FIND_SHIFT_DATES Took: ' || Time_Taken );
END;
$BODY$;
Please let me know how can I use the date_to_find input parameter in the Dynamic SQL in the Cursor in the above Function.
You can use unbound cursor, clause fetch to get data from cursor, and exit when not found to finish, like:
CREATE OR REPLACE FUNCTION example (p_name text) RETURNS void LANGUAGE 'plpgsql' AS $$
DECLARE
C1 refcursor;
res record;
BEGIN
OPEN c1 FOR EXECUTE 'SELECT * FROM pg_database WHERE datname like ''%'||p_name||'%''';
LOOP
FETCH c1 INTO res;
EXIT WHEN not found;
raise notice 'value datname: %',res.datname;
END LOOP;
CLOSE c1;
RETURN;
END; $$;
--in my case
select example ('test')
NOTICE: value datname: test
NOTICE: value datname: test_msmov
NOTICE: value datname: test_resources
NOTICE: value datname: test_load_table
NOTICE: value datname: test_resources2
Total query runtime: 63 msec
1 row retrieved.
You can use EXECUTE clause for open cursor, see the documentation of PostgreSQL
https://www.postgresql.org/docs/10/plpgsql-cursors.html#PLPGSQL-CURSOR-OPENING
Example:
OPEN curs1 FOR EXECUTE format('SELECT * FROM %I WHERE col1 = $1',tabname) USING keyvalue;

ERROR: missing FROM-clause entry for table "new"

I have a parent table layer_1_ and a number of child tables layer_1_points, layer_1_linestrings etc. which contain some geometry data. Each child table has its own geometry constraint. So, for example, layer_1_points has this constraint:
CONSTRAINT enforce_geotype_geom_geom CHECK (geometrytype(geom) = 'POINT'::text)
Whereas layer_1_linestrings table has this constraint:
CONSTRAINT enforce_geotype_geom_geom CHECK (geometrytype(geom) = 'LINESTRING'::text)
Many other layer tables have similar names: layer_2_, layer_3_, ..., layer_N_. And all of them have their own child tables. What I want to achive is that when a user inserts to a parent table (layer_N_), then this insert statement should be forwarded to a particular child table (layer_N_points etc.). So, for example, when I do:
INSERT INTO layer_1_ (geom) VALUES(ST_GeomFromText('POINT(0 0)', 3857))
I should actually insert to layer_1_points, because geom type is POINT. To achive all this I created this trigger function and the trigger itself:
CREATE OR REPLACE FUNCTION trigger_layer_insert()
RETURNS trigger AS
$$
DECLARE
var_geomtype text;
table_name text;
layer_id text := (TG_ARGV[0])::text;
BEGIN
var_geomtype := geometrytype(NEW.geom);
IF var_geomtype = 'POINT' THEN
table_name := (SELECT concat ('layer_', layer_id, '_points'));
ELSIF var_geomtype = 'MULTIPOINT' THEN
table_name := (SELECT concat ('layer_', layer_id, '_multipoints'));
ELSIF var_geomtype = 'LINESTRING' THEN
table_name := (SELECT concat ('layer_', layer_id, '_linestrings'));
ELSIF var_geomtype = 'MULTILINESTRING' THEN
table_name := (SELECT concat ('layer_', layer_id, '_multilinestrings'));
ELSIF var_geomtype = 'POLYGON' THEN
table_name := (SELECT concat ('layer_', layer_id, '_polygons'));
ELSIF var_geomtype = 'MULTIPOLYGON' THEN
table_name := (SELECT concat ('layer_', layer_id, '_multipolygons'));
END IF;
EXECUTE '
INSERT INTO ' || table_name || '
SELECT * FROM (SELECT NEW.*) AS t
';
RETURN NULL;
END;
$$
LANGUAGE 'plpgsql' VOLATILE;
CREATE TRIGGER trigger_layer_1_ BEFORE INSERT
ON layer_1_ FOR EACH ROW
EXECUTE PROCEDURE trigger_layer_insert(1);
However, when I do actual insert like:
INSERT INTO layer_1_ (geom) VALUES(ST_GeomFromText('POINT(0 0)', 3857))
I get an error message:
ERROR: missing FROM-clause entry for table "new"
LINE 3: SELECT * FROM (SELECT NEW.*) AS t
^
QUERY:
INSERT INTO layer_1_points
SELECT * FROM (SELECT NEW.*) AS t
So, what is wrong with SELECT NEW.* and how can I fix it? Thanks!
EDIT
I also tried this:
EXECUTE '
INSERT INTO ' || table_name || '
SELECT * FROM (SELECT NEW.*) AS t
' USING NEW;
But it has no effect.
When you execute something using PLPGSQL statement EXECUTE it runs in the different context so local variables is not visible there. To pass variable(s) the EXECUTE '<SQL script>' USING <variables list>; form is used:
EXECUTE 'insert into table(field1, field2) values ($1, $2)' USING var1, var2;
So the statement should be:
EXECUTE 'INSERT INTO ' || table_name || ' SELECT * FROM SELECT $1.*) AS t'
USING NEW;
But much more secure is using format function:
execute format('INSERT INTO %I SELECT * FROM SELECT $1.*) AS t', table_name)

how to reference a schema variable in plpgsql

I am trying to learn plpgsql code to automate some data cleaning in a database.
My current task is to replace all of '999' values in numeric fields with 'NaN'. What I am trying to do is:
1) find all columns in a schema that are numeric
2) loop through these and use 'update/replace'
My code is below. I think my main problem is finding out how to reference the schema.table in the update statement (but I am sure there are other things I have not done too well).
The error that I am getting is that the relation is not recognised.
Any assistance would be appreciated
Becky
CREATE OR REPLACE FUNCTION household.nodata_replace(schemanm text)
RETURNS VOID as $$
DECLARE
cname text;
tname text;
BEGIN
--FOR col IN
for cname,tname in SELECT column_name::text,table_name::text FROM information_schema.columns
where table_schema = schemanm and data_type in ('integer','double precision')
LOOP
RAISE NOTICE 'cname is: % from %', cname, tname;
EXECUTE 'update '||schemanm::regclass||'.' ||tname::regclass||
' set ' || quote_ident(cname) ||' = replace(' || quote_ident(cname) ||', 999, NaN);';
END LOOP;
END;
$$
LANGUAGE plpgsql;
try to rework your query like following:
EXECUTE 'update '|| (schemanm||'.'||tname)::regclass ||' set ' || quote_ident(cname) ||' = ''NaN'' WHERE '|| quote_ident(cname) ||' = 999;'
because casting to regclass is trying to search within registered relations. and schema is not relation.
or you can
EXECUTE 'update '|| quote_ident(schemanm)||'.'||quote_ident(tname) ||' set ' || quote_ident(cname) ||' = ''NaN'' WHERE '|| quote_ident(cname) ||' = 999;'
I would rather use format() for this. The placeholder %I takes care of properly quoting identifiers if needed.
replace() is for string manipulation not for replacing numbers. To assign the value NaN use set xxx = 'NaN' but you cannot do this for an integer value. Integers do not support NaN
So your dynamic SQL boils down to:
execute format('update %I.%I set %I = ''NaN'' where %I = 999, schemanm, tname, cname, cname);
But you will need to change your where clause to not include integer columns. You probably want to include numeric and real as well:
and data_type in ('numeric','double precision', 'real')
If you just want to mark the "absence of information", I would rather store null in those columns. In that case you don't need to distinguish between the different data types:
execute format('update %I.%I set %I = null where %I = 999, schemanm, tname, cname, cname);