I have a long and complex plpgsql function that creates a bunch of temporary tables nested within a while statement to get the optimal result. When the condition has been met I insert the result into an existing table, the function is far to long to post here but this is an example:
CREATE OR REPLACE FUNCTION public.test_function(id_input integer, val_input numeric)
RETURNS VOID AS
$BODY$
DECLARE
id_input numeric = $1;
val_input numeric = $2;
BEGIN
WHILE test_val < 0
LOOP
CREATE TEMP TABLE temp_table AS
SELECT a.existing_val - val_input AS new_val
FROM existing_table a
WHERE a.id = id_input;
test_val := (SELECT new_val FROM temp_table);
val_input := val_input + 1;
END LOOP;
INSERT INTO output_table (id, new_val)
SELECT a.id, a.new_val
FROM temp_table a;
END;
$BODY$
LANGUAGE plpgsql;
The function works if I call it like this SELECT test_function(1, 1000) However I would like run this function on a table with 60,000+ rows, like this:
SELECT test_function(a.id, a.val_input)
FROM data_table a;
It works when I use a subset of the data_table, say 1000 rows. However when I run it on the full table (60,000+ rows) I get the following error "AbortTransaction while in COMMIT state". After some reading I found out COMMITS, so in my case the inserts do not occur until the function has finished running which takes about 4 hours. So does anyone know what is going on?
As a workaround I tried nesting the function in a DO statement so the inserts are committed straight away:
DO
$do$
DECLARE
r data_table%rowtype;
BEGIN
FOR r IN
SELECT * FROM data_table
LOOP
SELECT public.test_function(r.id, r.val_input);
END LOOP;
END
$do$;
However then I get the following error "ERROR: query has no destination for result data", which I guess means I need to rewrite the function to use PERFORM instead of SELECT. However I have not had any luck with this as yet.
Any ideas?
Since you are not interested in the function result, you should use
PERFORM public.test_function(r.id, r.val_input);
instead of
SELECT public.test_function(r.id, r.val_input);
The latter syntax would only work if you add INTO some_variable as a destination for the query result.
Thank you all for your suggestions. I ended up using Jim Jones's suggestion and converting the function to a procedure which allowed me to use COMMIT after I did the INSERT. I also followed Jeremy's suggestion and moved from temp tables to CTE's. This solved the problem for me.
Related
I have a function that uses RECORD to temporarily store the data. I can use it - it's fine. My problem is that I can't hardcode columns I need to get from the RECORD. I must do it dynamically. Something line:
DECLARE
r1 RECORD;
r2 RECORD;
BEGIN
for r1 in Select column_name
from columns_to_process
where process_now = True
loop
for r2 in Select *
from my_data_table
where whatever
loop
-----------------------------
here I must call column by its name that is unknown at design time
-----------------------------
... do something with
r2.(r1.column_name)
end loop;
end loop;
END;
Does anyone know how to do it?
best regards
M
There is no need to select the all the qualifying rows and compute the total in a loop. Actually when working with SQL try to drop the word loop for your vocabulary; instead just use sum(column_name) in the select. The issue here is that you do not know what column to sum when the query is written, and all structural components(table names, columns names, operators, etc) must be known before submitting. You cannot use a variable for a structural component - in this case a column name. To do that you must use dynamic sql - i.e. SQL statement built by the process. The following accomplishes that: See example here.
create or replace function sum_something(
the_something text -- column name
, for_id my_table.id%type -- my_table.id
)
returns numeric
language plpgsql
as $$
declare
k_query_base constant text :=
$STMT$ Select sum(%I) from my_table where id = %s; $STMT$;
l_query text;
l_sum numeric;
begin
l_query = format(k_query_base, the_something, for_id);
raise notice E'Rumming Statememt:\n %',l_query; -- for prod raise Log
execute l_query into l_sum;
return l_sum;
end;
$$;
Well, after some time I figured out that I could use temporary table instead of RECORD. Doing so gives me all advantages of using dynamic queries so I can call any column by its name.
DECLARE
_my_var bigint;
BEGIN
create temporary table _my_temp_table as
Select _any, _column, _you, _need
from _my_table
where whatever = something;
execute 'Select ' || _any || ' from _my_temp_table' into _my_var;
... do whatever
END;
However I still believe that there should be a way to call records field by it's name.
I have this question, I was doing some migration from SQL Server to PostgreSQL 12.
The scenario, I am trying to accomplish:
The function should have a RETURN Statement, be it with SETOF 'tableType' or RETURN TABLE ( some number of columns )
The body starts with a count of records, if there is no record found based on input parameters, then simply Return Zero (0), else, return the entire set of record defined in the RETURN Statement.
The Equivalent part in SQL Server or Oracle is: They can just put a SELECT Statement inside a Procedure to accomplish this. But, its a kind of difficult in case of PostgreSQL.
Any suggestion, please.
What I could accomplish still now - If no record found, it will simply return NULL, may be using PERFORM, or may be selecting NULL as column name for the returning tableType columns.
I hope I am clear !
What I want is something like -
============================================================
CREATE OR REPLACE FUNCTION public.get_some_data(
id integer)
RETURNS TABLE ( id_1 integer, name character varying )
LANGUAGE 'plpgsql'
AS $BODY$
DECLARE
p_id alias for $1;
v_cnt integer:=0;
BEGIN
SELECT COUNT(1) FROM public.exampleTable e
WHERE id::integer = e.id::integer;
IF v_cnt= 0 THEN
SELECT 0;
ELSE
SELECT
a.id, a.name
public.exampleTable a
where a.id = p_id;
END;
$BODY$;
If you just want to return a set of a single table, using returns setof some_table is indeed the easiest way. The most basic SQL function to do that would be:
create function get_data()
returns setof some_table
as
$$
select *
from some_table;
$$
language sql;
PL/pgSQL isn't really necessary to put a SELECT statement into a function, but if you need to do other things, you need to use RETURN QUERY in a PL/pgSQL function:
create function get_data()
returns setof some_table
as
$$
begin
return query
select *
from some_table;
end;
$$
language plpgsql;
A function as exactly one return type. You can't have a function that sometimes returns an integer and sometimes returns thousands of rows with a dozen columns.
The only thing you could do, if you insist on returning something is something like this:
create function get_data()
returns setof some_table
as
$$
begin
return query
select *
from some_table;
if not found then
return query
select (null::some_table).*;
end if;
end;
$$
language plpgsql;
But I would consider the above an extremely ugly and confusing (not to say stupid) solution. I certainly wouldn't let that pass through a code review.
The caller of the function can test if something was returned in the same way I implemented that ugly hack: check the found variable after using the function.
One more hack to get as close as possible to what you want. But I will repeat what others have told you: You cannot do what you want directly. Just because MS SQL Server lets you get away poor coding does not mean Postgres is obligated to do so. As the link by #a_horse_with_no_name implies converting code is easy, once you migrate how you think about the problem in the first place. The closest you can get is return a tuple with a 0 id. The following is one way.
create or replace function public.get_some_data(
p_id integer)
returns table ( id integer, name character varying )
language plpgsql
as $$
declare
v_at_least_one boolean = false;
v_exp_rec record;
begin
for v_exp_rec in
select a.id, a.name
from public.exampletable a
where a.id = p_id
union all
select 0,null
loop
if v_exp_rec.id::integer > 0
or (v_exp_rec.id::integer = 0 and not v_at_least_one)
then
id = v_exp_rec.id;
name = v_exp_rec.name;
return next;
v_at_least_one = true;
end if;
end loop ;
return;
end
$$;
But that is still just a hack and assumes there in not valid row with id=0. A much better approach would by for the calling routing to check what the function returns (it has to do that in one way or another anyway) and let the function just return the data found instead of making up data. That is that mindset shift. Doing that you can reduce this function to a simple select statement:
create or replace function public.get_some_data2(
p_id integer)
returns table ( id integer, name character varying )
language sql strict
as $$
select a.id, a.name
from public.exampletable a
where a.id = p_id;
$$;
Or one of the other solutions offered.
I have a trigger function that is called by several tables when COLUMN A is updated, so that COLUMN B can be updated based on value from a different function. (More complicated to explain than it really is). The trigger function takes in col_a and col_b since they are different for the different tables.
IF needs_updated THEN
sql = format('($1).%2$s = dbo.foo(($1).%1$s); ', col_a, col_b);
EXECUTE sql USING NEW;
END IF;
When I try to run the above, the format produces this sql:
($1).NameText = dbo.foo(($1).Name);
When I execute the SQL with the USING I am expecting something like this to happen (which works when executed straight up without dynamic sql):
NEW.NameText = dbo.foo(NEW.Name);
Instead I get:
[42601] ERROR: syntax error at or near "$1"
How can I dynamically update the column on the record/composite type NEW?
This isn't going to work because NEW.NameText = dbo.foo(NEW.Name); isn't a correct sql query. And I cannot think of the way you could dynamically update variable attribute of NEW. My suggestion is to explicitly define behaviour for each of your tables:
IF TG_TABLE_SCHEMA = 'my_schema' THEN
IF TG_TABLE_NAME = 'my_table_1' THEN
NEW.a1 = foo(NEW.b1);
ELSE IF TG_TABLE_NAME = 'my_table_2' THEN
NEW.a2 = foo(NEW.b2);
... etc ...
END IF;
END IF;
First: This is a giant pain in plpgsql. So my best recommendation is to do this in some other PL, such as plpythonu or plperl. Doing this in either of those would be trivial. Even if you don't want to do the whole trigger in another PL, you could still do something like:
v_new RECORD;
BEGIN
v_new := plperl_function(NEW, column_a...)
The key to doing this in plpgsql is creating a CTE that has what you need in it:
c_new_old CONSTANT text := format(
'WITH
NEW AS (SELECT (r).* FROM (SELECT ($1)::%1$s r) s)
, OLD AS (SELECT (r).* FROM (SELECT ($2)::%1$s r) s
'
, TG_RELID::regclass
);
You will also need to define a v_new that is a plain record. You could then do something like:
-- Replace 2nd field in NEW with a new value
sql := c_new_old || $$SELECT row(NEW.a, $3, NEW.c) FROM NEW$$
EXECUTE sql INTO v_new USING NEW, OLD, new_value;
I have a postgresql function / stored proc that does the following:
1. calls another function and saves the value into a variable.
2. executes another sql statement using the value I got from step one as an argument.
My problem is that the query is not returning any data. No errors are returned either.
I'm just new to postgresql so I don't know the best way to debug... but I added a RAISE NOTICE command right after step 1, like so:
SELECT INTO active_id get_widget_id(widget_desc);
RAISE NOTICE 'Active ID is:(%)', active_id;
In the "Messages" section of the pgadmin3 screen, I see the debug message with the data:
NOTICE: Active ID is:(2)
I'm wondering whether or not the brackets are causing the problem for me.
Here's the sql I'm trying to run in step 2:
SELECT d.id, d.contact_id, d.priority, cp.contact
FROM widget_details d, contact_profile cp, contact_type ct
WHERE d.rule_id=active_id
AND d.active_yn = 't'
AND cp.id=d.contact_id
AND cp.contact_type_id=ct.id
AND ct.name = 'email'
Order by d.priority ASC
You'll notice that in my where clause I am referencing the variable "active_id".
I know that this query should return at least one row because when i run a straight sql select (vs using this function) and substitute the value 2 for the variable "active_id", I get back the data I'm looking for.
Any suggetions would be appreciated.
Thanks.
EDIT 1:
Here's the full function definition:
CREATE TYPE custom_return_type AS (
widgetnum integer,
contactid integer,
priority integer,
contactdetails character varying
);
CREATE OR REPLACE FUNCTION test(widget_desc integer)
RETURNS SETOF custom_return_type AS
$BODY$
DECLARE
active_id integer;
rec custom_return_type ;
BEGIN
SELECT INTO active_id get_widget_id(widget_desc);
RAISE NOTICE 'Active ID is:(%)', active_id;
FOR rec IN
SELECT d.id, d.contact_id, d.priority, cp.contact
FROM widget_details d, contact_profile cp, contact_type ct
WHERE d.rule_id=active_id
AND d.active_yn = 't'
AND cp.id=d.contact_id
AND cp.contact_type_id=ct.id
AND ct.name = 'email'
Order by d.priority ASC
LOOP
RETURN NEXT rec;
END LOOP;
END
$BODY$
That's several levels of too-complicated (edit: as it turns out that Erwin already explained to you last time you posted the same thing). Start by using RETURNS TABLE and RETURN QUERY:
CREATE OR REPLACE FUNCTION test(fmfm_number integer)
RETURNS TABLE (
widgetnum integer,
contactid integer,
priority integer,
contactdetails character varying
) AS
$BODY$
BEGIN
RETURN QUERY SELECT d.id, d.contact_id, d.priority, cp.contact
FROM widget_details d, contact_profile cp, contact_type ct
WHERE d.rule_id = get_widget_id(widget_desc)
AND d.active_yn = 't'
AND cp.id=d.contact_id
AND cp.contact_type_id=ct.id
AND ct.name = 'email'
Order by d.priority ASC;
END
$BODY$ LANGUAGE plpgsql;
at which point it's probably simple enough to be turned into a trivial SQL function or even a view. Hard to be sure, since the function doesn't make tons of sense as written:
You never use the parameter fmfm_number anywhere; and
widget_desc is never defined
so this function could never run. Clearly you haven't shown us the real source code, but some kind of "simplified" code that doesn't match the code you're really having issues with.
There is a difference between:
SELECT INTO ...
[http://www.postgresql.org/docs/current/interactive/sql-selectinto.html]
and
SELECT select_expressions INTO [STRICT] target FROM ...;
[http://www.postgresql.org/docs/current/interactive/plpgsql-statements.html#PLPGSQL-STATEMENTS-SQL-ONEROW]
I think you want:
SELECT get_widget_id(widget_desc) INTO active_id;
Im new to writing stored functions in postgresql and in general . I'm trying to write onw with an input parameter and return a set of results stored in a temporary table.
I do the following in my function .
1) Get a list of all the consumers and store their id's stored in a temp table.
2) Iterate over a particular table and retrieve values corresponding to each value from the above list and store in a temp table.
3)Return the temp table.
Here's the function that I've tried to write by myself ,
create or replace function getPumps(status varchar) returns setof record as $$ (setof record?)
DECLARE
cons_id integer[];
i integer;
temp table tmp_table;--Point B
BEGIN
select consumer_id into cons_id from db_consumer_pump_details;
FOR i in select * from cons_id LOOP
select objectid,pump_id,pump_serial_id,repdate,pumpmake,db_consumer_pump_details.status,db_consumer.consumer_name,db_consumer.wenexa_id,db_consumer.rr_no into tmp_table from db_consumer_pump_details inner join db_consumer on db_consumer.consumer_id=db_consumer_pump_details.consumer_id
where db_consumer_pump_details.consumer_id=i and db_consumer_pump_details.status=$1--Point A
order by db_consumer_pump_details.consumer_id,pump_id,createddate desc limit 2
END LOOP;
return tmp_table
END;
$$
LANGUAGE plpgsql;
However Im not sure about my approach and whether im right at the points A and B as I've marked in the code above.And getting a load of errors while trying to create the temporary table.
EDIT: got the function to work ,but I get the following error when I try to run the function.
ERROR: array value must start with "{" or dimension information
Here's my revised function.
create temp table tmp_table(objectid integer,pump_id integer,pump_serial_id varchar(50),repdate timestamp with time zone,pumpmake varchar(50),status varchar(2),consumer_name varchar(50),wenexa_id varchar(50),rr_no varchar(25));
select consumer_id into cons_id from db_consumer_pump_details;
FOR i in select * from cons_id LOOP
insert into tmp_table
select objectid,pump_id,pump_serial_id,repdate,pumpmake,db_consumer_pump_details.status,db_consumer.consumer_name,db_consumer.wenexa_id,db_consumer.rr_no from db_consumer_pump_details inner join db_consumer on db_consumer.consumer_id=db_consumer_pump_details.consumer_id where db_consumer_pump_details.consumer_id=i and db_consumer_pump_details.status=$1
order by db_consumer_pump_details.consumer_id,pump_id,createddate desc limit 2;
END LOOP;
return query (select * from tmp_table);
drop table tmp_table;
END;
$$
LANGUAGE plpgsql;
AFAIK one can't declare tables as variables in postgres. What you can do is create one in your funcion body and use it thourough (or even outside of function). Beware though as temporary tables aren't dropped until the end of the session or commit.
The way to go is to use RETURN NEXT or RETURN QUERY
As for the function result type I always found RETURNS TABLE to be more readable.
edit:
Your cons_id array is innecessary, just iterate the values returned by select.
Also you can have multiple return query statements in a single function to append result of the query to the result returned by function.
In your case:
CREATE OR REPLACE FUNCTION getPumps(status varchar)
RETURNS TABLE (objectid INTEGER,pump_id INTEGER,pump_serial_id INTEGER....)
AS
$$
BEGIN
FOR i in SELECT consumer_id FROM db_consumer_pump_details LOOP
RETURN QUERY(
SELECT objectid,pump_id,pump_serial_id,repdate,pumpmake,db_consumer_pump_details.status,db_consumer.consumer_name,db_consumer.wenexa_id,db_consumer.rr_no FROM db_consumer_pump_details INNER JOIN db_consumer ON db_consumer.consumer_id=db_consumer_pump_details.consumer_id
WHERE db_consumer_pump_details.consumer_id=i AND db_consumer_pump_details.status=$1
ORDER BY db_consumer_pump_details.consumer_id,pump_id,createddate DESC LIMIT 2
);
END LOOP;
END;
$$
edit2:
You probably want to take a look at this solution for groupwise-k-maximum problem as that's exactly what you're dealing with here.
it might be easier to just return a table (or query)
CREATE FUNCTION extended_sales(p_itemno int)
RETURNS TABLE(quantity int, total numeric) AS $$
BEGIN
RETURN QUERY SELECT quantity, quantity * price FROM sales
WHERE itemno = p_itemno;
END;
$$ LANGUAGE plpgsql;
(copied from postgresql docs)