CURSOR vs select statement in loop - postgresql

I just saw a simple example in another StackOverflow question that used a cursor to loop through a table. I would have just looped through the results of a select query instead of wrapping the select query in a cursor. What is the advantage of using a cursor?
(I couldn't include the example here because StackOverflow thought my question was mostly code, and demanded more details. I've run into that annoying restriction before. If I can ask my question clearly in just a few words, I should be able to. I'll see if I can find a link to that question, and if I can, I'll add the link here.)
Here is the original question where I saw CURSOR used.

What is the advantage of using a cursor?
The only advantage is that you have to write more code (if they pay you for each line of code).
do $$
declare
rec record;
cur cursor for select i from generate_series(1, 3) i;
begin
open cur;
loop
fetch cur into rec;
exit when rec is null;
raise notice '%', rec.i;
end loop;
close cur;
end
$$;
A loop through query results just opens a (virtual) cursor, fetches rows, checks range, exits when needed and closes the cursor for you.
do $$
declare
rec record;
begin
for rec in select i from generate_series(1, 3) i
loop
raise notice '%', rec.i;
end loop;
end
$$;

There are several ways:
Use an explicit cursor in PL/pgSQL and loop through it and process each result row.
Example:
OPEN c FOR SELECT id FROM a WHERE ok;
LOOP
UPDATE b SET a_ok = TRUE WHERE a_id = c.id;
END LOOP;
Use a FOR r IN SELECT ... LOOP in PL/pgSQL. This is effectively the same as 1. with a clearer syntax.
Example:
FOR c IN SELECT id FROM a WHERE ok LOOP
UPDATE b SET a_ok = TRUE WHERE a_id = c.id;
END LOOP;
Run a SELECT query without a cursor and process each result row on the client side, probably issuing a database query for each result.
Example (in pseudocode):
resultset := db_exec('SELECT id FROM a WHERE ok');
while (resultset.next()) {
db_exec('UPDATE b SET a_ok = TRUE WHERE a_id = ' || resultset.get('id'));
}
Use a JOIN.
Example:
UPDATE b SET a_ok = TRUE
FROM a
WHERE a.id = b.a_id AND a.ok;
Method 3. is the most terrible way conceivable to solve the problem, because it causes a lot of client-server round trips and has the database parse a gazillion statements.
Alas, it is often the way how SQL novices attack the problem. I call it home-grown nested loop join.
On top of all that, the client software will often snarf the complete result set from the first query into memory, which causes yet another problem.
Methods 1. and 2. are equivalent, except that 2. is more elegant. It saves the round trips and uses prepared statements under the hood, so the UPDATE doesn't have to be parsed all the time. Still, the executor has to run many times, and PL/pgSQL is known not to be particularly fast. It is also a kind of home-grown nested loop join.
Method 4 is the way to go. Not only is everything run in a single query, but PostgreSQL can also use a more effective join strategy if that is better.

Related

Is there any way to check whether a PostgreSQL record- or row-type variable contains a specific field from inside a function?

I have a trigger defined on several tables to fire after all INSERT, UPDATE, or DELETE, all using the same trigger function. The trigger function performs an expensive check, but I can speed it up significantly by filtering some of the intermediate steps of that check using either a WHERE machine_serial = NEW.machine_serial or WHERE machine_serial = OLD.machine_serial clause, depending on what type of statement fired the trigger. However, not all the tables actually have a machine_serial column, so I can't perform this filtering when the trigger is fired on one of those tables. I am currently trying to find a good solution to making the decision of whether to filter or not from within the trigger function, and I believe that simply checking whether NEW or OLD has the machine_serial field would be easiest, clearest, and fastest. I can't find any way to do that in the documentation though, but checking whether a RECORD contains a certain field seems like such a basic, commonplace operation for anyone that has to work with RECORDs that I assume that I've just got to be missing it somewhere - I can't imagine that it's just not possible.
For completeness, I'll go over the alternatives I've considered to the hypothetical does-RECORD-have-field check:
I could create two trigger functions, do_expensive_check_with_machine_serial() and do_expensive_check_without_machine_serial(), and use one or the other depending on whether the table has the machine_serial column. But if I or anyone after me needs to alter the logic in either one of these functions, they'll need to remember to alter the logic in the other one, too.
I could stick with the one trigger function I currently have, and figure out whether the firing table has machine_serial by just trying to access NEW.machine_serial or OLD.machine_serial. If that raises an exception, I can catch it and then I'll know the field isn't present. But the manual explicitly suggests avoiding using exception blocks unless absolutely necessary, due to performance impacts.
I could stick with the one trigger function I currently have, and just add a check like this: IF (TG_TABLE_SCHEMA = x AND TG_TABLE_NAME = y) OR (TG_TABLE_SCHEMA = w AND TG_TABLE_NAME = z) OR ...
, and just maintain that list of every table that has a machine_serial column. But then I and anyone that comes after me would need to alter that check in the trigger function any time the trigger is added to a new table, which is less than ideal.
Of course, the above three alternatives would all function, but they all feel like bad design choices to me. Maybe it's because I'm used to the dynamicness offered by Python, but if I used any of these alternatives, I would feel like I'm doing something wrong. And PostgreSQL is pretty good about offering lots of operators on all sorts of data types, so I just can't imagine that something as basic as checking whether a RECORD or ROW-type variable contains a certain field is impossible.
Before I show the solution, I have to say, so this requirement can be signal of some unhappy design. Maybe you try to implement some functionality that should not be implemented in triggers. Triggers are good, but too smart too generic too rich can be very slow and very hard to maintain and fix errors (but as every in life, there are exceptions from rules).
So first - you can look to system catalog:
CREATE FUNCTION public.foo_trg() RETURNS trigger
LANGUAGE plpgsql
AS $$
begin
raise notice 'a exists %', exists(select * from pg_attribute where attrelid = new.tableoid and attname = 'a');
raise notice 'd exists %', exists(select * from pg_attribute where attrelid = new.tableoid and attname = 'd');
return new;
end;
$$;
CREATE TABLE public.foo (
a integer,
b integer
);
CREATE TRIGGER foo_trg_insert
AFTER INSERT ON public.foo
FOR EACH ROW EXECUTE FUNCTION public.foo_trg();
(2022-09-02 06:18:41) postgres=# insert into foo values(1,2);
NOTICE: a exists t
NOTICE: d exists f
INSERT 0 1
Second solution is based on record to jsonb transformations:
CREATE OR REPLACE FUNCTION public.foo_trg()
RETURNS trigger
LANGUAGE plpgsql
AS $$
declare j jsonb;
begin
j := to_jsonb(new);
raise notice 'a exists %', j ? 'a';
raise notice 'd exists %', j ? 'd';
return new;
end;
$$
(2022-09-02 06:24:54) postgres=# insert into foo values(1,2);
NOTICE: a exists t
NOTICE: d exists f
INSERT 0 1
Second solution can be faster, because doesn't requires queries to system catalog. It hits just system catalog cache, but it doesn't work on some legacy PostgreSQL releases.

Second FOR loop not entered in PL/pgSQL

I encountered strange behaviour in PL/pgSQL for PostgreSQL. I am making a function that has two separate FOR loops. The first one runs perfectly but the second seems to be forgotten. This is how the code looks roughly:
Declare r RECORD;
BEGIN
FOR r in (select something from a)
LOOP
DO SOMETHING;
END LOOP;
r:=NULL;
FOR r in (select something from b)
LOOP
DO SOMETHING;
END LOOP;
END;
When put in separate functions both FOR loops work well. I also tried to use 2 separate record variables but that didn't work either.
I can't understand why the second loop is ignored.
Thanks!

Benchmarking/Performance of Postgresql Query

I want to measure the performance of postgresql code I wrote. In the code tables get created, selfwritten functions get called etc.
Looking around, I found EXPLAIN ANALYSE is the way to go.
However, as far as I understand it, the code only gets executed once. For a more realistic analysis I want to execute the code many many times and have the results of each iteration written somewhere, ideally in a table (for statistics later).
Is there a way to do this with a native postgresql function? If there is no native postgresql function, would I accomplish this with a simple loop? Further, how would I write out the information of every EXPLAIN ANALYZE iteration?
One way to do this is to write a function that runs an explain and then spool the output of that into a file (or insert that into a table).
E.g.:
create or replace function show_plan(to_explain text)
returns table (line_nr integer, line text)
as
$$
declare
l_plan_line record;
l_line integer;
begin
l_line := 1;
for l_plan_line in execute 'explain (analyze, verbose)'||to_explain loop
return query select l_line, l_plan_line."QUERY PLAN";
l_line := l_line + 1;
end loop;
end;
$$
language plpgsql;
Then you can use generate_series() to run a statement multiple times:
select g.i as run_nr, e.*
from show_plan('select * from foo') e
cross join generate_series(1,10) as g(i)
order by g.i, e.line_nr;
This will run the function 10 times with the passed SQL statement. The result can either be spooled to a file (how you do that depends on the SQL client you are using) or inserted into a table.
For an automatic analysis it's probably easer to use a more "parseable" explain format, e.g. XML or JSON. This is also easier to handle in the output as the plan is a single XML (or JSON) value instead of multiple text lines:
create or replace function show_plan_xml(to_explain text)
returns xml
as
$$
begin
return execut 'explain (analyze, verbose, format xml)'||to_explain;
end;
$$
language plpgsql;
Then use:
select g.i as run_nr, show_plan_xml('select * from foo')
from join generate_series(1,10) as g(i)
order by g.i;

Why doesn't this function actually update anything?

This is a SQL learning exercise. I have a 'tbl' with a single integer field. There are no indices on this table. I have this function:
CREATE OR REPLACE FUNCTION rcs()
RETURNS VOID AS $$
DECLARE
c CURSOR FOR SELECT * FROM tbl ORDER BY sequence;
s INTEGER;
r RECORD;
BEGIN
s := 0;
OPEN c;
LOOP
FETCH c INTO r;
EXIT WHEN NOT FOUND;
RAISE NOTICE 'got it';
r.sequence = s;
s := s + 1;
RAISE NOTICE '%', r.sequence;
END LOOP;
CLOSE c;
END;
$$ language 'plpgsql'
This loads and runs cleanly and the RAISE statements suggest that the 'sequence' field gets updated to 0, 1 etc. in accordance with the ORDER BY.
However, when I SELECT the table afterwards, the pre-existing values (which happen to all be '6') did not change.
Is this something to do with transactions? I tried fiddling around with COMMIT, etc. to no avail.
This is a freshly installed Postgresql 9.4.4 running on a Linode with no hackery of config files or anything like that, from the 'psql' command line.
EDIT: maybe it's because 'r' isn't actually the DB table, it's some kind of temporary copy of it? If so, please clarify, hopefully what I'm trying to achieve here is obvious (and I know it may be a bit nonsensical, but surely it's possible without resorting to reading the set into Java, etc.)
The actual problem: your function does not contain an UPDATE statement so nothing gets written to disk. r.sequence = s; simply assigns a new value to a variable that is held in memory.
To fix this, you need something like:
UPDATE tbl
set sequence = s -- change the actual column in the table
WHERE current of c; -- for the current row of your cursor
If where current of doesn't work, you need to switch that to a "regular" where clause:
UPDATE tbl
set sequence = s
WHERE tbl.pk_column = r.pk_column; -- the current primary key value
But a much more efficient solution is to do this in a single statement:
update tbl
set sequence = tu.new_sequence
from (
select t.pk_column,
row_number() over (order by t.sequence) as new_sequence
from tbl t
) tu
where tbl.pk_column = tu.pk_column;
You need to replace the column name pk_column with the real primary key (or unique) column of your table.

Is this generic MERGE/UPSERT function for PostgreSQL safe?

I have created a "merge" function which is supposed to execute either an UPDATE or an INSERT query, depending on existing data. Instead of writing an upsert-wrapper for each table (as in most of the available examples), this function takes entire SQL strings. Both of the SQL strings are automatically generated by our application.
The plan is to call the function like this:
-- hypothetical "settings" table, with a primary key of (user_id, setting):
SELECT merge(
$$UPDATE settings SET value = 'x' WHERE user_id = 42 AND setting = 'foo'$$,
$$INSERT INTO settings (user_id, setting, value) VALUES (42, 'foo', 'x')$$
);
Here's the full code of the merge() function:
CREATE OR REPLACE FUNCTION merge (update_sql TEXT, insert_sql TEXT) RETURNS TEXT AS
$func$
DECLARE
max_iterations INTEGER := 10;
i INTEGER := 0;
num_updated INTEGER;
BEGIN
-- usually returns before re-entering the loop
LOOP
-- first try the update
EXECUTE update_sql;
GET DIAGNOSTICS num_updated = ROW_COUNT;
IF num_updated > 0 THEN
RETURN 'UPDATE';
END IF;
-- nothing was updated: try the insert, watching out for concurrent inserts
BEGIN
EXECUTE insert_sql;
RETURN 'INSERT';
EXCEPTION WHEN unique_violation THEN
-- nop; just loop and try again from the top
END;
-- emergency brake
i := i + 1;
IF i >= max_iterations THEN
RAISE EXCEPTION 'merge(): tried looping % times, giving up now.', i;
EXIT;
END IF;
END LOOP;
END;
$func$
LANGUAGE plpgsql;
It appears to work well enough in my tests, but I'm not certain if I haven't missed anything crucial, especially regarding concurrent UPDATE/INSERT/DELETE queries, which may be issued without using this function. Did I overlook anything important?
Among the resources I consulted for this function are:
UPDATE/INSERT example 40.2 in the PostgreSQL manual
Why is UPSERT so complicated?
SO: Insert, on duplicate update (postgresql)
(Edit: one of the goals was to avoid locking the target table.)
The answer to your question depends your the context of how your application(s) will access the database. There are many ways to solve this as nicely discussed in depesz's post you cited by yourself. In addition you might want to also consider using writeable CTEs see here. Also the [question]Insert, on duplicate update in PostgreSQL? has some interesting discussions for your decision making process.