Transactions not working OK in a PostgreSQL 11 stored procedure - postgresql

Postgres version:
PostgreSQL 11.0 on x86_64-pc-linux-gnu,
compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit
I have this stored procedure as shown below. This is just a test.
In that procedure I have 2 transactions:
The first one is supposed to finish OK (i.e. I have written the code
so that it does not encounter any errors and so it will reach the COMMIT statement).
The second transaction is supposed to fail as I intentionally
introduce an error in it (either via this cast there, or via an INSERT which causes PK violation).
Also, yb.print_now is a simple function which is just logging (inserting) messages to another table.
When I run this stored procedure I am expecting the updates and the logging of messages done by
the first transaction to be persisted in the database, even though the 2nd transaction failed.
But this is not what happens, both transactions seem to be rolled back.
Why is this? Am I doing something wrong?
And 2 more questions which are very important to me. :
When an error occurs (say like on line marked ***) and when control reaches/jumps to the EXCEPTION block, I have the feeling that the transaction I was in is already rolled back before I even reach the EXCEPTION block.
So in the exception block I cannot do ROLLBACK or COMMIT or anything
related to the transaction. Is that feeling correct?
Say I want to commit all the stuff done, despite of the error, is there a way I can do that?
That's exactly what I want here. The error is an error... OK, but I want everything
which happened before I got the error to get committed.
How do I do this in Postgres 11?
CREATE OR REPLACE PROCEDURE yb.test123()
LANGUAGE plpgsql
AS $procedure$
DECLARE
var_cnt int;
c int;
BEGIN
START TRANSACTION; --- 1 ---
raise notice '001.';
PERFORM yb.print_now('===> 0010.');
var_cnt = 0;
update yb.mbb
set the_price = the_price + 1
where
the_id = 23164;
raise notice '002.';
PERFORM yb.print_now('===> 0020.');
raise notice '003.';
PERFORM yb.print_now('===> 0030.');
update yb.mbb
set the_price = the_price + 1
where
the_id = 23164;
COMMIT; --- 1 ---
START TRANSACTION; --- 2 ---
c = cast('###a1e3Z' as int); --- *** ---
raise notice '004.';
PERFORM yb.print_now('===> 0040.');
update yb.mbb
set the_price = the_price + 1
where
the_id = 23164;
-- insert into yb.mbb(the_id)
-- values (23164); -- this will throw duplicate PK error
raise notice '005.';
PERFORM yb.print_now('===> 0050.');
COMMIT; --- 2 ---
EXCEPTION
WHEN OTHERS THEN
raise notice 'We are in the exception block now.';
-- ROLLBACK;
-- COMMIT;
RETURN;
END
$procedure$;

The error happens right at the start of your procedure, in the statement
START TRANSACTION;
As the documentation says:
A new transaction is started automatically after a transaction is ended using these commands, so there is no separate START TRANSACTION command.
That should answer your first question.
As to the second, when you are in the exception branch, you have effectively rolled back the subtransaction that started with the BEGIN that belongs to the EXCEPTION clause (or after the last COMMIT). You are still in the transaction though, so you can issue COMMIT and ROLLBACK.
To your third question: No, there is no way to commit “everything up to the last exception”. You could only have that by wrapping every statement in a BEGIN ... EXCEPTION ... END block, but that would seriously hurt your performance (apart from making your code unreadable).
Use BEGIN ... EXCEPTION ... END blocks judiciously whenever you expect that a statement could fail.

Related

Postgres Exception handling in stored procedure to continue the process

Database is postgressql.
I want to call five stored procedure in on stored procedure that is master_call_all() and need to handle exception after every call. So if any error occur it raise the exception and not terminate the process.
Let say error occur in 3rd call scratch.sp_insert3() then after raising error next stored procedure will call. This thing I want to achieve after every call.
create or replace procedure scratch.master_call_all()
LANGUAGE plpgsql
AS $$
DECLARE
BEGIN
call scratch.sp_insert1();
call scratch.sp_insert2();
call scratch.sp_insert3();
call scratch.sp_insert4();
call scratch.sp_insert5();
commit;
END;
$$;
First off: In general this is an extremely bad plan. It is the anti-thesis of creating a Transaction (begin ... end).
Transactions are a fundamental concept of all database systems. The essential point of a transaction is that it bundles multiple steps
into a single, all-or-nothing operation. The intermediate states
between the steps are not visible to other concurrent transactions,
and if some failure occurs that prevents the transaction from
completing, then none of the steps affect the database at all.
Ref: Database Transactions
Now with that out of the way, what you are asking can be done, but it is not done cleanly. A code block in Postgres contains 3 sections declaration, execution, exception with declaration and exception optional. Further a block can be nested inside another and the nested block retains all 3 sections. So what you need is to place each ca;; statement into a nested block. Like:
create or replace procedure master_call_all()
language plpgsql
as $$
declare
begin
begin
call sp_insert1();
exception
when others then
raise notice 'Exception occurred calling sp_insert1()';
end ;
begin
call sp_insert2();
exception
when others then
raise notice 'Exception occurred calling sp_insert2()';
end ;
begin
call sp_insert3();
exception
when others then
raise notice 'Exception occurred calling sp_insert3()';
end ;
begin
call sp_insert4();
exception
when others then
raise notice 'Exception occured calling sp_insert4()';
end ;
begin
call sp_insert5();
exception
when others then
raise notice 'Exception occured calling sp_insert5()';
end ;
commit;
end;
$$;
See demo here. Unfortunately db<>fiddle does not show the messages from raise notice .... Clearly in the demo sp_insert1, and sp_insert3 will raise an exception. (To see the messages run in your environment.)

How do you handle error handling and commits in Postgres

I am using Postgres 13.5 and I am unsure how to combine commit and error handling in a stored procedure or DO block. I know that if I include the EXCEPTION clause in my block, then I cannot include a commit.
I am new to Postgres. It has also been over 15 years since I have written SQL that was working with transactions. When I was working with transactions I was using Oracle and recall using AUTONOMOUS_TRANSACTION to resolve some of these issues. I am just not sure how to do something like that in Postgres.
Here is a very simplified DO block. As I said above, I know that the Commits will cause the procedure to throw and exception. But, if I remove the EXCEPTION clause, then how will I trap an error if it happens? After reading many things, I still have not found a solution. So, I am not understanding something that will lead me to the solution.
Do
$$
DECLARE
v_Start timestamptz;
v_id integer;
v_message_type varchar(500);
Begin
select current_timestamp into start;
select q.id, q.message_type into (v_id, v_message_type) from message_queue;
call Load_data(v_id, v_message_type);
commit; -- if Load_Data completes successfully, I want to commmit the data
insert into log (id, message_type, Status, start, end)
values (v_id, v_message_type, 'Success', v_start, Currrent_Timestamp);
commit; -- commit the log issert for success
EXCEPTION
WHEN others THEN
insert into log (id, message_type, status, start, end, error_message)
values (v_id, v_message_type, 'Failue', v_start, Currrent_Timestamp, SQLERRM || '', ' ||
SQLSTATE );
commit; -- commit the log insert for failure.
end;
$$
Thanks!
Since this is a pattern that I will have to do tens of times, I want to understand the right way to do this.
Since you cannot use transaction management statements in a subtransaction, you will have to move part of the processing to the client side.
But your sample code doesn't need any transaction management at all! Simply remove all the COMMIT statements, and the procedure will work just as you want it to. Remember that PostgreSQL uses the autocommit mode, so your procedure call from the client will automatically run in its own transaction and commit when it is done.
But perhaps your sample code is simplified, and you would like more complicated processing (looping etc.) in your actual use cases. So let's discuss your options:
One option is to remove the EXCEPTION handler and move only that part to the client side: if the procedure causes an error, roll back and insert a log message. Another, perhaps cleaner, method is to move the whole transaction management to the client side. In that case, you would replace the complete procedure with client code and call load_data directly from client code.

How to roll back a transaction on error in PostgreSQL?

I'm writing a script for PostgreSQL and since I want it to be executed atomically, I'm wrapping it inside a transaction.
I expected the script to look something like this:
BEGIN
-- 1) Execute some valid actions;
-- 2) Execute some action that causes an error.
EXCEPTION
WHEN OTHERS THEN
ROLLBACK;
END; -- A.k.a. COMMIT;
However, in this case pgAdmin warns me about a syntax error right after the initial BEGIN. If I terminate the command there by appending a semicolon like so: BEGIN; it instead informs me about error near EXCEPTION.
I realize that perhaps I'm mixing up syntax for control structures and transactions, however I couldn't find any mention of how to roll back a failed transaction in the docs (nor in SO for that matter).
I also considered that perhaps the transaction is rolled back automatically on error, but it doesn't seem to be the case since the following script:
BEGIN;
-- 1) Execute some valid actions;
-- 2) Execute some action that causes an error.
COMMIT;
warns me that: ERROR: current transaction is aborted, commands ignored until end of transaction block and I have to then manually ROLLBACK; the transaction.
It seems I'm missing something fundamental here, but what?
EDIT:
I tried using DO as well like so:
DO $$
BEGIN
-- 1) Execute some valid actions;
-- 2) Execute some action that causes an error.
EXCEPTION
WHEN OTHERS THEN
ROLLBACK;
END; $$
pgAdmin hits me back with a: ERROR: cannot begin/end transactions in PL/pgSQL. HINT: Use a BEGIN block with an EXCEPTION clause instead. which confuses me to no end, because that is exactly what I am (I think) doing.
POST-ACCEPT EDIT:
Regarding Laurenz's comment: "Your SQL script would contain a COMMIT. That ends the transaction and rolls it back." - this is not the behavior that I observe. Please consider the following example (which is just a concrete version of an example I already provided in my original question):
BEGIN;
-- Just a simple, self-referencing table.
CREATE TABLE "Dummy" (
"Id" INT GENERATED ALWAYS AS IDENTITY,
"ParentId" INT NULL,
CONSTRAINT "PK_Dummy" PRIMARY KEY ("Id"),
CONSTRAINT "FK_Dummy_Dummy" FOREIGN KEY ("ParentId") REFERENCES "Dummy" ("Id")
);
-- Foreign key violation terminates the transaction.
INSERT INTO "Dummy" ("ParentId")
VALUES (99);
COMMIT;
When I execute the script above, I'm greeted with: ERROR: insert or update on table "Dummy" violates foreign key constraint "FK_Dummy_Dummy". DETAIL: Key (ParentId)=(99) is not present in table "Dummy". which is as expected.
However, if I then try to check whether my Dummy table was created or rolled back like so:
SELECT EXISTS (
SELECT FROM information_schema."tables"
WHERE "table_name" = 'Dummy');
instead of a simple false, I get the same error that I already mentioned twice: ERROR: current transaction is aborted, commands ignored until end of transaction block. Then I have to manually terminate the transaction via issuing ROLLBACK;.
So to me it seems that either the comment mentioned above is false or at least I'm heavily misinterpreting something here.
You cannot use ROLLBACK in PL/pgSQL, except in certain limited cases inside procedures.
You don't need to explicitly roll back in your PL/pgSQL code. Just let the exception propagate out of the PL/pgSQL code, and it will cause an error, which will cause the whole transaction to be rolled back.
Your comments suggest that this code is called from an SQL script. Then the solution would be to have a COMMIT in that SQL script at some place after the PL/pgSQL code. That would end the transaction and roll it back.
I think you must be using an older version, as the exact code from your question works without error for me:
(The above is with PostgreSQL 13.1, and pgAdmin 4.28.)
It also works fine for me, without the exception block:
As per this comment, you can remove the exception block within a function, and if an error occurs, the transaction run within it will automatically be rolled back. That appears to be the case, from my limited testing.

When is better to call UPDATE STATISTICS? before or after a COMMIT TRANS

I'm working on an ETL project, actively populating tables with data. Sometimes if something is missing, a whole block of data modifications must be undone, so I'm using a transaction. When something goes wrong, a rollback applies, if not, an UPDATE STATISTICS could help efficiency.
So my question is what would be more efficient, to UPDATE STATISTICS inside the TRANSACTION or after the COMMIT?
Currently working fine as:
BEGIN TRY
BEGIN TRAN
UPDATE stuffs SET ...
INSERT things VALUES(...
UPDATE STATISTICS stuffs
UPDATE STATISTICS things
END TRY
BEGIN CATCH
IF ##TRANCOUNT > 0 ROLLBACK TRAN
RAISERROR( ... -- RAISERROR prevents from executing past this point
END CATCH
IF ##TRANCOUNT > 0 COMMIT TRAN
But maybe this is better
BEGIN TRY
BEGIN TRAN
UPDATE stuffs SET ...
INSERT things VALUES(...
END TRY
BEGIN CATCH
IF ##TRANCOUNT > 0 ROLLBACK TRAN
RAISERROR( ... -- RAISERROR prevents from executing past this point
END CATCH
IF ##TRANCOUNT > 0 COMMIT TRAN
UPDATE STATISTICS stuffs
UPDATE STATISTICS things
I've tried both with virtually the same results, but with more data or more rollbacks could be different.
I will use the second one.
The transaction will be rolled back when it raises the error(s) if you put them in the try block.
I believe you still need 'update statistics' to improve efficiency even after a rollback applies to your transaction. If so, it is better to keep 'update statistics' after the transaction commits.

Are PostgreSQL functions transactional?

Is a PostgreSQL function such as the following automatically transactional?
CREATE OR REPLACE FUNCTION refresh_materialized_view(name)
RETURNS integer AS
$BODY$
DECLARE
_table_name ALIAS FOR $1;
_entry materialized_views%ROWTYPE;
_result INT;
BEGIN
EXECUTE 'TRUNCATE TABLE ' || _table_name;
UPDATE materialized_views
SET last_refresh = CURRENT_TIMESTAMP
WHERE table_name = _table_name;
RETURN 1;
END
$BODY$
LANGUAGE plpgsql VOLATILE SECURITY DEFINER;
In other words, if an error occurs during the execution of the function, will any changes be rolled back? If this isn't the default behavior, how can I make the function transactional?
PostgreSQL 12 update: there is limited support for top-level PROCEDUREs that can do transaction control. You still cannot manage transactions in regular SQL-callable functions, so the below remains true except when using the new top-level procedures.
Functions are part of the transaction they're called from. Their effects are rolled back if the transaction rolls back. Their work commits if the transaction commits. Any BEGIN ... EXCEPT blocks within the function operate like (and under the hood use) savepoints like the SAVEPOINT and ROLLBACK TO SAVEPOINT SQL statements.
The function either succeeds in its entirety or fails in its entirety, barring BEGIN ... EXCEPT error handling. If an error is raised within the function and not handled, the transaction calling the function is aborted. Aborted transactions cannot commit, and if they try to commit the COMMIT is treated as ROLLBACK, same as for any other transaction in error. Observe:
regress=# BEGIN;
BEGIN
regress=# SELECT 1/0;
ERROR: division by zero
regress=# COMMIT;
ROLLBACK
See how the transaction, which is in the error state due to the zero division, rolls back on COMMIT?
If you call a function without an explicit surounding transaction the rules are exactly the same as for any other Pg statement:
BEGIN;
SELECT refresh_materialized_view(name);
COMMIT;
(where COMMIT will fail if the SELECT raised an error).
PostgreSQL does not (yet) support autonomous transactions in functions, where the procedure/function could commit/rollback independently of the calling transaction. This can be simulated using a new session via dblink.
BUT, things that aren't transactional or are imperfectly transactional exist in PostgreSQL. If it has non-transactional behaviour in a normal BEGIN; do stuff; COMMIT; block, it has non-transactional behaviour in a function too. For example, nextval and setval, TRUNCATE, etc.
As my knowledge of PostgreSQL is less deeper than Craig Ringer´s I will try to give a shorter answer: Yes.
If you execute a function that has an error in it, none of the steps will impact in the database.
Also, if you execute a query in PgAdmin the same happen.
For example, if you execute in a query:
update your_table yt set column1 = 10 where yt.id=20;
select anything_that_do_not_exists;
The update in the row, id = 20 of your_table will not be saved in the database.
UPDATE Sep - 2018
To clarify the concept I have made a little example with non-transactional function nextval.
First, let´s create a sequence:
create sequence test_sequence start 100;
Then, let´s execute:
update your_table yt set column1 = 10 where yt.id=20;
select nextval('test_sequence');
select anything_that_do_not_exists;
Now, if we open another query and execute
select nextval('test_sequence');
We will get 101 because the first value (100) was used in the latter query (that is because the sequences are not transactional) although the update was not committed.
https://www.postgresql.org/docs/current/static/plpgsql-structure.html
It is important not to confuse the use of BEGIN/END for grouping statements in PL/pgSQL with the similarly-named SQL commands for transaction control. PL/pgSQL's BEGIN/END are only for grouping; they do not start or end a transaction. Functions and trigger procedures are always executed within a transaction established by an outer query — they cannot start or commit that transaction, since there would be no context for them to execute in. However, a block containing an EXCEPTION clause effectively forms a subtransaction that can be rolled back without affecting the outer transaction. For more about that see Section 39.6.6.
In the function level, it is not transnational. In other words, each statement in the function belongs to a single transaction, which is the default db auto commit value. Auto commit is true by default. But anyway, you have to call the function using
select schemaName.functionName()
The above statement 'select schemaName.functionName()' is a single transaction, let's name the transaction T1, and so the all the statements in the function belong to the transaction T1. In this way, the function is in a single transaction.
Postgres 14 update: All statements written in between the BEGIN and END block of a Procedure/Function is executed in a single transaction. Thus, any errors arising while execution of this block will cause automatic roll back of the transaction.
Additionally, the ATOMIC Transaction including triggers as well.