I am using Postgres 13.5 and I am unsure how to combine commit and error handling in a stored procedure or DO block. I know that if I include the EXCEPTION clause in my block, then I cannot include a commit.
I am new to Postgres. It has also been over 15 years since I have written SQL that was working with transactions. When I was working with transactions I was using Oracle and recall using AUTONOMOUS_TRANSACTION to resolve some of these issues. I am just not sure how to do something like that in Postgres.
Here is a very simplified DO block. As I said above, I know that the Commits will cause the procedure to throw and exception. But, if I remove the EXCEPTION clause, then how will I trap an error if it happens? After reading many things, I still have not found a solution. So, I am not understanding something that will lead me to the solution.
Do
$$
DECLARE
v_Start timestamptz;
v_id integer;
v_message_type varchar(500);
Begin
select current_timestamp into start;
select q.id, q.message_type into (v_id, v_message_type) from message_queue;
call Load_data(v_id, v_message_type);
commit; -- if Load_Data completes successfully, I want to commmit the data
insert into log (id, message_type, Status, start, end)
values (v_id, v_message_type, 'Success', v_start, Currrent_Timestamp);
commit; -- commit the log issert for success
EXCEPTION
WHEN others THEN
insert into log (id, message_type, status, start, end, error_message)
values (v_id, v_message_type, 'Failue', v_start, Currrent_Timestamp, SQLERRM || '', ' ||
SQLSTATE );
commit; -- commit the log insert for failure.
end;
$$
Thanks!
Since this is a pattern that I will have to do tens of times, I want to understand the right way to do this.
Since you cannot use transaction management statements in a subtransaction, you will have to move part of the processing to the client side.
But your sample code doesn't need any transaction management at all! Simply remove all the COMMIT statements, and the procedure will work just as you want it to. Remember that PostgreSQL uses the autocommit mode, so your procedure call from the client will automatically run in its own transaction and commit when it is done.
But perhaps your sample code is simplified, and you would like more complicated processing (looping etc.) in your actual use cases. So let's discuss your options:
One option is to remove the EXCEPTION handler and move only that part to the client side: if the procedure causes an error, roll back and insert a log message. Another, perhaps cleaner, method is to move the whole transaction management to the client side. In that case, you would replace the complete procedure with client code and call load_data directly from client code.
Related
I'm trying to translate an SQL Server transaction from a textbook into PostgreSQL. The original transaction is
BEGIN TRANSACTION
INSERT INTO Customers(cust_id,cust_name) VALUES('1000000010','Toys Emporium');
SAVE TRANSACTION StartOrder;
INSERT INTO Orders(order_num,order_date,cust_id) VALUES(20100,'1999/12/1','1000000010');
IF ##ERROR <> 0 ROLLBACK TRANSACTION StartOrder;
[additional inserts with same rollback omitted]
COMMIT TRANSACTION
But I keep getting response
ERROR: syntax error at or near "IF"
LINE 1: IF ##ERROR <> 0 THEN
^
PostgreSQL has the best architecture for developing SQL codes. So, inside the function, you can not use transaction commit or start. Transactions work only inside the procedures. But, you can create your best ACID structure only by using functions. Because the function begin keyword is equivalent to start transaction command and the end keyword of the function is equivalent to commit transaction command. One function = One transaction. But if you need to use one transaction for inserting data into two tables, you can use your insert functions for the same tables inside the one function. In this variant, your main function will be your main transaction. Inside the main function, if one of the internal functions will have an exception so, other internal functions will be rollbacked. For example, we use sub-functions and sub-sub functions inside the main function. If one of the sub-sub functions will have exceptions then your main function will be rollbacked. Using this mechanism you will control your transactions fully, without any problems.
I wrote your query in PostgreSQL, Example:
begin transaction;
INSERT INTO Customers(cust_id,cust_name) VALUES('1000000010','Toys Emporium');
savepoint StartOrder;
commit;
INSERT INTO Orders(order_num,order_date,cust_id) VALUES(20100,'1999/12/1','1000000010');
commit;
exception when others then
rollback to StartOrder;
commit;
I would love to be able to validate objects representing table rows using the database's existing constraints (triggers that raise exceptions and checks) without actually inserting them into the database.
Is there currently a way one could do this in postgres? At least with BEFORE INSERT triggers and CHECK, I assume it makes no sense with AFTER INSERT triggers.
The easiest way I can think or right now would be to:
Lock the table
Insert a new row
If exception raise to the API / else DELETE the row and call it valid
Unlock
But I can see several issues with this.
A simpler way is to insert within a transaction and not commit:
BEGIN;
INSERT INTO tbl(...) VALUES (...);
-- see effects ...
ROLLBACK;
No need for additional locking. The row is never visible to any other transaction with default transacton isolation level READ COMMITTED. (You might be stalling concurrent writes that confict with the tested row.)
Notable side-effect: Sequences of serial or IDENTITY columns are advanced even if the INSERT is never committed. But gaps in sequential numbers are to be expected anyway and nothing to worry about.
Be wary of triggers with side-effects. All "transactional" SQL effects are rolled back, even most DDL commands. But some special operations (like advancing sequences) are never rolled back.
Also, DEFERRED constraints do not kick in. The manual:
DEFERRED constraints are not checked until transaction commit.
If you need this a lot, work with a copy of your table, or even your database.
Strictly speaking, while any trigger / constraint / concurrent event is allowed, there is no other way to "validate objects" than to insert them into the actual target table in the actual target database at the actual point in time. Triggers, constraints, even default values, can interact with the current state of the whole DB. The more possibilities are ruled out and requirements are reduced, the more options we might have to emulate the test.
CREATE FUNCTION validate_function ( )
RETURNS trigger LANGUAGE plpgsql
AS $function$
DECLARE
valid_flag boolean := 't';
BEGIN
--Validation code
if valid_flag = 'f' then
RAISE EXCEPTION 'This record is not valid id %', id
USING HINT = 'Please enter valid record';
RETURN NULL;
else
RETURN NEW;
end if;
END;
$function$
CREATE TRIGGER validate_rec BEFORE INSERT OR UPDATE ON some_tbl
FOR EACH ROW EXECUTE FUNCTION validate_function();
With this function and trigger you validate inside the trigger. If the new record fails validation you set the valid_flag to false and then use that to raise exception. The RETURN NULL; is probably redundant and I am not sure it will be reached, but if it is it will also abort the insert or update. If the record is valid then you RETURN NEW and the insert/update completes.
I'm writing a script for PostgreSQL and since I want it to be executed atomically, I'm wrapping it inside a transaction.
I expected the script to look something like this:
BEGIN
-- 1) Execute some valid actions;
-- 2) Execute some action that causes an error.
EXCEPTION
WHEN OTHERS THEN
ROLLBACK;
END; -- A.k.a. COMMIT;
However, in this case pgAdmin warns me about a syntax error right after the initial BEGIN. If I terminate the command there by appending a semicolon like so: BEGIN; it instead informs me about error near EXCEPTION.
I realize that perhaps I'm mixing up syntax for control structures and transactions, however I couldn't find any mention of how to roll back a failed transaction in the docs (nor in SO for that matter).
I also considered that perhaps the transaction is rolled back automatically on error, but it doesn't seem to be the case since the following script:
BEGIN;
-- 1) Execute some valid actions;
-- 2) Execute some action that causes an error.
COMMIT;
warns me that: ERROR: current transaction is aborted, commands ignored until end of transaction block and I have to then manually ROLLBACK; the transaction.
It seems I'm missing something fundamental here, but what?
EDIT:
I tried using DO as well like so:
DO $$
BEGIN
-- 1) Execute some valid actions;
-- 2) Execute some action that causes an error.
EXCEPTION
WHEN OTHERS THEN
ROLLBACK;
END; $$
pgAdmin hits me back with a: ERROR: cannot begin/end transactions in PL/pgSQL. HINT: Use a BEGIN block with an EXCEPTION clause instead. which confuses me to no end, because that is exactly what I am (I think) doing.
POST-ACCEPT EDIT:
Regarding Laurenz's comment: "Your SQL script would contain a COMMIT. That ends the transaction and rolls it back." - this is not the behavior that I observe. Please consider the following example (which is just a concrete version of an example I already provided in my original question):
BEGIN;
-- Just a simple, self-referencing table.
CREATE TABLE "Dummy" (
"Id" INT GENERATED ALWAYS AS IDENTITY,
"ParentId" INT NULL,
CONSTRAINT "PK_Dummy" PRIMARY KEY ("Id"),
CONSTRAINT "FK_Dummy_Dummy" FOREIGN KEY ("ParentId") REFERENCES "Dummy" ("Id")
);
-- Foreign key violation terminates the transaction.
INSERT INTO "Dummy" ("ParentId")
VALUES (99);
COMMIT;
When I execute the script above, I'm greeted with: ERROR: insert or update on table "Dummy" violates foreign key constraint "FK_Dummy_Dummy". DETAIL: Key (ParentId)=(99) is not present in table "Dummy". which is as expected.
However, if I then try to check whether my Dummy table was created or rolled back like so:
SELECT EXISTS (
SELECT FROM information_schema."tables"
WHERE "table_name" = 'Dummy');
instead of a simple false, I get the same error that I already mentioned twice: ERROR: current transaction is aborted, commands ignored until end of transaction block. Then I have to manually terminate the transaction via issuing ROLLBACK;.
So to me it seems that either the comment mentioned above is false or at least I'm heavily misinterpreting something here.
You cannot use ROLLBACK in PL/pgSQL, except in certain limited cases inside procedures.
You don't need to explicitly roll back in your PL/pgSQL code. Just let the exception propagate out of the PL/pgSQL code, and it will cause an error, which will cause the whole transaction to be rolled back.
Your comments suggest that this code is called from an SQL script. Then the solution would be to have a COMMIT in that SQL script at some place after the PL/pgSQL code. That would end the transaction and roll it back.
I think you must be using an older version, as the exact code from your question works without error for me:
(The above is with PostgreSQL 13.1, and pgAdmin 4.28.)
It also works fine for me, without the exception block:
As per this comment, you can remove the exception block within a function, and if an error occurs, the transaction run within it will automatically be rolled back. That appears to be the case, from my limited testing.
Is a PostgreSQL function such as the following automatically transactional?
CREATE OR REPLACE FUNCTION refresh_materialized_view(name)
RETURNS integer AS
$BODY$
DECLARE
_table_name ALIAS FOR $1;
_entry materialized_views%ROWTYPE;
_result INT;
BEGIN
EXECUTE 'TRUNCATE TABLE ' || _table_name;
UPDATE materialized_views
SET last_refresh = CURRENT_TIMESTAMP
WHERE table_name = _table_name;
RETURN 1;
END
$BODY$
LANGUAGE plpgsql VOLATILE SECURITY DEFINER;
In other words, if an error occurs during the execution of the function, will any changes be rolled back? If this isn't the default behavior, how can I make the function transactional?
PostgreSQL 12 update: there is limited support for top-level PROCEDUREs that can do transaction control. You still cannot manage transactions in regular SQL-callable functions, so the below remains true except when using the new top-level procedures.
Functions are part of the transaction they're called from. Their effects are rolled back if the transaction rolls back. Their work commits if the transaction commits. Any BEGIN ... EXCEPT blocks within the function operate like (and under the hood use) savepoints like the SAVEPOINT and ROLLBACK TO SAVEPOINT SQL statements.
The function either succeeds in its entirety or fails in its entirety, barring BEGIN ... EXCEPT error handling. If an error is raised within the function and not handled, the transaction calling the function is aborted. Aborted transactions cannot commit, and if they try to commit the COMMIT is treated as ROLLBACK, same as for any other transaction in error. Observe:
regress=# BEGIN;
BEGIN
regress=# SELECT 1/0;
ERROR: division by zero
regress=# COMMIT;
ROLLBACK
See how the transaction, which is in the error state due to the zero division, rolls back on COMMIT?
If you call a function without an explicit surounding transaction the rules are exactly the same as for any other Pg statement:
BEGIN;
SELECT refresh_materialized_view(name);
COMMIT;
(where COMMIT will fail if the SELECT raised an error).
PostgreSQL does not (yet) support autonomous transactions in functions, where the procedure/function could commit/rollback independently of the calling transaction. This can be simulated using a new session via dblink.
BUT, things that aren't transactional or are imperfectly transactional exist in PostgreSQL. If it has non-transactional behaviour in a normal BEGIN; do stuff; COMMIT; block, it has non-transactional behaviour in a function too. For example, nextval and setval, TRUNCATE, etc.
As my knowledge of PostgreSQL is less deeper than Craig Ringer´s I will try to give a shorter answer: Yes.
If you execute a function that has an error in it, none of the steps will impact in the database.
Also, if you execute a query in PgAdmin the same happen.
For example, if you execute in a query:
update your_table yt set column1 = 10 where yt.id=20;
select anything_that_do_not_exists;
The update in the row, id = 20 of your_table will not be saved in the database.
UPDATE Sep - 2018
To clarify the concept I have made a little example with non-transactional function nextval.
First, let´s create a sequence:
create sequence test_sequence start 100;
Then, let´s execute:
update your_table yt set column1 = 10 where yt.id=20;
select nextval('test_sequence');
select anything_that_do_not_exists;
Now, if we open another query and execute
select nextval('test_sequence');
We will get 101 because the first value (100) was used in the latter query (that is because the sequences are not transactional) although the update was not committed.
https://www.postgresql.org/docs/current/static/plpgsql-structure.html
It is important not to confuse the use of BEGIN/END for grouping statements in PL/pgSQL with the similarly-named SQL commands for transaction control. PL/pgSQL's BEGIN/END are only for grouping; they do not start or end a transaction. Functions and trigger procedures are always executed within a transaction established by an outer query — they cannot start or commit that transaction, since there would be no context for them to execute in. However, a block containing an EXCEPTION clause effectively forms a subtransaction that can be rolled back without affecting the outer transaction. For more about that see Section 39.6.6.
In the function level, it is not transnational. In other words, each statement in the function belongs to a single transaction, which is the default db auto commit value. Auto commit is true by default. But anyway, you have to call the function using
select schemaName.functionName()
The above statement 'select schemaName.functionName()' is a single transaction, let's name the transaction T1, and so the all the statements in the function belong to the transaction T1. In this way, the function is in a single transaction.
Postgres 14 update: All statements written in between the BEGIN and END block of a Procedure/Function is executed in a single transaction. Thus, any errors arising while execution of this block will cause automatic roll back of the transaction.
Additionally, the ATOMIC Transaction including triggers as well.
Postgres automatically aborts transactions whenever any SQL statement terminates with an error, which includes any constraint violation. For example:
glyph=# create table foo (bar integer, constraint blug check(bar > 5));
CREATE TABLE
glyph=# begin;
BEGIN
glyph=# insert into foo values (10);
INSERT 0 1
glyph=# insert into foo values (1);
ERROR: new row for relation "foo" violates check constraint "blug"
STATEMENT: insert into foo values (1);
ERROR: new row for relation "foo" violates check constraint "blug"
No message has yet been issued to that effect, but the transaction is rolled back. My personal favorite line of this session is the following:
glyph=# commit;
ROLLBACK
... since "ROLLBACK" seems like an odd success-message for COMMIT. But, indeed, it's been rolled back, and there are no rows in the table:
glyph=# select * from foo;
bar
-----
(0 rows)
I know that I can create a ton of SAVEPOINTs and handle errors in SQL that way, but that involves more traffic to the database, more latency (I might have to handle an error from the SAVEPOINT after all), for relatively little benefit. I really just want to handle the error in my application language anyway (Python) with a try/except, so the only behavior I want out of the SQL is for errors not to trigger automatic rollbacks. What can I do?
I'm extremely new to PostgreSQL, but one of the examples in the PostgreSQL documentation for triggers / server-side programming looks like it does exactly what you're looking for.
See: http://www.postgresql.org/docs/9.2/static/trigger-example.html
Snippet from the page: "So the trigger acts as a not-null constraint but doesn't abort the transaction."
I know this is a very old ticket but (as of 2017) PostgreSQL still have this same behavior of auto-rolling back itself when something goes wrong in the commit. I'd like to share some thoughts here.
I don't know if we can change this behavior, and I don't need this, maybe for the best of delegating PostgreSQL to manage the rollback for us (he knows what he is doing, right ?). Rolling back means changing the data back to its original state before the failed transaction, that means altered or inserted data from triggers will also be discarded. In an ACID logic, this is what we want. Let say you are managing the rollback on the back-end yourself, if something goes wrong during your custom rollback or if the database is changed at the same time from external transactions during your rollback, the data becomes inconsistent and your whole structure most likely to collapse.
So knowing that PostgreSQL will manage its own rollback strategy, the question to ask is "how can I extend the rollback strategy ?". The thing you first should think of is "what caused the transaction to fail ?". In your try/catch structure, try to handle all the possible exceptions and run the transaction again or send feedback to the front-end application with some appropriate "don't do" messages. For me, this is the best way of handling things, it is less code, less overhead, more control, more user-friendly and your database will thank you.
A last point I want to shed light on, SQL standard is having a sqlstate code that can be use to communicate with back-end modules.
The failing operation during a transaction will return a sqlstate code, you can then use these codes to make appropriate drawbacks. You can make your own sqlstate codes, as long as it doesn't mess with the reserved ones (https://www.postgresql.org/message-id/20185.1359219138%40sss.pgh.pa.us).
For instance in a plpgsql function
...
$$
begin
...do something...if it goes wrong
raise exception 'custom exception message' using errcode='12345';
end
$$
...
This is a example using PDO in PHP (using the error code above) :
...
$pdo->beginTransaction();
try {
$s = $pdo->prepare('...');
$s->execute([$value]);
/**
* Simulate a null violation exception
* If it fails, PDO will not wait the commit
* and will throw the exception, the code below
* is not executed.
*/
$s->execute([null]);
/**
* If nothing wrong happened, we commit to change
* the database state.
*/
$pdo->commit();
}
catch (PDOException $e) {
/**
* It is important to also have the commit here.
* It will trigger PostgreSQL rollback.
* And make the pdo Object back to auto-commit mode.
*/
$pdo->commit();
if ($e->getCode() === '12345') {
send_feedback_to_client('please do not hack us.');
}
}
...
I would strongly suggest SqlAlchemy and use subtransactions. You can code like:
#some code
Session.begin(subtransactions=True)
#more stuff including sql activity then:
with Session.begin(nested=True):
# get the security
try:
foo = MyCodeToMakeFOO(args)
Session.add(foo)
Session.flush()
except:
log.error("Database hated your foo(%s) but I'll do the rest"%foo)
Most useful when the subtransaction is in a loop where you want to process the good records and log the bad ones.