I am currently migrating from MySQL to postgres using pgbouncer for my connection pool.
We select/insert/update/delete lots of data from postgres and all comes from remote sources so we try to make the data quality as good as possible before an insert but sometimes some bad data slips through.
This causes in postgres to report current transaction is aborted, commands ignored until end of transaction block
This is fine except that connection through pgbouncer will report this error for every query. I get the same logic if i connect directly to postgres instead of pgbouncer too. I'd expect it to roll back whichever transaction that caused this issue.
Is there a way to just rollback and continue working like normal? Everything i've read just says fix the query but in this case it's not always possible.
You need to use the ROLLBACK command. This will undo everything since the last BEGIN TRANSACTION or START TRANSACTION. Note that transactions do not nest; if you've begun multiple transactions without committing, this will roll back the outermost transaction.
This will drop you into autocommit mode. You may want to issue a new BEGIN TRANSACTION command to open a new transaction.
You should also be able to ROLLBACK TO SAVEPOINT, if you have a savepoint from before the error.
(If at all possible, it is preferred to just fix the query, but depending on what you're doing, that may be prohibitively difficult.)
Related
I'm experiencing performance drop in one of our Firebird stored procedures and I have no clue why. I have found the following code in the mentioned SP:
declare v_dummy integer;
...
in autonomous transaction do
begin
-- insert may fail, but that is not a problem because it means the record is already there
insert into my_table(my_field) values (:input_param);
when ANY do
v_dummy = 1;
end
I see few dozens of records in RDB$TRANSACTIONS table with STATE 3, no relevant records in MON$TRANSACTIONS table.
The question is, if the insert fails will the autonomous transaction be rolled back or does the "when ANY do" prevent the rollback and there will be an opened transaction? Can I just remove the exception handling, so it will be rolled back automatically without raising an exception and blocking the rest of the code?
Using a when any do inside an autonomous transaction block will not rollback the transaction, instead it will commit once the block ends because the exception does not escape the block.
However, this is probably the desired result: committing transactions in Firebird is (relatively) cheaper than rolling back. In fact, if a transaction rolls back when nothing was changed, Firebird will convert a rollback into a commit anyway.
I don't think this is the cause of your performance problem, but without reproducible example, it is hard to reason about this.
As an aside, transactions with state 3 are rolled back, and rolled back transactions have ended. MON$TRANSACTIONS only shows active transactions, so rolled back transactions will not be shown in that virtual table.
Asumming that a have a table called "t1" in a "db1" and other table called "t2" in a "db2", and i need to insert a record on both tables or fails.
Connected to the db1 i guess i shall type this
BEGIN;
PREPARE TRANSACTION 'pepe'; -- this says the manual that makes your transaction gets stored on disk, so what is the purpose if i can't use it from another database?)
insert into t1 (field) values ('a_value');
COMMIT PREPARED 'pepe'
Connected to the db2 i guess that
BEGIN;
PREPARE TRANSACTION 'pepe'; -- this fails (the name of the transacttion, what is the meaning, what is use for?)
-- It complains about this "ERROR: transaction identifier "pepe" is already in use"
insert into t2 (field) values ('another_value');
COMMIT PREPARED 'pepe'
As you may see i don't get how to use two phase commits on postgres.
TL;DR
I'm not getting how to perform syncronization commands on differents DB within the same RDBMS.
I have read at oficial postgres documentation that for syncronizing works across two or more unrelated postgres databases an implementation of the so called "two-phases commits" protocol is at our disposal.
So i start trying to see how people do actually use them within the postgres, i do not see any actual example, at most i get to this post of a guy that was trying to experiment with several postgres client connected to the differents databases in order to emulate the multiple process running in pararell doing things to the several dbs that should end in a gratefully (all commit) or dreadfully way (all rollback).
Other sources i have peek looking foward examples were:
https://en.wikipedia.org/wiki/Two-phase_commit_protocol (this source
explain well the protocol but really makes me wonder where or who is
my "Coordinator" and how to send messages to the "participants"... i
only got prepare transaction <id>, commit prepared <id> or
rollback prepared <id> commands at my disposal)
Two phase commit
https://dba.stackexchange.com/questions/145656/dependent-transaction-in-separate-database-connections
https://www.endpointdev.com/blog/2010/07/distributed-transactions-and-two-phase/
https://www.citusdata.com/blog/2017/11/22/how-citus-executes-distributed-transactions/
(From a golang client-app) https://github.com/go-pg/pg/issues/490
Please i'm really confused, i hope horse_with_no_name to appear here and enlightme (as happen in the past) or any other charity soul that can help me.
Thanks in advance!
Resolution (After Laurenz's Answer)
Connected to the db1, these are the sql lines to execute:
BEGIN;
-- DO THINGS TO BE DONE IN A ALL OR NOTHING FASHION
-- Stop point --
PREPARE TRANSACTION 't1';
COMMIT PREPARED 't1' || ROLLBACK PREPARED 't1' (decision requires awareness and coordination)
meanwhile connected to the db2 these will be the script to execute:
BEGIN;
-- DO THINGS TO BE DONE IN A ALL OR NOTHING FASHION
-- Stop point --
PREPARE TRANSACTION 't2';
COMMIT PREPARED 't2' || ROLLBACK PREPARED 't2'
The -- Stop point -- is where a coordinator process (for example
an application executing the statement, or a human behind a psql
client console or pgAdminII) shall stop the execution of both
scripts (actually not execute any further instruction, that is what i mean by stop).
Then, first on db1 (and then on db2, or viceversa) the
coordinator process (whatever been human or not) must run PREPARE TRANSACTION on each connection.
If one of then fails, then the coordinator must run ROLLBACK PREPARED on those database where the transaction was already prepared and ROLLBACK on the others.
If no one fails the coordinator must run COMMIT PREPARED on all involved databases, an operation that shall not fail ever (like existing the home when you are one step outside your house with all the things properly set to exit safely)
I think you misunderstood PREPARE TRANSACTION.
That statement ends work on the transaction, that is, it should be issued after all the work is done. The idea is that PREPARE TRANSACTION does everything that could potentially fail during a commit except for the commit itself. That is to guarantee that a subsequent COMMIT PREPARED cannot fail.
The idea is that processing is as follows:
Run START TRANSACTION on all database involved in the distributed transaction.
Do all the work. If there are errors, ROLLBACK all transactions.
Run PREPARE TRANSACTION on all databases. If that fails anywhere, run ROLLBACK PREPARED on those database where the transaction was already prepared and ROLLBACK on the others.
Once PREPARE TRANSACTION has succeeded everywhere, run COMMIT PREPARED on all involved databases.
That way, you can guarantee “all or nothing” across several databases.
One important component here that I haven't mentioned is the distributed transaction manager. It is a piece of software that persistently memorizes where in the above algorithm processing currently is so that it can clean up or continue committing after a crash.
Without a distributed transaction manager, two-phase commit is not worth a lot, and it is actually dangerous: if transactions get stuck in the “prepared” phase but are not committed yet, they will continue to hold locks and (in the case of PostgreSQL) block autovacuum work even through server restarts, as such transactions must needs be persistent.
This is difficult to get right.
I build a pgsql db, and using JDBC upload data to db. but sometime i upload wrong data.
so in pgsql it has some functions or sql statement can allow rollback to last step, without using backup file?
Thanks,
If you run things in a transaction, you may end the transaction with either "commit" or "rollback", hence in your script you may end up with an interactive question to the user: "does this seem OK?" and then either do a commit or rollback.
Beware that long-lasting "idle in transaction" may be harmful for the performance, so you should do rollback (or commit) after some timeout. Of course, it would be even better to ask the question to the user before doing any DB operations.
If something is already committed to the database, as far as I know it cannot be rolled back unless you have backups (dunno - perhaps it's theoretically possible to do it based on the WALs ... but probably not). Some years ago I was implementing a system with a "warm standby backup server" that would always be half an hour behind the production database - I could then issue a "distress signal" and the backup server would come online with the old state of the database, i.e. if I'd done some mistake like update users set password=md5('foo' || salt) instead of update users set password=md5('foo' || salt) where username='tobixen'. If the primary database went down, the "warm standby" would roll forward and then come online. I may post additional details about this if there is any interesst for it.
If you're expecting trouble, it's also usually possible to design the database and the scripts in such a manner that it's trivial to do a manual rollback.
Is there a known problem with SQLite giving a "database is locked" error for a second query in a single transaction when using Perl DBD::SQLite? Scenario: Linux, Perl DBI, AutoCommit => 0, a subroutine with two code blocks (using the blocks to localize variable names). In the first code block a query handle is created by prepare() on a select statement, it is executed() and the block closed. The second code block another query handle is created by prepare for an update statement, and frequently (30% of the time) SQLite/DBI gives a database locked error at this stage. I think the error happens during prepare() and not during the execute().
My work around is to commit after the first query. ( Calling finish on the first query did not help). I prefer not to commit for several reasons relating to elegance and performance. The original code has worked fine for many years with Postgres as the database. I tried sqlite_use_immediate_transaction with no effect.
In all other situations, I've found SQLite to perform very well, so I suspect this is an oversight in the DBD driver, rather than an issue with SQLite. Sadly, my current code is a big pile of scripts and modules, so I don't have a short, single file test case.
Not related to this in anyway is it: Transaction and Database Locking from the DBD::SQLite perldoc?
Transaction by AutoCommit or begin_work is nice and handy, but sometimes you may get an annoying "database is locked" error. This typically happens when someone begins a transaction, and tries to write to a database while other person is reading from the database (in another transaction). You might be surprised but SQLite doesn't lock a database when you just begin a normal (deferred) transaction to maximize concurrency. It reserves a lock when you issue a statement to write, but until you actually try to write with a commit statement, it allows other people to read from the database. However, reading from the database also requires shared lock, and that prevents to give you the exclusive lock you reserved, thus you get the "database is locked" error, and other people will get the same error if they try to write afterwards, as you still have a pending lock. busy_timeout doesn't help in this case.
To avoid this, set a transaction type explicitly. You can issue a begin immediate transaction (or begin exclusive transaction) for each transaction, or set sqlite_use_immediate_transaction database handle attribute to true (since 1.30_02) to always use an immediate transaction (even when you simply use begin_work or turn off the AutoCommit.).
my $dbh = DBI->connect("dbi:SQLite::memory:", "", "", {
sqlite_use_immediate_transaction => 1,
});
Note that this works only when all of the connections use the same (non-deferred) transaction. See http://sqlite.org/lockingv3.html for locking details.
I'm trying to debug an application (under PostgreSQL) and came across the following error:
"current transaction is aborted, commands ignored".
As far as I can understand a "transaction" is just a notion related to the underlying database connection.
If the connection has an auto commit "false", you can execute queries through the same Statement as long as it isn't failing. In which case you should rollback.
If auto commit is "true" then it doesn't matter as long as all your queries are considered atomic.
Using auto commit false, I get the aforementioned error by PostgreSQL even when a simple
select * from foo
fails, which makes me ask, under which SQLException(s) is a "transaction" considered invalid and should be rolled backed or not used for another query?
using MacOS 10.5, Java 1.5.0_16, PostgreSQL 8.3 with JDBC driver 8.1-407.jdbc3
That error means that one of the queries sent in a transaction has failed, so the rest of the queries are ignored until the end of the current transaction (which will automatically be a rollback). To PostgreSQL the transaction has failed, and it will be rolled back in any case after the error with one exception. You have to take appropriate measures, one of
discard the statement and start anew.
use SAVEPOINTs in the transaction to be able to get back to that point in time and try another path. (This is the exception)
Enable query logging to see which query is the failing one and why.
In any case the exact answer to your question is that any SQLException should mean a rollback happened when the end of transaction command is sent, that is when a COMMIT or ROLLBACK (or END) is issued. This is how it works, if you use savepoints you'll still be bound by the same rules, you'll just be able to get back to where you saved and try something else.
It seems to be a characteristic behaviour of PostgreSQL that is not shared by most other DBMS. In general (outside of PostgreSQL), you can have one operation fail because of an error and then, in the same transaction, can try alternative actions that will succeed, compensating for the error. One example: consider a merging (insert/update) operation. If you try to INSERT the new record but find that it already exists, you can switch to an UPDATE operation that changes the existing record instead. This works fine in all the main DBMS. I'm not certain that it does not work in PostgreSQL, but the descriptions I've seen elsewhere, as well as in this question, suggest that when the attempted INSERT means that any further activity in the transaction is doomed to fail too. Which is at best draconian and at worst 'unusable'.