Can someone please explain to me the following situation:
I am using Postgresql 12 as main rdbms in my project, there are several background jobs accessing and writing to the database in parallel, also there are some user interactions (which of course produce updates and inserts to the database from the front of application)
Periodically i am getting exceptions like this one:
SQLSTATE[40P01]: Deadlock detected: 7 ERROR: deadlock detected
DETAIL: Process 18046 waits for ShareLock on transaction 212488; blocked by process 31036.
Process 31036 waits for ShareLock on transaction 212489; blocked by process 18046.
HINT: See server log for query details.
CONTEXT: while updating tuple (1637,16) in relation "my_table"
Inside my application i don't lock manually any rows or tables during my transactions. But i have 'large' transactions that can modify a lot of rows in single operation frequently. So the questions are:
Does ordinary transactions produce table-wide locks, or row-wide locks? (I assume yes, unless this whole situation is magic)
Shouldn't the rdbms resolve automatically this kind of problems when two queries are trying to modify the same resource, if they are wrapped inside transaction?
If answer to the second question is "no" then how i should handle that kind of situations?
re 1) DML statements only lock the rows that are modified. There is no lock escalation in Postgres where the whole table is locked for writes. There is a "table lock" but that is only there to prevent concurrent DDL - a SELECT will also acquire that. Those share locks don't prevent DML on the table.
re 2) no, the DBMS can not resolve this because a deadlock means tx1 is waiting for a lock to be released from tx2 and tx2 is waiting for a lock to be released by tx1. How would the DBMS know what to do? The only way the DBMS can solve this is by choosing one of the two sessions as a victim and kill the transaction (which is the error you see).
re 3) the usual approach to avoiding deadlocks is to always update rows in the same order. Which usually turns the deadlock into a simple "lock wait" for the second transaction.
Assume the following UPDATE sequence
tx1 tx2
-------------------------------
update id = 1 |
| update id = 2
update id = 2 |
(tx1 waits) |
| update id = 1
(now we have a deadlock)
If you always update the rows in e.g. ascending order this changes to:
tx1 tx2
-------------------------------
update id = 1 |
| update id = 1
| (waits)
update id = 2 |
|
|
commit; |
(locks released) |
|
| update id = 2
| commit;
So you don't get a deadlock, just a wait for the second transaction.
All SQL statements that affect tables will take a lock on the table (albeit not necessarily a strong one). But that doesn't seem to be your problem here.
All SQL statements that modify a row (or SELECT ... FOR UPDATE) will lock the affected rows. Your two transactions probably blocked on a row-level lock.
Yes; that is what the error message shows. PostgreSQL has resolved the deadlock by killing one of the involved transactions.
If transactoin 1 holds a lock that transaction 2 is waiting for and vice versa, there is no other way to resolve the situation. The only way to release a lock is to end the transaction that holds it.
You should catch the error on your application code and retry the database transaction. A deadlock is a transient error.
If you get a lot of deadlocks, you should try to reduce them. What helps is to keep your transactions short and small. If that is not an option, make sure that all transactions that lock several rows lock them in the same order.
Related
In Java application I frequently see such errors:
org.springframework.r2dbc.UncategorizedR2dbcException: executeMany; SQL [select * from table1 where id=:id and column1 <> :a for update]; could not serialize access due to concurrent update; nested exception is io.r2dbc.postgresql.ExceptionFactory$PostgresqlTransientException: [40001] could not serialize access due to concurrent update
Transaction with query select * from table1 where id=:id and column1 <> :a for update was rollbacked.
Transaction isolation level - REPEATABLE READ.
How can I see what has locked this row? Lock is very short (milliseconds).
I see no helpful information in Postgres log and application log.
The problem here is not a concurrent lock, but a concurrent data modification. It is perfectly normal to get that error, and if you get it, you should simply repeat the failed transaction.
There is no way to find out which concurrent transaction updated the row unless you log all DML statements.
If you get a lot of these errors, you might consider switching to pessimistic locking using SELECT ... FOR NO KEY UPDATE.
I have a table with 500k elements. Now I want to add a new column
(type boolean, nullable = false) without a default value.
The query to do so is running like for ever.
I'm using PostgreSQL 12.1, compiled by Visual C++ build 1914, 64-bit on my Windows 2012 Server
In pgAdmin I can see the query is blocked by PID 0. But when I execute this query, I can't see the query with pid = 0
SELECT *
FROM pg_stat_activity
Can someone help me here? Why is the query blocked and how can I fix this to add a new column to my table.
UPDATE attempt:
SELECT *
FROM pg_prepared_xacts
Update
It works after rollback all prepared transaction.
ROLLBACK PREPARED 'gid goes here';
You have got stale prepared transactions. I say that as in "you have got the measles", because it is a disease for a database.
Such prepared transactions keep holding locks and block autovacuum progress, so they will bring your database to its knees if you don't take action. In addition, such transactions are persisted, so even a restart of the database won't get rid of them.
Remove them with
ROLLBACK PREPARED 'gid goes here'; /* use the transaction names shown in the view */
If you use prepared transactions, you need a distributed transaction manager. That is a piece of software that keeps track of all prepared transactions and their state and persists that information, so that no distributed transaction can become stale. Even if there is a crash, the distributed transaction manager will resolve in-doubt transactions in all involved databases.
If you don't have that, don't use prepared transactions. You now know why. Best is to set max_prepared_transactions to 0 in that case.
I'm pretty new to PostgreSQL and I'm sure I'm missing something here.
The scenario is with version 11, executing a big drop table and insert transaction on a given table with the nodejs driver, which may take 30 minutes.
While doing that, if I try to query with select on that table using the jdbc driver, the query execution waits for the transaction to finish. If I close the transaction (by finishing it or by forcing it to exit), the jdbc query becomes responsive.
I thought I can read a table with one connection while performing a transaction with another one.
What am I missing here?
Should I keep the table (without dropping it at the beginning of the transaction) ?
DROP TABLE takes an ACCESS EXCLUSIVE lock on the table, which is there precisely to prevent it from taking place concurrently with any other operation on the table. After all, DROP TABLE physically removes the table.
Since all locks are held until the end of the database transaction, all access to the dropped table is blocked until the transaction ends.
Of course the files are only removed when the transaction commits, so you might wonder why PostgreSQL doesn't let concurrent transactions read in the mean time. But that would mean that COMMIT may be blocked by a concurrent reader, or a SELECT might cause a system error in the middle of reading, both of which don't sound appealing.
I'm running a Postgresql 9.5.2 server, and I'm occasionally seeing a message like:
ERROR: deadlock detected
Detail: Process 1234 waits for ShareLock on transaction 3042999324; blocked by process 5678.
Process 5678 waits for ShareLock on transaction 3042999328; blocked by process 1234.
Hint: See server log for query details.
Where: while locking tuple (5389,30) in relation "asset"
If it contains any information about the row or column that's causing the deadlock, it will help me debug the big ugly common-table expression that's causing the error in the first place.
I figured it out while looking up the correct terminology to use while asking my question: tuple refers to the row's ctid, a system column on every row indicating the physical location of the version of the row in question. (When a row is updated, Postgresql keeps the old version around for a while in order to fulfill ACID guarantees.)
You can select the data simply with:
SELECT * from "asset" where ctid = '(5389,30)';
However, if you wait too long (like I did), an autovacuum job might clean up that version of the row if it's no longer in use.
Let's assume in SQL window 1 I do:
-- query 1
BEGIN TRANSACTION;
UPDATE post SET title = 'edited' WHERE id = 1;
-- note that there is no explicit commit
Then from another window (window 2) I do:
-- query 2
SELECT * FROM post WHERE id = 1;
I get:
1 | original title
Which is fine as the default isolation level is READ COMMITTED and because query 1 is never committed, the change it performs is not readable until I explicitly commit from window 1.
In fact if I, in window 1, do:
COMMIT TRANSACTION;
I can then see the change if I re-run query 2.
1 | edited
My question is:
Why is query 2 returning fine the first time I run it? I was expecting it to block as the transaction in window 1 was not committed yet and the lock placed on row with id = 1 was (should be) an unreleased exclusive one that should block a read like the one performed in window 2. All the rest makes sense to me but I was expecting the SELECT to get stuck until an explicit commit in window 1 was executed.
The behaviour you describe is normal and expected in any transactional relational database.
If PostgreSQL showed you the value edited for the first SELECT it'd be wrong to do so - that's called a "dirty read", and is bad news in databases.
PostgreSQL would be allowed to wait at the SELECT until you committed or rolled back, but it isn't required to by the SQL standard, you haven't told it you want to wait, and it doesn't have to wait for any technical reason, so it returns the data you asked for immediately. After all, until it's committed, that update only kind-of exists - it still might or might not happen.
If PostgreSQL always waited here, then you'd quickly land up with a situation where only one connection could be doing anything with the database at a time. Not pretty for performance, and totally unnecessary the vast majority of the time.
If you want to wait for a concurrent UPDATE (or DELETE), you'd use SELECT ... FOR SHARE. (But be aware that this won't work for INSERT).
Details:
SELECT without a FOR UPDATE or FOR SHARE clause does not take any row level locks. So it sees whatever is the current committed row, and is not affected by any in-flight transactions that might be modifying that row. The concepts are explained in the MVCC section of the docs. The general idea is that PostgreSQL is copy-on-write, with versioning that allows it to return the correct copy based on what the transaction or statement could "see" at the time it started - what PostgreSQL calls a "snapshot".
In the default READ COMMITTED isolation snapshots are taken at the statement level, so if you SELECT a row, COMMIT a change to it from another transaction, and SELECT it again you'll see different values even within one transation. You can use SNAPSHOT isolation if you don't want to see changes committed after the transaction begins, or SERIALIZABLE isolation to add further protection against certain kinds of transaction inter-dependencies.
See the transaction isolation chapter in the documentation.
If you want a SELECT to wait for in-progress transactions to commit or rollback changes to rows being selected, you must use SELECT ... FOR SHARE. This will block on the lock taken by an UPDATE or DELETE until the transaction that took the lock rolls back or commits.
INSERT is different, though - the tuples just don't exist to other transactions until commit. The only way to wait for concurrent INSERTs is to take an EXCLUSIVE table-level lock, so you know nobody else is changing the table while you read it. Usually the need to do that means you have a design problem in the application though - your app should not care if there are uncommitted inserts still in flight.
See the explicit locking chapter of the documentation.
In PostgreSQL's MVCC implementation, the principle is reading does not block writing and vice-versa. The manual:
The main advantage of using the MVCC model of concurrency control
rather than locking is that in MVCC locks acquired for querying
(reading) data do not conflict with locks acquired for writing data,
and so reading never blocks writing and writing never blocks reading.
PostgreSQL maintains this guarantee even when providing the strictest
level of transaction isolation through the use of an innovative
Serializable Snapshot Isolation (SSI) level.
Each transaction only sees (mostly) what has been committed before the transaction began.
That does not mean there'd be no locking. Not at all. For many operations various kinds of locks are acquired. And various strategies are applied to resolve possible conflicts.