AccessExclusive lock origin - postgresql

I am running the following query in first transaction:
BEGIN ISOLATION LEVEL repeatable read;
SELECT balance from "Users" WHERE id = 1 FOR UPDATE;
UPDATE "Users" SET "balance"="balance"+1 WHERE id = 1;
Then the second, from different connection which is exactly same. On SELECT ... FOR UPDATE it's waiting.
Then I run the following to see the locks
select t.relname,l.locktype,l.tuple,page,virtualtransaction,pid,mode,granted from pg_locks l, pg_stat_all_tables t where l.relation=t.relid order by relation asc;
It shows row locks (ROW SHARE, ROW EXCLUSIVE), which is fine. But I also see
Users tuple 9 0 11/17085 199957 AccessExclusiveLock TRUE
According to documentation AccessExclusiveLock comes from:
Acquired by the ALTER TABLE, DROP TABLE, TRUNCATE, REINDEX, CLUSTER,
and VACUUM FULL commands. This is also the default lock mode for LOCK
TABLE statements that do not specify a mode explicitly.
I am not explicitly doing any of them and I can't find in docs how implicitly this lock is acquired. Moreover, what means tuple lock type and 9?
update #1:
I used the following query to get more info:
SELECT a.datname,
l.relation::regclass,
l.transactionid,
l.mode,
l.GRANTED,
a.usename,
a.query,
a.query_start,
age(now(), a.query_start) AS "age",
a.pid
FROM pg_stat_activity a
JOIN pg_locks l ON l.pid = a.pid
ORDER BY a.query_start;
So indeed database is same, query is the one that is blocked.
mydb "Users" AccessExclusiveLock TRUE appuserdev SELECT balance from "Users" WHERE id = 1 for update; 2021-08-30 08:06:38.864007+00 00:08:06.978082 205464

The docs you are reading apply to AccessExclusiveLock on tables. But what you are seeing is not a table lock. I don't think the lock modes for non-tables are explicitly documented anywhere, but you can generally figure it out through analogy.
On the other hand, I also don't see what you see. I see an ExclusiveLock, not an AccessExclusiveLock, being held on the tuple.

Related

Is it possible to guarantee the order row locks are obtained across multiple UPDATE CTEs in PostgreSQL 14?

On PostgreSQL 14
To avoid deadlocks, the documentation provides the following guidance:
The best defense against deadlocks is generally to avoid them by being certain that all applications using a database acquire locks on multiple objects in a consistent order.
The documentation on CTEs states:
The sub-statements in WITH are executed concurrently with each other and with the main query. Therefore, when using data-modifying statements in WITH, the order in which the specified updates actually happen is unpredictable.
I assume this also means the order in which the locks are obtained for the specified updates is also unpredictable.
Does this apply even if there is a dependency between CTEs? I assumed so as I could not find any documentation that states otherwise. Perhaps because the query planer has the potential to rewrite the query in such a way that the dependency no longer exists?
I did some experimenting:
create table table_a (id int primary key, fu int);
create table table_b (id int primary key, bar int);
WITH update_a AS (
UPDATE table_a
SET fu = fu + 1
WHERE id = 7
RETURNING
8 AS b_id,
fu,
pg_sleep(5)
),
update_b AS (
UPDATE table_b
SET bar = bar + 1
FROM update_a
WHERE id = b_id
RETURNING
bar,
pg_sleep(3)
)
SELECT fu, bar FROM update_a, update_b;
I executed the above query and on another connection I executed:
update table_b set bar = 0; \timing on \watch 1
The results were:
UPDATE 1 | 1.392 ms
UPDATE 1 | 1.123 ms
UPDATE 1 | 1.648 ms
UPDATE 1 | 1.419 ms
UPDATE 1 | 2903.621 ms
Which shows the second connection was able to repeatably obtain a lock and update table_b without waiting for the first 5 seconds, but then had to wait to obtain the lock for ~3 seconds afterwards. This aligns with the pg_sleep(5) and pg_sleep(3) which I assume means PSQL did not lock the table_b row until after the update_a CTE completed.
However, I also noticed the pg_locks table contains a RowExclusiveLock for both table_a and table_b immediately after executing the first query, and remained the same throughout the entire 8 seconds. Though I confess to not knowing how to really read this table:
pid
virtualtransaction
transactionid
relname
locktype
mode
granted
waitstart
2318
3/139
NULL
table_a
relation
RowExclusiveLock
t
NULL
2318
3/139
NULL
table_a_pkey
relation
RowExclusiveLock
t
NULL
2318
3/139
NULL
table_b
relation
RowExclusiveLock
t
NULL
2318
3/139
NULL
table_b_pkey
relation
RowExclusiveLock
t
NULL
2318
3/139
NULL
NULL
virtualxid
ExclusiveLock
t
NULL
2318
3/139
13304
NULL
transactionid
ExclusiveLock
t
NULL
If I rewrite the query to remove the dependency by omitting FROM updated_b, the behavior is inconsistent and seems to change based on table ordering in the SELECT statement or other factors.
Perhaps I got lucky? Again, I could not find this documented anywhere so I assume this behavior is not guaranteed.
If so, does this mean if one wanted to update two rows from different tables in a single query using CTEs, there is no way to guarantee the order the rows are locked, therefore allowing for the possibility of a deadlock?
(I realize I there are alternate solutions such using multiple queries, functions/DO statements, Repeatable Read isolation level, etc., but I am curious about the specific behavior of CTEs and locking order.)

postgres delete query is hanging forever, although its associated transaction is in progress and all locks are granted

I have a query doing some rolling delete from old entries from my database:
delete from my_table where last_updated_at < now() - '1 day'::interval
This query gets triggered every 15 Minutes. As the query stopped working some time ago, we started investigating our database and noticed that there is one transaction having all required locks to do its work, but it hangs and blocks all subsequent ones.
The results of our analysis follow:
Using this script we got the following results:
It clearly shows, that one of the delete queries, whose pid is 2674 somehow blocked all other delete queries causing a "traffic jam". In order to see, whether the process 2674 is waiting for any locks, we issued following query:
select relation::regclass, * from pg_locks l
join pg_stat_activity a on l.pid = a.pid
where not granted and a.pid = 2674 order by query_start ;
but the result was empty. A quick look in the lock table revealed to us, that the transaction id associated with this pid is 1838047967:
select l.transactionid from pg_locks l where l.pid = 2674 and l.transactionid is not null;
1838047967
Quick check of the transaction status shows, that the transaction is running:
SELECT pg_xact_status (xid8 '1838047967');
in progress
The problem is, that it has not been doing anything for more than 2 days.
Funny enough, after canceling the backend 2674 via command
SELECT pg_cancel_backend(2674);
the next backend showed in the tree above, with pid 4679, overtakes the role of the canceled backend and all queries listed in this post yield similar results for this new backend (i.e. in progress transaction with all locks granted)
Following is the result of the query:
SELECT a.pid, pg_blocking_pids(a.pid), a.query, a.query_start,
l.transactionid,
l.mode,
l.GRANTED
FROM pg_stat_activity a
join pg_locks l on l.pid = a.pid
WHERE backend_type = 'client backend'
order by a.query_start ;
after canceling backend 2674, with backend 4679 being the offending one (marked in blue). It is clearly visible that backend 4679 got all locks and is blocking subsequently the backend with pid 4555 and therefore the latter cannot acquire ShareLock.
The entry in pg_stat_activity for this backed looks like following:
To summarize:
We have an in progress transaction holding all required locks to do its delete work, but this transaction has been hanging for more than 2 days, and it blocked all subsequent transactions. The question is, what can we check to find out what is causing this transaction to hang.
Postgres Version: 14.3

In certain cases, can UPDATE transactions fail rather than block waiting for "FOR UPDATE lock"?

I'm using Postgres 9.6.5.
The docs say:
13.3. Explicit Locking
13.3.2. Row-level Locks" -> "Row-level Lock Modes" -> "FOR UPDATE":
FOR UPDATE causes the rows retrieved by the SELECT statement to be locked as though for update. This prevents them from being locked, modified or deleted by other transactions until the current transaction ends. That is, other transactions that attempt UPDATE, DELETE, SELECT FOR UPDATE, SELECT FOR NO KEY UPDATE, SELECT FOR SHARE or SELECT FOR KEY SHARE of these rows will be blocked until the current transaction ends; conversely, SELECT FOR UPDATE will wait for a concurrent transaction that has run any of those commands on the same row, and will then lock and return the updated row (or no row, if the row was deleted). ...
The mode is also acquired by any DELETE on a row, and also by an UPDATE that modifies the values on certain columns. Currently, the set of columns considered for the UPDATE case are those that have a unique index on them that can be used in a foreign key (so partial indexes and expressional indexes are not considered), but this may change in the future.
Regarding UPDATEs on rows that are locked via "SELECT FOR UPDATE" in another transaction, I read the above as follows: other transactions that attempt UPDATE of these rows will be blocked until the current transaction ( which did "SELECT FOR UPDATE" for those rows ) ends, unless the columns in these rows being UPDATE'ed are those that don't have a unique index on them that can be used in a foreign key.
Is this correct ? If so, if I have a table "program" with a text column "stage" ( this column doesn't fit "have a unique index on them that can be used in a foreign key" ), and I have a transaction that does "SELECT FOR UPDATE" for some rows followed by UPDATE'ing "stage" in these rows, is it correct that other concurrent transactions UPDATE'ing "stage" on these rows can fail, rather than block until the former transaction ends ?
Your transactions can fail if a deadlock is detected in one of the UPDATE or SELECT...FOR UPDATE, for example:
Transaction 1
BEGIN;
SELECT * FROM T1 FOR UPDATE;
SELECT * FROM T2 FOR UPDATE;
-- Intentionally, no rollback nor commit yet
Transaction 2
BEGIN;
SELECT * FROM T2 FOR UPDATE; -- blocks T2 first
SELECT * FROM T1 FOR UPDATE; -- exception will happen here
The moment the second transaction tries to lock T1 you'll get:
ERROR: 40P01: deadlock detected
DETAIL: Process 15981 waits for ShareLock on transaction 3538264; blocked by process 15942.
Process 15942 waits for ShareLock on transaction 3538265; blocked by process 15981.
HINT: See server log for query details.
CONTEXT: while locking tuple (0,1) in relation "t1"
LOCATION: DeadLockReport, deadlock.c:956
Time: 1001.728 ms

postgresql is SELECT FOR UPDATE over multiple rows atomic?

Lets say that there are multiple parallel transactions that all do the same query:
SELECT * FROM table1 FOR UPDATE;
Can this result in a deadlock?
To put it in another way. Is the operation "lock all rows" in the above statement atomic or are the locks acquired along the way while the the records are processed?
Yes, it can result in a deadlock.
This is pretty easy to demonstrate. Set up a test table:
CREATE TABLE t AS SELECT i FROM generate_series(1,1000000) s(i);
... and then run these two queries in parallel:
SELECT i FROM t ORDER BY i FOR UPDATE;
SELECT i FROM t ORDER BY i DESC FOR UPDATE;
You can prevent deadlocks by ensuring that all processes acquire their locks in the same order. Alternatively, if you want to lock every record in the table, you can do it atomically with a table lock:
LOCK t IN ROW SHARE MODE;

Postgresql - Why is DROP VIEW command hanging?

I want to perform a simple DROP VIEW ... but it hangs.
I have run this query SELECT * FROM pg_locks WHERE NOT granted taken from this page on Lock Monitoring.
However the following query they suggest returns no results:
SELECT bl.pid AS blocked_pid,
a.usename AS blocked_user,
kl.pid AS blocking_pid,
ka.usename AS blocking_user,
a.query AS blocked_statement
FROM pg_catalog.pg_locks bl
JOIN pg_catalog.pg_stat_activity a ON a.pid = bl.pid
JOIN pg_catalog.pg_locks kl ON kl.transactionid = bl.transactionid AND kl.pid != bl.pid
JOIN pg_catalog.pg_stat_activity ka ON ka.pid = kl.pid
WHERE NOT bl.granted;
Where should I look now ?
Finally I figure out what was wrong. Here are the steps to find the root cause:
Solution
Step 1 : List requested locks not granted
select * from pg_locks where not granted;
In my case, an attempt to lock, with the mode AccessExclusiveLock, the view I want to drop was not granted. This is why my DROP VIEW... hangs.
Step 2 : Find which other process(es) held a conflicting lock
select * from pg_locks where relation = <oid_of_view>
Here I list all processes locking or trying to lock on my view. I found out two processes, the one that want to drop the view and... another one.
Step 3 : Find out what other process(es) is/are doing now
select xact_start,query_start,backend_start,state_change,state from pg_stat_activity where pid in (<list_of_other_process(es)_pid>);
I had only one process holding a lock in my case. Surprisingly, its state was : idle in transaction
I was not able to drop the view because another process was idle in transaction. I simply kill it to solve my issue. For example, if the procpid was 8484 and let's suppose my postgresql server runs on a Linux box, then in the shell, I execute the following command:
$ kill -9 8484
Discussion
If you face similar issue, you can quickly find out what's going on by reproducing steps 1,2,3. You may need to customize Step 2 in order to find other conflicting process(es).
References
Lock Monitoring
Lock Dependency Information
View Postgresql Locks
I had a similar problem but the accepted answer didn't work for me as I do not have admin access to kill any process. Instead, this is how I managed to solve the problem:
Issue SELECT * FROM pg_stat_activity; to get the stats about the PostgreSQL activities.
In query column, look for the queries that read from that view. You may choose to narrow down your search by only looking into the rows related to your username (using username column) or query_start if you know when the issue emerged. There could be more than one row associated with your unwanted view.
Identify all pid from the rows in the above step and plug them into SELECT pg_terminate_backend(<pid>); (instead of <pid>) one by one and run them.
Now you should be able to drop your view.
Please note that as you terminate the backend processes using pg_terminate_backend(), you may face some errors. The reason is that terminating some process may automatically end other processes. Therefore, some of the identified PIDs might be invalid by the time.
As a summary, this solution from comments worked for me:
Step 1: Find the pid:
select * from pg_stat_activity
where pid in
(select pid from pg_locks
where relation =
(select relation from pg_locks where not granted));
Step 2: kill pid:
kill -9 pid