I have two query that deadlock together
PERFORM id
FROM stack
WHERE id IN (SELECT tmp.stkid FROM tmp_push_bulk tmp WHERE tmp.stkid > 0)
ORDER BY id
FOR UPDATE OF stack
And
PERFORM stk.id
FROM stack stk
WHERE stk.referer IN (
SELECT tmp.id
FROM tmp_renew_stk tmp
)
ORDER BY stk.id
FOR UPDATE OF stk
The error is:
- PG (20:46:37) [14786]: Execute command failed: ERROR: deadlock detected
DETAIL: Process 14797 waits for ShareLock on transaction 183495696; blocked by process 24303.
Process 24303 waits for ShareLock on transaction 183495704; blocked by process 14797.
HINT: See server log for query details.
I also think that every process lock its row in ordering of id column, so deadlock is impossible. Can anyone tell me why?
It might be that the IN expression locks rows in unspecific order (untested). I would generally replace IN on bigger sets with a JOIN where possible. This is faster to begin with, thereby minimizing the chance for deadlocks. I would try:
Update: According to your comment you have many duplicates. Assuming dupes in the temp table I suggest a subquery for three purposes:
Fold duplicates.
Using DISTINCT it's particularly cheap to sort rows in the subquery at the same.
Apply WHERE conditon AND tmp.stkid > 0
PERFORM s.id
FROM stack s
JOIN (
SELECT DISTINCT stkid
FROM tmp_push_bulk
WHERE stkid > 0
) tmp ON tmp.stkid = s.id
ORDER BY 1
FOR UPDATE OF s;
And:
PERFORM s.id
FROM stack s
JOIN (
SELECT DISTINCT id
FROM tmp_renew_stk
) tmp ON tmp.id = s.referer
ORDER BY 1
FOR UPDATE OF s;
Related
I am getting a deadlock error on my code. The issue is that this deadlock error is happening on the very first query of the transaction. This query joins two tables, TableA and TableB and should lock a single row in TableA with id==table_a_id, and all the rows on TableB that have a foreign key for table_a_id.
The query looks as follows (I am using SQLAlchemy, this output is from printing the equivalent query from it and will have its code below as well):
SELECT TableB.id AS TableB_id
FROM TableA JOIN TableB ON TableA.id = TableB.table_a_id
WHERE TableB.id = %(id_1)s FOR UPDATE
The query looks as follows in SQLAlchemy syntax:
query = (
database.query(TableB.id)
.select_from(TableA)
.filter_by(id=table_a_id)
.join((TableB, TableA.id == TableB.table_a_id))
.with_for_update()
)
return query.all()
My question is, will this query atomically lock all those rows from both tables? If so, why would I get a deadlock already exactly on this query, given it's the first query of the transaction?
The query will lock the rows one after the other as they are selected. The exact order will depend on the execution plan. Perhaps you can add FOR UPDATE OF table_name to lock rows only in the table where you need them locked.
I have two more ideas:
rewrite the query so that it locks the rows in a certain order:
WITH b AS MATERIALIZED (
SELECT id, table_a_id
FROM tableb
WHERE id = 42
FOR NO KEY UPDATE
)
SELECT b.id
FROM tablea
WHERE EXISTS (SELECT 1 FROM b
WHERE tablea.id = b.table_a_id)
ORDER BY tablea.id
FOR NO KEY UPDATE;
Performance may not be as good, but if everybody selects like that, you won't get a deadlock.
lock the tables:
LOCK TABLE tablea, tableb IN EXCLUSIVE MODE;
That lock will prevent concurrent row locks and data modifications, so you will be safe from a deadlock.
Only do that as a last-ditch effort, and don't do it too often. If you frequently take high table locks like that, you keep autovacuum from running and endanger the health of your database.
I have a table that is a queue of tasks with each task requiring exclusive access to several resources. I want my query to select a single task that doesn't need resources claimed by other similar sessions.
If each task had to work on a single resource I would've written something like this:
select *
from tasks t
inner join resources r
on r.id = t.resource_id
order by t.submitted_ts
limit 1
for update skip locked
But since I have multiple resources I somehow have to lock them all:
select *
from tasks t
inner join task_details td
on t.id = td.task_id
inner join resources r
on r.id = td.resource_id
order by t.submitted_ts, t.id
limit ???
for update skip locked
I cannot limit by 1, since I need to lock all joined rows of resources.
It also seems to me that I should try and lock all rows of resources, so it must be not skip locked, but nowait for resources and skip locked for tasks.
First I had to create a helper function that either locks all linked rows or not:
create or replace function try_lock_resources(p_task_id bigint)
returns boolean
language plpgsql
as $$
begin
perform *
from task_details td
join resources r
on td.resource_id = r.resource_id
where td.task_id = p_task_id
for update of r nowait;
return true;
exception when lock_not_available then
return false;
end;
$$;
Then I needed to invoke this function for each row:
select *
from tasks
where processing_status = 'Unprocessed'
and try_lock_resources(task_id)
order by created_ts
limit 1
for update skip locked
After this query is run, only the returned row and its associated resources are locked. I verified that an identical query from another session returns the first unprocessed tasks that has no resources in common with the one returned by the first session.
P.S.: the original answer used a different query (which you shouldn't use as is):
with unprocessed_tasks as materialized (
select *
from tasks t
where processing_status = 'Unprocessed'
order by created_ts
)
select *
from unprocessed_tasks
where try_lock_resources(task_id)
limit 1
for update skip locked
The problem with this query is that the following could (and did happen):
session A runs the query, locks task X and starts working on it
session B starts running the query, the materialized CTE is run first, returning task X among other tasks
session A commits the transaction and releases all locks
session B finishes running the query, locks task X and starts working on it
LIMIT clause applies to joined table.
Instead of table A use subquery with it's own LIMIT.
SELECT
"table a"."учебный год",
"table b".семестр
FROM
(SELECT
"Учебный год"."учебный год"
FROM
"Учебный год"
ORDER BY
"Учебный год"."учебный год"
LIMIT 1) "table a"
INNER JOIN "Семестр" "table b" ON "table b"."учебный год" = "table a"."учебный год"
I'm on PostgreSQL 9.3. I'm the only one working on the database, and my code run queries sequentially for unit tests.
Most of the times the following UPDATE query run without problem, but sometimes it makes locks on the PostgreSQL server. And then the query seems to never ends, while it takes only 3 sec normally.
I must precise that the query run in a unit test context, i.e. data is exactly the same whereas the lock happens or not. The code is the only process that updates the data.
I know there may be lock problems with PostgreSQL when using update query for a self updating table. And most over when a LEFT JOIN is used.
I also know that a LEFT JOIN query can be replaced with a NOT EXISTS query for an UPDATE but in my case the LEFT JOIN is much faster because there is few data to update, while a NOT EXISTS should visit quite all row candidates.
So my question is: what PostgreSQL commands (like Explicit Locking LOCK on table) or options (like SELECT FOR UPDATE) I should use in order to ensure to run my query without never-ending lock.
Query:
-- for each places of scenario #1 update all owners that
-- are different from scenario #0
UPDATE t_territories AS upt
SET id_owner = diff.id_owner
FROM (
-- list of owners in the source that are different from target
SELECT trg.id_place, src.id_owner
FROM t_territories AS trg
LEFT JOIN t_territories AS src
ON (src.id_scenario = 0)
AND (src.id_place = trg.id_place)
WHERE (trg.id_scenario = 1)
AND (trg.id_owner IS DISTINCT FROM src.id_owner)
-- FOR UPDATE -- bug SQL : FOR UPDATE cannot be applied to the nullable side of an outer join
) AS diff
WHERE (upt.id_scenario = 1)
AND (upt.id_place = diff.id_place)
Table structure:
CREATE TABLE t_territories
(
id_scenario integer NOT NULL,
id_place integer NOT NULL,
id_owner integer,
CONSTRAINT t_territories_pk PRIMARY KEY (id_scenario, id_place),
CONSTRAINT t_territories_fkey_owner FOREIGN KEY (id_owner)
REFERENCES t_owner (id) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE RESTRICT
)
I think that your query was locked by another query. You can find this query by
SELECT
COALESCE(blockingl.relation::regclass::text,blockingl.locktype) as locked_item,
now() - blockeda.query_start AS waiting_duration, blockeda.pid AS blocked_pid,
blockeda.query as blocked_query, blockedl.mode as blocked_mode,
blockinga.pid AS blocking_pid, blockinga.query as blocking_query,
blockingl.mode as blocking_mode
FROM pg_catalog.pg_locks blockedl
JOIN pg_stat_activity blockeda ON blockedl.pid = blockeda.pid
JOIN pg_catalog.pg_locks blockingl ON(
( (blockingl.transactionid=blockedl.transactionid) OR
(blockingl.relation=blockedl.relation AND blockingl.locktype=blockedl.locktype)
) AND blockedl.pid != blockingl.pid)
JOIN pg_stat_activity blockinga ON blockingl.pid = blockinga.pid
AND blockinga.datid = blockeda.datid
WHERE NOT blockedl.granted
AND blockinga.datname = current_database()
This query I've found here http://big-elephants.com/2013-09/exploring-query-locks-in-postgres/
Also can use ACCESS EXCLUSIVE LOCK to prevent any query to read and write table t_territories
LOCK t_territories IN ACCESS EXCLUSIVE MODE;
More info about locks here https://www.postgresql.org/docs/9.1/static/explicit-locking.html
I can list the partitions with
SELECT
child.relname AS child_schema
FROM pg_inherits
JOIN pg_class child ON pg_inherits.inhrelid = child.oid ;
Is it guaranteed that they are listed in creation order? Because then only an additional LIMIT 1 is required. Else this will print the oldest, the one with the lowest number in its name: (my partitions are named name_1 name_2 name_3 ...)
SELECT
MIN ( trim(leading 'name_' from child.relname)::int ) AS child_schema
FROM pg_inherits
JOIN pg_class child ON pg_inherits.inhrelid = child.oid ;
Then I need to create a script which uses the result to execute DROP TABLE? Is there no easier way?
Is it guaranteed that they are listed in creation order?
No. This is likely as long as sequential scans and no dropped tables, but if you change the query and the plan changes, you could get rather unexpected results ordering-wise. Also I would expect that once free space is re-used, the ordering may change as well.
Your current trim query is the best way. Stick with it.
Problem
On a DB2 (version 9.5) the SQL statement
SELECT o.Id FROM Table1 o, Table2 x WHERE [...] FOR UPDATE WITH RR
gives me the error message SQLSTATE=42829 (The FOR UPDATE clause is not allowed because the table specified by the cursor cannot be modified).
Additional info
I need to specify WITH RR, because I'm running on isolation level READ_COMMITTED, but I need my query to block while there is another process running the same query.
Solution so far...
If I instead query like this:
SELECT t.Id FROM Table t WHERE t.Id IN (
SELECT o.Id FROM Table1 o, Table2 x WHERE [...]
) FOR UPDATE WITH RR
everything works fine.
New problem
But now I occasionally get deadlock exceptions when multiple processes perform this query simultaneously.
Question
Is there a way to formulate the FOR UPDATE query without introducing a place where a deadlock can occur?
First, for having isolation level READ_COMMITTED you do not need to specify WITH RR, because this results in the isolation level SERIALIZABLE. To specify WITH RS (Read Stability) is enough.
To propagate the FOR UPDATE WITH RS to the inner select you have to specify additionally USE AND KEEP UPDATE LOCKS.
So the complete statement looks like this:
SELECT t.Id FROM Table t WHERE t.Id IN (
SELECT o.Id FROM Table1 o, Table2 x WHERE [...]
) FOR UPDATE WITH RS USE AND KEEP UPDATE LOCKS
I made some tests on a DB2 via JDBC and it worked without deadlocks.