I use PostgreSQL 9.2, and I do not use explicit locking anywhere, neither LOCK statement nor SELECT ... FOR UPDATE. However, recently I got ERROR: 40P01: deadlock detected. The query where deadlock was detected is wrapped in transaction block though. Anyway, how comes it?
You don't need any explicit LOCK to go into a deadlock. Here's a very simple demo from scratch with only INSERTs:
create table a(i int primary key);
create table b(i int primary key);
Session #1 does:
begin;
insert into a values(1);
Then session #2 does:
begin;
insert into b values(1);
insert into a values(1);
-- here it goes into waiting for session #1 to finish its transaction
Then session #1 does:
insert into b values(1);
And then the deadlock occurs:
ERROR: deadlock detected
DETAIL: Process 9571 waits for ShareLock on
transaction 4150; blocked by process 9501.
Process 9501 waits for
ShareLock on transaction 4149; blocked by process 9571.
HINT: See
server log for query details.
The same could happen with simple UPDATEs or a combination of UPDATEs and INSERTs.
These operations take implicit locks, and if they happen in different sessions in different orders, they may deadlock.
I would suspect hash indexes first.
Switch any hash-indexes you have to B-tree
Use Serializable isolation level if it seems appropriate.
Related
I am trying to create indexes on multiple (1000) partitioned tables. As I'm using Postgres 10.2, I would have to do this for each of the partition separately, having to execute 1000 queries for the same.
I have figured how to do it, and it does work on environments where the table size(s) and number of transactions are very less. Below is the query to be executed for one of the table (which is to be repeated for all the tables ( user_2, user_3, etc.)
CREATE INDEX IF NOT EXISTS user_1_idx_location_id
ON users.user_1 ( user_id, ( user_data->>'locationId') );
where user_data is a jsonb column
This query does not work for large tables, with high number of transactions - when I run it for all the tables at once. Error thrown:
ERROR: SQL State : 40P01
Error Code : 0
Message : ERROR: deadlock detected
Detail: Process 77999 waits for ShareLock on relation 1999264 of database 16311; blocked by process 77902.
Process 77902 waits for RowExclusiveLock on relation 1999077 of database 16311; blocked by process 77999
I am able to run it in small batches (of 25 each) - still encountering the issue at times, but running successfully when I retry it once or twice. Smaller the batch, lesser the chances of a deadlock.
I would think this happens because all the user tables ( user_1, user_2, etc) are linked to the parent table: user. I don't want to lock the entire table for the index creation (since in theory only one table is being modified at a time). Why does this happen and is there any way around this, to ensure that the index is created without the deadlocks ?
This worked:
CREATE INDEX CONCURRENTLY IF NOT EXISTS user_1_idx_location_id
ON users.user_1 ( user_id, ( user_data->>'locationId') );
When the following transaction is run concurrently on different connections it sometimes errors with
trigger "my_trigger" for relation "my_table" already exists
What am I doing wrong?
BEGIN;
DROP TRIGGER IF EXISTS my_trigger ON my_table;
CREATE TRIGGER my_trigger
AFTER INSERT ON my_table
REFERENCING NEW TABLE AS new_table
FOR EACH STATEMENT EXECUTE PROCEDURE my_function();
COMMIT;
I am trying to set up a system where I can add triggers to notify about data changes in specific tables. If a table already has such a trigger then skip it. Otherwise CREATE all CRUD triggers. This logic needs to run sequentially in case of concurrent requests.
After trying ISOLATION LEVEL SERIALIZABLE I noticed that any conflicting transactions are failed and dropped (I would need to manually check sql status and retry). But what I want is to queue up these transactions and run afterwards one by one in the order they're sent.
At the moment I am trying to achieve this by having a my_triggers (table_name TEXT) table that has a BEFORE INSERT OR DELETE trigger. Within this trigger I do the actual table trigger upsert logic. Inserts or deletes on my_triggers are made with LOCK TABLE my_triggers IN ACCESS EXCLUSIVE MODE ... which should queue up conflicting CRUD transactions ?!
What happens is following:
BEGIN....DROP TRIGGER IF EXISTS....CREATE TRIGGER....COMMIT;
..BEGIN....DROP TRIGGER IF EXISTS....CREATE TRIGGER--------EXCEPTION.
Both transactions starts when trigger is not present.
Both succeed in drop trigger because of "IF EXISTS" statement.
First transaction starts creating a trigger. For that a SHARE ROW EXCLUSIVE lock is placed on table my_table. The lock SHARE ROW EXCLUSIVE conflicts with it self so no other transaction is allowed to create a trigger until the first one completes.
Second transaction blocks on CREATE TRIGGER.
First transaction completes.
Second transaction proceeds with CREATE TRIGGER but it already exists. Exception is raised.
What you need is adding a LOCK before DROP TRIGGER statement. This way you will ensure the trigger is dropped and not created in concurrent transaction.
BEGIN;
LOCK TABLE my_table IN SHARE ROW EXCLUSIVE MODE ;
DROP TRIGGER IF EXISTS my_trigger ON my_table;
CREATE TRIGGER my_trigger
AFTER INSERT ON my_table
REFERENCING NEW TABLE AS new_table
FOR EACH STATEMENT EXECUTE PROCEDURE my_function();
COMMIT;
ideally, Queries wait only for ExclusiveLock on a table but we saw a weird behaviour in our prod infra. select queries were waiting for accessShareLock on a view in postgres, any reason why it can be possible? Here is the output of lock monitoring query, pg_locks table of postgres
https://docs.google.com/spreadsheets/d/1xt0sfYicrDiPEdd3QdVVEm--cdHI3QGHSKb2jP2ofjI/edit#gid=1512744594
The only lock that conflicts with an AccessShareLock is an AccessExclusiveLock, which is tanken by statements like TRUNCATE, DROP TABLE or ALTER TABLE. So either one of these statements is currently running or is waiting for a transaction to finish (locks are queued).
Avoid long database transactions, and the problem will go away.
I am trying to get a better grasp on how transactions work in PostgreSQL. I did a lot of research, but I could not find any answers on the following question.
question 1
I have two transactions with isolation set to read committed, the default. I also have the following table:
create table test(a integer primary key);
Let's start the first transaction:
begin;
insert into test(a) values(1);
Now let's start the second transaction and do the same:
begin;
insert into test(a) values(1);
Now I notice that the second transaction is blocking until the first transaction either commits or rollbacks. Why is that? Why isn't it possible in the second transaction to simply continue after the insert and throw a unique-key-constraint-exception when the transaction is requested to be committed instead of throwing the exception directly after the insert call?
question 2
Now, a second scenario. Let's start from scratch with the first transaction:
begin;
insert into test(a) values(1);
delete from test where a = 1;
Now let's go to the second transaction:
begin;
insert into test(a) values(1);
Now I notice that the second transaction is also blocking. Why is it blocking on a row which does not exists anyway?
Why is it blocking on a row which does not exists anyway?
Because both transactions are inserting the same value for the primary key. The second transactions needs to wait for the outcome of the first transaction to know whether it can succeed or not. If the first transactions commits, the second will fail with a primary key violation. If the first transaction rolls back, the insert in the second transaction succeeds.
I have a question. Transaction isolation level is set to serializable. When the one user opens a transaction and INSERTs or UPDATEs data in "table1" and then another user opens a transaction and tries to INSERT data to the same table, does the second user need to wait 'til the first user commits the transaction?
Generally, no. The second transaction is inserting only, so unless there is a unique index check or other trigger that needs to take place, the data can be inserted unconditionally. In the case of a unique index (including primary key), it will block if both transactions are updating rows with the same value, e.g.:
-- Session 1 -- Session 2
CREATE TABLE t (x INT PRIMARY KEY);
BEGIN;
INSERT INTO t VALUES (1);
BEGIN;
INSERT INTO t VALUES (1); -- blocks here
COMMIT;
-- finally completes with duplicate key error
Things are less obvious in the case of updates that may affect insertions by the other transaction. I understand PostgreSQL does not yet support "true" serialisability in this case. I do not know how commonly supported it is by other SQL systems.
See http://www.postgresql.org/docs/current/interactive/mvcc.html
The second user will be blocked until the first user commits or rolls back his/her changes.