I want to ask you for help with my problem. I have a program that triggers asynchronous computations in parallel and waits in the loop until they are finished.
I am using Postgres as a database where I have created the table computation_status that contains the following data when computations is triggered:
computation
finished
COMPUTATION_A
null
COMPUTATION_B
null
COMPUTATION_C
null
Then I am waiting in the loop until all computations are finished. This loop acceps notifications for each computation that is finished and triggers SQL transactions to update its status and check if there is any other computation running. For example:
T1:
BEGIN_TRANSACTION
update computation_status set finished = NOW() where and computation = 'COMPUTATION_A'
select exists (select 1 from computation_status where finished is null)
COMMIT
T2:
BEGIN_TRANSACTION
update computation_status set finished = NOW() where and computation = 'COMPUTATION_B'
select exists (select 1 from computation_status where finished is null)
COMMIT
T3:
BEGIN_TRANSACTION
update computation_status set finished = NOW() where and computation = 'COMPUTATION_C'
select exists (select 1 from computation_status where finished is null)
COMMIT
And when the last computation is finished the program exits the waiting loop.
What level of isolation should I use to avoid these problems? I know I should at least use the READ_COMMITED isolation level to prevent non-repeatable reads, but is that enough? Or is it also possible that phantom reads will occur and I should use REPETABLE_READ? (I'm not sure if an UPDATE is counted as a READ too).
I want to avoid the problem that for example computations A and B will be finished at the same time as the last ones. Then T1 will set A=finished and read that B is not finished and T2 will set B=finished and read that A is not fished and this will cause a problem in my application because it will end up in an infinite loop.
To avoid race conditions here, you have to effectively serialize the transactions.
The only isolation level where that would work reliably would be SERIALIZABLE. However, that incurs a performance penalty, and you have to be ready to repeat transactions in case a serialization error is thrown. If more than one of these transactions are running concurrently, a serialization error will be thrown.
The alternative would be to use locks, but that is not very appealing: using row locks would lead to deadlocks, and using table locks would block autovacuum, which would eventually bring your system down.
Related
I am implementing a lightweight jobs system in my highly-concurrent application. For simplicity, I will be using Postgres to manage the state of all jobs in the system.
I have a table where processes can mark specific jobs as "running" via a boolean flag is_running. If multiple processes attempt to run the same job, only one of the processes should "win".
I want to have an atomic statement that the application processes will call in an attempt to exclusively mark the job as running.
UPDATE jobs SET is_running = true WHERE is_running = false AND id = 1
If there a is a single row with (id=1, is_running=false), and multiple processes attempt to execute the above statement, is it possible for more than one process to actually set is_running=true?
I only want one of the processes to see an updated row count of 1 - all other processes should see an updated row count of 0.
Your approach is safe and free from race conditions, because PostgreSQL will reevaluate the WHERE condition after it had to wait for a lock caused by a concurrent modification. It will then see the changed value and skip the row.
I have a python script that executes multiple sql scripts (one after another) in Redshift. Some of the tables in these sql scripts can be queried multiple times. For ex. Table t1 can be SELECTed in one script and can be dropped/recreated in another script. This whole process is running in one transaction. Now, sometimes, I am getting deadlock detected error and the whole transaction is rolled back. If there is a deadlock on a table, I would like to wait for the table to be released and then retry the sql execution. For other types of errors, I would like to rollback the transaction. From the documentation, it looks like the table lock isn't released until end of transaction. I would like to achieve all or no data changes (which is accomplished by using transaction) but also would like to handle deadlocks. Any suggestion on how this can be accomplished?
I would execute all of the SQL you are referring to in one transaction with a retry loop. Below is the logic I use to handle concurrency issues and retry (pseudocode for brevity). I do not have the system wait indefinitely for the lock to be released. Instead I handle it in the application by retrying over time.
begin transaction
while not successful and count < 5
try
execute sql
commit
except
if error code is '40P01' or '55P03'
# Deadlock or lock not available
sleep a random time (200 ms to 1 sec) * number of retries
else if error code is '40001' or '25P02'
# "In failed sql transaction" or serialized transaction failure
rollback
sleep a random time (200 ms to 1 sec) * number of retries
begin transaction
else if error message is 'There is no active transaction'
sleep a random time (200 ms to 1 sec) * number of retries
begin transaction
increment count
The key components are catching every type of error, knowing which cases require a rollback, and having an exponential backoff for retries.
I've been running into occasional deadlocks in my application because two transactions which need to update the same rows but in different orders (ex, transaction A updates rows X then Y, while transaction B updates rows Y then X).
For various Reasons, the traditional approaches to resolving avoiding this kind of deadlock – locking, or updating rows in a consistent order – are less than ideal.
Since the updates I'm trying to perform are otherwise idempotent and order-independent, is it safe and reasonable to simply catch these occasional deadlocks at the application level and retry the transaction?
For example:
def process_update(update):
attempt = 0
while attempt < 10:
try:
execute("SAVEPOINT foo")
for row in update:
execute("UPDATE mytable SET … WHERE …", row)
execute("RELEASE SAVEPOINT foo")
break
except Deadlock:
execute("ROLLBACK TO SAVEPOINT foo")
attempt += 1
raise Exception("Too many retries")
Is this a reasonable idea? Or are there costs associated with Postgres' deadlock detection that might make it dangerous?
I did a lot of research and experimentation into this for a system that's running 50 to 100 concurrent processes on the same tables. There are a number of transaction failures that can happen besides basic deadlocks. My case includes both read committed and serializable transactions. There's no situation where handling this at the application level caused any issues. Fortunately Postgres will fail immediately, so the only performance hit is to the application, nothing significant to the database.
The key components are catching every type of error, knowing which cases require a rollback, and having an exponential backoff for retries. I found that immediate retries or static sleep times cause processes to simply deadlock each other repeatedly and cause a bit of a domino effect, which makes sense.
This is the complete logic my system requires to handle every concurrency issue (pseudocode):
begin transaction (either read committed or serializable)
while not successful and count < 5
try
execute sql
commit
except
if error code is '40P01' or '55P03'
# Deadlock or lock not available
sleep a random time (200 ms to 1 sec) * number of retries
else if error code is '40001' or '25P02'
# "In failed sql transaction" or serialized transaction failure
rollback
sleep a random time (200 ms to 1 sec) * number of retries
begin transaction
else if error message is 'There is no active transaction'
sleep a random time (200 ms to 1 sec) * number of retries
begin transaction
increment count
this is a problem I faced in my operating system's exam. I could not figure out the right answer for it. Can someone help.Given is a code for synchronization where many threads are trying to access a global counter g using lock-
if(lock==1)
wait(); //sleep this thread until some other thread wakes up this thread
else
lock=1; //enter in protected area
//access global counter g//
lock=0;
//wake up some other thread which is waiting for the lock to be released
What is the problem in above synchronization? Choose anyone of the options given below
The synchronization is fine and will run correctly.
Will only run on uni-processor systems but not on multiprocessor systems.
Will not run on any system
Can’t say. Need more data
The answer is 3. This code fails both at safety and liveness as long as threads can be preempted. For safety, consider the following interleaving of operations with two threads t1 and t2:
t1 checks lock, skips to the else statement
OS preempts t1 and schedules t2
t2 checks lock, skips to the else statement
And we have two threads in the critical section. This is why you need some sort of atomic test-and-set operation, or the ability to disable preemption, to do it properly.
For liveness, consider the following interleaving of operations with two threads t1 and t2:
t1 checks lock, skips to the else statement
t1 sets lock to 1
OS preempts t1 and schedules t2
t2 checks lock, finds 1
OS preemtps t2 and schedules t1
t1 sets lock to 0
t1 finds no thread waiting and does nothing else
OS schedules t2 again
t2 starts waiting...
And thus t2 is (potentially) waiting forever. The solution is for the synchronization primitive to keep track of wake-ups (e.g., a semaphore) or require that testing the condition and waiting is done atomically (e.g., mutexes and condition variables).
I'm using Postgres 9.1. I'm wondering if using multiple SELECT FOR UPDATES in the same transaction could potentially cause a race condition.
2 concurrent transactions:
transaction 1: select for update on table 1 -- successfully acquires lock
transaction 2: select for update on table 2 -- successfully acquires lock
transaction 2: select for update on table 1 -- waiting for lock release from transaction 1
transaction 1: select for update on table 2 -- waiting for lock release from transaction 2
What happens in this situation? Does one of the waiting transactions eventually time out? If so, is there a way to configure the timeout duration?
edit: is deadlock_timeout the configuration I am looking for?
Yes, you should look for the deadlock_timeout in the docs.
But your scenario doesn't means that there will be a deadlock, 'cos PostgreSQL is using row-level locks and it is not clear whether your transactions are concurring for the same rows.
Another option is to use serialization level higher then default READ COMMITTED. But in this case your application should be ready to receive exceptions with SQLCODE=40001:
ERROR: could not serialize access due to concurrent update
This is expected, you should just re-try transaction as is.
A very good overview of Serializable isolation level you can find on the wiki.
PostgreSQL will detect the deadlock on step 4 and will fail the transaction. Here's what happened when I tried it in psql (only showing step 4):
template1=# SELECT * FROM table2 FOR UPDATE;
ERROR: deadlock detected
DETAIL: Process 17536 waits for ShareLock on transaction 166946; blocked by process 18880.
Process 18880 waits for ShareLock on transaction 166944; blocked by process 17536.
HINT: See server log for query details.
template1=#
This happens after 1s, which is the default timeout. The other answer has more information about this.