I have a Python Webapp with Flask and SQLAlchemy, and there's a system update process that occurs in multiple threads. When I run it, I'm getting a DeadlocK from Postgres.
The queries that appear in the logs are the following.
ERROR: deadlock detected
DETAIL: Process 2269053 waits for ShareLock on transaction 42979254; blocked by process 2269014.
Process 2269014 waits for ShareLock on transaction 42979253; blocked by process 2269053.
Process 2269053: UPDATE sequence SET item_list='{"item_list": [162, 164]}' WHERE sequence.id = 1978
Process 2269014: UPDATE sequence SET item_list='{"item_list": [162, 165]}' WHERE sequence.id = 1977
HINT: See server log for query details.
while updating tuple (102,44) in relation "sequence"
STATEMENT: UPDATE sequence SET item_list='{"item_list": [162, 164]}' WHERE sequence.id = 1978
I see that they are 2 different PK's, from my understanding when an update is performed only the row is locked and the statements are from 2 different rows. There's clearly something I'm misunderstanding so I wanted to ask if someone could help me clarify why this deadlocks happens and how can I solve it?
Thank you
The missing information is that a transaction can span multiple statements, and each of these can take locks. So each of the quoted UPDATE statements blocked on a lock taken by some statement in the other transaction.
For example, the previous statement of process 2269053 might have updated the row with id 1977, and the previous statement of process 2269014 might have updated the row with id 1978. There are of course numerous other possibilities.
You should figure out which part of your application issued these statements (application log file?) and look what these transactions did before. You might have to crank up application or database logging to get that information, if you cannot reconstruct it by looking at the code.
Related
Let me open by saying: yes, I am aware of Determine if a transaction is active (Postgres)
Unfortunately the sole answer to that question is far too specific to the use case provided, and doesn't actually indicate whether or not a transaction is active.
The select txid_current(); trick suggested by How to check for pending operations in a PostgreSQL transaction doesn't appear to work - I always get the same transaction ID from adjacent calls to that function. Possibly this is because I'm trying to test it from pgAdmin, which is transparently starting transactions...? (Note: I don't actually care whether there are any pending changes or active locks, so looking at pg_locks isn't helpful - what if nothing's been touched since the transaction was started?)
So: How can I determine in PostgreSQL PL/pgSQL code if a transaction is currently active?
One possible use case is: the SP/FN in question will be doing its own explicit transaction management, and calling it with a transaction already active will greatly interfere with that. I want to raise an error so that the coding mistake of calling this SP/FN in a transaction can be corrected.
There are other use cases, though.
Ideally what I'm looking for is an equivalent to MSSQL's ##TRANCOUNT (though I don't really care how deeply the transactions may be nested...)
Postgres runs PL/pgSQL inside the transaction. Thus you can't control transaction from inside PL/pgSQL. Code will look like:
begin;
select plpgsql_fn();
do '/*same any plpgsql*/';
end;
So answering your question:
If you have PL/pgSQL running ATM, you have your transaction active ATM...
Of course you can do some trick, like starting/ending work over dblink or such. but then you can check select txid_current(); over the dblink successfully...
If you want to determine if there have been any data modifications in your transaction, call txid_current_if_assigned(). It returns NULL if nothing has been modified yet.
If you only want to know if you are inside some transaction, you can save yourself the trouble, because you always are.
Before PostgreSQL v11, you cannot use transaction control statements in a function.
I haven't found a clean way to do that, but you can always call BEGIN and if it succeeds it means there is no transaction in progress (don't forget to rollback). If it fails with "there is already a transaction in progress" this means you are within transaction (better not to rollback then).
I have a python app ran on heroku that utilizes a standard postgresql heroku db ($50 version). There are 4 tables within the db. My app queries for one primary key within the main table based off input from the users of my app.
The querying worked great at first however now I'm finding it becoming too slow after about 40-50 minutes without restart my dyno. The queries will take 2,000ms after a while and take several seconds to load in front of the users. I'm newer to programming and this is my second app. I'm wondering what would make queries slower with time instead of constant. They are so fast at first. What are best practices for psycopg2 within an app to ensure the db doesn't get hung up? Here is an example of one of the queries (all others have similar syntax throughout the script):
if eventText=="Mc3 my champs":
user=event.source.user_id
profile= line_bot_api.get_profile(user)
name=str((profile.display_name))
cur=None
try:
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
# get the user's information if it exists
cur.execute("""SELECT lineid, summoner_name, champ_data FROM prestige_data WHERE lineid = %(lineid)s LIMIT 1""", {"lineid": user})
rows = cur.fetchall()
for row in rows:
champs = row[2]
prestige=(calculate_prestige(champs))
champs = json.loads(champs)
champsdict=dict.items(champs)
champs_sorted=sorted(champsdict, key=lambda student: student[1], reverse=True)
l=('\n'.join(map(str,champs_sorted)))
hello=str(l).replace('(', '').replace(')', '')
yay=str(hello).replace("'", "").replace("'", "")
msg=(yay+'\n'+"---------------------------"+'\n'+name+'\n'+"Prestige:"+(str(prestige)))
line_bot_api.reply_message(
event.reply_token,
TextSendMessage(text=msg))
break # we should only have one result, but we'll stop just in case
# The user does not exist in the database already
else:
msg = "Oops! You need to add some champs first. Try 'Mc3 inputchamp'."
line_bot_api.reply_message(
event.reply_token,
TextSendMessage(text=msg))
except BaseException:
if cur is not None:
conn.rollback()
finally:
if cur is not None:
cur.close()
While I know I did not frame that question well (only been programming for a month), I have found one issue potentially causing this that warrants documentation on here.
I had a suspicion the concurrency issue was caused when incorrect data was not found in the queries. In this situation, my conn would not rollback, commit, nor close in the above example.
Per psycopg2s documentation, even select queries need to be committed or rolled back or the transaction will stand. This in turn will keep your heroku dyno worker focused on the transaction for 30 seconds causing h12. So make sure you commit or rollback each query, regardless of the outcome, with in an application to ensure you do not get an idle transaction.
My queries are spiffy now but the issue persists. I'm not sure what slowly but surely idles my waitress workers. I think some ancillary process is somehow started in one of the class modules I've created which goes indefinitely taking hold of each worker until they are all focused on the transaction which leads to h12.
Would love someones input if they've had a similar experience. I don't want to have a cron job reboot the app every 10 minutes to make this self functioning.
I have seen another database system that offers to suspend transaction. The current transaction is kept intact but put on hold while your code is allowed to work with the database to effect immediate permanent changes to rows. Then you can resume transaction, continuing where you left off with the same locks and other transaction protections in place as if you'd never interrupted it.
For example, say an customer is placing an order, in a transaction. During that transaction, the customer notices their phone number needs to be updated, so we change that data. Next, customer decides to cancel the not-yet-completed order. A rollback of the order has the unintended consequence of also undoing the phone number change. So it would be nice if we could:
Suspend the transaction for the order.
Update the phone number, committed to the database.
Resume the transaction for the order.
Is there some way to suspend a transaction in Postgres? In JDBC?
If a transaction cannot continue, it must roll back.
If your transaction has a point at which you don't know how to carry on, then your transaction logic is flawed, you need to reorganize it - either split into multiple transactions (or sub-transactions, aka save points), or take out the parts that do not belong to the transaction logic.
Is there some way to suspend a transaction in Postgres?
No, no such thing. And the data integrity principle is unconditional as to time.
No.
The closest things are
prepared transactions: this allows (with some conditions) for a transaction to be saved, and then later rolled back or committed.
savepoints: this allows for "nested transactions", where portions of transactions can be rolled back .
Neither of these fit exactly what you are looking for. It seems that our example has two operations that do not need to be part of the same transaction at all, since the phone number update appears to be unrelated to the success of the order. (Also, a long-running transaction is a bad idea....your order should probably be a state machine implemented without long-running transaction.)
Workaround – open second connection
In JDBC, you could just open a second connection to the database.
Do your separate work on that second connection and close. The first connection is still open and remains in its same state. Any active transaction in that first connection remains.
In PostgreSQL, are DEFERRED triggers executed before (within) the completion of the transaction or just after it?
The documentation says:
DEFERRABLE
NOT DEFERRABLE
This controls whether the constraint can be deferred. A constraint
that is not deferrable will be checked immediately after every
command. Checking of constraints that are deferrable can be postponed
until the end of the transaction (using the SET CONSTRAINTS command).
It doesn't specify if it is still inside the transaction or out. My personal experience says that it is inside the transaction and I need it to be outside!
Are DEFERRED (or INITIALLY DEFERRED) triggers executed inside of the transaction? And if they are, how can I postpone their execution to the time when the transaction is completed?
To give you a hint what I'm after, I'm using pg_notify and RabbitMQ (PostgreSQL LISTEN Exchange) to send out messages. I process such messages in an external application. Right now I have a trigger which notifies the external app of the newly inserted records by including the record's id in the message. But in a non-deterministic way, once in a while, when I try to select a record by its id at hand, the record can not be found. That's because the transaction is not complete yet and the record is not actually added to the table. If I can only postpone the execution of the trigger for after the completion of the transaction, everything will work out.
In order to get better answers let me explain the situation even closer to the real world. The actual scenario is a little more complicated than what I explained before. The source code can be found here if anyone's interested. Becuase of reasons that I'm not gonna dig into, I have to send the notification from another database so the notification is actually sent like:
PERFORM * FROM dblink('hq','SELECT pg_notify(''' || channel || ''', ''' || payload || ''')');
Which I'm sure makes the whole situation much more complicated.
Triggers (including all sorts of deferred triggers) fire inside the transaction.
But that is not the problem here, because notifications are delivered between transactions anyway.
The manual on NOTIFY:
NOTIFY interacts with SQL transactions in some important ways.
Firstly, if a NOTIFY is executed inside a transaction, the notify
events are not delivered until and unless the transaction is
committed. This is appropriate, since if the transaction is aborted,
all the commands within it have had no effect, including NOTIFY. But
it can be disconcerting if one is expecting the notification events to
be delivered immediately. Secondly, if a listening session receives a
notification signal while it is within a transaction, the notification
event will not be delivered to its connected client until just after
the transaction is completed (either committed or aborted). Again, the
reasoning is that if a notification were delivered within a
transaction that was later aborted, one would want the notification to
be undone somehow — but the server cannot "take back" a notification
once it has sent it to the client. So notification events are only
delivered between transactions. The upshot of this is that
applications using NOTIFY for real-time signaling should try to keep
their transactions short.
Bold emphasis mine.
pg_notify() is just a convenient wrapper function for the SQL NOTIFY command.
If some rows cannot be found after a notification has been received, there must be a different cause! Go find it. Likely candidates:
Concurrent transactions interfering
Triggers doing something more or different than you think they do.
All sorts of programming errors.
Either way, like the manual suggests, keep transactions that send notifications short.
dblink
Update: Transaction control in a PROCEDURE or DO statement in Postgres 11 or later makes this a lot simpler. Just COMMIT; to (also) send waiting notifications.
Original answer (mostly for Postgres 10 or older):
PERFORM * FROM dblink('hq','SELECT pg_notify(''' || channel || ''', ''' || payload || ''')');
... which should be rewritten with format() to simplify and make the syntax secure:
PRERFORM dblink('hq', format('NOTIFY %I, %L', channel, payload));
dblink is a game-changer here, because it opens a separate transaction in the other database. This is sometimes used to fake autonomous transaction.
Does Postgres support nested or autonomous transactions?
How do I do large non-blocking updates in PostgreSQL?
dblink() waits for the remote command to finish. So the remote transaction will most probably commit first. The manual:
The function returns the row(s) produced by the query.
If you can send notification from the same transaction instead, that would be a clean solution.
Workaround for dblink
If notifications have to be sent from a different transaction, there is a workaround with dblink_send_query():
dblink_send_query sends a query to be executed asynchronously, that is, without immediately waiting for the result.
DO -- or plpgsql function
$$
BEGIN
-- do stuff
PERFORM dblink_connect ('hq', 'your_connstr_or_foreign_server_here');
PERFORM dblink_send_query('con1', format('SELECT pg_sleep(3); NOTIFY %I, %L ', 'Channel', 'payload'));
PERFORM dblink_disconnect('con1');
END
$$;
If you do this right before the end of the transaction, your local transaction gets 3 seconds (pg_sleep(3)) head start to commit. Chose an appropriate number of seconds.
There is an inherent uncertainty to this approach, since you get no error message if anything goes wrong. For a secure solution you need a different design. After successfully sending the command, chances for it to still fail are extremely slim, though. The chance that successful notifications are missed seem much higher, but that's built into your current solution already.
Safe alternative
A safer alternative would be to write to a queue table and poll it like discussed in #Bohemian's answer. This related answer demonstrates how to poll safely:
Postgres UPDATE … LIMIT 1
I'm posting this as an answer, assuming the actual problem you are trying to solve is deferring execution of an external process until after the transaction is completed (rather than the X-Y "problem" you're trying to solve using trigger Kung Fu).
Having the database tell an app to do something is a broken pattern. It's broken because:
There's no fallback if the app doesn't get the message, eg because it's down, network explodes, whatever. Even the app replying with an acknowledgment (which it can't), wouldn't fix this problem (see next point)
There's no sensible way to retry the work if the app gets the message but fails to complete it (for any of lots of reasons)
In contrast, using the database as a persistant queue, and having the app poll it for work, and take the work off the queue when work is complete, has none of the above problems.
There are lots of ways to achieve this. The one I prefer is to have some process (usually trigger on insert, update and delete) put data into a "queue" table. Have another process poll that table for work to do, and delete from the table when work is complete.
It also adds some other benefits:
The production and consumption of work is decoupled, which means you can safely kill and restart your app (which must happen from time to time, eg deploying) - the queue table will happily grow while the app is down, and will drain when the app is back up. You can even replace the app with an entirely new one
If for whatever reason you want to initiate processing of certain items, you can just manually insert rows into the queue table. I used this technique myself to initiate the processing of all items in a database that needed initialising by being put on the queue once. Importantly, I didn't need to do a perfunctory update to every row just to fire the trigger
Getting to your question, a slight delay can be introduced by adding a timestamp column to the queue table and having the poll query only select rows that are older than (say) 1 second, which gives the database time to complete its transaction
You can't overload the app. The app will read only as much work as it can handle. If your queue is growing, you need a faster app, or more apps If multiple consumers are operating, concurrency can be solved by (for example) adding a "token" column to the queue table
Queues that are backed by database tables is the basis of how persistent queues are implemented in commercial grade queue-based platforms, so the pattern is well tested, used and understood.
Leave the database to do what it does best, and the only thing it does well: Manage data. Don't try to make your database server into an app server.
I am using the Play! framework, and have a difficulty with in the following scenario.
I have a server process which has a 'read-only' transaction. This to prevent any possible database lock due to execution as it is a complicated procedure. There are one or two record to be stored, but I do that as a job, as I found doing them in the main thread could result in a deadlock under higher load.
However, in one occasion I need to create an object and subsequently use it.
However, when I create the object using a Job, wait for the resulting id (with a Promise return) and then search in the database for it, it cannot be found.
Is there an easy way to have the JPA search 'afresh' in the DB at this point? I implemented a 5 sec. pause to test, so I am sue it is not because the procedure hadn't finished yet.
Check if there is a transaction wrapped around your INSERT and if there is one check that the transaction is COMMITed.