I recently inherited an application that uses PostgreSQL and I have been troubleshooting issues relating to saving records in the database.
I know that PostgreSQL allows me to record the transaction ID in the log by including the special value of %x in the log_line_prefix. What I noticed, however, is that the first statement that occurs within a transaction always gets logged with a zero.
If I perform the following operations in psql,
begin;
insert into numbers (1);
insert into numbers (2);
commit;
the query log will contain the following entries:
2016-09-20 03:07:40 UTC 0 LOG: statement: begin;
2016-09-20 03:07:53 UTC 0 LOG: statement: insert into numbers values (1);
2016-09-20 03:07:58 UTC 689 LOG: statement: insert into numbers values (2);
2016-09-20 03:08:03 UTC 689 LOG: statement: commit;
My log format is %t %x and as you can see, the transaction ID for the first insert statement is 0, but it changes to 689 when I execute the second insert.
Can anyone explain why after starting a transaction PostgreSQL doesn't log the right transaction ID on the first statement? Or if I'm just doing this wrong, is there a more reliable way of identifying which queries were part of a single transaction by looking at the log file?
The transaction ID is assigned after the statement starts, so log_statement doesn't capture it. BEGIN doesn't assign a transaction ID, it's delayed until the first write operation.
Use the virtual txid instead, which is assigned immediately. The placeholder is %v. These are assigned immediately, but are not persistent and are backend-local.
I find it useful to log both. The txid because it matches up to xmin and xmax system column contents, etc; the vtxid to help me group up operations done in transactions.
Related
I'm trying to rename a database using:
ALTER DATABASE xxx RENAME TO yyy
I got an error message saying there is another open session. I killed it using:
SELECT pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = 'xxx' AND pid <> pg_backend_pid();
However, I then got an error message saying there are 2 prepared transactions pending. I made sure to kill all processes in the background with:
SELECT pg_cancel_backend(pid) FROM pg_stat_activity WHERE pid <> pg_backend_pid();
But still, I am not able to rename the database.
I then saw there is a view by postgres called pg_prepared_xacts that shows me the prepared transactions (shortened the gid for a better overview), and indeed, there are two of them:
transaction|gid |prepared |owner|database|
-----------+--------------+-----------------------------+-----+--------+
5697779 |4871251_EAAAAC|2022-08-05 15:50:59.782 +0200|xxx |xxx |
5487701 |4871251_DAAAAA|2022-07-08 08:05:36.066 +0200|xxx |xxx |
According to the Postgres documentation on prepared transactions, I can execute a Rollback on the transaction id - which is what I did:
ROLLBACK PREPARED '5697779';
I made sure to execute the ROLLBACK with the same user, but it shows an error message saying that the transaction does not exist...
How can I get rid of it / kill it in order to be able to rename the database?
Thank you for reading and taking time to respond.
From here Prepared transaction:
transaction_id
An arbitrary identifier that later identifies this transaction for COMMIT PREPARED or ROLLBACK PREPARED. The identifier must be written as a string literal, and must be less than 200 bytes long. It must not be the same as the identifier used for any currently prepared transaction.
and from here Rollback prepared:
transaction_id
The transaction identifier of the transaction that is to be rolled back.
Lastly from here pg_prepared_xacts:
gid text
Global transaction identifier that was assigned to the transaction
So to rollback the transaction you need the global identifier assigned in the PREPARE TRANSACTION as shown in the gid column in pg_prepared_xacts.
I have multiple Insert Statementes:
INSERT INTO table1(item_name) VALUES ('item1');
INSERT INTO table1(item_name) VALUES ('item2');
INSERT INTO table1(item_name) VALUES ('item3');
the first value already exists in the table. So i get an error: ERROR: duplicate key value violates unique constraint when the first statement is executed and the execution of the other statements stops.
How can i force Postgres to execute the other statements despite the error? I know that INSERT ON CONFLICT in this case would be a solution, my question is more general: How to continue with the following statements if an error occurs.
If you send several SQL statements, separated by semicolon, to the PostgreSQL server in a single query, they are executed in a single transaction. So if one of them fails, all are undone and processing terminates.
You have to execute each statement separately for that.
If you use psql CLI instead of pgadmin, you can use ON_ERROR_ROLLBACK setting.
When set to on, if a statement in a transaction block generates an
error, the error is ignored and the transaction continues. When set to
interactive, such errors are only ignored in interactive sessions, and
not when reading script files. When set to off (the default), a
statement in a transaction block that generates an error aborts the
entire transaction. The error rollback mode works by issuing an
implicit SAVEPOINT for you, just before each command that is in a
transaction block, and then rolling back to the savepoint if the
command fails.
Example:
select * from t;
x
---
1
(1 row)
\set ON_ERROR_ROLLBACK on
begin;
BEGIN
insert into t values(1);
ERROR: duplicate key value violates unique constraint "t_pkey"
DETAIL: Key (x)=(1) already exists.
insert into t values(2);
INSERT 0 1
insert into t values(3);
INSERT 0 1
commit;
COMMIT
select * from t;
x
---
1
2
3
(3 rows)
I'm using Npgsql 3.2 and it seems that the driver is now preparing statements automatically.
I see the following in the PostgreSQL logs:
LOG: execute <unnamed>:SELECT * FROM database.update_item($1, $2, $3 , $4)
I may be wrong but this is not activated on the connection string and I see no previous 'prepare' call in the logs.
What am I missing ?
Also I'm using Dapper 1.50.2 on top of Npgsql to query the database.
This is not yet implemented at this level but I see that there are some talks about such a feature on GitHub.
I'm using a READ COMMITTED transaction to update a row in the database. The row is updated twice with 2 disctinct statements.
When playing the statements one by one in a pgadmin query window it works fine.
When the statements are played by the driver through execute, the first statement seems to put locks on the record, and the second statement hang.
Here is the queries ran in the query window (runs to completion) :
BEGIN;
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
SELECT * FROM database.update_item(params...);
SELECT * from database.update_item(params...);
ROLLBACK;
The corresponding PG logs:
LOG: statement: BEGIN;
LOG: statement: SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
LOG: statement: SELECT * FROM database.update_item(params...);
LOG: statement: SELECT * from database.update_item(params...);
LOG: statement: ROLLBACK;
The PostgreSQL logs when played by Npgsql:
LOG: statement: BEGIN
LOG: statement: SET TRANSACTION ISOLATION LEVEL READ COMMITTED
LOG: execute <unnamed>: select * from database.update_item(params...)
LOG: execute <unnamed>: select * from database.update_item(params...) <-- hangs
Any help appreciated.
EDIT 1:
Additional information: The update_item() plpgsql function is updating the row with an EXECUTE statement.
EDIT 2:
Added PG logs for query window.
First, Npgsql's automatic preparation feature is opt-in - it's not activated without the connection string. Even then, the same SQL needs to be executed several times (5 by default) before Npgsql prepares it. See the documentation for more details.
Regarding your deadlock, are you running your code concurrently? In other words, do you have multiple transactions at the same time, updating the same rows? If so, then this is probably the expected PostgreSQL behavior and has nothing to do with Npgsql. When you update rows in a transaction, these rows are locked until the transaction is committed. The fix in general is to update rows in the same order. See the PostgreSQL documentation for more details.
In postgres log:
2016-12-23 15:28:14 +07 [17281-351 trns: 4280939, vtrns: 3/20] postgres#deadlocks HINT: See server log for query details.
2016-12-23 15:28:14 +07 [17281-352 trns: 4280939, vtrns: 3/20] postgres#deadlocks CONTEXT: while locking tuple (0,79) in relation "account"
2016-12-23 15:28:14 +07 [17281-353 trns: 4280939, vtrns: 3/20] postgres#deadlocks STATEMENT: SELECT id FROM account where id=$1 for update;
when I provoke a deadlock I can see text: tuple (0,79).
As I know, a tuple just is several rows in table. But I don't understand what (0,79) means. I have only 2 rows in table account, it's just play and self-learning application.
So what does (0,79) means?
This is the data type of the system column ctid. A tuple ID is a pair
(block number, tuple index within block) that identifies the physical
location of the row within its table.
read https://www.postgresql.org/docs/current/static/datatype-oid.html
It means block number 0, row index 79
also read http://rachbelaid.com/introduction-to-postgres-physical-storage/
also run SELECT id,ctid FROM account where id=$1 with right $1 to check out...
I have a stored procedure that retrieves sensitive information from an SQL Server 2008 database. I would like to modify the procedure so that any time it is called, it records information about who called it in a separate table.
I thought something like the following would work:
declare #account varchar(255);
set #account = (SELECT SYSTEM_USER);
INSERT into AUDIT_LOG(ACCOUNT, TSTAMP)
VALUES(#account, getdate())
;
--Now fetch data
SELECT x,y,z from sensitive_info;
My issue is that the client application can issue a call to this stored procedure and get the sensitive information, but not commit the connection and the INSERT never occurs!
Is there some way to force the INSERT to happen before the SELECT?
I am using SQL Server 2008.
Thanks,
Carl
You only COMMIT if a transaction has been started.
So you can test for an open transaction first and disallow the read. This will ensure that no transaction is open to be rolled back. I've used XACT_STATE() here
Using SET XACT_ABORT ON and TRY/CATCH too will mean that the INSERT for logging must happen too before the read happens. Any errors at all on INSERT will go to the CATCH block. So no read and the logging fail can itself be logged too.
So: this is your guarantee of "read only if logged"
Having an explicit transaction doesn't help: the INSERT is an atomic action anyway. And if the called opens a transaction the log entry can be rolled back
CREATE PROC getSecretStuff
AS
SET NOCOUNT, XACT_ABORT ON;
BEGIN TRY
IF XACT_STATE() <> 0
RAISERRROR ('Call not allowed in an active transaction', 16, 1)
INSERT into AUDIT_LOG(ACCOUNT, TSTAMP)
VALUES(SYSTEM_USER, getdate());
--Now fetch data
SELECT x,y,z from sensitive_info;
END TRY
BEGIN CATCH
-- error handling etc
END CATCH
GO
Why not use the build in auditing functionality?
Have you tried using expicit transactions and doing the select after the commit statement?
On you insert a record in a table you should be albe to get the SCOPE_IDENTITY() of the ast inserted value. Before doing SELECT x,y,z from sensitive_info; you can check if SCOPE_IDENTITY() > 0 then only execute SELECT statement.