Postgresql deadlock issue - postgresql

I have received the following deadlock error in pg_log:
2016-01-15 09:52:48.648 EST,"name","name",11694,"ip:40273",56988e35.2dae,1,"UPDATE",2016-01-15 01:14:13 EST,10/3886,49775,ERROR,40P01,
"deadlock detected",
"Process 11694 waits for ShareLock on transaction 49774; blocked by process 11685.
Process 11685 waits for ShareLock on transaction 49775; blocked by process 11694.
Process 11694: update bb_batter_season_stat set a_field=a_field+$1, ab_int=ab_int+$2, abvsl=abvsl+$3, abvsr=abvsr+$4, bbvsl=bbvsl+$5, bbvsr=bbvsr+$6, cs=cs+$7,
cs_field=cs_field+$8, ... where bb_players_id=$53
Process 11685: update bb_batter_season_stat set a_field=a_field+$1, ab_int=ab_int+$2, abvsl=abvsl+$3, abvsr=abvsr+$4, bbvsl=bbvsl+$5, bbvsr=bbvsr+$6, cs=cs+$7,
cs_field=cs_field+$8, ... where bb_players_id=$53","See server log for query details.",,,,
"update bb_batter_season_stat set a_field=a_field+$1, ab_int=ab_int+$2, abvsl=abvsl+$3,
abvsr=abvsr+$4, bbvsl=bbvsl+$5, bbvsr=bbvsr+$6, cs=cs+$7, cs_field=cs_field+$8, ... where bb_players_id=$53",,,""
I can't understand why is it happening. Two processes running same query and deadlock happens.
The table schema is:
CREATE TABLE bb_batter_season_stat (
id SERIAL NOT NULL ,
bb_players_id INTEGER NOT NULL ,
G SMALLINT ,
ABvsL SMALLINT ,
ABvsR SMALLINT ,
RvsL SMALLINT ,
RvsR SMALLINT ,
HvsL SMALLINT ,
HvsR SMALLINT ,
d2BvsL SMALLINT ,
d2BvsR SMALLINT ,
...
PRIMARY KEY(id) ,
FOREIGN KEY(bb_players_id) REFERENCES bb_players(id) );
CREATE INDEX bb_batter_season_stat_FKIndex1 ON bb_batter_season_stat (bb_players_id);

I suppose that you have at least 2 records in bb_batter_season_stat that have the same bb_players_id.
Transaction one locked for update the first record.
Transaction two locked the second.
Transaction one tried to lock the the second, but was blocked waiting for transaction two.
Transaction two tried to lock the first record, but was blocked waiting for transaction one, creating a deadlock.
To avoid this you should force locking of the records in the order of primary key. For example using select for update:
with ids as (
select id
from bb_batter_season_stat
where bb_players_id=?
order by id
for update
)
update bb_batter_season_stat
set a_field=a_field+$1, ab_int=ab_int+$2, …
where id in (select id from ids);

Related

pglogical logical plugin missing DML records

Created pglogical plugin replication slot.
SELECT 'init' FROM pg_create_logical_replication_slot('demo_slot', 'pglogical_output');
Created table which have primary key.
create table pglogical_test(id serial primary key,name text);
Inserted few rows into table
insert into pglogical_test(name) values('hi');
Used below query to check the replication slot data
SELECT *
FROM pg_logical_slot_peek_changes('demo_slot', NULL, NULL,
'min_proto_version', '1', 'max_proto_version', '1',
'startup_params_format', '1', 'proto_format', 'json');
Slot is missing actions of I(insert),U(update) & D(delete). Only B(begin) & C(commit) is available.Sample slot data is as below:
{"action":"S", "params": {"max_proto_version":"1","min_proto_version":"1","coltypes":"f","pg_version_num":"120009","pg_version":"12.9 (Debian 12.9-1.pgdg110+1)","pg_catversion":"201909212","database_encoding":"UTF8","encoding":"SQL_ASCII","forward_changeset_origins":"t","walsender_pid":"884","pglogical_version":"2.4.1","pglogical_version_num":"20401","binary.internal_basetypes":"f","binary.binary_basetypes":"f","binary.basetypes_major_version":"1200","binary.sizeof_int":"4","binary.sizeof_long":"8","binary.sizeof_datum":"8","binary.maxalign":"8","binary.bigendian":"f","binary.float4_byval":"t","binary.float8_byval":"t","binary.integer_datetimes":"f","binary.binary_pg_version":"1200","no_txinfo":"f"}}
{"action":"B", "has_catalog_changes":"t", "xid":"529", "first_lsn":"0/189E960", "commit_time":"2023-02-01 09:14:20.965952+00"}
{"action":"C", "final_lsn":"0/18C2190", "end_lsn":"0/18C28E8"}
{"action":"B", "has_catalog_changes":"f", "xid":"530", "first_lsn":"0/18C28E8", "commit_time":"2023-02-01 09:14:29.654792+00"}
{"action":"C", "final_lsn":"0/18C2A30", "end_lsn":"0/18C2A60"}
Followed documentation from https://github.com/2ndQuadrant/pglogical/blob/REL2_x_STABLE/internals-doc/OUTPUT.md
Why action I ,U & D are missing from pg_logical_slot_peek_changes ?

DB2 measure execution time of triggers

How can I best measure the execution time of triggers in DB2 for insert or update.
It is needed for some performance issues, some of them are behaving very slow.
CREATE OR REPLACE TRIGGER CHECK
NO CASCADE BEFORE INSERT ON DAG
REFERENCING NEW AS OBJ
FOR EACH ROW MODE DB2SQL
WHEN(
xyz)
)
SIGNAL SQLSTATE xxx)
For compiled triggers (that is, with BEGIN ... END body):
SELECT
T.TRIGNAME
, M.SECTION_NUMBER
, M.STMT_EXEC_TIME
, M.NUM_EXEC_WITH_METRICS
-- Other M metrics
, M.*
, M.STMT_TEXT
FROM SYSCAT.TRIGGERS T
JOIN SYSCAT.TRIGDEP D
ON (D.TRIGSCHEMA, D.TRIGNAME) = (T.TRIGSCHEMA, T.TRIGNAME)
JOIN TABLE (MON_GET_PKG_CACHE_STMT (NULL, NULL, NULL, -2)) M
ON (M.PACKAGE_SCHEMA, M.PACKAGE_NAME) = (D.BSCHEMA, D.BNAME)
WHERE D.BTYPE = 'K'
-- Or use whatever filter on your triggers
AND (T.TABSCHEMA, T.TABNAME) = ('MYSCHEMA', 'MYTABLE')
ORDER BY 1, 2
For inlined triggers (that is, with BEGIN ATOMIC ... END body):
No way to get separate statistics for them. They are compiled and optimized with the corresponding statement fired them.

How to use transaction with PQexecPrepared libpq

I'm new to postgresql and would like to ask how to do a transaction using BEGIN, COMMIT and PQexecPrepared. In my program, I have to update many tables before COMMIT. I do understand that we need to use:
1. PQexec(conn,"BEGIN");
2. execute some queries
3. PQexec(conn,"COMMIT");
I had first tried by using PQexecParams, it worked:
PQexec(conn,"BEGIN");
PQexecParams(conn, "INSERT INTO Cars (Id,Name, Price) VALUES ($1,$2,$3)",
3, NULL, parValues, NULL , NULL, 0 );
PQexec(conn,"COMMIT");
However when I had tried by using PQexecPrepared, my table Cars wasn't updated after COMMIT (of course it worked in autocommit mode without BEGIN &COMMIT )
PQexec(conn,"BEGIN");
PQprepare(conn,"teststmt", "INSERT INTO Cars (Id,Name, Price) VALUES ($1,$2,$3)", 3, NULL );
PQexecPrepared(conn, "teststmt", 3, parValues,NULL, NULL,0);
PQexec(conn,"COMMIT");
Do you have any advice in this case?

T-Sql update and avoid conflict

I'm trying to migrate a Tomcat app from using Postgres 9.5 to SQL Server 2016 and I've got a problem statement I can't seem to duplicate.
It's basically an upsert but one of the complications is the request supplies arguments to do the update, but when there is conflict I need to use some of the existing values from conflicting rows to insert/update.
The primary keys in the table can sometimes cause a conflict, which requires updating rows and deleting the old ones.
The table schema in MS SQL looks like:
CREATE TABLE [dbo].[signup](
[site_key] [varchar](32) NOT NULL,
[list_id] [bigint] NOT NULL,
[email_address] [varchar](256) NOT NULL,
[customer_id] [bigint] NULL,
[attribute1] [varchar](64) NULL,
[date1] [datetime] NOT NULL,
[date2] [datetime] NULL,
CONSTRAINT [pk_signup] PRIMARY KEY CLUSTERED
(
[site_key] ASC,
[list_id] ASC,
[email_address] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
The old Postgres SQL looked like this:
WITH updated_rows AS (
INSERT INTO signup
(site_key, list_id, email_address, customer_id, attribute1, date1, date2)
SELECT site_key, list_id, :emailAddress, customer_id, attribute1, date1, date2
FROM signup WHERE customer_id = :customerId and email_address <> :emailAddress
ON CONFLICT (site_key, list_id, email_address) DO UPDATE SET customer_id = excluded.customer_id
RETURNING site_key, customer_id, email_address, list_id
)
DELETE FROM signup AS signup_delete USING updated_rows
WHERE
signup_delete.site_key = updated_rows.site_key
AND signup_delete.customer_id = updated_rows.customer_id
AND signup_delete.list_id = updated_rows.list_id
AND signup_delete.email_address <> :emailAddress;
Two arguments are supplied, customer id and email address, shown here as Spring NamedParameterJdbcTemplate values :customerId and :emailAddress
It's trying to change the email address of the customer id to be the supplied one, but sometimes the supplied email address already exists in the primary key constraint.
In which case it needs to change the existing customer id to be supplied one, and remove the rows with that don't match the new email address.
I also need to try and maintain isolation so that nothing can change the data whilst I'm updating.
I'm trying to do it with a MERGE statement but I can't seem to get it to work, it's complaining I cant use values that aren't in the clause scope, but I think I've probably got other issues here too.
This is what I had so far. It doesn't even address the deleting part - only the upserting, but I can't even get this part to work. I was planning to use the OUTPUT from this as input to something to delete the rows similar to the postgres version.
WITH source AS (
SELECT cs.[site_key] as existing_site_key,
cs.list_id as existing_list_id,
cs.email_address as existing_email,
cs.customer_id as existing_customer_id,
cs.attribute1 as existing_attribute1,
cs.date1 as existing_date1,
cs.date2 as existing_date2,
cs2.email_address as conflicting_email,
cs2.customer_id AS conflicting_customer_id
FROM [dbo].[signup] cs
LEFT JOIN [dbo].[signup] cs2 ON cs2.email_address = :emailAddress
AND cs.site_key = cs2.site_key
AND cs.list_id = cs2.list_id
WHERE cs.customer_id = :customerId
)
MERGE signup WITH (HOLDLOCK) AS target
USING source
ON ( source.conflicting_customer_id is not null )
WHEN MATCHED AND source.existing_site_key = target.site_key AND source.existing_list_id = target.list_id AND source.conflicting_email = target.email_address THEN UPDATE
SET customer_id = :customerId
WHEN NOT MATCHED BY target AND source.existing_site_key = target.site_key AND source.existing_list_id = target.list_id AND source.conflicting_customer_id = :customerId THEN INSERT
(site_key, list_id, email_address, customer_id, attribute1, date1, date2) VALUES
(source.existing_site_key, source.existing_list_id, :emailAddress, source.customer_id, source.existing_attribute1, source.existing_date1, source.existing_date2)
Thanks,
mikee

H2 Optimize select statement / shutdown defrag

Test Case:
drop table master;
create table master(id int primary key, fk1 int, fk2 int, fk3 int, dataS varchar(255), data1 int, data2 int, data3 int, data4 int,data5 int,data6 int,data7 int,data8 int,data9 int,b1 boolean,b2 boolean,b3 boolean,b4 boolean,b5 boolean,b6 boolean,b7 boolean,b8 boolean,b9 boolean,b10 boolean,b11 boolean,b12 boolean,b13 boolean,b14 boolean,b15 boolean,b16 boolean,b17 boolean,b18 boolean,b19 boolean,b20 boolean,b21 boolean,b22 boolean,b23 boolean,b24 boolean,b25 boolean,b26 boolean,b27 boolean,b28 boolean,b29 boolean,b30 boolean,b31 boolean,b32 boolean,b33 boolean,b34 boolean,b35 boolean,b36 boolean,b37 boolean,b38 boolean,b39 boolean,b40 boolean,b41 boolean,b42 boolean,b43 boolean,b44 boolean,b45 boolean,b46 boolean,b47 boolean,b48 boolean,b49 boolean,b50 boolean);
create index idx_comp on master(fk1,fk2,fk3);
#loop 5000000 insert into master values(?, mod(?,100), mod(?,5), ?,'Hello World Hello World Hello World',?, ?, ?,?, ?, ?, ?, ?, ?,true,true,true,true,true,true,false,false,false,true,true,true,true,true,true,true,false,false,false,true,true,true,true,true,true,true,false,false,false,true,true,true,true,true,true,true,false,false,false,true,true,true,true,true,true,true,false,false,false,true);
1.The following select statement takes up to 30seconds. Is there a way to optimize the response time?
SELECT count(*), SUM(CONVERT(b1,INT)) ,SUM(CONVERT(b2,INT)),SUM(CONVERT(b3,INT)),SUM(CONVERT(b4,INT)),SUM(CONVERT(b5,INT)),SUM(CONVERT(b6,INT)),SUM(CONVERT(b7,INT)),SUM(CONVERT(b8,INT)),SUM(CONVERT(b9,INT)),SUM(CONVERT(b10,INT)),SUM(CONVERT(b11,INT)),SUM(CONVERT(b12,INT)),SUM(CONVERT(b13,INT)),SUM(CONVERT(b14,INT)),SUM(CONVERT(b15,INT)),SUM(CONVERT(b16,INT))
FROM master
WHERE fk1=53 AND fk2=3
2.I tried shutdown defrag. But this statement took about 40min for my test case. After shutdown defrag the select takes up to 15seconds. If i execute the statement again it takes under 1sec. Even if stop and start the server, the statement takes about 1sec.
Has H2 a persistent Cache?
Infrastructure: WebBrowser <-> H2 Console Server <-> H2 DB: h2 1.3.158
According to the profiler output, the main problem (93%) is reading from the disk. I ran this in the H2 Console:
#prof_start;
SELECT ... FROM master WHERE fk1=53 AND fk2=3;
#prof_stop;
and got:
Profiler: top 3 stack trace(s) of 48039 ms [build-158]:
4084/4376 (93%):
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:338)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
at org.h2.store.FileStore.readFully(FileStore.java:285)
at org.h2.store.PageStore.readPage(PageStore.java:1253)
at org.h2.store.PageStore.getPage(PageStore.java:707)
at org.h2.index.PageDataIndex.getPage(PageDataIndex.java:225)
at org.h2.index.PageDataNode.getRowWithKey(PageDataNode.java:269)
at org.h2.index.PageDataNode.getRowWithKey(PageDataNode.java:270)
According to EXPLAIN ANALYZE SELECT it's reading over 55'000 pages from the disk (2 KB each page; 110 MB) for this query. I'm not sure how other databases perform for such a query. But I guess if possible the query should be changed so that it reads less data.
Is it possible to have a temporary table/view that already has the datatype conversions done? If it's feasible to have that update itself from the main table occassionally (once a night or so), then you've got a lot of processing power that goes into the conversion done already.
If that's not feasible, you may want to do multiple sub-selects, one for each "b" column, where you only pull where b# = 1. Then do a COUNT instead of a SUM, which should be faster as well. For instance:
SELECT (count1+count2) AS Count,
(SELECT COUNT(*) FROM master WHERE fk1=53 AND fk2=3 AND b1=1) AS count1
(SELECT COUNT(*) FROM master WHERE fk1=53 AND fk2=3 AND b2=1) AS count2
I'm not sure if that exact syntax works in your program, but hopefully as a generic SQL idea it gets you on the right track.