Statement timeout error in postgreSQL for large data - postgresql
We have recently migrated to PostgreSQL 15 from Oracle.
My table structure is as in image below. Only indices are primary keys.
We have this program for finding duplicate( for an entity if any staging record exists in main, then mark as duplicate in staging detail)
Main table has approx. 500 million of data. The below query works when staging table has approx. 1 million records (1 million ~ 500 million comparison), but gives statement timeout error for more than 1 million records. No new index was created in PostgreSQL. Do we have option to increase memory or timeout time in Postgres or any new index can be created to run this query for large data?
Edit : edited query with unique_id >= 15684 condition
SELECT FD.file_id, FD.LINE_NO
from staging_header FH
left join staging_detail FD on FH.file_id = FD.file_id
left join(main_header H left join main_detail D
on H.unique_id = D.unique_id)
ON (FD.TXN_DATE = D.transaction_date and FD.MID = D.merchant_id and FD.TID = D.terminal_id and FD.TXN_REF_NO = D.tx_ref_no
where FH.entity_id = 'XXXX' and FH.entity_id = H.entity_id and unique_id >= 15684
and FH.PROCESS_FLAG = 'N' and FH.PROCESS_STATUS = '00'
and FD.PROCESS_FLAG = 'N' and FD.PROCESS_STATUS = '00'
We have approx. 76 month wise partitions for main_detail table on transaction_date column.
Output of EXPLAIN ANALYSE:
Gather (cost=686297.66..56604907.59 rows=1 width=16) (actual time=391652.590..397957.305 rows=0 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Parallel Hash Join (cost=685297.66..56603907.49 rows=1 width=16) (actual time=391629.181..391630.151 rows=0
loops=3)
Hash Cond: ((d.transaction_date = fd.txn_date) AND ((d.merchant_id)::text = (fd.mid)::text) AND ((d.termina
l_id)::text = (fd.tid)::text) AND ((d.tx_ref_no)::text = (fd.txn_ref_no)::text) AND ((d.auth_code)::text = (fd.auth_
code)::text))
-> Hash Join (cost=748.33..53211966.04 rows=61120806 width=74) (actual time=48280.346..300238.985 rows=14
2758659 loops=3)
Hash Cond: (d.unique_id = h.unique_id)
-> Parallel Append (cost=0.00..51719090.37 rows=568323296 width=77) (actual time=0.242..248105.291
rows=454658663 loops=3)
-> Parallel Seq Scan on main_detail_202204 d_68 (cost=0.00..2765248.17 rows=31161217
width=79) (actual time=0.131..36385.007 rows=74786917 loops=1)
-> Parallel Seq Scan on main_detail_202205 d_69 (cost=0.00..2570672.60 rows=28713160
width=83) (actual time=0.139..34076.709 rows=68911584 loops=1)
-> Parallel Seq Scan on main_detail_202210 d_74 (cost=0.00..2525110.23 rows=27849423
width=88) (actual time=0.066..34458.420 rows=66838615 loops=1)
-> Parallel Seq Scan on main_detail_202208 d_72 (cost=0.00..2437750.12 rows=27017712
width=85) (actual time=1.309..31526.598 rows=64842508 loops=1)
-> Parallel Seq Scan on main_detail_202209 d_73 (cost=0.00..2417680.45 rows=26714745
width=88) (actual time=0.389..30640.660 rows=64115389 loops=1)
-> Parallel Seq Scan on main_detail_202207 d_71 (cost=0.00..2380965.98 rows=26395098
width=85) (actual time=1.878..30367.172 rows=63348237 loops=1)
-> Parallel Seq Scan on main_detail_202206 d_70 (cost=0.00..2288957.10 rows=25457710
width=84) (actual time=1.225..30613.453 rows=61098506 loops=1)
-> Parallel Seq Scan on main_detail_202203 d_67 (cost=0.00..2228805.72 rows=27047072
width=79) (actual time=1.204..30554.095 rows=64912973 loops=1)
-> Parallel Seq Scan on main_detail_202211 d_75 (cost=0.00..2114584.45 rows=23342545
width=89) (actual time=2.361..28424.133 rows=56022109 loops=1)
-> Parallel Seq Scan on main_detail_202110 d_62 (cost=0.00..1929836.90 rows=23170790
width=79) (actual time=1.908..24234.816 rows=55609894 loops=1)
-> Parallel Seq Scan on main_detail_202112 d_64 (cost=0.00..1916823.43 rows=23095743
width=80) (actual time=0.309..24250.450 rows=55429782 loops=1)
-> Parallel Seq Scan on main_detail_202202 d_66 (cost=0.00..1841871.05 rows=22302905
width=81) (actual time=0.920..23430.150 rows=53526972 loops=1)
-> Parallel Seq Scan on main_detail_202111 d_63 (cost=0.00..1811233.87 rows=21760687
width=80) (actual time=1.175..10110.061 rows=17408550 loops=3)
-> Parallel Seq Scan on main_detail_202201 d_65 (cost=0.00..1781216.65 rows=21494365
width=81) (actual time=1.924..14664.887 rows=25793238 loops=2)
-> Parallel Seq Scan on main_detail_202108 d_60 (cost=0.00..1672640.07 rows=19995007
width=78) (actual time=1.034..21265.332 rows=47988018 loops=1)
-> Parallel Seq Scan on main_detail_202109 d_61 (cost=0.00..1642747.75 rows=19685775
width=78) (actual time=0.495..20735.015 rows=47245860 loops=1)
-> Parallel Seq Scan on main_detail_202107 d_59 (cost=0.00..1582179.23 rows=18876623
width=75) (actual time=1.217..24656.746 rows=45303895 loops=1)
-> Parallel Seq Scan on main_detail_202106 d_58 (cost=0.00..1416963.00 rows=16879900
width=75) (actual time=0.461..18096.119 rows=40511759 loops=1)
-> Parallel Seq Scan on main_detail_202104 d_56 (cost=0.00..1247189.85 rows=14815785
width=73) (actual time=1.271..16250.747 rows=35557883 loops=1)
-> Parallel Seq Scan on main_detail_202105 d_57 (cost=0.00..1034296.03 rows=12273802
width=75) (actual time=0.279..12897.524 rows=29457126 loops=1)
-> Parallel Seq Scan on main_detail_202001 d_41 (cost=0.00..660224.22 rows=7901322 wi
dth=62) (actual time=0.851..8385.719 rows=18963171 loops=1)
-> Parallel Seq Scan on main_detail_202103 d_55 (cost=0.00..636403.65 rows=7600365 wi
dth=59) (actual time=0.636..7917.474 rows=18240877 loops=1)
-> Parallel Seq Scan on main_detail_201912 d_40 (cost=0.00..634131.68 rows=7575968 wi
dth=62) (actual time=0.981..7876.041 rows=18182322 loops=1)
-> Parallel Seq Scan on main_detail_202002 d_42 (cost=0.00..618985.88 rows=7403088 wi
dth=61) (actual time=0.443..7823.615 rows=17767413 loops=1)
-> Parallel Seq Scan on main_detail_202101 d_53 (cost=0.00..594448.23 rows=7099623 wi
dth=60) (actual time=0.579..7617.154 rows=17039096 loops=1)
-> Parallel Seq Scan on main_detail_202012 d_52 (cost=0.00..592949.22 rows=7075422 wi
dth=60) (actual time=0.289..7599.207 rows=16981011 loops=1)
-> Parallel Seq Scan on main_detail_202102 d_54 (cost=0.00..557941.45 rows=6666945 wi
dth=59) (actual time=1.097..8469.829 rows=16000668 loops=1)
-> Parallel Seq Scan on main_detail_202011 d_51 (cost=0.00..551561.22 rows=6580722 wi
dth=60) (actual time=2.323..7845.248 rows=15793732 loops=1)
-> Parallel Seq Scan on main_detail_202010 d_50 (cost=0.00..541516.32 rows=6465732 wi
dth=60) (actual time=3.071..7509.806 rows=15517757 loops=1)
-> Parallel Seq Scan on main_detail_202003 d_43 (cost=0.00..500525.41 rows=5982441 wi
dth=61) (actual time=0.505..6566.428 rows=14357859 loops=1)
-> Parallel Seq Scan on main_detail_202009 d_49 (cost=0.00..498229.27 rows=5952427 wi
dth=60) (actual time=1.340..6354.120 rows=14285824 loops=1)
-> Parallel Seq Scan on main_detail_202008 d_48 (cost=0.00..473451.19 rows=5649219 wi
dth=60) (actual time=1.899..6152.809 rows=13558126 loops=1)
-> Parallel Seq Scan on main_detail_202007 d_47 (cost=0.00..442856.52 rows=5280052 wi
dth=60) (actual time=0.688..5718.016 rows=12672124 loops=1)
-> Parallel Seq Scan on main_detail_202006 d_46 (cost=0.00..431238.51 rows=5139651 wi
dth=60) (actual time=1.022..5570.241 rows=12335163 loops=1)
-> Parallel Seq Scan on main_detail_202005 d_45 (cost=0.00..322049.82 rows=3843882 wi
dth=59) (actual time=0.514..4179.456 rows=9225317 loops=1)
-> Parallel Seq Scan on main_detail_202004 d_44 (cost=0.00..170375.59 rows=2033059 wi
dth=59) (actual time=1.270..2200.745 rows=4879342 loops=1)
-> Parallel Seq Scan on main_detail_201911 d_39 (cost=0.00..99748.34 rows=1126534 wid
th=60) (actual time=0.442..1238.540 rows=2703682 loops=1)
-> Parallel Seq Scan on main_detail_201904 d_32 (cost=0.00..77520.76 rows=877776 widt
h=60) (actual time=1.233..1038.713 rows=2106663 loops=1)
-> Parallel Seq Scan on main_detail_201905 d_33 (cost=0.00..75244.24 rows=852324 widt
h=60) (actual time=0.033..1018.941 rows=2045578 loops=1)
-> Parallel Seq Scan on main_detail_201908 d_36 (cost=0.00..71581.46 rows=807346 widt
h=60) (actual time=0.034..756.542 rows=1937631 loops=1)
-> Parallel Seq Scan on main_detail_201907 d_35 (cost=0.00..70736.10 rows=798410 widt
h=60) (actual time=0.031..815.549 rows=1916185 loops=1)
-> Parallel Seq Scan on main_detail_201906 d_34 (cost=0.00..70287.73 rows=795673 widt
h=60) (actual time=4.525..1071.337 rows=1909616 loops=1)
-> Parallel Seq Scan on main_detail_201910 d_38 (cost=0.00..64894.83 rows=731183 widt
h=60) (actual time=4.229..1106.733 rows=1754839 loops=1)
-> Parallel Seq Scan on main_detail_201909 d_37 (cost=0.00..64030.31 rows=722131 widt
h=60) (actual time=0.036..1115.049 rows=1733114 loops=1)
-> Parallel Seq Scan on main_detail_202212 d_76 (cost=0.00..62668.10 rows=734710 widt
h=70) (actual time=0.502..941.721 rows=1763305 loops=1)
-> Parallel Seq Scan on main_detail_201704 d_8 (cost=0.00..20424.27 rows=244827 width
=59) (actual time=0.023..253.853 rows=587584 loops=1)
-> Parallel Seq Scan on main_detail_201805 d_21 (cost=0.00..18971.21 rows=222421 widt
h=60) (actual time=0.365..255.920 rows=534441 loops=1)
-> Parallel Seq Scan on main_detail_201806 d_22 (cost=0.00..17918.28 rows=211428 widt
h=60) (actual time=0.022..244.113 rows=507427 loops=1)
-> Parallel Seq Scan on main_detail_201804 d_20 (cost=0.00..17845.21 rows=210521 widt
h=60) (actual time=0.033..227.176 rows=505301 loops=1)
-> Parallel Seq Scan on main_detail_201803 d_19 (cost=0.00..17609.88 rows=208088 widt
h=60) (actual time=0.018..201.712 rows=499411 loops=1)
-> Parallel Seq Scan on main_detail_201810 d_26 (cost=0.00..17272.90 rows=203890 widt
h=60) (actual time=3.561..241.027 rows=489337 loops=1)
-> Parallel Seq Scan on main_detail_201811 d_27 (cost=0.00..16898.90 rows=199690 widt
h=60) (actual time=0.016..226.327 rows=479255 loops=1)
-> Parallel Seq Scan on main_detail_201801 d_17 (cost=0.00..16872.47 rows=199347 widt
h=60) (actual time=0.011..228.096 rows=478432 loops=1)
-> Parallel Seq Scan on main_detail_201809 d_25 (cost=0.00..16693.76 rows=197076 widt
h=60) (actual time=0.011..232.691 rows=472983 loops=1)
-> Parallel Seq Scan on main_detail_201808 d_24 (cost=0.00..16615.35 rows=196135 widt
h=60) (actual time=0.527..256.811 rows=470725 loops=1)
-> Parallel Seq Scan on main_detail_201705 d_9 (cost=0.00..16518.85 rows=196985 width
=60) (actual time=0.633..247.533 rows=472764 loops=1)
-> Parallel Seq Scan on main_detail_201710 d_14 (cost=0.00..16385.53 rows=192653 widt
h=61) (actual time=0.011..242.029 rows=462368 loops=1)
-> Parallel Seq Scan on main_detail_201807 d_23 (cost=0.00..16300.88 rows=192788 widt
h=60) (actual time=0.011..212.557 rows=462691 loops=1)
-> Parallel Seq Scan on main_detail_201712 d_16 (cost=0.00..15956.17 rows=189018 widt
h=60) (actual time=0.377..233.791 rows=453642 loops=1)
-> Parallel Seq Scan on main_detail_201903 d_31 (cost=0.00..15630.37 rows=184637 widt
h=60) (actual time=2.076..195.545 rows=443129 loops=1)
-> Parallel Seq Scan on main_detail_201802 d_18 (cost=0.00..15556.97 rows=183697 widt
h=60) (actual time=0.019..200.397 rows=440872 loops=1)
-> Parallel Seq Scan on main_detail_201812 d_28 (cost=0.00..15281.62 rows=180662 widt
h=60) (actual time=2.067..209.129 rows=433590 loops=1)
-> Parallel Seq Scan on main_detail_201709 d_13 (cost=0.00..14963.41 rows=177641 widt
h=60) (actual time=0.011..207.096 rows=426339 loops=1)
-> Parallel Seq Scan on main_detail_201901 d_29 (cost=0.00..14355.48 rows=169448 widt
h=60) (actual time=0.453..200.458 rows=406674 loops=1)
-> Parallel Seq Scan on main_detail_201708 d_12 (cost=0.00..14190.58 rows=168458 widt
h=60) (actual time=0.676..187.462 rows=404299 loops=1)
-> Parallel Seq Scan on main_detail_201711 d_15 (cost=0.00..13645.50 rows=161350 widt
h=60) (actual time=0.487..190.109 rows=387239 loops=1)
-> Parallel Seq Scan on main_detail_201902 d_30 (cost=0.00..13521.67 rows=159367 widt
h=60) (actual time=0.643..183.340 rows=382480 loops=1)
-> Parallel Seq Scan on main_detail_201706 d_10 (cost=0.00..13430.53 rows=159553 widt
h=60) (actual time=0.981..186.323 rows=382928 loops=1)
-> Parallel Seq Scan on main_detail_201707 d_11 (cost=0.00..13163.89 rows=156389 widt
h=60) (actual time=2.020..183.935 rows=375334 loops=1)
-> Parallel Seq Scan on main_detail_201703 d_7 (cost=0.00..926.89 rows=14989 width=60
) (actual time=0.009..10.364 rows=25481 loops=1)
-> Parallel Seq Scan on main_detail_201612 d_4 (cost=0.00..14.23 rows=223 width=61) (
actual time=0.009..0.556 rows=379 loops=1)
-> Parallel Seq Scan on main_detail_201610 d_2 (cost=0.00..10.29 rows=29 width=408) (
actual time=0.000..0.003 rows=0 loops=1)
-> Parallel Seq Scan on main_detail_202301 d_77 (cost=0.00..10.29 rows=29 width=408)
(actual time=0.000..0.001 rows=0 loops=1)
-> Parallel Seq Scan on main_detail_202302 d_78 (cost=0.00..10.29 rows=29 width=408)
(actual time=0.001..0.001 rows=0 loops=1)
-> Parallel Seq Scan on main_detail_202303 d_79 (cost=0.00..10.29 rows=29 width=408)
(actual time=0.000..0.001 rows=0 loops=1)
-> Parallel Seq Scan on main_detail_202304 d_80 (cost=0.00..10.29 rows=29 width=408)
(actual time=0.000..0.003 rows=0 loops=1)
-> Parallel Seq Scan on main_detail_202305 d_81 (cost=0.00..10.29 rows=29 width=408)
(actual time=0.000..0.000 rows=0 loops=1)
-> Parallel Seq Scan on main_detail_202306 d_82 (cost=0.00..10.29 rows=29 width=408)
(actual time=0.000..0.000 rows=0 loops=1)
-> Parallel Seq Scan on main_detail_202307 d_83 (cost=0.00..10.29 rows=29 width=408)
(actual time=0.000..0.000 rows=0 loops=1)
-> Parallel Seq Scan on main_detail_202308 d_84 (cost=0.00..10.29 rows=29 width=408)
(actual time=0.000..0.000 rows=0 loops=1)
-> Parallel Seq Scan on main_detail_202309 d_85 (cost=0.00..10.29 rows=29 width=408)
(actual time=0.000..0.002 rows=0 loops=1)
-> Parallel Seq Scan on main_detail_202310 d_86 (cost=0.00..10.29 rows=29 width=408)
(actual time=0.000..0.000 rows=0 loops=1)
-> Parallel Seq Scan on main_detail_202312 d_87 (cost=0.00..10.29 rows=29 width=408)
(actual time=0.000..0.001 rows=0 loops=1)
-> Parallel Seq Scan on main_detail_201702 d_6 (cost=0.00..7.06 rows=106 width=61) (a
ctual time=0.009..0.457 rows=180 loops=1)
-> Parallel Seq Scan on main_detail_before_2016 d_1 (cost=0.00..2.43 rows=43 width=21
6) (actual time=0.010..0.027 rows=73 loops=1)
-> Parallel Seq Scan on main_detail_201701 d_5 (cost=0.00..2.35 rows=35 width=60) (ac
tual time=0.318..0.336 rows=59 loops=1)
-> Parallel Seq Scan on main_detail_201611 d_3 (cost=0.00..1.02 rows=2 width=408) (ac
tual time=0.453..0.458 rows=4 loops=1)
-> Hash (cost=717.83..717.83 rows=2440 width=13) (actual time=100.803..100.806 rows=4086 loops=3)
Buckets: 4096 Batches: 1 Memory Usage: 212kB
-> Bitmap Heap Scan on main_header h (cost=47.54..717.83 rows=2440 width=13) (actual
time=1.429..99.670 rows=4086 loops=3)
Recheck Cond: ((entity_id)::text = 'W002'::text)
Filter: (unique_id >= 15684)
Heap Blocks: exact=421
-> Bitmap Index Scan on idx_pos_cb_hdr_t99 (cost=0.00..46.93 rows=4086 width=0) (actual
time=1.002..1.002 rows=4086 loops=3)
Index Cond: ((entity_id)::text = 'W002'::text)
-> Parallel Hash (cost=661416.16..661416.16 rows=606763 width=101) (actual time=7268.024..7268.037 rows=1
894307 loops=3)
Buckets: 65536 (originally 65536) Batches: 128 (originally 32) Memory Usage: 7136kB
-> Hash Join (cost=7.85..661416.16 rows=606763 width=101) (actual time=3559.203..5547.343 rows=1894
307 loops=3)
Hash Cond: (fd.file_id = fh.file_id)
-> Parallel Seq Scan on staging_detail fd (cost=0.00..642716.64 rows=6977776
width=96) (actual time=1.887..4871.854 rows=5580857 loops=3)
Filter: (((process_flag)::text = 'N'::text) AND ((process_status)::text = '00'::text))
Rows Removed by Filter: 162
-> Hash (cost=7.62..7.62 rows=18 width=13) (actual time=0.154..0.157 rows=46 loops=3)
Buckets: 1024 Batches: 1 Memory Usage: 11kB
-> Seq Scan on staging_header fh (cost=0.00..7.62 rows=18 width=13) (ac
tual time=0.082..0.126 rows=46 loops=3)
Filter: (((entity_id)::text = 'W002'::text) AND ((process_flag)::text = 'N'::text)
AND ((process_status)::text = '00'::text))
Rows Removed by Filter: 121
Planning Time: 65.403 ms
Execution Time: 397960.649 ms
(117 rows)
Related
postgres IN clause not using index
When I am using IN, the query is not using index and scanning mview on which view1 is built. But same query when = is used, uses index and returns fast. How to make IN use index and complete fast. Here its using "=" clause explain analyze select * from schema1.view1 cr where c_cd = (select distinct c_cd from schema1.mview1 where cd = 'XY87296C'); Subquery Scan on res (cost=110.44..110.52 rows=1 width=271) (actual time=0.130..0.139 rows=5 loops=1) Filter: (res.rnk = 1) InitPlan 1 (returns $0) -> Unique (cost=60.80..60.90 rows=20 width=6) (actual time=0.041..0.045 rows=1 loops=1) -> Sort (cost=60.80..60.85 rows=20 width=6) (actual time=0.041..0.042 rows=3 loops=1) Sort Key: mview1.c_cd Sort Method: quicksort Memory: 25kB -> Index Scan using ix_cd_n on mview1 (cost=0.43..60.37 rows=20 width=6) (actual time=0.030..0.034 rows=3 loops=1) Index Cond: ((cd)::text = 'XY87296C'::text) -> WindowAgg (cost=49.53..49.59 rows=2 width=1287) (actual time=0.128..0.134 rows=5 loops=1) -> Sort (cost=49.53..49.54 rows=2 width=271) (actual time=0.119..0.120 rows=5 loops=1) Sort Key: cerer.rer_cd, cerer.r_dt DESC, cerer.rere_dt DESC, cerer.rere1_dt DESC Sort Method: quicksort Memory: 27kB -> Nested Loop (cost=0.84..49.52 rows=2 width=271) (actual time=0.101..0.107 rows=5 loops=1) Join Filter: (cerer.r_dt = mx.r_dt) -> Index Scan using ix_c_cd on cerer cerer (cost=0.42..24.51 rows=5 width=271) (actual time=0.069..0.070 rows=5 loops=1) Index Cond: ((c_cd)::text = ($0)::text) -> Materialize (cost=0.42..24.66 rows=5 width=13) (actual time=0.006..0.006 rows=1 loops=5) -> Subquery Scan on mx (cost=0.42..24.63 rows=5 width=13) (actual time=0.026..0.027 rows=1 loops=1) -> GroupAggregate (cost=0.42..24.58 rows=5 width=13) (actual time=0.026..0.026 rows=1 loops=1) Group Key: cerer.c_cd -> Index Scan using ix_c_cd on cerer (cost=0.42..24.51 rows=5 width=13) (actual time=0.011..0.018 rows=5 loops=1) Index Cond: ((c_cd)::text = ($0)::text) Planning Time: 0.812 ms Execution Time: 0.471 ms Here its using IN clause explain analyze select * from schema1.view1 cr where c_cd in (select distinct c_cd from schema1.mview1 where cd = 'XY87296C'); Merge Join (cost=74280.97..74345.71 rows=1 width=271) (actual time=1415.760..1415.861 rows=5 loops=1) Merge Cond: ((res.c_cd)::text = (mview1.c_cd)::text) -> Subquery Scan on res (cost=74220.16..74284.52 rows=8 width=271) (actual time=1214.729..1395.978 rows=144485 loops=1) Filter: (res.rnk = 1) Rows Removed by Filter: 534 -> WindowAgg (cost=74220.16..74264.41 rows=1609 width=1287) (actual time=1214.726..1383.997 rows=145019 loops=1) -> Sort (cost=74220.16..74224.19 rows=1609 width=271) (actual time=1214.706..1239.776 rows=145020 loops=1) Sort Key: cerer.c_cd, cerer.rer_cd, cerer.r_dt DESC, cerer.rere_dt DESC, cerer.rere1_dt DESC Sort Method: quicksort Memory: 111598kB -> Gather (cost=40122.36..74134.47 rows=1609 width=271) (actual time=308.210..409.188 rows=205422 loops=1) Workers Planned: 2 Workers Launched: 2 -> Hash Join (cost=39122.36..72973.57 rows=670 width=271) (actual time=301.493..400.779 rows=68474 loops=3) Hash Cond: (((cerer.c_cd)::text = (mx.c_cd)::text) AND (cerer.r_dt = mx.r_dt)) -> Parallel Seq Scan on cerer cerer (cost=0.00..33147.12 rows=134112 width=271) (actual time=0.004..20.591 rows=107290 loops=3) -> Hash (cost=38055.93..38055.93 rows=71095 width=13) (actual time=300.894..300.896 rows=108049 loops=3) Buckets: 131072 Batches: 1 Memory Usage: 6089kB -> Subquery Scan on mx (cost=36634.04..38055.93 rows=71095 width=13) (actual time=246.132..278.589 rows=108049 loops=3) -> HashAggregate (cost=36634.04..37344.99 rows=71095 width=13) (actual time=246.130..269.294 rows=108049 loops=3) Group Key: cerer.c_cd -> Seq Scan on cerer (cost=0.00..35024.69 rows=321869 width=13) (actual time=0.019..60.328 rows=321869 loops=3) -> Unique (cost=60.80..60.90 rows=20 width=6) (actual time=0.056..0.060 rows=1 loops=1) -> Sort (cost=60.80..60.85 rows=20 width=6) (actual time=0.055..0.057 rows=3 loops=1) Sort Key: mview1.c_cd Sort Method: quicksort Memory: 25kB -> Index Scan using ix_consume_customer_health_coverage_cd_n on mview1 (cost=0.43..60.37 rows=20 width=6) (actual time=0.040..0.045 rows=3 loops=1) Index Cond: ((cd)::text = 'XY87296C'::text) Planning Time: 0.770 ms Execution Time: 1424.928 ms
Postgres hash join batches explosion
We are having some struggle identifying why Postgres is using too much batches to resolve a join.
Here it is the output of explain analyze of a problematic execution:
https://explain.dalibo.com/plan/xNJ#plan
Limit (cost=20880.87..20882.91 rows=48 width=205) (actual time=10722.953..10723.358 rows=48 loops=1)
-> Unique (cost=20880.87..21718.12 rows=19700 width=205) (actual time=10722.951..10723.356 rows=48 loops=1)
-> Sort (cost=20880.87..20930.12 rows=19700 width=205) (actual time=10722.950..10722.990 rows=312 loops=1)
Sort Key: titlemetadata_titlemetadata.creation_date DESC, titlemetadata_titlemetadata.id, titlemetadata_titlemetadata.title_type, titlemetadata_titlemetadata.original_title, titlemetadata_titlemetadata.alternative_ids, titlemetadata_titlemetadata.metadata,
titlemetadata_titlemetadata.is_adult, titlemetadata_titlemetadata.is_kids, titlemetadata_titlemetadata.last_modified, titlemetadata_titlemetadata.year, titlemetadata_titlemetadata.runtime, titlemetadata_titlemetadata.rating, titlemetadata_titlemetadata.video_provider, tit
lemetadata_titlemetadata.series_id_id, titlemetadata_titlemetadata.season_number, titlemetadata_titlemetadata.episode_number
Sort Method: quicksort Memory: 872kB
-> Hash Right Join (cost=13378.20..19475.68 rows=19700 width=205) (actual time=1926.352..10709.970 rows=2909 loops=1)
Hash Cond: (t4.titlemetadata_id = t3.id)
Filter: ((hashed SubPlan 1) OR (hashed SubPlan 2))
Rows Removed by Filter: 63248
-> Seq Scan on video_provider_offer t4 (cost=0.00..5454.90 rows=66290 width=16) (actual time=0.024..57.893 rows=66390 loops=1)
-> Hash (cost=11314.39..11314.39 rows=22996 width=221) (actual time=489.530..489.530 rows=60096 loops=1)
Buckets: 65536 (originally 32768) Batches: 32768 (originally 1) Memory Usage: 11656kB
-> Hash Right Join (cost=5380.95..11314.39 rows=22996 width=221) (actual time=130.024..225.271 rows=60096 loops=1)
Hash Cond: (video_provider_offer.titlemetadata_id = titlemetadata_titlemetadata.id)
-> Seq Scan on video_provider_offer (cost=0.00..5454.90 rows=66290 width=16) (actual time=0.011..32.950 rows=66390 loops=1)
-> Hash (cost=5129.28..5129.28 rows=20133 width=213) (actual time=129.897..129.897 rows=55793 loops=1)
Buckets: 65536 (originally 32768) Batches: 2 (originally 1) Memory Usage: 7877kB
-> Merge Left Join (cost=1.72..5129.28 rows=20133 width=213) (actual time=0.041..93.057 rows=55793 loops=1)
Merge Cond: (titlemetadata_titlemetadata.id = t3.series_id_id)
-> Index Scan using titlemetadata_titlemetadata_pkey on titlemetadata_titlemetadata (cost=1.30..4130.22 rows=20133 width=205) (actual time=0.028..62.949 rows=43921 loops=1)
Filter: ((NOT is_adult) AND (NOT (hashed SubPlan 3)) AND (((title_type)::text = 'MOV'::text) OR ((title_type)::text = 'TVS'::text) OR ((title_type)::text = 'TVP'::text) OR ((title_type)::text = 'EVT'::text)))
Rows Removed by Filter: 14121
SubPlan 3
-> Seq Scan on cable_operator_cableoperatorexcludedtitle u0_2 (cost=0.00..1.01 rows=1 width=8) (actual time=0.006..0.006 rows=0 loops=1)
Filter: (cable_operator_id = 54)
-> Index Scan using titlemetadata_titlemetadata_series_id_id_73453db4_uniq on titlemetadata_titlemetadata t3 (cost=0.41..3901.36 rows=58037 width=16) (actual time=0.011..9.375 rows=12887 loops=1)
SubPlan 1
-> Hash Join (cost=44.62..885.73 rows=981 width=8) (actual time=0.486..36.806 rows=5757 loops=1)
Hash Cond: (w2.device_id = w3.id)
-> Nested Loop (cost=43.49..866.20 rows=2289 width=16) (actual time=0.441..33.096 rows=20180 loops=1)
-> Nested Loop (cost=43.06..414.98 rows=521 width=8) (actual time=0.426..9.952 rows=2909 loops=1)
Join Filter: (w1.id = w0.video_provider_id)
-> Nested Loop (cost=42.65..54.77 rows=13 width=24) (actual time=0.399..0.532 rows=15 loops=1)
-> HashAggregate (cost=42.50..42.95 rows=45 width=16) (actual time=0.390..0.403 rows=45 loops=1)
Group Key: v0.id
-> Nested Loop (cost=13.34..42.39 rows=45 width=16) (actual time=0.095..0.364 rows=45 loops=1)
-> Hash Semi Join (cost=13.19..32.72 rows=45 width=8) (actual time=0.084..0.229 rows=45 loops=1)
Hash Cond: (v1.id = u0.id)
-> Seq Scan on cable_operator_cableoperatorprovider v1 (cost=0.00..17.36 rows=636 width=16) (actual time=0.010..0.077 rows=636 loops=1)
-> Hash (cost=12.63..12.63 rows=45 width=8) (actual time=0.046..0.046 rows=45 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Index Scan using cable_operator_cableoperatorprovider_4d6e54b3 on cable_operator_cableoperatorprovider u0 (cost=0.28..12.63 rows=45 width=8) (actual time=0.016..0.035 rows=45 loops=1)
Index Cond: (cable_operator_id = 54)
-> Index Only Scan using video_provider_videoprovider_pkey on video_provider_videoprovider v0 (cost=0.15..0.20 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=45)
Index Cond: (id = v1.provider_id)
Heap Fetches: 45
-> Index Scan using video_provider_videoprovider_pkey on video_provider_videoprovider w1 (cost=0.15..0.25 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (id = v0.id)
Filter: ((video_provider_type)::text = 'VOD'::text)
Rows Removed by Filter: 1
-> Index Scan using video_provider_offer_da942d2e on video_provider_offer w0 (cost=0.42..27.22 rows=39 width=16) (actual time=0.026..0.585 rows=194 loops=15)
Index Cond: (video_provider_id = v0.id)
Filter: (((end_date > '2021-09-02 19:23:00-03'::timestamp with time zone) OR (end_date IS NULL)) AND (access_criteria && '{vtv_mas,TBX_LOGIN,urn:spkg:tve:fox-premium,urn:tve:mcp,AMCHD,AMC_CONSORCIO,ANIMAL_PLANET,ASUNTOS_PUBLI
COS,ASUNTOS_PUBLICOS_CONSORCIO,CINECANALLIVE,CINECANAL_CONSORCIO,DISCOVERY,DISCOVERY_KIDS_CONSORCIO,DISCOVERY_KIDS_OD,DISNEY,DISNEY_CH_CONSORCIO,DISNEY_XD,DISNEY_XD_CONSORCIO,EL_CANAL_HD,EL_CANAL_HD_CONSORCIO,EL_GOURMET_CONSORCIO,ESPN,ESPN2_HD_CONSORCIO,ESPN3_HD_CONSORCIO
,ESPNMAS_HD_CONSORCIO,ESPN_BASIC,ESPN_HD_CONSORCIO,ESPN_PLAY,EUROPALIVE,EUROPA_EUROPA,EUROPA_EUROPA_CONSORCIO,FILMANDARTS_DISPOSITIVOS,FILMS_ARTS,FILM_AND_ARTS_CONSORCIO,FOXLIFE,FOX_LIFE_CONSORCIO,FOX_SPORTS_1_DISPOSITIVOS,FOX_SPORTS_2_DISPOSITIVOS,FOX_SPORTS_2_HD_CONSORC
IO,FOX_SPORTS_3_DISPOSITIVOS,FOX_SPORTS_3_HD_CONSORCIO,FOX_SPORTS_HD_CONSORCIO,FRANCE24_DISPOSITIVOS,FRANCE_24_CONSORCIO,GOURMET,GOURMET_DISPOSITIVOS,HOME_HEALTH,INVESTIGATION_DISCOVERY,MAS_CHIC,NATGEOKIDS_DISPOSITIVOS,NATGEO_CONSORCIO,NATGEO_DISPOSITIVOS,NATGEO_KIDS_CONS
ORCIO,PASIONES,PASIONES_CONSORCIO,SVOD_TYC_BASIC,TBX_LOGIN,TCC_2_CONSORCIO,TCC_2_HD,TLC,TVE,TVE_CONSORCIO,TYC_SPORTS_CONSORCIO,VTV_LIVE,clarosports,discoverykids,espnplay_south_alt,urn:spkg:tve:fox-basic,urn:tve:babytv,urn:tve:cinecanal,urn:tve:discoverykids,urn:tve:foxli
fe,urn:tve:fp,urn:tve:fx,urn:tve:natgeo,urn:tve:natgeokids,urn:tve:natgeowild,urn:tve:thefilmzone}'::character varying(50)[]) AND ((((content_type)::text = 'VOD'::text) AND ((start_date < '2021-09-02 19:23:00-03'::timestamp with time zone) OR (start_date IS NULL))) OR ((c
ontent_type)::text = 'LIV'::text)))
Rows Removed by Filter: 5
-> Index Only Scan using video_provider_offer_devices_offer_id_device_id_key on video_provider_offer_devices w2 (cost=0.42..0.81 rows=6 width=16) (actual time=0.004..0.007 rows=7 loops=2909)
Index Cond: (offer_id = w0.id)
Heap Fetches: 17828
-> Hash (cost=1.10..1.10 rows=3 width=8) (actual time=0.029..0.029 rows=2 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on platform_device_device w3 (cost=0.00..1.10 rows=3 width=8) (actual time=0.024..0.027 rows=2 loops=1)
Filter: ((device_code)::text = ANY ('{ANDROID,ott_dual_tcc,ott_k2_tcc}'::text[]))
Rows Removed by Filter: 5
SubPlan 2
-> Hash Join (cost=44.62..885.73 rows=981 width=8) (actual time=0.410..33.580 rows=5757 loops=1)
Hash Cond: (w2_1.device_id = w3_1.id)
-> Nested Loop (cost=43.49..866.20 rows=2289 width=16) (actual time=0.375..29.886 rows=20180 loops=1)
-> Nested Loop (cost=43.06..414.98 rows=521 width=8) (actual time=0.366..9.134 rows=2909 loops=1)
Join Filter: (w1_1.id = w0_1.video_provider_id)
-> Nested Loop (cost=42.65..54.77 rows=13 width=24) (actual time=0.343..0.476 rows=15 loops=1)
-> HashAggregate (cost=42.50..42.95 rows=45 width=16) (actual time=0.333..0.347 rows=45 loops=1)
Group Key: v0_1.id
-> Nested Loop (cost=13.34..42.39 rows=45 width=16) (actual time=0.083..0.311 rows=45 loops=1)
-> Hash Semi Join (cost=13.19..32.72 rows=45 width=8) (actual time=0.076..0.202 rows=45 loops=1)
Hash Cond: (v1_1.id = u0_1.id)
-> Seq Scan on cable_operator_cableoperatorprovider v1_1 (cost=0.00..17.36 rows=636 width=16) (actual time=0.005..0.057 rows=636 loops=1)
-> Hash (cost=12.63..12.63 rows=45 width=8) (actual time=0.038..0.038 rows=45 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Index Scan using cable_operator_cableoperatorprovider_4d6e54b3 on cable_operator_cableoperatorprovider u0_1 (cost=0.28..12.63 rows=45 width=8) (actual time=0.007..0.020 rows=45 loops=1)
Index Cond: (cable_operator_id = 54)
-> Index Only Scan using video_provider_videoprovider_pkey on video_provider_videoprovider v0_1 (cost=0.15..0.20 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=45)
Index Cond: (id = v1_1.provider_id)
Heap Fetches: 45
-> Index Scan using video_provider_videoprovider_pkey on video_provider_videoprovider w1_1 (cost=0.15..0.25 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (id = v0_1.id)
Filter: ((video_provider_type)::text = 'VOD'::text)
Rows Removed by Filter: 1
-> Index Scan using video_provider_offer_da942d2e on video_provider_offer w0_1 (cost=0.42..27.22 rows=39 width=16) (actual time=0.022..0.536 rows=194 loops=15)
Index Cond: (video_provider_id = v0_1.id)
Filter: (((end_date > '2021-09-02 19:23:00-03'::timestamp with time zone) OR (end_date IS NULL)) AND (access_criteria && '{vtv_mas,TBX_LOGIN,urn:spkg:tve:fox-premium,urn:tve:mcp,AMCHD,AMC_CONSORCIO,ANIMAL_PLANET,ASUNTOS_PUBLI
COS,ASUNTOS_PUBLICOS_CONSORCIO,CINECANALLIVE,CINECANAL_CONSORCIO,DISCOVERY,DISCOVERY_KIDS_CONSORCIO,DISCOVERY_KIDS_OD,DISNEY,DISNEY_CH_CONSORCIO,DISNEY_XD,DISNEY_XD_CONSORCIO,EL_CANAL_HD,EL_CANAL_HD_CONSORCIO,EL_GOURMET_CONSORCIO,ESPN,ESPN2_HD_CONSORCIO,ESPN3_HD_CONSORCIO
,ESPNMAS_HD_CONSORCIO,ESPN_BASIC,ESPN_HD_CONSORCIO,ESPN_PLAY,EUROPALIVE,EUROPA_EUROPA,EUROPA_EUROPA_CONSORCIO,FILMANDARTS_DISPOSITIVOS,FILMS_ARTS,FILM_AND_ARTS_CONSORCIO,FOXLIFE,FOX_LIFE_CONSORCIO,FOX_SPORTS_1_DISPOSITIVOS,FOX_SPORTS_2_DISPOSITIVOS,FOX_SPORTS_2_HD_CONSORC
IO,FOX_SPORTS_3_DISPOSITIVOS,FOX_SPORTS_3_HD_CONSORCIO,FOX_SPORTS_HD_CONSORCIO,FRANCE24_DISPOSITIVOS,FRANCE_24_CONSORCIO,GOURMET,GOURMET_DISPOSITIVOS,HOME_HEALTH,INVESTIGATION_DISCOVERY,MAS_CHIC,NATGEOKIDS_DISPOSITIVOS,NATGEO_CONSORCIO,NATGEO_DISPOSITIVOS,NATGEO_KIDS_CONS
ORCIO,PASIONES,PASIONES_CONSORCIO,SVOD_TYC_BASIC,TBX_LOGIN,TCC_2_CONSORCIO,TCC_2_HD,TLC,TVE,TVE_CONSORCIO,TYC_SPORTS_CONSORCIO,VTV_LIVE,clarosports,discoverykids,espnplay_south_alt,urn:spkg:tve:fox-basic,urn:tve:babytv,urn:tve:cinecanal,urn:tve:discoverykids,urn:tve:foxli
fe,urn:tve:fp,urn:tve:fx,urn:tve:natgeo,urn:tve:natgeokids,urn:tve:natgeowild,urn:tve:thefilmzone}'::character varying(50)[]) AND ((((content_type)::text = 'VOD'::text) AND ((start_date < '2021-09-02 19:23:00-03'::timestamp with time zone) OR (start_date IS NULL))) OR ((c
ontent_type)::text = 'LIV'::text)))
Rows Removed by Filter: 5
-> Index Only Scan using video_provider_offer_devices_offer_id_device_id_key on video_provider_offer_devices w2_1 (cost=0.42..0.81 rows=6 width=16) (actual time=0.003..0.006 rows=7 loops=2909)
Index Cond: (offer_id = w0_1.id)
Heap Fetches: 17828
-> Hash (cost=1.10..1.10 rows=3 width=8) (actual time=0.015..0.015 rows=2 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on platform_device_device w3_1 (cost=0.00..1.10 rows=3 width=8) (actual time=0.010..0.011 rows=2 loops=1)
Filter: ((device_code)::text = ANY ('{ANDROID,ott_dual_tcc,ott_k2_tcc}'::text[]))
Rows Removed by Filter: 5
Planning time: 8.255 ms
Execution time: 10723.830 ms
(100 rows)
The weird part is that the same query, sometimes just uses a single batch. Here is an example: https://explain.dalibo.com/plan/zTv#plan
Here is the work_mem being used:
show work_mem;
work_mem
----------
8388kB
(1 row)
I'm not interested in changing the query to be more performant, but in understanding why is the different behavior.
I've found this thread apparently related with this, but I don't quite understand what are they talking about: https://www.postgresql.org/message-id/flat/CA%2BhUKGKWWmf%3DWELLG%3DaUGbcugRaSQbtm0tKYiBut-B2rVKX63g%40mail.gmail.com
Can anyone tell me why is this different behavior? The underlying data is the same in both cases.
If the hash is done in memory, there will only be a single batch. A difference with the original hash batch numbers is due to Postgres choosing to increase the number of batches in order to reduce memory consumption. You might find this EXPLAIN glossary useful (disclaimer: I'm one of the authors), here is the page on Hash Batches which also links to the PostgreSQL source code (it's very nicely documented in plain English). While not a perfect heuristic, you can see that the memory required for the operations with multiple batches are around or above your work_mem setting. They can be lower than it, due to operations on disk generally requiring less memory overall. I'm not 100% sure why in your exact case one was chosen over the other, but it does look like there are some very slight row estimate differences, which might be a good place to start. As of PostgreSQL 13 there is also now a hash_mem_multiplier setting that can be used to give more memory to hashes without doing so for other operations (like sorts).
We where able to solve the problem just by doing VACUUM FULL ANALYZE;. After that, everything started to work as expected (https://explain.depesz.com/s/eoqH#html) Side note: we where not aware that we should do this on daily basis.
Differing PSQL Planners with same Indexes
I have been trying to speed up my psql queries to squeeze out as much speed as possible. With a few indexes I installed on my local system I got good speeds. I installed these on the remote system but had different results. The screenshot for the planners follow: Local Planner: QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------------------------- Hash Join (cost=19.54..67.37 rows=12 width=133) (actual time=0.771..0.862 rows=12 loops=1) Hash Cond: ((sensor_lookup.sensorid)::text = (sensor.sensorid)::text) Buffers: shared hit=25 -> Nested Loop (cost=3.01..50.81 rows=12 width=119) (actual time=0.193..0.271 rows=12 loops=1) Buffers: shared hit=19 -> Nested Loop (cost=2.60..26.10 rows=1 width=320) (actual time=0.163..0.228 rows=1 loops=1) Buffers: shared hit=15 -> Nested Loop (cost=2.60..25.02 rows=1 width=98) (actual time=0.156..0.217 rows=1 loops=1) Buffers: shared hit=14 -> Nested Loop (cost=0.27..13.80 rows=1 width=68) (actual time=0.097..0.151 rows=1 loops=1) Buffers: shared hit=7 -> Index Scan using meta_pkey on meta (cost=0.27..4.29 rows=1 width=45) (actual time=0.029..0.031 rows=1 loops=1) Index Cond: (stationid = 'WYTOR02'::bpchar) Buffers: shared hit=3 -> Seq Scan on meta_lookup (cost=0.00..9.50 rows=1 width=31) (actual time=0.064..0.116 rows=1 loops=1) Filter: ((stationid)::bpchar = 'WYTOR02'::bpchar) Rows Removed by Filter: 439 Buffers: shared hit=4 -> Bitmap Heap Scan on datetime_lookup (cost=2.33..11.21 rows=1 width=38) (actual time=0.054..0.060 rows=1 loops=1) Recheck Cond: (stationid = 'WYTOR02'::bpchar) Filter: ((productid)::text = 'qc60'::text) Rows Removed by Filter: 5 Heap Blocks: exact=5 Buffers: shared hit=7 -> Bitmap Index Scan on idx_16 (cost=0.00..2.32 rows=6 width=0) (actual time=0.033..0.033 rows=6 loops=1) Index Cond: (stationid = 'WYTOR02'::bpchar) Buffers: shared hit=2 -> Seq Scan on product (cost=0.00..1.07 rows=1 width=222) (actual time=0.006..0.008 rows=1 loops=1) Filter: ((productid)::text = 'qc60'::text) Rows Removed by Filter: 5 Buffers: shared hit=1 -> Index Scan using idx_15 on sensor_lookup (cost=0.41..24.59 rows=12 width=30) (actual time=0.027..0.034 rows=12 loops=1) Index Cond: ((stationid = 'WYTOR02'::bpchar) AND ((productid)::text = 'qc60'::text)) Buffers: shared hit=4 -> Hash (cost=10.68..10.68 rows=468 width=27) (actual time=0.547..0.548 rows=468 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 34kB Buffers: shared hit=6 -> Seq Scan on sensor (cost=0.00..10.68 rows=468 width=27) (actual time=0.013..0.208 rows=468 loops=1) Buffers: shared hit=6 Planning time: 1.655 ms Execution time: 1.106 ms Remote Planner: QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------------------------- Hash Join (cost=26.67..102.51 rows=12 width=133) (actual time=0.644..0.719 rows=12 loops=1) Hash Cond: ((sensor_lookup.sensorid)::text = (sensor.sensorid)::text) Buffers: shared hit=29 -> Nested Loop (cost=9.14..84.82 rows=12 width=119) (actual time=0.161..0.227 rows=12 loops=1) Buffers: shared hit=19 -> Nested Loop (cost=4.60..38.12 rows=1 width=108) (actual time=0.128..0.187 rows=1 loops=1) Buffers: shared hit=15 -> Nested Loop (cost=4.60..37.03 rows=1 width=98) (actual time=0.116..0.173 rows=1 loops=1) Buffers: shared hit=14 -> Nested Loop (cost=0.27..17.80 rows=1 width=68) (actual time=0.081..0.132 rows=1 loops=1) Buffers: shared hit=7 -> Index Scan using meta_pkey on meta (cost=0.27..8 .29 rows=1 width=45) (actual time=0.011..0.012 rows=1 loops=1) Index Cond: (stationid = 'WYTOR02'::bpchar) Buffers: shared hit=3 -> Seq Scan on meta_lookup (cost=0.00..9.50 rows=1 width=31) (actual time=0.067..0.117 rows=1 loops=1) Filter: ((stationid)::bpchar = 'WYTOR02'::bpchar) Rows Removed by Filter: 439 Buffers: shared hit=4 -> Bitmap Heap Scan on datetime_lookup (cost=4.33..19.22 rows=1 width=38) (actual time=0.031..0.036 rows=1 loops=1) Recheck Cond: (stationid = 'WYTOR02'::bpchar) Filter: ((productid)::text = 'qc60'::text) Rows Removed by Filter: 5 Heap Blocks: exact=5 Buffers: shared hit=7 -> Bitmap Index Scan on idx_16 (cost=0.00..4.33 rows=6 width=0) (actual time=0.019..0.019 rows=6 loops=1) Index Cond: (stationid = 'WYTOR02'::bpchar) Buffers: shared hit=2 -> Seq Scan on product (cost=0.00..1.07 rows=1 width=10) (actual time=0.010..0.012 rows=1 loops=1) Filter: ((productid)::text = 'qc60'::text) Rows Removed by Filter: 5 Buffers: shared hit=1 -> Bitmap Heap Scan on sensor_lookup (cost=4.54..46.58 rows=12 width=30) (actual time=0.030..0.032 rows=12 loops=1) Recheck Cond: ((stationid = 'WYTOR02'::bpchar) AND ((productid)::text = 'qc60'::text)) Heap Blocks: exact=1 Buffers: shared hit=4 -> Bitmap Index Scan on idx_15 (cost=0.00..4.54 rows=12 width=0) (actual time=0.021..0.021 rows=12 loops=1) Index Cond: ((stationid = 'WYTOR02'::bpchar) AND ((productid)::text = 'qc60'::text)) Buffers: shared hit=3 -> Hash (cost=11.68..11.68 rows=468 width=27) (actual time=0.440..0.440 rows=468 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 34kB Buffers: shared hit=7 -> Seq Scan on sensor (cost=0.00..11.68 rows=468 width=27) (actual time=0.004..0.174 rows=468 loops=1) Buffers: shared hit=7 Planning time: 2.572 ms Execution time: 0.947 ms Even though the difference is 1ms, these calls are done thousands of time so the difference adds up. The difference seems to ne that the reomote is doing a Bitmap Heap Scan as opposed to an Index Scan. Though I'm not sure these differences account for the planning time it is a difference between matching systems. The settings in the postgresql.conf are the same so what can I look at to see why these are different? Both the local and remote servers have the same Postgresql and Ubuntu versions: Ubuntu 18.04.1 psql (PostgreSQL) 10.15 (Ubuntu 10.15-0ubuntu0.18.04.1)
Query is slow with indexes - how to understand execution plans?
I need help to understand why my query is slower when I use index than without any index. I ran explain analyze command, and below are execution plans option 1 - with index, and option 2 - without index.
Can someone explain to me why index makes performances worse in those execution plans?
PS. When I add 10 million rows to table (original size 2M), situation is turning in favor of index, and in that case query with index is 3x faster).
OPTION 1 WITH INDEX FOR LEFT JOIN invoice_id+acct_level ON TABLE cost_invoice_facepage AND CONDITION (cdb.invoice_id = invoice_id) AND (acct_level = 1)
Append (cost=48.87..38583.97 rows=163773 width=371) (actual time=1.269..1516.564 rows=379129 loops=1)
-> Nested Loop (cost=48.87..10520.11 rows=36504 width=362) (actual time=1.268..5.986 rows=579 loops=1)
-> Hash Left Join (cost=44.66..9918.22 rows=507 width=322) (actual time=1.160..5.497 rows=579 loops=1)
Hash Cond: (cd.gl_string_id = gs.id)
-> Nested Loop Left Join (cost=0.85..9873.07 rows=507 width=262) (actual time=0.485..4.473 rows=579 loops=1)
Filter: ((c.gl_rule_type IS NULL) OR ((cd.charge_id IS NOT NULL) AND (c.gl_rule_type_id <> ALL ('{60,70}'::integer[]))))
-> Index Scan using cost_invoice_charge_invoice_id_idx on cost_invoice_charge c (cost=0.43..1204.53 rows=1188 width=243) (actual time=0.467..2.664 rows=579 loops=1)
Index Cond: (invoice_id = 14517)
Filter: ((chg_amt <> '0'::numeric) AND ((gl_rule_type IS NULL) OR (gl_rule_type_id <> ALL ('{60,70}'::integer[]))))
Rows Removed by Filter: 3364
-> Index Scan using "gl_charge_detail.charge_id->cost_invoice_info_only.id" on gl_charge_detail cd (cost=0.42..7.28 rows=1 width=27) (actual time=0.002..0.002 rows=1 loops=579)
Index Cond: (c.id = charge_id)
-> Hash (cost=31.69..31.69 rows=969 width=64) (actual time=0.657..0.657 rows=969 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 103kB
-> Seq Scan on gl_strings gs (cost=0.00..31.69 rows=969 width=64) (actual time=0.026..0.389 rows=969 loops=1)
-> Materialize (cost=4.22..145.78 rows=72 width=44) (actual time=0.000..0.000 rows=1 loops=579)
-> Hash Left Join (cost=4.22..145.42 rows=72 width=44) (actual time=0.100..0.102 rows=1 loops=1)
Hash Cond: (f.vendor_id = vn.id)
-> Nested Loop (cost=0.57..141.57 rows=72 width=31) (actual time=0.027..0.029 rows=1 loops=1)
-> Index Scan using cost_invoice_header_id_idx on cost_invoice_header ch (cost=0.29..8.31 rows=1 width=4) (actual time=0.012..0.013 rows=1 loops=1)
Index Cond: (id = 14517)
Filter: (status_code <> ALL ('{100,101,102,490}'::integer[]))
-> Index Scan using "invoice_id+acct_level" on cost_invoice_facepage f (cost=0.29..132.55 rows=72 width=31) (actual time=0.013..0.013 rows=1 loops=1)
Index Cond: ((invoice_id = 14517) AND (acct_level = 1))
-> Hash (cost=2.73..2.73 rows=73 width=17) (actual time=0.061..0.061 rows=73 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on appdata_vendor_common vn (cost=0.00..2.73 rows=73 width=17) (actual time=0.020..0.038 rows=73 loops=1)
-> Hash Left Join (cost=2276.48..25607.26 rows=127269 width=374) (actual time=204.117..1486.717 rows=378550 loops=1)
Hash Cond: (f_1.vendor_id = vn_1.id)
-> Nested Loop Left Join (cost=2272.84..25250.14 rows=127269 width=361) (actual time=204.072..1328.491 rows=378550 loops=1)
-> Gather (cost=2272.55..24385.32 rows=2663 width=338) (actual time=204.055..335.965 rows=378550 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Hash Left Join (cost=1272.55..23119.02 rows=1110 width=338) (actual time=127.365..321.126 rows=126183 loops=3)
Hash Cond: (cdb.gl_string_id = gs_1.id)
-> Hash Join (cost=1228.74..23072.30 rows=1110 width=278) (actual time=126.126..263.315 rows=126183 loops=3)
Hash Cond: (cdb.charge_id = c_1.id)
-> Parallel Seq Scan on gl_charge_detail_ban cdb (cost=0.00..20581.15 rows=480915 width=43) (actual time=0.270..109.543 rows=384732 loops=3)
-> Hash (cost=1194.13..1194.13 rows=2769 width=239) (actual time=7.232..7.232 rows=3929 loops=3)
Buckets: 4096 Batches: 1 Memory Usage: 635kB
-> Index Scan using cost_invoice_charge_invoice_id_idx on cost_invoice_charge c_1 (cost=0.43..1194.13 rows=2769 width=239) (actual time=0.070..4.686 rows=3929 loops=3)
Index Cond: (invoice_id = 14517)
Filter: (chg_amt <> '0'::numeric)
Rows Removed by Filter: 14
-> Hash (cost=31.69..31.69 rows=969 width=64) (actual time=1.127..1.127 rows=969 loops=3)
Buckets: 1024 Batches: 1 Memory Usage: 103kB
-> Seq Scan on gl_strings gs_1 (cost=0.00..31.69 rows=969 width=64) (actual time=0.165..0.714 rows=969 loops=3)
-> Index Scan using "invoice_id+acct_level" on cost_invoice_facepage f_1 (cost=0.29..0.31 rows=1 width=31) (actual time=0.001..0.002 rows=1 loops=378550)
Index Cond: ((cdb.invoice_id = invoice_id) AND (acct_level = 1))
-> Hash (cost=2.73..2.73 rows=73 width=17) (actual time=0.035..0.035 rows=73 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on appdata_vendor_common vn_1 (cost=0.00..2.73 rows=73 width=17) (actual time=0.014..0.021 rows=73 loops=1)
Planning Time: 3.636 ms
Execution Time: 1550.844 ms
and
OPTION 2 WITHOUT INDEXES
Append (cost=48.58..43257.20 rows=163773 width=371) (actual time=7.965..831.408 rows=379129 loops=1)
-> Nested Loop (cost=48.58..12251.68 rows=36504 width=362) (actual time=7.965..14.476 rows=579 loops=1)
-> Hash Left Join (cost=44.66..9918.22 rows=507 width=322) (actual time=0.588..6.245 rows=579 loops=1)
Hash Cond: (cd.gl_string_id = gs.id)
-> Nested Loop Left Join (cost=0.85..9873.07 rows=507 width=262) (actual time=0.245..5.442 rows=579 loops=1)
Filter: ((c.gl_rule_type IS NULL) OR ((cd.charge_id IS NOT NULL) AND (c.gl_rule_type_id <> ALL ('{60,70}'::integer[]))))
-> Index Scan using cost_invoice_charge_invoice_id_idx on cost_invoice_charge c (cost=0.43..1204.53 rows=1188 width=243) (actual time=0.231..3.003 rows=579 loops=1)
Index Cond: (invoice_id = 14517)
Filter: ((chg_amt <> '0'::numeric) AND ((gl_rule_type IS NULL) OR (gl_rule_type_id <> ALL ('{60,70}'::integer[]))))
Rows Removed by Filter: 3364
-> Index Scan using "gl_charge_detail.charge_id->cost_invoice_info_only.id" on gl_charge_detail cd (cost=0.42..7.28 rows=1 width=27) (actual time=0.003..0.003 rows=1 loops=579)
Index Cond: (c.id = charge_id)
-> Hash (cost=31.69..31.69 rows=969 width=64) (actual time=0.331..0.331 rows=969 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 103kB
-> Seq Scan on gl_strings gs (cost=0.00..31.69 rows=969 width=64) (actual time=0.017..0.183 rows=969 loops=1)
-> Materialize (cost=3.93..1877.35 rows=72 width=44) (actual time=0.013..0.013 rows=1 loops=579)
-> Hash Left Join (cost=3.93..1876.99 rows=72 width=44) (actual time=7.370..7.698 rows=1 loops=1)
Hash Cond: (f.vendor_id = vn.id)
-> Nested Loop (cost=0.29..1873.14 rows=72 width=31) (actual time=7.307..7.635 rows=1 loops=1)
-> Index Scan using cost_invoice_header_id_idx on cost_invoice_header ch (cost=0.29..8.31 rows=1 width=4) (actual time=0.011..0.013 rows=1 loops=1)
Index Cond: (id = 14517)
Filter: (status_code <> ALL ('{100,101,102,490}'::integer[]))
-> Seq Scan on cost_invoice_facepage f (cost=0.00..1864.12 rows=72 width=31) (actual time=7.293..7.619 rows=1 loops=1)
Filter: ((invoice_id = 14517) AND (acct_level = 1))
Rows Removed by Filter: 40340
-> Hash (cost=2.73..2.73 rows=73 width=17) (actual time=0.045..0.045 rows=73 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on appdata_vendor_common vn (cost=0.00..2.73 rows=73 width=17) (actual time=0.022..0.028 rows=73 loops=1)
-> Hash Left Join (cost=4248.29..28548.92 rows=127269 width=374) (actual time=234.692..789.334 rows=378550 loops=1)
Hash Cond: (cdb.invoice_id = f_1.invoice_id)
-> Gather (cost=2272.55..24385.32 rows=2663 width=338) (actual time=216.507..376.349 rows=378550 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Hash Left Join (cost=1272.55..23119.02 rows=1110 width=338) (actual time=128.932..389.669 rows=126183 loops=3)
Hash Cond: (cdb.gl_string_id = gs_1.id)
-> Hash Join (cost=1228.74..23072.30 rows=1110 width=278) (actual time=127.984..308.092 rows=126183 loops=3)
Hash Cond: (cdb.charge_id = c_1.id)
-> Parallel Seq Scan on gl_charge_detail_ban cdb (cost=0.00..20581.15 rows=480915 width=43) (actual time=0.163..117.001 rows=384732 loops=3)
-> Hash (cost=1194.13..1194.13 rows=2769 width=239) (actual time=8.779..8.779 rows=3929 loops=3)
Buckets: 4096 Batches: 1 Memory Usage: 635kB
-> Index Scan using cost_invoice_charge_invoice_id_idx on cost_invoice_charge c_1 (cost=0.43..1194.13 rows=2769 width=239) (actual time=0.050..5.563 rows=3929 loops=3)
Index Cond: (invoice_id = 14517)
Filter: (chg_amt <> '0'::numeric)
Rows Removed by Filter: 14
-> Hash (cost=31.69..31.69 rows=969 width=64) (actual time=0.829..0.829 rows=969 loops=3)
Buckets: 1024 Batches: 1 Memory Usage: 103kB
-> Seq Scan on gl_strings gs_1 (cost=0.00..31.69 rows=969 width=64) (actual time=0.184..0.534 rows=969 loops=3)
-> Hash (cost=1804.87..1804.87 rows=13670 width=44) (actual time=18.101..18.101 rows=13705 loops=1)
Buckets: 16384 Batches: 1 Memory Usage: 1198kB
-> Hash Left Join (cost=3.64..1804.87 rows=13670 width=44) (actual time=0.075..14.009 rows=13705 loops=1)
Hash Cond: (f_1.vendor_id = vn_1.id)
-> Seq Scan on cost_invoice_facepage f_1 (cost=0.00..1763.26 rows=13670 width=31) (actual time=0.017..6.216 rows=13705 loops=1)
Filter: (acct_level = 1)
Rows Removed by Filter: 26636
-> Hash (cost=2.73..2.73 rows=73 width=17) (actual time=0.052..0.052 rows=73 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on appdata_vendor_common vn_1 (cost=0.00..2.73 rows=73 width=17) (actual time=0.013..0.027 rows=73 loops=1)
Planning Time: 3.365 ms
Execution Time: 863.941 ms
Look at the line that is driving the iteration over the index scan: Gather (cost=2272.55..24385.32 rows=2663 width=338) (actual time=204.055..335.965 rows=378550 loops=1) It thinks the index scan will get iterated 2663 times (with a different value of invoice_id for each one) but it really gets iterated 378550 times, (this latter number is where the 'loops' field on the index scan comes from), a difference of 140 fold. Every time you hit the index, you need to re-descend from the root to the leaf, locking and unlocking pages as you go. While this is not terribly expensive, it does add up if you do it 378550 times. It gets to be faster to process the table in bulk into a private hash table. But since the estimated row count is so wrong, PostgreSQL doesn't realize that in this case.
Very slow SELECT from partitioned table in PostgreSQL
I use Postgresql 9.5. The settings are default. I have split a table into ~7000 partitions. Then I've inserted one row. When I query SELECT * FROM "Offer";, it is running 1,5 seconds. When I query SELECT * FROM "Offer" WHERE bid=4793;, where bid -- partition's constraint (one table per bid), it is running 1 second. Here is an EXPLAIN ANALYZE for second query: Append (cost=0.00..12.14 rows=2 width=596) (actual time=0.014..0.014 rows=1 loops=1) -> Seq Scan on "Offer" (cost=0.00..1.01 rows=1 width=344) (actual time=0.011..0.011 rows=0 loops=1) Filter: (bid = 4793) Rows Removed by Filter: 1 -> Seq Scan on "Offer-4793" (cost=0.00..11.12 rows=1 width=848) (actual time=0.002..0.002 rows=1 loops=1) Filter: (bid = 4793) Planning time: 996.243 ms Execution time: 0.261 ms Why so slow? What can I use to profile it? I have only one guess -- postgresql does not keep partitioning constrains in RAM and reads them from HDD every time. Expecting some help! UPDATE: I've tried to create cascading partitioning (as #jmelesky has written). Results are worse: Append (cost=0.00..12.24 rows=5 width=848) (actual time=0.013..0.013 rows=1 loops=1) -> Seq Scan on "Offer" (cost=0.00..1.11 rows=1 width=848) (actual time=0.006..0.006 rows=0 loops=1) Filter: (bid = 4793) -> Seq Scan on "Offer-ddd-3" (cost=0.00..0.00 rows=1 width=848) (actual time=0.001..0.001 rows=0 loops=1) Filter: (bid = 4793) -> Seq Scan on "Offer-dd-33" (cost=0.00..0.00 rows=1 width=848) (actual time=0.000..0.000 rows=0 loops=1) Filter: (bid = 4793) -> Seq Scan on "Offer-d-336" (cost=0.00..0.00 rows=1 width=848) (actual time=0.000..0.000 rows=0 loops=1) Filter: (bid = 4793) -> Seq Scan on "Offer-4793" (cost=0.00..11.12 rows=1 width=848) (actual time=0.006..0.006 rows=1 loops=1) Filter: (bid = 4793) Planning time: 1449.872 ms Execution time: 0.354 ms