I am using PostgreSQL 12.5, I have a partitioned table tbl_message with some columns and I have a query to fetch data from that table
SELECT m.* FROM tbl_message m
WHERE m.hash IS NOT NULL AND m.involved_accounts #> array['account_id']::text[]
ORDER BY m.col1 DESC, m.col2 DESC, m.col3 DESC
LIMIT 25 OFFSET 0
About the columns
hash is NULLABLE TEXT
involved_accounts is TEXT[]
col1 is BIGINT NOT NULL
col2 is BIGINT NOT NULL
col3 is SMALLINT NOT NULL
I have indexes:
CREATE INDEX message_hash_index ON tbl_message(hash); -- for the first filter
CREATE INDEX message_involved_accounts_index ON tbl_message USING GIN(involved_accounts); -- for the second filter
CREATE INDEX message_order_index ON tbl_message(col1 desc, col2 desc, col3 desc); -- for the order
-- each of col1, col2, col3 also has their own index
The table at this moment has above 41 million records at time of wring and will keep increasing by ~12k records per day.
So when I perform the query which I wrote above, most of the query returns result after few milliseconds which is nice but with some account_id it takes 2-5 minutes and yes, it depends on the input param account_id even tho the account_id has the exactly same format and same length.
Some Query Plan:
16 secs to count
explain analyze select count(1) from message where hash is not null;
Finalize Aggregate (cost=7711827.60..7711827.61 rows=1 width=8) (actual time=13175.393..15703.182 rows=1 loops=1)
-> Gather (cost=7711827.18..7711827.59 rows=4 width=8) (actual time=13173.672..15703.160 rows=5 loops=1)
Workers Planned: 4
Workers Launched: 4
-> Partial Aggregate (cost=7710827.18..7710827.19 rows=1 width=8) (actual time=13150.783..13150.835 rows=1 loops=5)
-> Parallel Append (cost=0.56..7684791.29 rows=10414354 width=0) (actual time=187.908..12815.261 rows=8326720 loops=5)
-> Parallel Bitmap Heap Scan on message_16 (cost=904.65..47077.65 rows=17956 width=0) (actual time=208.622..330.643 rows=70319 loops=1)
Recheck Cond: (hash IS NOT NULL)
-> Bitmap Index Scan on message_16_hash_idx (cost=0.00..886.70 rows=71823 width=0) (actual time=19.628..19.628 rows=75445 loops=1)
Index Cond: (hash IS NOT NULL)
-> Parallel Index Only Scan using message_1_hash_idx on message_1 (cost=0.56..2390108.45 rows=3461250 width=0) (actual time=37.516..4434.148 rows=2768141 loops=5)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 7548376
-> Parallel Index Only Scan using message_2_hash_idx on message_2 (cost=0.56..2322145.34 rows=3050950 width=0) (actual time=68.249..5701.835 rows=4065667 loops=3)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 7338697
-> Parallel Index Only Scan using message_0_hash_idx on message_0 (cost=0.56..1693903.33 rows=2472619 width=0) (actual time=93.264..6489.499 rows=4946515 loops=2)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 5398655
-> Parallel Index Only Scan using message_3_hash_idx on message_3 (cost=0.56..1028833.21 rows=1171119 width=0) (actual time=0.101..4072.223 rows=2332674 loops=2)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 3156901
-> Parallel Index Only Scan using message_7_hash_idx on message_7 (cost=0.42..49070.09 rows=44645 width=0) (actual time=0.043..171.102 rows=138257 loops=1)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 88474
-> Parallel Index Only Scan using message_5_hash_idx on message_5 (cost=0.42..29203.74 rows=29492 width=0) (actual time=0.039..77.370 rows=70787 loops=1)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 42342
-> Parallel Index Only Scan using message_8_hash_idx on message_8 (cost=0.42..10828.02 rows=34819 width=0) (actual time=0.031..47.104 rows=84533 loops=1)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 16210
-> Parallel Index Only Scan using message_14_hash_idx on message_14 (cost=0.42..1439.52 rows=38392 width=0) (actual time=0.021..24.691 rows=93447 loops=1)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 0
-> Parallel Index Only Scan using message_13_hash_idx on message_13 (cost=0.42..1281.84 rows=34058 width=0) (actual time=0.022..18.290 rows=82294 loops=1)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 0
-> Parallel Index Only Scan using message_11_hash_idx on message_11 (cost=0.42..1236.68 rows=33152 width=0) (actual time=0.018..16.634 rows=79022 loops=1)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 0
-> Parallel Index Only Scan using message_12_hash_idx on message_12 (cost=0.42..1218.59 rows=32624 width=0) (actual time=0.022..16.128 rows=77633 loops=1)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 0
-> Parallel Index Only Scan using message_15_hash_idx on message_15 (cost=0.42..1184.22 rows=30335 width=0) (actual time=0.018..18.168 rows=76143 loops=1)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 0
-> Parallel Index Only Scan using message_9_hash_idx on message_9 (cost=0.42..1108.65 rows=29601 width=0) (actual time=0.026..14.513 rows=71693 loops=1)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 0
-> Parallel Index Only Scan using message_10_hash_idx on message_10 (cost=0.42..1072.47 rows=28584 width=0) (actual time=0.065..13.795 rows=68969 loops=1)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 0
-> Parallel Index Only Scan using message_6_hash_idx on message_6 (cost=0.42..27106.57 rows=26636 width=0) (actual time=0.055..71.713 rows=63436 loops=1)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 39933
-> Parallel Index Only Scan using message_4_hash_idx on message_4 (cost=0.42..25901.16 rows=25408 width=0) (actual time=153.397..217.151 rows=60984 loops=1)
Index Cond: (hash IS NOT NULL)
Heap Fetches: 37519
Planning Time: 1.106 ms
JIT:
Functions: 187
Options: Inlining true, Optimization true, Expressions true, Deforming true
Timing: Generation 4.864 ms, Inlining 168.951 ms, Optimization 400.175 ms, Emission 346.037 ms, Total 920.027 ms
Execution Time: 15704.360 ms
The query took 350s with account_id let say account_id_1
EXPLAIN ANALYZE SELECT m.* FROM message AS m
WHERE ((m.hash IS NOT NULL)) AND m.involved_accounts #> array['account_id_1']::text[]
ORDER BY m.col1 DESC, m.col2 DESC, m.col3 DESC
LIMIT 25 OFFSET 0;
Limit (cost=7.60..1047.21 rows=25 width=385) (actual time=336357.537..347776.125 rows=3 loops=1)
-> Merge Append (cost=7.60..13086668.58 rows=314704 width=385) (actual time=336357.535..347776.119 rows=3 loops=1)
Sort Key: m.col1 DESC, m.col2 DESC, m.col3 DESC
-> Index Scan using message_0_col1_col2_col3_idx on message_0 m (cost=0.43..2833000.49 rows=11544 width=387) (actual time=58029.607..58029.607 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 10423545
-> Index Scan using message_1_col1_col2_col3_idx on message_1 m_1 (cost=0.43..4014646.75 rows=47555 width=386) (actual time=92869.149..92869.149 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 14668913
-> Index Scan using message_2_col1_col2_col3_idx on message_2 m_2 (cost=0.43..3655774.25 rows=115975 width=384) (actual time=85838.196..85838.197 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 13940182
-> Index Scan using message_3_col1_col2_col3_idx on message_3 m_3 (cost=0.43..1481300.56 rows=64980 width=383) (actual time=40045.022..40045.022 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 6086597
-> Index Scan using message_4_col1_col2_col3_idx on message_4 m_4 (cost=0.42..61507.63 rows=4762 width=388) (actual time=3343.082..3343.082 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 490989
-> Index Scan using message_5_col1_col2_col3_idx on message_5 m_5 (cost=0.42..67689.29 rows=5429 width=388) (actual time=3268.888..3268.889 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 532991
-> Index Scan using message_6_col1_col2_col3_idx on message_6 m_6 (cost=0.42..57305.88 rows=5145 width=391) (actual time=2823.064..2823.064 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 437871
-> Index Scan using message_7_col1_col2_col3_idx on message_7 m_7 (cost=0.42..83700.77 rows=9928 width=400) (actual time=3580.900..3580.901 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 571980
-> Index Scan using message_8_col1_col2_col3_idx on message_8 m_8 (cost=0.42..81686.36 rows=7241 width=388) (actual time=2135.704..5567.318 rows=1 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 654258
-> Index Scan using message_9_col1_col2_col3_idx on message_9 m_9 (cost=0.42..88054.55 rows=5908 width=384) (actual time=6070.489..6070.489 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 784701
-> Index Scan using message_10_col1_col2_col3_idx on message_10 m_10 (cost=0.42..81750.93 rows=5481 width=384) (actual time=2397.042..5914.743 rows=1 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 727217
-> Index Scan using message_11_col1_col2_col3_idx on message_11 m_11 (cost=0.42..101345.12 rows=5739 width=381) (actual time=2499.117..6968.372 rows=1 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 921600
-> Index Scan using message_12_col1_col2_col3_idx on message_12 m_12 (cost=0.42..97795.14 rows=5269 width=382) (actual time=7395.400..7395.400 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 884379
-> Index Scan using message_13_col1_col2_col3_idx on message_13 m_13 (cost=0.42..93591.47 rows=5588 width=385) (actual time=6870.456..6870.456 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 826472
-> Index Scan using message_14_col1_col2_col3_idx on message_14 m_14 (cost=0.42..110115.43 rows=5559 width=381) (actual time=8402.985..8402.985 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 994695
-> Index Scan using message_15_col1_col2_col3_idx on message_15 m_15 (cost=0.42..88758.30 rows=4254 width=381) (actual time=6204.999..6204.999 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 788500
-> Index Scan using message_16_col1_col2_col3_idx on message_16 m_16 (cost=0.42..80640.09 rows=4347 width=385) (actual time=4583.407..4583.407 rows=0 loops=1)
Filter: ((hash IS NOT NULL) AND (involved_accounts #> '{account_id_1}'::text[]))
Rows Removed by Filter: 655792
Planning Time: 15.418 ms
Execution Time: 347776.246 ms
The query took only 33ms to execute when I changed the account id. You can see the query plan actually changed
Limit (cost=11574.91..11574.98 rows=25 width=386) (actual time=33.602..33.616 rows=25 loops=1)
-> Sort (cost=11574.91..11590.99 rows=6431 width=386) (actual time=33.600..33.612 rows=25 loops=1)
Sort Key: m.col1 DESC, m.col2 DESC, m.col3 DESC
Sort Method: quicksort Memory: 45kB
-> Append (cost=174.56..11393.43 rows=6431 width=386) (actual time=2.111..33.577 rows=32 loops=1)
-> Bitmap Heap Scan on message_0 m (cost=174.56..2492.36 rows=1981 width=387) (actual time=2.111..2.126 rows=4 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 1
Heap Blocks: exact=5
-> Bitmap Index Scan on message_0_involved_accounts_idx (cost=0.00..174.07 rows=2089 width=0) (actual time=2.095..2.096 rows=5 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_1 m_1 (cost=436.76..2339.70 rows=1618 width=386) (actual time=4.851..4.852 rows=1 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Heap Blocks: exact=1
-> Bitmap Index Scan on message_1_involved_accounts_idx (cost=0.00..436.35 rows=1714 width=0) (actual time=4.845..4.845 rows=1 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_2 m_2 (cost=272.40..3376.68 rows=2445 width=384) (actual time=3.159..3.231 rows=11 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 36
Heap Blocks: exact=43
-> Bitmap Index Scan on message_2_involved_accounts_idx (cost=0.00..271.79 rows=2799 width=0) (actual time=3.082..3.083 rows=47 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_3 m_3 (cost=307.83..759.88 rows=313 width=383) (actual time=3.596..3.596 rows=0 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 35
Heap Blocks: exact=24
-> Bitmap Index Scan on message_3_involved_accounts_idx (cost=0.00..307.75 rows=407 width=0) (actual time=3.509..3.509 rows=37 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_4 m_4 (cost=244.45..281.07 rows=4 width=388) (actual time=3.003..3.003 rows=0 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 14
Heap Blocks: exact=10
-> Bitmap Index Scan on message_4_involved_accounts_idx (cost=0.00..244.45 rows=33 width=0) (actual time=2.962..2.962 rows=14 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_5 m_5 (cost=469.97..509.92 rows=5 width=388) (actual time=5.837..5.868 rows=4 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 9
Heap Blocks: exact=12
-> Bitmap Index Scan on message_5_involved_accounts_idx (cost=0.00..469.97 rows=36 width=0) (actual time=5.818..5.818 rows=13 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_6 m_6 (cost=157.57..197.52 rows=5 width=391) (actual time=1.789..1.797 rows=1 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 13
Heap Blocks: exact=11
-> Bitmap Index Scan on message_6_involved_accounts_idx (cost=0.00..157.57 rows=36 width=0) (actual time=1.754..1.754 rows=14 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_7 m_7 (cost=185.16..238.43 rows=12 width=400) (actual time=2.097..2.105 rows=1 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 9
Heap Blocks: exact=6
-> Bitmap Index Scan on message_7_involved_accounts_idx (cost=0.00..185.16 rows=48 width=0) (actual time=2.076..2.077 rows=10 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_8 m_8 (cost=193.93..242.76 rows=6 width=388) (actual time=2.177..2.198 rows=3 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 19
Heap Blocks: exact=17
-> Bitmap Index Scan on message_8_involved_accounts_idx (cost=0.00..193.93 rows=44 width=0) (actual time=2.134..2.134 rows=22 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_9 m_9 (cost=4.79..62.50 rows=5 width=384) (actual time=0.073..0.081 rows=2 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 20
Heap Blocks: exact=15
-> Bitmap Index Scan on message_9_involved_accounts_idx (cost=0.00..4.79 rows=52 width=0) (actual time=0.027..0.027 rows=22 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_10 m_10 (cost=4.86..72.53 rows=6 width=384) (actual time=0.055..0.068 rows=1 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 18
Heap Blocks: exact=14
-> Bitmap Index Scan on message_10_involved_accounts_idx (cost=0.00..4.85 rows=61 width=0) (actual time=0.018..0.018 rows=19 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_11 m_11 (cost=4.86..72.55 rows=5 width=381) (actual time=0.039..0.049 rows=1 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 12
Heap Blocks: exact=8
-> Bitmap Index Scan on message_11_involved_accounts_idx (cost=0.00..4.86 rows=61 width=0) (actual time=0.018..0.018 rows=13 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_12 m_12 (cost=4.84..70.32 rows=5 width=382) (actual time=0.041..0.041 rows=0 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 9
Heap Blocks: exact=5
-> Bitmap Index Scan on message_12_involved_accounts_idx (cost=0.00..4.84 rows=59 width=0) (actual time=0.020..0.020 rows=9 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_13 m_13 (cost=4.81..65.85 rows=5 width=385) (actual time=0.022..0.022 rows=0 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 2
Heap Blocks: exact=1
-> Bitmap Index Scan on message_13_involved_accounts_idx (cost=0.00..4.81 rows=55 width=0) (actual time=0.017..0.017 rows=2 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_14 m_14 (cost=4.90..78.14 rows=6 width=381) (actual time=0.031..0.046 rows=3 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 7
Heap Blocks: exact=8
-> Bitmap Index Scan on message_14_involved_accounts_idx (cost=0.00..4.90 rows=66 width=0) (actual time=0.017..0.017 rows=10 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_15 m_15 (cost=4.80..63.61 rows=5 width=381) (actual time=0.037..0.037 rows=0 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 7
Heap Blocks: exact=5
-> Bitmap Index Scan on message_15_involved_accounts_idx (cost=0.00..4.79 rows=53 width=0) (actual time=0.017..0.018 rows=7 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
-> Bitmap Heap Scan on message_16 m_16 (cost=388.63..437.46 rows=5 width=385) (actual time=4.446..4.446 rows=0 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_2}'::text[])
Filter: (hash IS NOT NULL)
Rows Removed by Filter: 3
Heap Blocks: exact=3
-> Bitmap Index Scan on message_16_involved_accounts_idx (cost=0.00..388.63 rows=44 width=0) (actual time=4.434..4.434 rows=3 loops=1)
Index Cond: (involved_accounts #> '{account_id_2}'::text[])
Planning Time: 2.543 ms
Execution Time: 33.921 ms
I also noticed the faster plan is only use Index Condition of involved_accounts and then filter the hash is not null later. I removed the condition hash is not null from query and it runs really fast.
Limit (cost=887468.62..887471.62 rows=25 width=419) (actual time=6709.798..7131.047 rows=25 loops=1)
-> Gather Merge (cost=887468.62..999118.78 rows=932480 width=419) (actual time=5084.627..5505.872 rows=25 loops=1)
Workers Planned: 4
Workers Launched: 4
-> Sort (cost=886468.57..887051.37 rows=233120 width=419) (actual time=5050.402..5050.438 rows=20 loops=5)
Sort Key: m_2.col123 DESC
Sort Method: top-N heapsort Memory: 47kB
Worker 0: Sort Method: top-N heapsort Memory: 50kB
Worker 1: Sort Method: top-N heapsort Memory: 47kB
Worker 2: Sort Method: top-N heapsort Memory: 44kB
Worker 3: Sort Method: top-N heapsort Memory: 46kB
-> Parallel Append (cost=551.27..879890.09 rows=233120 width=419) (actual time=1332.139..4917.573 rows=189949 loops=5)
-> Parallel Bitmap Heap Scan on message_2 m_2 (cost=991.69..132230.25 rows=30251 width=418) (actual time=859.916..2407.663 rows=65051 loops=2)
Recheck Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Bitmap Index Scan on message_2_involved_accounts_idx (cost=0.00..961.44 rows=121005 width=0) (actual time=39.265..39.266 rows=130102 loops=1)
Index Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Parallel Bitmap Heap Scan on message_16 m_16 (cost=809.64..36172.66 rows=9759 width=418) (actual time=1657.269..1890.565 rows=40724 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Bitmap Index Scan on message_16_involved_accounts_idx (cost=0.00..799.88 rows=39037 width=0) (actual time=24.626..24.626 rows=40724 loops=1)
Index Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Parallel Bitmap Heap Scan on message_3 m_3 (cost=745.03..97514.37 rows=22720 width=416) (actual time=1642.602..4089.925 rows=90358 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Bitmap Index Scan on message_3_involved_accounts_idx (cost=0.00..722.31 rows=90881 width=0) (actual time=23.948..23.949 rows=90358 loops=1)
Index Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Parallel Bitmap Heap Scan on message_9 m_9 (cost=551.27..56916.12 rows=16790 width=418) (actual time=1634.792..1905.398 rows=66960 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Bitmap Index Scan on message_9_involved_accounts_idx (cost=0.00..534.48 rows=67158 width=0) (actual time=18.669..18.669 rows=66960 loops=1)
Index Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Parallel Bitmap Heap Scan on message_11 m_11 (cost=546.24..58071.66 rows=16627 width=415) (actual time=15.659..1026.270 rows=66403 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Bitmap Index Scan on message_11_involved_accounts_idx (cost=0.00..529.61 rows=66508 width=0) (actual time=8.589..8.590 rows=66403 loops=1)
Index Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Parallel Bitmap Heap Scan on message_14 m_14 (cost=492.64..54205.42 rows=15004 width=415) (actual time=21.116..1559.622 rows=60123 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Bitmap Index Scan on message_14_involved_accounts_idx (cost=0.00..477.64 rows=60018 width=0) (actual time=13.919..13.919 rows=60123 loops=1)
Index Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Parallel Bitmap Heap Scan on message_12 m_12 (cost=463.37..50438.38 rows=14096 width=416) (actual time=18.859..1621.024 rows=57992 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Bitmap Index Scan on message_12_involved_accounts_idx (cost=0.00..449.27 rows=56383 width=0) (actual time=12.007..12.008 rows=57992 loops=1)
Index Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Parallel Bitmap Heap Scan on message_13 m_13 (cost=460.58..49813.39 rows=14006 width=419) (actual time=20.554..1259.837 rows=54899 loops=1)
Recheck Cond: (involved_accounts #> '{account_id_1}'::text[])
-> Bitmap Index Scan on message_13_involved_accounts_idx (cost=0.00..446.57 rows=56023 width=0) (actual time=13.220..13.220 rows=54899 loops=1)
Index Cond: (involved_accounts #> '{account_id_1}'::text[])
Planning Time: 21.657 ms
JIT:
Functions: 342
Options: Inlining true, Optimization true, Expressions true, Deforming true
Timing: Generation 36.420 ms, Inlining 220.761 ms, Optimization 5049.366 ms, Emission 2849.535 ms, Total 8156.083 ms
Execution Time: 7157.161 ms
Due to limitation in characters of SOF, I remove some lines when scanning partitioned table. The message table currently has 16 sub-tables
Thank you for your time
I don't know what you meant by "Expect 3 rows found, actual 25 rows", that sounds like your middle plan (only backwards), not one of simpler ones I asked for and without the LIMIT.
But based on the three plans you have shown, I think I can put together the picture. The vast majority of rows have hash IS NOT NULL. But the specific case of rows associated with 'account_id_1', the vast majority have hash IS NULL. This leads to poor row estimates, which in turn leads to a poor plan.
This is the kind of thing that could generally be fixed by extended statistics covering both hash and involved_accounts, but not in this case. IS NOT NULL does not refer to a specific value, so can't be covered by the extended statistics. And involved_accounts is an array column, and those also are not well covered by extended statistics.
You could use extended statistics on an expression, introduced in v14:
create statistics h_and_ia on (hash is not null and involved_accounts#>ARRAY['account_id_1']) from j;
But the problem is that that extended statistics only gets picked if you write the query in a rather unnatural way:
WHERE (hash is not null and involved_accounts#>ARRAY['account_id_1']) IS TRUE
And if you are going to write it in an unnatural way, you might as well target (for disabling) the ORDER BY index specifically, by changing the order by:
ORDER BY m.col1+0 DESC, m.col2 DESC, m.col3 DESC
Or of course drop that index, if you doesn't serve any other purpose.
Related
I have a query with joins to rather large tables, but do not understand the slow performance of it.
Especially this part of the query plan seems weird to me (complete plan and query below):
-> Bitmap Heap Scan on order_line (cost=65.45..11521.37 rows=3228 width=20) (actual time=22.555..7764.120 rows=6250 loops=12)
Recheck Cond: (product_id = catalogue_product.id)
Heap Blocks: exact=71735
Buffers: shared hit=55299 read=16686
-> Bitmap Index Scan on order_line_product_id_e620902d (cost=0.00..64.65 rows=3228 width=0) (actual time=21.532..21.532 rows=6269 loops=12)
Index Cond: (product_id = catalogue_product.id)
Buffers: shared hit=143 read=107
Why does it need to recheck product_id = catalogue_product.id which is the same as in index and then take so much time?
As far as i understand recheck is needed if a) only part of the condition can be covered by index or b) bitmap is too big and must be compressed - but then there should be a lossy=x entry, right?
Complete query:
SELECT ("order_order"."date_placed" AT TIME ZONE 'UTC')::date, "partner_partner"."odoo_id", "catalogue_product"."odoo_id", SUM("order_line"."quantity") AS "orders"
FROM "order_line"
INNER JOIN "order_order" ON ("order_line"."order_id" = "order_order"."id")
INNER JOIN "catalogue_product" ON ("order_line"."product_id" = "catalogue_product"."id")
INNER JOIN "partner_stockrecord" ON ("order_line"."stockrecord_id" = "partner_stockrecord"."id")
INNER JOIN "partner_partner" ON ("partner_stockrecord"."partner_id" = "partner_partner"."id")
WHERE (("order_order"."date_placed" AT TIME ZONE 'UTC')::date IN ('2022-11-22'::DATE)
AND "catalogue_product"."odoo_id" IN (6241, 6499, 6500, 49195, 44753, 44754, 53427, 6452, 44755, 44787, 6427, 6428)
AND "partner_partner"."odoo_id" IS NOT NULL AND NOT ("order_order"."status" IN ('Pending', 'PaymentDeclined', 'Canceled')))
GROUP BY ("order_order"."date_placed" AT TIME ZONE 'UTC')::date, "partner_partner"."odoo_id", "catalogue_product"."odoo_id", "order_line"."id"
ORDER BY "order_line"."id" ASC
Complete plan:
GroupAggregate (cost=141002.93..141003.41 rows=16 width=24) (actual time=93629.346..93629.369 rows=52 loops=1)
Group Key: order_line.id, ((timezone('UTC'::text, order_order.date_placed))::date), partner_partner.odoo_id, catalogue_product.odoo_id
Buffers: shared hit=56537 read=16693
-> Sort (cost=141002.93..141002.97 rows=16 width=20) (actual time=93629.331..93629.335 rows=52 loops=1)
Sort Key: order_line.id, partner_partner.odoo_id, catalogue_product.odoo_id
Sort Method: quicksort Memory: 29kB
Buffers: shared hit=56537 read=16693
-> Hash Join (cost=2319.22..141002.61 rows=16 width=20) (actual time=859.917..93629.204 rows=52 loops=1)
Hash Cond: (partner_stockrecord.partner_id = partner_partner.id)
Buffers: shared hit=56537 read=16693
-> Nested Loop (cost=2318.11..141001.34 rows=16 width=24) (actual time=859.853..93628.903 rows=52 loops=1)
Buffers: shared hit=56536 read=16693
-> Hash Join (cost=2317.69..140994.41 rows=16 width=24) (actual time=859.824..93627.791 rows=52 loops=1)
Hash Cond: (order_line.order_id = order_order.id)
Buffers: shared hit=56328 read=16693
-> Nested Loop (cost=108.94..138731.32 rows=20700 width=20) (actual time=1.566..93206.434 rows=74999 loops=1)
Buffers: shared hit=55334 read=16686
-> Bitmap Heap Scan on catalogue_product (cost=43.48..87.52 rows=12 width=8) (actual time=0.080..0.183 rows=12 loops=1)
Recheck Cond: (odoo_id = ANY ('{6241,6499,6500,49195,44753,44754,53427,6452,44755,44787,6427,6428}'::integer[]))
Heap Blocks: exact=11
Buffers: shared hit=35
-> Bitmap Index Scan on catalogue_product_odoo_id_c5e41bad (cost=0.00..43.48 rows=12 width=0) (actual time=0.072..0.072 rows=12 loops=1)
Index Cond: (odoo_id = ANY ('{6241,6499,6500,49195,44753,44754,53427,6452,44755,44787,6427,6428}'::integer[]))
Buffers: shared hit=24
-> Bitmap Heap Scan on order_line (cost=65.45..11521.37 rows=3228 width=20) (actual time=22.555..7764.120 rows=6250 loops=12)
Recheck Cond: (product_id = catalogue_product.id)
Heap Blocks: exact=71735
Buffers: shared hit=55299 read=16686
-> Bitmap Index Scan on order_line_product_id_e620902d (cost=0.00..64.65 rows=3228 width=0) (actual time=21.532..21.532 rows=6269 loops=12)
Index Cond: (product_id = catalogue_product.id)
Buffers: shared hit=143 read=107
-> Hash (cost=2194.42..2194.42 rows=1147 width=12) (actual time=365.766..365.766 rows=1313 loops=1)
Buckets: 2048 Batches: 1 Memory Usage: 73kB
Buffers: shared hit=994 read=7
-> Index Scan using order_date_placed_utc_date_idx on order_order (cost=0.43..2194.42 rows=1147 width=12) (actual time=0.050..365.158 rows=1313 loops=1)
Index Cond: ((timezone('UTC'::text, date_placed))::date = '2022-11-22'::date)
Filter: ((status)::text <> ALL ('{Pending,PaymentDeclined,Canceled}'::text[]))
Rows Removed by Filter: 253
Buffers: shared hit=994 read=7
-> Index Scan using partner_stockrecord_pkey on partner_stockrecord (cost=0.41..0.43 rows=1 width=8) (actual time=0.017..0.017 rows=1 loops=52)
Index Cond: (id = order_line.stockrecord_id)
Buffers: shared hit=208
-> Hash (cost=1.05..1.05 rows=5 width=8) (actual time=0.028..0.028 rows=5 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
Buffers: shared hit=1
-> Seq Scan on partner_partner (cost=0.00..1.05 rows=5 width=8) (actual time=0.013..0.015 rows=5 loops=1)
Filter: (odoo_id IS NOT NULL)
Buffers: shared hit=1
Planning time: 3.275 ms
Execution time: 93629.781 ms
It doesn't have to do any rechecks. That line in the plan comes from the planner, not from the run-time part. (you can tell because if you just do EXPLAIN without ANALYZE, the line still appears.) At planning time, it doesn't know whether any of the bitmap will overflow, so it has to be prepared to do the recheck, even if that turns out not to be necessary to execute it at run time. The slowness almost certainly comes from the time spent reading 16686 random pages, which could be made clear by turning on track_io_timing.
We are having some struggle identifying why Postgres is using too much batches to resolve a join.
Here it is the output of explain analyze of a problematic execution:
https://explain.dalibo.com/plan/xNJ#plan
Limit (cost=20880.87..20882.91 rows=48 width=205) (actual time=10722.953..10723.358 rows=48 loops=1)
-> Unique (cost=20880.87..21718.12 rows=19700 width=205) (actual time=10722.951..10723.356 rows=48 loops=1)
-> Sort (cost=20880.87..20930.12 rows=19700 width=205) (actual time=10722.950..10722.990 rows=312 loops=1)
Sort Key: titlemetadata_titlemetadata.creation_date DESC, titlemetadata_titlemetadata.id, titlemetadata_titlemetadata.title_type, titlemetadata_titlemetadata.original_title, titlemetadata_titlemetadata.alternative_ids, titlemetadata_titlemetadata.metadata,
titlemetadata_titlemetadata.is_adult, titlemetadata_titlemetadata.is_kids, titlemetadata_titlemetadata.last_modified, titlemetadata_titlemetadata.year, titlemetadata_titlemetadata.runtime, titlemetadata_titlemetadata.rating, titlemetadata_titlemetadata.video_provider, tit
lemetadata_titlemetadata.series_id_id, titlemetadata_titlemetadata.season_number, titlemetadata_titlemetadata.episode_number
Sort Method: quicksort Memory: 872kB
-> Hash Right Join (cost=13378.20..19475.68 rows=19700 width=205) (actual time=1926.352..10709.970 rows=2909 loops=1)
Hash Cond: (t4.titlemetadata_id = t3.id)
Filter: ((hashed SubPlan 1) OR (hashed SubPlan 2))
Rows Removed by Filter: 63248
-> Seq Scan on video_provider_offer t4 (cost=0.00..5454.90 rows=66290 width=16) (actual time=0.024..57.893 rows=66390 loops=1)
-> Hash (cost=11314.39..11314.39 rows=22996 width=221) (actual time=489.530..489.530 rows=60096 loops=1)
Buckets: 65536 (originally 32768) Batches: 32768 (originally 1) Memory Usage: 11656kB
-> Hash Right Join (cost=5380.95..11314.39 rows=22996 width=221) (actual time=130.024..225.271 rows=60096 loops=1)
Hash Cond: (video_provider_offer.titlemetadata_id = titlemetadata_titlemetadata.id)
-> Seq Scan on video_provider_offer (cost=0.00..5454.90 rows=66290 width=16) (actual time=0.011..32.950 rows=66390 loops=1)
-> Hash (cost=5129.28..5129.28 rows=20133 width=213) (actual time=129.897..129.897 rows=55793 loops=1)
Buckets: 65536 (originally 32768) Batches: 2 (originally 1) Memory Usage: 7877kB
-> Merge Left Join (cost=1.72..5129.28 rows=20133 width=213) (actual time=0.041..93.057 rows=55793 loops=1)
Merge Cond: (titlemetadata_titlemetadata.id = t3.series_id_id)
-> Index Scan using titlemetadata_titlemetadata_pkey on titlemetadata_titlemetadata (cost=1.30..4130.22 rows=20133 width=205) (actual time=0.028..62.949 rows=43921 loops=1)
Filter: ((NOT is_adult) AND (NOT (hashed SubPlan 3)) AND (((title_type)::text = 'MOV'::text) OR ((title_type)::text = 'TVS'::text) OR ((title_type)::text = 'TVP'::text) OR ((title_type)::text = 'EVT'::text)))
Rows Removed by Filter: 14121
SubPlan 3
-> Seq Scan on cable_operator_cableoperatorexcludedtitle u0_2 (cost=0.00..1.01 rows=1 width=8) (actual time=0.006..0.006 rows=0 loops=1)
Filter: (cable_operator_id = 54)
-> Index Scan using titlemetadata_titlemetadata_series_id_id_73453db4_uniq on titlemetadata_titlemetadata t3 (cost=0.41..3901.36 rows=58037 width=16) (actual time=0.011..9.375 rows=12887 loops=1)
SubPlan 1
-> Hash Join (cost=44.62..885.73 rows=981 width=8) (actual time=0.486..36.806 rows=5757 loops=1)
Hash Cond: (w2.device_id = w3.id)
-> Nested Loop (cost=43.49..866.20 rows=2289 width=16) (actual time=0.441..33.096 rows=20180 loops=1)
-> Nested Loop (cost=43.06..414.98 rows=521 width=8) (actual time=0.426..9.952 rows=2909 loops=1)
Join Filter: (w1.id = w0.video_provider_id)
-> Nested Loop (cost=42.65..54.77 rows=13 width=24) (actual time=0.399..0.532 rows=15 loops=1)
-> HashAggregate (cost=42.50..42.95 rows=45 width=16) (actual time=0.390..0.403 rows=45 loops=1)
Group Key: v0.id
-> Nested Loop (cost=13.34..42.39 rows=45 width=16) (actual time=0.095..0.364 rows=45 loops=1)
-> Hash Semi Join (cost=13.19..32.72 rows=45 width=8) (actual time=0.084..0.229 rows=45 loops=1)
Hash Cond: (v1.id = u0.id)
-> Seq Scan on cable_operator_cableoperatorprovider v1 (cost=0.00..17.36 rows=636 width=16) (actual time=0.010..0.077 rows=636 loops=1)
-> Hash (cost=12.63..12.63 rows=45 width=8) (actual time=0.046..0.046 rows=45 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Index Scan using cable_operator_cableoperatorprovider_4d6e54b3 on cable_operator_cableoperatorprovider u0 (cost=0.28..12.63 rows=45 width=8) (actual time=0.016..0.035 rows=45 loops=1)
Index Cond: (cable_operator_id = 54)
-> Index Only Scan using video_provider_videoprovider_pkey on video_provider_videoprovider v0 (cost=0.15..0.20 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=45)
Index Cond: (id = v1.provider_id)
Heap Fetches: 45
-> Index Scan using video_provider_videoprovider_pkey on video_provider_videoprovider w1 (cost=0.15..0.25 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (id = v0.id)
Filter: ((video_provider_type)::text = 'VOD'::text)
Rows Removed by Filter: 1
-> Index Scan using video_provider_offer_da942d2e on video_provider_offer w0 (cost=0.42..27.22 rows=39 width=16) (actual time=0.026..0.585 rows=194 loops=15)
Index Cond: (video_provider_id = v0.id)
Filter: (((end_date > '2021-09-02 19:23:00-03'::timestamp with time zone) OR (end_date IS NULL)) AND (access_criteria && '{vtv_mas,TBX_LOGIN,urn:spkg:tve:fox-premium,urn:tve:mcp,AMCHD,AMC_CONSORCIO,ANIMAL_PLANET,ASUNTOS_PUBLI
COS,ASUNTOS_PUBLICOS_CONSORCIO,CINECANALLIVE,CINECANAL_CONSORCIO,DISCOVERY,DISCOVERY_KIDS_CONSORCIO,DISCOVERY_KIDS_OD,DISNEY,DISNEY_CH_CONSORCIO,DISNEY_XD,DISNEY_XD_CONSORCIO,EL_CANAL_HD,EL_CANAL_HD_CONSORCIO,EL_GOURMET_CONSORCIO,ESPN,ESPN2_HD_CONSORCIO,ESPN3_HD_CONSORCIO
,ESPNMAS_HD_CONSORCIO,ESPN_BASIC,ESPN_HD_CONSORCIO,ESPN_PLAY,EUROPALIVE,EUROPA_EUROPA,EUROPA_EUROPA_CONSORCIO,FILMANDARTS_DISPOSITIVOS,FILMS_ARTS,FILM_AND_ARTS_CONSORCIO,FOXLIFE,FOX_LIFE_CONSORCIO,FOX_SPORTS_1_DISPOSITIVOS,FOX_SPORTS_2_DISPOSITIVOS,FOX_SPORTS_2_HD_CONSORC
IO,FOX_SPORTS_3_DISPOSITIVOS,FOX_SPORTS_3_HD_CONSORCIO,FOX_SPORTS_HD_CONSORCIO,FRANCE24_DISPOSITIVOS,FRANCE_24_CONSORCIO,GOURMET,GOURMET_DISPOSITIVOS,HOME_HEALTH,INVESTIGATION_DISCOVERY,MAS_CHIC,NATGEOKIDS_DISPOSITIVOS,NATGEO_CONSORCIO,NATGEO_DISPOSITIVOS,NATGEO_KIDS_CONS
ORCIO,PASIONES,PASIONES_CONSORCIO,SVOD_TYC_BASIC,TBX_LOGIN,TCC_2_CONSORCIO,TCC_2_HD,TLC,TVE,TVE_CONSORCIO,TYC_SPORTS_CONSORCIO,VTV_LIVE,clarosports,discoverykids,espnplay_south_alt,urn:spkg:tve:fox-basic,urn:tve:babytv,urn:tve:cinecanal,urn:tve:discoverykids,urn:tve:foxli
fe,urn:tve:fp,urn:tve:fx,urn:tve:natgeo,urn:tve:natgeokids,urn:tve:natgeowild,urn:tve:thefilmzone}'::character varying(50)[]) AND ((((content_type)::text = 'VOD'::text) AND ((start_date < '2021-09-02 19:23:00-03'::timestamp with time zone) OR (start_date IS NULL))) OR ((c
ontent_type)::text = 'LIV'::text)))
Rows Removed by Filter: 5
-> Index Only Scan using video_provider_offer_devices_offer_id_device_id_key on video_provider_offer_devices w2 (cost=0.42..0.81 rows=6 width=16) (actual time=0.004..0.007 rows=7 loops=2909)
Index Cond: (offer_id = w0.id)
Heap Fetches: 17828
-> Hash (cost=1.10..1.10 rows=3 width=8) (actual time=0.029..0.029 rows=2 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on platform_device_device w3 (cost=0.00..1.10 rows=3 width=8) (actual time=0.024..0.027 rows=2 loops=1)
Filter: ((device_code)::text = ANY ('{ANDROID,ott_dual_tcc,ott_k2_tcc}'::text[]))
Rows Removed by Filter: 5
SubPlan 2
-> Hash Join (cost=44.62..885.73 rows=981 width=8) (actual time=0.410..33.580 rows=5757 loops=1)
Hash Cond: (w2_1.device_id = w3_1.id)
-> Nested Loop (cost=43.49..866.20 rows=2289 width=16) (actual time=0.375..29.886 rows=20180 loops=1)
-> Nested Loop (cost=43.06..414.98 rows=521 width=8) (actual time=0.366..9.134 rows=2909 loops=1)
Join Filter: (w1_1.id = w0_1.video_provider_id)
-> Nested Loop (cost=42.65..54.77 rows=13 width=24) (actual time=0.343..0.476 rows=15 loops=1)
-> HashAggregate (cost=42.50..42.95 rows=45 width=16) (actual time=0.333..0.347 rows=45 loops=1)
Group Key: v0_1.id
-> Nested Loop (cost=13.34..42.39 rows=45 width=16) (actual time=0.083..0.311 rows=45 loops=1)
-> Hash Semi Join (cost=13.19..32.72 rows=45 width=8) (actual time=0.076..0.202 rows=45 loops=1)
Hash Cond: (v1_1.id = u0_1.id)
-> Seq Scan on cable_operator_cableoperatorprovider v1_1 (cost=0.00..17.36 rows=636 width=16) (actual time=0.005..0.057 rows=636 loops=1)
-> Hash (cost=12.63..12.63 rows=45 width=8) (actual time=0.038..0.038 rows=45 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Index Scan using cable_operator_cableoperatorprovider_4d6e54b3 on cable_operator_cableoperatorprovider u0_1 (cost=0.28..12.63 rows=45 width=8) (actual time=0.007..0.020 rows=45 loops=1)
Index Cond: (cable_operator_id = 54)
-> Index Only Scan using video_provider_videoprovider_pkey on video_provider_videoprovider v0_1 (cost=0.15..0.20 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=45)
Index Cond: (id = v1_1.provider_id)
Heap Fetches: 45
-> Index Scan using video_provider_videoprovider_pkey on video_provider_videoprovider w1_1 (cost=0.15..0.25 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (id = v0_1.id)
Filter: ((video_provider_type)::text = 'VOD'::text)
Rows Removed by Filter: 1
-> Index Scan using video_provider_offer_da942d2e on video_provider_offer w0_1 (cost=0.42..27.22 rows=39 width=16) (actual time=0.022..0.536 rows=194 loops=15)
Index Cond: (video_provider_id = v0_1.id)
Filter: (((end_date > '2021-09-02 19:23:00-03'::timestamp with time zone) OR (end_date IS NULL)) AND (access_criteria && '{vtv_mas,TBX_LOGIN,urn:spkg:tve:fox-premium,urn:tve:mcp,AMCHD,AMC_CONSORCIO,ANIMAL_PLANET,ASUNTOS_PUBLI
COS,ASUNTOS_PUBLICOS_CONSORCIO,CINECANALLIVE,CINECANAL_CONSORCIO,DISCOVERY,DISCOVERY_KIDS_CONSORCIO,DISCOVERY_KIDS_OD,DISNEY,DISNEY_CH_CONSORCIO,DISNEY_XD,DISNEY_XD_CONSORCIO,EL_CANAL_HD,EL_CANAL_HD_CONSORCIO,EL_GOURMET_CONSORCIO,ESPN,ESPN2_HD_CONSORCIO,ESPN3_HD_CONSORCIO
,ESPNMAS_HD_CONSORCIO,ESPN_BASIC,ESPN_HD_CONSORCIO,ESPN_PLAY,EUROPALIVE,EUROPA_EUROPA,EUROPA_EUROPA_CONSORCIO,FILMANDARTS_DISPOSITIVOS,FILMS_ARTS,FILM_AND_ARTS_CONSORCIO,FOXLIFE,FOX_LIFE_CONSORCIO,FOX_SPORTS_1_DISPOSITIVOS,FOX_SPORTS_2_DISPOSITIVOS,FOX_SPORTS_2_HD_CONSORC
IO,FOX_SPORTS_3_DISPOSITIVOS,FOX_SPORTS_3_HD_CONSORCIO,FOX_SPORTS_HD_CONSORCIO,FRANCE24_DISPOSITIVOS,FRANCE_24_CONSORCIO,GOURMET,GOURMET_DISPOSITIVOS,HOME_HEALTH,INVESTIGATION_DISCOVERY,MAS_CHIC,NATGEOKIDS_DISPOSITIVOS,NATGEO_CONSORCIO,NATGEO_DISPOSITIVOS,NATGEO_KIDS_CONS
ORCIO,PASIONES,PASIONES_CONSORCIO,SVOD_TYC_BASIC,TBX_LOGIN,TCC_2_CONSORCIO,TCC_2_HD,TLC,TVE,TVE_CONSORCIO,TYC_SPORTS_CONSORCIO,VTV_LIVE,clarosports,discoverykids,espnplay_south_alt,urn:spkg:tve:fox-basic,urn:tve:babytv,urn:tve:cinecanal,urn:tve:discoverykids,urn:tve:foxli
fe,urn:tve:fp,urn:tve:fx,urn:tve:natgeo,urn:tve:natgeokids,urn:tve:natgeowild,urn:tve:thefilmzone}'::character varying(50)[]) AND ((((content_type)::text = 'VOD'::text) AND ((start_date < '2021-09-02 19:23:00-03'::timestamp with time zone) OR (start_date IS NULL))) OR ((c
ontent_type)::text = 'LIV'::text)))
Rows Removed by Filter: 5
-> Index Only Scan using video_provider_offer_devices_offer_id_device_id_key on video_provider_offer_devices w2_1 (cost=0.42..0.81 rows=6 width=16) (actual time=0.003..0.006 rows=7 loops=2909)
Index Cond: (offer_id = w0_1.id)
Heap Fetches: 17828
-> Hash (cost=1.10..1.10 rows=3 width=8) (actual time=0.015..0.015 rows=2 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on platform_device_device w3_1 (cost=0.00..1.10 rows=3 width=8) (actual time=0.010..0.011 rows=2 loops=1)
Filter: ((device_code)::text = ANY ('{ANDROID,ott_dual_tcc,ott_k2_tcc}'::text[]))
Rows Removed by Filter: 5
Planning time: 8.255 ms
Execution time: 10723.830 ms
(100 rows)
The weird part is that the same query, sometimes just uses a single batch. Here is an example: https://explain.dalibo.com/plan/zTv#plan
Here is the work_mem being used:
show work_mem;
work_mem
----------
8388kB
(1 row)
I'm not interested in changing the query to be more performant, but in understanding why is the different behavior.
I've found this thread apparently related with this, but I don't quite understand what are they talking about: https://www.postgresql.org/message-id/flat/CA%2BhUKGKWWmf%3DWELLG%3DaUGbcugRaSQbtm0tKYiBut-B2rVKX63g%40mail.gmail.com
Can anyone tell me why is this different behavior? The underlying data is the same in both cases.
If the hash is done in memory, there will only be a single batch.
A difference with the original hash batch numbers is due to Postgres choosing to increase the number of batches in order to reduce memory consumption.
You might find this EXPLAIN glossary useful (disclaimer: I'm one of the authors), here is the page on Hash Batches which also links to the PostgreSQL source code (it's very nicely documented in plain English).
While not a perfect heuristic, you can see that the memory required for the operations with multiple batches are around or above your work_mem setting. They can be lower than it, due to operations on disk generally requiring less memory overall.
I'm not 100% sure why in your exact case one was chosen over the other, but it does look like there are some very slight row estimate differences, which might be a good place to start.
As of PostgreSQL 13 there is also now a hash_mem_multiplier setting that can be used to give more memory to hashes without doing so for other operations (like sorts).
We where able to solve the problem just by doing VACUUM FULL ANALYZE;.
After that, everything started to work as expected (https://explain.depesz.com/s/eoqH#html)
Side note: we where not aware that we should do this on daily basis.
I was working on optimising a query, with dumb luck I tried something and it improved the query but I am unable to explain why.
Below is the query with poor performance
with ctedata1 as(
select
sum(total_visit_count) as total_visit_count,
sum(sh_visit_count) as sh_visit_count,
sum(ec_visit_count) as ec_visit_count,
sum(total_like_count) as total_like_count,
sum(sh_like_count) as sh_like_count,
sum(ec_like_count) as ec_like_count,
sum(total_order_count) as total_order_count,
sum(sh_order_count) as sh_order_count,
sum(ec_order_count) as ec_order_count,
sum(total_sales_amount) as total_sales_amount,
sum(sh_sales_amount) as sh_sales_amount,
sum(ec_sales_amount) as ec_sales_amount,
sum(ec_order_online_count) as ec_order_online_count,
sum(ec_sales_online_amount) as ec_sales_online_amount,
sum(ec_order_in_store_count) as ec_order_in_store_count,
sum(ec_sales_in_store_amount) as ec_sales_in_store_amount,
table2.im_name,
table2.brand as kpibrand,
table2.id_region as kpiregion
from
table2
where
deleted_at is null
and id_region = any('{1}')
group by
im_name,
kpiregion,
kpibrand ),
ctedata2 as (
select
ctedata1.*,
rank() over (partition by (kpiregion,
kpibrand)
order by
coalesce(ctedata1.total_sales_amount, 0) desc) rank,
count(*) over (partition by (kpiregion,
kpibrand)) as total_count
from
ctedata1 )
select
table1.id_pf_item,
table1.product_id,
table1.color_code,
table1.l1_code,
table1.local_title as product_name,
table1.id_region,
table1.gender,
case
when table1.created_at is null then '1970/01/01 00:00:00'
else table1.created_at
end as created_at,
(
select
count(distinct id_outfit)
from
table3
left join table4 on
table3.id_item = table4.id_item
and table4.deleted_at is null
where
table3.deleted_at is null
and table3.id_pf_item = table1.id_pf_item) as outfit_count,
count(*) over() as total_matched,
case
when table1.v8_im_name = '' then table1.im_name
else table1.v8_im_name
end as im_name,
case
when table1.id_region != 1 then null
else
case
when table1.sales_start_at is null then '1970/01/01 00:00:00'
else table1.sales_start_at
end
end as sales_start_date,
table1.category_ids,
array_to_string(table1.intermediate_category_ids, ','),
table1.image_url,
table1.brand,
table1.pdp_url,
coalesce(ctedata2.total_visit_count, 0) as total_visit_count,
coalesce(ctedata2.sh_visit_count, 0) as sh_visit_count,
coalesce(ctedata2.ec_visit_count, 0) as ec_visit_count,
coalesce(ctedata2.total_like_count, 0) as total_like_count,
coalesce(ctedata2.sh_like_count, 0) as sh_like_count,
coalesce(ctedata2.ec_like_count, 0) as ec_like_count,
coalesce(ctedata2.total_order_count, 0) as total_order_count,
coalesce(ctedata2.sh_order_count, 0) as sh_order_count,
coalesce(ctedata2.ec_order_count, 0) as ec_order_count,
coalesce(ctedata2.total_sales_amount, 0) as total_sales_amount,
coalesce(ctedata2.sh_sales_amount, 0) as sh_sales_amount,
coalesce(ctedata2.ec_sales_amount, 0) as ec_sales_amount,
coalesce(ctedata2.ec_order_online_count, 0) as ec_order_online_count,
coalesce(ctedata2.ec_sales_online_amount, 0) as ec_sales_online_amount,
coalesce(ctedata2.ec_order_in_store_count, 0) as ec_order_in_store_count,
coalesce(ctedata2.ec_sales_in_store_amount, 0) as ec_sales_in_store_amount,
ctedata2.rank,
ctedata2.total_count,
table1.department,
table1.seasons
from
table1
left join ctedata2 on
table1.im_name = ctedata2.im_name
and table1.brand = ctedata2.kpibrand
where
table1.deleted_at is null
and table1.id_region = any('{1}')
and lower(table1.brand) = any('{"brand1","brand2"}')
and 'season1' = any(lower(seasons::text)::text[])
and table1.department = 'Department1'
order by
total_sales_amount desc offset 0
limit 100
The explain output for above query is
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=172326.55..173435.38 rows=1 width=952) (actual time=85664.201..85665.970 rows=100 loops=1)
CTE ctedata1
-> GroupAggregate (cost=0.42..80478.71 rows=43468 width=530) (actual time=0.063..708.069 rows=73121 loops=1)
Group Key: table2.im_name, table2.id_region, table2.brand
-> Index Scan using udx_table2_im_name_id_region_brand_target_date_key on table2 (cost=0.42..59699.18 rows=391708 width=146) (actual time=0.029..308.582 rows=391779 loops=1)
Filter: ((deleted_at IS NULL) AND (id_region = ANY ('{1}'::integer[])))
Rows Removed by Filter: 20415
CTE ctedata2
-> WindowAgg (cost=16104.06..17842.78 rows=43468 width=628) (actual time=1012.994..1082.057 rows=73121 loops=1)
-> WindowAgg (cost=16104.06..17082.09 rows=43468 width=620) (actual time=945.755..1014.656 rows=73121 loops=1)
-> Sort (cost=16104.06..16212.73 rows=43468 width=612) (actual time=945.747..963.254 rows=73121 loops=1)
Sort Key: ctedata1.kpiregion, ctedata1.kpibrand, (COALESCE(ctedata1.total_sales_amount, '0'::numeric)) DESC
Sort Method: external merge Disk: 6536kB
-> CTE Scan on ctedata1 (cost=0.00..869.36 rows=43468 width=612) (actual time=0.069..824.841 rows=73121 loops=1)
-> Result (cost=74005.05..75113.88 rows=1 width=952) (actual time=85664.199..85665.950 rows=100 loops=1)
-> Sort (cost=74005.05..74005.05 rows=1 width=944) (actual time=85664.072..85664.089 rows=100 loops=1)
Sort Key: (COALESCE(ctedata2.total_sales_amount, '0'::numeric)) DESC
Sort Method: top-N heapsort Memory: 76kB
-> WindowAgg (cost=10960.95..74005.04 rows=1 width=944) (actual time=85658.049..85661.393 rows=3151 loops=1)
-> Nested Loop Left Join (cost=10960.95..74005.02 rows=1 width=927) (actual time=1075.219..85643.595 rows=3151 loops=1)
Join Filter: (((table1.im_name)::text = ctedata2.im_name) AND ((table1.brand)::text = ctedata2.kpibrand))
Rows Removed by Join Filter: 230402986
-> Bitmap Heap Scan on table1 (cost=10960.95..72483.64 rows=1 width=399) (actual time=45.466..278.376 rows=3151 loops=1)
Recheck Cond: (id_region = ANY ('{1}'::integer[]))
Filter: ((deleted_at IS NULL) AND (department = 'Department1'::text) AND (lower((brand)::text) = ANY ('{brand1, brand2}'::text[])) AND ('season1'::text = ANY ((lower((seasons)::text))::text[])))
Rows Removed by Filter: 106335
Heap Blocks: exact=42899
-> Bitmap Index Scan on table1_im_name_id_region_key (cost=0.00..10960.94 rows=110619 width=0) (actual time=38.307..38.307 rows=109486 loops=1)
Index Cond: (id_region = ANY ('{1}'::integer[]))
-> CTE Scan on ctedata2 (cost=0.00..869.36 rows=43468 width=592) (actual time=0.325..21.721 rows=73121 loops=3151)
SubPlan 3
-> Aggregate (cost=1108.80..1108.81 rows=1 width=8) (actual time=0.018..0.018 rows=1 loops=100)
-> Nested Loop Left Join (cost=5.57..1108.57 rows=93 width=4) (actual time=0.007..0.016 rows=3 loops=100)
-> Bitmap Heap Scan on table3 (cost=5.15..350.95 rows=93 width=4) (actual time=0.005..0.008 rows=3 loops=100)
Recheck Cond: (id_pf_item = table1.id_pf_item)
Filter: (deleted_at IS NULL)
Heap Blocks: exact=107
-> Bitmap Index Scan on idx_id_pf_item (cost=0.00..5.12 rows=93 width=0) (actual time=0.003..0.003 rows=3 loops=100)
Index Cond: (id_pf_item = table1.id_pf_item)
-> Index Scan using index_table4_id_item on table4 (cost=0.42..8.14 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=303)
Index Cond: (table3.id_item = id_item)
Filter: (deleted_at IS NULL)
Rows Removed by Filter: 0
Planning time: 1.023 ms
Execution time: 85669.512 ms
I changed
and lower(table1.brand) = any('{"brand1","brand2"}')
in the query to
and table1.brand = any('{"Brand1","Brand2"}')
and the plan changed to
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=173137.44..188661.06 rows=14 width=952) (actual time=1444.123..1445.653 rows=100 loops=1)
CTE ctedata1
-> GroupAggregate (cost=0.42..80478.71 rows=43468 width=530) (actual time=0.040..769.982 rows=73121 loops=1)
Group Key: table2.im_name, table2.id_region, table2.brand
-> Index Scan using udx_table2_item_im_name_id_region_brand_target_date_key on table2 (cost=0.42..59699.18 rows=391708 width=146) (actual time=0.021..350.774 rows=391779 loops=1)
Filter: ((deleted_at IS NULL) AND (id_region = ANY ('{1}'::integer[])))
Rows Removed by Filter: 20415
CTE ctedata2
-> WindowAgg (cost=16104.06..17842.78 rows=43468 width=628) (actual time=1088.905..1153.749 rows=73121 loops=1)
-> WindowAgg (cost=16104.06..17082.09 rows=43468 width=620) (actual time=1020.017..1089.117 rows=73121 loops=1)
-> Sort (cost=16104.06..16212.73 rows=43468 width=612) (actual time=1020.011..1037.170 rows=73121 loops=1)
Sort Key: ctedata1.kpiregion, ctedata1.kpibrand, (COALESCE(ctedata1.total_sales_amount, '0'::numeric)) DESC
Sort Method: external merge Disk: 6536kB
-> CTE Scan on ctedata1 (cost=0.00..869.36 rows=43468 width=612) (actual time=0.044..891.653 rows=73121 loops=1)
-> Result (cost=74815.94..90339.56 rows=14 width=952) (actual time=1444.121..1445.635 rows=100 loops=1)
-> Sort (cost=74815.94..74815.98 rows=14 width=944) (actual time=1444.053..1444.065 rows=100 loops=1)
Sort Key: (COALESCE(ctedata2.total_sales_amount, '0'::numeric)) DESC
Sort Method: top-N heapsort Memory: 76kB
-> WindowAgg (cost=72207.31..74815.68 rows=14 width=944) (actual time=1439.128..1441.885 rows=3151 loops=1)
-> Hash Right Join (cost=72207.31..74815.40 rows=14 width=927) (actual time=1307.531..1437.246 rows=3151 loops=1)
Hash Cond: ((ctedata2.im_name = (table1.im_name)::text) AND (ctedata2.kpibrand = (table1.brand)::text))
-> CTE Scan on ctedata2 (cost=0.00..869.36 rows=43468 width=592) (actual time=1088.911..1209.646 rows=73121 loops=1)
-> Hash (cost=72207.10..72207.10 rows=14 width=399) (actual time=216.850..216.850 rows=3151 loops=1)
Buckets: 4096 (originally 1024) Batches: 1 (originally 1) Memory Usage: 1249kB
-> Bitmap Heap Scan on table1 (cost=10960.95..72207.10 rows=14 width=399) (actual time=46.434..214.246 rows=3151 loops=1)
Recheck Cond: (id_region = ANY ('{1}'::integer[]))
Filter: ((deleted_at IS NULL) AND (department = 'Department1'::text) AND ((brand)::text = ANY ('{Brand1, Brand2}'::text[])) AND ('season1'::text = ANY ((lower((seasons)::text))::text[])))
Rows Removed by Filter: 106335
Heap Blocks: exact=42899
-> Bitmap Index Scan on table1_im_name_id_region_key (cost=0.00..10960.94 rows=110619 width=0) (actual time=34.849..34.849 rows=109486 loops=1)
Index Cond: (id_region = ANY ('{1}'::integer[]))
SubPlan 3
-> Aggregate (cost=1108.80..1108.81 rows=1 width=8) (actual time=0.015..0.015 rows=1 loops=100)
-> Nested Loop Left Join (cost=5.57..1108.57 rows=93 width=4) (actual time=0.006..0.014 rows=3 loops=100)
-> Bitmap Heap Scan on table3 (cost=5.15..350.95 rows=93 width=4) (actual time=0.004..0.006 rows=3 loops=100)
Recheck Cond: (id_pf_item = table1.id_pf_item)
Filter: (deleted_at IS NULL)
Heap Blocks: exact=107
-> Bitmap Index Scan on idx_id_pf_item (cost=0.00..5.12 rows=93 width=0) (actual time=0.003..0.003 rows=3 loops=100)
Index Cond: (id_pf_item = table1.id_pf_item)
-> Index Scan using index_table4_id_item on table4 (cost=0.42..8.14 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=303)
Index Cond: (table3.id_item = id_item)
Filter: (deleted_at IS NULL)
Rows Removed by Filter: 0
Planning time: 0.760 ms
Execution time: 1448.848 ms
My Observation
The join strategy for table1 left join ctedata2 changes after the lower() function is avoided. The strategy changes from nested loop left join to hash right join.
The CTE Scan node on ctedata2 is executed only once in the better performing query.
Postgres Version
9.6
Please help me to understand this behaviour. I will supply additional info if required.
It is almost not worthwhile taking a deep dive into the inner workings of a nearly-obsolete version. That time and energy is probably better spent jollying along an upgrade.
But the problem is pretty plain. Your scan on table1 is estimated dreadfully, although 14 times less dreadful in the better plan.
-> Bitmap Heap Scan on table1 (cost=10960.95..72483.64 rows=1 width=399) (actual time=45.466..278.376 rows=3151 loops=1)
-> Bitmap Heap Scan on table1 (cost=10960.95..72207.10 rows=14 width=399) (actual time=46.434..214.246 rows=3151 loops=1)
Your use of lower(), apparently without reason, surely contributes to the poor estimation. And dynamically converting a string into an array certainly doesn't help either. If it were stored as a real array in the first place, the statistics system could get its hands on it and generate more reasonable estimates.
When ch table is queried direcly on rp_fk column, it uses index for almost all of its partitions.
explain analyze select * from oc.ch ch where rp_fk = 'abc123';
Append (cost=0.14..9469.61 rows=2357 width=10008) (actual time=0.164..0.166 rows=0 loops=1)
-> Index Scan using ch_2016_rp_fk_idx1 on ch_2016 ch (cost=0.14..8.16 rows=1 width=11097) (actual time=0.015..0.016 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Index Scan using ch_2017_rp_fk_idx1 on ch_2017 ch_1 (cost=0.14..8.16 rows=1 width=10477) (actual time=0.022..0.022 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Index Scan using ch_2018_rp_fk_idx1 on ch_2018 ch_2 (cost=0.56..4.58 rows=1 width=12295) (actual time=0.031..0.031 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Index Scan using ch_2019_rp_fk_idx1 on ch_2019 ch_3 (cost=0.69..4.71 rows=1 width=12299) (actual time=0.025..0.025 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Index Scan using ch_2020_rp_fk_idx1 on ch_2020 ch_4 (cost=0.69..6030.67 rows=1510 width=10259) (actual time=0.029..0.029 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Bitmap Heap Scan on ch_2021 ch_5 (cost=55.08..3392.36 rows=841 width=9566) (actual time=0.018..0.019 rows=0 loops=1)
Recheck Cond: (rp_fk = 'abc123'::bpchar)
-> Bitmap Index Scan on ch_2021_rp_fk_idx1 (cost=0.00..54.87 rows=841 width=0) (actual time=0.016..0.016 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Index Scan using ch_2022_rp_fk_idx on ch_2022 ch_6 (cost=0.12..8.14 rows=1 width=12241) (actual time=0.013..0.013 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Seq Scan on ch_def ch_7 (cost=0.00..1.05 rows=1 width=12167) (actual time=0.009..0.009 rows=0 loops=1)
Filter: (rp_fk = 'abc123'::bpchar)
Rows Removed by Filter: 4
Planning Time: 24.637 ms
Execution Time: 0.538 ms
But when joining ch table with p table from where it gets the rp_fk, its doing a sequential scan of the ch_2018 and ch_2019 partitions. This is where its taking most of the time, whereas 2020 and 2021 paritions are scanned using index. Any insight as to why its using seq scan on these 2 partitions? All the partitions have same indexes and have upto date vacuum analyze. random_page_cost is set to 1.
explain analyze select * FROM oc.ch ch JOIN op.p p ON ch.rp_fk = p.p_pk WHERE ch.del_flg = '0'::bpchar and hf_p_num_cd ='112642817-002';
Nested Loop (cost=0.42..13134696.50 rows=96 width=12593) (actual time=193878.564..193878.569 rows=0 loops=1)
-> Index Scan using ix_p_hf_p_num_cd on p p (cost=0.42..8.44 rows=1 width=1420) (actual time=0.025..0.028 rows=1 loops=1)
Index Cond: ((hf_p_num_cd)::text = '112642817-002'::text)
-> Append (cost=0.00..12666318.27 rows=46836978 width=11167) (actual time=193878.532..193878.535 rows=0 loops=1)
-> Seq Scan on ch_2016 ch (cost=0.00..11.57 rows=13 width=11097) (actual time=0.072..0.072 rows=0 loops=1)
Filter: ((del_flg = '0'::bpchar) AND (p.p_pk = rp_fk))
Rows Removed by Filter: 38
-> Index Scan using ch_2017_rp_fk_idx1 on ch_2017 ch_1 (cost=0.14..20.94 rows=40 width=10477) (actual time=0.016..0.016 rows=0 loops=1)
Index Cond: (rp_fk = p.p_pk)
Filter: (del_flg = '0'::bpchar)
-> Seq Scan on ch_2018 ch_2 (cost=0.00..4005273.43 rows=15109962 width=12295) (actual time=25051.825..25051.825 rows=0 loops=1)
Filter: ((del_flg = '0'::bpchar) AND (p.p_pk = rp_fk))
Rows Removed by Filter: 15111645
-> Seq Scan on ch_2019 ch_3 (cost=0.00..8406638.77 rows=31721918 width=12299) (actual time=168826.481..168826.481 rows=0 loops=1)
Filter: ((del_flg = '0'::bpchar) AND (p.p_pk = rp_fk))
Rows Removed by Filter: 31722679
-> Index Scan using ch_2020_rp_fk_idx1 on ch_2020 ch_4 (cost=0.69..14893.73 rows=3729 width=10259) (actual time=0.057..0.057 rows=0 loops=1)
Index Cond: (rp_fk = p.p_pk)
Filter: (del_flg = '0'::bpchar)
-> Bitmap Heap Scan on ch_2021 ch_5 (cost=82.74..5285.74 rows=1314 width=9566) (actual time=0.033..0.033 rows=0 loops=1)
Recheck Cond: (rp_fk = p.p_pk)
Filter: (del_flg = '0'::bpchar)
-> Bitmap Index Scan on ch_2021_rp_fk_idx1 (cost=0.00..82.42 rows=1314 width=0) (actual time=0.022..0.022 rows=0 loops=1)
Index Cond: (rp_fk = p.p_pk)
-> Index Scan using ch_2022_rp_fk_idx on ch_2022 ch_6 (cost=0.12..8.14 rows=1 width=12241) (actual time=0.014..0.014 rows=0 loops=1)
Filter: ((del_flg = '0'::bpchar) AND (p.p_pk = rp_fk))
-> Seq Scan on ch_def ch_7 (cost=0.00..1.06 rows=1 width=12167) (actual time=0.017..0.017 rows=0 loops=1)
Filter: ((del_flg = '0'::bpchar) AND (p.p_pk = rp_fk))
Rows Removed by Filter: 4
Planning Time: 4.183 ms
Execution Time: 193878.918 ms
It seems like 'abc123' is a very rare value for rp_fk, so PostgreSQL plans an index scan.
But it seems there are values that are much more frequent. In the second query, the optimizer does not know what value the rp_fk for hf_p_num_cd = '112642817-002' is going to be, so it goes with some average estimate, which turns out to be much higher for many partitions. Hence the sequential scan.
I would split the query in two parts and query the partitioned table with the constants found in the first query. Then the optimizer knows more and will plan better.
If you are certain that an index scan will always win, you can force the planner's hand:
BEGIN;
SET LOCAL enable_seqscan = off;
SET LOCAL enable_bitmapscan = off;
SELECT ...
COMMIT;
I have been trying to speed up my psql queries to squeeze out as much speed as possible. With a few indexes I installed on my local system I got good speeds. I installed these on the remote system but had different results. The screenshot for the planners follow:
Local Planner:
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
Hash Join (cost=19.54..67.37 rows=12 width=133) (actual time=0.771..0.862 rows=12 loops=1)
Hash Cond: ((sensor_lookup.sensorid)::text = (sensor.sensorid)::text)
Buffers: shared hit=25
-> Nested Loop (cost=3.01..50.81 rows=12 width=119) (actual time=0.193..0.271 rows=12 loops=1)
Buffers: shared hit=19
-> Nested Loop (cost=2.60..26.10 rows=1 width=320) (actual time=0.163..0.228 rows=1 loops=1)
Buffers: shared hit=15
-> Nested Loop (cost=2.60..25.02 rows=1 width=98) (actual time=0.156..0.217 rows=1 loops=1)
Buffers: shared hit=14
-> Nested Loop (cost=0.27..13.80 rows=1 width=68) (actual time=0.097..0.151 rows=1 loops=1)
Buffers: shared hit=7
-> Index Scan using meta_pkey on meta (cost=0.27..4.29 rows=1 width=45) (actual time=0.029..0.031 rows=1 loops=1)
Index Cond: (stationid = 'WYTOR02'::bpchar)
Buffers: shared hit=3
-> Seq Scan on meta_lookup (cost=0.00..9.50 rows=1 width=31) (actual time=0.064..0.116 rows=1 loops=1)
Filter: ((stationid)::bpchar = 'WYTOR02'::bpchar)
Rows Removed by Filter: 439
Buffers: shared hit=4
-> Bitmap Heap Scan on datetime_lookup (cost=2.33..11.21 rows=1 width=38) (actual time=0.054..0.060 rows=1 loops=1)
Recheck Cond: (stationid = 'WYTOR02'::bpchar)
Filter: ((productid)::text = 'qc60'::text)
Rows Removed by Filter: 5
Heap Blocks: exact=5
Buffers: shared hit=7
-> Bitmap Index Scan on idx_16 (cost=0.00..2.32 rows=6 width=0) (actual time=0.033..0.033 rows=6 loops=1)
Index Cond: (stationid = 'WYTOR02'::bpchar)
Buffers: shared hit=2
-> Seq Scan on product (cost=0.00..1.07 rows=1 width=222) (actual time=0.006..0.008 rows=1 loops=1)
Filter: ((productid)::text = 'qc60'::text)
Rows Removed by Filter: 5
Buffers: shared hit=1
-> Index Scan using idx_15 on sensor_lookup (cost=0.41..24.59 rows=12 width=30) (actual time=0.027..0.034 rows=12 loops=1)
Index Cond: ((stationid = 'WYTOR02'::bpchar) AND ((productid)::text = 'qc60'::text))
Buffers: shared hit=4
-> Hash (cost=10.68..10.68 rows=468 width=27) (actual time=0.547..0.548 rows=468 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 34kB
Buffers: shared hit=6
-> Seq Scan on sensor (cost=0.00..10.68 rows=468 width=27) (actual time=0.013..0.208 rows=468 loops=1)
Buffers: shared hit=6
Planning time: 1.655 ms
Execution time: 1.106 ms
Remote Planner:
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
Hash Join (cost=26.67..102.51 rows=12 width=133) (actual time=0.644..0.719 rows=12 loops=1)
Hash Cond: ((sensor_lookup.sensorid)::text = (sensor.sensorid)::text)
Buffers: shared hit=29
-> Nested Loop (cost=9.14..84.82 rows=12 width=119) (actual time=0.161..0.227 rows=12 loops=1)
Buffers: shared hit=19
-> Nested Loop (cost=4.60..38.12 rows=1 width=108) (actual time=0.128..0.187 rows=1 loops=1)
Buffers: shared hit=15
-> Nested Loop (cost=4.60..37.03 rows=1 width=98) (actual time=0.116..0.173 rows=1 loops=1)
Buffers: shared hit=14
-> Nested Loop (cost=0.27..17.80 rows=1 width=68) (actual time=0.081..0.132 rows=1 loops=1)
Buffers: shared hit=7
-> Index Scan using meta_pkey on meta (cost=0.27..8
.29 rows=1 width=45) (actual time=0.011..0.012 rows=1 loops=1)
Index Cond: (stationid = 'WYTOR02'::bpchar)
Buffers: shared hit=3
-> Seq Scan on meta_lookup (cost=0.00..9.50 rows=1 width=31) (actual time=0.067..0.117 rows=1 loops=1)
Filter: ((stationid)::bpchar = 'WYTOR02'::bpchar)
Rows Removed by Filter: 439
Buffers: shared hit=4
-> Bitmap Heap Scan on datetime_lookup (cost=4.33..19.22 rows=1 width=38) (actual time=0.031..0.036 rows=1 loops=1)
Recheck Cond: (stationid = 'WYTOR02'::bpchar)
Filter: ((productid)::text = 'qc60'::text)
Rows Removed by Filter: 5
Heap Blocks: exact=5
Buffers: shared hit=7
-> Bitmap Index Scan on idx_16 (cost=0.00..4.33 rows=6 width=0) (actual time=0.019..0.019 rows=6 loops=1)
Index Cond: (stationid = 'WYTOR02'::bpchar)
Buffers: shared hit=2
-> Seq Scan on product (cost=0.00..1.07 rows=1 width=10) (actual time=0.010..0.012 rows=1 loops=1)
Filter: ((productid)::text = 'qc60'::text)
Rows Removed by Filter: 5
Buffers: shared hit=1
-> Bitmap Heap Scan on sensor_lookup (cost=4.54..46.58 rows=12 width=30) (actual time=0.030..0.032 rows=12 loops=1)
Recheck Cond: ((stationid = 'WYTOR02'::bpchar) AND ((productid)::text = 'qc60'::text))
Heap Blocks: exact=1
Buffers: shared hit=4
-> Bitmap Index Scan on idx_15 (cost=0.00..4.54 rows=12 width=0) (actual time=0.021..0.021 rows=12 loops=1)
Index Cond: ((stationid = 'WYTOR02'::bpchar) AND ((productid)::text = 'qc60'::text))
Buffers: shared hit=3
-> Hash (cost=11.68..11.68 rows=468 width=27) (actual time=0.440..0.440 rows=468 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 34kB
Buffers: shared hit=7
-> Seq Scan on sensor (cost=0.00..11.68 rows=468 width=27) (actual time=0.004..0.174 rows=468 loops=1)
Buffers: shared hit=7
Planning time: 2.572 ms
Execution time: 0.947 ms
Even though the difference is 1ms, these calls are done thousands of time so the difference adds up. The difference seems to ne that the reomote is doing a Bitmap Heap Scan as opposed to an Index Scan. Though I'm not sure these differences account for the planning time it is a difference between matching systems. The settings in the postgresql.conf are the same so what can I look at to see why these are different?
Both the local and remote servers have the same Postgresql and Ubuntu versions:
Ubuntu 18.04.1
psql (PostgreSQL) 10.15 (Ubuntu 10.15-0ubuntu0.18.04.1)