postgres seq scan on two partitions - postgresql

Question

When ch table is queried direcly on rp_fk column, it uses index for almost all of its partitions.
explain analyze select * from oc.ch ch where rp_fk = 'abc123';
Append (cost=0.14..9469.61 rows=2357 width=10008) (actual time=0.164..0.166 rows=0 loops=1)
-> Index Scan using ch_2016_rp_fk_idx1 on ch_2016 ch (cost=0.14..8.16 rows=1 width=11097) (actual time=0.015..0.016 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Index Scan using ch_2017_rp_fk_idx1 on ch_2017 ch_1 (cost=0.14..8.16 rows=1 width=10477) (actual time=0.022..0.022 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Index Scan using ch_2018_rp_fk_idx1 on ch_2018 ch_2 (cost=0.56..4.58 rows=1 width=12295) (actual time=0.031..0.031 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Index Scan using ch_2019_rp_fk_idx1 on ch_2019 ch_3 (cost=0.69..4.71 rows=1 width=12299) (actual time=0.025..0.025 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Index Scan using ch_2020_rp_fk_idx1 on ch_2020 ch_4 (cost=0.69..6030.67 rows=1510 width=10259) (actual time=0.029..0.029 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Bitmap Heap Scan on ch_2021 ch_5 (cost=55.08..3392.36 rows=841 width=9566) (actual time=0.018..0.019 rows=0 loops=1)
Recheck Cond: (rp_fk = 'abc123'::bpchar)
-> Bitmap Index Scan on ch_2021_rp_fk_idx1 (cost=0.00..54.87 rows=841 width=0) (actual time=0.016..0.016 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Index Scan using ch_2022_rp_fk_idx on ch_2022 ch_6 (cost=0.12..8.14 rows=1 width=12241) (actual time=0.013..0.013 rows=0 loops=1)
Index Cond: (rp_fk = 'abc123'::bpchar)
-> Seq Scan on ch_def ch_7 (cost=0.00..1.05 rows=1 width=12167) (actual time=0.009..0.009 rows=0 loops=1)
Filter: (rp_fk = 'abc123'::bpchar)
Rows Removed by Filter: 4
Planning Time: 24.637 ms
Execution Time: 0.538 ms
But when joining ch table with p table from where it gets the rp_fk, its doing a sequential scan of the ch_2018 and ch_2019 partitions. This is where its taking most of the time, whereas 2020 and 2021 paritions are scanned using index. Any insight as to why its using seq scan on these 2 partitions? All the partitions have same indexes and have upto date vacuum analyze. random_page_cost is set to 1.
explain analyze select * FROM oc.ch ch JOIN op.p p ON ch.rp_fk = p.p_pk WHERE ch.del_flg = '0'::bpchar and hf_p_num_cd ='112642817-002';
Nested Loop (cost=0.42..13134696.50 rows=96 width=12593) (actual time=193878.564..193878.569 rows=0 loops=1)
-> Index Scan using ix_p_hf_p_num_cd on p p (cost=0.42..8.44 rows=1 width=1420) (actual time=0.025..0.028 rows=1 loops=1)
Index Cond: ((hf_p_num_cd)::text = '112642817-002'::text)
-> Append (cost=0.00..12666318.27 rows=46836978 width=11167) (actual time=193878.532..193878.535 rows=0 loops=1)
-> Seq Scan on ch_2016 ch (cost=0.00..11.57 rows=13 width=11097) (actual time=0.072..0.072 rows=0 loops=1)
Filter: ((del_flg = '0'::bpchar) AND (p.p_pk = rp_fk))
Rows Removed by Filter: 38
-> Index Scan using ch_2017_rp_fk_idx1 on ch_2017 ch_1 (cost=0.14..20.94 rows=40 width=10477) (actual time=0.016..0.016 rows=0 loops=1)
Index Cond: (rp_fk = p.p_pk)
Filter: (del_flg = '0'::bpchar)
-> Seq Scan on ch_2018 ch_2 (cost=0.00..4005273.43 rows=15109962 width=12295) (actual time=25051.825..25051.825 rows=0 loops=1)
Filter: ((del_flg = '0'::bpchar) AND (p.p_pk = rp_fk))
Rows Removed by Filter: 15111645
-> Seq Scan on ch_2019 ch_3 (cost=0.00..8406638.77 rows=31721918 width=12299) (actual time=168826.481..168826.481 rows=0 loops=1)
Filter: ((del_flg = '0'::bpchar) AND (p.p_pk = rp_fk))
Rows Removed by Filter: 31722679
-> Index Scan using ch_2020_rp_fk_idx1 on ch_2020 ch_4 (cost=0.69..14893.73 rows=3729 width=10259) (actual time=0.057..0.057 rows=0 loops=1)
Index Cond: (rp_fk = p.p_pk)
Filter: (del_flg = '0'::bpchar)
-> Bitmap Heap Scan on ch_2021 ch_5 (cost=82.74..5285.74 rows=1314 width=9566) (actual time=0.033..0.033 rows=0 loops=1)
Recheck Cond: (rp_fk = p.p_pk)
Filter: (del_flg = '0'::bpchar)
-> Bitmap Index Scan on ch_2021_rp_fk_idx1 (cost=0.00..82.42 rows=1314 width=0) (actual time=0.022..0.022 rows=0 loops=1)
Index Cond: (rp_fk = p.p_pk)
-> Index Scan using ch_2022_rp_fk_idx on ch_2022 ch_6 (cost=0.12..8.14 rows=1 width=12241) (actual time=0.014..0.014 rows=0 loops=1)
Filter: ((del_flg = '0'::bpchar) AND (p.p_pk = rp_fk))
-> Seq Scan on ch_def ch_7 (cost=0.00..1.06 rows=1 width=12167) (actual time=0.017..0.017 rows=0 loops=1)
Filter: ((del_flg = '0'::bpchar) AND (p.p_pk = rp_fk))
Rows Removed by Filter: 4
Planning Time: 4.183 ms
Execution Time: 193878.918 ms

Answer 1

It seems like 'abc123' is a very rare value for rp_fk, so PostgreSQL plans an index scan.
But it seems there are values that are much more frequent. In the second query, the optimizer does not know what value the rp_fk for hf_p_num_cd = '112642817-002' is going to be, so it goes with some average estimate, which turns out to be much higher for many partitions. Hence the sequential scan.
I would split the query in two parts and query the partitioned table with the constants found in the first query. Then the optimizer knows more and will plan better.
If you are certain that an index scan will always win, you can force the planner's hand:
BEGIN;
SET LOCAL enable_seqscan = off;
SET LOCAL enable_bitmapscan = off;
SELECT ...
COMMIT;