Very slow search for NULL values with index - postgresql

Question

I have a Postgres table with ~50 columns and ~75 million records.
It has the following index among others:
"index_shipments_on_buyer_supplier_id" btree (buyer_supplier_id)
EXPLAIN shows it wants to use a sequential scan:
db=# EXPLAIN SELECT COUNT(*) FROM "shipments" WHERE (buyer_supplier_id IS NULL)
db-# ;
QUERY PLAN
--------------------------------------------------------------------------------------------------
Finalize Aggregate (cost=15427130.32..15427130.33 rows=1 width=8)
-> Gather (cost=15427130.11..15427130.32 rows=2 width=8)
Workers Planned: 2
-> Partial Aggregate (cost=15426130.11..15426130.12 rows=1 width=8)
-> Parallel Seq Scan on shipments (cost=0.00..15354385.03 rows=28698029 width=0)
Filter: (buyer_supplier_id IS NULL)
(6 rows)
Now force use of the index:
db=# set enable_seqscan = false;
SET
db=# EXPLAIN SELECT COUNT(*) FROM "shipments" WHERE (buyer_supplier_id IS NULL);
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------
Finalize Aggregate (cost=17314493.48..17314493.49 rows=1 width=8)
-> Gather (cost=17314493.26..17314493.47 rows=2 width=8)
Workers Planned: 2
-> Partial Aggregate (cost=17313493.26..17313493.27 rows=1 width=8)
-> Parallel Bitmap Heap Scan on shipments (cost=1922711.90..17241748.19 rows=28698029 width=0)
Recheck Cond: (buyer_supplier_id IS NULL)
-> Bitmap Index Scan on index_shipments_on_buyer_supplier_id (cost=0.00..1905493.08 rows=68875269 width=0)
Index Cond: (buyer_supplier_id IS NULL)
(8 rows)
db=# EXPLAIN ANALYZE SELECT COUNT(*) FROM "shipments" WHERE (buyer_supplier_id IS NULL);
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Finalize Aggregate (cost=17314493.48..17314493.49 rows=1 width=8) (actual time=795551.977..795573.311 rows=1 loops=1)
-> Gather (cost=17314493.26..17314493.47 rows=2 width=8) (actual time=795528.063..795573.304 rows=3 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Partial Aggregate (cost=17313493.26..17313493.27 rows=1 width=8) (actual time=795519.276..795519.277 rows=1 loops=3)
-> Parallel Bitmap Heap Scan on shipments (cost=1922711.90..17241748.19 rows=28698029 width=0) (actual time=7642.771..794473.494 rows=5439073 loops=3)
Recheck Cond: (buyer_supplier_id IS NULL)
Rows Removed by Index Recheck: 10948389
Heap Blocks: exact=14343 lossy=3993510
-> Bitmap Index Scan on index_shipments_on_buyer_supplier_id (cost=0.00..1905493.08 rows=68875269 width=0) (actual time=7633.652..7633.652 rows=62174015 loops=1)
Index Cond: (buyer_supplier_id IS NULL)
Planning time: 0.102 ms
Execution time: 795573.347 ms
(13 rows)
I don't understand why getting a COUNT of NULL buyer_supplier_ids should be so taxing on the system. What am I missing here, and how can I make this count fast?

Answer 1

Postgres organizes indexes with nulls placed last by default. Check https://www.postgresql.org/docs/current/indexes-ordering.html for more info
In your case, if the table has high cardinality for buyers_supplier_id it will have to scroll through the entire index to look for nulls hence the planner might be deciding to use seq scan.
To fix this
You can either recreate the index with nulls first option or you can also create a partial index with buyers_supplier_id is null condition as #a_horse_with_no_name mentioned.
Another thing to look into is index bloat. If this table is frequently getting updated and has not been through a vacuum index might start getting bloated reducing the performance.