Why postgres choose wrong execution plan - postgresql

Question

I have a simple query
select count(*)
from taxi_order.ta_orders o
inner join public.t_bases b on b.id = o.id_base
where o.c_phone2 = '012356789'
and b.id_organization = 1
and o.c_date_end < '2017-12-01'::date
group by date_trunc('month', o.c_date_end);
Most of time this query runs fast in less then 100 ms but sometimes it starts run very slow up to 4 seconds for some c_phone2, id_organization combinations.
Execution plan for fast case:
HashAggregate (cost=7005.05..7005.62 rows=163 width=8)
Group Key: date_trunc('month'::text, o.c_date_end)
-> Hash Join (cost=94.30..7004.23 rows=163 width=8)
Hash Cond: (o.id_base = b.id)
-> Index Scan using ix_ta_orders_c_phone2 on ta_orders o (cost=0.57..6899.41 rows=2806 width=12)
Index Cond: ((c_phone2)::text = $3)
Filter: (c_date_end < $4)
-> Hash (cost=93.26..93.26 rows=133 width=4)
-> Bitmap Heap Scan on t_bases b (cost=4.71..93.26 rows=133 width=4)
Recheck Cond: (id_organization = $2)
-> Bitmap Index Scan on ix_t_bases_id_organization (cost=0.00..4.68 rows=133 width=0)
Index Cond: (id_organization = $2)
Execution plan for slow case:
HashAggregate (cost=6604.97..6604.98 rows=1 width=8)
Group Key: date_trunc('month'::text, o.c_date_end)
-> Nested Loop (cost=2195.33..6604.97 rows=1 width=8)
-> Bitmap Heap Scan on t_bases b (cost=2.29..7.78 rows=3 width=4)
Recheck Cond: (id_organization = $2)
-> Bitmap Index Scan on ix_t_bases_id_organization (cost=0.00..2.29 rows=3 width=0)
Index Cond: (id_organization = $2)
-> Bitmap Heap Scan on ta_orders o (cost=2193.04..2199.06 rows=3 width=12)
Recheck Cond: (((c_phone2)::text = $3) AND (id_base = b.id) AND (c_date_end < $4))
-> BitmapAnd (cost=2193.04..2193.04 rows=3 width=0)
-> Bitmap Index Scan on ix_ta_orders_c_phone2 (cost=0.00..58.84 rows=3423 width=0)
Index Cond: ((c_phone2)::text = $3)
-> Bitmap Index Scan on ix_ta_orders_id_base_date_end (cost=0.00..2133.66 rows=83472 width=0)
Index Cond: ((id_base = b.id) AND (c_date_end < $4))
Why query planer chooses so slow ineffective plan sometimes?
EDIT
Schema for tables:
craete table taxi_order.ta_orders (
id bigserial not null,
id_base integer not null,
c_phone2 character varying(30),
c_date_end timestamp with time zone,
...
CONSTRAINT pk_ta_orders PRIMARY KEY (id),
CONSTRAINT fk_ta_orders_t_bases REFERENCES public.t_bases (id)
);
craete table public.t_bases (
id serial not null,
id_organization integer not null,
...
CONSTRAINT pk_t_bases PRIMARY KEY (id)
);
ta_orders ~ 100M rows, t_bases ~ 2K rows.
EDIT2
Explain analyze for slow case:
HashAggregate (cost=6355.29..6355.29 rows=1 width=8) (actual time=4075.847..4075.847 rows=1 loops=1)
Group Key: date_trunc('month'::text, o.c_date_end)
-> Nested Loop (cost=2112.10..6355.28 rows=1 width=8) (actual time=114.871..4075.803 rows=2 loops=1)
-> Bitmap Heap Scan on t_bases b (cost=2.29..7.78 rows=3 width=4) (actual time=0.061..0.375 rows=133 loops=1)
Recheck Cond: (id_organization = $2)
Heap Blocks: exact=45
-> Bitmap Index Scan on ix_t_bases_id_organization (cost=0.00..2.29 rows=3 width=0) (actual time=0.045..0.045 rows=133 loops=1)
Index Cond: (id_organization = $2)
-> Bitmap Heap Scan on ta_orders o (cost=2109.81..2115.83 rows=3 width=12) (actual time=30.638..30.638 rows=0 loops=133)
Recheck Cond: (((c_phone2)::text = $3) AND (id_base = b.id) AND (c_date_end < $4))
Heap Blocks: exact=2
-> BitmapAnd (cost=2109.81..2109.81 rows=3 width=0) (actual time=30.635..30.635 rows=0 loops=133)
-> Bitmap Index Scan on ix_ta_orders_c_phone2 (cost=0.00..58.85 rows=3427 width=0) (actual time=0.032..0.032 rows=6 loops=133)
Index Cond: ((c_phone2)::text = $3)
-> Bitmap Index Scan on ix_ta_orders_id_base_date_end (cost=0.00..2050.42 rows=80216 width=0) (actual time=30.108..30.108 rows=94206 loops=133)
Index Cond: ((id_base = b.id) AND (c_date_end < $4))
Explain analyze for fast case:
HashAggregate (cost=7005.05..7005.62 rows=163 width=8) (actual time=0.927..0.928 rows=1 loops=1)
Group Key: date_trunc('month'::text, o.c_date_end)
-> Hash Join (cost=94.30..7004.23 rows=163 width=8) (actual time=0.903..0.913 rows=2 loops=1)
Hash Cond: (o.id_base = b.id)
-> Index Scan using ix_ta_orders_c_phone2 on ta_orders o (cost=0.57..6899.41 rows=2806 width=12) (actual time=0.591..0.604 rows=4 loops=1)
Index Cond: ((c_phone2)::text = $3)
Filter: (c_date_end < $4)
Rows Removed by Filter: 2
-> Hash (cost=93.26..93.26 rows=133 width=4) (actual time=0.237..0.237 rows=133 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 13kB
-> Bitmap Heap Scan on t_bases b (cost=4.71..93.26 rows=133 width=4) (actual time=0.058..0.196 rows=133 loops=1)
Recheck Cond: (id_organization = $2)
Heap Blocks: exact=45
-> Bitmap Index Scan on ix_t_bases_id_organization (cost=0.00..4.68 rows=133 width=0) (actual time=0.044..0.044 rows=133 loops=1)
Index Cond: (id_organization = $2)
I know I can create separate index for every query to speed it up. But I want to know what is the reason for choosing wrong plan? What is wrong with my statistic?

Answer 1

You'd have to give us EXPLAIN (ANALYZE, BUFFERS) output for a definitive answer.
The difference between the plans is that the second plan chooses a nested loop join because it estimates that only very few rows will be selected from t_bases. Since you complain that the query is slow, that estimate is probably wrong, resulting in too many loops over the inner table.
Try to improve your table statistics by running ANALYZE, perhaps after increasing default_statistics_target.
A multi-column index on ta_orders(c_phone2, id_base, c_date_end) would improve the execution time for the nested loop plan.

Answer 2

Not sure, but I can suggest a possible improvement to your query: remove the inner join. You're not selecting anything from that table, so why bother querying it? You should be able to add where o.id_base = ? to your query.
If you want this query to run quickly every time you should add the following index to ta_orders: (id_base, c_phone2, c_date_end). It's important that the column with the > or < where clause is at the end (otherwise Postgres will not be able to make use of it).