postgres IN clause not using index - postgresql

When I am using IN, the query is not using index and scanning mview on which view1 is built. But same query when = is used, uses index and returns fast. How to make IN use index and complete fast.
Here its using "=" clause
explain analyze select * from schema1.view1 cr
where c_cd = (select distinct c_cd from schema1.mview1 where cd = 'XY87296C');
Subquery Scan on res (cost=110.44..110.52 rows=1 width=271) (actual time=0.130..0.139 rows=5 loops=1)
Filter: (res.rnk = 1)
InitPlan 1 (returns $0)
-> Unique (cost=60.80..60.90 rows=20 width=6) (actual time=0.041..0.045 rows=1 loops=1)
-> Sort (cost=60.80..60.85 rows=20 width=6) (actual time=0.041..0.042 rows=3 loops=1)
Sort Key: mview1.c_cd
Sort Method: quicksort Memory: 25kB
-> Index Scan using ix_cd_n on mview1 (cost=0.43..60.37 rows=20 width=6) (actual time=0.030..0.034 rows=3 loops=1)
Index Cond: ((cd)::text = 'XY87296C'::text)
-> WindowAgg (cost=49.53..49.59 rows=2 width=1287) (actual time=0.128..0.134 rows=5 loops=1)
-> Sort (cost=49.53..49.54 rows=2 width=271) (actual time=0.119..0.120 rows=5 loops=1)
Sort Key: cerer.rer_cd, cerer.r_dt DESC, cerer.rere_dt DESC, cerer.rere1_dt DESC
Sort Method: quicksort Memory: 27kB
-> Nested Loop (cost=0.84..49.52 rows=2 width=271) (actual time=0.101..0.107 rows=5 loops=1)
Join Filter: (cerer.r_dt = mx.r_dt)
-> Index Scan using ix_c_cd on cerer cerer (cost=0.42..24.51 rows=5 width=271) (actual time=0.069..0.070 rows=5 loops=1)
Index Cond: ((c_cd)::text = ($0)::text)
-> Materialize (cost=0.42..24.66 rows=5 width=13) (actual time=0.006..0.006 rows=1 loops=5)
-> Subquery Scan on mx (cost=0.42..24.63 rows=5 width=13) (actual time=0.026..0.027 rows=1 loops=1)
-> GroupAggregate (cost=0.42..24.58 rows=5 width=13) (actual time=0.026..0.026 rows=1 loops=1)
Group Key: cerer.c_cd
-> Index Scan using ix_c_cd on cerer (cost=0.42..24.51 rows=5 width=13) (actual time=0.011..0.018 rows=5 loops=1)
Index Cond: ((c_cd)::text = ($0)::text)
Planning Time: 0.812 ms
Execution Time: 0.471 ms
Here its using IN clause
explain analyze select * from schema1.view1 cr
where c_cd in (select distinct c_cd from schema1.mview1 where cd = 'XY87296C');
Merge Join (cost=74280.97..74345.71 rows=1 width=271) (actual time=1415.760..1415.861 rows=5 loops=1)
Merge Cond: ((res.c_cd)::text = (mview1.c_cd)::text)
-> Subquery Scan on res (cost=74220.16..74284.52 rows=8 width=271) (actual time=1214.729..1395.978 rows=144485 loops=1)
Filter: (res.rnk = 1)
Rows Removed by Filter: 534
-> WindowAgg (cost=74220.16..74264.41 rows=1609 width=1287) (actual time=1214.726..1383.997 rows=145019 loops=1)
-> Sort (cost=74220.16..74224.19 rows=1609 width=271) (actual time=1214.706..1239.776 rows=145020 loops=1)
Sort Key: cerer.c_cd, cerer.rer_cd, cerer.r_dt DESC, cerer.rere_dt DESC, cerer.rere1_dt DESC
Sort Method: quicksort Memory: 111598kB
-> Gather (cost=40122.36..74134.47 rows=1609 width=271) (actual time=308.210..409.188 rows=205422 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Hash Join (cost=39122.36..72973.57 rows=670 width=271) (actual time=301.493..400.779 rows=68474 loops=3)
Hash Cond: (((cerer.c_cd)::text = (mx.c_cd)::text) AND (cerer.r_dt = mx.r_dt))
-> Parallel Seq Scan on cerer cerer (cost=0.00..33147.12 rows=134112 width=271) (actual time=0.004..20.591 rows=107290 loops=3)
-> Hash (cost=38055.93..38055.93 rows=71095 width=13) (actual time=300.894..300.896 rows=108049 loops=3)
Buckets: 131072 Batches: 1 Memory Usage: 6089kB
-> Subquery Scan on mx (cost=36634.04..38055.93 rows=71095 width=13) (actual time=246.132..278.589 rows=108049 loops=3)
-> HashAggregate (cost=36634.04..37344.99 rows=71095 width=13) (actual time=246.130..269.294 rows=108049 loops=3)
Group Key: cerer.c_cd
-> Seq Scan on cerer (cost=0.00..35024.69 rows=321869 width=13) (actual time=0.019..60.328 rows=321869 loops=3)
-> Unique (cost=60.80..60.90 rows=20 width=6) (actual time=0.056..0.060 rows=1 loops=1)
-> Sort (cost=60.80..60.85 rows=20 width=6) (actual time=0.055..0.057 rows=3 loops=1)
Sort Key: mview1.c_cd
Sort Method: quicksort Memory: 25kB
-> Index Scan using ix_consume_customer_health_coverage_cd_n on mview1 (cost=0.43..60.37 rows=20 width=6) (actual time=0.040..0.045 rows=3 loops=1)
Index Cond: ((cd)::text = 'XY87296C'::text)
Planning Time: 0.770 ms
Execution Time: 1424.928 ms

Related

Postgres hash join batches explosion

We are having some struggle identifying why Postgres is using too much batches to resolve a join.
Here it is the output of explain analyze of a problematic execution:
https://explain.dalibo.com/plan/xNJ#plan
Limit (cost=20880.87..20882.91 rows=48 width=205) (actual time=10722.953..10723.358 rows=48 loops=1)
-> Unique (cost=20880.87..21718.12 rows=19700 width=205) (actual time=10722.951..10723.356 rows=48 loops=1)
-> Sort (cost=20880.87..20930.12 rows=19700 width=205) (actual time=10722.950..10722.990 rows=312 loops=1)
Sort Key: titlemetadata_titlemetadata.creation_date DESC, titlemetadata_titlemetadata.id, titlemetadata_titlemetadata.title_type, titlemetadata_titlemetadata.original_title, titlemetadata_titlemetadata.alternative_ids, titlemetadata_titlemetadata.metadata,
titlemetadata_titlemetadata.is_adult, titlemetadata_titlemetadata.is_kids, titlemetadata_titlemetadata.last_modified, titlemetadata_titlemetadata.year, titlemetadata_titlemetadata.runtime, titlemetadata_titlemetadata.rating, titlemetadata_titlemetadata.video_provider, tit
lemetadata_titlemetadata.series_id_id, titlemetadata_titlemetadata.season_number, titlemetadata_titlemetadata.episode_number
Sort Method: quicksort Memory: 872kB
-> Hash Right Join (cost=13378.20..19475.68 rows=19700 width=205) (actual time=1926.352..10709.970 rows=2909 loops=1)
Hash Cond: (t4.titlemetadata_id = t3.id)
Filter: ((hashed SubPlan 1) OR (hashed SubPlan 2))
Rows Removed by Filter: 63248
-> Seq Scan on video_provider_offer t4 (cost=0.00..5454.90 rows=66290 width=16) (actual time=0.024..57.893 rows=66390 loops=1)
-> Hash (cost=11314.39..11314.39 rows=22996 width=221) (actual time=489.530..489.530 rows=60096 loops=1)
Buckets: 65536 (originally 32768) Batches: 32768 (originally 1) Memory Usage: 11656kB
-> Hash Right Join (cost=5380.95..11314.39 rows=22996 width=221) (actual time=130.024..225.271 rows=60096 loops=1)
Hash Cond: (video_provider_offer.titlemetadata_id = titlemetadata_titlemetadata.id)
-> Seq Scan on video_provider_offer (cost=0.00..5454.90 rows=66290 width=16) (actual time=0.011..32.950 rows=66390 loops=1)
-> Hash (cost=5129.28..5129.28 rows=20133 width=213) (actual time=129.897..129.897 rows=55793 loops=1)
Buckets: 65536 (originally 32768) Batches: 2 (originally 1) Memory Usage: 7877kB
-> Merge Left Join (cost=1.72..5129.28 rows=20133 width=213) (actual time=0.041..93.057 rows=55793 loops=1)
Merge Cond: (titlemetadata_titlemetadata.id = t3.series_id_id)
-> Index Scan using titlemetadata_titlemetadata_pkey on titlemetadata_titlemetadata (cost=1.30..4130.22 rows=20133 width=205) (actual time=0.028..62.949 rows=43921 loops=1)
Filter: ((NOT is_adult) AND (NOT (hashed SubPlan 3)) AND (((title_type)::text = 'MOV'::text) OR ((title_type)::text = 'TVS'::text) OR ((title_type)::text = 'TVP'::text) OR ((title_type)::text = 'EVT'::text)))
Rows Removed by Filter: 14121
SubPlan 3
-> Seq Scan on cable_operator_cableoperatorexcludedtitle u0_2 (cost=0.00..1.01 rows=1 width=8) (actual time=0.006..0.006 rows=0 loops=1)
Filter: (cable_operator_id = 54)
-> Index Scan using titlemetadata_titlemetadata_series_id_id_73453db4_uniq on titlemetadata_titlemetadata t3 (cost=0.41..3901.36 rows=58037 width=16) (actual time=0.011..9.375 rows=12887 loops=1)
SubPlan 1
-> Hash Join (cost=44.62..885.73 rows=981 width=8) (actual time=0.486..36.806 rows=5757 loops=1)
Hash Cond: (w2.device_id = w3.id)
-> Nested Loop (cost=43.49..866.20 rows=2289 width=16) (actual time=0.441..33.096 rows=20180 loops=1)
-> Nested Loop (cost=43.06..414.98 rows=521 width=8) (actual time=0.426..9.952 rows=2909 loops=1)
Join Filter: (w1.id = w0.video_provider_id)
-> Nested Loop (cost=42.65..54.77 rows=13 width=24) (actual time=0.399..0.532 rows=15 loops=1)
-> HashAggregate (cost=42.50..42.95 rows=45 width=16) (actual time=0.390..0.403 rows=45 loops=1)
Group Key: v0.id
-> Nested Loop (cost=13.34..42.39 rows=45 width=16) (actual time=0.095..0.364 rows=45 loops=1)
-> Hash Semi Join (cost=13.19..32.72 rows=45 width=8) (actual time=0.084..0.229 rows=45 loops=1)
Hash Cond: (v1.id = u0.id)
-> Seq Scan on cable_operator_cableoperatorprovider v1 (cost=0.00..17.36 rows=636 width=16) (actual time=0.010..0.077 rows=636 loops=1)
-> Hash (cost=12.63..12.63 rows=45 width=8) (actual time=0.046..0.046 rows=45 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Index Scan using cable_operator_cableoperatorprovider_4d6e54b3 on cable_operator_cableoperatorprovider u0 (cost=0.28..12.63 rows=45 width=8) (actual time=0.016..0.035 rows=45 loops=1)
Index Cond: (cable_operator_id = 54)
-> Index Only Scan using video_provider_videoprovider_pkey on video_provider_videoprovider v0 (cost=0.15..0.20 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=45)
Index Cond: (id = v1.provider_id)
Heap Fetches: 45
-> Index Scan using video_provider_videoprovider_pkey on video_provider_videoprovider w1 (cost=0.15..0.25 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (id = v0.id)
Filter: ((video_provider_type)::text = 'VOD'::text)
Rows Removed by Filter: 1
-> Index Scan using video_provider_offer_da942d2e on video_provider_offer w0 (cost=0.42..27.22 rows=39 width=16) (actual time=0.026..0.585 rows=194 loops=15)
Index Cond: (video_provider_id = v0.id)
Filter: (((end_date > '2021-09-02 19:23:00-03'::timestamp with time zone) OR (end_date IS NULL)) AND (access_criteria && '{vtv_mas,TBX_LOGIN,urn:spkg:tve:fox-premium,urn:tve:mcp,AMCHD,AMC_CONSORCIO,ANIMAL_PLANET,ASUNTOS_PUBLI
COS,ASUNTOS_PUBLICOS_CONSORCIO,CINECANALLIVE,CINECANAL_CONSORCIO,DISCOVERY,DISCOVERY_KIDS_CONSORCIO,DISCOVERY_KIDS_OD,DISNEY,DISNEY_CH_CONSORCIO,DISNEY_XD,DISNEY_XD_CONSORCIO,EL_CANAL_HD,EL_CANAL_HD_CONSORCIO,EL_GOURMET_CONSORCIO,ESPN,ESPN2_HD_CONSORCIO,ESPN3_HD_CONSORCIO
,ESPNMAS_HD_CONSORCIO,ESPN_BASIC,ESPN_HD_CONSORCIO,ESPN_PLAY,EUROPALIVE,EUROPA_EUROPA,EUROPA_EUROPA_CONSORCIO,FILMANDARTS_DISPOSITIVOS,FILMS_ARTS,FILM_AND_ARTS_CONSORCIO,FOXLIFE,FOX_LIFE_CONSORCIO,FOX_SPORTS_1_DISPOSITIVOS,FOX_SPORTS_2_DISPOSITIVOS,FOX_SPORTS_2_HD_CONSORC
IO,FOX_SPORTS_3_DISPOSITIVOS,FOX_SPORTS_3_HD_CONSORCIO,FOX_SPORTS_HD_CONSORCIO,FRANCE24_DISPOSITIVOS,FRANCE_24_CONSORCIO,GOURMET,GOURMET_DISPOSITIVOS,HOME_HEALTH,INVESTIGATION_DISCOVERY,MAS_CHIC,NATGEOKIDS_DISPOSITIVOS,NATGEO_CONSORCIO,NATGEO_DISPOSITIVOS,NATGEO_KIDS_CONS
ORCIO,PASIONES,PASIONES_CONSORCIO,SVOD_TYC_BASIC,TBX_LOGIN,TCC_2_CONSORCIO,TCC_2_HD,TLC,TVE,TVE_CONSORCIO,TYC_SPORTS_CONSORCIO,VTV_LIVE,clarosports,discoverykids,espnplay_south_alt,urn:spkg:tve:fox-basic,urn:tve:babytv,urn:tve:cinecanal,urn:tve:discoverykids,urn:tve:foxli
fe,urn:tve:fp,urn:tve:fx,urn:tve:natgeo,urn:tve:natgeokids,urn:tve:natgeowild,urn:tve:thefilmzone}'::character varying(50)[]) AND ((((content_type)::text = 'VOD'::text) AND ((start_date < '2021-09-02 19:23:00-03'::timestamp with time zone) OR (start_date IS NULL))) OR ((c
ontent_type)::text = 'LIV'::text)))
Rows Removed by Filter: 5
-> Index Only Scan using video_provider_offer_devices_offer_id_device_id_key on video_provider_offer_devices w2 (cost=0.42..0.81 rows=6 width=16) (actual time=0.004..0.007 rows=7 loops=2909)
Index Cond: (offer_id = w0.id)
Heap Fetches: 17828
-> Hash (cost=1.10..1.10 rows=3 width=8) (actual time=0.029..0.029 rows=2 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on platform_device_device w3 (cost=0.00..1.10 rows=3 width=8) (actual time=0.024..0.027 rows=2 loops=1)
Filter: ((device_code)::text = ANY ('{ANDROID,ott_dual_tcc,ott_k2_tcc}'::text[]))
Rows Removed by Filter: 5
SubPlan 2
-> Hash Join (cost=44.62..885.73 rows=981 width=8) (actual time=0.410..33.580 rows=5757 loops=1)
Hash Cond: (w2_1.device_id = w3_1.id)
-> Nested Loop (cost=43.49..866.20 rows=2289 width=16) (actual time=0.375..29.886 rows=20180 loops=1)
-> Nested Loop (cost=43.06..414.98 rows=521 width=8) (actual time=0.366..9.134 rows=2909 loops=1)
Join Filter: (w1_1.id = w0_1.video_provider_id)
-> Nested Loop (cost=42.65..54.77 rows=13 width=24) (actual time=0.343..0.476 rows=15 loops=1)
-> HashAggregate (cost=42.50..42.95 rows=45 width=16) (actual time=0.333..0.347 rows=45 loops=1)
Group Key: v0_1.id
-> Nested Loop (cost=13.34..42.39 rows=45 width=16) (actual time=0.083..0.311 rows=45 loops=1)
-> Hash Semi Join (cost=13.19..32.72 rows=45 width=8) (actual time=0.076..0.202 rows=45 loops=1)
Hash Cond: (v1_1.id = u0_1.id)
-> Seq Scan on cable_operator_cableoperatorprovider v1_1 (cost=0.00..17.36 rows=636 width=16) (actual time=0.005..0.057 rows=636 loops=1)
-> Hash (cost=12.63..12.63 rows=45 width=8) (actual time=0.038..0.038 rows=45 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Index Scan using cable_operator_cableoperatorprovider_4d6e54b3 on cable_operator_cableoperatorprovider u0_1 (cost=0.28..12.63 rows=45 width=8) (actual time=0.007..0.020 rows=45 loops=1)
Index Cond: (cable_operator_id = 54)
-> Index Only Scan using video_provider_videoprovider_pkey on video_provider_videoprovider v0_1 (cost=0.15..0.20 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=45)
Index Cond: (id = v1_1.provider_id)
Heap Fetches: 45
-> Index Scan using video_provider_videoprovider_pkey on video_provider_videoprovider w1_1 (cost=0.15..0.25 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (id = v0_1.id)
Filter: ((video_provider_type)::text = 'VOD'::text)
Rows Removed by Filter: 1
-> Index Scan using video_provider_offer_da942d2e on video_provider_offer w0_1 (cost=0.42..27.22 rows=39 width=16) (actual time=0.022..0.536 rows=194 loops=15)
Index Cond: (video_provider_id = v0_1.id)
Filter: (((end_date > '2021-09-02 19:23:00-03'::timestamp with time zone) OR (end_date IS NULL)) AND (access_criteria && '{vtv_mas,TBX_LOGIN,urn:spkg:tve:fox-premium,urn:tve:mcp,AMCHD,AMC_CONSORCIO,ANIMAL_PLANET,ASUNTOS_PUBLI
COS,ASUNTOS_PUBLICOS_CONSORCIO,CINECANALLIVE,CINECANAL_CONSORCIO,DISCOVERY,DISCOVERY_KIDS_CONSORCIO,DISCOVERY_KIDS_OD,DISNEY,DISNEY_CH_CONSORCIO,DISNEY_XD,DISNEY_XD_CONSORCIO,EL_CANAL_HD,EL_CANAL_HD_CONSORCIO,EL_GOURMET_CONSORCIO,ESPN,ESPN2_HD_CONSORCIO,ESPN3_HD_CONSORCIO
,ESPNMAS_HD_CONSORCIO,ESPN_BASIC,ESPN_HD_CONSORCIO,ESPN_PLAY,EUROPALIVE,EUROPA_EUROPA,EUROPA_EUROPA_CONSORCIO,FILMANDARTS_DISPOSITIVOS,FILMS_ARTS,FILM_AND_ARTS_CONSORCIO,FOXLIFE,FOX_LIFE_CONSORCIO,FOX_SPORTS_1_DISPOSITIVOS,FOX_SPORTS_2_DISPOSITIVOS,FOX_SPORTS_2_HD_CONSORC
IO,FOX_SPORTS_3_DISPOSITIVOS,FOX_SPORTS_3_HD_CONSORCIO,FOX_SPORTS_HD_CONSORCIO,FRANCE24_DISPOSITIVOS,FRANCE_24_CONSORCIO,GOURMET,GOURMET_DISPOSITIVOS,HOME_HEALTH,INVESTIGATION_DISCOVERY,MAS_CHIC,NATGEOKIDS_DISPOSITIVOS,NATGEO_CONSORCIO,NATGEO_DISPOSITIVOS,NATGEO_KIDS_CONS
ORCIO,PASIONES,PASIONES_CONSORCIO,SVOD_TYC_BASIC,TBX_LOGIN,TCC_2_CONSORCIO,TCC_2_HD,TLC,TVE,TVE_CONSORCIO,TYC_SPORTS_CONSORCIO,VTV_LIVE,clarosports,discoverykids,espnplay_south_alt,urn:spkg:tve:fox-basic,urn:tve:babytv,urn:tve:cinecanal,urn:tve:discoverykids,urn:tve:foxli
fe,urn:tve:fp,urn:tve:fx,urn:tve:natgeo,urn:tve:natgeokids,urn:tve:natgeowild,urn:tve:thefilmzone}'::character varying(50)[]) AND ((((content_type)::text = 'VOD'::text) AND ((start_date < '2021-09-02 19:23:00-03'::timestamp with time zone) OR (start_date IS NULL))) OR ((c
ontent_type)::text = 'LIV'::text)))
Rows Removed by Filter: 5
-> Index Only Scan using video_provider_offer_devices_offer_id_device_id_key on video_provider_offer_devices w2_1 (cost=0.42..0.81 rows=6 width=16) (actual time=0.003..0.006 rows=7 loops=2909)
Index Cond: (offer_id = w0_1.id)
Heap Fetches: 17828
-> Hash (cost=1.10..1.10 rows=3 width=8) (actual time=0.015..0.015 rows=2 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on platform_device_device w3_1 (cost=0.00..1.10 rows=3 width=8) (actual time=0.010..0.011 rows=2 loops=1)
Filter: ((device_code)::text = ANY ('{ANDROID,ott_dual_tcc,ott_k2_tcc}'::text[]))
Rows Removed by Filter: 5
Planning time: 8.255 ms
Execution time: 10723.830 ms
(100 rows)
The weird part is that the same query, sometimes just uses a single batch. Here is an example: https://explain.dalibo.com/plan/zTv#plan
Here is the work_mem being used:
show work_mem;
work_mem
----------
8388kB
(1 row)
I'm not interested in changing the query to be more performant, but in understanding why is the different behavior.
I've found this thread apparently related with this, but I don't quite understand what are they talking about: https://www.postgresql.org/message-id/flat/CA%2BhUKGKWWmf%3DWELLG%3DaUGbcugRaSQbtm0tKYiBut-B2rVKX63g%40mail.gmail.com
Can anyone tell me why is this different behavior? The underlying data is the same in both cases.

If the hash is done in memory, there will only be a single batch.
A difference with the original hash batch numbers is due to Postgres choosing to increase the number of batches in order to reduce memory consumption.
You might find this EXPLAIN glossary useful (disclaimer: I'm one of the authors), here is the page on Hash Batches which also links to the PostgreSQL source code (it's very nicely documented in plain English).
While not a perfect heuristic, you can see that the memory required for the operations with multiple batches are around or above your work_mem setting. They can be lower than it, due to operations on disk generally requiring less memory overall.
I'm not 100% sure why in your exact case one was chosen over the other, but it does look like there are some very slight row estimate differences, which might be a good place to start.
As of PostgreSQL 13 there is also now a hash_mem_multiplier setting that can be used to give more memory to hashes without doing so for other operations (like sorts).

We where able to solve the problem just by doing VACUUM FULL ANALYZE;.
After that, everything started to work as expected (https://explain.depesz.com/s/eoqH#html)
Side note: we where not aware that we should do this on daily basis.

How does a string operation on a column in a filter condition of a Postgresql query have on the plan it chooses

I was working on optimising a query, with dumb luck I tried something and it improved the query but I am unable to explain why.
Below is the query with poor performance
with ctedata1 as(
select
sum(total_visit_count) as total_visit_count,
sum(sh_visit_count) as sh_visit_count,
sum(ec_visit_count) as ec_visit_count,
sum(total_like_count) as total_like_count,
sum(sh_like_count) as sh_like_count,
sum(ec_like_count) as ec_like_count,
sum(total_order_count) as total_order_count,
sum(sh_order_count) as sh_order_count,
sum(ec_order_count) as ec_order_count,
sum(total_sales_amount) as total_sales_amount,
sum(sh_sales_amount) as sh_sales_amount,
sum(ec_sales_amount) as ec_sales_amount,
sum(ec_order_online_count) as ec_order_online_count,
sum(ec_sales_online_amount) as ec_sales_online_amount,
sum(ec_order_in_store_count) as ec_order_in_store_count,
sum(ec_sales_in_store_amount) as ec_sales_in_store_amount,
table2.im_name,
table2.brand as kpibrand,
table2.id_region as kpiregion
from
table2
where
deleted_at is null
and id_region = any('{1}')
group by
im_name,
kpiregion,
kpibrand ),
ctedata2 as (
select
ctedata1.*,
rank() over (partition by (kpiregion,
kpibrand)
order by
coalesce(ctedata1.total_sales_amount, 0) desc) rank,
count(*) over (partition by (kpiregion,
kpibrand)) as total_count
from
ctedata1 )
select
table1.id_pf_item,
table1.product_id,
table1.color_code,
table1.l1_code,
table1.local_title as product_name,
table1.id_region,
table1.gender,
case
when table1.created_at is null then '1970/01/01 00:00:00'
else table1.created_at
end as created_at,
(
select
count(distinct id_outfit)
from
table3
left join table4 on
table3.id_item = table4.id_item
and table4.deleted_at is null
where
table3.deleted_at is null
and table3.id_pf_item = table1.id_pf_item) as outfit_count,
count(*) over() as total_matched,
case
when table1.v8_im_name = '' then table1.im_name
else table1.v8_im_name
end as im_name,
case
when table1.id_region != 1 then null
else
case
when table1.sales_start_at is null then '1970/01/01 00:00:00'
else table1.sales_start_at
end
end as sales_start_date,
table1.category_ids,
array_to_string(table1.intermediate_category_ids, ','),
table1.image_url,
table1.brand,
table1.pdp_url,
coalesce(ctedata2.total_visit_count, 0) as total_visit_count,
coalesce(ctedata2.sh_visit_count, 0) as sh_visit_count,
coalesce(ctedata2.ec_visit_count, 0) as ec_visit_count,
coalesce(ctedata2.total_like_count, 0) as total_like_count,
coalesce(ctedata2.sh_like_count, 0) as sh_like_count,
coalesce(ctedata2.ec_like_count, 0) as ec_like_count,
coalesce(ctedata2.total_order_count, 0) as total_order_count,
coalesce(ctedata2.sh_order_count, 0) as sh_order_count,
coalesce(ctedata2.ec_order_count, 0) as ec_order_count,
coalesce(ctedata2.total_sales_amount, 0) as total_sales_amount,
coalesce(ctedata2.sh_sales_amount, 0) as sh_sales_amount,
coalesce(ctedata2.ec_sales_amount, 0) as ec_sales_amount,
coalesce(ctedata2.ec_order_online_count, 0) as ec_order_online_count,
coalesce(ctedata2.ec_sales_online_amount, 0) as ec_sales_online_amount,
coalesce(ctedata2.ec_order_in_store_count, 0) as ec_order_in_store_count,
coalesce(ctedata2.ec_sales_in_store_amount, 0) as ec_sales_in_store_amount,
ctedata2.rank,
ctedata2.total_count,
table1.department,
table1.seasons
from
table1
left join ctedata2 on
table1.im_name = ctedata2.im_name
and table1.brand = ctedata2.kpibrand
where
table1.deleted_at is null
and table1.id_region = any('{1}')
and lower(table1.brand) = any('{"brand1","brand2"}')
and 'season1' = any(lower(seasons::text)::text[])
and table1.department = 'Department1'
order by
total_sales_amount desc offset 0
limit 100
The explain output for above query is
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=172326.55..173435.38 rows=1 width=952) (actual time=85664.201..85665.970 rows=100 loops=1)
CTE ctedata1
-> GroupAggregate (cost=0.42..80478.71 rows=43468 width=530) (actual time=0.063..708.069 rows=73121 loops=1)
Group Key: table2.im_name, table2.id_region, table2.brand
-> Index Scan using udx_table2_im_name_id_region_brand_target_date_key on table2 (cost=0.42..59699.18 rows=391708 width=146) (actual time=0.029..308.582 rows=391779 loops=1)
Filter: ((deleted_at IS NULL) AND (id_region = ANY ('{1}'::integer[])))
Rows Removed by Filter: 20415
CTE ctedata2
-> WindowAgg (cost=16104.06..17842.78 rows=43468 width=628) (actual time=1012.994..1082.057 rows=73121 loops=1)
-> WindowAgg (cost=16104.06..17082.09 rows=43468 width=620) (actual time=945.755..1014.656 rows=73121 loops=1)
-> Sort (cost=16104.06..16212.73 rows=43468 width=612) (actual time=945.747..963.254 rows=73121 loops=1)
Sort Key: ctedata1.kpiregion, ctedata1.kpibrand, (COALESCE(ctedata1.total_sales_amount, '0'::numeric)) DESC
Sort Method: external merge Disk: 6536kB
-> CTE Scan on ctedata1 (cost=0.00..869.36 rows=43468 width=612) (actual time=0.069..824.841 rows=73121 loops=1)
-> Result (cost=74005.05..75113.88 rows=1 width=952) (actual time=85664.199..85665.950 rows=100 loops=1)
-> Sort (cost=74005.05..74005.05 rows=1 width=944) (actual time=85664.072..85664.089 rows=100 loops=1)
Sort Key: (COALESCE(ctedata2.total_sales_amount, '0'::numeric)) DESC
Sort Method: top-N heapsort Memory: 76kB
-> WindowAgg (cost=10960.95..74005.04 rows=1 width=944) (actual time=85658.049..85661.393 rows=3151 loops=1)
-> Nested Loop Left Join (cost=10960.95..74005.02 rows=1 width=927) (actual time=1075.219..85643.595 rows=3151 loops=1)
Join Filter: (((table1.im_name)::text = ctedata2.im_name) AND ((table1.brand)::text = ctedata2.kpibrand))
Rows Removed by Join Filter: 230402986
-> Bitmap Heap Scan on table1 (cost=10960.95..72483.64 rows=1 width=399) (actual time=45.466..278.376 rows=3151 loops=1)
Recheck Cond: (id_region = ANY ('{1}'::integer[]))
Filter: ((deleted_at IS NULL) AND (department = 'Department1'::text) AND (lower((brand)::text) = ANY ('{brand1, brand2}'::text[])) AND ('season1'::text = ANY ((lower((seasons)::text))::text[])))
Rows Removed by Filter: 106335
Heap Blocks: exact=42899
-> Bitmap Index Scan on table1_im_name_id_region_key (cost=0.00..10960.94 rows=110619 width=0) (actual time=38.307..38.307 rows=109486 loops=1)
Index Cond: (id_region = ANY ('{1}'::integer[]))
-> CTE Scan on ctedata2 (cost=0.00..869.36 rows=43468 width=592) (actual time=0.325..21.721 rows=73121 loops=3151)
SubPlan 3
-> Aggregate (cost=1108.80..1108.81 rows=1 width=8) (actual time=0.018..0.018 rows=1 loops=100)
-> Nested Loop Left Join (cost=5.57..1108.57 rows=93 width=4) (actual time=0.007..0.016 rows=3 loops=100)
-> Bitmap Heap Scan on table3 (cost=5.15..350.95 rows=93 width=4) (actual time=0.005..0.008 rows=3 loops=100)
Recheck Cond: (id_pf_item = table1.id_pf_item)
Filter: (deleted_at IS NULL)
Heap Blocks: exact=107
-> Bitmap Index Scan on idx_id_pf_item (cost=0.00..5.12 rows=93 width=0) (actual time=0.003..0.003 rows=3 loops=100)
Index Cond: (id_pf_item = table1.id_pf_item)
-> Index Scan using index_table4_id_item on table4 (cost=0.42..8.14 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=303)
Index Cond: (table3.id_item = id_item)
Filter: (deleted_at IS NULL)
Rows Removed by Filter: 0
Planning time: 1.023 ms
Execution time: 85669.512 ms
I changed
and lower(table1.brand) = any('{"brand1","brand2"}')
in the query to
and table1.brand = any('{"Brand1","Brand2"}')
and the plan changed to
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=173137.44..188661.06 rows=14 width=952) (actual time=1444.123..1445.653 rows=100 loops=1)
CTE ctedata1
-> GroupAggregate (cost=0.42..80478.71 rows=43468 width=530) (actual time=0.040..769.982 rows=73121 loops=1)
Group Key: table2.im_name, table2.id_region, table2.brand
-> Index Scan using udx_table2_item_im_name_id_region_brand_target_date_key on table2 (cost=0.42..59699.18 rows=391708 width=146) (actual time=0.021..350.774 rows=391779 loops=1)
Filter: ((deleted_at IS NULL) AND (id_region = ANY ('{1}'::integer[])))
Rows Removed by Filter: 20415
CTE ctedata2
-> WindowAgg (cost=16104.06..17842.78 rows=43468 width=628) (actual time=1088.905..1153.749 rows=73121 loops=1)
-> WindowAgg (cost=16104.06..17082.09 rows=43468 width=620) (actual time=1020.017..1089.117 rows=73121 loops=1)
-> Sort (cost=16104.06..16212.73 rows=43468 width=612) (actual time=1020.011..1037.170 rows=73121 loops=1)
Sort Key: ctedata1.kpiregion, ctedata1.kpibrand, (COALESCE(ctedata1.total_sales_amount, '0'::numeric)) DESC
Sort Method: external merge Disk: 6536kB
-> CTE Scan on ctedata1 (cost=0.00..869.36 rows=43468 width=612) (actual time=0.044..891.653 rows=73121 loops=1)
-> Result (cost=74815.94..90339.56 rows=14 width=952) (actual time=1444.121..1445.635 rows=100 loops=1)
-> Sort (cost=74815.94..74815.98 rows=14 width=944) (actual time=1444.053..1444.065 rows=100 loops=1)
Sort Key: (COALESCE(ctedata2.total_sales_amount, '0'::numeric)) DESC
Sort Method: top-N heapsort Memory: 76kB
-> WindowAgg (cost=72207.31..74815.68 rows=14 width=944) (actual time=1439.128..1441.885 rows=3151 loops=1)
-> Hash Right Join (cost=72207.31..74815.40 rows=14 width=927) (actual time=1307.531..1437.246 rows=3151 loops=1)
Hash Cond: ((ctedata2.im_name = (table1.im_name)::text) AND (ctedata2.kpibrand = (table1.brand)::text))
-> CTE Scan on ctedata2 (cost=0.00..869.36 rows=43468 width=592) (actual time=1088.911..1209.646 rows=73121 loops=1)
-> Hash (cost=72207.10..72207.10 rows=14 width=399) (actual time=216.850..216.850 rows=3151 loops=1)
Buckets: 4096 (originally 1024) Batches: 1 (originally 1) Memory Usage: 1249kB
-> Bitmap Heap Scan on table1 (cost=10960.95..72207.10 rows=14 width=399) (actual time=46.434..214.246 rows=3151 loops=1)
Recheck Cond: (id_region = ANY ('{1}'::integer[]))
Filter: ((deleted_at IS NULL) AND (department = 'Department1'::text) AND ((brand)::text = ANY ('{Brand1, Brand2}'::text[])) AND ('season1'::text = ANY ((lower((seasons)::text))::text[])))
Rows Removed by Filter: 106335
Heap Blocks: exact=42899
-> Bitmap Index Scan on table1_im_name_id_region_key (cost=0.00..10960.94 rows=110619 width=0) (actual time=34.849..34.849 rows=109486 loops=1)
Index Cond: (id_region = ANY ('{1}'::integer[]))
SubPlan 3
-> Aggregate (cost=1108.80..1108.81 rows=1 width=8) (actual time=0.015..0.015 rows=1 loops=100)
-> Nested Loop Left Join (cost=5.57..1108.57 rows=93 width=4) (actual time=0.006..0.014 rows=3 loops=100)
-> Bitmap Heap Scan on table3 (cost=5.15..350.95 rows=93 width=4) (actual time=0.004..0.006 rows=3 loops=100)
Recheck Cond: (id_pf_item = table1.id_pf_item)
Filter: (deleted_at IS NULL)
Heap Blocks: exact=107
-> Bitmap Index Scan on idx_id_pf_item (cost=0.00..5.12 rows=93 width=0) (actual time=0.003..0.003 rows=3 loops=100)
Index Cond: (id_pf_item = table1.id_pf_item)
-> Index Scan using index_table4_id_item on table4 (cost=0.42..8.14 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=303)
Index Cond: (table3.id_item = id_item)
Filter: (deleted_at IS NULL)
Rows Removed by Filter: 0
Planning time: 0.760 ms
Execution time: 1448.848 ms
My Observation
The join strategy for table1 left join ctedata2 changes after the lower() function is avoided. The strategy changes from nested loop left join to hash right join.
The CTE Scan node on ctedata2 is executed only once in the better performing query.
Postgres Version
9.6
Please help me to understand this behaviour. I will supply additional info if required.

It is almost not worthwhile taking a deep dive into the inner workings of a nearly-obsolete version. That time and energy is probably better spent jollying along an upgrade.
But the problem is pretty plain. Your scan on table1 is estimated dreadfully, although 14 times less dreadful in the better plan.
-> Bitmap Heap Scan on table1 (cost=10960.95..72483.64 rows=1 width=399) (actual time=45.466..278.376 rows=3151 loops=1)
-> Bitmap Heap Scan on table1 (cost=10960.95..72207.10 rows=14 width=399) (actual time=46.434..214.246 rows=3151 loops=1)
Your use of lower(), apparently without reason, surely contributes to the poor estimation. And dynamically converting a string into an array certainly doesn't help either. If it were stored as a real array in the first place, the statistics system could get its hands on it and generate more reasonable estimates.

Query is slow with indexes - how to understand execution plans?

I need help to understand why my query is slower when I use index than without any index. I ran explain analyze command, and below are execution plans option 1 - with index, and option 2 - without index.
Can someone explain to me why index makes performances worse in those execution plans?
PS. When I add 10 million rows to table (original size 2M), situation is turning in favor of index, and in that case query with index is 3x faster).
OPTION 1 WITH INDEX FOR LEFT JOIN invoice_id+acct_level ON TABLE cost_invoice_facepage AND CONDITION (cdb.invoice_id = invoice_id) AND (acct_level = 1)
Append (cost=48.87..38583.97 rows=163773 width=371) (actual time=1.269..1516.564 rows=379129 loops=1)
-> Nested Loop (cost=48.87..10520.11 rows=36504 width=362) (actual time=1.268..5.986 rows=579 loops=1)
-> Hash Left Join (cost=44.66..9918.22 rows=507 width=322) (actual time=1.160..5.497 rows=579 loops=1)
Hash Cond: (cd.gl_string_id = gs.id)
-> Nested Loop Left Join (cost=0.85..9873.07 rows=507 width=262) (actual time=0.485..4.473 rows=579 loops=1)
Filter: ((c.gl_rule_type IS NULL) OR ((cd.charge_id IS NOT NULL) AND (c.gl_rule_type_id <> ALL ('{60,70}'::integer[]))))
-> Index Scan using cost_invoice_charge_invoice_id_idx on cost_invoice_charge c (cost=0.43..1204.53 rows=1188 width=243) (actual time=0.467..2.664 rows=579 loops=1)
Index Cond: (invoice_id = 14517)
Filter: ((chg_amt <> '0'::numeric) AND ((gl_rule_type IS NULL) OR (gl_rule_type_id <> ALL ('{60,70}'::integer[]))))
Rows Removed by Filter: 3364
-> Index Scan using "gl_charge_detail.charge_id->cost_invoice_info_only.id" on gl_charge_detail cd (cost=0.42..7.28 rows=1 width=27) (actual time=0.002..0.002 rows=1 loops=579)
Index Cond: (c.id = charge_id)
-> Hash (cost=31.69..31.69 rows=969 width=64) (actual time=0.657..0.657 rows=969 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 103kB
-> Seq Scan on gl_strings gs (cost=0.00..31.69 rows=969 width=64) (actual time=0.026..0.389 rows=969 loops=1)
-> Materialize (cost=4.22..145.78 rows=72 width=44) (actual time=0.000..0.000 rows=1 loops=579)
-> Hash Left Join (cost=4.22..145.42 rows=72 width=44) (actual time=0.100..0.102 rows=1 loops=1)
Hash Cond: (f.vendor_id = vn.id)
-> Nested Loop (cost=0.57..141.57 rows=72 width=31) (actual time=0.027..0.029 rows=1 loops=1)
-> Index Scan using cost_invoice_header_id_idx on cost_invoice_header ch (cost=0.29..8.31 rows=1 width=4) (actual time=0.012..0.013 rows=1 loops=1)
Index Cond: (id = 14517)
Filter: (status_code <> ALL ('{100,101,102,490}'::integer[]))
-> Index Scan using "invoice_id+acct_level" on cost_invoice_facepage f (cost=0.29..132.55 rows=72 width=31) (actual time=0.013..0.013 rows=1 loops=1)
Index Cond: ((invoice_id = 14517) AND (acct_level = 1))
-> Hash (cost=2.73..2.73 rows=73 width=17) (actual time=0.061..0.061 rows=73 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on appdata_vendor_common vn (cost=0.00..2.73 rows=73 width=17) (actual time=0.020..0.038 rows=73 loops=1)
-> Hash Left Join (cost=2276.48..25607.26 rows=127269 width=374) (actual time=204.117..1486.717 rows=378550 loops=1)
Hash Cond: (f_1.vendor_id = vn_1.id)
-> Nested Loop Left Join (cost=2272.84..25250.14 rows=127269 width=361) (actual time=204.072..1328.491 rows=378550 loops=1)
-> Gather (cost=2272.55..24385.32 rows=2663 width=338) (actual time=204.055..335.965 rows=378550 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Hash Left Join (cost=1272.55..23119.02 rows=1110 width=338) (actual time=127.365..321.126 rows=126183 loops=3)
Hash Cond: (cdb.gl_string_id = gs_1.id)
-> Hash Join (cost=1228.74..23072.30 rows=1110 width=278) (actual time=126.126..263.315 rows=126183 loops=3)
Hash Cond: (cdb.charge_id = c_1.id)
-> Parallel Seq Scan on gl_charge_detail_ban cdb (cost=0.00..20581.15 rows=480915 width=43) (actual time=0.270..109.543 rows=384732 loops=3)
-> Hash (cost=1194.13..1194.13 rows=2769 width=239) (actual time=7.232..7.232 rows=3929 loops=3)
Buckets: 4096 Batches: 1 Memory Usage: 635kB
-> Index Scan using cost_invoice_charge_invoice_id_idx on cost_invoice_charge c_1 (cost=0.43..1194.13 rows=2769 width=239) (actual time=0.070..4.686 rows=3929 loops=3)
Index Cond: (invoice_id = 14517)
Filter: (chg_amt <> '0'::numeric)
Rows Removed by Filter: 14
-> Hash (cost=31.69..31.69 rows=969 width=64) (actual time=1.127..1.127 rows=969 loops=3)
Buckets: 1024 Batches: 1 Memory Usage: 103kB
-> Seq Scan on gl_strings gs_1 (cost=0.00..31.69 rows=969 width=64) (actual time=0.165..0.714 rows=969 loops=3)
-> Index Scan using "invoice_id+acct_level" on cost_invoice_facepage f_1 (cost=0.29..0.31 rows=1 width=31) (actual time=0.001..0.002 rows=1 loops=378550)
Index Cond: ((cdb.invoice_id = invoice_id) AND (acct_level = 1))
-> Hash (cost=2.73..2.73 rows=73 width=17) (actual time=0.035..0.035 rows=73 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on appdata_vendor_common vn_1 (cost=0.00..2.73 rows=73 width=17) (actual time=0.014..0.021 rows=73 loops=1)
Planning Time: 3.636 ms
Execution Time: 1550.844 ms
and
OPTION 2 WITHOUT INDEXES
Append (cost=48.58..43257.20 rows=163773 width=371) (actual time=7.965..831.408 rows=379129 loops=1)
-> Nested Loop (cost=48.58..12251.68 rows=36504 width=362) (actual time=7.965..14.476 rows=579 loops=1)
-> Hash Left Join (cost=44.66..9918.22 rows=507 width=322) (actual time=0.588..6.245 rows=579 loops=1)
Hash Cond: (cd.gl_string_id = gs.id)
-> Nested Loop Left Join (cost=0.85..9873.07 rows=507 width=262) (actual time=0.245..5.442 rows=579 loops=1)
Filter: ((c.gl_rule_type IS NULL) OR ((cd.charge_id IS NOT NULL) AND (c.gl_rule_type_id <> ALL ('{60,70}'::integer[]))))
-> Index Scan using cost_invoice_charge_invoice_id_idx on cost_invoice_charge c (cost=0.43..1204.53 rows=1188 width=243) (actual time=0.231..3.003 rows=579 loops=1)
Index Cond: (invoice_id = 14517)
Filter: ((chg_amt <> '0'::numeric) AND ((gl_rule_type IS NULL) OR (gl_rule_type_id <> ALL ('{60,70}'::integer[]))))
Rows Removed by Filter: 3364
-> Index Scan using "gl_charge_detail.charge_id->cost_invoice_info_only.id" on gl_charge_detail cd (cost=0.42..7.28 rows=1 width=27) (actual time=0.003..0.003 rows=1 loops=579)
Index Cond: (c.id = charge_id)
-> Hash (cost=31.69..31.69 rows=969 width=64) (actual time=0.331..0.331 rows=969 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 103kB
-> Seq Scan on gl_strings gs (cost=0.00..31.69 rows=969 width=64) (actual time=0.017..0.183 rows=969 loops=1)
-> Materialize (cost=3.93..1877.35 rows=72 width=44) (actual time=0.013..0.013 rows=1 loops=579)
-> Hash Left Join (cost=3.93..1876.99 rows=72 width=44) (actual time=7.370..7.698 rows=1 loops=1)
Hash Cond: (f.vendor_id = vn.id)
-> Nested Loop (cost=0.29..1873.14 rows=72 width=31) (actual time=7.307..7.635 rows=1 loops=1)
-> Index Scan using cost_invoice_header_id_idx on cost_invoice_header ch (cost=0.29..8.31 rows=1 width=4) (actual time=0.011..0.013 rows=1 loops=1)
Index Cond: (id = 14517)
Filter: (status_code <> ALL ('{100,101,102,490}'::integer[]))
-> Seq Scan on cost_invoice_facepage f (cost=0.00..1864.12 rows=72 width=31) (actual time=7.293..7.619 rows=1 loops=1)
Filter: ((invoice_id = 14517) AND (acct_level = 1))
Rows Removed by Filter: 40340
-> Hash (cost=2.73..2.73 rows=73 width=17) (actual time=0.045..0.045 rows=73 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on appdata_vendor_common vn (cost=0.00..2.73 rows=73 width=17) (actual time=0.022..0.028 rows=73 loops=1)
-> Hash Left Join (cost=4248.29..28548.92 rows=127269 width=374) (actual time=234.692..789.334 rows=378550 loops=1)
Hash Cond: (cdb.invoice_id = f_1.invoice_id)
-> Gather (cost=2272.55..24385.32 rows=2663 width=338) (actual time=216.507..376.349 rows=378550 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Hash Left Join (cost=1272.55..23119.02 rows=1110 width=338) (actual time=128.932..389.669 rows=126183 loops=3)
Hash Cond: (cdb.gl_string_id = gs_1.id)
-> Hash Join (cost=1228.74..23072.30 rows=1110 width=278) (actual time=127.984..308.092 rows=126183 loops=3)
Hash Cond: (cdb.charge_id = c_1.id)
-> Parallel Seq Scan on gl_charge_detail_ban cdb (cost=0.00..20581.15 rows=480915 width=43) (actual time=0.163..117.001 rows=384732 loops=3)
-> Hash (cost=1194.13..1194.13 rows=2769 width=239) (actual time=8.779..8.779 rows=3929 loops=3)
Buckets: 4096 Batches: 1 Memory Usage: 635kB
-> Index Scan using cost_invoice_charge_invoice_id_idx on cost_invoice_charge c_1 (cost=0.43..1194.13 rows=2769 width=239) (actual time=0.050..5.563 rows=3929 loops=3)
Index Cond: (invoice_id = 14517)
Filter: (chg_amt <> '0'::numeric)
Rows Removed by Filter: 14
-> Hash (cost=31.69..31.69 rows=969 width=64) (actual time=0.829..0.829 rows=969 loops=3)
Buckets: 1024 Batches: 1 Memory Usage: 103kB
-> Seq Scan on gl_strings gs_1 (cost=0.00..31.69 rows=969 width=64) (actual time=0.184..0.534 rows=969 loops=3)
-> Hash (cost=1804.87..1804.87 rows=13670 width=44) (actual time=18.101..18.101 rows=13705 loops=1)
Buckets: 16384 Batches: 1 Memory Usage: 1198kB
-> Hash Left Join (cost=3.64..1804.87 rows=13670 width=44) (actual time=0.075..14.009 rows=13705 loops=1)
Hash Cond: (f_1.vendor_id = vn_1.id)
-> Seq Scan on cost_invoice_facepage f_1 (cost=0.00..1763.26 rows=13670 width=31) (actual time=0.017..6.216 rows=13705 loops=1)
Filter: (acct_level = 1)
Rows Removed by Filter: 26636
-> Hash (cost=2.73..2.73 rows=73 width=17) (actual time=0.052..0.052 rows=73 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on appdata_vendor_common vn_1 (cost=0.00..2.73 rows=73 width=17) (actual time=0.013..0.027 rows=73 loops=1)
Planning Time: 3.365 ms
Execution Time: 863.941 ms

Look at the line that is driving the iteration over the index scan:
Gather (cost=2272.55..24385.32 rows=2663 width=338) (actual time=204.055..335.965 rows=378550 loops=1)
It thinks the index scan will get iterated 2663 times (with a different value of invoice_id for each one) but it really gets iterated 378550 times, (this latter number is where the 'loops' field on the index scan comes from), a difference of 140 fold. Every time you hit the index, you need to re-descend from the root to the leaf, locking and unlocking pages as you go. While this is not terribly expensive, it does add up if you do it 378550 times. It gets to be faster to process the table in bulk into a private hash table. But since the estimated row count is so wrong, PostgreSQL doesn't realize that in this case.

Explain postgres query, why is the query that much longer with WHERE and LIMIT

I'm using postgres v9.6.5. I have a query which seems not that complicated and was wondering why is it so "slow" (it's not really that slow, but I don't have a lot of data actually - like a few thousand rows).
Here is the query:
SELECT o0.*
FROM "orders" AS o0
JOIN "balances" AS b1 ON b1."id" = o0."balance_id"
JOIN "users" AS u3 ON u3."id" = b1."user_id"
WHERE (u3."partner_id" = 3)
ORDER BY o0."id" DESC LIMIT 10;
And that's query plan:
Limit (cost=0.43..12.84 rows=10 width=148) (actual time=0.062..53.866 rows=4 loops=1)
-> Nested Loop (cost=0.43..4750.03 rows=3826 width=148) (actual time=0.061..53.864 rows=4 loops=1)
Join Filter: (b1.user_id = u3.id)
Rows Removed by Join Filter: 67404
-> Nested Loop (cost=0.43..3945.32 rows=17856 width=152) (actual time=0.025..38.457 rows=16852 loops=1)
-> Index Scan Backward using orders_pkey on orders o0 (cost=0.29..897.80 rows=17856 width=148) (actual time=0.016..11.558 rows=16852 loops=1)
-> Index Scan using balances_pkey on balances b1 (cost=0.14..0.16 rows=1 width=8) (actual time=0.001..0.001 rows=1 loops=16852)
Index Cond: (id = o0.balance_id)
-> Materialize (cost=0.00..1.19 rows=3 width=4) (actual time=0.000..0.000 rows=4 loops=16852)
-> Seq Scan on users u3 (cost=0.00..1.18 rows=3 width=4) (actual time=0.023..0.030 rows=4 loops=1)
Filter: (partner_id = 3)
Rows Removed by Filter: 12
Planning time: 0.780 ms
Execution time: 54.053 ms
I actually tried without LIMIT and I got quite different plan:
Sort (cost=874.23..883.80 rows=3826 width=148) (actual time=11.361..11.362 rows=4 loops=1)
Sort Key: o0.id DESC
Sort Method: quicksort Memory: 26kB
-> Hash Join (cost=3.77..646.55 rows=3826 width=148) (actual time=11.300..11.346 rows=4 loops=1)
Hash Cond: (o0.balance_id = b1.id)
-> Seq Scan on orders o0 (cost=0.00..537.56 rows=17856 width=148) (actual time=0.012..8.464 rows=16852 loops=1)
-> Hash (cost=3.55..3.55 rows=18 width=4) (actual time=0.125..0.125 rows=24 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Hash Join (cost=1.21..3.55 rows=18 width=4) (actual time=0.046..0.089 rows=24 loops=1)
Hash Cond: (b1.user_id = u3.id)
-> Seq Scan on balances b1 (cost=0.00..1.84 rows=84 width=8) (actual time=0.011..0.029 rows=96 loops=1)
-> Hash (cost=1.18..1.18 rows=3 width=4) (actual time=0.028..0.028 rows=4 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on users u3 (cost=0.00..1.18 rows=3 width=4) (actual time=0.014..0.021 rows=4 loops=1)
Filter: (partner_id = 3)
Rows Removed by Filter: 12
Planning time: 0.569 ms
Execution time: 11.420 ms
And also without WHERE (but with LIMIT):
Limit (cost=0.43..4.74 rows=10 width=148) (actual time=0.023..0.066 rows=10 loops=1)
-> Nested Loop (cost=0.43..7696.26 rows=17856 width=148) (actual time=0.022..0.065 rows=10 loops=1)
Join Filter: (b1.user_id = u3.id)
Rows Removed by Join Filter: 139
-> Nested Loop (cost=0.43..3945.32 rows=17856 width=152) (actual time=0.009..0.029 rows=10 loops=1)
-> Index Scan Backward using orders_pkey on orders o0 (cost=0.29..897.80 rows=17856 width=148) (actual time=0.007..0.015 rows=10 loops=1)
-> Index Scan using balances_pkey on balances b1 (cost=0.14..0.16 rows=1 width=8) (actual time=0.001..0.001 rows=1 loops=10)
Index Cond: (id = o0.balance_id)
-> Materialize (cost=0.00..1.21 rows=14 width=4) (actual time=0.001..0.001 rows=15 loops=10)
-> Seq Scan on users u3 (cost=0.00..1.14 rows=14 width=4) (actual time=0.005..0.007 rows=16 loops=1)
Planning time: 0.286 ms
Execution time: 0.097 ms
As you can see, without WHERE it's much faster. Can someone provide me with some information where can I look for explanations for those plans to better understand them? And also what can I do to make those queries faster (or I shouldn't worry cause with like 100 times more data they will still be fast enough? - 50ms is fine for me tbh)

PostgreSQL thinks that it will be fastest if it scans orders in the correct order until it finds a matching users entry that satisfies the WHERE condition.
However, it seems that the data distribution is such that it has to scan almost 17000 orders before it finds a match.
Since PostgreSQL doesn't know how values correlate across tables, there is nothing much you can do to change that.
You can force PostgreSQL to plan the query without the LIMIT clause like this:
SELECT *
FROM (<your query without ORDER BY and LIMIT> OFFSET 0) q
ORDER BY id DESC LIMIT 10;
With a top-N-sort this should perform better.

Postgresql: Query is slow (9.5)

this is my query. I have more than 600000 rows of my table sale. I think the numbers of data is not too much, but it takes approximately 50s.
Here's the query.
I need faster.
SELECT
st.product_id,
prd.price_in_stock,
prd.product_name_eng,
prd.currency_id,
prd.product_name_kh,
(SELECT quantity
FROM daily_stock
WHERE stock_id = st.stock_id
AND DATE (stock_date) = DATE (now())
) AS pre_stock_quantity,
(SELECT SUM (trrc.quantity)
FROM transfer trs
INNER JOIN transfer_detail trd
ON trs.transfer_id = trd.transfer_id
INNER JOIN transfer_received trrc
ON trrc.transfer_detail_id = trd.transfer_detail_id
WHERE
trs.transfer_from = st.branch_id
AND trd.product_id = st.product_id
AND DATE (trrc.received_date) = DATE (NOW())
) AS trasfered_quantity,
(SELECT SUM (trrc.quantity)
FROM transfer trs
INNER JOIN transfer_detail trd
ON trs.transfer_id = trd.transfer_id
INNER JOIN transfer_received trrc
ON trrc.transfer_detail_id = trd.transfer_detail_id
WHERE
trs.transfer_to = st.branch_id
AND trd.product_id = st.product_id
AND DATE (trrc.received_date) = DATE (NOW())
) AS received_quantity,
(SELECT (SUM (smallest_devisor (sd.product_id)
* sd.quantity
/ getDevisor (sd.product_id, sd.unit_id)))
/ smallest_devisor (sd.product_id) AS sold_quantity
FROM
sale_detail sd
INNER JOIN sale sa ON sa.sale_id = sd.sale_id
WHERE
sa.branch_id = st.branch_id
AND sd.product_id = st.product_id
AND DATE (sa.sale_date) = DATE (NOW())
GROUP BY
sd.product_id
),
(SELECT (SUM(smallest_devisor (sd.product_id)
* rp.quantity
/ getDevisor (sd.product_id, sd.unit_id))
) / smallest_devisor (sd.product_id) AS returned_quantity
FROM
returned_product rp
INNER JOIN sale_detail sd ON sd.sale_detail_id = rp.sale_detail_id
WHERE
rp.branch_id = st.branch_id
AND sd.product_id = st.product_id
AND DATE (rp.returned_date) = DATE (NOW())
GROUP BY
sd.product_id
),
(
SELECT
SUM (quantity) AS imported_quantity
FROM
import
WHERE
branch_id = st.branch_id
AND product_id = st.product_id
AND DATE (import_date) = DATE (NOW())
),
st.quantity AS post_stock_quantity
FROM
stock st
INNER JOIN product prd ON prd.product_id = st.product_id
LEFT JOIN import imp ON imp.product_id = st.product_id
WHERE
st.branch_id = 'BR0000';
analyze is here, I've seen the nested loop has taken cost much, but I don't know what to do more. I don't understand well about them, pls explain me some.
Hash Left Join (cost=2.45..210929.10 rows=4 width=99) (actual time=845.040..68556.076 rows=22 loops=1)
Hash Cond: ((st.product_id)::text = (imp.product_id)::text)
Buffers: shared hit=2362098 read=499796, temp read=63910 written=63781
-> Hash Join (cost=1.11..11.36 rows=4 width=99) (actual time=0.130..0.160 rows=5 loops=1)
Hash Cond: ((st.product_id)::text = (prd.product_id)::text)
Buffers: shared hit=11
-> Seq Scan on stock st (cost=0.00..10.19 rows=4 width=23) (actual time=0.025..0.041 rows=5 loops=1)
Filter: ((branch_id)::text = 'BR0000'::text)
Rows Removed by Filter: 10
Buffers: shared hit=10
-> Hash (cost=1.05..1.05 rows=5 width=114) (actual time=0.058..0.058 rows=5 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
Buffers: shared hit=1
-> Seq Scan on product prd (cost=0.00..1.05 rows=5 width=114) (actual time=0.037..0.041 rows=5 loops=1)
Buffers: shared hit=1
-> Hash (cost=1.15..1.15 rows=15 width=38) (actual time=0.041..0.041 rows=22 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
Buffers: shared hit=1
-> Seq Scan on import imp (cost=0.00..1.15 rows=15 width=38) (actual time=0.009..0.019 rows=22 loops=1)
Buffers: shared hit=1
SubPlan 1
-> Seq Scan on daily_stock (cost=0.00..48.25 rows=1 width=8) (actual time=0.010..0.011 rows=1 loops=22)
Filter: ((stock_id = st.stock_id) AND (date(stock_date) = date(now())))
Rows Removed by Filter: 29
Buffers: shared hit=22
SubPlan 2
-> Aggregate (cost=52.41..52.42 rows=1 width=8) (actual time=0.010..0.010 rows=1 loops=22)
-> Nested Loop (cost=0.15..52.41 rows=1 width=8) (actual time=0.008..0.008 rows=0 loops=22)
-> Nested Loop (cost=0.00..45.56 rows=1 width=60) (actual time=0.001..0.001 rows=0 loops=22)
Join Filter: (trd.transfer_detail_id = trrc.transfer_detail_id)
-> Seq Scan on transfer_received trrc (cost=0.00..28.00 rows=4 width=12) (actual time=0.000..0.000 rows=0 loops=22)
Filter: (date(received_date) = date(now()))
-> Materialize (cost=0.00..17.39 rows=3 width=56) (never executed)
-> Seq Scan on transfer_detail trd (cost=0.00..17.38 rows=3 width=56) (never executed)
Filter: ((product_id)::text = (st.product_id)::text)
-> Index Scan using transfer_pkey on transfer trs (cost=0.15..6.83 rows=1 width=52) (never executed)
Index Cond: ((transfer_id)::text = (trd.transfer_id)::text)
Filter: ((transfer_from)::text = (st.branch_id)::text)
SubPlan 3
-> Aggregate (cost=52.41..52.42 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=22)
-> Nested Loop (cost=0.15..52.41 rows=1 width=8) (actual time=0.005..0.005 rows=0 loops=22)
-> Nested Loop (cost=0.00..45.56 rows=1 width=60) (actual time=0.000..0.000 rows=0 loops=22)
Join Filter: (trd_1.transfer_detail_id = trrc_1.transfer_detail_id)
-> Seq Scan on transfer_received trrc_1 (cost=0.00..28.00 rows=4 width=12) (actual time=0.000..0.000 rows=0 loops=22)
Filter: (date(received_date) = date(now()))
-> Materialize (cost=0.00..17.39 rows=3 width=56) (never executed)
-> Seq Scan on transfer_detail trd_1 (cost=0.00..17.38 rows=3 width=56) (never executed)
Filter: ((product_id)::text = (st.product_id)::text)
-> Index Scan using transfer_pkey on transfer trs_1 (cost=0.15..6.83 rows=1 width=52) (never executed)
Index Cond: ((transfer_id)::text = (trd_1.transfer_id)::text)
Filter: ((transfer_to)::text = (st.branch_id)::text)
SubPlan 4
-> GroupAggregate (cost=32059.02..52563.95 rows=1 width=18) (actual time=3116.048..3116.048 rows=0 loops=22)
Group Key: sd.product_id
Buffers: shared hit=2362020 read=499796, temp read=63910 written=63781
-> Hash Join (cost=32059.02..52045.08 rows=1007 width=18) (actual time=830.903..988.696 rows=53521 loops=22)
Hash Cond: ((sd.sale_id)::text = (sa.sale_id)::text)
Buffers: shared hit=7084 read=499796, temp read=63910 written=63781
-> Seq Scan on sale_detail sd (cost=0.00..19220.90 rows=201358 width=35) (actual time=0.079..193.761 rows=292880 loops=22)
Filter: ((product_id)::text = (st.product_id)::text)
Rows Removed by Filter: 512552
Buffers: shared hit=3520 read=197846
-> Hash (cost=32008.68..32008.68 rows=4027 width=17) (actual time=552.758..552.758 rows=147183 loops=22)
Buckets: 131072 (originally 4096) Batches: 4 (originally 1) Memory Usage: 3137kB
Buffers: shared hit=3564 read=301950, temp written=10934
-> Seq Scan on sale sa (cost=0.00..32008.68 rows=4027 width=17) (actual time=400.100..500.394 rows=147183 loops=22)
Filter: (((branch_id)::text = (st.branch_id)::text) AND (date(sale_date) = date(now())))
Rows Removed by Filter: 658225
Buffers: shared hit=3564 read=301950
SubPlan 5
-> GroupAggregate (cost=0.42..10.66 rows=1 width=18) (actual time=0.041..0.041 rows=0 loops=22)
Group Key: sd_1.product_id
Buffers: shared hit=22
-> Nested Loop (cost=0.42..9.88 rows=1 width=18) (actual time=0.040..0.040 rows=0 loops=22)
Buffers: shared hit=22
-> Seq Scan on returned_product rp (cost=0.00..1.43 rows=1 width=12) (actual time=0.026..0.026 rows=0 loops=22)
Filter: (((branch_id)::text = (st.branch_id)::text) AND (date(returned_date) = date(now())))
Rows Removed by Filter: 19
Buffers: shared hit=22
-> Index Scan using sale_detail_pkey on sale_detail sd_1 (cost=0.42..8.45 rows=1 width=18) (never executed)
Index Cond: (sale_detail_id = rp.sale_detail_id)
Filter: ((product_id)::text = (st.product_id)::text)
SubPlan 6
-> Aggregate (cost=1.38..1.39 rows=1 width=8) (actual time=0.027..0.027 rows=1 loops=22)
Buffers: shared hit=22
-> Seq Scan on import (cost=0.00..1.38 rows=1 width=8) (actual time=0.018..0.018 rows=0 loops=22)
Filter: (((branch_id)::text = (st.branch_id)::text) AND ((product_id)::text = (st.product_id)::text) AND (date(import_date) = date(now())))
Rows Removed by Filter: 22
Buffers: shared hit=22
Planning time: 4.046 ms
Execution time: 68557.361 ms