We are having some struggle identifying why Postgres is using too much batches to resolve a join.
Here it is the output of explain analyze of a problematic execution:
https://explain.dalibo.com/plan/xNJ#plan
Limit (cost=20880.87..20882.91 rows=48 width=205) (actual time=10722.953..10723.358 rows=48 loops=1)
-> Unique (cost=20880.87..21718.12 rows=19700 width=205) (actual time=10722.951..10723.356 rows=48 loops=1)
-> Sort (cost=20880.87..20930.12 rows=19700 width=205) (actual time=10722.950..10722.990 rows=312 loops=1)
Sort Key: titlemetadata_titlemetadata.creation_date DESC, titlemetadata_titlemetadata.id, titlemetadata_titlemetadata.title_type, titlemetadata_titlemetadata.original_title, titlemetadata_titlemetadata.alternative_ids, titlemetadata_titlemetadata.metadata,
titlemetadata_titlemetadata.is_adult, titlemetadata_titlemetadata.is_kids, titlemetadata_titlemetadata.last_modified, titlemetadata_titlemetadata.year, titlemetadata_titlemetadata.runtime, titlemetadata_titlemetadata.rating, titlemetadata_titlemetadata.video_provider, tit
lemetadata_titlemetadata.series_id_id, titlemetadata_titlemetadata.season_number, titlemetadata_titlemetadata.episode_number
Sort Method: quicksort Memory: 872kB
-> Hash Right Join (cost=13378.20..19475.68 rows=19700 width=205) (actual time=1926.352..10709.970 rows=2909 loops=1)
Hash Cond: (t4.titlemetadata_id = t3.id)
Filter: ((hashed SubPlan 1) OR (hashed SubPlan 2))
Rows Removed by Filter: 63248
-> Seq Scan on video_provider_offer t4 (cost=0.00..5454.90 rows=66290 width=16) (actual time=0.024..57.893 rows=66390 loops=1)
-> Hash (cost=11314.39..11314.39 rows=22996 width=221) (actual time=489.530..489.530 rows=60096 loops=1)
Buckets: 65536 (originally 32768) Batches: 32768 (originally 1) Memory Usage: 11656kB
-> Hash Right Join (cost=5380.95..11314.39 rows=22996 width=221) (actual time=130.024..225.271 rows=60096 loops=1)
Hash Cond: (video_provider_offer.titlemetadata_id = titlemetadata_titlemetadata.id)
-> Seq Scan on video_provider_offer (cost=0.00..5454.90 rows=66290 width=16) (actual time=0.011..32.950 rows=66390 loops=1)
-> Hash (cost=5129.28..5129.28 rows=20133 width=213) (actual time=129.897..129.897 rows=55793 loops=1)
Buckets: 65536 (originally 32768) Batches: 2 (originally 1) Memory Usage: 7877kB
-> Merge Left Join (cost=1.72..5129.28 rows=20133 width=213) (actual time=0.041..93.057 rows=55793 loops=1)
Merge Cond: (titlemetadata_titlemetadata.id = t3.series_id_id)
-> Index Scan using titlemetadata_titlemetadata_pkey on titlemetadata_titlemetadata (cost=1.30..4130.22 rows=20133 width=205) (actual time=0.028..62.949 rows=43921 loops=1)
Filter: ((NOT is_adult) AND (NOT (hashed SubPlan 3)) AND (((title_type)::text = 'MOV'::text) OR ((title_type)::text = 'TVS'::text) OR ((title_type)::text = 'TVP'::text) OR ((title_type)::text = 'EVT'::text)))
Rows Removed by Filter: 14121
SubPlan 3
-> Seq Scan on cable_operator_cableoperatorexcludedtitle u0_2 (cost=0.00..1.01 rows=1 width=8) (actual time=0.006..0.006 rows=0 loops=1)
Filter: (cable_operator_id = 54)
-> Index Scan using titlemetadata_titlemetadata_series_id_id_73453db4_uniq on titlemetadata_titlemetadata t3 (cost=0.41..3901.36 rows=58037 width=16) (actual time=0.011..9.375 rows=12887 loops=1)
SubPlan 1
-> Hash Join (cost=44.62..885.73 rows=981 width=8) (actual time=0.486..36.806 rows=5757 loops=1)
Hash Cond: (w2.device_id = w3.id)
-> Nested Loop (cost=43.49..866.20 rows=2289 width=16) (actual time=0.441..33.096 rows=20180 loops=1)
-> Nested Loop (cost=43.06..414.98 rows=521 width=8) (actual time=0.426..9.952 rows=2909 loops=1)
Join Filter: (w1.id = w0.video_provider_id)
-> Nested Loop (cost=42.65..54.77 rows=13 width=24) (actual time=0.399..0.532 rows=15 loops=1)
-> HashAggregate (cost=42.50..42.95 rows=45 width=16) (actual time=0.390..0.403 rows=45 loops=1)
Group Key: v0.id
-> Nested Loop (cost=13.34..42.39 rows=45 width=16) (actual time=0.095..0.364 rows=45 loops=1)
-> Hash Semi Join (cost=13.19..32.72 rows=45 width=8) (actual time=0.084..0.229 rows=45 loops=1)
Hash Cond: (v1.id = u0.id)
-> Seq Scan on cable_operator_cableoperatorprovider v1 (cost=0.00..17.36 rows=636 width=16) (actual time=0.010..0.077 rows=636 loops=1)
-> Hash (cost=12.63..12.63 rows=45 width=8) (actual time=0.046..0.046 rows=45 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Index Scan using cable_operator_cableoperatorprovider_4d6e54b3 on cable_operator_cableoperatorprovider u0 (cost=0.28..12.63 rows=45 width=8) (actual time=0.016..0.035 rows=45 loops=1)
Index Cond: (cable_operator_id = 54)
-> Index Only Scan using video_provider_videoprovider_pkey on video_provider_videoprovider v0 (cost=0.15..0.20 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=45)
Index Cond: (id = v1.provider_id)
Heap Fetches: 45
-> Index Scan using video_provider_videoprovider_pkey on video_provider_videoprovider w1 (cost=0.15..0.25 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (id = v0.id)
Filter: ((video_provider_type)::text = 'VOD'::text)
Rows Removed by Filter: 1
-> Index Scan using video_provider_offer_da942d2e on video_provider_offer w0 (cost=0.42..27.22 rows=39 width=16) (actual time=0.026..0.585 rows=194 loops=15)
Index Cond: (video_provider_id = v0.id)
Filter: (((end_date > '2021-09-02 19:23:00-03'::timestamp with time zone) OR (end_date IS NULL)) AND (access_criteria && '{vtv_mas,TBX_LOGIN,urn:spkg:tve:fox-premium,urn:tve:mcp,AMCHD,AMC_CONSORCIO,ANIMAL_PLANET,ASUNTOS_PUBLI
COS,ASUNTOS_PUBLICOS_CONSORCIO,CINECANALLIVE,CINECANAL_CONSORCIO,DISCOVERY,DISCOVERY_KIDS_CONSORCIO,DISCOVERY_KIDS_OD,DISNEY,DISNEY_CH_CONSORCIO,DISNEY_XD,DISNEY_XD_CONSORCIO,EL_CANAL_HD,EL_CANAL_HD_CONSORCIO,EL_GOURMET_CONSORCIO,ESPN,ESPN2_HD_CONSORCIO,ESPN3_HD_CONSORCIO
,ESPNMAS_HD_CONSORCIO,ESPN_BASIC,ESPN_HD_CONSORCIO,ESPN_PLAY,EUROPALIVE,EUROPA_EUROPA,EUROPA_EUROPA_CONSORCIO,FILMANDARTS_DISPOSITIVOS,FILMS_ARTS,FILM_AND_ARTS_CONSORCIO,FOXLIFE,FOX_LIFE_CONSORCIO,FOX_SPORTS_1_DISPOSITIVOS,FOX_SPORTS_2_DISPOSITIVOS,FOX_SPORTS_2_HD_CONSORC
IO,FOX_SPORTS_3_DISPOSITIVOS,FOX_SPORTS_3_HD_CONSORCIO,FOX_SPORTS_HD_CONSORCIO,FRANCE24_DISPOSITIVOS,FRANCE_24_CONSORCIO,GOURMET,GOURMET_DISPOSITIVOS,HOME_HEALTH,INVESTIGATION_DISCOVERY,MAS_CHIC,NATGEOKIDS_DISPOSITIVOS,NATGEO_CONSORCIO,NATGEO_DISPOSITIVOS,NATGEO_KIDS_CONS
ORCIO,PASIONES,PASIONES_CONSORCIO,SVOD_TYC_BASIC,TBX_LOGIN,TCC_2_CONSORCIO,TCC_2_HD,TLC,TVE,TVE_CONSORCIO,TYC_SPORTS_CONSORCIO,VTV_LIVE,clarosports,discoverykids,espnplay_south_alt,urn:spkg:tve:fox-basic,urn:tve:babytv,urn:tve:cinecanal,urn:tve:discoverykids,urn:tve:foxli
fe,urn:tve:fp,urn:tve:fx,urn:tve:natgeo,urn:tve:natgeokids,urn:tve:natgeowild,urn:tve:thefilmzone}'::character varying(50)[]) AND ((((content_type)::text = 'VOD'::text) AND ((start_date < '2021-09-02 19:23:00-03'::timestamp with time zone) OR (start_date IS NULL))) OR ((c
ontent_type)::text = 'LIV'::text)))
Rows Removed by Filter: 5
-> Index Only Scan using video_provider_offer_devices_offer_id_device_id_key on video_provider_offer_devices w2 (cost=0.42..0.81 rows=6 width=16) (actual time=0.004..0.007 rows=7 loops=2909)
Index Cond: (offer_id = w0.id)
Heap Fetches: 17828
-> Hash (cost=1.10..1.10 rows=3 width=8) (actual time=0.029..0.029 rows=2 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on platform_device_device w3 (cost=0.00..1.10 rows=3 width=8) (actual time=0.024..0.027 rows=2 loops=1)
Filter: ((device_code)::text = ANY ('{ANDROID,ott_dual_tcc,ott_k2_tcc}'::text[]))
Rows Removed by Filter: 5
SubPlan 2
-> Hash Join (cost=44.62..885.73 rows=981 width=8) (actual time=0.410..33.580 rows=5757 loops=1)
Hash Cond: (w2_1.device_id = w3_1.id)
-> Nested Loop (cost=43.49..866.20 rows=2289 width=16) (actual time=0.375..29.886 rows=20180 loops=1)
-> Nested Loop (cost=43.06..414.98 rows=521 width=8) (actual time=0.366..9.134 rows=2909 loops=1)
Join Filter: (w1_1.id = w0_1.video_provider_id)
-> Nested Loop (cost=42.65..54.77 rows=13 width=24) (actual time=0.343..0.476 rows=15 loops=1)
-> HashAggregate (cost=42.50..42.95 rows=45 width=16) (actual time=0.333..0.347 rows=45 loops=1)
Group Key: v0_1.id
-> Nested Loop (cost=13.34..42.39 rows=45 width=16) (actual time=0.083..0.311 rows=45 loops=1)
-> Hash Semi Join (cost=13.19..32.72 rows=45 width=8) (actual time=0.076..0.202 rows=45 loops=1)
Hash Cond: (v1_1.id = u0_1.id)
-> Seq Scan on cable_operator_cableoperatorprovider v1_1 (cost=0.00..17.36 rows=636 width=16) (actual time=0.005..0.057 rows=636 loops=1)
-> Hash (cost=12.63..12.63 rows=45 width=8) (actual time=0.038..0.038 rows=45 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Index Scan using cable_operator_cableoperatorprovider_4d6e54b3 on cable_operator_cableoperatorprovider u0_1 (cost=0.28..12.63 rows=45 width=8) (actual time=0.007..0.020 rows=45 loops=1)
Index Cond: (cable_operator_id = 54)
-> Index Only Scan using video_provider_videoprovider_pkey on video_provider_videoprovider v0_1 (cost=0.15..0.20 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=45)
Index Cond: (id = v1_1.provider_id)
Heap Fetches: 45
-> Index Scan using video_provider_videoprovider_pkey on video_provider_videoprovider w1_1 (cost=0.15..0.25 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (id = v0_1.id)
Filter: ((video_provider_type)::text = 'VOD'::text)
Rows Removed by Filter: 1
-> Index Scan using video_provider_offer_da942d2e on video_provider_offer w0_1 (cost=0.42..27.22 rows=39 width=16) (actual time=0.022..0.536 rows=194 loops=15)
Index Cond: (video_provider_id = v0_1.id)
Filter: (((end_date > '2021-09-02 19:23:00-03'::timestamp with time zone) OR (end_date IS NULL)) AND (access_criteria && '{vtv_mas,TBX_LOGIN,urn:spkg:tve:fox-premium,urn:tve:mcp,AMCHD,AMC_CONSORCIO,ANIMAL_PLANET,ASUNTOS_PUBLI
COS,ASUNTOS_PUBLICOS_CONSORCIO,CINECANALLIVE,CINECANAL_CONSORCIO,DISCOVERY,DISCOVERY_KIDS_CONSORCIO,DISCOVERY_KIDS_OD,DISNEY,DISNEY_CH_CONSORCIO,DISNEY_XD,DISNEY_XD_CONSORCIO,EL_CANAL_HD,EL_CANAL_HD_CONSORCIO,EL_GOURMET_CONSORCIO,ESPN,ESPN2_HD_CONSORCIO,ESPN3_HD_CONSORCIO
,ESPNMAS_HD_CONSORCIO,ESPN_BASIC,ESPN_HD_CONSORCIO,ESPN_PLAY,EUROPALIVE,EUROPA_EUROPA,EUROPA_EUROPA_CONSORCIO,FILMANDARTS_DISPOSITIVOS,FILMS_ARTS,FILM_AND_ARTS_CONSORCIO,FOXLIFE,FOX_LIFE_CONSORCIO,FOX_SPORTS_1_DISPOSITIVOS,FOX_SPORTS_2_DISPOSITIVOS,FOX_SPORTS_2_HD_CONSORC
IO,FOX_SPORTS_3_DISPOSITIVOS,FOX_SPORTS_3_HD_CONSORCIO,FOX_SPORTS_HD_CONSORCIO,FRANCE24_DISPOSITIVOS,FRANCE_24_CONSORCIO,GOURMET,GOURMET_DISPOSITIVOS,HOME_HEALTH,INVESTIGATION_DISCOVERY,MAS_CHIC,NATGEOKIDS_DISPOSITIVOS,NATGEO_CONSORCIO,NATGEO_DISPOSITIVOS,NATGEO_KIDS_CONS
ORCIO,PASIONES,PASIONES_CONSORCIO,SVOD_TYC_BASIC,TBX_LOGIN,TCC_2_CONSORCIO,TCC_2_HD,TLC,TVE,TVE_CONSORCIO,TYC_SPORTS_CONSORCIO,VTV_LIVE,clarosports,discoverykids,espnplay_south_alt,urn:spkg:tve:fox-basic,urn:tve:babytv,urn:tve:cinecanal,urn:tve:discoverykids,urn:tve:foxli
fe,urn:tve:fp,urn:tve:fx,urn:tve:natgeo,urn:tve:natgeokids,urn:tve:natgeowild,urn:tve:thefilmzone}'::character varying(50)[]) AND ((((content_type)::text = 'VOD'::text) AND ((start_date < '2021-09-02 19:23:00-03'::timestamp with time zone) OR (start_date IS NULL))) OR ((c
ontent_type)::text = 'LIV'::text)))
Rows Removed by Filter: 5
-> Index Only Scan using video_provider_offer_devices_offer_id_device_id_key on video_provider_offer_devices w2_1 (cost=0.42..0.81 rows=6 width=16) (actual time=0.003..0.006 rows=7 loops=2909)
Index Cond: (offer_id = w0_1.id)
Heap Fetches: 17828
-> Hash (cost=1.10..1.10 rows=3 width=8) (actual time=0.015..0.015 rows=2 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on platform_device_device w3_1 (cost=0.00..1.10 rows=3 width=8) (actual time=0.010..0.011 rows=2 loops=1)
Filter: ((device_code)::text = ANY ('{ANDROID,ott_dual_tcc,ott_k2_tcc}'::text[]))
Rows Removed by Filter: 5
Planning time: 8.255 ms
Execution time: 10723.830 ms
(100 rows)
The weird part is that the same query, sometimes just uses a single batch. Here is an example: https://explain.dalibo.com/plan/zTv#plan
Here is the work_mem being used:
show work_mem;
work_mem
----------
8388kB
(1 row)
I'm not interested in changing the query to be more performant, but in understanding why is the different behavior.
I've found this thread apparently related with this, but I don't quite understand what are they talking about: https://www.postgresql.org/message-id/flat/CA%2BhUKGKWWmf%3DWELLG%3DaUGbcugRaSQbtm0tKYiBut-B2rVKX63g%40mail.gmail.com
Can anyone tell me why is this different behavior? The underlying data is the same in both cases.
If the hash is done in memory, there will only be a single batch.
A difference with the original hash batch numbers is due to Postgres choosing to increase the number of batches in order to reduce memory consumption.
You might find this EXPLAIN glossary useful (disclaimer: I'm one of the authors), here is the page on Hash Batches which also links to the PostgreSQL source code (it's very nicely documented in plain English).
While not a perfect heuristic, you can see that the memory required for the operations with multiple batches are around or above your work_mem setting. They can be lower than it, due to operations on disk generally requiring less memory overall.
I'm not 100% sure why in your exact case one was chosen over the other, but it does look like there are some very slight row estimate differences, which might be a good place to start.
As of PostgreSQL 13 there is also now a hash_mem_multiplier setting that can be used to give more memory to hashes without doing so for other operations (like sorts).
We where able to solve the problem just by doing VACUUM FULL ANALYZE;.
After that, everything started to work as expected (https://explain.depesz.com/s/eoqH#html)
Side note: we where not aware that we should do this on daily basis.
I use a simple query with a WHERE on a fixed value. This query does a left join on a temporary view. For some reason this query is performing very badly. I guess that the view is being executed for every row and not only for the selected rows. When I replace the fixed value with a value from a temporary table, the query performs MUCH better (about 15-20 times faster). Why is this?
I use postgresql version 9.2.15.
I added a temporary table 'wrkovz91-selecties' with only 1 record to pass the selection-value of the WHERE instruction to the query.
The view 'view_wrk_012_000001' that is joined in the query is pretty 'heavy', because it contains a nested other view ('view_wrk_013_000006').
First create the temporary table and add one record:
CREATE TEMPORARY TABLE WRKOVZ91_SELECTIES
(
NR DECIMAL(006) NOT NULL,
KLANTNR DECIMAL(008),
CONSTRAINT WRKOVZ91_SELECTIES_KEY_001 PRIMARY KEY (NR)
);
INSERT INTO WRKOVZ91_SELECTIES
(NR, KLANTNR) VALUES (1, 1);
Then create the temporary view (with a nested view):
CREATE TEMPORARY VIEW view_wrk_013_000006 AS
SELECT vrr.klantnr AS klantnr,
UPPER(vrr.artnr) AS artnr,
vrr.partijnr AS partijnr,
SUM(vrr.kollo) AS colli_op_voorraad
FROM voorraad vrr
WHERE (vrr.status = 'A'::text)
GROUP BY 1,
2,
3;
CREATE TEMPORARY VIEW view_wrk_012_000001 AS
SELECT COALESCE(wrd.klantnr,0) AS klantnr,
COALESCE(wrd.wrkopdrnr,0) AS wrkopdrnr,
MIN(COALESCE(wrd.recept_benodigd_colli,0) *
COALESCE(wrk.te_produceren,0)) AS vrije_voorraad_ind,
MIN(COALESCE(v40.colli_op_voorraad,0)) AS hulpveld
FROM wrkopdr wrk
LEFT JOIN wrkopdrd wrd ON wrd.klantnr = wrk.klantnr
AND wrd.wrkopdrnr = wrk.wrkopdrnr
LEFT JOIN view_wrk_013_000006 v40 ON v40.klantnr =
COALESCE(wrd.klantnr_grondstof,0)
AND v40.artnr = COALESCE(wrd.artnr_grondstof,'')
AND v40.partijnr = COALESCE(wrd.partijnr_grondstof,0)
LEFT JOIN artikel art ON art.klantnr = COALESCE(wrd.klantnr_grondstof,0)
AND art.artnr = COALESCE(wrd.artnr_grondstof,'')
WHERE wrk.status = 'A'::text
AND art.voorraadhoudend_jn = 'J'::text
GROUP BY 1,
2;
The query that performs badly is simple:
SELECT WRK.KLANTNR,
WRK.WRKOPDRNR,
WRK.MIL_UITVOER_DATUM
FROM WRKOPDR WRK
LEFT JOIN VIEW_WRK_012_000001 V38 ON V38.KLANTNR = WRK.KLANTNR AND
V38.WRKOPDRNR = WRK.WRKOPDRNR
LEFT JOIN WRKOVZ91_SELECTIES S02 ON S02.NR = 1
WHERE WRK.KLANTNR = 1
LIMIT 9999;
The query that performs MUCH better is (note the small difference):
SELECT WRK.KLANTNR,
WRK.WRKOPDRNR,
WRK.MIL_UITVOER_DATUM
FROM WRKOPDR WRK
LEFT JOIN VIEW_WRK_012_000001 V38 ON V38.KLANTNR = WRK.KLANTNR AND
V38.WRKOPDRNR = WRK.WRKOPDRNR
LEFT JOIN WRKOVZ91_SELECTIES S02 ON S02.NR = 1
WHERE WRK.KLANTNR = S02.KLANTNR
LIMIT 9999;
I cannot understand why the slow query is performing so badly. It takes in my test-data about 219 secondes. The fast query is taking only 12 secondes. The only difference between the 2 queries is the selection-value in the WHERE.
Does anyone have an explanation for this behaviour?
In addition, the output of the explain analyse of the slow query is:
Limit (cost=19460.18..19972.73 rows=9999 width=18) (actual time=221573.343..221585.953 rows=9999 loops=1)
-> Hash Left Join (cost=19460.18..20913.07 rows=28344 width=18) (actual time=221573.341..221583.701 rows=9999 loops=1)
Hash Cond: ((wrk.klantnr = v38.klantnr) AND (wrk.wrkopdrnr = v38.wrkopdrnr))
-> Seq Scan on wrkopdr wrk (cost=0.00..1240.30 rows=28344 width=18) (actual time=0.055..5.490 rows=9999 loops=1)
Filter: (klantnr = 1::numeric)
-> Hash (cost=19460.17..19460.17 rows=1 width=64) (actual time=221573.254..221573.254 rows=2621 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 113kB
-> Subquery Scan on v38 (cost=19460.15..19460.17 rows=1 width=64) (actual time=221570.429..221572.158 rows=2621 loops=1)
-> HashAggregate (cost=19460.15..19460.16 rows=1 width=52) (actual time=221570.429..221571.499 rows=2621 loops=1)
-> Nested Loop Left Join (cost=14049.90..19460.14 rows=1 width=52) (actual time=225.848..221495.813 rows=6801 loops=1)
Join Filter: ((vrr.klantnr = COALESCE(wrd.klantnr_grondstof, 0::numeric)) AND ((upper(vrr.artnr)) = COALESCE(wrd.artnr_grondstof, ''::text)) AND (vrr.partijnr = COALESCE(wrd.partijnr_grondstof, 0::numeric)))
Rows Removed by Join Filter: 308209258
-> Nested Loop (cost=1076.77..6011.60 rows=1 width=43) (actual time=9.506..587.824 rows=6801 loops=1)
-> Hash Right Join (cost=1076.77..5828.70 rows=69 width=43) (actual time=9.428..204.601 rows=7861 loops=1)
Hash Cond: ((wrd.klantnr = wrk.klantnr) AND (wrd.wrkopdrnr = wrk.wrkopdrnr))
Filter: (COALESCE(wrd.klantnr, 0::numeric) = 1::numeric)
Rows Removed by Filter: 1
-> Seq Scan on wrkopdrd wrd (cost=0.00..3117.73 rows=116873 width=38) (actual time=0.013..65.472 rows=116873 loops=1)
-> Hash (cost=1026.34..1026.34 rows=3362 width=16) (actual time=9.324..9.324 rows=3362 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 161kB
-> Bitmap Heap Scan on wrkopdr wrk (cost=98.31..1026.34 rows=3362 width=16) (actual time=0.850..7.843 rows=3362 loops=1)
Recheck Cond: (status = 'A'::text)
-> Bitmap Index Scan on wrkopdr_key_002 (cost=0.00..97.47 rows=3362 width=0) (actual time=0.763..0.763 rows=3362 loops=1)
Index Cond: (status = 'A'::text)
-> Index Scan using artikel_key_001 on artikel art (cost=0.00..2.64 rows=1 width=17) (actual time=0.037..0.043 rows=1 loops=7861)
Index Cond: ((klantnr = COALESCE(wrd.klantnr_grondstof, 0::numeric)) AND (artnr = COALESCE(wrd.artnr_grondstof, ''::text)))
Filter: (voorraadhoudend_jn = 'J'::text)
Rows Removed by Filter: 0
-> HashAggregate (cost=12973.14..13121.70 rows=11885 width=25) (actual time=0.027..19.661 rows=45319 loops=6801)
-> Seq Scan on voorraad vrr (cost=0.00..12340.71 rows=63243 width=25) (actual time=0.456..122.855 rows=62655 loops=1)
Filter: (status = 'A'::text)
Rows Removed by Filter: 113953
Total runtime: 221587.386 ms
(33 rows)
The output of the explain analyse of the fast query is:
Limit (cost=56294.55..57997.06 rows=142 width=18) (actual time=445.371..12739.474 rows=9999 loops=1)
-> Nested Loop Left Join (cost=56294.55..57997.06 rows=142 width=18) (actual time=445.368..12736.035 rows=9999 loops=1)
Join Filter: ((v38.klantnr = wrk.klantnr) AND (v38.wrkopdrnr = wrk.wrkopdrnr))
Rows Removed by Join Filter: 26206807
-> Nested Loop (cost=0.00..1532.01 rows=142 width=18) (actual time=0.055..18.652 rows=9999 loops=1)
Join Filter: (wrk.klantnr = s02.klantnr)
-> Index Scan using wrkovz91_selecties_key_001 on wrkovz91_selecties s02 (cost=0.00..8.27 rows=1 width=14) (actual time=0.021..0.021 rows=1 loops=1)
Index Cond: (nr = 1::numeric)
-> Seq Scan on wrkopdr wrk (cost=0.00..1169.44 rows=28344 width=18) (actual time=0.026..11.905 rows=9999 loops=1)
-> Materialize (cost=56294.55..56296.25 rows=68 width=64) (actual time=0.044..0.380 rows=2621 loops=9999)
-> Subquery Scan on v38 (cost=56294.55..56295.91 rows=68 width=64) (actual time=441.797..443.503 rows=2621 loops=1)
-> HashAggregate (cost=56294.55..56295.23 rows=68 width=52) (actual time=441.795..442.848 rows=2621 loops=1)
-> Hash Left Join (cost=14525.30..56293.70 rows=68 width=52) (actual time=255.847..433.386 rows=6801 loops=1)
Hash Cond: ((COALESCE(wrd.klantnr_grondstof, 0::numeric) = v40.klantnr) AND (COALESCE(wrd.artnr_grondstof, ''::text) = v40.artnr) AND (COALESCE(wrd.partijnr_grondstof, 0::numeric) = v40.partijnr))
-> Nested Loop (cost=1076.77..42541.70 rows=68 width=43) (actual time=10.356..171.502 rows=6801 loops=1)
-> Hash Right Join (cost=1076.77..5794.04 rows=13863 width=43) (actual time=10.286..91.471 rows=7862 loops=1)
Hash Cond: ((wrd.klantnr = wrk.klantnr) AND (wrd.wrkopdrnr = wrk.wrkopdrnr))
-> Seq Scan on wrkopdrd wrd (cost=0.00..3117.73 rows=116873 width=38) (actual time=0.014..38.276 rows=116873 loops=1)
-> Hash (cost=1026.34..1026.34 rows=3362 width=16) (actual time=10.179..10.179 rows=3362 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 161kB
-> Bitmap Heap Scan on wrkopdr wrk (cost=98.31..1026.34 rows=3362 width=16) (actual time=0.835..8.667 rows=3362 loops=1)
Recheck Cond: (status = 'A'::text)
-> Bitmap Index Scan on wrkopdr_key_002 (cost=0.00..97.47 rows=3362 width=0) (actual time=0.748..0.748 rows=3362 loops=1)
Index Cond: (status = 'A'::text)
-> Index Scan using artikel_key_001 on artikel art (cost=0.00..2.64 rows=1 width=17) (actual time=0.009..0.009 rows=1 loops=7862)
Index Cond: ((klantnr = COALESCE(wrd.klantnr_grondstof, 0::numeric)) AND (artnr = COALESCE(wrd.artnr_grondstof, ''::text)))
Filter: (voorraadhoudend_jn = 'J'::text)
Rows Removed by Filter: 0
-> Hash (cost=13240.55..13240.55 rows=11885 width=74) (actual time=245.430..245.430 rows=45319 loops=1)
Buckets: 2048 Batches: 1 Memory Usage: 2645kB
-> Subquery Scan on v40 (cost=12973.14..13240.55 rows=11885 width=74) (actual time=186.156..222.042 rows=45319 loops=1)
-> HashAggregate (cost=12973.14..13121.70 rows=11885 width=25) (actual time=186.154..209.633 rows=45319 loops=1)
-> Seq Scan on voorraad vrr (cost=0.00..12340.71 rows=63243 width=25) (actual time=0.453..126.361 rows=62655 loops=1)
Filter: (status = 'A'::text)
Rows Removed by Filter: 113953
Total runtime: 12742.125 ms
(36 rows)
I am having these results for analyze for a simple query that does not return more than 150 records from tables less than 200 records most of them, as I have a table that stores latest value and the other fields are FK of the data.
Update: see the new results from same query some our later. The site is not public and/or there should be not users right now as it is in development.
explain analyze
SELECT lv.station_id,
s.name AS station_name,
s.latitude,
s.longitude,
s.elevation,
lv.element_id,
e.symbol AS element_symbol,
u.symbol,
e.name AS element_name,
lv.last_datetime AS datetime,
lv.last_value AS valor,
s.basin_id,
s.municipality_id
FROM (((element_station lv /*350 records*/
JOIN stations s ON ((lv.station_id = s.id))) /*40 records*/
JOIN elements e ON ((lv.element_id = e.id))) /*103 records*/
JOIN units u ON ((e.unit_id = u.id))) /* 32 records */
WHERE s.id = lv.station_id AND e.id = lv.element_id AND lv.interval_id = 6 and
lv.last_datetime >= ((now() - '06:00:00'::interval) - '01:00:00'::interval)
I have already tried VACUUM and after that some is saved, but again after some times it goes up. I have implemented an index on the fields.
Nested Loop (cost=0.29..2654.66 rows=1 width=92) (actual time=1219.390..35296.253 rows=157 loops=1)
Join Filter: (e.unit_id = u.id)
Rows Removed by Join Filter: 4867
-> Nested Loop (cost=0.29..2652.93 rows=1 width=92) (actual time=1219.383..35294.083 rows=157 loops=1)
Join Filter: (lv.element_id = e.id)
Rows Removed by Join Filter: 16014
-> Nested Loop (cost=0.29..2648.62 rows=1 width=61) (actual time=1219.301..35132.373 rows=157 loops=1)
-> Seq Scan on element_station lv (cost=0.00..2640.30 rows=1 width=20) (actual time=1219.248..1385.517 rows=157 loops=1)
Filter: ((interval_id = 6) AND (last_datetime >= ((now() - '06:00:00'::interval) - '01:00:00'::interval)))
Rows Removed by Filter: 168
-> Index Scan using stations_pkey on stations s (cost=0.29..8.31 rows=1 width=45) (actual time=3.471..214.941 rows=1 loops=157)
Index Cond: (id = lv.station_id)
-> Seq Scan on elements e (cost=0.00..3.03 rows=103 width=35) (actual time=0.003..0.999 rows=103 loops=157)
-> Seq Scan on units u (cost=0.00..1.32 rows=32 width=8) (actual time=0.002..0.005 rows=32 loops=157)
Planning time: 8.312 ms
Execution time: 35296.427 ms
update, same query running it tonight; no changes:
Sort (cost=601.74..601.88 rows=55 width=92) (actual time=1.822..1.841 rows=172 loops=1)
Sort Key: lv.last_datetime DESC
Sort Method: quicksort Memory: 52kB
-> Nested Loop (cost=11.60..600.15 rows=55 width=92) (actual time=0.287..1.680 rows=172 loops=1)
-> Hash Join (cost=11.31..248.15 rows=55 width=51) (actual time=0.263..0.616 rows=172 loops=1)
Hash Cond: (e.unit_id = u.id)
-> Hash Join (cost=9.59..245.60 rows=75 width=51) (actual time=0.225..0.528 rows=172 loops=1)
Hash Cond: (lv.element_id = e.id)
-> Bitmap Heap Scan on element_station lv (cost=5.27..240.25 rows=75 width=20) (actual time=0.150..0.359 rows=172 loops=1)
Recheck Cond: ((last_datetime >= ((now() - '06:00:00'::interval) - '01:00:00'::interval)) AND (interval_id = 6))
Heap Blocks: exact=22
-> Bitmap Index Scan on element_station_latest (cost=0.00..5.25 rows=75 width=0) (actual time=0.136..0.136 rows=226 loops=1)
Index Cond: ((last_datetime >= ((now() - '06:00:00'::interval) - '01:00:00'::interval)) AND (interval_id = 6))
-> Hash (cost=3.03..3.03 rows=103 width=35) (actual time=0.062..0.062 rows=103 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 15kB
-> Seq Scan on elements e (cost=0.00..3.03 rows=103 width=35) (actual time=0.006..0.031 rows=103 loops=1)
-> Hash (cost=1.32..1.32 rows=32 width=8) (actual time=0.019..0.019 rows=32 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Seq Scan on units u (cost=0.00..1.32 rows=32 width=8) (actual time=0.003..0.005 rows=32 loops=1)
-> Index Scan using stations_pkey on stations s (cost=0.29..6.39 rows=1 width=45) (actual time=0.005..0.006 rows=1 loops=172)
Index Cond: (id = lv.station_id)
Planning time: 2.390 ms
Execution time: 2.009 ms
The problem is the misestimate of the number of rows in the sequential scan on element_station. Either autoanalyze has kicked in and calculated new statistics for the table or the data changed.
The problem is probably that PostgreSQL doesn't know the result of
((now() - '06:00:00'::interval) - '01:00:00'::interval)
at query planning time.
If that is possible for you, do it in two steps: First, calculate the expression above (either in PostgreSQL or on the client side). Then run the query with the result as a constant. That will make it easier for PostgreSQL to estimate the result count.
Please let me know if you need the table definitions. As I'm sure is obvious, I have several tables holding information about a user (partydevicestatus, groupgps) each have a foreign key relationship to the user held in partyrelationship and each row has an identifier of the "group" that user's in.
With this query, I simply want to, for a particular group (in this example, 6) get the user details, position and device info for each user.
I can clearly see with the explain that the Sort is the issue here, due to having 2 columns with a lot of data. However I have an index on both the columns being sorted on and it has yielded no improvement. I'm almost certain this is a terribly optimised query but I am not experienced enough with PostgreSQL to find a better one?
SELECT DISTINCT ON("public".groupgps.groupmember)
"public".groupgps.groupgps,
"public".groupgps.groupmember,
"public".groupgps.messagetype,
"public".groupgps.lat,
"public".groupgps.lon,
"public".groupgps.date_stamp,
"public".partyrelationship.to_party,
"public".partyrelationship.to_name,
"public".partyrelationship.image_url,
"public".partyrelationship.partyrelationship,
"public".partydevicestatus.connection_type,
"public".partydevicestatus.battery_level,
"public".partydevicestatus.charging_state,
"public".partydevicestatus.timestamp
FROM "public".groupgps
INNER JOIN "public".partyrelationship
ON "public".partyrelationship.partyrelationship = "public".groupgps.groupmember
INNER JOIN "public".partysettings
ON "public".partysettings.groupmember = "public".groupgps.groupmember
LEFT JOIN "public".partydevicestatus
ON "public".partydevicestatus.groupmember_id = "public".groupgps.groupmember
WHERE "public".partyrelationship.from_party = 6
AND "public".partysettings.gps_tracking_enabled = true
ORDER BY "public".groupgps.groupmember, "public".groupgps.date_stamp DESC
Explain Result
Unique (cost=1368961.43..1390701.85 rows=25 width=192) (actual time=24622.609..27043.061 rows=4 loops=1)
-> Sort (cost=1368961.43..1379831.64 rows=4348083 width=192) (actual time=24622.601..26604.659 rows=2221853 loops=1)
Sort Key: groupgps.groupmember, groupgps.date_stamp DESC
Sort Method: external merge Disk: 431400kB
-> Hash Left Join (cost=50.64..87013.93 rows=4348083 width=192) (actual time=0.499..3011.806 rows=2221853 loops=1)
Hash Cond: (groupgps.groupmember = partydevicestatus.groupmember_id)
-> Hash Join (cost=31.66..29732.32 rows=77101 width=167) (actual time=0.153..2242.950 rows=109041 loops=1)
Hash Cond: (groupgps.groupmember = partyrelationship.partyrelationship)
-> Seq Scan on groupgps (cost=0.00..24372.00 rows=1217200 width=50) (actual time=0.005..1933.528 rows=1217025 loops=1)
-> Hash (cost=31.48..31.48 rows=14 width=125) (actual time=0.141..0.141 rows=5 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Hash Join (cost=10.31..31.48 rows=14 width=125) (actual time=0.092..0.138 rows=5 loops=1)
Hash Cond: (partysettings.groupmember = partyrelationship.partyrelationship)
-> Seq Scan on partysettings (cost=0.00..20.75 rows=75 width=8) (actual time=0.003..0.038 rows=75 loops=1)
Filter: gps_tracking_enabled
-> Hash (cost=9.79..9.79 rows=42 width=117) (actual time=0.076..0.076 rows=42 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 11kB
-> Seq Scan on partyrelationship (cost=0.00..9.79 rows=42 width=117) (actual time=0.007..0.058 rows=42 loops=1)
Filter: (from_party = 6)
Rows Removed by Filter: 181
-> Hash (cost=12.88..12.88 rows=488 width=29) (actual time=0.341..0.341 rows=489 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 41kB
-> Seq Scan on partydevicestatus (cost=0.00..12.88 rows=488 width=29) (actual time=0.023..0.163 rows=489 loops=1)
Planning time: 0.878 ms
Execution time: 27218.016 ms