We are having a problem with a specific query that sometimes takes over 620.000 ms on execute, but this is random because if we execute the same query again, it takes aprox. 79ms
This is the query:
INSERT INTO product_store(id_store, cod_product, account_id, stock, product,active)
SELECT s.id_store, t.cod_product, t.account_id, 0, i.product, i.active
FROM tmp_import_product t, store s, product i
WHERE t.id_process_import_product = 99988
AND s.account_id = t.account_id
AND s.active = 'S'
AND i.account_id = t.account_id
AND i.cod_product = t.cod_product
AND i.manage_stock = 'S'
AND i.variant_stock = 'N'
AND t.existent = 'S'
AND t.error = 'N'
AND NOT EXISTS(SELECT 1 FROM product_store i WHERE i.cod_product = t.cod_product AND i.account_id = t.account_id AND i.id_store = s.id_store)
And this is the plan
Insert on product_store (cost=1.57..296.01 rows=1 width=733) (actual rows=0 loops=1)
Buffers: shared hit=532388600 read=21556 dirtied=186
-> Nested Loop (cost=1.57..296.01 rows=1 width=733) (actual rows=0 loops=1)
Buffers: shared hit=532388600 read=21556 dirtied=186
-> Nested Loop (cost=1.14..291.09 rows=1 width=76) (actual rows=22200 loops=1)
Join Filter: (t.account_id = s.account_id)
Buffers: shared hit=74893
-> Nested Loop (cost=0.85..290.40 rows=2 width=76) (actual rows=7400 loops=1)
Buffers: shared hit=37893
-> Index Scan using ix_tmp_import_product_2 on tmp_import_product t (cost=0.42..120.32 rows=64 width=21) (actual rows=7400 loops=1)
Index Cond: (t.id_process_import_product = '96680'::bigint)
Filter: ((t.existent = 'S'::bpchar) AND (t.error = 'N'::bpchar))
Rows Removed by Filter: 102
Buffers: shared hit=3515
-> Index Scan using pk_product on product i (cost=0.43..2.66 rows=1 width=64) (actual rows=1 loops=7400)
Index Cond: (((i.cod_product)::text = t.cod_product) AND (i.account_id = t.account_id))
Filter: ((i.manage_stock = 'S'::bpchar) AND (i.variant_stock = 'N'::bpchar))
Rows Removed by Filter: 0
Buffers: shared hit=34291
-> Index Scan using store_account_id_active_index on store s (cost=0.28..0.32 rows=2 width=16) (actual rows=3 loops=7400)
Index Cond: ((s.account_id = i.account_id) AND (s.active = 'S'::bpchar))
Buffers: shared hit=37000
-> Index Scan using idx_product_store_account_id_id_store on product_store i_1 (cost=0.43..4.90 rows=1 width=24) (actual rows=1 loops=22200)
Index Cond: ((i_1.id_store = s.id_store) AND (i_1.account_id = t.account_id))
Filter: ((i_1.cod_product)::text = t.cod_product)
Rows Removed by Filter: 141420
Buffers: shared hit=532306240 read=21556 dirtied=186
Trigger t_insert_product_stock: time=0.000 calls=1
I think that because tmp_import_product has a bad estimation, it affects the next joins.
How can I check if the stats of the table are updated? (already check last autovacuum and it is running) How can I improve this query? Could it be something else?
Related
So I have identical databases with identical set of data, one is on production instance with 4GbRam another is i docker image.
I'm running same query:
select * from product p
inner join product_manufacturers pm on pm.product_id = p.id
inner join manufacturers m on pm.manufacturer_id = m.id
inner join brand b on p.brand_id = b.id
inner join tags t on t.product_id = p.id
inner join groups g on g.manufacturer_id = pm.id
inner join group_options gp on g.id = gp.group_id
inner join images i on i.product_id = p.id
where pm.enabled = true
and pm.available = true
and pm.launched = true
and p.available = true
and p.enabled = true
and p.id in (49, 77, 6, 12, 36)
order by b.id
Locally query execution time is 4 seconds and returned rows amount are 120k.
But on production execution time is 8 minutes and amount of returned rows are the same (as tables are identical).
What can be the issue, as I already told databases structure the same (same indexes etc.) and they have identical data, so why the execution time so different.... Also, when I'm runing the query on production directly through pgsql and storing result to the file, postgres execute query instantly but process of storing data to the file taking 8 minutes... what can be wrong with my instance???
I know a lot of unknowns here, but it's all what I have at the moment.
Local explain:
Nested Loop (cost=5.83..213.82 rows=1399 width=1820) (actual time=4.861..2352.400 rows=119503 loops=1)
Join Filter: (pm.product_id = t.product_id)
Buffers: shared hit=42646 read=85
-> Nested Loop (cost=5.55..107.33 rows=202 width=1797) (actual time=4.548..280.433 rows=12743 loops=1)
Buffers: shared hit=4420 read=82
-> Nested Loop (cost=5.26..81.32 rows=30 width=1763) (actual time=3.508..44.939 rows=950 loops=1)
Buffers: shared hit=931 read=38
-> Nested Loop (cost=4.98..77.05 rows=7 width=1697) (actual time=2.768..23.855 rows=166 loops=1)
Buffers: shared hit=436 read=35
-> Nested Loop (cost=4.71..75.14 rows=4 width=1677) (actual time=2.196..15.537 rows=83 loops=1)
Buffers: shared hit=195 read=26
-> Nested Loop (cost=4.56..74.36 rows=4 width=1399) (actual time=2.155..13.079 rows=83 loops=1)
Buffers: shared hit=30 read=25
-> Nested Loop (cost=0.27..40.11 rows=3 width=646) (actual time=1.803..3.986 rows=5 loops=1)
Join Filter: (p.brand_id = b.id)
Rows Removed by Join Filter: 158
Buffers: shared hit=17 read=4
-> Index Scan using product_pkey on product p (cost=0.27..34.93 rows=3 width=407) (actual time=0.830..1.995 rows=5 loops=1)
Index Cond: (id = ANY ('{49,77,6,12,36}'::integer[]))
Filter: (available AND enabled)
Buffers: shared hit=16 read=2
-> Materialize (cost=0.00..3.57 rows=38 width=239) (actual time=0.066..0.254 rows=33 loops=5)
Buffers: shared hit=1 read=2
-> Seq Scan on brand b (cost=0.00..3.38 rows=38 width=239) (actual time=0.298..0.461 rows=40 loops=1)
Buffers: shared hit=1 read=2
-> Bitmap Heap Scan on product_manufacturers pm (cost=4.29..11.41 rows=1 width=753) (actual time=0.674..1.615 rows=17 loops=5)
Recheck Cond: (product_id = p.id)
Filter: (available AND launched)
Rows Removed by Filter: 5
Heap Blocks: exact=21
Buffers: shared hit=13 read=21
-> Bitmap Index Scan on idx_e093f35a40e556f5 (cost=0.00..4.29 rows=2 width=0) (actual time=0.019..0.020 rows=27 loops=5)
Index Cond: (product_id = p.id)
Buffers: shared hit=9 read=1
-> Index Scan using manufacturer_pkey on manufacturers m (cost=0.14..0.20 rows=1 width=278) (actual time=0.011..0.011 rows=1 loops=83)
Index Cond: (id = pm.manufacturer_id)
Filter: enabled
Buffers: shared hit=165 read=1
-> Index Scan using idx_237d25c5d3e1ebb8 on groups g (cost=0.28..0.46 rows=2 width=20) (actual time=0.055..0.067 rows=2 loops=83)
Index Cond: (manufacturer_id = pm.id)
Buffers: shared hit=241 read=9
-> Index Scan using idx_4a5244b740e556f5 on icons i (cost=0.28..0.57 rows=4 width=66) (actual time=0.021..0.053 rows=6 loops=166)
Index Cond: (product_id = pm.product_id)
Buffers: shared hit=495 read=3
-> Index Scan using group_options_unique_idx on group_options gp (cost=0.29..0.80 rows=7 width=34) (actual time=0.023..0.104 rows=13 loops=950)
Index Cond: (group_id = g.id)
Buffers: shared hit=3489 read=44
-> Index Scan using idx_7edc9c5340e556f5 on tags t (cost=0.28..0.45 rows=6 width=19) (actual time=0.008..0.057 rows=9 loops=12743)
Index Cond: (product_id = i.product_id)
Buffers: shared hit=38226 read=3
Planning Time: 65.124 ms
Execution Time: 2926.918 ms
and here is production explain:
Nested Loop (cost=1.81..201.75 rows=1344 width=1760) (actual time=0.170..115.850 rows=119503 loops=1)
Join Filter: (pm.product_id = t.product_id)
Buffers: shared hit=43045
-> Nested Loop (cost=1.53..88.32 rows=211 width=1737) (actual time=0.145..13.585 rows=12743 loops=1)
Buffers: shared hit=4816
-> Nested Loop (cost=1.25..64.37 rows=29 width=1703) (actual time=0.120..2.492 rows=950 loops=1)
Buffers: shared hit=954
-> Nested Loop (cost=0.97..60.22 rows=7 width=1637) (actual time=0.103..1.468 rows=166 loops=1)
Buffers: shared hit=456
-> Nested Loop (cost=0.69..58.32 rows=4 width=1617) (actual time=0.084..0.950 rows=83 loops=1)
Join Filter: (p.brand_id = b.id)
Rows Removed by Join Filter: 598
Buffers: shared hit=206
-> Nested Loop (cost=0.69..53.64 rows=4 width=1411) (actual time=0.072..0.705 rows=83 loops=1)
Buffers: shared hit=205
-> Nested Loop (cost=0.55..52.62 rows=5 width=1153) (actual time=0.058..0.338 rows=83 loops=1)
Buffers: shared hit=39
-> Index Scan using product_pkey on product p (cost=0.27..22.60 rows=4 width=403) (actual time=0.039..0.072 rows=5 loops=1)
Index Cond: (id = ANY ('{49,77,6,12,36}'::integer[]))
Filter: (available AND enabled)
Buffers: shared hit=16
-> Index Scan using idx_e093f35a40e556f5 on product_manufacturers pm (cost=0.28..7.50 rows=1 width=750) (actual time=0.012..0.043 rows=17 loops=5)
Index Cond: (product_id = p.id)
Filter: (available AND launched)
Rows Removed by Filter: 5
Buffers: shared hit=23
-> Index Scan using manufacturer_pkey on manufacturer m (cost=0.14..0.20 rows=1 width=258) (actual time=0.003..0.003 rows=1 loops=83)
Index Cond: (id = pm.manufacturer_id)
Filter: enabled
Buffers: shared hit=166
-> Materialize (cost=0.00..2.56 rows=37 width=206) (actual time=0.000..0.001 rows=8 loops=83)
Buffers: shared hit=1
-> Seq Scan on brand b (cost=0.00..2.37 rows=37 width=206) (actual time=0.006..0.012 rows=27 loops=1)
Buffers: shared hit=1
-> Index Scan using idx_237d25c5d3e1ebb8 on groups g (cost=0.28..0.45 rows=2 width=20) (actual time=0.003..0.004 rows=2 loops=83)
Index Cond: (manufacturer_id = pm.id)
Buffers: shared hit=250
-> Index Scan using idx_4a5244b740e556f5 on icons i (cost=0.28..0.55 rows=4 width=66) (actual time=0.002..0.003 rows=6 loops=166)
Index Cond: (product_id = pm.product_id)
Buffers: shared hit=498
-> Index Scan using group_options_unique_idx on group_options gp (cost=0.29..0.76 rows=7 width=34) (actual time=0.003..0.007 rows=13 loops=950)
Index Cond: (group_id = g.id)
Buffers: shared hit=3862
-> Index Scan using idx_7edc9c5340e556f5 on tags t (cost=0.28..0.46 rows=6 width=19) (actual time=0.003..0.004 rows=9 loops=12743)
Index Cond: (product_id = i.product_id)
Buffers: shared hit=38229
Planning Time: 11.001 ms
Execution Time: 129.339 ms
Regarding indexes, I did not add any specific indexes yet, this query used in background and I did not try to optimize it yet as 3-4 seconds was allowable time but here it is, all the indexes I have:
CREATE UNIQUE INDEX product_pkey ON product USING btree (id);
CREATE INDEX idx_2645e26644f5d008 ON product USING btree (brand_id);
CREATE UNIQUE INDEX brand_pkey ON brand USING btree (id);
CREATE UNIQUE INDEX icon_pkey ON icons USING btree (id);
CREATE INDEX idx_4a5244b740e556f5 ON icons USING btree (product_id);
CREATE UNIQUE INDEX tags_pkey ON tags USING btree (id);
CREATE INDEX idx_7edc9c5340e556f5 ON tags USING btree (product_id);
CREATE INDEX tag_value_index ON tags USING btree (value);
CREATE UNIQUE INDEX manufacturer_pkey ON manufacturers USING btree (id);
CREATE UNIQUE INDEX groups_pkey ON groups_pkey USING btree (id);
CREATE INDEX idx_237d25c5d3e1ebb8 ON groups USING btree (manufacturer_id);
CREATE UNIQUE INDEX group_options_pkey ON group_options USING btree (id);
CREATE INDEX idx_2a964c28de23a8e3 ON group_options USING btree (group_id);
CREATE UNIQUE INDEX group_options_unique_idx ON group_options USING btree (group_id, option_id);
CREATE UNIQUE INDEX product_manufacturer_pkey ON product_manufacturers USING btree (id);
CREATE INDEX idx_e093f35a40e556f5 ON product_manufacturers USING btree (product_id);
CREATE UNIQUE INDEX manufacturer_unique_idx ON product_manufacturers USING btree (manufacturer_id, product_id);
CREATE INDEX idx_e093f35ad3e1ebb8 ON product_manufacturers USING btree (manufacturer_id)
I have a view req_res which joins 2 tables - Request and Response inner joined on requestId. Request has primary key - ID column.
When I query (Query 1):
select max(ID) from req_res where ID > 1000000 and ID < 2000000;
Explain plan: hash join: Index scan of Request.ID and seqential scan of Response.request_id
query duration: 30s
When I lower the boundaries to 900k (Query 2):
select max(ID) from req_res where ID > 1000000 and ID < 1900000;
Plan: nested loop: Index scan of Request.ID and Index only scan of Response.request_id
query duration: 3s
When I play with first query and disable hash join - set enable_hashjoin=off; I get Merge join plan. When I disable also the merge join plan with set enable_mergejoin=off; I get nested loop, which completes in 3 seconds (instead of 30 using hash join).
Size of the Request table is ~70 Mil records. Most of the requests have response counterpart, but some of them don't.
Version: PostgreSQL 10.10
req_res DDL:
CREATE OR REPLACE VIEW public.req_res
AS SELECT req.id,
res.req_id,
res.body::character varying(500),
res.time,
res.duration,
res.balance,
res.header::character varying(100),
res.created_at
FROM response res
JOIN request req ON req.req_id = res.req_id;
Query 1 Plan:
Aggregate (cost=2834115.70..2834115.71 rows=1 width=8) (actual time=154709.729..154709.730 rows=1 loops=1)
Buffers: shared hit=467727 read=685320 dirtied=214, temp read=240773 written=239751
-> Hash Join (cost=2493060.64..2831172.33 rows=1177346 width=8) (actual time=143800.101..154147.080 rows=1198706 loops=1)
Hash Cond: (req.req_id = res.req_id)
Buffers: shared hit=467727 read=685320 dirtied=214, temp read=240773 written=239751
-> Append (cost=0.44..55619.59 rows=1177346 width=16) (actual time=0.957..2354.648 rows=1200001 loops=1)
Buffers: shared hit=438960 read=32014
-> Index Scan using "5_5_req_pkey" on _hyper_2_5_chunk rs (cost=0.44..19000.10 rows=399803 width=16) (actual time=0.956..546.231 rows=399999 loops=1)
Index Cond: ((id >= 49600001) AND (id <= 50800001))
Buffers: shared hit=178872 read=10742
-> Index Scan using "7_7_req_pkey" on _hyper_2_7_chunk rs_1 (cost=0.44..36619.50 rows=777543 width=16) (actual time=0.037..767.744 rows=800002 loops=1)
Index Cond: ((id >= 49600001) AND (id <= 50800001))
Buffers: shared hit=260088 read=21272
-> Hash (cost=1367864.98..1367864.98 rows=68583298 width=8) (actual time=143681.850..143681.850 rows=68568554 loops=1)
Buckets: 262144 Batches: 512 Memory Usage: 7278kB
Buffers: shared hit=28764 read=653306 dirtied=214, temp written=233652
-> Append (cost=0.00..1367864.98 rows=68583298 width=8) (actual time=0.311..99590.021 rows=68568554 loops=1)
Buffers: shared hit=28764 read=653306 dirtied=214
-> Seq Scan on _hyper_3_2_chunk wt (cost=0.00..493704.44 rows=24941244 width=8) (actual time=0.309..14484.420 rows=24950147 loops=1)
Buffers: shared hit=661 read=243631
-> Seq Scan on _hyper_3_6_chunk wt_1 (cost=0.00..503935.04 rows=24978804 width=8) (actual time=0.334..14487.931 rows=24963020 loops=1)
Buffers: shared hit=168 read=253979
-> Seq Scan on _hyper_3_8_chunk wt_2 (cost=0.00..370225.50 rows=18663250 width=8) (actual time=0.327..10837.291 rows=18655387 loops=1)
Buffers: shared hit=27935 read=155696 dirtied=214
Planning time: 3.986 ms
Execution time: 154709.859 ms
Query 2 Plan:
Finalize Aggregate (cost=2634042.50..2634042.51 rows=1 width=8) (actual time=5525.626..5525.627 rows=1 loops=1)
Buffers: shared hit=8764620 read=12779
-> Gather (cost=2634042.29..2634042.50 rows=2 width=8) (actual time=5525.609..5525.705 rows=3 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=8764620 read=12779
-> Partial Aggregate (cost=2633042.29..2633042.30 rows=1 width=8) (actual time=5515.507..5515.508 rows=1 loops=3)
Buffers: shared hit=8764620 read=12779
-> Nested Loop (cost=0.88..2632023.83 rows=407382 width=8) (actual time=5.383..5261.979 rows=332978 loops=3)
Buffers: shared hit=8764620 read=12779
-> Append (cost=0.44..40514.98 rows=407383 width=16) (actual time=0.035..924.498 rows=333334 loops=3)
Buffers: shared hit=446706
-> Parallel Index Scan using "5_5_req_pkey" on _hyper_2_5_chunk rs (cost=0.44..16667.91 rows=166585 width=16) (actual time=0.033..169.854 rows=133333 loops=3)
Index Cond: ((id >= 49600001) AND (id <= 50600001))
Buffers: shared hit=190175
-> Parallel Index Scan using "7_7_req_pkey" on _hyper_2_7_chunk rs_1 (cost=0.44..23847.07 rows=240798 width=16) (actual time=0.039..336.091 rows=200001 loops=3)
Index Cond: ((id >= 49600001) AND (id <= 50600001))
Buffers: shared hit=256531
-> Append (cost=0.44..6.33 rows=3 width=8) (actual time=0.011..0.011 rows=1 loops=1000001)
Buffers: shared hit=8317914 read=12779
-> Index Only Scan using "2_2_response_pkey" on _hyper_3_2_chunk wt (cost=0.44..2.11 rows=1 width=8) (actual time=0.003..0.003 rows=0 loops=1000001)
Index Cond: (req_id = req.req_id)
Heap Fetches: 0
Buffers: shared hit=3000005
-> Index Only Scan using "6_6_response_pkey" on _hyper_3_6_chunk wt_1 (cost=0.44..2.11 rows=1 width=8) (actual time=0.003..0.003 rows=1 loops=1000001)
Index Cond: (req_id = req.req_id)
Heap Fetches: 192906
Buffers: shared hit=3551440 read=7082
-> Index Only Scan using "8_8_response_pkey" on _hyper_3_8_chunk wt_2 (cost=0.44..2.10 rows=1 width=8) (actual time=0.003..0.003 rows=1 loops=443006)
Index Cond: (req_id = req.req_id)
Heap Fetches: 162913
Buffers: shared hit=1766469 read=5697
Planning time: 0.839 ms
Execution time: 5525.814 ms
I use a simple query with a WHERE on a fixed value. This query does a left join on a temporary view. For some reason this query is performing very badly. I guess that the view is being executed for every row and not only for the selected rows. When I replace the fixed value with a value from a temporary table, the query performs MUCH better (about 15-20 times faster). Why is this?
I use postgresql version 9.2.15.
I added a temporary table 'wrkovz91-selecties' with only 1 record to pass the selection-value of the WHERE instruction to the query.
The view 'view_wrk_012_000001' that is joined in the query is pretty 'heavy', because it contains a nested other view ('view_wrk_013_000006').
First create the temporary table and add one record:
CREATE TEMPORARY TABLE WRKOVZ91_SELECTIES
(
NR DECIMAL(006) NOT NULL,
KLANTNR DECIMAL(008),
CONSTRAINT WRKOVZ91_SELECTIES_KEY_001 PRIMARY KEY (NR)
);
INSERT INTO WRKOVZ91_SELECTIES
(NR, KLANTNR) VALUES (1, 1);
Then create the temporary view (with a nested view):
CREATE TEMPORARY VIEW view_wrk_013_000006 AS
SELECT vrr.klantnr AS klantnr,
UPPER(vrr.artnr) AS artnr,
vrr.partijnr AS partijnr,
SUM(vrr.kollo) AS colli_op_voorraad
FROM voorraad vrr
WHERE (vrr.status = 'A'::text)
GROUP BY 1,
2,
3;
CREATE TEMPORARY VIEW view_wrk_012_000001 AS
SELECT COALESCE(wrd.klantnr,0) AS klantnr,
COALESCE(wrd.wrkopdrnr,0) AS wrkopdrnr,
MIN(COALESCE(wrd.recept_benodigd_colli,0) *
COALESCE(wrk.te_produceren,0)) AS vrije_voorraad_ind,
MIN(COALESCE(v40.colli_op_voorraad,0)) AS hulpveld
FROM wrkopdr wrk
LEFT JOIN wrkopdrd wrd ON wrd.klantnr = wrk.klantnr
AND wrd.wrkopdrnr = wrk.wrkopdrnr
LEFT JOIN view_wrk_013_000006 v40 ON v40.klantnr =
COALESCE(wrd.klantnr_grondstof,0)
AND v40.artnr = COALESCE(wrd.artnr_grondstof,'')
AND v40.partijnr = COALESCE(wrd.partijnr_grondstof,0)
LEFT JOIN artikel art ON art.klantnr = COALESCE(wrd.klantnr_grondstof,0)
AND art.artnr = COALESCE(wrd.artnr_grondstof,'')
WHERE wrk.status = 'A'::text
AND art.voorraadhoudend_jn = 'J'::text
GROUP BY 1,
2;
The query that performs badly is simple:
SELECT WRK.KLANTNR,
WRK.WRKOPDRNR,
WRK.MIL_UITVOER_DATUM
FROM WRKOPDR WRK
LEFT JOIN VIEW_WRK_012_000001 V38 ON V38.KLANTNR = WRK.KLANTNR AND
V38.WRKOPDRNR = WRK.WRKOPDRNR
LEFT JOIN WRKOVZ91_SELECTIES S02 ON S02.NR = 1
WHERE WRK.KLANTNR = 1
LIMIT 9999;
The query that performs MUCH better is (note the small difference):
SELECT WRK.KLANTNR,
WRK.WRKOPDRNR,
WRK.MIL_UITVOER_DATUM
FROM WRKOPDR WRK
LEFT JOIN VIEW_WRK_012_000001 V38 ON V38.KLANTNR = WRK.KLANTNR AND
V38.WRKOPDRNR = WRK.WRKOPDRNR
LEFT JOIN WRKOVZ91_SELECTIES S02 ON S02.NR = 1
WHERE WRK.KLANTNR = S02.KLANTNR
LIMIT 9999;
I cannot understand why the slow query is performing so badly. It takes in my test-data about 219 secondes. The fast query is taking only 12 secondes. The only difference between the 2 queries is the selection-value in the WHERE.
Does anyone have an explanation for this behaviour?
In addition, the output of the explain analyse of the slow query is:
Limit (cost=19460.18..19972.73 rows=9999 width=18) (actual time=221573.343..221585.953 rows=9999 loops=1)
-> Hash Left Join (cost=19460.18..20913.07 rows=28344 width=18) (actual time=221573.341..221583.701 rows=9999 loops=1)
Hash Cond: ((wrk.klantnr = v38.klantnr) AND (wrk.wrkopdrnr = v38.wrkopdrnr))
-> Seq Scan on wrkopdr wrk (cost=0.00..1240.30 rows=28344 width=18) (actual time=0.055..5.490 rows=9999 loops=1)
Filter: (klantnr = 1::numeric)
-> Hash (cost=19460.17..19460.17 rows=1 width=64) (actual time=221573.254..221573.254 rows=2621 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 113kB
-> Subquery Scan on v38 (cost=19460.15..19460.17 rows=1 width=64) (actual time=221570.429..221572.158 rows=2621 loops=1)
-> HashAggregate (cost=19460.15..19460.16 rows=1 width=52) (actual time=221570.429..221571.499 rows=2621 loops=1)
-> Nested Loop Left Join (cost=14049.90..19460.14 rows=1 width=52) (actual time=225.848..221495.813 rows=6801 loops=1)
Join Filter: ((vrr.klantnr = COALESCE(wrd.klantnr_grondstof, 0::numeric)) AND ((upper(vrr.artnr)) = COALESCE(wrd.artnr_grondstof, ''::text)) AND (vrr.partijnr = COALESCE(wrd.partijnr_grondstof, 0::numeric)))
Rows Removed by Join Filter: 308209258
-> Nested Loop (cost=1076.77..6011.60 rows=1 width=43) (actual time=9.506..587.824 rows=6801 loops=1)
-> Hash Right Join (cost=1076.77..5828.70 rows=69 width=43) (actual time=9.428..204.601 rows=7861 loops=1)
Hash Cond: ((wrd.klantnr = wrk.klantnr) AND (wrd.wrkopdrnr = wrk.wrkopdrnr))
Filter: (COALESCE(wrd.klantnr, 0::numeric) = 1::numeric)
Rows Removed by Filter: 1
-> Seq Scan on wrkopdrd wrd (cost=0.00..3117.73 rows=116873 width=38) (actual time=0.013..65.472 rows=116873 loops=1)
-> Hash (cost=1026.34..1026.34 rows=3362 width=16) (actual time=9.324..9.324 rows=3362 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 161kB
-> Bitmap Heap Scan on wrkopdr wrk (cost=98.31..1026.34 rows=3362 width=16) (actual time=0.850..7.843 rows=3362 loops=1)
Recheck Cond: (status = 'A'::text)
-> Bitmap Index Scan on wrkopdr_key_002 (cost=0.00..97.47 rows=3362 width=0) (actual time=0.763..0.763 rows=3362 loops=1)
Index Cond: (status = 'A'::text)
-> Index Scan using artikel_key_001 on artikel art (cost=0.00..2.64 rows=1 width=17) (actual time=0.037..0.043 rows=1 loops=7861)
Index Cond: ((klantnr = COALESCE(wrd.klantnr_grondstof, 0::numeric)) AND (artnr = COALESCE(wrd.artnr_grondstof, ''::text)))
Filter: (voorraadhoudend_jn = 'J'::text)
Rows Removed by Filter: 0
-> HashAggregate (cost=12973.14..13121.70 rows=11885 width=25) (actual time=0.027..19.661 rows=45319 loops=6801)
-> Seq Scan on voorraad vrr (cost=0.00..12340.71 rows=63243 width=25) (actual time=0.456..122.855 rows=62655 loops=1)
Filter: (status = 'A'::text)
Rows Removed by Filter: 113953
Total runtime: 221587.386 ms
(33 rows)
The output of the explain analyse of the fast query is:
Limit (cost=56294.55..57997.06 rows=142 width=18) (actual time=445.371..12739.474 rows=9999 loops=1)
-> Nested Loop Left Join (cost=56294.55..57997.06 rows=142 width=18) (actual time=445.368..12736.035 rows=9999 loops=1)
Join Filter: ((v38.klantnr = wrk.klantnr) AND (v38.wrkopdrnr = wrk.wrkopdrnr))
Rows Removed by Join Filter: 26206807
-> Nested Loop (cost=0.00..1532.01 rows=142 width=18) (actual time=0.055..18.652 rows=9999 loops=1)
Join Filter: (wrk.klantnr = s02.klantnr)
-> Index Scan using wrkovz91_selecties_key_001 on wrkovz91_selecties s02 (cost=0.00..8.27 rows=1 width=14) (actual time=0.021..0.021 rows=1 loops=1)
Index Cond: (nr = 1::numeric)
-> Seq Scan on wrkopdr wrk (cost=0.00..1169.44 rows=28344 width=18) (actual time=0.026..11.905 rows=9999 loops=1)
-> Materialize (cost=56294.55..56296.25 rows=68 width=64) (actual time=0.044..0.380 rows=2621 loops=9999)
-> Subquery Scan on v38 (cost=56294.55..56295.91 rows=68 width=64) (actual time=441.797..443.503 rows=2621 loops=1)
-> HashAggregate (cost=56294.55..56295.23 rows=68 width=52) (actual time=441.795..442.848 rows=2621 loops=1)
-> Hash Left Join (cost=14525.30..56293.70 rows=68 width=52) (actual time=255.847..433.386 rows=6801 loops=1)
Hash Cond: ((COALESCE(wrd.klantnr_grondstof, 0::numeric) = v40.klantnr) AND (COALESCE(wrd.artnr_grondstof, ''::text) = v40.artnr) AND (COALESCE(wrd.partijnr_grondstof, 0::numeric) = v40.partijnr))
-> Nested Loop (cost=1076.77..42541.70 rows=68 width=43) (actual time=10.356..171.502 rows=6801 loops=1)
-> Hash Right Join (cost=1076.77..5794.04 rows=13863 width=43) (actual time=10.286..91.471 rows=7862 loops=1)
Hash Cond: ((wrd.klantnr = wrk.klantnr) AND (wrd.wrkopdrnr = wrk.wrkopdrnr))
-> Seq Scan on wrkopdrd wrd (cost=0.00..3117.73 rows=116873 width=38) (actual time=0.014..38.276 rows=116873 loops=1)
-> Hash (cost=1026.34..1026.34 rows=3362 width=16) (actual time=10.179..10.179 rows=3362 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 161kB
-> Bitmap Heap Scan on wrkopdr wrk (cost=98.31..1026.34 rows=3362 width=16) (actual time=0.835..8.667 rows=3362 loops=1)
Recheck Cond: (status = 'A'::text)
-> Bitmap Index Scan on wrkopdr_key_002 (cost=0.00..97.47 rows=3362 width=0) (actual time=0.748..0.748 rows=3362 loops=1)
Index Cond: (status = 'A'::text)
-> Index Scan using artikel_key_001 on artikel art (cost=0.00..2.64 rows=1 width=17) (actual time=0.009..0.009 rows=1 loops=7862)
Index Cond: ((klantnr = COALESCE(wrd.klantnr_grondstof, 0::numeric)) AND (artnr = COALESCE(wrd.artnr_grondstof, ''::text)))
Filter: (voorraadhoudend_jn = 'J'::text)
Rows Removed by Filter: 0
-> Hash (cost=13240.55..13240.55 rows=11885 width=74) (actual time=245.430..245.430 rows=45319 loops=1)
Buckets: 2048 Batches: 1 Memory Usage: 2645kB
-> Subquery Scan on v40 (cost=12973.14..13240.55 rows=11885 width=74) (actual time=186.156..222.042 rows=45319 loops=1)
-> HashAggregate (cost=12973.14..13121.70 rows=11885 width=25) (actual time=186.154..209.633 rows=45319 loops=1)
-> Seq Scan on voorraad vrr (cost=0.00..12340.71 rows=63243 width=25) (actual time=0.453..126.361 rows=62655 loops=1)
Filter: (status = 'A'::text)
Rows Removed by Filter: 113953
Total runtime: 12742.125 ms
(36 rows)
I need help optimizing this PostgreSQL Query. I've already created some indexes but the only index being used is "cp_campaign_task_ap_index".
EXPLAIN ANALYZE SELECT * FROM campaign_phones
INNER JOIN campaigns ON campaigns.id = campaign_phones.campaign_id
INNER JOIN campaign_ports ON campaign_ports.campaign_id = campaigns.id
WHERE campaign_phones.task_id IS NULL
AND campaign_phones.assigned_port = 2
AND (campaigns.auto_start IS TRUE)
AND (campaigns.starts_at_date::date <= '2018-07-08'
AND campaigns.ends_at_date::date >= '2018-07-08')
AND (campaign_ports.gateway_port_id = 611)
AND (campaign_ports.op_mode != 1)
ORDER BY campaigns.last_sent_at ASC NULLS FIRST LIMIT 1;`
The output of the command is:
Limit (cost=26031.86..26031.87 rows=1 width=475) (actual time=2335.421..2335.421 rows=1 loops=1)
-> Sort (cost=26031.86..26047.26 rows=6158 width=475) (actual time=2335.419..2335.419 rows=1 loops=1)
Sort Key: campaigns.last_sent_at NULLS FIRST
Sort Method: top-N heapsort Memory: 25kB
-> Nested Loop (cost=136.10..26001.07 rows=6158 width=475) (actual time=1.176..1510.276 rows=36666 loops=1)
Join Filter: (campaigns.id = campaign_phones.campaign_id)
-> Nested Loop (cost=0.00..28.28 rows=2 width=218) (actual time=0.163..0.435 rows=4 loops=1)
Join Filter: (campaigns.id = campaign_ports.campaign_id)
Rows Removed by Join Filter: 113
-> Seq Scan on campaign_ports (cost=0.00..21.48 rows=9 width=55) (actual time=0.017..0.318 rows=9 loops=1)
Filter: ((op_mode <> 1) AND (gateway_port_id = 611))
Rows Removed by Filter: 823
-> Materialize (cost=0.00..5.74 rows=8 width=163) (actual time=0.001..0.008 rows=13 loops=9)
-> Seq Scan on campaigns (cost=0.00..5.70 rows=8 width=163) (actual time=0.011..0.050 rows=13 loops=1)
Filter: ((auto_start IS TRUE) AND ((starts_at_date)::date <= '2018-07-08'::date) AND ((ends_at_date)::date >= '2018-07-08'::date))
Rows Removed by Filter: 22
-> Bitmap Heap Scan on campaign_phones (cost=136.10..12931.82 rows=4366 width=249) (actual time=43.079..302.895 rows=9166 loops=4)
Recheck Cond: ((campaign_id = campaign_ports.campaign_id) AND (task_id IS NULL) AND (assigned_port = 2))
Heap Blocks: exact=6686
-> Bitmap Index Scan on cp_campaign_task_ap_index (cost=0.00..135.01 rows=4366 width=0) (actual time=8.884..8.884 rows=9167 loops=4)
Index Cond: ((campaign_id = campaign_ports.campaign_id) AND (task_id IS NULL) AND (assigned_port = 2))
Planning time: 1.115 ms
Execution time: 2335.563 ms
The "campaign_phones" relation could have many rows, perhaps a million.
I don't know where to start optimizing, perhaps creating indexes or changing query structure.
Thanks.
I have two queries one is using OR expression and is running very fast. The other query is similar but is using IN expression instead of OR and is running very slow. I would appreciate if you could let me know how to make the query using IN as fast as the one using OR. The table has 15 million records
SELECT e.id
FROM events e,
resources r
WHERE e.resource_id = r.id
AND resource_type_id IN (19872817,
282)
ORDER BY occurrence_date DESC LIMIT 100
Limit (cost=0.85..228363.80 rows=100 width=12) (actual time=238.668..57470.017 rows=19 loops=1)
-> Nested Loop (cost=0.85..26211499.28 rows=11478 width=12) (actual time=238.667..57470.010 rows=19 loops=1)
Join Filter: (e.resource_id = r.id)
Rows Removed by Join Filter: 507548495
-> Index Scan using eventoccurrencedateindex on events e (cost=0.43..603333.83 rows=15380258 width=16) (actual time=0.023..2798.538 rows=15380258 loops=1)
-> Materialize (cost=0.42..36.16 rows=111 width=4) (actual time=0.000..0.001 rows=33 loops=15380258)
-> Index Scan using resources_type_fk_index on resources r (cost=0.42..35.60 rows=111 width=4) (actual time=0.014..0.107 rows=33 loops=1)
Index Cond: (resource_type_id = ANY ('{19872817,282}'::integer[]))
Total runtime: 57470.057 ms
SELECT e.id
FROM events e,
resources r
WHERE e.resource_id = r.id
AND (resource_type_id = '19872817' OR resource_type_id = '282')
ORDER BY occurrence_date DESC LIMIT 100
Limit (cost=10.17..14.22 rows=100 width=12) (actual time=0.060..0.181 rows=100 loops=1)
-> Nested Loop (cost=10.17..34747856.23 rows=858030913 width=12) (actual time=0.059..0.167 rows=100 loops=1)
Join Filter: (((e.resource_id = r.id) AND (r.resource_type_id = 19872817)) OR (r.resource_type_id = 282))
-> Index Scan using eventoccurrencedateindex on events e (cost=0.43..603333.83 rows=15380258 width=16) (actual time=0.018..0.019 rows=4 loops=1)
-> Materialize (cost=9.74..349.92 rows=111 width=8) (actual time=0.009..0.023 rows=25
loops=4)
-> Bitmap Heap Scan on resources r (cost=9.74..349.36 rows=111 width=8) (actual time=0.034..0.081 rows=33 loops=1)
Recheck Cond: ((resource_type_id = 19872817) OR (resource_type_id = 282))
-> BitmapOr (cost=9.74..9.74 rows=111 width=0) (actual time=0.023..0.023 rows=0 loops=1)
-> Bitmap Index Scan on resources_type_fk_index (cost=0.00..4.84 rows=56 width=0) (actual time=0.009..0.009 rows=0 loops=1)
Index Cond: (resource_type_id = 19872817)
-> Bitmap Index Scan on resources_type_fk_index (cost=0.00..4.84 rows=56 width=0) (actual time=0.014..0.014 rows=33 loops=1)
Index Cond: (resource_type_id = 282)" "Total runtime: 0.242 ms
This is strange in the or version:
Join Filter: (
((e.resource_id = r.id) AND (r.resource_type_id = 19872817))
OR
(r.resource_type_id = 282)
)
It does e.resource_id = r.id AND r.resource_type_id = 19872817 first and then OR r.resource_type_id = 282 which is wrong. Are you sure you issued the correct condition in that query? Notice that there must be parenthesis wrapping the OR:
e.resource_id = r.id
AND
(r.resource_type_id = 19872817 OR r.resource_type_id = 282)