query execution x100 slower on prod than locally - postgresql

So I have identical databases with identical set of data, one is on production instance with 4GbRam another is i docker image.
I'm running same query:
select * from product p
inner join product_manufacturers pm on pm.product_id = p.id
inner join manufacturers m on pm.manufacturer_id = m.id
inner join brand b on p.brand_id = b.id
inner join tags t on t.product_id = p.id
inner join groups g on g.manufacturer_id = pm.id
inner join group_options gp on g.id = gp.group_id
inner join images i on i.product_id = p.id
where pm.enabled = true
and pm.available = true
and pm.launched = true
and p.available = true
and p.enabled = true
and p.id in (49, 77, 6, 12, 36)
order by b.id
Locally query execution time is 4 seconds and returned rows amount are 120k.
But on production execution time is 8 minutes and amount of returned rows are the same (as tables are identical).
What can be the issue, as I already told databases structure the same (same indexes etc.) and they have identical data, so why the execution time so different.... Also, when I'm runing the query on production directly through pgsql and storing result to the file, postgres execute query instantly but process of storing data to the file taking 8 minutes... what can be wrong with my instance???
I know a lot of unknowns here, but it's all what I have at the moment.
Local explain:
Nested Loop (cost=5.83..213.82 rows=1399 width=1820) (actual time=4.861..2352.400 rows=119503 loops=1)
Join Filter: (pm.product_id = t.product_id)
Buffers: shared hit=42646 read=85
-> Nested Loop (cost=5.55..107.33 rows=202 width=1797) (actual time=4.548..280.433 rows=12743 loops=1)
Buffers: shared hit=4420 read=82
-> Nested Loop (cost=5.26..81.32 rows=30 width=1763) (actual time=3.508..44.939 rows=950 loops=1)
Buffers: shared hit=931 read=38
-> Nested Loop (cost=4.98..77.05 rows=7 width=1697) (actual time=2.768..23.855 rows=166 loops=1)
Buffers: shared hit=436 read=35
-> Nested Loop (cost=4.71..75.14 rows=4 width=1677) (actual time=2.196..15.537 rows=83 loops=1)
Buffers: shared hit=195 read=26
-> Nested Loop (cost=4.56..74.36 rows=4 width=1399) (actual time=2.155..13.079 rows=83 loops=1)
Buffers: shared hit=30 read=25
-> Nested Loop (cost=0.27..40.11 rows=3 width=646) (actual time=1.803..3.986 rows=5 loops=1)
Join Filter: (p.brand_id = b.id)
Rows Removed by Join Filter: 158
Buffers: shared hit=17 read=4
-> Index Scan using product_pkey on product p (cost=0.27..34.93 rows=3 width=407) (actual time=0.830..1.995 rows=5 loops=1)
Index Cond: (id = ANY ('{49,77,6,12,36}'::integer[]))
Filter: (available AND enabled)
Buffers: shared hit=16 read=2
-> Materialize (cost=0.00..3.57 rows=38 width=239) (actual time=0.066..0.254 rows=33 loops=5)
Buffers: shared hit=1 read=2
-> Seq Scan on brand b (cost=0.00..3.38 rows=38 width=239) (actual time=0.298..0.461 rows=40 loops=1)
Buffers: shared hit=1 read=2
-> Bitmap Heap Scan on product_manufacturers pm (cost=4.29..11.41 rows=1 width=753) (actual time=0.674..1.615 rows=17 loops=5)
Recheck Cond: (product_id = p.id)
Filter: (available AND launched)
Rows Removed by Filter: 5
Heap Blocks: exact=21
Buffers: shared hit=13 read=21
-> Bitmap Index Scan on idx_e093f35a40e556f5 (cost=0.00..4.29 rows=2 width=0) (actual time=0.019..0.020 rows=27 loops=5)
Index Cond: (product_id = p.id)
Buffers: shared hit=9 read=1
-> Index Scan using manufacturer_pkey on manufacturers m (cost=0.14..0.20 rows=1 width=278) (actual time=0.011..0.011 rows=1 loops=83)
Index Cond: (id = pm.manufacturer_id)
Filter: enabled
Buffers: shared hit=165 read=1
-> Index Scan using idx_237d25c5d3e1ebb8 on groups g (cost=0.28..0.46 rows=2 width=20) (actual time=0.055..0.067 rows=2 loops=83)
Index Cond: (manufacturer_id = pm.id)
Buffers: shared hit=241 read=9
-> Index Scan using idx_4a5244b740e556f5 on icons i (cost=0.28..0.57 rows=4 width=66) (actual time=0.021..0.053 rows=6 loops=166)
Index Cond: (product_id = pm.product_id)
Buffers: shared hit=495 read=3
-> Index Scan using group_options_unique_idx on group_options gp (cost=0.29..0.80 rows=7 width=34) (actual time=0.023..0.104 rows=13 loops=950)
Index Cond: (group_id = g.id)
Buffers: shared hit=3489 read=44
-> Index Scan using idx_7edc9c5340e556f5 on tags t (cost=0.28..0.45 rows=6 width=19) (actual time=0.008..0.057 rows=9 loops=12743)
Index Cond: (product_id = i.product_id)
Buffers: shared hit=38226 read=3
Planning Time: 65.124 ms
Execution Time: 2926.918 ms
and here is production explain:
Nested Loop (cost=1.81..201.75 rows=1344 width=1760) (actual time=0.170..115.850 rows=119503 loops=1)
Join Filter: (pm.product_id = t.product_id)
Buffers: shared hit=43045
-> Nested Loop (cost=1.53..88.32 rows=211 width=1737) (actual time=0.145..13.585 rows=12743 loops=1)
Buffers: shared hit=4816
-> Nested Loop (cost=1.25..64.37 rows=29 width=1703) (actual time=0.120..2.492 rows=950 loops=1)
Buffers: shared hit=954
-> Nested Loop (cost=0.97..60.22 rows=7 width=1637) (actual time=0.103..1.468 rows=166 loops=1)
Buffers: shared hit=456
-> Nested Loop (cost=0.69..58.32 rows=4 width=1617) (actual time=0.084..0.950 rows=83 loops=1)
Join Filter: (p.brand_id = b.id)
Rows Removed by Join Filter: 598
Buffers: shared hit=206
-> Nested Loop (cost=0.69..53.64 rows=4 width=1411) (actual time=0.072..0.705 rows=83 loops=1)
Buffers: shared hit=205
-> Nested Loop (cost=0.55..52.62 rows=5 width=1153) (actual time=0.058..0.338 rows=83 loops=1)
Buffers: shared hit=39
-> Index Scan using product_pkey on product p (cost=0.27..22.60 rows=4 width=403) (actual time=0.039..0.072 rows=5 loops=1)
Index Cond: (id = ANY ('{49,77,6,12,36}'::integer[]))
Filter: (available AND enabled)
Buffers: shared hit=16
-> Index Scan using idx_e093f35a40e556f5 on product_manufacturers pm (cost=0.28..7.50 rows=1 width=750) (actual time=0.012..0.043 rows=17 loops=5)
Index Cond: (product_id = p.id)
Filter: (available AND launched)
Rows Removed by Filter: 5
Buffers: shared hit=23
-> Index Scan using manufacturer_pkey on manufacturer m (cost=0.14..0.20 rows=1 width=258) (actual time=0.003..0.003 rows=1 loops=83)
Index Cond: (id = pm.manufacturer_id)
Filter: enabled
Buffers: shared hit=166
-> Materialize (cost=0.00..2.56 rows=37 width=206) (actual time=0.000..0.001 rows=8 loops=83)
Buffers: shared hit=1
-> Seq Scan on brand b (cost=0.00..2.37 rows=37 width=206) (actual time=0.006..0.012 rows=27 loops=1)
Buffers: shared hit=1
-> Index Scan using idx_237d25c5d3e1ebb8 on groups g (cost=0.28..0.45 rows=2 width=20) (actual time=0.003..0.004 rows=2 loops=83)
Index Cond: (manufacturer_id = pm.id)
Buffers: shared hit=250
-> Index Scan using idx_4a5244b740e556f5 on icons i (cost=0.28..0.55 rows=4 width=66) (actual time=0.002..0.003 rows=6 loops=166)
Index Cond: (product_id = pm.product_id)
Buffers: shared hit=498
-> Index Scan using group_options_unique_idx on group_options gp (cost=0.29..0.76 rows=7 width=34) (actual time=0.003..0.007 rows=13 loops=950)
Index Cond: (group_id = g.id)
Buffers: shared hit=3862
-> Index Scan using idx_7edc9c5340e556f5 on tags t (cost=0.28..0.46 rows=6 width=19) (actual time=0.003..0.004 rows=9 loops=12743)
Index Cond: (product_id = i.product_id)
Buffers: shared hit=38229
Planning Time: 11.001 ms
Execution Time: 129.339 ms
Regarding indexes, I did not add any specific indexes yet, this query used in background and I did not try to optimize it yet as 3-4 seconds was allowable time but here it is, all the indexes I have:
CREATE UNIQUE INDEX product_pkey ON product USING btree (id);
CREATE INDEX idx_2645e26644f5d008 ON product USING btree (brand_id);
CREATE UNIQUE INDEX brand_pkey ON brand USING btree (id);
CREATE UNIQUE INDEX icon_pkey ON icons USING btree (id);
CREATE INDEX idx_4a5244b740e556f5 ON icons USING btree (product_id);
CREATE UNIQUE INDEX tags_pkey ON tags USING btree (id);
CREATE INDEX idx_7edc9c5340e556f5 ON tags USING btree (product_id);
CREATE INDEX tag_value_index ON tags USING btree (value);
CREATE UNIQUE INDEX manufacturer_pkey ON manufacturers USING btree (id);
CREATE UNIQUE INDEX groups_pkey ON groups_pkey USING btree (id);
CREATE INDEX idx_237d25c5d3e1ebb8 ON groups USING btree (manufacturer_id);
CREATE UNIQUE INDEX group_options_pkey ON group_options USING btree (id);
CREATE INDEX idx_2a964c28de23a8e3 ON group_options USING btree (group_id);
CREATE UNIQUE INDEX group_options_unique_idx ON group_options USING btree (group_id, option_id);
CREATE UNIQUE INDEX product_manufacturer_pkey ON product_manufacturers USING btree (id);
CREATE INDEX idx_e093f35a40e556f5 ON product_manufacturers USING btree (product_id);
CREATE UNIQUE INDEX manufacturer_unique_idx ON product_manufacturers USING btree (manufacturer_id, product_id);
CREATE INDEX idx_e093f35ad3e1ebb8 ON product_manufacturers USING btree (manufacturer_id)

Related

Join on UUID column not using index for multiple values

I have a PostgreSQL database that I cloned.
Database 1 has varchar(36) as primary keys
Database 2 (the clone) has UUID as primary keys.
Both contain the same data. What I don't understand is why queries on Database 1 will use the index but Database 2 will not. Here's the query:
EXPLAIN (ANALYZE, BUFFERS)
select * from table1
INNER JOIN table2 on table1.id = table2.table1_id
where table1.id in (
'541edffc-7179-42db-8c99-727be8c9ffec',
'eaac06d3-e44e-4e4a-8e11-1cdc6e562996'
);
Database 1
Nested Loop (cost=16.13..7234.96 rows=14 width=803) (actual time=0.072..0.112 rows=8 loops=1)
Buffers: shared hit=23
-> Index Scan using table1_pk on table1 (cost=0.56..17.15 rows=2 width=540) (actual time=0.042..0.054 rows=2 loops=1)
" Index Cond: ((id)::text = ANY ('{541edffc-7179-42db-8c99-727be8c9ffec,eaac06d3-e44e-4e4a-8e11-1cdc6e562996}'::text[]))"
Buffers: shared hit=12
-> Bitmap Heap Scan on table2 (cost=15.57..3599.86 rows=904 width=263) (actual time=0.022..0.023 rows=4 loops=2)
Recheck Cond: ((table1_id)::text = (table1.id)::text)
Heap Blocks: exact=3
Buffers: shared hit=11
-> Bitmap Index Scan on table2_table1_id_fk (cost=0.00..15.34 rows=904 width=0) (actual time=0.019..0.019 rows=4 loops=2)
Index Cond: ((table1_id)::text = (table1.id)::text)
Buffers: shared hit=8
Planning:
Buffers: shared hit=416
Planning Time: 1.869 ms
Execution Time: 0.330 ms
Database 2
Gather (cost=1000.57..1801008.91 rows=14 width=740) (actual time=11.580..42863.893 rows=8 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=863 read=631539 dirtied=631979 written=2523
-> Nested Loop (cost=0.56..1800007.51 rows=6 width=740) (actual time=28573.119..42856.696 rows=3 loops=3)
Buffers: shared hit=863 read=631539 dirtied=631979 written=2523
-> Parallel Seq Scan on table1 (cost=0.00..678896.46 rows=1 width=519) (actual time=28573.112..42855.524 rows=1 loops=3)
" Filter: (id = ANY ('{541edffc-7179-42db-8c99-727be8c9ffec,eaac06d3-e44e-4e4a-8e11-1cdc6e562996}'::uuid[]))"
Rows Removed by Filter: 2976413
Buffers: shared hit=854 read=631536 dirtied=631979 written=2523
-> Index Scan using table2_table1_id_fk on table2 (cost=0.56..1117908.70 rows=320236 width=221) (actual time=1.736..1.745 rows=4 loops=2)
Index Cond: (table1_id = table1.id)
Buffers: shared hit=9 read=3
Planning:
Buffers: shared hit=376 read=15
Planning Time: 43.594 ms
Execution Time: 42864.044 ms
Some notes:
The query is orders of magnitude faster in Database 1
Having only one ID in the WHERE clause activates the index in both databases
Casting to ::uuid has no impact
I understand that these results are because the query planner calculates that the cost of the index in the UUID (Database 2) case is too high. But I'm trying to understand why it thinks that and if there's something I can do.

PostgreSQL optimizing query - maybee nested loop is the problem

I have this very slow query:
SELECT DISTINCT et.id
FROM elementtype et
where et.id = any
(SELECT elementtypeid
FROM
(SELECT ic.elementtypeid
FROM
(SELECT categoryid
FROM issue
WHERE clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'
AND deleteddate IS NULL
GROUP BY categoryid) i
JOIN issuecategory ic ON ic.id = i.categoryid
UNION SELECT tc.elementtypeid
FROM
(SELECT categoryid
FROM task
WHERE clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'
AND deleteddate IS NULL
GROUP BY categoryid) t
JOIN taskcategory tc ON tc.id = t.categoryid) icc)
I have tried to change the ANY operator with IN, made an join instead of IN (in line 3 of the query) but it is still very slow, when the result is not cached.
I think it might be the nested loop making the problem - but I dont know if I can get rid of it - and why et only
As you can see, I use a couple of indexes _idx an of course primary keys on every table.
the elementtype table has ~6000 rows
the issue sub-query with these conditions (not group by) returns ~33000 rows
the task sub-query with these conditions (not group by) returns ~148000 rows
Is there any way to optimize the query?
EDIT:
As requested by #a_horse_with_no_name I add a query plan using the command he/she surgested. The best way to post it in here, is is using an image, I think:
QUERY PLAN
Unique (cost=473976.82..474453.63 rows=4453 width=16) (actual time=69897.728..69897.737 rows=1 loops=1)
Buffers: shared hit=61346 read=19651
-> Merge Join (cost=473976.82..474442.49 rows=4453 width=16) (actual time=69897.724..69897.731 rows=1 loops=1)
Merge Cond: (et.id = ic.elementtypeid)
Buffers: shared hit=61346 read=19651
-> Index Only Scan using elementtype_pkey on elementtype et (cost=0.28..384.47 rows=5879 width=16) (actual time=0.021..32.618 rows=1784 loops=1)
Heap Fetches: 1784
Buffers: shared hit=1699 read=54
-> Sort (cost=473976.54..473987.67 rows=4453 width=16) (actual time=69863.461..69863.464 rows=1 loops=1)
Sort Key: ic.elementtypeid
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=59647 read=19597
-> HashAggregate (cost=473617.61..473662.14 rows=4453 width=16) (actual time=69863.432..69863.436 rows=1 loops=1)
Group Key: ic.elementtypeid
Buffers: shared hit=59647 read=19597
-> Append (cost=107927.43..473606.48 rows=4453 width=16) (actual time=114.259..69863.317 rows=55 loops=1)
Buffers: shared hit=59647 read=19597
-> Hash Join (cost=107927.43..109170.43 rows=3625 width=16) (actual time=114.257..208.716 rows=46 loops=1)
Hash Cond: (ic.id = issue.categoryid)
Buffers: shared hit=15431
-> Seq Scan on issuecategory ic (cost=0.00..1100.36 rows=54336 width=32) (actual time=0.011..47.327 rows=54336 loops=1)
Buffers: shared hit=557
-> Hash (cost=107882.12..107882.12 rows=3625 width=16) (actual time=113.850..113.850 rows=46 loops=1)
Buckets: 4096 Batches: 1 Memory Usage: 35kB
Buffers: shared hit=14874
-> HashAggregate (cost=107809.62..107845.87 rows=3625 width=16) (actual time=113.738..113.795 rows=46 loops=1)
Group Key: issue.categoryid
Buffers: shared hit=14874
-> Bitmap Heap Scan on issue (cost=1801.41..107730.88 rows=31493 width=16) (actual time=7.279..81.266 rows=33670 loops=1)
Recheck Cond: (clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'::uuid)
Filter: (deleteddate IS NULL)
Rows Removed by Filter: 1362
Heap Blocks: exact=14636
Buffers: shared hit=14874
-> Bitmap Index Scan on issue_clientid_ix (cost=0.00..1793.54 rows=32681 width=0) (actual time=5.165..5.166 rows=35064 loops=1)
Index Cond: (clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'::uuid)
Buffers: shared hit=238
-> Nested Loop (cost=360635.19..364391.52 rows=828 width=16) (actual time=69603.779..69654.505 rows=9 loops=1)
Buffers: shared hit=44216 read=19597
-> HashAggregate (cost=360634.78..360643.06 rows=828 width=16) (actual time=69592.635..69592.657 rows=9 loops=1)
Group Key: task.categoryid
Buffers: shared hit=44198 read=19579
-> Bitmap Heap Scan on task (cost=3438.67..360280.46 rows=141728 width=16) (actual time=33.283..69416.182 rows=147931 loops=1)
Recheck Cond: (clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'::uuid)
Filter: (deleteddate IS NULL)
Rows Removed by Filter: 2329
Heap Blocks: exact=63193
Buffers: shared hit=44198 read=19579
-> Bitmap Index Scan on task_clientid_ix (cost=0.00..3403.24 rows=148091 width=0) (actual time=20.865..20.866 rows=150975 loops=1)
Index Cond: (clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'::uuid)
Buffers: shared hit=584
-> Index Scan using taskcategory_pkey on taskcategory tc (cost=0.42..4.52 rows=1 width=32) (actual time=6.865..6.865 rows=1 loops=9)
Index Cond: (id = task.categoryid)
Buffers: shared hit=18 read=18
Planning time: 1.173 ms
Execution time: 69899.380 ms
EDIT2:
issuecategory has index on id, clintid, elementypeid
issue has index on clientid, deleteddate and categoryid
taskcategory has index on id, clientid, elementtypeid,
task has index on clientid, id, deleteddate, categoryid
The problem is the bitmap heap scans. They seem to be jumping to a lot of different parts of the disk to fetch the data they need.
The best solution is probably to create indexes on (clientid, categoryid, deleteddate) on each table, or maybe (clientid, categoryid) where deleteddate is null. This will allow those bitmap heap scans to be replaced with index-only scans (assuming your tables are vacuumed well enough).
Other approaches would be to CLUSTER the tables so that rows with the same clientid are physically grouped together, or increase effective_io_concurrency so more IO can be done at the same time (assuming your storage system has multiple spindles in RAID/JBOD, or whatever the SSD equivalent to that is).

select max(id) from joined table inefficient query plan

I have a view req_res which joins 2 tables - Request and Response inner joined on requestId. Request has primary key - ID column.
When I query (Query 1):
select max(ID) from req_res where ID > 1000000 and ID < 2000000;
Explain plan: hash join: Index scan of Request.ID and seqential scan of Response.request_id
query duration: 30s
When I lower the boundaries to 900k (Query 2):
select max(ID) from req_res where ID > 1000000 and ID < 1900000;
Plan: nested loop: Index scan of Request.ID and Index only scan of Response.request_id
query duration: 3s
When I play with first query and disable hash join - set enable_hashjoin=off; I get Merge join plan. When I disable also the merge join plan with set enable_mergejoin=off; I get nested loop, which completes in 3 seconds (instead of 30 using hash join).
Size of the Request table is ~70 Mil records. Most of the requests have response counterpart, but some of them don't.
Version: PostgreSQL 10.10
req_res DDL:
CREATE OR REPLACE VIEW public.req_res
AS SELECT req.id,
res.req_id,
res.body::character varying(500),
res.time,
res.duration,
res.balance,
res.header::character varying(100),
res.created_at
FROM response res
JOIN request req ON req.req_id = res.req_id;
Query 1 Plan:
Aggregate (cost=2834115.70..2834115.71 rows=1 width=8) (actual time=154709.729..154709.730 rows=1 loops=1)
Buffers: shared hit=467727 read=685320 dirtied=214, temp read=240773 written=239751
-> Hash Join (cost=2493060.64..2831172.33 rows=1177346 width=8) (actual time=143800.101..154147.080 rows=1198706 loops=1)
Hash Cond: (req.req_id = res.req_id)
Buffers: shared hit=467727 read=685320 dirtied=214, temp read=240773 written=239751
-> Append (cost=0.44..55619.59 rows=1177346 width=16) (actual time=0.957..2354.648 rows=1200001 loops=1)
Buffers: shared hit=438960 read=32014
-> Index Scan using "5_5_req_pkey" on _hyper_2_5_chunk rs (cost=0.44..19000.10 rows=399803 width=16) (actual time=0.956..546.231 rows=399999 loops=1)
Index Cond: ((id >= 49600001) AND (id <= 50800001))
Buffers: shared hit=178872 read=10742
-> Index Scan using "7_7_req_pkey" on _hyper_2_7_chunk rs_1 (cost=0.44..36619.50 rows=777543 width=16) (actual time=0.037..767.744 rows=800002 loops=1)
Index Cond: ((id >= 49600001) AND (id <= 50800001))
Buffers: shared hit=260088 read=21272
-> Hash (cost=1367864.98..1367864.98 rows=68583298 width=8) (actual time=143681.850..143681.850 rows=68568554 loops=1)
Buckets: 262144 Batches: 512 Memory Usage: 7278kB
Buffers: shared hit=28764 read=653306 dirtied=214, temp written=233652
-> Append (cost=0.00..1367864.98 rows=68583298 width=8) (actual time=0.311..99590.021 rows=68568554 loops=1)
Buffers: shared hit=28764 read=653306 dirtied=214
-> Seq Scan on _hyper_3_2_chunk wt (cost=0.00..493704.44 rows=24941244 width=8) (actual time=0.309..14484.420 rows=24950147 loops=1)
Buffers: shared hit=661 read=243631
-> Seq Scan on _hyper_3_6_chunk wt_1 (cost=0.00..503935.04 rows=24978804 width=8) (actual time=0.334..14487.931 rows=24963020 loops=1)
Buffers: shared hit=168 read=253979
-> Seq Scan on _hyper_3_8_chunk wt_2 (cost=0.00..370225.50 rows=18663250 width=8) (actual time=0.327..10837.291 rows=18655387 loops=1)
Buffers: shared hit=27935 read=155696 dirtied=214
Planning time: 3.986 ms
Execution time: 154709.859 ms
Query 2 Plan:
Finalize Aggregate (cost=2634042.50..2634042.51 rows=1 width=8) (actual time=5525.626..5525.627 rows=1 loops=1)
Buffers: shared hit=8764620 read=12779
-> Gather (cost=2634042.29..2634042.50 rows=2 width=8) (actual time=5525.609..5525.705 rows=3 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=8764620 read=12779
-> Partial Aggregate (cost=2633042.29..2633042.30 rows=1 width=8) (actual time=5515.507..5515.508 rows=1 loops=3)
Buffers: shared hit=8764620 read=12779
-> Nested Loop (cost=0.88..2632023.83 rows=407382 width=8) (actual time=5.383..5261.979 rows=332978 loops=3)
Buffers: shared hit=8764620 read=12779
-> Append (cost=0.44..40514.98 rows=407383 width=16) (actual time=0.035..924.498 rows=333334 loops=3)
Buffers: shared hit=446706
-> Parallel Index Scan using "5_5_req_pkey" on _hyper_2_5_chunk rs (cost=0.44..16667.91 rows=166585 width=16) (actual time=0.033..169.854 rows=133333 loops=3)
Index Cond: ((id >= 49600001) AND (id <= 50600001))
Buffers: shared hit=190175
-> Parallel Index Scan using "7_7_req_pkey" on _hyper_2_7_chunk rs_1 (cost=0.44..23847.07 rows=240798 width=16) (actual time=0.039..336.091 rows=200001 loops=3)
Index Cond: ((id >= 49600001) AND (id <= 50600001))
Buffers: shared hit=256531
-> Append (cost=0.44..6.33 rows=3 width=8) (actual time=0.011..0.011 rows=1 loops=1000001)
Buffers: shared hit=8317914 read=12779
-> Index Only Scan using "2_2_response_pkey" on _hyper_3_2_chunk wt (cost=0.44..2.11 rows=1 width=8) (actual time=0.003..0.003 rows=0 loops=1000001)
Index Cond: (req_id = req.req_id)
Heap Fetches: 0
Buffers: shared hit=3000005
-> Index Only Scan using "6_6_response_pkey" on _hyper_3_6_chunk wt_1 (cost=0.44..2.11 rows=1 width=8) (actual time=0.003..0.003 rows=1 loops=1000001)
Index Cond: (req_id = req.req_id)
Heap Fetches: 192906
Buffers: shared hit=3551440 read=7082
-> Index Only Scan using "8_8_response_pkey" on _hyper_3_8_chunk wt_2 (cost=0.44..2.10 rows=1 width=8) (actual time=0.003..0.003 rows=1 loops=443006)
Index Cond: (req_id = req.req_id)
Heap Fetches: 162913
Buffers: shared hit=1766469 read=5697
Planning time: 0.839 ms
Execution time: 5525.814 ms

PostgreSQL: Sequential scan despite having indexes

I have the following two tables.
person_addresses
address_normalization
The person_addresses table has a field named address_id as the primary key and address_normalization has the corresponding field address_id which has an index on it.
Now, when I explain the following query, I see a sequential scan.
SELECT
count(*)
FROM
mp_member2.person_addresses pa
JOIN mp_member2.address_normalization an ON
an.address_id = pa.address_id
WHERE
an.sr_modification_time >= 1550692189468;
-- Result: 2654
Please refer to the following screenshot.
You see that there is a sequential scan after the hash join. I'm not sure I understand this part; why would a sequential scan follow a hash join.
And as seen in the query above, the set of records returned is also low.
Is this expected behaviour or am I doing something wrong?
Update #1: I also have indices on the sr_modification_time fields of both the tables
Update #2: Full execution plan
Aggregate (cost=206944.74..206944.75 rows=1 width=0) (actual time=2807.844..2807.844 rows=1 loops=1)
Buffers: shared hit=4629 read=82217
-> Hash Join (cost=2881.95..206825.15 rows=47836 width=0) (actual time=0.775..2807.160 rows=2654 loops=1)
Hash Cond: (pa.address_id = an.address_id)
Buffers: shared hit=4629 read=82217
-> Seq Scan on person_addresses pa (cost=0.00..135924.93 rows=4911993 width=8) (actual time=0.005..1374.610 rows=4911993 loops=1)
Buffers: shared hit=4588 read=82217
-> Hash (cost=2432.05..2432.05 rows=35992 width=18) (actual time=0.756..0.756 rows=1005 loops=1)
Buckets: 4096 Batches: 1 Memory Usage: 41kB
Buffers: shared hit=41
-> Index Scan using mp_member2_address_normalization_mod_time on address_normalization an (cost=0.43..2432.05 rows=35992 width=18) (actual time=0.012..0.424 rows=1005 loops=1)
Index Cond: (sr_modification_time >= 1550692189468::bigint)
Buffers: shared hit=41
Planning time: 0.244 ms
Execution time: 2807.885 ms
Update #3: I tried with a newer timestamp and it used an index scan.
EXPLAIN (
ANALYZE
, buffers
, format TEXT
) SELECT
COUNT(*)
FROM
mp_member2.person_addresses pa
JOIN mp_member2.address_normalization an ON
an.address_id = pa.address_id
WHERE
an.sr_modification_time >= 1557507300342;
-- count: 1364
Query Plan:
Aggregate (cost=295.48..295.49 rows=1 width=0) (actual time=2.770..2.770 rows=1 loops=1)
Buffers: shared hit=1404
-> Nested Loop (cost=4.89..295.43 rows=19 width=0) (actual time=0.038..2.491 rows=1364 loops=1)
Buffers: shared hit=1404
-> Index Scan using mp_member2_address_normalization_mod_time on address_normalization an (cost=0.43..8.82 rows=14 width=18) (actual time=0.009..0.142 rows=341 loops=1)
Index Cond: (sr_modification_time >= 1557507300342::bigint)
Buffers: shared hit=14
-> Bitmap Heap Scan on person_addresses pa (cost=4.46..20.43 rows=4 width=8) (actual time=0.004..0.005 rows=4 loops=341)
Recheck Cond: (address_id = an.address_id)
Heap Blocks: exact=360
Buffers: shared hit=1390
-> Bitmap Index Scan on idx_mp_member2_person_addresses_address_id (cost=0.00..4.46 rows=4 width=0) (actual time=0.003..0.003 rows=4 loops=341)
Index Cond: (address_id = an.address_id)
Buffers: shared hit=1030
Planning time: 0.214 ms
Execution time: 2.816 ms
That is the expected behavior because you don't have index for sr_modification_time so after create the hash join db has to scan the whole set to check each row for the sr_modification_time value
You should create:
index for (sr_modification_time)
or composite index for (address_id , sr_modification_time )

postgres two column sort low performance

I've got a query that performs multiple joins. I try to get only those positions of each keyword that are latest in results.
Here is the query:
SELECT DISTINCT ON (p.keyword_id)
a.id AS account_id,
w.parent_id AS parent_id,
w.name AS name,
p.position AS position
FROM websites w
JOIN accounts a ON w.account_id = a.id
JOIN keywords k ON k.website_id = w.parent_id
JOIN positions p ON p.website_id = w.parent_id
WHERE a.amount > 0 AND w.parent_id NOTNULL AND (round((a.amount / a.payment_renewal_period), 2) BETWEEN 1 AND 19)
ORDER BY p.keyword_id, p.created_at DESC;
Plan with costs for that query is as follows:
Unique (cost=73673.65..76630.38 rows=264 width=40) (actual time=30777.117..49143.023 rows=259 loops=1)
-> Sort (cost=73673.65..75152.02 rows=591347 width=40) (actual time=30777.116..47352.373 rows=10891486 loops=1)
Sort Key: p.keyword_id, p.created_at DESC
Sort Method: external merge Disk: 512672kB
-> Merge Join (cost=219.59..812.26 rows=591347 width=40) (actual time=3.487..3827.028 rows=10891486 loops=1)
Merge Cond: (w.parent_id = k.website_id)
-> Nested Loop (cost=128.46..597.73 rows=1268 width=44) (actual time=3.378..108.915 rows=61582 loops=1)
-> Nested Loop (cost=2.28..39.86 rows=1 width=28) (actual time=0.026..0.216 rows=7 loops=1)
-> Index Scan using index_websites_on_parent_id on websites w (cost=0.14..15.08 rows=4 width=28) (actual time=0.004..0.023 rows=7 loops=1)
Index Cond: (parent_id IS NOT NULL)
-> Bitmap Heap Scan on accounts a (cost=2.15..6.18 rows=1 width=4) (actual time=0.019..0.020 rows=1 loops=7)
Recheck Cond: (id = w.account_id)
Filter: ((amount > '0'::numeric) AND (round((amount / (payment_renewal_period)::numeric), 2) >= '1'::numeric) AND (round((amount / (payment_renewal_period)::numeric), 2) <= '19'::numeric))
Heap Blocks: exact=7
-> Bitmap Index Scan on accounts_pkey (cost=0.00..2.15 rows=1 width=0) (actual time=0.006..0.006 rows=1 loops=7)
Index Cond: (id = w.account_id)
-> Bitmap Heap Scan on positions p (cost=126.18..511.57 rows=4631 width=16) (actual time=0.994..8.226 rows=8797 loops=7)
Recheck Cond: (website_id = w.parent_id)
Heap Blocks: exact=1004
-> Bitmap Index Scan on index_positions_on_5_columns (cost=0.00..125.02 rows=4631 width=0) (actual time=0.965..0.965 rows=8797 loops=7)
Index Cond: (website_id = w.parent_id)
-> Sort (cost=18.26..18.92 rows=264 width=4) (actual time=0.106..1013.966 rows=10891487 loops=1)
Sort Key: k.website_id
Sort Method: quicksort Memory: 37kB
-> Seq Scan on keywords k (cost=0.00..7.64 rows=264 width=4) (actual time=0.005..0.039 rows=263 loops=1)
Planning time: 1.081 ms
Execution time: 49184.222 ms
The thing is when I run query with w.id instead of w.parent_id in join positions part total cost decreases to
Unique (cost=3621.07..3804.99 rows=264 width=40) (actual time=128.430..139.550 rows=259 loops=1)
-> Sort (cost=3621.07..3713.03 rows=36784 width=40) (actual time=128.429..135.444 rows=40385 loops=1)
Sort Key: p.keyword_id, p.created_at DESC
Sort Method: external sort Disk: 2000kB
-> Merge Join (cost=128.73..831.59 rows=36784 width=40) (actual time=25.521..63.299 rows=40385 loops=1)
Merge Cond: (k.website_id = w.id)
-> Index Only Scan using index_keywords_on_website_id_deleted_at on keywords k (cost=0.27..24.23 rows=264 width=4) (actual time=0.137..0.274 rows=263 loops=1)
Heap Fetches: 156
-> Materialize (cost=128.46..606.85 rows=1268 width=44) (actual time=3.772..49.587 rows=72242 loops=1)
-> Nested Loop (cost=128.46..603.68 rows=1268 width=44) (actual time=3.769..30.530 rows=61582 loops=1)
-> Nested Loop (cost=2.28..45.80 rows=1 width=32) (actual time=0.047..0.204 rows=7 loops=1)
-> Index Scan using websites_pkey on websites w (cost=0.14..21.03 rows=4 width=32) (actual time=0.007..0.026 rows=7 loops=1)
Filter: (parent_id IS NOT NULL)
Rows Removed by Filter: 4
-> Bitmap Heap Scan on accounts a (cost=2.15..6.18 rows=1 width=4) (actual time=0.018..0.019 rows=1 loops=7)
Recheck Cond: (id = w.account_id)
Filter: ((amount > '0'::numeric) AND (round((amount / (payment_renewal_period)::numeric), 2) >= '1'::numeric) AND (round((amount / (payment_renewal_period)::numeric), 2) <= '19'::numeric))
Heap Blocks: exact=7
-> Bitmap Index Scan on accounts_pkey (cost=0.00..2.15 rows=1 width=0) (actual time=0.004..0.004 rows=1 loops=7)
Index Cond: (id = w.account_id)
-> Bitmap Heap Scan on positions p (cost=126.18..511.57 rows=4631 width=16) (actual time=0.930..2.341 rows=8797 loops=7)
Recheck Cond: (website_id = w.parent_id)
Heap Blocks: exact=1004
-> Bitmap Index Scan on index_positions_on_5_columns (cost=0.00..125.02 rows=4631 width=0) (actual time=0.906..0.906 rows=8797 loops=7)
Index Cond: (website_id = w.parent_id)
Planning time: 1.124 ms
Execution time: 157.167 ms
Indexes on websites
Indexes:
"websites_pkey" PRIMARY KEY, btree (id)
"index_websites_on_account_id" btree (account_id)
"index_websites_on_deleted_at" btree (deleted_at)
"index_websites_on_domain_id" btree (domain_id)
"index_websites_on_parent_id" btree (parent_id)
"index_websites_on_processed_at" btree (processed_at)
Indexes on positions
Indexes:
"positions_pkey" PRIMARY KEY, btree (id)
"index_positions_on_5_columns" UNIQUE, btree (website_id, keyword_id, created_at, engine_id, region_id)
"overlap_index" btree (keyword_id, created_at)
The second EXPLAIN output shows more than 200 times fewer rows, so it is hardly surprising that sorting is much faster.
You will notice that the sort spills to disk in both cases (Sort Method: external merge Disk: ...kB). If you can keep the sort in memory by raising work_mem, it will be much faster.
But the first sort is so large that you won't be able to fit it in memory.
Ideas to speed up the query:
An index on (keyword_id, created_at)for positions. Not sure if that helps though.
Do the filtering first, like this:
SELECT
a.id AS account_id,
w.parent_id AS parent_id,
w.name AS name,
p.position AS position
FROM (SELECT DISTINCT ON (keyword_id)
positions,
website_id,
keyword_id,
created_at
FROM positions
ORDER BY keyword_id, created_at DESC) p
JOIN ...
WHERE ...
ORDER BY p.keyword_id, p.created_at DESC;
Remark: The DISTINCT ON is somewhat strange, since you do not ORDER BY the values of the SELECT list, so the result values are not well defined.