I have the following two tables.
person_addresses
address_normalization
The person_addresses table has a field named address_id as the primary key and address_normalization has the corresponding field address_id which has an index on it.
Now, when I explain the following query, I see a sequential scan.
SELECT
count(*)
FROM
mp_member2.person_addresses pa
JOIN mp_member2.address_normalization an ON
an.address_id = pa.address_id
WHERE
an.sr_modification_time >= 1550692189468;
-- Result: 2654
Please refer to the following screenshot.
You see that there is a sequential scan after the hash join. I'm not sure I understand this part; why would a sequential scan follow a hash join.
And as seen in the query above, the set of records returned is also low.
Is this expected behaviour or am I doing something wrong?
Update #1: I also have indices on the sr_modification_time fields of both the tables
Update #2: Full execution plan
Aggregate (cost=206944.74..206944.75 rows=1 width=0) (actual time=2807.844..2807.844 rows=1 loops=1)
Buffers: shared hit=4629 read=82217
-> Hash Join (cost=2881.95..206825.15 rows=47836 width=0) (actual time=0.775..2807.160 rows=2654 loops=1)
Hash Cond: (pa.address_id = an.address_id)
Buffers: shared hit=4629 read=82217
-> Seq Scan on person_addresses pa (cost=0.00..135924.93 rows=4911993 width=8) (actual time=0.005..1374.610 rows=4911993 loops=1)
Buffers: shared hit=4588 read=82217
-> Hash (cost=2432.05..2432.05 rows=35992 width=18) (actual time=0.756..0.756 rows=1005 loops=1)
Buckets: 4096 Batches: 1 Memory Usage: 41kB
Buffers: shared hit=41
-> Index Scan using mp_member2_address_normalization_mod_time on address_normalization an (cost=0.43..2432.05 rows=35992 width=18) (actual time=0.012..0.424 rows=1005 loops=1)
Index Cond: (sr_modification_time >= 1550692189468::bigint)
Buffers: shared hit=41
Planning time: 0.244 ms
Execution time: 2807.885 ms
Update #3: I tried with a newer timestamp and it used an index scan.
EXPLAIN (
ANALYZE
, buffers
, format TEXT
) SELECT
COUNT(*)
FROM
mp_member2.person_addresses pa
JOIN mp_member2.address_normalization an ON
an.address_id = pa.address_id
WHERE
an.sr_modification_time >= 1557507300342;
-- count: 1364
Query Plan:
Aggregate (cost=295.48..295.49 rows=1 width=0) (actual time=2.770..2.770 rows=1 loops=1)
Buffers: shared hit=1404
-> Nested Loop (cost=4.89..295.43 rows=19 width=0) (actual time=0.038..2.491 rows=1364 loops=1)
Buffers: shared hit=1404
-> Index Scan using mp_member2_address_normalization_mod_time on address_normalization an (cost=0.43..8.82 rows=14 width=18) (actual time=0.009..0.142 rows=341 loops=1)
Index Cond: (sr_modification_time >= 1557507300342::bigint)
Buffers: shared hit=14
-> Bitmap Heap Scan on person_addresses pa (cost=4.46..20.43 rows=4 width=8) (actual time=0.004..0.005 rows=4 loops=341)
Recheck Cond: (address_id = an.address_id)
Heap Blocks: exact=360
Buffers: shared hit=1390
-> Bitmap Index Scan on idx_mp_member2_person_addresses_address_id (cost=0.00..4.46 rows=4 width=0) (actual time=0.003..0.003 rows=4 loops=341)
Index Cond: (address_id = an.address_id)
Buffers: shared hit=1030
Planning time: 0.214 ms
Execution time: 2.816 ms
That is the expected behavior because you don't have index for sr_modification_time so after create the hash join db has to scan the whole set to check each row for the sr_modification_time value
You should create:
index for (sr_modification_time)
or composite index for (address_id , sr_modification_time )
Related
I have a PostgreSQL database that I cloned.
Database 1 has varchar(36) as primary keys
Database 2 (the clone) has UUID as primary keys.
Both contain the same data. What I don't understand is why queries on Database 1 will use the index but Database 2 will not. Here's the query:
EXPLAIN (ANALYZE, BUFFERS)
select * from table1
INNER JOIN table2 on table1.id = table2.table1_id
where table1.id in (
'541edffc-7179-42db-8c99-727be8c9ffec',
'eaac06d3-e44e-4e4a-8e11-1cdc6e562996'
);
Database 1
Nested Loop (cost=16.13..7234.96 rows=14 width=803) (actual time=0.072..0.112 rows=8 loops=1)
Buffers: shared hit=23
-> Index Scan using table1_pk on table1 (cost=0.56..17.15 rows=2 width=540) (actual time=0.042..0.054 rows=2 loops=1)
" Index Cond: ((id)::text = ANY ('{541edffc-7179-42db-8c99-727be8c9ffec,eaac06d3-e44e-4e4a-8e11-1cdc6e562996}'::text[]))"
Buffers: shared hit=12
-> Bitmap Heap Scan on table2 (cost=15.57..3599.86 rows=904 width=263) (actual time=0.022..0.023 rows=4 loops=2)
Recheck Cond: ((table1_id)::text = (table1.id)::text)
Heap Blocks: exact=3
Buffers: shared hit=11
-> Bitmap Index Scan on table2_table1_id_fk (cost=0.00..15.34 rows=904 width=0) (actual time=0.019..0.019 rows=4 loops=2)
Index Cond: ((table1_id)::text = (table1.id)::text)
Buffers: shared hit=8
Planning:
Buffers: shared hit=416
Planning Time: 1.869 ms
Execution Time: 0.330 ms
Database 2
Gather (cost=1000.57..1801008.91 rows=14 width=740) (actual time=11.580..42863.893 rows=8 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=863 read=631539 dirtied=631979 written=2523
-> Nested Loop (cost=0.56..1800007.51 rows=6 width=740) (actual time=28573.119..42856.696 rows=3 loops=3)
Buffers: shared hit=863 read=631539 dirtied=631979 written=2523
-> Parallel Seq Scan on table1 (cost=0.00..678896.46 rows=1 width=519) (actual time=28573.112..42855.524 rows=1 loops=3)
" Filter: (id = ANY ('{541edffc-7179-42db-8c99-727be8c9ffec,eaac06d3-e44e-4e4a-8e11-1cdc6e562996}'::uuid[]))"
Rows Removed by Filter: 2976413
Buffers: shared hit=854 read=631536 dirtied=631979 written=2523
-> Index Scan using table2_table1_id_fk on table2 (cost=0.56..1117908.70 rows=320236 width=221) (actual time=1.736..1.745 rows=4 loops=2)
Index Cond: (table1_id = table1.id)
Buffers: shared hit=9 read=3
Planning:
Buffers: shared hit=376 read=15
Planning Time: 43.594 ms
Execution Time: 42864.044 ms
Some notes:
The query is orders of magnitude faster in Database 1
Having only one ID in the WHERE clause activates the index in both databases
Casting to ::uuid has no impact
I understand that these results are because the query planner calculates that the cost of the index in the UUID (Database 2) case is too high. But I'm trying to understand why it thinks that and if there's something I can do.
I have this very slow query:
SELECT DISTINCT et.id
FROM elementtype et
where et.id = any
(SELECT elementtypeid
FROM
(SELECT ic.elementtypeid
FROM
(SELECT categoryid
FROM issue
WHERE clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'
AND deleteddate IS NULL
GROUP BY categoryid) i
JOIN issuecategory ic ON ic.id = i.categoryid
UNION SELECT tc.elementtypeid
FROM
(SELECT categoryid
FROM task
WHERE clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'
AND deleteddate IS NULL
GROUP BY categoryid) t
JOIN taskcategory tc ON tc.id = t.categoryid) icc)
I have tried to change the ANY operator with IN, made an join instead of IN (in line 3 of the query) but it is still very slow, when the result is not cached.
I think it might be the nested loop making the problem - but I dont know if I can get rid of it - and why et only
As you can see, I use a couple of indexes _idx an of course primary keys on every table.
the elementtype table has ~6000 rows
the issue sub-query with these conditions (not group by) returns ~33000 rows
the task sub-query with these conditions (not group by) returns ~148000 rows
Is there any way to optimize the query?
EDIT:
As requested by #a_horse_with_no_name I add a query plan using the command he/she surgested. The best way to post it in here, is is using an image, I think:
QUERY PLAN
Unique (cost=473976.82..474453.63 rows=4453 width=16) (actual time=69897.728..69897.737 rows=1 loops=1)
Buffers: shared hit=61346 read=19651
-> Merge Join (cost=473976.82..474442.49 rows=4453 width=16) (actual time=69897.724..69897.731 rows=1 loops=1)
Merge Cond: (et.id = ic.elementtypeid)
Buffers: shared hit=61346 read=19651
-> Index Only Scan using elementtype_pkey on elementtype et (cost=0.28..384.47 rows=5879 width=16) (actual time=0.021..32.618 rows=1784 loops=1)
Heap Fetches: 1784
Buffers: shared hit=1699 read=54
-> Sort (cost=473976.54..473987.67 rows=4453 width=16) (actual time=69863.461..69863.464 rows=1 loops=1)
Sort Key: ic.elementtypeid
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=59647 read=19597
-> HashAggregate (cost=473617.61..473662.14 rows=4453 width=16) (actual time=69863.432..69863.436 rows=1 loops=1)
Group Key: ic.elementtypeid
Buffers: shared hit=59647 read=19597
-> Append (cost=107927.43..473606.48 rows=4453 width=16) (actual time=114.259..69863.317 rows=55 loops=1)
Buffers: shared hit=59647 read=19597
-> Hash Join (cost=107927.43..109170.43 rows=3625 width=16) (actual time=114.257..208.716 rows=46 loops=1)
Hash Cond: (ic.id = issue.categoryid)
Buffers: shared hit=15431
-> Seq Scan on issuecategory ic (cost=0.00..1100.36 rows=54336 width=32) (actual time=0.011..47.327 rows=54336 loops=1)
Buffers: shared hit=557
-> Hash (cost=107882.12..107882.12 rows=3625 width=16) (actual time=113.850..113.850 rows=46 loops=1)
Buckets: 4096 Batches: 1 Memory Usage: 35kB
Buffers: shared hit=14874
-> HashAggregate (cost=107809.62..107845.87 rows=3625 width=16) (actual time=113.738..113.795 rows=46 loops=1)
Group Key: issue.categoryid
Buffers: shared hit=14874
-> Bitmap Heap Scan on issue (cost=1801.41..107730.88 rows=31493 width=16) (actual time=7.279..81.266 rows=33670 loops=1)
Recheck Cond: (clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'::uuid)
Filter: (deleteddate IS NULL)
Rows Removed by Filter: 1362
Heap Blocks: exact=14636
Buffers: shared hit=14874
-> Bitmap Index Scan on issue_clientid_ix (cost=0.00..1793.54 rows=32681 width=0) (actual time=5.165..5.166 rows=35064 loops=1)
Index Cond: (clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'::uuid)
Buffers: shared hit=238
-> Nested Loop (cost=360635.19..364391.52 rows=828 width=16) (actual time=69603.779..69654.505 rows=9 loops=1)
Buffers: shared hit=44216 read=19597
-> HashAggregate (cost=360634.78..360643.06 rows=828 width=16) (actual time=69592.635..69592.657 rows=9 loops=1)
Group Key: task.categoryid
Buffers: shared hit=44198 read=19579
-> Bitmap Heap Scan on task (cost=3438.67..360280.46 rows=141728 width=16) (actual time=33.283..69416.182 rows=147931 loops=1)
Recheck Cond: (clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'::uuid)
Filter: (deleteddate IS NULL)
Rows Removed by Filter: 2329
Heap Blocks: exact=63193
Buffers: shared hit=44198 read=19579
-> Bitmap Index Scan on task_clientid_ix (cost=0.00..3403.24 rows=148091 width=0) (actual time=20.865..20.866 rows=150975 loops=1)
Index Cond: (clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'::uuid)
Buffers: shared hit=584
-> Index Scan using taskcategory_pkey on taskcategory tc (cost=0.42..4.52 rows=1 width=32) (actual time=6.865..6.865 rows=1 loops=9)
Index Cond: (id = task.categoryid)
Buffers: shared hit=18 read=18
Planning time: 1.173 ms
Execution time: 69899.380 ms
EDIT2:
issuecategory has index on id, clintid, elementypeid
issue has index on clientid, deleteddate and categoryid
taskcategory has index on id, clientid, elementtypeid,
task has index on clientid, id, deleteddate, categoryid
The problem is the bitmap heap scans. They seem to be jumping to a lot of different parts of the disk to fetch the data they need.
The best solution is probably to create indexes on (clientid, categoryid, deleteddate) on each table, or maybe (clientid, categoryid) where deleteddate is null. This will allow those bitmap heap scans to be replaced with index-only scans (assuming your tables are vacuumed well enough).
Other approaches would be to CLUSTER the tables so that rows with the same clientid are physically grouped together, or increase effective_io_concurrency so more IO can be done at the same time (assuming your storage system has multiple spindles in RAID/JBOD, or whatever the SSD equivalent to that is).
I have a query joining two tables partitioned on timestamp column. Both tables are filtered on current date partition. But query is unusually slow with APPEND Cost of the driving table very high.
Query and Plan : https://explain.dalibo.com/plan/wVA
Nested Loop (cost=0.56..174042.82 rows=16 width=494) (actual time=0.482..20.133 rows=1713 loops=1)
Output: tran.transaction_status, mgwor.apx_transaction_id, org.organisation_name, mgwor.order_status, mgwor.request_date, mgwor.response_date, (date_part('epoch'::text, mgwor.response_date) - date_part('epoch'::text, mgwor.request_date))
Buffers: shared hit=5787 dirtied=3
-> Nested Loop (cost=0.42..166837.32 rows=16 width=337) (actual time=0.459..7.803 rows=1713 loops=1)
Output: mgwor.apx_transaction_id, mgwor.order_status, mgwor.request_date, mgwor.response_date, org.organisation_name
Join Filter: ((account.account_id)::text = (mgwor.account_id)::text)
Rows Removed by Join Filter: 3007
Buffers: shared hit=589
-> Nested Loop (cost=0.27..40.66 rows=4 width=54) (actual time=0.203..0.483 rows=2 loops=1)
Output: account.account_id, org.organisation_name
Join Filter: ((account.organisation_id)::text = (org.organisation_id)::text)
Rows Removed by Join Filter: 289
Buffers: shared hit=27
-> Index Scan using account_pkey on mdm.account (cost=0.27..32.55 rows=285 width=65) (actual time=0.013..0.122 rows=291 loops=1)
Output: account.account_id, account.account_created_at, account.account_name, account.account_status, account.account_valid_until, account.currency_id, account.organisation_id, account.organisation_psp_id, account."account_threeDS_required", account.account_use_webhook, account.account_webhook_url, account.account_webhook_max_attempt, account.reporting_account_id, account.card_type, account.country_id, account.product_id
Buffers: shared hit=24
-> Materialize (cost=0.00..3.84 rows=1 width=55) (actual time=0.000..0.000 rows=1 loops=291)
Output: org.organisation_name, org.organisation_id
Buffers: shared hit=3
-> Seq Scan on mdm.organisation_smd org (cost=0.00..3.84 rows=1 width=55) (actual time=0.017..0.023 rows=1 loops=1)
Output: org.organisation_name, org.organisation_id
Filter: ((org.organisation_name)::text = 'ABC'::text)
Rows Removed by Filter: 67
Buffers: shared hit=3
-> Materialize (cost=0.15..166576.15 rows=3835 width=473) (actual time=0.127..2.826 rows=2360 loops=2)
Output: mgwor.apx_transaction_id, mgwor.order_status, mgwor.request_date, mgwor.response_date, mgwor.account_id
Buffers: shared hit=562
-> Append (cost=0.15..166556.97 rows=3835 width=473) (actual time=0.252..3.661 rows=2360 loops=1)
Buffers: shared hit=562
Subplans Removed: 1460
-> Bitmap Heap Scan on public.mgworderrequest_part_20200612 mgwor (cost=50.98..672.23 rows=2375 width=91) (actual time=0.251..2.726 rows=2360 loops=1)
Output: mgwor.apx_transaction_id, mgwor.order_status, mgwor.request_date, mgwor.response_date, mgwor.account_id
Recheck Cond: ((mgwor.request_type)::text = ANY ('{CARD,CARD_PAYMENT}'::text[]))
Filter: ((mgwor.request_date >= date(now())) AND (mgwor.request_date < (date(now()) + 1)))
Heap Blocks: exact=549
Buffers: shared hit=562
-> Bitmap Index Scan on mgworderrequest_part_20200612_request_type_idx (cost=0.00..50.38 rows=2375 width=0) (actual time=0.191..0.192 rows=2361 loops=1)
Index Cond: ((mgwor.request_type)::text = ANY ('{CARD,CARD_PAYMENT}'::text[]))
Buffers: shared hit=13
-> Append (cost=0.14..435.73 rows=1461 width=316) (actual time=0.005..0.006 rows=1 loops=1713)
Buffers: shared hit=5198 dirtied=3
Subplans Removed: 1460
-> Index Scan using transaction_part_20200612_pkey on public.transaction_part_20200612 tran (cost=0.29..0.87 rows=1 width=42) (actual time=0.004..0.005 rows=1 loops=1713)
Output: tran.transaction_status, tran.transaction_id
Index Cond: (((tran.transaction_id)::text = (mgwor.apx_transaction_id)::text) AND (tran.transaction_created_at >= date(now())) AND (tran.transaction_created_at < (date(now()) + 1)))
Filter: (tran.transaction_status IS NOT NULL)
Buffers: shared hit=5198 dirtied=3
Planning Time: 19535.308 ms
Execution Time: 21.006 ms
Partition pruning is working on both the tables.
Am I missing something obvious here?
Thanks,
VA
I don't know why the cost estimate for the append is so large, but presumably you are really worried about how long this takes, not how large the estimate is. As noted, the actual time is going to planning, not to execution.
A likely explanation is that it was waiting on a lock. Time spent waiting on a table lock for a partition table (but not for the parent table) gets attributed to planning time.
I have a view req_res which joins 2 tables - Request and Response inner joined on requestId. Request has primary key - ID column.
When I query (Query 1):
select max(ID) from req_res where ID > 1000000 and ID < 2000000;
Explain plan: hash join: Index scan of Request.ID and seqential scan of Response.request_id
query duration: 30s
When I lower the boundaries to 900k (Query 2):
select max(ID) from req_res where ID > 1000000 and ID < 1900000;
Plan: nested loop: Index scan of Request.ID and Index only scan of Response.request_id
query duration: 3s
When I play with first query and disable hash join - set enable_hashjoin=off; I get Merge join plan. When I disable also the merge join plan with set enable_mergejoin=off; I get nested loop, which completes in 3 seconds (instead of 30 using hash join).
Size of the Request table is ~70 Mil records. Most of the requests have response counterpart, but some of them don't.
Version: PostgreSQL 10.10
req_res DDL:
CREATE OR REPLACE VIEW public.req_res
AS SELECT req.id,
res.req_id,
res.body::character varying(500),
res.time,
res.duration,
res.balance,
res.header::character varying(100),
res.created_at
FROM response res
JOIN request req ON req.req_id = res.req_id;
Query 1 Plan:
Aggregate (cost=2834115.70..2834115.71 rows=1 width=8) (actual time=154709.729..154709.730 rows=1 loops=1)
Buffers: shared hit=467727 read=685320 dirtied=214, temp read=240773 written=239751
-> Hash Join (cost=2493060.64..2831172.33 rows=1177346 width=8) (actual time=143800.101..154147.080 rows=1198706 loops=1)
Hash Cond: (req.req_id = res.req_id)
Buffers: shared hit=467727 read=685320 dirtied=214, temp read=240773 written=239751
-> Append (cost=0.44..55619.59 rows=1177346 width=16) (actual time=0.957..2354.648 rows=1200001 loops=1)
Buffers: shared hit=438960 read=32014
-> Index Scan using "5_5_req_pkey" on _hyper_2_5_chunk rs (cost=0.44..19000.10 rows=399803 width=16) (actual time=0.956..546.231 rows=399999 loops=1)
Index Cond: ((id >= 49600001) AND (id <= 50800001))
Buffers: shared hit=178872 read=10742
-> Index Scan using "7_7_req_pkey" on _hyper_2_7_chunk rs_1 (cost=0.44..36619.50 rows=777543 width=16) (actual time=0.037..767.744 rows=800002 loops=1)
Index Cond: ((id >= 49600001) AND (id <= 50800001))
Buffers: shared hit=260088 read=21272
-> Hash (cost=1367864.98..1367864.98 rows=68583298 width=8) (actual time=143681.850..143681.850 rows=68568554 loops=1)
Buckets: 262144 Batches: 512 Memory Usage: 7278kB
Buffers: shared hit=28764 read=653306 dirtied=214, temp written=233652
-> Append (cost=0.00..1367864.98 rows=68583298 width=8) (actual time=0.311..99590.021 rows=68568554 loops=1)
Buffers: shared hit=28764 read=653306 dirtied=214
-> Seq Scan on _hyper_3_2_chunk wt (cost=0.00..493704.44 rows=24941244 width=8) (actual time=0.309..14484.420 rows=24950147 loops=1)
Buffers: shared hit=661 read=243631
-> Seq Scan on _hyper_3_6_chunk wt_1 (cost=0.00..503935.04 rows=24978804 width=8) (actual time=0.334..14487.931 rows=24963020 loops=1)
Buffers: shared hit=168 read=253979
-> Seq Scan on _hyper_3_8_chunk wt_2 (cost=0.00..370225.50 rows=18663250 width=8) (actual time=0.327..10837.291 rows=18655387 loops=1)
Buffers: shared hit=27935 read=155696 dirtied=214
Planning time: 3.986 ms
Execution time: 154709.859 ms
Query 2 Plan:
Finalize Aggregate (cost=2634042.50..2634042.51 rows=1 width=8) (actual time=5525.626..5525.627 rows=1 loops=1)
Buffers: shared hit=8764620 read=12779
-> Gather (cost=2634042.29..2634042.50 rows=2 width=8) (actual time=5525.609..5525.705 rows=3 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=8764620 read=12779
-> Partial Aggregate (cost=2633042.29..2633042.30 rows=1 width=8) (actual time=5515.507..5515.508 rows=1 loops=3)
Buffers: shared hit=8764620 read=12779
-> Nested Loop (cost=0.88..2632023.83 rows=407382 width=8) (actual time=5.383..5261.979 rows=332978 loops=3)
Buffers: shared hit=8764620 read=12779
-> Append (cost=0.44..40514.98 rows=407383 width=16) (actual time=0.035..924.498 rows=333334 loops=3)
Buffers: shared hit=446706
-> Parallel Index Scan using "5_5_req_pkey" on _hyper_2_5_chunk rs (cost=0.44..16667.91 rows=166585 width=16) (actual time=0.033..169.854 rows=133333 loops=3)
Index Cond: ((id >= 49600001) AND (id <= 50600001))
Buffers: shared hit=190175
-> Parallel Index Scan using "7_7_req_pkey" on _hyper_2_7_chunk rs_1 (cost=0.44..23847.07 rows=240798 width=16) (actual time=0.039..336.091 rows=200001 loops=3)
Index Cond: ((id >= 49600001) AND (id <= 50600001))
Buffers: shared hit=256531
-> Append (cost=0.44..6.33 rows=3 width=8) (actual time=0.011..0.011 rows=1 loops=1000001)
Buffers: shared hit=8317914 read=12779
-> Index Only Scan using "2_2_response_pkey" on _hyper_3_2_chunk wt (cost=0.44..2.11 rows=1 width=8) (actual time=0.003..0.003 rows=0 loops=1000001)
Index Cond: (req_id = req.req_id)
Heap Fetches: 0
Buffers: shared hit=3000005
-> Index Only Scan using "6_6_response_pkey" on _hyper_3_6_chunk wt_1 (cost=0.44..2.11 rows=1 width=8) (actual time=0.003..0.003 rows=1 loops=1000001)
Index Cond: (req_id = req.req_id)
Heap Fetches: 192906
Buffers: shared hit=3551440 read=7082
-> Index Only Scan using "8_8_response_pkey" on _hyper_3_8_chunk wt_2 (cost=0.44..2.10 rows=1 width=8) (actual time=0.003..0.003 rows=1 loops=443006)
Index Cond: (req_id = req.req_id)
Heap Fetches: 162913
Buffers: shared hit=1766469 read=5697
Planning time: 0.839 ms
Execution time: 5525.814 ms
I'm using postgres 10, and have the following query
select
count(task.id) over() as _total_ ,
json_agg(u.*) as users,
task.*
from task
left outer join taskuserlink_history tu on (task.id = tu.taskid)
left outer join "user" u on (tu.userId = u.id)
group by task.id offset 10 limit 10;
this query takes approx 800ms to execute
if I remove the count(task.id) over() as _total_ , line, then it executes in 250ms
I have to confess being a complete sql noob, so the query itself may be completely borked
I was wondering if anyone could point to the flaws in the query, and make suggestions on how to speed it up.
The number of tasks is approx 15k, with an average of 5 users per task, linked through taskuserlink
I have looked at the pgadmin "explain" diagram
but to be honest can't really figure it out yet ;)
the table definitions are
task , with id (int) as primary column
taskuserlink_history, with taskId (int) and userId (int) (both as foreign key constraints, indexed)
user, with id (int) as primary column
the query plan is as follows
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=4.74..12.49 rows=10 width=44) (actual time=1178.016..1178.043 rows=10 loops=1)
Buffers: shared hit=3731, temp read=6655 written=6914
-> WindowAgg (cost=4.74..10248.90 rows=13231 width=44) (actual time=1178.014..1178.040 rows=10 loops=1)
Buffers: shared hit=3731, temp read=6655 written=6914
-> GroupAggregate (cost=4.74..10083.51 rows=13231 width=36) (actual time=0.417..1049.294 rows=13255 loops=1)
Group Key: task.id
Buffers: shared hit=3731
-> Nested Loop Left Join (cost=4.74..9586.77 rows=66271 width=36) (actual time=0.103..309.372 rows=66162 loops=1)
Join Filter: (taskuserlink_history.userid = user_archive.id)
Rows Removed by Join Filter: 1182904
Buffers: shared hit=3731
-> Merge Left Join (cost=0.58..5563.22 rows=66271 width=8) (actual time=0.044..73.598 rows=66162 loops=1)
Merge Cond: (task.id = taskuserlink_history.taskid)
Buffers: shared hit=3629
-> Index Only Scan using task_pkey on task (cost=0.29..1938.30 rows=13231 width=4) (actual time=0.026..7.683 rows=13255 loops=1)
Heap Fetches: 13255
Buffers: shared hit=1810
-> Index Scan using taskuserlink_history_task_fk_idx on taskuserlink_history (cost=0.29..2764.46 rows=66271 width=8) (actual time=0.015..40.109 rows=66162 loops=1)
Filter: (timeend IS NULL)
Rows Removed by Filter: 13368
Buffers: shared hit=1819
-> Materialize (cost=4.17..50.46 rows=4 width=36) (actual time=0.000..0.001 rows=19 loops=66162)
Buffers: shared hit=102
-> Bitmap Heap Scan on user_archive (cost=4.17..50.44 rows=4 width=36) (actual time=0.050..0.305 rows=45 loops=1)
Recheck Cond: (archived_at IS NULL)
Heap Blocks: exact=11
Buffers: shared hit=102
-> Bitmap Index Scan on user_unique_username (cost=0.00..4.16 rows=4 width=0) (actual time=0.014..0.014 rows=46 loops=1)
Buffers: shared hit=1
SubPlan 1
-> Aggregate (cost=8.30..8.31 rows=1 width=8) (actual time=0.003..0.003 rows=1 loops=45)
Buffers: shared hit=90
-> Index Scan using task_assignedto_idx on task task_1 (cost=0.29..8.30 rows=1 width=4) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (assignedtoid = user_archive.id)
Buffers: shared hit=90
Planning time: 0.989 ms
Execution time: 1191.451 ms
(37 rows)
without the window function it is
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=4.74..12.36 rows=10 width=36) (actual time=0.510..1.763 rows=10 loops=1)
Buffers: shared hit=91
-> GroupAggregate (cost=4.74..10083.51 rows=13231 width=36) (actual time=0.509..1.759 rows=10 loops=1)
Group Key: task.id
Buffers: shared hit=91
-> Nested Loop Left Join (cost=4.74..9586.77 rows=66271 width=36) (actual time=0.073..0.744 rows=50 loops=1)
Join Filter: (taskuserlink_history.userid = user_archive.id)
Rows Removed by Join Filter: 361
Buffers: shared hit=91
-> Merge Left Join (cost=0.58..5563.22 rows=66271 width=8) (actual time=0.029..0.161 rows=50 loops=1)
Merge Cond: (task.id = taskuserlink_history.taskid)
Buffers: shared hit=7
-> Index Only Scan using task_pkey on task (cost=0.29..1938.30 rows=13231 width=4) (actual time=0.016..0.031 rows=11 loops=1)
Heap Fetches: 11
Buffers: shared hit=4
-> Index Scan using taskuserlink_history_task_fk_idx on taskuserlink_history (cost=0.29..2764.46 rows=66271 width=8) (actual time=0.009..0.081 rows=50 loops=1)
Filter: (timeend IS NULL)
Rows Removed by Filter: 11
Buffers: shared hit=3
-> Materialize (cost=4.17..50.46 rows=4 width=36) (actual time=0.001..0.009 rows=8 loops=50)
Buffers: shared hit=84
-> Bitmap Heap Scan on user_archive (cost=4.17..50.44 rows=4 width=36) (actual time=0.040..0.382 rows=38 loops=1)
Recheck Cond: (archived_at IS NULL)
Heap Blocks: exact=7
Buffers: shared hit=84
-> Bitmap Index Scan on user_unique_username (cost=0.00..4.16 rows=4 width=0) (actual time=0.012..0.012 rows=46 loops=1)
Buffers: shared hit=1
SubPlan 1
-> Aggregate (cost=8.30..8.31 rows=1 width=8) (actual time=0.005..0.005 rows=1 loops=38)
Buffers: shared hit=76
-> Index Scan using task_assignedto_idx on task task_1 (cost=0.29..8.30 rows=1 width=4) (actual time=0.003..0.003 rows=0 loops=38)
Index Cond: (assignedtoid = user_archive.id)
Buffers: shared hit=76
Planning time: 0.895 ms
Execution time: 1.890 ms
(35 rows)|
I believe the LIMIT clause is making the difference. LIMIT is limiting the number of rows returned, not neccessarily the work involved:
Your second query can be aborted early after 20 rows have been constructed (10 for OFFSET and 10 for LIMIT).
However, your first query needs to go through the whole set to calculate the count(task.id).
Not what you were asking, but I say it anyway:
"user" is not a table, but a view. That is were both queries actually get slower than they should be (The "Materialize" in the plan).
Using OFFSET for paging calls for trouble because it will get slow when the OFFSET increases
Using OFFSET and LIMIT without an ORDER BY is most likely not what you want. The result sets might not be identical on consecutive calls.