Efficiently querying view with left join - postgresql

PostgreSQL 14.6 on x86_64-pc-linux-gnu, compiled by gcc, a 12d889e8a6 p ce8d8b4729, 64-bit
I have an organizations table and a (much smaller) partner_members table that associate some organizations with a partner_member_id.
There is also a convenience view to list organizations with their (potential) partner IDs, defined like this:
select
o.id,
o.name,
o.email,
o.created,
p.member_id AS partner_member_id
from organizations o
left join partner_members p on o.id= p.organization_id
However, this leads to an admin query that queries this view ending up like this:
select count(*) OVER (),"id","name","email","created"
from (
select
o.id,
o.name,
o.email,
o.created,
p.member_id AS partner_member_id
from organizations o
left join partner_members p on o.id= p.organization_id
) _
where ("name" ilike '%example#example.com%')
or ("email" ilike '%example#example.com%')
or ("partner_member_id" ilike '%example#example.com%')
or ("id" ilike '%example#example.com%')
order by "created" desc
offset 0 limit 50;
… which is super slow, since the partner_member_id constraint isn't “pushed” into the sub query, which means that the filtering happens way too late.
Is there a way to make a query such as this efficient, or is this convenience view a no-go here?
Here is the plan:
Limit (cost=12842.32..12848.77 rows=50 width=74) (actual time=2344.828..2385.234 rows=0 loops=1)
Buffers: shared hit=5246, temp read=3088 written=3120
-> WindowAgg (cost=12842.32..12853.80 rows=89 width=74) (actual time=2344.826..2385.232 rows=0 loops=1)
Buffers: shared hit=5246, temp read=3088 written=3120
-> Gather Merge (cost=12842.32..12852.69 rows=89 width=66) (actual time=2344.822..2385.226 rows=0 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=5246, temp read=3088 written=3120
-> Sort (cost=11842.30..11842.39 rows=37 width=66) (actual time=2322.988..2323.050 rows=0 loops=3)
Sort Key: o.created DESC
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=5246, temp read=3088 written=3120
Worker 0: Sort Method: quicksort Memory: 25kB
Worker 1: Sort Method: quicksort Memory: 25kB
-> Parallel Hash Left Join (cost=3368.61..11841.33 rows=37 width=66) (actual time=2322.857..2322.917 rows=0 loops=3)
Hash Cond: ((o.id)::text = p.organization_id)
Filter: (((o.name)::text ~~* '%example#example.com%'::text) OR ((o.email)::text ~~* '%example#example.com%'::text) OR (p.member_id ~~* '%example#example.com%'::text) OR ((o.id)::text ~~* '%example#example.com%'::text))
Rows Removed by Filter: 73800
Buffers: shared hit=5172, temp read=3088 written=3120
-> Parallel Seq Scan on organizations o (cost=0.00..4813.65 rows=92365 width=66) (actual time=0.020..200.111 rows=73800 loops=3)
Buffers: shared hit=3890
-> Parallel Hash (cost=1926.05..1926.05 rows=71005 width=34) (actual time=108.608..108.610 rows=40150 loops=3)
Buckets: 32768 Batches: 4 Memory Usage: 2432kB
Buffers: shared hit=1216, temp written=620
-> Parallel Seq Scan on partner_members p (cost=0.00..1926.05 rows=71005 width=34) (actual time=0.028..43.757 rows=40150 loops=3)
Buffers: shared hit=1216
Planning:
Buffers: shared hit=24
Planning Time: 1.837 ms
Execution Time: 2385.319 ms

Related

Join on UUID column not using index for multiple values

I have a PostgreSQL database that I cloned.
Database 1 has varchar(36) as primary keys
Database 2 (the clone) has UUID as primary keys.
Both contain the same data. What I don't understand is why queries on Database 1 will use the index but Database 2 will not. Here's the query:
EXPLAIN (ANALYZE, BUFFERS)
select * from table1
INNER JOIN table2 on table1.id = table2.table1_id
where table1.id in (
'541edffc-7179-42db-8c99-727be8c9ffec',
'eaac06d3-e44e-4e4a-8e11-1cdc6e562996'
);
Database 1
Nested Loop (cost=16.13..7234.96 rows=14 width=803) (actual time=0.072..0.112 rows=8 loops=1)
Buffers: shared hit=23
-> Index Scan using table1_pk on table1 (cost=0.56..17.15 rows=2 width=540) (actual time=0.042..0.054 rows=2 loops=1)
" Index Cond: ((id)::text = ANY ('{541edffc-7179-42db-8c99-727be8c9ffec,eaac06d3-e44e-4e4a-8e11-1cdc6e562996}'::text[]))"
Buffers: shared hit=12
-> Bitmap Heap Scan on table2 (cost=15.57..3599.86 rows=904 width=263) (actual time=0.022..0.023 rows=4 loops=2)
Recheck Cond: ((table1_id)::text = (table1.id)::text)
Heap Blocks: exact=3
Buffers: shared hit=11
-> Bitmap Index Scan on table2_table1_id_fk (cost=0.00..15.34 rows=904 width=0) (actual time=0.019..0.019 rows=4 loops=2)
Index Cond: ((table1_id)::text = (table1.id)::text)
Buffers: shared hit=8
Planning:
Buffers: shared hit=416
Planning Time: 1.869 ms
Execution Time: 0.330 ms
Database 2
Gather (cost=1000.57..1801008.91 rows=14 width=740) (actual time=11.580..42863.893 rows=8 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=863 read=631539 dirtied=631979 written=2523
-> Nested Loop (cost=0.56..1800007.51 rows=6 width=740) (actual time=28573.119..42856.696 rows=3 loops=3)
Buffers: shared hit=863 read=631539 dirtied=631979 written=2523
-> Parallel Seq Scan on table1 (cost=0.00..678896.46 rows=1 width=519) (actual time=28573.112..42855.524 rows=1 loops=3)
" Filter: (id = ANY ('{541edffc-7179-42db-8c99-727be8c9ffec,eaac06d3-e44e-4e4a-8e11-1cdc6e562996}'::uuid[]))"
Rows Removed by Filter: 2976413
Buffers: shared hit=854 read=631536 dirtied=631979 written=2523
-> Index Scan using table2_table1_id_fk on table2 (cost=0.56..1117908.70 rows=320236 width=221) (actual time=1.736..1.745 rows=4 loops=2)
Index Cond: (table1_id = table1.id)
Buffers: shared hit=9 read=3
Planning:
Buffers: shared hit=376 read=15
Planning Time: 43.594 ms
Execution Time: 42864.044 ms
Some notes:
The query is orders of magnitude faster in Database 1
Having only one ID in the WHERE clause activates the index in both databases
Casting to ::uuid has no impact
I understand that these results are because the query planner calculates that the cost of the index in the UUID (Database 2) case is too high. But I'm trying to understand why it thinks that and if there's something I can do.

Postgresql Limit 1 causing large number of loops

We have a transactions table, approx. 10m rows and growing. Each customer we have specifies many rules which group certain transactions together based on locations, associated products, sale customer, etc. Off of these rules we produce reports each night which allows them to see the price the customer is paying for products vs their purchase prices from different price lists, these lists are changing daily and each date on the transaction we have to either find their set yearly price or the price that was effective at the date of the transaction.
These price lists can change historically and do all the time as do new historic transactions which are added so within each financial year we have to continue to regenerate these reports.
We are having a problem with the two types of price list/price joins we have to do. The first is on the set yearly price list.
I have removed the queries which bring the transactions in and put into a table called transaction_data_6787.
EXPLAIN analyze
SELECT *
FROM transaction_data_6787 t
inner JOIN LATERAL
(
SELECT p."Price"
FROM "Prices" p
INNER JOIN "PriceLists" pl on p."PriceListId" = pl."Id"
WHERE (pl."CustomerId" = 20)
AND (pl."Year" = 2020)
AND (pl."PriceListTypeId" = 2)
AND p."ProductId" = t.product_id
limit 1
) AS prices ON true
Nested Loop (cost=0.70..133877.20 rows=5394 width=165) (actual time=0.521..193.638 rows=5394 loops=1) -> Seq Scan on transaction_data_6787 t (cost=0.00..159.94 rows=5394 width=145) (actual time=0.005..0.593 rows=5394 loops=1) -> Limit (cost=0.70..24.77 rows=1 width=20) (actual time=0.035..0.035 rows=1 loops=5394)
-> Nested Loop (cost=0.70..24.77 rows=1 width=20) (actual time=0.035..0.035 rows=1 loops=5394)
-> Index Scan using ix_prices_covering on "Prices" p (cost=0.42..8.44 rows=1 width=16) (actual time=0.006..0.015 rows=23 loops=5394)
Index Cond: (("ProductId" = t.product_id))
-> Index Scan using ix_pricelists_covering on "PriceLists" pl (cost=0.28..8.30 rows=1 width=12) (actual time=0.001..0.001 rows=0 loops=122443)
Index Cond: (("Id" = p."PriceListId") AND ("CustomerId" = 20) AND ("PriceListTypeId" = 2))
Filter: ("Year" = 2020)
Rows Removed by Filter: 0 Planning Time: 0.307 ms Execution Time: 193.982 ms
If I remove the LIMIT 1, the execution time drops to 3ms and the 122443 loops on ix_pricelists_covering don't happen. The reason we are doing a lateral join is the price query is dynamically built and sometimes when not joining on the annual price list we join on the effective price lists. This looks like the below:
EXPLAIN analyze
SELECT *
FROM transaction_data_6787 t
inner JOIN LATERAL
(
SELECT p."Price"
FROM "Prices" p
INNER JOIN "PriceLists" pl on p."PriceListId" = pl."Id"
WHERE (pl."CustomerId" = 20)
AND (pl."PriceListTypeId" = 1)
AND p."ProductId" = t.product_id
and pl."ValidFromDate" <= t.transaction_date
ORDER BY pl."ValidFromDate" desc
limit 1
) AS prices ON true
This is killing our performance, some queries are taking 20 seconds plus when we don't order by date desc/limit 1 it completes in ms but we could get duplicate prices back.
We are happy to rewrite if a better way of joining the most recent record. We have thousands of price lists and 100k or prices and there could be 100s if not 1000s of effective prices for each transaction and we need to ensure we get the one which was most recently effective for a product at the date of the transaction.
I have found if I denormalise the price lists/prices into a single table and add an index with ValidFromDate DESC it seems to eliminate the loops but I am hesitant to denormalise and have to maintain that data, these reports can be run adhoc as well as batch jobs and we would have to maintain that data in real time.
Updated Explain/Analyze:
I've added below the query which joins on prices which need to get the most recently effective for the transaction date. I see now that when <= date clause and limit 1 is removed it's actually spinning up multiple workers which is why it seems faster.
I am still seeing the slower query doing a large number of loops, 200k+ (when limit 1/<= date is included).
Maybe the better question is what can we do instead of a lateral join which will allow us to join the effective prices for transactions in the most efficient/performant way possible. I am hoping to avoid denormalising and maintainint that data but if it is the only way we'll do it. If there is a way to rewrite this and not denormalise then I'd really appreciate any insight.
Nested Loop (cost=14.21..76965.60 rows=5394 width=10) (actual time=408.948..408.950 rows=0 loops=1)
Output: t.transaction_id, pr."Price"
Buffers: shared hit=688022
-> Seq Scan on public.transaction_data_6787 t (cost=0.00..159.94 rows=5394 width=29) (actual time=0.018..0.682 rows=5394 loops=1)
Output: t.transaction_id
Buffers: shared hit=106
-> Limit (cost=14.21..14.22 rows=1 width=10) (actual time=0.075..0.075 rows=0 loops=5394)
Output: pr."Price", pl."ValidFromDate"
Buffers: shared hit=687916
-> Sort (cost=14.21..14.22 rows=1 width=10) (actual time=0.075..0.075 rows=0 loops=5394)
Output: pr."Price", pl."ValidFromDate"
Sort Key: pl."ValidFromDate" DESC
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=687916
-> Nested Loop (cost=0.70..14.20 rows=1 width=10) (actual time=0.074..0.074 rows=0 loops=5394)
Output: pr."Price", pl."ValidFromDate"
Inner Unique: true
Buffers: shared hit=687916
-> Index Only Scan using ix_prices_covering on public."Prices" pr (cost=0.42..4.44 rows=1 width=10) (actual time=0.007..0.019 rows=51 loops=5394)
Output: pr."ProductId", pr."ValidFromDate", pr."Id", pr."Price", pr."PriceListId"
Index Cond: (pr."ProductId" = t.product_id)
Heap Fetches: 0
Buffers: shared hit=17291
-> Index Scan using ix_pricelists_covering on public."PriceLists" pl (cost=0.28..8.30 rows=1 width=8) (actual time=0.001..0.001 rows=0 loops=273678)
Output: pl."Id", pl."Name", pl."CustomerId", pl."ValidFromDate", pl."PriceListTypeId"
Index Cond: ((pl."Id" = pr."PriceListId") AND (pl."CustomerId" = 20) AND (pl."PriceListTypeId" = 1))
Filter: (pl."ValidFromDate" <= t.transaction_date)
Rows Removed by Filter: 0
Buffers: shared hit=670625
Planning Time: 1.254 ms
Execution Time: 409.088 ms
Gather (cost=6395.67..7011.99 rows=68 width=10) (actual time=92.481..92.554 rows=0 loops=1)
Output: t.transaction_id, pr."Price"
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=1466 read=2
-> Hash Join (cost=5395.67..6005.19 rows=28 width=10) (actual time=75.126..75.129 rows=0 loops=3)
Output: t.transaction_id, pr."Price"
Inner Unique: true
Hash Cond: (pr."PriceListId" = pl."Id")
Join Filter: (pl."ValidFromDate" <= t.transaction_date)
Rows Removed by Join Filter: 41090
Buffers: shared hit=1466 read=2
Worker 0: actual time=64.707..64.709 rows=0 loops=1
Buffers: shared hit=462
Worker 1: actual time=72.545..72.547 rows=0 loops=1
Buffers: shared hit=550 read=1
-> Merge Join (cost=5374.09..5973.85 rows=3712 width=18) (actual time=26.804..61.492 rows=91226 loops=3)
Output: t.transaction_id, t.transaction_date, pr."Price", pr."PriceListId"
Merge Cond: (pr."ProductId" = t.product_id)
Buffers: shared hit=1325 read=2
Worker 0: actual time=17.677..51.590 rows=83365 loops=1
Buffers: shared hit=400
Worker 1: actual time=24.995..59.395 rows=103814 loops=1
Buffers: shared hit=488 read=1
-> Parallel Index Only Scan using ix_prices_covering on public."Prices" pr (cost=0.42..7678.38 rows=79544 width=29) (actual time=0.036..12.136 rows=42281 loops=3)
Output: pr."ProductId", pr."ValidFromDate", pr."Id", pr."Price", pr."PriceListId"
Heap Fetches: 0
Buffers: shared hit=989 read=2
Worker 0: actual time=0.037..9.660 rows=36873 loops=1
Buffers: shared hit=285
Worker 1: actual time=0.058..13.459 rows=47708 loops=1
Buffers: shared hit=373 read=1
-> Sort (cost=494.29..507.78 rows=5394 width=29) (actual time=9.037..14.700 rows=94555 loops=3)
Output: t.transaction_id, t.product_id, t.transaction_date
Sort Key: t.product_id
Sort Method: quicksort Memory: 614kB
Worker 0: Sort Method: quicksort Memory: 614kB
Worker 1: Sort Method: quicksort Memory: 614kB
Buffers: shared hit=336
Worker 0: actual time=6.608..12.034 rows=86577 loops=1
Buffers: shared hit=115
Worker 1: actual time=8.973..14.598 rows=107126 loops=1
Buffers: shared hit=115
-> Seq Scan on public.transaction_data_6787 t (cost=0.00..159.94 rows=5394 width=29) (actual time=0.020..2.948 rows=5394 loops=3)
Output: t.transaction_id, t.product_id, t.transaction_date
Buffers: shared hit=318
Worker 0: actual time=0.017..2.078 rows=5394 loops=1
Buffers: shared hit=106
Worker 1: actual time=0.027..2.976 rows=5394 loops=1
Buffers: shared hit=106
-> Hash (cost=21.21..21.21 rows=30 width=8) (actual time=0.145..0.145 rows=35 loops=3)
Output: pl."Id", pl."ValidFromDate"
Buckets: 1024 Batches: 1 Memory Usage: 10kB
Buffers: shared hit=53
Worker 0: actual time=0.137..0.138 rows=35 loops=1
Buffers: shared hit=18
Worker 1: actual time=0.149..0.150 rows=35 loops=1
Buffers: shared hit=18
-> Bitmap Heap Scan on public."PriceLists" pl (cost=4.59..21.21 rows=30 width=8) (actual time=0.067..0.114 rows=35 loops=3)
Output: pl."Id", pl."ValidFromDate"
Recheck Cond: (pl."CustomerId" = 20)
Filter: (pl."PriceListTypeId" = 1)
Rows Removed by Filter: 6
Heap Blocks: exact=15
Buffers: shared hit=53
Worker 0: actual time=0.068..0.108 rows=35 loops=1
Buffers: shared hit=18
Worker 1: actual time=0.066..0.117 rows=35 loops=1
Buffers: shared hit=18
-> Bitmap Index Scan on "IX_PriceLists_CustomerId" (cost=0.00..4.58 rows=41 width=0) (actual time=0.049..0.049 rows=41 loops=3)
Index Cond: (pl."CustomerId" = 20)
Buffers: shared hit=8
Worker 0: actual time=0.053..0.054 rows=41 loops=1
Buffers: shared hit=3
Worker 1: actual time=0.048..0.048 rows=41 loops=1
Buffers: shared hit=3
Planning Time: 2.236 ms
Execution Time: 92.814 ms

PostgreSQL optimizing query - maybee nested loop is the problem

I have this very slow query:
SELECT DISTINCT et.id
FROM elementtype et
where et.id = any
(SELECT elementtypeid
FROM
(SELECT ic.elementtypeid
FROM
(SELECT categoryid
FROM issue
WHERE clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'
AND deleteddate IS NULL
GROUP BY categoryid) i
JOIN issuecategory ic ON ic.id = i.categoryid
UNION SELECT tc.elementtypeid
FROM
(SELECT categoryid
FROM task
WHERE clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'
AND deleteddate IS NULL
GROUP BY categoryid) t
JOIN taskcategory tc ON tc.id = t.categoryid) icc)
I have tried to change the ANY operator with IN, made an join instead of IN (in line 3 of the query) but it is still very slow, when the result is not cached.
I think it might be the nested loop making the problem - but I dont know if I can get rid of it - and why et only
As you can see, I use a couple of indexes _idx an of course primary keys on every table.
the elementtype table has ~6000 rows
the issue sub-query with these conditions (not group by) returns ~33000 rows
the task sub-query with these conditions (not group by) returns ~148000 rows
Is there any way to optimize the query?
EDIT:
As requested by #a_horse_with_no_name I add a query plan using the command he/she surgested. The best way to post it in here, is is using an image, I think:
QUERY PLAN
Unique (cost=473976.82..474453.63 rows=4453 width=16) (actual time=69897.728..69897.737 rows=1 loops=1)
Buffers: shared hit=61346 read=19651
-> Merge Join (cost=473976.82..474442.49 rows=4453 width=16) (actual time=69897.724..69897.731 rows=1 loops=1)
Merge Cond: (et.id = ic.elementtypeid)
Buffers: shared hit=61346 read=19651
-> Index Only Scan using elementtype_pkey on elementtype et (cost=0.28..384.47 rows=5879 width=16) (actual time=0.021..32.618 rows=1784 loops=1)
Heap Fetches: 1784
Buffers: shared hit=1699 read=54
-> Sort (cost=473976.54..473987.67 rows=4453 width=16) (actual time=69863.461..69863.464 rows=1 loops=1)
Sort Key: ic.elementtypeid
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=59647 read=19597
-> HashAggregate (cost=473617.61..473662.14 rows=4453 width=16) (actual time=69863.432..69863.436 rows=1 loops=1)
Group Key: ic.elementtypeid
Buffers: shared hit=59647 read=19597
-> Append (cost=107927.43..473606.48 rows=4453 width=16) (actual time=114.259..69863.317 rows=55 loops=1)
Buffers: shared hit=59647 read=19597
-> Hash Join (cost=107927.43..109170.43 rows=3625 width=16) (actual time=114.257..208.716 rows=46 loops=1)
Hash Cond: (ic.id = issue.categoryid)
Buffers: shared hit=15431
-> Seq Scan on issuecategory ic (cost=0.00..1100.36 rows=54336 width=32) (actual time=0.011..47.327 rows=54336 loops=1)
Buffers: shared hit=557
-> Hash (cost=107882.12..107882.12 rows=3625 width=16) (actual time=113.850..113.850 rows=46 loops=1)
Buckets: 4096 Batches: 1 Memory Usage: 35kB
Buffers: shared hit=14874
-> HashAggregate (cost=107809.62..107845.87 rows=3625 width=16) (actual time=113.738..113.795 rows=46 loops=1)
Group Key: issue.categoryid
Buffers: shared hit=14874
-> Bitmap Heap Scan on issue (cost=1801.41..107730.88 rows=31493 width=16) (actual time=7.279..81.266 rows=33670 loops=1)
Recheck Cond: (clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'::uuid)
Filter: (deleteddate IS NULL)
Rows Removed by Filter: 1362
Heap Blocks: exact=14636
Buffers: shared hit=14874
-> Bitmap Index Scan on issue_clientid_ix (cost=0.00..1793.54 rows=32681 width=0) (actual time=5.165..5.166 rows=35064 loops=1)
Index Cond: (clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'::uuid)
Buffers: shared hit=238
-> Nested Loop (cost=360635.19..364391.52 rows=828 width=16) (actual time=69603.779..69654.505 rows=9 loops=1)
Buffers: shared hit=44216 read=19597
-> HashAggregate (cost=360634.78..360643.06 rows=828 width=16) (actual time=69592.635..69592.657 rows=9 loops=1)
Group Key: task.categoryid
Buffers: shared hit=44198 read=19579
-> Bitmap Heap Scan on task (cost=3438.67..360280.46 rows=141728 width=16) (actual time=33.283..69416.182 rows=147931 loops=1)
Recheck Cond: (clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'::uuid)
Filter: (deleteddate IS NULL)
Rows Removed by Filter: 2329
Heap Blocks: exact=63193
Buffers: shared hit=44198 read=19579
-> Bitmap Index Scan on task_clientid_ix (cost=0.00..3403.24 rows=148091 width=0) (actual time=20.865..20.866 rows=150975 loops=1)
Index Cond: (clientid = '833e1f2f-ff44-4aca-bd12-0e4f67969a11'::uuid)
Buffers: shared hit=584
-> Index Scan using taskcategory_pkey on taskcategory tc (cost=0.42..4.52 rows=1 width=32) (actual time=6.865..6.865 rows=1 loops=9)
Index Cond: (id = task.categoryid)
Buffers: shared hit=18 read=18
Planning time: 1.173 ms
Execution time: 69899.380 ms
EDIT2:
issuecategory has index on id, clintid, elementypeid
issue has index on clientid, deleteddate and categoryid
taskcategory has index on id, clientid, elementtypeid,
task has index on clientid, id, deleteddate, categoryid
The problem is the bitmap heap scans. They seem to be jumping to a lot of different parts of the disk to fetch the data they need.
The best solution is probably to create indexes on (clientid, categoryid, deleteddate) on each table, or maybe (clientid, categoryid) where deleteddate is null. This will allow those bitmap heap scans to be replaced with index-only scans (assuming your tables are vacuumed well enough).
Other approaches would be to CLUSTER the tables so that rows with the same clientid are physically grouped together, or increase effective_io_concurrency so more IO can be done at the same time (assuming your storage system has multiple spindles in RAID/JBOD, or whatever the SSD equivalent to that is).

Postgres: performance when computing counts of LEFT JOIN records

I have a Postgres query that works but performs slower than expected.
SELECT
"post"."id",
COUNT(DISTINCT l.id) AS num_likes,
COUNT(DISTINCT ul.id) AS num_user_likes,
COUNT(DISTINCT c.id) AS num_comments
FROM "post"
LEFT JOIN "like" AS "l" ON "l"."post_id" = "post"."id"
LEFT JOIN "like" AS "ul" ON "ul"."post_id" = "post"."id" AND "ul"."user_id" = 1
LEFT JOIN "comment" AS "c" ON "c"."post_id" = "post"."id"
GROUP BY "post"."id"
The query is quite fast if I omit one of the LEFT JOIN statements, but becomes 5-10x slower once I add the third. From my basic understanding of joins, shouldn't Postgres be joining these tables to post separately? Why the spike after the third join?
How may I rewrite this query to be more performant?
Running EXPLAIN (ANALYZE, BUFFERS) on the query locally yields:
GroupAggregate (cost=26.31..114.09 rows=12 width=28) (actual time=11.466..11.580 rows=15 loops=1)
Group Key: post.id
Buffers: shared hit=13
-> Merge Left Join (cost=26.31..80.52 rows=3345 width=16) (actual time=0.171..6.298 rows=20443 loops=1)
Merge Cond: (post.id = l.post_id)
Buffers: shared hit=13
-> Merge Left Join (cost=8.49..11.98 rows=217 width=12) (actual time=0.085..0.682 rows=2042 loops=1)
Merge Cond: (post.id = ul.post_id)
Buffers: shared hit=4
-> Sort (cost=5.40..5.53 rows=51 width=8) (actual time=0.061..0.067 rows=60 loops=1)
Sort Key: post.id
Sort Method: quicksort Memory: 27kB
Buffers: shared hit=3
-> Hash Right Join (cost=2.27..3.96 rows=51 width=8) (actual time=0.027..0.048 rows=60 loops=1)
Hash Cond: (l.post_id = post.id)
Buffers: shared hit=3
-> Seq Scan on like l (cost=0.00..1.51 rows=51 width=8) (actual time=0.005..0.009 rows=49 loops=1)
Buffers: shared hit=1
-> Hash (cost=2.12..2.12 rows=12 width=4) (actual time=0.017..0.017 rows=15 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
Buffers: shared hit=2
-> Seq Scan on post (cost=0.00..2.12 rows=12 width=4) (actual time=0.009..0.012 rows=15 loops=1)
Buffers: shared hit=2
-> Sort (cost=3.08..3.21 rows=51 width=8) (actual time=0.021..0.212 rows=2030 loops=1)
Sort Key: ul.post_id
Sort Method: quicksort Memory: 27kB
Buffers: shared hit=1
-> Seq Scan on like ul (cost=0.00..1.64 rows=51 width=8) (actual time=0.004..0.012 rows=49 loops=1)
Filter: (user_id = 1)
Buffers: shared hit=1
-> Sort (cost=17.82..18.28 rows=185 width=8) (actual time=0.084..1.506 rows=20438 loops=1)
Sort Key: c.post_id
Sort Method: quicksort Memory: 34kB
Buffers: shared hit=9
-> Seq Scan on comment c (cost=0.00..10.85 rows=185 width=8) (actual time=0.004..0.045 rows=192 loops=1)
Buffers: shared hit=9
Planning Time: 0.319 ms
Execution Time: 11.624 ms

postgres window function trebles query time

I'm using postgres 10, and have the following query
select
count(task.id) over() as _total_ ,
json_agg(u.*) as users,
task.*
from task
left outer join taskuserlink_history tu on (task.id = tu.taskid)
left outer join "user" u on (tu.userId = u.id)
group by task.id offset 10 limit 10;
this query takes approx 800ms to execute
if I remove the count(task.id) over() as _total_ , line, then it executes in 250ms
I have to confess being a complete sql noob, so the query itself may be completely borked
I was wondering if anyone could point to the flaws in the query, and make suggestions on how to speed it up.
The number of tasks is approx 15k, with an average of 5 users per task, linked through taskuserlink
I have looked at the pgadmin "explain" diagram
but to be honest can't really figure it out yet ;)
the table definitions are
task , with id (int) as primary column
taskuserlink_history, with taskId (int) and userId (int) (both as foreign key constraints, indexed)
user, with id (int) as primary column
the query plan is as follows
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=4.74..12.49 rows=10 width=44) (actual time=1178.016..1178.043 rows=10 loops=1)
Buffers: shared hit=3731, temp read=6655 written=6914
-> WindowAgg (cost=4.74..10248.90 rows=13231 width=44) (actual time=1178.014..1178.040 rows=10 loops=1)
Buffers: shared hit=3731, temp read=6655 written=6914
-> GroupAggregate (cost=4.74..10083.51 rows=13231 width=36) (actual time=0.417..1049.294 rows=13255 loops=1)
Group Key: task.id
Buffers: shared hit=3731
-> Nested Loop Left Join (cost=4.74..9586.77 rows=66271 width=36) (actual time=0.103..309.372 rows=66162 loops=1)
Join Filter: (taskuserlink_history.userid = user_archive.id)
Rows Removed by Join Filter: 1182904
Buffers: shared hit=3731
-> Merge Left Join (cost=0.58..5563.22 rows=66271 width=8) (actual time=0.044..73.598 rows=66162 loops=1)
Merge Cond: (task.id = taskuserlink_history.taskid)
Buffers: shared hit=3629
-> Index Only Scan using task_pkey on task (cost=0.29..1938.30 rows=13231 width=4) (actual time=0.026..7.683 rows=13255 loops=1)
Heap Fetches: 13255
Buffers: shared hit=1810
-> Index Scan using taskuserlink_history_task_fk_idx on taskuserlink_history (cost=0.29..2764.46 rows=66271 width=8) (actual time=0.015..40.109 rows=66162 loops=1)
Filter: (timeend IS NULL)
Rows Removed by Filter: 13368
Buffers: shared hit=1819
-> Materialize (cost=4.17..50.46 rows=4 width=36) (actual time=0.000..0.001 rows=19 loops=66162)
Buffers: shared hit=102
-> Bitmap Heap Scan on user_archive (cost=4.17..50.44 rows=4 width=36) (actual time=0.050..0.305 rows=45 loops=1)
Recheck Cond: (archived_at IS NULL)
Heap Blocks: exact=11
Buffers: shared hit=102
-> Bitmap Index Scan on user_unique_username (cost=0.00..4.16 rows=4 width=0) (actual time=0.014..0.014 rows=46 loops=1)
Buffers: shared hit=1
SubPlan 1
-> Aggregate (cost=8.30..8.31 rows=1 width=8) (actual time=0.003..0.003 rows=1 loops=45)
Buffers: shared hit=90
-> Index Scan using task_assignedto_idx on task task_1 (cost=0.29..8.30 rows=1 width=4) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (assignedtoid = user_archive.id)
Buffers: shared hit=90
Planning time: 0.989 ms
Execution time: 1191.451 ms
(37 rows)
without the window function it is
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=4.74..12.36 rows=10 width=36) (actual time=0.510..1.763 rows=10 loops=1)
Buffers: shared hit=91
-> GroupAggregate (cost=4.74..10083.51 rows=13231 width=36) (actual time=0.509..1.759 rows=10 loops=1)
Group Key: task.id
Buffers: shared hit=91
-> Nested Loop Left Join (cost=4.74..9586.77 rows=66271 width=36) (actual time=0.073..0.744 rows=50 loops=1)
Join Filter: (taskuserlink_history.userid = user_archive.id)
Rows Removed by Join Filter: 361
Buffers: shared hit=91
-> Merge Left Join (cost=0.58..5563.22 rows=66271 width=8) (actual time=0.029..0.161 rows=50 loops=1)
Merge Cond: (task.id = taskuserlink_history.taskid)
Buffers: shared hit=7
-> Index Only Scan using task_pkey on task (cost=0.29..1938.30 rows=13231 width=4) (actual time=0.016..0.031 rows=11 loops=1)
Heap Fetches: 11
Buffers: shared hit=4
-> Index Scan using taskuserlink_history_task_fk_idx on taskuserlink_history (cost=0.29..2764.46 rows=66271 width=8) (actual time=0.009..0.081 rows=50 loops=1)
Filter: (timeend IS NULL)
Rows Removed by Filter: 11
Buffers: shared hit=3
-> Materialize (cost=4.17..50.46 rows=4 width=36) (actual time=0.001..0.009 rows=8 loops=50)
Buffers: shared hit=84
-> Bitmap Heap Scan on user_archive (cost=4.17..50.44 rows=4 width=36) (actual time=0.040..0.382 rows=38 loops=1)
Recheck Cond: (archived_at IS NULL)
Heap Blocks: exact=7
Buffers: shared hit=84
-> Bitmap Index Scan on user_unique_username (cost=0.00..4.16 rows=4 width=0) (actual time=0.012..0.012 rows=46 loops=1)
Buffers: shared hit=1
SubPlan 1
-> Aggregate (cost=8.30..8.31 rows=1 width=8) (actual time=0.005..0.005 rows=1 loops=38)
Buffers: shared hit=76
-> Index Scan using task_assignedto_idx on task task_1 (cost=0.29..8.30 rows=1 width=4) (actual time=0.003..0.003 rows=0 loops=38)
Index Cond: (assignedtoid = user_archive.id)
Buffers: shared hit=76
Planning time: 0.895 ms
Execution time: 1.890 ms
(35 rows)|
I believe the LIMIT clause is making the difference. LIMIT is limiting the number of rows returned, not neccessarily the work involved:
Your second query can be aborted early after 20 rows have been constructed (10 for OFFSET and 10 for LIMIT).
However, your first query needs to go through the whole set to calculate the count(task.id).
Not what you were asking, but I say it anyway:
"user" is not a table, but a view. That is were both queries actually get slower than they should be (The "Materialize" in the plan).
Using OFFSET for paging calls for trouble because it will get slow when the OFFSET increases
Using OFFSET and LIMIT without an ORDER BY is most likely not what you want. The result sets might not be identical on consecutive calls.