Related
I have table (over 100 millions records) on PostgreSQL 13.1
CREATE TABLE report
(
id serial primary key,
license_plate_id integer,
datetime timestamp
);
Indexes (for test I create both of them):
create index report_lp_datetime_index on report (license_plate_id, datetime);
create index report_lp_datetime_desc_index on report (license_plate_id desc, datetime desc);
So, my question is why query like
select * from report r
where r.license_plate_id in (1,2,4,5,6,7,8,10,15,22,34,75)
order by datetime desc
limit 100
Is very slow (~10sec). But query without order statement is fast (milliseconds).
Explain:
explain (analyze, buffers, format text) select * from report r
where r.license_plate_id in (1,2,4,5,6,7,8,10,15,22,34, 75,374,57123)
limit 100
Limit (cost=0.57..400.38 rows=100 width=316) (actual time=0.037..0.216 rows=100 loops=1)
Buffers: shared hit=103
-> Index Scan using report_lp_id_idx on report r (cost=0.57..44986.97 rows=11252 width=316) (actual time=0.035..0.202 rows=100 loops=1)
Index Cond: (license_plate_id = ANY ('{1,2,4,5,6,7,8,10,15,22,34,75,374,57123}'::integer[]))
Buffers: shared hit=103
Planning Time: 0.228 ms
Execution Time: 0.251 ms
explain (analyze, buffers, format text) select * from report r
where r.license_plate_id in (1,2,4,5,6,7,8,10,15,22,34,75,374,57123)
order by datetime desc
limit 100
Limit (cost=44193.63..44193.88 rows=100 width=316) (actual time=4921.030..4921.047 rows=100 loops=1)
Buffers: shared hit=11455 read=671
-> Sort (cost=44193.63..44221.76 rows=11252 width=316) (actual time=4921.028..4921.035 rows=100 loops=1)
Sort Key: datetime DESC
Sort Method: top-N heapsort Memory: 128kB
Buffers: shared hit=11455 read=671
-> Bitmap Heap Scan on report r (cost=151.18..43763.59 rows=11252 width=316) (actual time=54.422..4911.927 rows=12148 loops=1)
Recheck Cond: (license_plate_id = ANY ('{1,2,4,5,6,7,8,10,15,22,34,75,374,57123}'::integer[]))
Heap Blocks: exact=12063
Buffers: shared hit=11455 read=671
-> Bitmap Index Scan on report_lp_id_idx (cost=0.00..148.37 rows=11252 width=0) (actual time=52.631..52.632 rows=12148 loops=1)
Index Cond: (license_plate_id = ANY ('{1,2,4,5,6,7,8,10,15,22,34,75,374,57123}'::integer[]))
Buffers: shared hit=59 read=4
Planning Time: 0.427 ms
Execution Time: 4921.128 ms
You seem to have rather slow storage, if reading 671 8kB-blocks from disk takes a couple of seconds.
The way to speed this up is to reorder the table in the same way as the index, so that you can find the required rows in the same or adjacent table blocks:
CLUSTER report_lp_id_idx USING report_lp_id_idx;
Be warned that rewriting the table in this way causes downtime – the table will not be available while it is being rewritten. Moreover, PostgreSQL does not maintain the table order, so subsequent data modifications will cause performance to gradually deteriorate, so that after a while you will have to run CLUSTER again.
But if you need this query to be fast no matter what, CLUSTER is the way to go.
Your two indices do exactly the same thing, so you can remove the second one, it's useless.
To optimize your query, the order of the fields inside the index must be reversed:
create index report_lp_datetime_index on report (datetime,license_plate_id);
BEGIN;
CREATE TABLE foo (d INTEGER, i INTEGER);
INSERT INTO foo SELECT random()*100000, random()*1000 FROM generate_series(1,1000000) s;
CREATE INDEX foo_d_i ON foo(d DESC,i);
COMMIT;
VACUUM ANALYZE foo;
EXPLAIN ANALYZE SELECT * FROM foo WHERE i IN (1,2,4,5,6,7,8,10,15,22,34,75) ORDER BY d DESC LIMIT 100;
Limit (cost=0.42..343.92 rows=100 width=8) (actual time=0.076..9.359 rows=100 loops=1)
-> Index Only Scan Backward using foo_d_i on foo (cost=0.42..40976.43 rows=11929 width=8) (actual time=0.075..9.339 rows=100 loops=1)
Filter: (i = ANY ('{1,2,4,5,6,7,8,10,15,22,34,75}'::integer[]))
Rows Removed by Filter: 9016
Heap Fetches: 0
Planning Time: 0.339 ms
Execution Time: 9.387 ms
Note the index is not used to optimize the WHERE clause. It is used here as a compact and fast way to store references to the rows ordered by date DESC, so the ORDER BY can do an index-only scan and avoid sorting. By adding column id to the index, an index-only scan can be performed to test the condition on id, without hitting the table for every row. Since there is a low LIMIT value it does not need to scan the whole index, it only scans it in date DESC order until it finds enough rows satisfying the WHERE condition to return the result.
It will be faster if you create the index in date DESC order, this could be useful if you use ORDER BY date DESC + LIMIT in other queries too.
You forget that OP's table has a third column, and he is using SELECT *. So that wouldn't be an index-only scan.
Easy to work around. The optimum way to do this query would be an index-only scan to filter on WHERE conditions, then LIMIT, then hit the table to get the rows. For some reason if "select *" is used postgres takes the id column from the table instead of taking it from the index, which results in lots of unnecessary heap fetches for rows whose id is rejected by the WHERE condition.
Easy to work around, by doing it manually. I've also added another bogus column to make sure the SELECT * hits the table.
EXPLAIN (ANALYZE,buffers) SELECT * FROM foo
JOIN (SELECT d,i FROM foo WHERE i IN (1,2,4,5,6,7,8,10,15,22,34,75) ORDER BY d DESC LIMIT 100) f USING (d,i)
ORDER BY d DESC LIMIT 100;
Limit (cost=0.85..1281.94 rows=1 width=17) (actual time=0.052..3.618 rows=100 loops=1)
Buffers: shared hit=453
-> Nested Loop (cost=0.85..1281.94 rows=1 width=17) (actual time=0.050..3.594 rows=100 loops=1)
Buffers: shared hit=453
-> Limit (cost=0.42..435.44 rows=100 width=8) (actual time=0.037..2.953 rows=100 loops=1)
Buffers: shared hit=53
-> Index Only Scan using foo_d_i on foo foo_1 (cost=0.42..51936.43 rows=11939 width=8) (actual time=0.037..2.935 rows=100 loops=1)
Filter: (i = ANY ('{1,2,4,5,6,7,8,10,15,22,34,75}'::integer[]))
Rows Removed by Filter: 9010
Heap Fetches: 0
Buffers: shared hit=53
-> Index Scan using foo_d_i on foo (cost=0.42..8.45 rows=1 width=17) (actual time=0.005..0.005 rows=1 loops=100)
Index Cond: ((d = foo_1.d) AND (i = foo_1.i))
Buffers: shared hit=400
Execution Time: 3.663 ms
Another option is to just add the primary key to the date,license_plate index.
SELECT * FROM foo JOIN (SELECT id FROM foo WHERE i IN (1,2,4,5,6,7,8,10,15,22,34,75) ORDER BY d DESC LIMIT 100) f USING (id) ORDER BY d DESC LIMIT 100;
Limit (cost=1357.98..1358.23 rows=100 width=17) (actual time=3.920..3.947 rows=100 loops=1)
Buffers: shared hit=473
-> Sort (cost=1357.98..1358.23 rows=100 width=17) (actual time=3.919..3.931 rows=100 loops=1)
Sort Key: foo.d DESC
Sort Method: quicksort Memory: 32kB
Buffers: shared hit=473
-> Nested Loop (cost=0.85..1354.66 rows=100 width=17) (actual time=0.055..3.858 rows=100 loops=1)
Buffers: shared hit=473
-> Limit (cost=0.42..509.41 rows=100 width=8) (actual time=0.039..3.116 rows=100 loops=1)
Buffers: shared hit=73
-> Index Only Scan using foo_d_i_id on foo foo_1 (cost=0.42..60768.43 rows=11939 width=8) (actual time=0.039..3.093 rows=100 loops=1)
Filter: (i = ANY ('{1,2,4,5,6,7,8,10,15,22,34,75}'::integer[]))
Rows Removed by Filter: 9010
Heap Fetches: 0
Buffers: shared hit=73
-> Index Scan using foo_pkey on foo (cost=0.42..8.44 rows=1 width=17) (actual time=0.006..0.006 rows=1 loops=100)
Index Cond: (id = foo_1.id)
Buffers: shared hit=400
Execution Time: 3.972 ms
Edit
After thinking about it... since the LIMIT restricts the output to 100 rows ordered by date desc, wouldn't it be nice if we could get the 100 most recent rows for each license_plate_id, put all that into a top-n sort, and only keep the best 100 for all license_plate_ids? That would avoid reading and throwing away a lot of rows from the index. Even if that's much faster than hitting the table, it will still load up these index pages in RAM and clog up your buffers with stuff you don't actually need to keep in cache. Let's use LATERAL JOIN:
EXPLAIN (ANALYZE,BUFFERS)
SELECT * FROM foo
JOIN (SELECT d,i FROM
(VALUES (1),(2),(4),(5),(6),(7),(8),(10),(15),(22),(34),(75)) idlist
CROSS JOIN LATERAL
(SELECT d,i FROM foo WHERE i=idlist.column1 ORDER BY d DESC LIMIT 100) f2
ORDER BY d DESC LIMIT 100
) f3 USING (d,i)
ORDER BY d DESC LIMIT 100;
It's even faster: 2ms, and it uses the index on (license_plate_id,date) instead of the other way around. Also, and this is important, since each subquery in the lateral hits only the index pages that contain rows that will actually be selected, while the previous queries hit much more index pages. So you save on RAM buffers.
If you don't need the index on (date,license_plate_id) and don't want to keep a useless index, that could be interesting since this query doesn't use it. On the other hand, if you need the index on (date,license_plate_id) for something else and want to keep it, then... maybe not.
Please post results for the winning query 🔥
In my query, I just want to call data with exact where conditions. These where conditions were created in index. Bu the explain shows bit index-scan. I couldn't understand why.
My query looks like below:
Select
r.spend,
r.date,
...
from metadata m
inner join
report r
on m.org_id = r.org_id and m.country_or_region = r.country_or_region and m.campaign_id = r.campaign_id and m.keyword_id = r.keyword_id
where r.org_id = 1 and m.keyword_type = 'KEYWORD'
offset 0 limit 20
Indexes:
Metadata(org_id, keyword_type, country_or_region, campaign_id, keyword_id);
Report(org_id, country_or_region, campaign_id, keyword_id, date);
Explain Analyze:
"Limit (cost=811883.21..910327.87 rows=20 width=8) (actual time=18120.268..18235.831 rows=20 loops=1)"
" -> Gather (cost=811883.21..2702020.67 rows=384 width=8) (actual time=18120.267..18235.791 rows=20 loops=1)"
" Workers Planned: 2"
" Workers Launched: 2"
" -> Parallel Hash Join (cost=810883.21..2700982.27 rows=160 width=8) (actual time=18103.440..18103.496 rows=14 loops=3)"
" Hash Cond: (((r.country_or_region)::text = (m.country_or_region)::text) AND (r.campaign_id = m.campaign_id) AND (r.keyword_id = m.keyword_id))"
" -> Parallel Bitmap Heap Scan on report r (cost=260773.11..2051875.83 rows=3939599 width=35) (actual time=552.601..8532.962 rows=3162553 loops=3)"
" Recheck Cond: (org_id = 479360)"
" Rows Removed by Index Recheck: 21"
" Heap Blocks: exact=20484 lossy=84350"
" -> Bitmap Index Scan on idx_kr_org_date_camp (cost=0.00..258409.35 rows=9455038 width=0) (actual time=539.329..539.329 rows=9487660 loops=1)"
" Index Cond: (org_id = 479360)"
" -> Parallel Hash (cost=527278.08..527278.08 rows=938173 width=26) (actual time=7425.062..7425.062 rows=727133 loops=3)"
" Buckets: 65536 Batches: 64 Memory Usage: 2656kB"
" -> Parallel Bitmap Heap Scan on metadata m (cost=88007.61..527278.08 rows=938173 width=26) (actual time=1007.028..7119.233 rows=727133 loops=3)"
" Recheck Cond: ((org_id = 479360) AND ((keyword_type)::text = 'KEYWORD'::text))"
" Rows Removed by Index Recheck: 3"
" Heap Blocks: exact=14585 lossy=11054"
" -> Bitmap Index Scan on idx_primary (cost=0.00..87444.71 rows=2251615 width=0) (actual time=1014.631..1014.631 rows=2181399 loops=1)"
" Index Cond: ((org_id = 479360) AND ((keyword_type)::text = 'KEYWORD'::text))"
"Planning Time: 0.492 ms"
"Execution Time: 18235.879 ms"
In here, I just want to call 20 items. It should be more effective?
The Bitmap Index Scan happens when the result set will have a high selectivity rate with respect to the search conditions (i.e., there is a high percentage of rows that satisfy the search criteria). In this case, the planner will plan to scan the entire index, forming a bitmap of which pages on disk to pull the data out from (which happens during the Bitmap Heap Scan step). This is better than a Sequential Scan because it only scans the relevant pages on disk, skipping the pages that it knows relevant data does not exist. Depending on the statistics available to the optimizer, it may not be advantageous to do an Index Scan or an Index-Only Scan, but it is still better than a Sequential Scan.
To complete the answer to the question, an Index-Only Scan is a scan of the index that will pull the relevant data without having to visit the actual table. This is because the relevant data is already in the index. Take, for example, this table:
postgres=# create table foo (id int primary key, name text);
CREATE TABLE
postgres=# insert into foo values (generate_series(1,1000000),'foo');
INSERT 0 1000000
There is an index on the id column of this table, and suppose we call the following query:
postgres=# EXPLAIN ANALYZE SELECT * FROM foo WHERE id < 100;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------
Index Scan using foo_pkey on foo (cost=0.42..10.25 rows=104 width=8) (actual time=0.012..1.027 rows=99 loops=1)
Index Cond: (id < 100)
Planning Time: 0.190 ms
Execution Time: 2.067 ms
(4 rows)
This query results in an Index scan because it scans the index for the rows that have id < 100, and then visits the actual table on disk to pull the other columns included in the * portion of the SELECT query.
However, suppose we call the following query (notice SELECT id instead of SELECT *):
postgres=# EXPLAIN ANALYZE SELECT id FROM foo WHERE id < 100;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Index Only Scan using foo_pkey on foo (cost=0.42..10.25 rows=104 width=4) (actual time=0.019..0.996 rows=99 loops=1)
Index Cond: (id < 100)
Heap Fetches: 99
Planning Time: 0.098 ms
Execution Time: 1.980 ms
(5 rows)
This results in an Index-only scan because only the id column is requested, and that is included (naturally) in the index, so there's no need to visit the actual table on disk to retrieve anything else. This saves time, but its occurrence is very limited.
To answer your question about limiting to 20 results, the limiting occurs after the Bitmap Index Scan has occurred, so the runtime will still be the same whether you limit to 20, 40, or some other value. In the case of an Index/Index-Only Scan, the executor will stop scanning after it has acquired enough rows as specified by the LIMIT clause. In your case, with the Bitmap Heap Scan, this isn’t possible
I have a single small table(~400k rows), the table is indexed by collection_id and contains a JSON column with several GIN indexes defined, one of them is on the value tagline.id.
The query to get all the objects with a specific tagline.id sometimes is VERY slow:
explain (analyze, buffers)
SELECT "objects_object"."created",
"objects_object"."modified",
"objects_object"."_id",
"objects_object"."id",
"objects_object"."collection_id",
"objects_object"."data",
"objects_object"."search",
"objects_object"."location"::bytea
FROM "objects_object"
WHERE ("objects_object"."collection_id" IN (3381, 3321, 3312, 3262, 3068, 2684, 2508, 2159, 2158, 2154, 2157, 2156)
AND (("objects_object"."data" #>> ARRAY['tagline','id']))::float IN ('8')
AND ("objects_object"."data" -> 'tagline') ? 'id')
ORDER BY "objects_object"."created" DESC,
"objects_object"."id" ASC
LIMIT 101;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=8.46..8.47 rows=1 width=1239) (actual time=5513.374..5513.399 rows=101 loops=1)
Buffers: shared hit=4480 read=6261
-> Sort (cost=8.46..8.47 rows=1 width=1239) (actual time=5513.372..5513.389 rows=101 loops=1)
Sort Key: created DESC, id
Sort Method: top-N heapsort Memory: 247kB
Buffers: shared hit=4480 read=6261
-> Index Scan using index_tagline_id_float_51a27976 on objects_object (cost=0.42..8.45 rows=1 width=1239) (actual time=943.689..5513.002 rows=235 loops=1)
Index Cond: (((data #>> '{tagline,id}'::text[]))::double precision = '8'::double precision)
Filter: (collection_id = ANY ('{3381,3321,3312,3262,3068,2684,2508,2159,2158,2154,2157,2156}'::integer[]))
Rows Removed by Filter: 47295
Buffers: shared hit=4480 read=6261
Planning time: 0.244 ms
Execution time: 5513.439 ms
(13 rows)
If executed multiple times the execution time drops to ~ 5 ms.
What is taking so long? Why after the first time the execution time drops that much?
I don't think it's memory related since the default memory(4MB) is much higher than the required(247Kb).
EDIT:
Index definitions:
SELECT indexdef FROM pg_indexes
WHERE indexname = 'index_tagline_id_float_51a27976';
indexdef
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE INDEX index_tagline_id_float_51a27976 ON public.objects_object USING btree ((((data #>> ARRAY['tagline'::text, 'id'::text]))::double precision)) WHERE ((data -> 'tagline'::text) ? 'id'::text)
(1 row)
SELECT indexdef FROM pg_indexes
WHERE indexname = 'objects_object_collection_id_6f1559f5';
indexdef
---------------------------------------------------------------------------------------------------------
CREATE INDEX objects_object_collection_id_6f1559f5 ON public.objects_object USING btree (collection_id)
(1 row)
EDIT:
After adding the index test:
select indexdef from pg_indexes where indexname='test';
indexdef
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE INDEX test ON public.objects_object USING btree ((((data #>> ARRAY['tagline'::text, 'id'::text]))::double precision), collection_id) WHERE ((data -> 'tagline'::text) ? 'id'::text)
(1 row)
The execution time decreased, but so the buffer shared hit, not sure this improved the performance then:
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=8.46..8.47 rows=1 width=1238) (actual time=1721.260..1721.281 rows=101 loops=1)
Buffers: shared hit=5460 read=5115
-> Sort (cost=8.46..8.47 rows=1 width=1238) (actual time=1721.257..1721.270 rows=101 loops=1)
Sort Key: created DESC, id
Sort Method: top-N heapsort Memory: 298kB
Buffers: shared hit=5460 read=5115
-> Index Scan using test on objects_object (cost=0.42..8.45 rows=1 width=1238) (actual time=1682.637..1720.793 rows=235 loops=1)
Index Cond: (((data #>> '{tagline,id}'::text[]))::double precision = '8'::double precision)
Filter: (collection_id = ANY ('{3381,3321,3312,3262,3068,2684,2508,2159,2158,2154,2157,2156}'::integer[]))
Rows Removed by Filter: 47295
Buffers: shared hit=5454 read=5115
Planning time: 238.364 ms
Execution time: 1762.996 ms
(13 rows)
The problem seems to be that collection_id should be part of the index condition, not on the filtering, this would avoid getting from the (slow) data storage a large amount of data.
Why is the index not working as expected?
UPDATE:
Apparently the order of the parameters impacted on the query plan,
i rewrote the index as:
select indexdef from pg_indexes where indexname='test';
indexdef
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE INDEX test ON public.objects_object USING btree (collection_id, (((data #>> ARRAY['tagline'::text, 'id'::text]))::double precision)) WHERE ((data -> 'tagline'::text) ? 'id'::text)
Running now the query we can see the lower number of read records:
Limit (cost=57.15..57.16 rows=1 width=1177) (actual time=1.043..1.059 rows=101 loops=1)
Buffers: shared hit=101 read=10
-> Sort (cost=57.15..57.16 rows=1 width=1177) (actual time=1.040..1.047 rows=101 loops=1)
Sort Key: created DESC, id
Sort Method: top-N heapsort Memory: 304kB
Buffers: shared hit=101 read=10
-> Index Scan using test on objects_object (cost=0.42..57.14 rows=1 width=1177) (actual time=0.094..0.670 rows=232 loops=1)
Index Cond: ((collection_id = ANY ('{3381,3321,3312,3262,3068,2684,2508,2159,2158,2154,2157,2156}'::integer[])) AND (((data #>> '{tagline,id}'::text[]))::double precision = '8'::double precisio
n))
Buffers: shared hit=95 read=10
Planning time: 416.365 ms
Execution time: 43.463 ms
(11 rows)
This particular query could be sped up with the following index:
CREATE INDEX ON public.objects_object (
((data #>> ARRAY['tagline'::text, 'id'::text])::double precision),
collection_id
) WHERE (data -> 'tagline') ? 'id';
This will avoid the filter in the index scan, which is where most of the time is spent.
I am really struggling to optimize this query:
SELECT wins / (wins + COUNT(loosers.match_id) + 0.) winrate, wins + COUNT(loosers.match_id) matches, winners.winning_champion_one_id, winners.winning_champion_two_id, winners.winning_champion_three_id, winners.winning_champion_four_id, winners.winning_champion_five_id
FROM
(
SELECT COUNT(match_id) wins, winning_champion_one_id, winning_champion_two_id, winning_champion_three_id, winning_champion_four_id, winning_champion_five_id FROM matches
WHERE
157 IN (winning_champion_one_id, winning_champion_two_id, winning_champion_three_id, winning_champion_four_id, winning_champion_five_id)
GROUP BY winning_champion_one_id, winning_champion_two_id, winning_champion_three_id, winning_champion_four_id, winning_champion_five_id
) winners
LEFT OUTER JOIN matches loosers ON
winners.winning_champion_one_id = loosers.loosing_champion_one_id AND
winners.winning_champion_two_id = loosers.loosing_champion_two_id AND
winners.winning_champion_three_id = loosers.loosing_champion_three_id AND
winners.winning_champion_four_id = loosers.loosing_champion_four_id AND
winners.winning_champion_five_id = loosers.loosing_champion_five_id
GROUP BY winners.wins, winners.winning_champion_one_id, winners.winning_champion_two_id, winners.winning_champion_three_id, winners.winning_champion_four_id, winners.winning_champion_five_id
HAVING wins + COUNT(loosers.match_id) >= 20
ORDER BY winrate DESC, matches DESC
LIMIT 1;
And this is the output of EXPLAIN (BUFFERS, ANALYZE):
Limit (cost=72808.80..72808.80 rows=1 width=58) (actual time=1478.749..1478.749 rows=1 loops=1)
Buffers: shared hit=457002
-> Sort (cost=72808.80..72837.64 rows=11535 width=58) (actual time=1478.747..1478.747 rows=1 loops=1)
" Sort Key: ((((count(matches.match_id)))::numeric / ((((count(matches.match_id)) + count(loosers.match_id)))::numeric + '0'::numeric))) DESC, (((count(matches.match_id)) + count(loosers.match_id))) DESC"
Sort Method: top-N heapsort Memory: 25kB
Buffers: shared hit=457002
-> HashAggregate (cost=72462.75..72751.12 rows=11535 width=58) (actual time=1448.941..1478.643 rows=83 loops=1)
" Group Key: (count(matches.match_id)), matches.winning_champion_one_id, matches.winning_champion_two_id, matches.winning_champion_three_id, matches.winning_champion_four_id, matches.winning_champion_five_id"
Filter: (((count(matches.match_id)) + count(loosers.match_id)) >= 20)
Rows Removed by Filter: 129131
Buffers: shared hit=457002
-> Nested Loop Left Join (cost=9857.76..69867.33 rows=115352 width=26) (actual time=288.086..1309.687 rows=146610 loops=1)
Buffers: shared hit=457002
-> HashAggregate (cost=9857.33..11010.85 rows=115352 width=18) (actual time=288.056..408.317 rows=129214 loops=1)
" Group Key: matches.winning_champion_one_id, matches.winning_champion_two_id, matches.winning_champion_three_id, matches.winning_champion_four_id, matches.winning_champion_five_id"
Buffers: shared hit=22174
-> Bitmap Heap Scan on matches (cost=1533.34..7455.69 rows=160109 width=18) (actual time=26.618..132.844 rows=161094 loops=1)
Recheck Cond: ((157 = winning_champion_one_id) OR (157 = winning_champion_two_id) OR (157 = winning_champion_three_id) OR (157 = winning_champion_four_id) OR (157 = winning_champion_five_id))
Heap Blocks: exact=21594
Buffers: shared hit=22174
-> BitmapOr (cost=1533.34..1533.34 rows=164260 width=0) (actual time=22.190..22.190 rows=0 loops=1)
Buffers: shared hit=580
-> Bitmap Index Scan on matches_winning_champion_one_id_index (cost=0.00..35.03 rows=4267 width=0) (actual time=0.045..0.045 rows=117 loops=1)
Index Cond: (157 = winning_champion_one_id)
Buffers: shared hit=3
-> Bitmap Index Scan on matches_winning_champion_two_id_index (cost=0.00..47.22 rows=5772 width=0) (actual time=0.665..0.665 rows=3010 loops=1)
Index Cond: (157 = winning_champion_two_id)
Buffers: shared hit=13
-> Bitmap Index Scan on matches_winning_champion_three_id_index (cost=0.00..185.53 rows=22840 width=0) (actual time=3.824..3.824 rows=23893 loops=1)
Index Cond: (157 = winning_champion_three_id)
Buffers: shared hit=89
-> Bitmap Index Scan on matches_winning_champion_four_id_index (cost=0.00..537.26 rows=66257 width=0) (actual time=8.069..8.069 rows=67255 loops=1)
Index Cond: (157 = winning_champion_four_id)
Buffers: shared hit=244
-> Bitmap Index Scan on matches_winning_champion_five_id_index (cost=0.00..528.17 rows=65125 width=0) (actual time=9.577..9.577 rows=67202 loops=1)
Index Cond: (157 = winning_champion_five_id)
Buffers: shared hit=231
-> Index Scan using matches_loosing_champion_ids_index on matches loosers (cost=0.43..0.49 rows=1 width=18) (actual time=0.006..0.006 rows=0 loops=129214)
Index Cond: ((matches.winning_champion_one_id = loosing_champion_one_id) AND (matches.winning_champion_two_id = loosing_champion_two_id) AND (matches.winning_champion_three_id = loosing_champion_three_id) AND (matches.winning_champion_four_id = loosing_champion_four_id) AND (matches.winning_champion_five_id = loosing_champion_five_id))
Buffers: shared hit=434828
Planning time: 0.584 ms
Execution time: 1479.779 ms
Table and index definitions:
create table matches (
match_id bigint not null,
winning_champion_one_id smallint,
winning_champion_two_id smallint,
winning_champion_three_id smallint,
winning_champion_four_id smallint,
winning_champion_five_id smallint,
loosing_champion_one_id smallint,
loosing_champion_two_id smallint,
loosing_champion_three_id smallint,
loosing_champion_four_id smallint,
loosing_champion_five_id smallint,
constraint matches_match_id_pk primary key (match_id)
);
create index matches_winning_champion_one_id_index on matches (winning_champion_one_id);
create index matches_winning_champion_two_id_index on matches (winning_champion_two_id);
create index matches_winning_champion_three_id_index on matches (winning_champion_three_id);
create index matches_winning_champion_four_id_index on matches (winning_champion_four_id);
create index matches_winning_champion_five_id_index on matches (winning_champion_five_id);
create index matches_loosing_champion_ids_index on matches (loosing_champion_one_id, loosing_champion_two_id, loosing_champion_three_id, loosing_champion_four_id, loosing_champion_five_id);
create index matches_loosing_champion_one_id_index on matches (loosing_champion_one_id);
create index matches_loosing_champion_two_id_index on matches (loosing_champion_two_id);
create index matches_loosing_champion_three_id_index on matches (loosing_champion_three_id);
create index matches_loosing_champion_four_id_index on matches (loosing_champion_four_id);
create index matches_loosing_champion_five_id_index on matches (loosing_champion_five_id);
The table can have 100m+ rows. At the moment it does have about 20m rows.
Current size of table and indexes:
public.matches, 2331648 rows, 197 MB
public.matches_riot_match_id_pk, 153 MB
public.matches_loosing_champion_ids_index, 136 MB
public.matches_loosing_champion_four_id_index, 113 MB
public.matches_loosing_champion_five_id_index, 113 MB
public.matches_winning_champion_one_id_index, 113 MB
public.matches_winning_champion_five_id_index, 113 MB
public.matches_winning_champion_three_id_index, 112 MB
public.matches_loosing_champion_three_id_index, 112 MB
public.matches_winning_champion_four_id_index, 112 MB
public.matches_loosing_champion_one_id_index, 112 MB
public.matches_winning_champion_two_id_index, 112 MB
public.matches_loosing_champion_two_id_index, 112 MB
These are the only changes I made to postgresql.conf:
max_connections = 50
shared_buffers = 6GB
effective_cache_size = 18GB
work_mem = 125829kB
maintenance_work_mem = 1536MB
min_wal_size = 1GB
max_wal_size = 2GB
checkpoint_completion_target = 0.7
wal_buffers = 16MB
default_statistics_target = 100
max_parallel_workers_per_gather = 8
min_parallel_relation_size = 1
There is probably something I do overlook.
EDIT:
For anyone wondering. The best approach was the UNION ALL approach. The suggested schema of Erwin unfortunately doesn't work well. Here is the EXPLAIN (ANALYZE, BUFFERS) output of the suggested schema:
Limit (cost=2352157.06..2352157.06 rows=1 width=48) (actual time=1976.709..1976.710 rows=1 loops=1)
Buffers: shared hit=653004
-> Sort (cost=2352157.06..2352977.77 rows=328287 width=48) (actual time=1976.708..1976.708 rows=1 loops=1)
" Sort Key: (((((count(*)))::numeric * 1.0) / (((count(*)) + l.loss))::numeric)) DESC, (((count(*)) + l.loss)) DESC"
Sort Method: top-N heapsort Memory: 25kB
Buffers: shared hit=653004
-> Nested Loop (cost=2.10..2350515.62 rows=328287 width=48) (actual time=0.553..1976.294 rows=145 loops=1)
Buffers: shared hit=653004
-> GroupAggregate (cost=1.67..107492.42 rows=492431 width=16) (actual time=0.084..1409.450 rows=154547 loops=1)
Group Key: w.winner
Buffers: shared hit=188208
-> Merge Join (cost=1.67..100105.96 rows=492431 width=8) (actual time=0.061..1301.578 rows=199530 loops=1)
Merge Cond: (tm.team_id = w.winner)
Buffers: shared hit=188208
-> Index Only Scan using team_member_champion_team_idx on team_member tm (cost=0.56..8978.79 rows=272813 width=8) (actual time=0.026..103.842 rows=265201 loops=1)
Index Cond: (champion_id = 157)
Heap Fetches: 0
Buffers: shared hit=176867
-> Index Only Scan using match_winner_loser_idx on match w (cost=0.43..79893.82 rows=2288093 width=8) (actual time=0.013..597.331 rows=2288065 loops=1)
Heap Fetches: 0
Buffers: shared hit=11341
-> Subquery Scan on l (cost=0.43..4.52 rows=1 width=8) (actual time=0.003..0.003 rows=0 loops=154547)
Filter: (((count(*)) + l.loss) > 19)
Rows Removed by Filter: 0
Buffers: shared hit=464796
-> GroupAggregate (cost=0.43..4.49 rows=2 width=16) (actual time=0.003..0.003 rows=0 loops=154547)
Group Key: l_1.loser
Buffers: shared hit=464796
-> Index Only Scan using match_loser_winner_idx on match l_1 (cost=0.43..4.46 rows=2 width=8) (actual time=0.002..0.002 rows=0 loops=154547)
Index Cond: (loser = w.winner)
Heap Fetches: 0
Buffers: shared hit=464796
Planning time: 0.634 ms
Execution time: 1976.792 ms
And now with the UNION ALL approach and the new schema:
Limit (cost=275211.80..275211.80 rows=1 width=48) (actual time=3540.420..3540.421 rows=1 loops=1)
Buffers: shared hit=199478
CTE t
-> Index Only Scan using team_member_champion_team_idx on team_member (cost=0.56..8978.79 rows=272813 width=8) (actual time=0.027..103.732 rows=265201 loops=1)
Index Cond: (champion_id = 157)
Heap Fetches: 0
Buffers: shared hit=176867
-> Sort (cost=266233.01..266233.51 rows=200 width=48) (actual time=3540.417..3540.417 rows=1 loops=1)
" Sort Key: ((((count((true)))::numeric * 1.0) / (count(*))::numeric)) DESC, (count(*)) DESC"
Sort Method: top-N heapsort Memory: 25kB
Buffers: shared hit=199478
-> HashAggregate (cost=266228.01..266232.01 rows=200 width=48) (actual time=3455.112..3540.301 rows=145 loops=1)
Group Key: t.team_id
Filter: (count(*) > 19)
Rows Removed by Filter: 265056
Buffers: shared hit=199478
-> Append (cost=30088.37..254525.34 rows=936214 width=9) (actual time=315.399..3137.115 rows=386575 loops=1)
Buffers: shared hit=199478
-> Merge Join (cost=30088.37..123088.80 rows=492454 width=9) (actual time=315.398..1583.746 rows=199530 loops=1)
Merge Cond: (match.winner = t.team_id)
Buffers: shared hit=188208
-> Index Only Scan using match_winner_loser_idx on match (cost=0.43..79893.82 rows=2288093 width=8) (actual time=0.033..583.016 rows=2288065 loops=1)
Heap Fetches: 0
Buffers: shared hit=11341
-> Sort (cost=30087.94..30769.97 rows=272813 width=8) (actual time=315.333..402.516 rows=310184 loops=1)
Sort Key: t.team_id
Sort Method: quicksort Memory: 24720kB
Buffers: shared hit=176867
-> CTE Scan on t (cost=0.00..5456.26 rows=272813 width=8) (actual time=0.030..240.150 rows=265201 loops=1)
Buffers: shared hit=176867
-> Merge Join (cost=30088.37..122074.39 rows=443760 width=9) (actual time=134.118..1410.484 rows=187045 loops=1)
Merge Cond: (match_1.loser = t_1.team_id)
Buffers: shared hit=11270
-> Index Only Scan using match_loser_winner_idx on match match_1 (cost=0.43..79609.82 rows=2288093 width=8) (actual time=0.025..589.773 rows=2288060 loops=1)
Heap Fetches: 0
Buffers: shared hit=11270
-> Sort (cost=30087.94..30769.97 rows=272813 width=8) (actual time=134.076..219.529 rows=303364 loops=1)
Sort Key: t_1.team_id
Sort Method: quicksort Memory: 24720kB
-> CTE Scan on t t_1 (cost=0.00..5456.26 rows=272813 width=8) (actual time=0.003..60.179 rows=265201 loops=1)
Planning time: 0.401 ms
Execution time: 3548.072 ms
Your query and explain output don't look so bad. Still, a couple of observations:
An index-only scan instead of an index scan on matches_loosing_champion_ids_index would be faster. The reason you don't see that: the useless count(match_id).
5 bitmap index scans + BitmapOR step are pretty fast but a single bitmap index scan would be faster.
The most expensive part in this query plan is the Nested Loop Left Join. Might be different for other players.
With your schema
Query 1: LEFT JOIN LATERAL
This way, we aggregate before we join and don't need another GROUP BY. Also fewer join operations. And count(*) should unblock index-only scans:
SELECT player1, player2, player3, player4, player5
, ((win * 1.0) / (win + loss))::numeric(5,5) AS winrate
, win + loss AS matches
FROM (
SELECT winning_champion_one_id AS player1
, winning_champion_two_id AS player2
, winning_champion_three_id AS player3
, winning_champion_four_id AS player4
, winning_champion_five_id AS player5
, COUNT(*) AS win -- see below
FROM matches
WHERE 157 IN (winning_champion_one_id
, winning_champion_two_id
, winning_champion_three_id
, winning_champion_four_id
, winning_champion_five_id)
GROUP BY 1,2,3,4,5
) w
LEFT JOIN LATERAL (
SELECT COUNT(*) AS loss -- see below
FROM matches
WHERE loosing_champion_one_id = w.player1
AND loosing_champion_two_id = w.player2
AND loosing_champion_three_id = w.player3
AND loosing_champion_four_id = w.player4
AND loosing_champion_five_id = w.player5
GROUP BY loosing_champion_one_id
, loosing_champion_two_id
, loosing_champion_three_id
, loosing_champion_four_id
, loosing_champion_five_id
) l ON true
WHERE win + loss > 19
ORDER BY winrate DESC, matches DESC
LIMIT 1;
count(*):
is slightly shorter and faster in Postgres, doing the same as count(match_id) here, because match_id is never NULL.
Removing the only reference to match_id allows an index-only scan on matches_loosing_champion_ids_index! Some other preconditions must be met ...
Query 2: UNION ALL
Another way around the expensive Nested Loop Left Join, and a single GROUP BY. But we add 5 more bitmap index scans. May or may not be faster:
SELECT player1, player2, player3, player4, player5
,(count(win) * 1.0) / count(*) AS winrate -- I would round ...
, count(*) AS matches
FROM (
SELECT winning_champion_one_id AS player1
, winning_champion_two_id AS player2
, winning_champion_three_id AS player3
, winning_champion_four_id AS player4
, winning_champion_five_id AS player5
, TRUE AS win
FROM matches
WHERE 157 IN (winning_champion_one_id
, winning_champion_two_id
, winning_champion_three_id
, winning_champion_four_id
, winning_champion_five_id)
UNION ALL
SELECT loosing_champion_one_id
, loosing_champion_two_id
, loosing_champion_three_id
, loosing_champion_four_id
, loosing_champion_five_id
, NULL AS win -- following "count(win)" ignores NULL values
FROM matches
WHERE 157 IN (loosing_champion_one_id
, loosing_champion_two_id
, loosing_champion_three_id
, loosing_champion_four_id
, loosing_champion_five_id)
) m
GROUP BY 1,2,3,4,5
HAVING count(*) > 19 -- min 20 matches
-- AND count(win) > 0 -- min 1 win -- see below!
ORDER BY winrate DESC, matches DESC
LIMIT 1;
AND count(win) > 0 is commented out, because it's redundant, while you pick the single best winrate anyways.
Different schema
I really would use a different schema to begin with:
CREATE TABLE team (
team_id serial PRIMARY KEY -- or bigserial if you expect > 2^31 distinct teams
-- more attributes?
);
CREATE TABLE player (
player_id smallserial PRIMARY KEY
-- more attributes?
);
CREATE TABLE team_member (
team_id integer REFERENCES team
, player_id smallint REFERENCES player
, team_pos smallint NOT NULL CHECK (team_pos BETWEEN 1 AND 5) -- only if position matters
, PRIMARY KEY (team_id, player_id)
, UNIQUE (team_id, team_pos)
);
CREATE INDEX team_member_player_team_idx on team_member (player_id, team_id);
-- Enforce 5 players per team. Various options, different question.
CREATE TABLE match (
match_id bigserial PRIMARY KEY
, winner integer NOT NULL REFERENCES team
, loser integer NOT NULL REFERENCES team
, CHECK (winner <> loser) -- wouldn't make sense
);
CREATE INDEX match_winner_loser_idx ON match (winner, loser);
CREATE INDEX match_loser_winner_idx ON match (loser, winner);
Subsidiary tables add to the disk footprint, but the main table is a bit smaller. And most importantly, you need fewer indexes, which should be substantially smaller overall for your cardinalities.
Query
We don't need any other indexes for this equivalent query. Much simpler and presumably faster now:
SELECT winner
, win * 1.0/ (win + loss) AS winrate
, win + loss AS matches
FROM (
SELECT w.winner, count(*) AS win
FROM team_member tm
JOIN match w ON w.winner = tm.team_id
WHERE tm.player_id = 157
GROUP BY w.winner
) w
LEFT JOIN LATERAL (
SELECT count(*) AS loss
FROM match l
WHERE l.loser = w.winner
GROUP BY l.loser
) l ON true
WHERE win + loss > 19
ORDER BY winrate DESC, matches DESC
LIMIT 1;
Join the result to team_member to get individual players.
You can also try the corresponding UNION ALL technique from above:
WITH t AS (
SELECT team_id
FROM team_member
WHERE player_id = 157 -- provide player here
)
SELECT team_id
,(count(win) * 1.0) / count(*) AS winrate
, count(*) AS matches
FROM (
SELECT t.team_id, TRUE AS win
FROM t JOIN match ON winner = t.team_id
UNION ALL
SELECT t.team_id, NULL AS win
FROM t JOIN match ON loser = t.team_id
) m
GROUP BY 1
HAVING count(*) > 19 -- min 20 matches
ORDER BY winrate DESC, matches DESC
LIMIT 1;
bloom index
I briefly considered a bloom index for your predicate:
WHERE 157 IN (loosing_champion_one_id
, loosing_champion_two_id
, loosing_champion_three_id
, loosing_champion_four_id
, loosing_champion_five_id)
Didn't test, probably won't pay for just 5 smallint columns.
The execution plan looks pretty good in my opinion.
What you could try is to see if performance improves when a nested loop join is avoided.
To test this, run
SET enable_nestloop = off;
before your query and see if that improves the speed.
Other than that, I cannot think of any improvements.
I have heard PostgreSQL being used in cases where the tables run into billion of rows and having satisfactory response times too. But here's my simple experiment to check this out. I have a table with 6 columns and has 4255700 entries exactly. I have used pgtune to tune the configuration according to my setup. Now, when I run a simple "*select * from tab1*", it is taking 173.425 seconds to fetch all the rows. Is this the normal behavior? I have this single table in the DB.
The table definition is as follows -
CREATE TABLE file_group_permissions
(
fgp_id serial NOT NULL,
file_id integer NOT NULL,
pg_id integer NOT NULL,
policy_id integer,
tag_id integer,
inst_id integer,
CONSTRAINT file_group_permissions_pkey PRIMARY KEY (fgp_id)
)
WITH (
OIDS=FALSE
);
ALTER TABLE file_group_permissions
OWNER TO sa;
-- Index: fgp_file_idx
-- DROP INDEX fgp_file_idx;
CREATE INDEX fgp_file_idx
ON file_group_permissions
USING btree
(file_id);
ALTER TABLE file_group_permissions CLUSTER ON fgp_file_idx;
-- Index: fgp_inst_idx
-- DROP INDEX fgp_inst_idx;
CREATE INDEX fgp_inst_idx
ON file_group_permissions
USING btree
(inst_id);
-- Index: fgp_tag_idx
-- DROP INDEX fgp_tag_idx;
CREATE INDEX fgp_tag_idx
ON file_group_permissions
USING btree
(tag_id);
-- Index: pgfgp_idx
-- DROP INDEX pgfgp_idx;
CREATE INDEX pgfgp_idx
ON file_group_permissions
USING btree
(pg_id);
Output of EXPLAIN (ANALYZE, BUFFERS) select * from file_group_permissions -
"Seq Scan on file_group_permissions (cost=0.00..69662.00 rows=4255700
width=24) (actual time=0.019..580.273 rows=4255700 loops=1)"
" Buffers: shared hit=2432 read=24673"
"Planning time: 0.070 ms"
"Execution time: 903.325 ms"
I have a MacBook Pro with 16 Gigs of RAM and 512 Gigs of SSD. I have configured the PostgreSQL to use 2Gigs of RAM.
Edit
EXPLAIN (ANALYZE, BUFFERS) select pg_id, count(distinct file_id) from file_group_permissions where pg_id in (6117,6115,6116,6113,6114) group by 1;
"GroupAggregate (cost=0.44..102028.21 rows=208 width=8) (actual time=4970.884..5013.423 rows=3 loops=1)"
" Group Key: pg_id"
" Buffers: shared hit=50891, temp read=4824 written=4824"
" -> Index Scan using pgfgp_idx on file_group_permissions (cost=0.44..85511.31 rows=3302964 width=8) (actual time=0.062..1080.926 rows=3323389 loops=1)"
" Index Cond: (pg_id = ANY('{6117,6115,6116,6113,6114}'::integer[]))"
" Buffers: shared hit=50891"
"Planning time: 0.219 ms"
"Execution time: 5013.495 ms"
EDIT1
I separated this table into a new DB and followed the suggestions (composite indices and postgresql conf), here's the new plan -
"GroupAggregate (cost=478307.10..502996.67 rows=209 width=8) (actual
time=7500.426..7528.021 rows=3 loops=1)"
" Group Key: pg_id"
" Buffers: shared read=27105, temp read=12137 written=12137"
" -> Sort (cost=478307.10..486536.26 rows=3291664 width=8) (actual
time=2944.597..3647.248 rows=3323389 loops=1)"
" Sort Key: pg_id"
" Sort Method: external sort Disk: 58488kB"
" Buffers: shared read=27105, temp read=7311 written=7311"
" -> Seq Scan on file_group_permissions (cost=0.00..96260.12
rows=3291664 width=8) (actual time=0.016..1516.743 rows=3323389 loops=1)"
" Filter: (pg_id = ANY
('{6117,6115,6116,6113,6114}'::integer[]))"
" Rows Removed by Filter: 932311"
" Buffers: shared read=27105"
"Planning time: 0.514 ms"
"Execution time: 7540.243 ms"
This table is simply jinxed and it is hurting the performance everywhere it is being joined.
I have created a test table, just like yours, and I have populated it with exactly same amount of records:
insert into file_group_permissions (file_id,pg_id,policy_id,tag_id,inst_id)
select
trunc(random()*10000) as file_id,
trunc(random()*10000) as pg_id,
trunc(random()*10000) as policy_id,
trunc(random()*10000) as tag_id,
trunc(random()*10000) as inst_id
from generate_series(1,4255700) g
When I run your query, it executes pretty fast:
EXPLAIN (ANALYZE, BUFFERS)
select pg_id, count(distinct file_id)
from file_group_permissions
where pg_id in (6117,6115,6116,6113,6114) group by 1;
GroupAggregate (cost=0.43..8.32 rows=5 width=8) (actual time=0.339..1.608 rows=5 loops=1)
Buffers: shared hit=2158
-> Index Scan using pgfgp_idx on file_group_permissions (cost=0.43..8.24 rows=5 width=8) (actual time=0.018..1.170 rows=2147 loops=1)
Index Cond: (pg_id = ANY ('{6117,6115,6116,6113,6114}'::integer[]))
Buffers: shared hit=2158
Total runtime: 1.633 ms
I have notice this line in your execution plan:
Buffers: shared hit=50891, temp read=4824 written=4824
temp read=4824 written=4824 tells us that the database is "using" disk somehow to perform the scan operation. Perhaps you have to tweak some other postgresql.conf parameters, like these of mine:
shared_buffers = 1GB
temp_buffers = 32MB
work_mem = 32MB
effective_cache_size = 1GB