In our time dimension in data warehouse, we have a lot of columns with boolean flags, for example:
is_ytd (is year to date)
is_mtd (is month to date)
is_current_date
is_current_month
is_current_year
Would it be a good indexing strategy to create partial index on all such columns? Something like:
CREATE INDEX tdim_is_current_month
ON calendar (is_current_month)
WHERE is_current_month;
Our time dimension has 136 columns, 7 000 rows, 53 columns with boolean indicator.
Why we use flags instead of deriving desired date range from current_date?
Make life easier
Enforce consistency
Speed-up queries
Provide not-so-easy-to-derive indicators
Make use of other tools easier
Ad1)
Once you join time dimension (and this is almost always when analyzing any fact table in data warehouse), it is much easier to write where is_current_year instead of where extract(year from time_date) = extract(year from current_date)
Ad2)
Example: It sounds simple to figure out what year to date (YTD) is. We can start with: time_date between date_trunc('year', current_date) and current_date. But some people would actually exclude current_date (this make sense, because today is not finished). In such case we would use: time_date between date_trunc('year', current_date) and (current_date - 1). And going further - what would happen if for some reason DW is not updated for couple of days. Maybe then you would like YTD linked to day where you have last completed data from all source systems. When you have common definition of what YTD means than you reduce risk of different meanings.
Ad 3)
I think that it should be faster to filter data based on indexed boolean flag in column than filter based on on-the-fly calculated expression.
Ad 4)
Some flags are not so easy to create - for example we do have flags is_first_workday_in_month, is_last_workday_in_month.
Ad 5)
In some tools it is easier to use existing columns than SQL expressions. For example when creating dimensions for OLAP cube is is much easier to add table column as level of hierarchy than constructing such level with SQL expression.
Testing indexes for boolean flags
I tested all indexed flags and run explain analyze for simmple query with one fact table and time dimension (named calendar):
select count(*) from fact_table join calendar using(time_key)
For most flags I get Index scan:
"Aggregate (cost=4022.80..4022.81 rows=1 width=0) (actual time=38.642..38.642 rows=1 loops=1)"
" -> Hash Join (cost=13.12..4019.73 rows=1230 width=0) (actual time=38.640..38.640 rows=0 loops=1)"
" Hash Cond: (fact_table.time_key = calendar.time_key)"
" -> Seq Scan on fact_table (cost=0.00..3249.95 rows=198495 width=2) (actual time=0.006..17.769 rows=198495 loops=1)"
" -> Hash (cost=12.58..12.58 rows=43 width=2) (actual time=0.054..0.054 rows=43 loops=1)"
" Buckets: 1024 Batches: 1 Memory Usage: 2kB"
" -> Index Scan using cal_is_qtd on calendar (cost=0.00..12.58 rows=43 width=2) (actual time=0.014..0.049 rows=43 loops=1)"
" Index Cond: (is_qtd = true)"
"Total runtime: 38.679 ms"
For some flags I get bitmap heap scan combined with bitmap index scan:
"Aggregate (cost=13341.07..13341.08 rows=1 width=0) (actual time=100.972..100.973 rows=1 loops=1)"
" -> Hash Join (cost=6656.54..13001.52 rows=135820 width=0) (actual time=5.729..86.972 rows=198495 loops=1)"
" Hash Cond: (fact_table.time_key = calendar.time_key)"
" -> Seq Scan on fact_table (cost=0.00..3249.95 rows=198495 width=2) (actual time=0.012..22.667 rows=198495 loops=1)"
" -> Hash (cost=6597.19..6597.19 rows=4748 width=2) (actual time=5.706..5.706 rows=4748 loops=1)"
" Buckets: 1024 Batches: 1 Memory Usage: 158kB"
" -> Bitmap Heap Scan on calendar (cost=97.05..6597.19 rows=4748 width=2) (actual time=0.440..4.971 rows=4748 loops=1)"
" Filter: is_past_quarter"
" -> Bitmap Index Scan on cal_is_past_quarter (cost=0.00..95.86 rows=3249 width=0) (actual time=0.395..0.395 rows=4748 loops=1)"
" Index Cond: (is_past_quarter = true)"
"Total runtime: 101.013 ms"
Only for two flags I get seq scan:
"Aggregate (cost=17195.33..17195.34 rows=1 width=0) (actual time=122.108..122.108 rows=1 loops=1)"
" -> Hash Join (cost=9231.13..16699.10 rows=198495 width=0) (actual time=23.960..108.018 rows=198495 loops=1)"
" Hash Cond: (fact_table.time_key = calendar.time_key)"
" -> Seq Scan on fact_table (cost=0.00..3249.95 rows=198495 width=2) (actual time=0.012..22.153 rows=198495 loops=1)"
" -> Hash (cost=9144.39..9144.39 rows=6939 width=2) (actual time=23.935..23.935 rows=6939 loops=1)"
" Buckets: 1024 Batches: 1 Memory Usage: 231kB"
" -> Seq Scan on calendar (cost=0.00..9144.39 rows=6939 width=2) (actual time=17.427..22.908 rows=6939 loops=1)"
" Filter: is_eoq"
"Total runtime: 122.138 ms"
If is_current_month = true represents more than a few percent of the rows then the index will not be used. 7,000 rows is too few to even bother.
Maybe this is more of a comment than an answer....
Given that the query planner/optimizer gets the cardinalities and join type correct, the execution time of any query involving a join between your fact table and your time dimension will be determined by the size of the fact table.
Your time dimension will either be in cache all the time or fully read in a few ms. You will have bigger variations than that depending on the current load! The rest of the execution time does not have to do with the time dimension.
Having said that, I'm all for using every trick in the bag to help the query planner/optimizer come up with good enough estimates. Sometimes this means creating or disabling constraints and creating unnecessary indexes.
Related
I have product_details table with 30+ Million records. product attributes text type data is stored into column Value1.
Front end(web) users search for product details and it will be queried on column Value1.
create table product_details(
key serial primary key ,
product_key int,
attribute_key int ,
Value1 text[],
Value2 int[],
status text);
I created gin index on column Value1 to improve search query performance.
query execution improved a lot for many queries.
Tables and indexes are here
Below is one of query used by application for search.
select p.key from (select x.product_key,
x.value1,
x.attribute_key,
x.status
from product_details x
where value1 IS NOT NULL
) as pr_d
join attribute_type at on at.key = pr_d.attribute_key
join product p on p.key = pr_d.product_key
where value1_search(pr_d.value1) ilike '%B s%'
and at.type = 'text'
and at.status = 'active'
and pr_d.status = 'active'
and 1 = 1
and p.product_type_key=1
and 1 = 1
group by p.key
query is executed in 2 or 3 secs if we search %B % or any single or two char words and below is query plan
Group (cost=180302.82..180302.83 rows=1 width=4) (actual time=49.006..49.021 rows=65 loops=1)
Group Key: p.key
-> Sort (cost=180302.82..180302.83 rows=1 width=4) (actual time=49.005..49.009 rows=69 loops=1)
Sort Key: p.key
Sort Method: quicksort Memory: 28kB
-> Nested Loop (cost=0.99..180302.81 rows=1 width=4) (actual time=3.491..48.965 rows=69 loops=1)
Join Filter: (x.attribute_key = at.key)
Rows Removed by Join Filter: 10051
-> Nested Loop (cost=0.99..180270.15 rows=1 width=8) (actual time=3.396..45.211 rows=69 loops=1)
-> Index Scan using products_product_type_key_status on product p (cost=0.43..4420.58 rows=1413 width=4) (actual time=0.024..1.473 rows=1630 loops=1)
Index Cond: (product_type_key = 1)
-> Index Scan using product_details_product_attribute_key_status on product_details x (cost=0.56..124.44 rows=1 width=8) (actual time=0.026..0.027 rows=0 loops=1630)
Index Cond: ((product_key = p.key) AND (status = 'active'))
Filter: ((value1 IS NOT NULL) AND (value1_search(value1) ~~* '%B %'::text))
Rows Removed by Filter: 14
-> Seq Scan on attribute_type at (cost=0.00..29.35 rows=265 width=4) (actual time=0.002..0.043 rows=147 loops=69)
Filter: ((value_type = 'text') AND (status = 'active'))
Rows Removed by Filter: 115
Planning Time: 0.732 ms
Execution Time: 49.089 ms
But if i search for %B s%, query took 75 secs and below is query plan (second time query execution took 63 sec)
In below query plan, DB engine didn't consider index for scan as in above query plan indexes were used. Not sure why ?
Group (cost=8057.69..8057.70 rows=1 width=4) (actual time=62138.730..62138.737 rows=12 loops=1)
Group Key: p.key
-> Sort (cost=8057.69..8057.70 rows=1 width=4) (actual time=62138.728..62138.732 rows=14 loops=1)
Sort Key: p.key
Sort Method: quicksort Memory: 25kB
-> Nested Loop (cost=389.58..8057.68 rows=1 width=4) (actual time=2592.685..62138.710 rows=14 loops=1)
-> Hash Join (cost=389.15..4971.85 rows=368 width=4) (actual time=298.280..62129.956 rows=831 loops=1)
Hash Cond: (x.attribute_type = at.key)
-> Bitmap Heap Scan on product_details x (cost=356.48..4937.39 rows=681 width=8) (actual time=298.117..62128.452 rows=831 loops=1)
Recheck Cond: (value1_search(value1) ~~* '%B s%'::text)
Rows Removed by Index Recheck: 26168889
Filter: ((value1 IS NOT NULL) AND (status = 'active'))
Rows Removed by Filter: 22
Heap Blocks: exact=490 lossy=527123
-> Bitmap Index Scan on product_details_value1_gin (cost=0.00..356.31 rows=1109 width=0) (actual time=251.596..251.596 rows=2846970 loops=1)
Index Cond: (value1_search(value1) ~~* '%B s%'::text)
-> Hash (cost=29.35..29.35 rows=265 width=4) (actual time=0.152..0.153 rows=269 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 18kB
-> Seq Scan on attribute_type at (cost=0.00..29.35 rows=265 width=4) (actual time=0.010..0.122 rows=269 loops=1)
Filter: ((value_type = 'text') AND (status = 'active'))
Rows Removed by Filter: 221
-> Index Scan using product_pkey on product p (cost=0.43..8.39 rows=1 width=4) (actual time=0.009..0.009 rows=0 loops=831)
Index Cond: (key = x.product_key)
Filter: (product_type_key = 1)
Rows Removed by Filter: 1
Planning Time: 0.668 ms
Execution Time: 62138.794 ms
Any suggestions pls to improve query for search %B s%
thanks
ilike '%B %' has no usable trigrams in it. The planner knows this, and punishes the pg_trgm index plan so much that the planner then goes with an entirely different plan instead.
But ilike '%B s%' does have one usable trigram in it, ' s'. It turns out that this trigram sucks because it is extremely common in the searched data, but the planner currently has no way to accurately estimate how much it sucks.
Even worse, this large number matches means your full bitmap can't fit in work_mem so it goes lossy. Then it needs to recheck all the tuples in any page which contains even one tuple that has the ' s' trigram in it, which looks like it is most of the pages in your table.
The first thing to do is to increase your work_mem to the point you stop getting lossy blocks. If most of your time is spent in the CPU applying the recheck condition, this should help tremendously. If most of your time is spent reading the product_details from disk (so that the recheck has the data it needs to run) then it won't help much. If you had done EXPLAIN (ANALYZE, BUFFERS) with track_io_timing turned on, then we would already know which is which.
Another thing you could do is have the application inspect the search parameter, and if it looks like two letters (with or without a space between), then forcibly disable that index usage, or just throw an error if there is no good reason to do that type of search. For example, changing the part of the query to look like this will disable the index:
where value1_search(pr_d.value1)||'' ilike '%B s%'
Another thing would be to rethink your data representation. '%B s%' is a peculiar thing to search for. Why would anyone search for that? Does it have some special meaning within the context of your data, which is not obvious to the outside observer? Maybe you could represent it in a different way that gets along better with pg_trgm.
Finally, you could try to improve the planning for GIN indexes generally by explicitly estimating how many tuples are going to fail recheck (due to inherent lossiness of the index, not due to overrunning work_mem). This would be a major undertaking, and you would be unlikely to see it in production for at least a couple years, if ever.
I am working on calculating the overlap of two tables/layers in a PostGIS database. One set is a grid of hexagons for which I want to calculate the fraction of overlap with another set of polygons, for each of the hexagons. The multipolygon set also has a few polygons that overlap, so I need to dissolve/union those. Before I was doing this in FME, but it kept running out of memory for some of the larger polygons. That's why I want to do this in the database (and PostGIS should be very much capable of doing that).
Here is what I have so far, and it works, and memory is not running out now:
EXPLAIN ANALYZE
WITH rh_union AS (
SELECT (ST_Dump(ST_Union(rh.geometry))).geom AS geometry
FROM relevant_habitats rh
WHERE rh.assessment_area_id=1
)
SELECT h.receptor_id,
SUM(ROUND((ST_Area(ST_Intersection(rhu.geometry, h.geometry)) / ST_Area(h.geometry))::numeric,3)) AS frac_overlap
FROM rh_union rhu, hexagons h
WHERE h.zoom_level=1 AND ST_Intersects(rhu.geometry, h.geometry)
GROUP BY h.receptor_id
So I first break the multipolygon and union what I can. Then calculate the overlay of the hexagons with the polygons. Then calculate the (sum of all small pieces of) area.
Now, my question is:
is this an efficient way of doing this? Or would there be a better way?
(And a side question: is it correct to first ST_Union and then ST_Dump?)
--Update with EXPLAIN ANALYZE
Output for a single area:
"QUERY PLAN"
"GroupAggregate (cost=1996736.77..15410052.20 rows=390140 width=36) (actual time=571.303..1137.657 rows=685 loops=1)"
" Group Key: h.receptor_id"
" -> Sort (cost=1996736.77..1998063.55 rows=530712 width=188) (actual time=571.090..620.379 rows=806 loops=1)"
" Sort Key: h.receptor_id"
" Sort Method: external merge Disk: 42152kB"
" -> Nested Loop (cost=55.53..1848314.51 rows=530712 width=188) (actual time=382.769..424.643 rows=806 loops=1)"
" -> Result (cost=55.25..1321.51 rows=1000 width=32) (actual time=382.550..383.696 rows=65 loops=1)"
" -> ProjectSet (cost=55.25..61.51 rows=1000 width=32) (actual time=382.544..383.652 rows=65 loops=1)"
" -> Aggregate (cost=55.25..55.26 rows=1 width=32) (actual time=381.323..381.325 rows=1 loops=1)"
" -> Index Scan using relevant_habitats_pkey on relevant_habitats rh (cost=0.28..28.75 rows=12 width=130244) (actual time=0.026..0.048 rows=12 loops=1)"
" Index Cond: (assessment_area_id = 94)"
" -> Index Scan using idx_hexagons_geometry_gist on hexagons h (cost=0.29..1846.45 rows=53 width=156) (actual time=0.315..0.624 rows=12 loops=65)"
" Index Cond: (geometry && (((st_dump((st_union(rh.geometry))))).geom))"
" Filter: (((zoom_level)::integer = 1) AND st_intersects((((st_dump((st_union(rh.geometry))))).geom), geometry))"
" Rows Removed by Filter: 19"
"Planning Time: 0.390 ms"
"Execution Time: 1372.443 ms"
Update 2: now the output of EXPLAIN ANALYZE on the second select (SELECT h.receptor_id~) and the CTE replaced by a (temp) table:
"QUERY PLAN"
"GroupAggregate (cost=2691484.47..20931829.74 rows=390140 width=36) (actual time=29.455..927.945 rows=685 loops=1)"
" Group Key: h.receptor_id"
" -> Sort (cost=2691484.47..2693288.89 rows=721768 width=188) (actual time=28.382..31.514 rows=806 loops=1)"
" Sort Key: h.receptor_id"
" Sort Method: quicksort Memory: 336kB"
" -> Nested Loop (cost=0.29..2488035.20 rows=721768 width=188) (actual time=0.189..27.852 rows=806 loops=1)"
" -> Seq Scan on rh_union rhu (cost=0.00..23.60 rows=1360 width=32) (actual time=0.016..0.050 rows=65 loops=1)"
" -> Index Scan using idx_hexagons_geometry_gist on hexagons h (cost=0.29..1828.89 rows=53 width=156) (actual time=0.258..0.398 rows=12 loops=65)"
" Index Cond: (geometry && rhu.geometry)"
" Filter: (((zoom_level)::integer = 1) AND st_intersects(rhu.geometry, geometry))"
" Rows Removed by Filter: 19"
"Planning Time: 0.481 ms"
"Execution Time: 928.583 ms"
You want a metric describing the extent of overlap of a table of polygons on other polygons on another table.
This query returns the id of overlapping polygons within the same table and an indicator as a percentage.
SELECT
CONCAT(a.polyid,' ',b.polyid) AS intersecting_polys,
CONCAT(a.attribute_x,' ',b.attribute_x) AS attribute,
ROUND((100 * (ST_Area(ST_Intersection(a.geom, b.geom))) / ST_Area(a.geom))::NUMERIC,0) AS pc_overlap
FROM your_schema.your_table AS a
LEFT JOIN your_schema.your_table AS b
ON (a.geom && b.geom AND ST_Relate(a.geom, b.geom, '2********'))
WHERE a.polyid != b.polyid
;
Note the term
ON (a.geom && b.geom AND ST_Relate(a.geom, b.geom, '2********')) is used instead of ST_Covers. You might want to experiment which is correct in your use case.
I have a query that is being ran on PGSQL, and when queried at a fast rate for large data sets, it is taking a long time to run because it isn't making use of the available indexes. I found that changing the filter from multiple OR's to an IN clause causes the right index to be used. Is there a way I can force the index to be used even when using OR's?
Query with Disjunction:
SELECT field1, field2,..., fieldN
FROM table1 WHERE
((((field9='val1' OR field9='val2') OR field9='val3') OR field9='val4')
AND (field6='val5'));
Query Plan:
"Bitmap Heap Scan on table1 (cost=18.85..19.88 rows=1 width=395) (actual time=0.017..0.017 rows=0 loops=1)"
" Recheck Cond: (((field6)::text = 'val5'::text) AND (((field9)::text = 'val1'::text) OR ((field9)::text = 'val2'::text) OR ((field9)::text = 'val3'::text) OR ((field9)::text = 'val4'::text)))"
" -> BitmapAnd (cost=18.85..18.85 rows=1 width=0) (actual time=0.016..0.016 rows=0 loops=1)"
" -> Bitmap Index Scan on idx_field6_field9 (cost=0.00..9.01 rows=611 width=0) (actual time=0.015..0.015 rows=0 loops=1)"
" Index Cond: ((field6)::text = 'val5'::text)"
" -> BitmapOr (cost=9.59..9.59 rows=516 width=0) (never executed)"
" -> Bitmap Index Scan on idx_id_field9 (cost=0.00..2.40 rows=129 width=0) (never executed)"
" Index Cond: ((field9)::text = 'val1'::text)"
" -> Bitmap Index Scan on idx_id_field9 (cost=0.00..2.40 rows=129 width=0) (never executed)"
" Index Cond: ((field9)::text = 'val2'::text)"
" -> Bitmap Index Scan on idx_id_field9 (cost=0.00..2.40 rows=129 width=0) (never executed)"
" Index Cond: ((field9)::text = 'val3'::text)"
" -> Bitmap Index Scan on idx_id_field9 (cost=0.00..2.40 rows=129 width=0) (never executed)"
" Index Cond: ((field9)::text = 'val4'::text)"
"Planning time: 0.177 ms"
"Execution time: 0.061 ms"
Query with IN
SELECT field1, field2,..., fieldN
FROM table1
WHERE
((field9 IN ('val1', 'val2', 'val3', 'val4'))
AND (field6='val5'));
Query Plan:
"Index Scan using idx_field6_field9 on table1 (cost=0.43..6.77 rows=1 width=395) (actual time=0.032..0.032 rows=0 loops=1)"
" Index Cond: (((field6)::text = 'val5'::text) AND ((field9)::text = ANY ('{val1,val2,val3,val4}'::text[])))"
"Planning time: 0.145 ms"
"Execution time: 0.055 ms"
There is an index on field 6 and field 9 which the second query uses as expected, which the first one also should. Field9 is also kind of like a state field, so its cardinality is extremely low - there's only like 9 different values across the whole table. Unfortunately, it isn't straightforward to change the query to use an IN clause, so getting PG to use the right plan would be ideal.
There is no way you can get the fast plan (single index scan) using the OR condition. You'll have to rewrite the query.
You want to know why, which is always difficult to explain. With optimizations like that, there are usually two reasons:
Nobody got around to do it.
This requires extra effort every time a query with an OR is planned:
Are there several conditions linked with OR that have the same expression on one side?
Both plans, the original and the rewritten one, would have to be estimated. It may well be that the BitmapOr is the most efficient way to process the query.
This price would have to be paid by every query with OR in it.
I am not saying that it is a bad idea to add an optimization like this, but there are two sides to the coin.
Please, observe:
(Forgot to add order, the plan is updated)
The query:
EXPLAIN ANALYZE
SELECT DISTINCT(id), special, customer, business_no, bill_to_name, bill_to_address1, bill_to_address2, bill_to_postal_code, ship_to_name, ship_to_address1, ship_to_address2, ship_to_postal_code,
purchase_order_no, ship_date::text, calc_discount_text(o) AS discount, discount_absolute, delivery, hst_percents, sub_total, total_before_hst, hst, total, total_discount, terms, rep, ship_via,
item_count, version, to_char(modified, 'YYYY-MM-DD HH24:MI:SS') AS "modified", to_char(created, 'YYYY-MM-DD HH24:MI:SS') AS "created"
FROM invoices o
LEFT JOIN reps ON reps.rep_id = o.rep_id
LEFT JOIN terms ON terms.terms_id = o.terms_id
LEFT JOIN shipVia ON shipVia.ship_via_id = o.ship_via_id
JOIN invoiceItems items ON items.invoice_id = o.id
WHERE items.qty < 5
ORDER BY modified
LIMIT 100
The result:
Limit (cost=2931740.10..2931747.85 rows=100 width=635) (actual time=414307.004..414387.899 rows=100 loops=1)
-> Unique (cost=2931740.10..3076319.37 rows=1865539 width=635) (actual time=414307.001..414387.690 rows=100 loops=1)
-> Sort (cost=2931740.10..2936403.95 rows=1865539 width=635) (actual time=414307.000..414325.058 rows=2956 loops=1)
Sort Key: (to_char(o.modified, 'YYYY-MM-DD HH24:MI:SS'::text)), o.id, o.special, o.customer, o.business_no, o.bill_to_name, o.bill_to_address1, o.bill_to_address2, o.bill_to_postal_code, o.ship_to_name, o.ship_to_address1, o.ship_to_address2, (...)
Sort Method: external merge Disk: 537240kB
-> Hash Join (cost=11579.63..620479.38 rows=1865539 width=635) (actual time=1535.805..131378.864 rows=1872673 loops=1)
Hash Cond: (items.invoice_id = o.id)
-> Seq Scan on invoiceitems items (cost=0.00..78363.45 rows=1865539 width=4) (actual time=0.110..4591.117 rows=1872673 loops=1)
Filter: (qty < 5)
Rows Removed by Filter: 1405763
-> Hash (cost=5498.18..5498.18 rows=64996 width=635) (actual time=1530.786..1530.786 rows=64996 loops=1)
Buckets: 1024 Batches: 64 Memory Usage: 598kB
-> Hash Left Join (cost=113.02..5498.18 rows=64996 width=635) (actual time=0.214..1043.207 rows=64996 loops=1)
Hash Cond: (o.ship_via_id = shipvia.ship_via_id)
-> Hash Left Join (cost=75.35..4566.81 rows=64996 width=607) (actual time=0.154..754.957 rows=64996 loops=1)
Hash Cond: (o.terms_id = terms.terms_id)
-> Hash Left Join (cost=37.67..3800.33 rows=64996 width=579) (actual time=0.071..506.145 rows=64996 loops=1)
Hash Cond: (o.rep_id = reps.rep_id)
-> Seq Scan on invoices o (cost=0.00..2868.96 rows=64996 width=551) (actual time=0.010..235.977 rows=64996 loops=1)
-> Hash (cost=22.30..22.30 rows=1230 width=36) (actual time=0.044..0.044 rows=4 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 1kB
-> Seq Scan on reps (cost=0.00..22.30 rows=1230 width=36) (actual time=0.027..0.032 rows=4 loops=1)
-> Hash (cost=22.30..22.30 rows=1230 width=36) (actual time=0.067..0.067 rows=3 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 1kB
-> Seq Scan on terms (cost=0.00..22.30 rows=1230 width=36) (actual time=0.001..0.007 rows=3 loops=1)
-> Hash (cost=22.30..22.30 rows=1230 width=36) (actual time=0.043..0.043 rows=4 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 1kB
-> Seq Scan on shipvia (cost=0.00..22.30 rows=1230 width=36) (actual time=0.027..0.032 rows=4 loops=1)
Total runtime: 414488.582 ms
This is, obviously, awful. I am pretty new to interpreting query plans and would like to know how to extract the useful performance improvement hints from such a plan.
EDIT 1
Two kinds of entities are involved in this query - invoices and invoice items having the 1-many relationship.
An invoice item specifies the quantity of it within the parent invoice.
The given query returns 100 invoices which have at least one item with the quantity of less than 5.
That should explain why I need DISTINCT - an invoice may have several items satisfying the filter, but I do not want that same invoice returned multiple times. Hence the usage of DISTINCT. However, I am perfectly aware that there may be better means to accomplish the same semantics than using DISTINCT - I am more than willing to learn about them.
EDIT 2
Please, find below the indexes on the invoiceItems table at the time of the query:
CREATE INDEX invoiceitems_invoice_id_idx ON invoiceitems (invoice_id);
CREATE INDEX invoiceitems_invoice_id_name_index ON invoiceitems (invoice_id, name varchar_pattern_ops);
CREATE INDEX invoiceitems_name_index ON invoiceitems (name varchar_pattern_ops);
CREATE INDEX invoiceitems_qty_index ON invoiceitems (qty);
EDIT 3
The advice given by https://stackoverflow.com/users/808806/yieldsfalsehood as to how eliminate DISTINCT (and why) turns out to be a really good one. Here is the new query:
EXPLAIN ANALYZE
SELECT id, special, customer, business_no, bill_to_name, bill_to_address1, bill_to_address2, bill_to_postal_code, ship_to_name, ship_to_address1, ship_to_address2, ship_to_postal_code,
purchase_order_no, ship_date::text, calc_discount_text(o) AS discount, discount_absolute, delivery, hst_percents, sub_total, total_before_hst, hst, total, total_discount, terms, rep, ship_via,
item_count, version, to_char(modified, 'YYYY-MM-DD HH24:MI:SS') AS "modified", to_char(created, 'YYYY-MM-DD HH24:MI:SS') AS "created"
FROM invoices o
LEFT JOIN reps ON reps.rep_id = o.rep_id
LEFT JOIN terms ON terms.terms_id = o.terms_id
LEFT JOIN shipVia ON shipVia.ship_via_id = o.ship_via_id
WHERE EXISTS (SELECT 1 FROM invoiceItems items WHERE items.invoice_id = id AND items.qty < 5)
ORDER BY modified DESC
LIMIT 100
Here is the new plan:
Limit (cost=64717.14..64717.39 rows=100 width=635) (actual time=7830.347..7830.869 rows=100 loops=1)
-> Sort (cost=64717.14..64827.01 rows=43949 width=635) (actual time=7830.334..7830.568 rows=100 loops=1)
Sort Key: (to_char(o.modified, 'YYYY-MM-DD HH24:MI:SS'::text))
Sort Method: top-N heapsort Memory: 76kB
-> Hash Left Join (cost=113.46..63037.44 rows=43949 width=635) (actual time=2.322..6972.679 rows=64467 loops=1)
Hash Cond: (o.ship_via_id = shipvia.ship_via_id)
-> Hash Left Join (cost=75.78..50968.72 rows=43949 width=607) (actual time=0.650..3809.276 rows=64467 loops=1)
Hash Cond: (o.terms_id = terms.terms_id)
-> Hash Left Join (cost=38.11..50438.25 rows=43949 width=579) (actual time=0.550..3527.558 rows=64467 loops=1)
Hash Cond: (o.rep_id = reps.rep_id)
-> Nested Loop Semi Join (cost=0.43..49796.28 rows=43949 width=551) (actual time=0.015..3200.735 rows=64467 loops=1)
-> Seq Scan on invoices o (cost=0.00..2868.96 rows=64996 width=551) (actual time=0.002..317.954 rows=64996 loops=1)
-> Index Scan using invoiceitems_invoice_id_idx on invoiceitems items (cost=0.43..7.61 rows=42 width=4) (actual time=0.030..0.030 rows=1 loops=64996)
Index Cond: (invoice_id = o.id)
Filter: (qty < 5)
Rows Removed by Filter: 1
-> Hash (cost=22.30..22.30 rows=1230 width=36) (actual time=0.213..0.213 rows=4 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 1kB
-> Seq Scan on reps (cost=0.00..22.30 rows=1230 width=36) (actual time=0.183..0.192 rows=4 loops=1)
-> Hash (cost=22.30..22.30 rows=1230 width=36) (actual time=0.063..0.063 rows=3 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 1kB
-> Seq Scan on terms (cost=0.00..22.30 rows=1230 width=36) (actual time=0.044..0.050 rows=3 loops=1)
-> Hash (cost=22.30..22.30 rows=1230 width=36) (actual time=0.096..0.096 rows=4 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 1kB
-> Seq Scan on shipvia (cost=0.00..22.30 rows=1230 width=36) (actual time=0.071..0.079 rows=4 loops=1)
Total runtime: 7832.750 ms
Is it the best I can count on? I have restarted the server (to clean the database caches) and rerun the query without EXPLAIN ANALYZE. It takes almost 5 seconds. Can it be improved even further? I have 65,000 invoices and 3,278,436 invoice items.
EDIT 4
Found it. I was ordering by a computation result, modified = to_char(modified, 'YYYY-MM-DD HH24:MI:SS'). Adding an index on the modified invoice field and ordering by the field itself brings the result to under 100 ms !
The final plan is:
Limit (cost=1.18..1741.92 rows=100 width=635) (actual time=3.002..27.065 rows=100 loops=1)
-> Nested Loop Left Join (cost=1.18..765042.09 rows=43949 width=635) (actual time=2.989..25.989 rows=100 loops=1)
-> Nested Loop Left Join (cost=1.02..569900.41 rows=43949 width=607) (actual time=0.413..16.863 rows=100 loops=1)
-> Nested Loop Left Join (cost=0.87..386185.48 rows=43949 width=579) (actual time=0.333..15.694 rows=100 loops=1)
-> Nested Loop Semi Join (cost=0.72..202470.54 rows=43949 width=551) (actual time=0.017..13.965 rows=100 loops=1)
-> Index Scan Backward using invoices_modified_index on invoices o (cost=0.29..155543.23 rows=64996 width=551) (actual time=0.003..4.543 rows=100 loops=1)
-> Index Scan using invoiceitems_invoice_id_idx on invoiceitems items (cost=0.43..7.61 rows=42 width=4) (actual time=0.079..0.079 rows=1 loops=100)
Index Cond: (invoice_id = o.id)
Filter: (qty < 5)
Rows Removed by Filter: 1
-> Index Scan using reps_pkey on reps (cost=0.15..4.17 rows=1 width=36) (actual time=0.007..0.008 rows=1 loops=100)
Index Cond: (rep_id = o.rep_id)
-> Index Scan using terms_pkey on terms (cost=0.15..4.17 rows=1 width=36) (actual time=0.003..0.004 rows=1 loops=100)
Index Cond: (terms_id = o.terms_id)
-> Index Scan using shipvia_pkey on shipvia (cost=0.15..4.17 rows=1 width=36) (actual time=0.006..0.008 rows=1 loops=100)
Index Cond: (ship_via_id = o.ship_via_id)
Total runtime: 27.572 ms
It is amazing! Thank you all for the help.
For starters, it's pretty standard to post explain plans to http://explain.depesz.com - that'll add some pretty formatting to it, give you a nice way to distribute the plan, and let you anonymize plans that might contain sensitive data. Even if you're not distributing the plan it makes it a lot easier to understand what's happening and can sometimes illustrate exactly where a bottleneck is.
There are countless resources that cover interpreting the details of postgres explain plans (see https://wiki.postgresql.org/wiki/Using_EXPLAIN). There are a lot of little details that get taken in to account when the database chooses a plan, but there are some general concepts that can make it easier. First, get a grasp of the page-based layout of data and indexes (you don't need to know the details of the page format, just how data and indexes get split in to pages). From there, get a feel for the two basic data access methods - full table scans and index scans - and with a little thought it should start to become clear the different situations where one would be preferred to the other (also keep in mind that an index scan isn't even always possible). At that point you can start looking in to some of the different configuration items that affect plan selection in the context of how they might tip the scale in favor of a table scan or an index scan.
Once you've got that down, move on up the plan and read in to the details of the different nodes you find - in this plan you've got a lot of hash joins, so read up on that to start with. Then, to compare apples to apples, disable hash joins entirely ("set enable_hashjoin = false;") and run your explain analyze again. Now what join method do you see? Read up on that. Compare the estimated cost of that method with the estimated cost of the hash join. Why might they be different? The estimated cost of the second plan will be higher than this first plan (otherwise it would have been preferred in the first place) but what about the real time that it takes to run the second plan? Is it lower or higher?
Finally, to address this plan specifically. With regards to that sort that's taking a long time: distinct is not a function. "DISTINCT(id)" does not say "give me all the rows that are distinct on only the column id", instead it is sorting the rows and taking the unique values based on all columns in the output (i.e. it is equivalent to writing "distinct id ..."). You should probably re-consider if you actually need that distinct in there. Normalization will tend to design away the need for distincts, and while they will occasionally be needed, whether they really are super truly needed is not always true.
You begin by chasing down the node that takes the longest, and start optimizing there. In your case, that appears to be
Seq Scan on invoiceitems items
You should add an index there, and problem also to the other tables.
You could also try increasing work_mem to get rid of the external sort.
When you have done that, the new plan will probably look completely differently, so then start over.
Running a Heroku "Crane" PostgreSQL instance (Version 9.1.6)
I have a table with sales points; currency amounts are in local currency. I have a currency conversion table which contains the conversion factors between each currency and the euro, for any given day. I want to sum up the sales, returns, giveaways, and revenue (in dollars) for a given book (product). So I join to the currency conversion table once to convert the local currency to euros, and again to convert euros to dollars (remember that the rates are different based on the settlement date of the sale). So every sale point to be considered will be twice-joined to the currency conversions; experimentation has shown me that that is the main slow-down factor.
So I'm trying to optimize the following query:
SELECT
sum(paid_sales - paid_returns) as paid_units,
sum(royalty_amt*(uu_cc.rate / sp_cc.rate)) as royalty_amt,
sum(free_sales - free_returns) as free_units,
sum(lent_units) as lent_units
FROM "sales_points"
join currency_conversions sp_cc
on sp_cc.date = sales_points.settlement_date
and sp_cc.currency = sales_points.currency
join currency_conversions uu_cc
on uu_cc.date = sales_points.settlement_date
and uu_cc.currency = 'USD'
WHERE "sales_points"."book_id" = 234
LIMIT 1
I have created the following index:
CREATE INDEX index_currency_conversions_on_date_and_currency
ON currency_conversions
USING btree (date, currency COLLATE pg_catalog."default");
and yet EXPLAIN (after running ANALYZE) tells me it is doing a sequential scan of the currency conversions table. In case it matters, date is of type 'date' and currency is of type 'char var(255)'.
Here is the query plan:
Limit (cost=7285.04..7285.04 rows=1 width=39) (actual time=103.166..103.167 rows=1 loops=1)
Buffers: shared hit=916
-> Aggregate (cost=7285.04..7285.04 rows=1 width=39) (actual time=103.163..103.163 rows=1 loops=1)
Buffers: shared hit=916
-> Hash Join (cost=584.15..7256.29 rows=6388 width=39) (actual time=60.513..92.084 rows=5840 loops=1)
Hash Cond: (sp_cc.date = uu_cc.date)
Buffers: shared hit=916
-> Hash Join (cost=351.63..6985.45 rows=6388 width=39) (actual time=52.454..72.418 rows=5840 loops=1)
Hash Cond: ((sales_points.settlement_date = sp_cc.date) AND ((sales_points.currency)::text = (sp_cc.currency)::text))
Buffers: shared hit=763
-> Bitmap Heap Scan on sales_points (cost=54.09..6630.06 rows=6446 width=30) (actual time=0.912..7.020 rows=5840 loops=1)
Recheck Cond: (book_id = 234)
Buffers: shared hit=610
-> Bitmap Index Scan on index_sales_points_on_book_id (cost=0.00..53.77 rows=6446 width=0) (actual time=0.809..0.809 rows=6521 loops=1)
Index Cond: (book_id = 234)
Buffers: shared hit=22
-> Hash (cost=214.95..214.95 rows=20649 width=16) (actual time=51.502..51.502 rows=20649 loops=1)
Buckets: 4096 Batches: 1 Memory Usage: 968kB
Buffers: shared hit=153
-> Seq Scan on currency_conversions sp_cc (cost=0.00..214.95 rows=20649 width=16) (actual time=0.007..21.153 rows=20649 loops=1)
Buffers: shared hit=153
-> Hash (cost=225.27..225.27 rows=2071 width=12) (actual time=8.040..8.040 rows=2071 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 89kB
Buffers: shared hit=153
-> Seq Scan on currency_conversions uu_cc (cost=0.00..225.27 rows=2071 width=12) (actual time=0.021..5.963 rows=2071 loops=1)
Filter: ((currency)::text = 'USD'::text)
Buffers: shared hit=153
Total runtime: 103.306 ms
Does anyone know why it is not using my index?
The multi-column index is something of a mistake here. You probably want two separate indexes on the two columns since this gives the planner more flexibility.
Your current index cannot be used with your query since it requires querying on date from the table (the btree is first on date, secondarily on currency). If the columns were in the other order it might be useful but it could not be used where date would be more selective.
Your best option is to have separate indexes for the two fields. This way the planner can choose which index is expected to be more selective for the query at hand, rather than having to take or leave an index which may be of dubious value for a given query.
Also note that PostgreSQL can do bitmap index scans across multiple indexes, allowing it to use both indexes concurrently if necessary.