patial Index basing on data of second table - postgresql

Question

I have a master/detail Table situation. For every entry at the master table I have dozens at the detail table.
Lets say these are my tables:
+-----------------------
| Master
+-----------------------
| master_key integer,
| insert_date timestamp
+-----------------------
+-----------------------
| Detail
+-----------------------
| detail_key integer,
| master_key integer
| quantity numeric
| amount numeric
+-----------------------
And my most used Query is something like
SELECT extract(year from insert_date) AS Insert_Year, extract(month from insert_date) AS Insert_Month, sum(quantity) AS Quantity, sum(amount) AS Amount
FROM Master, Detail
WHERE (amount not null) and (insert_date <= '2016-12-31') and (insert_date >= '2015-01-01') and (Detail.master_key=Master.master_key)
GROUP BY Insert_Year, Insert_Month
ORDER BY Insert_Year ASC, Insert_Month ASC;
This query becomes to slow because there are tons of data for many years in both tables.
Of cause I have Indexes at both tables and EXPLAIN ANALYZE tells me that the INDEX scan takes mode than 80% of the hole Execution Time.
"Sort (cost=44013.52..44013.53 rows=1 width=19) (actual time=17073.129..17073.129 rows=16 loops=1)"
" Sort Key: (date_part('year'::text, master.insert_date)), (date_part('month'::text, master.insert_date))"
" Sort Method: quicksort Memory: 26kB"
" -> HashAggregate (cost=44013.49..44013.51 rows=1 width=19) (actual time=17073.046..17073.053 rows=16 loops=1)"
" Group Key: date_part('year'::text, master.insert_date), date_part('month'::text, master.insert_date)"
" -> Nested Loop (cost=0.43..43860.32 rows=15317 width=19) (actual time=0.056..15951.178 rows=843647 loops=1)"
" -> Seq Scan on master (cost=0.00..18881.38 rows=3127 width=12) (actual time=0.027..636.202 rows=182338 loops=1)"
" Filter: ((date(insert_date) >= '2015-01-01'::date) AND (date(insert_date) <= '2016-12-31'::date))"
" Rows Removed by Filter: 443031"
" -> Index Scan using idx_detail_master_key on detail (cost=0.43..7.89 rows=7 width=15) (actual time=0.055..0.077 rows=5 loops=182338)"
" Index Cond: (master_key = master.master_key)"
" Filter: (amount IS NOT NULL)"
" Rows Removed by Filter: 2"
"Planning time: 105.317 ms"
"Execution time: 17073.396 ms"
So my idea was to reduce the index sizes by defining them partial. In most cases only data of the last 2 years are queried.
So I tried something like that:
CREATE INDEX idx_detail_table_master_keys
ON detail (master_key)
WHERE master_key in (SELECT master_key FROM master WHERE (extract( year from insert_date) = 2016) or (extract( year from insert_date) = 2015))
Of cause this is not the final version it should just be a proof of concept and it failed. PGAdmin Tells me that I'm not allowed to use subselects on Index creation.
So my question is: Is it posiple to create a partial Index, basing on the data of an other table?
And of cause I would be thankful for any Tips speeding constellations like this up.
regards

Answer 1

It is not possible to create a partial index, basing on the data of another table because relational databases like postgresql have three ways of joining: nested loops, hash-join and sort-merge. All of these methods load the tables of a join seperately. Since the database optimizer decides which of those methods will be used and also in which direction the join will be executed, it doesn't make sense to create a table index that covers data of another table. This is why you can't define such indexes. A more detailed description about this topics can be found here: http://use-the-index-luke.com/sql/join (and the following sections of the online book)
For further optimization see the comment of Gabriel's Messanger