Limits of Postgres Query Optimization (Already using Index-Only Scans) - postgresql

Question

I have a Postgres query that has already been optimized, but we're hitting 100% CPU usage under peak load, so I wanted to see if there's more that can yet be done in optimizing the database interactions. It already is using two index-only scans in the join, so my question is if there's much more to be done on the Postgres side of things.
The database is an Amazon-hosted Postgres RDS db.m3.2xlarge instance (8 vCPUs and 30 GB of memory) running 9.4.1, and the results below are from a period with low CPU usage and minimal connections (around 15). Peak usage is around 300 simultaneous connections, and that's when we're maxing our CPU (which kills performance on everything).
Here's the query and the EXPLAIN:
Query:
EXPLAIN (ANALYZE, BUFFERS)
SELECT m.valdate, p.index_name, m.market_data_closing, m.available_date
FROM md.market_data_closing m
JOIN md.primitive p on (m.primitive_id = p.index_id)
where p.index_name = ?
order by valdate desc
;
Output:
Sort (cost=183.80..186.22 rows=967 width=44) (actual time=44.590..54.788 rows=11133 loops=1)
Sort Key: m.valdate
Sort Method: quicksort Memory: 1254kB
Buffers: shared hit=181
-> Nested Loop (cost=0.85..135.85 rows=967 width=44) (actual time=0.041..32.853 rows=11133 loops=1)
Buffers: shared hit=181
-> Index Only Scan using primitive_index_name_index_id_idx on primitive p (cost=0.29..4.30 rows=1 width=25) (actual time=0.018..0.019 rows=1 loops=1)
Index Cond: (index_name = '?'::text)
Heap Fetches: 0
Buffers: shared hit=3
-> Index Only Scan using market_data_closing_primitive_id_valdate_available_date_mar_idx on market_data_closing m (cost=0.56..109.22 rows=2233 width=27) (actual time=0.016..12.059 rows=11133 loops=1)
Index Cond: (primitive_id = p.index_id)
Heap Fetches: 42
Buffers: shared hit=178
Planning time: 0.261 ms
Execution time: 64.957 ms
Here are the table sizes:
md.primitive: 14283 rows
md.market_data_closing: 13544087 rows
For reference, here is the underlying spec for the tables and indices:
CREATE TABLE md.primitive(
index_id serial NOT NULL,
index_name text NOT NULL UNIQUE,
index_description text not NULL,
index_source_code text NOT NULL DEFAULT 'MAN',
index_source_spec json NOT NULL DEFAULT '{}',
frequency text NULL,
primitive_type text NULL,
is_maintained boolean NOT NULL default true,
create_dt timestamp NOT NULL,
create_user text NOT NULL,
update_dt timestamp not NULL,
update_user text not NULL,
PRIMARY KEY
(
index_id
)
) ;
CREATE INDEX ON md.primitive
(
index_name ASC,
index_id ASC
);
CREATE TABLE md.market_data_closing(
valdate timestamp NOT NULL,
primitive_id int references md.primitive,
market_data_closing decimal(28, 10) not NULL,
available_date timestamp NULL,
pricing_source text not NULL,
create_dt timestamp NOT NULL,
create_user text NOT NULL,
update_dt timestamp not NULL,
update_user text not NULL,
PRIMARY KEY
(
valdate,
primitive_id
)
) ;
CREATE INDEX ON md.market_data_closing
(
primitive_id ASC,
valdate DESC,
available_date DESC,
market_data_closing ASC
);
What else can be done?

Answer 1

It seems the nested loop is taking an absurd amount of time and primitive table is returning only one row. You can try eliminating the nested loop by doing something like this:
SELECT m.valdate, m.market_data_closing, m.available_date
FROM md.market_data_closing m
WHERE m.primitive_id = (SELECT p.index_id
FROM md.primitive p
WHERE p.index_name = ?
OFFSET 0 -- probably not needed, try it)
ORDER BY valdate DESC;
This does not return p.index_name but that can be easily fixed by selecting it as a const.

Answer 2

For next generations reading this: the problem seems to be with index
md.market_data_closing(
...
PRIMARY KEY
(
valdate,
primitive_id
)
This seems to be an incorrect index. Should be:
md.market_data_closing(
...
PRIMARY KEY
(
primitive_id,
valdate
)
Explanation why. This kind of query:
...
JOIN md.primitive p on (m.primitive_id = p.index_id)
...
Will only be effective only if primitive_id is the first field.
Also
order by validate
Will be more effective if validate is second.
Why?
Because index is a B-tree structure.
(
valdate,
primitive_id
)
results in
valdate1
primitive_id1
primitive_id2
primitive_id3
valdate2
primitive_id1
primitive_id2
Using this tree you can't search by primitive_id1 effectively
But
(
primitive_id,
valdate
)
results in
primitive_id1
valdate1
valdate2
valdate3
primitive_id2
valdate1
valdate2
Which is effective for looking up by primitive_id.
There is another solution to this problem:
If you don't want to change the index, you add a strict equal condition on valdate.
Say 'valdate = some_date', this will make your index effective.