Postgres using wrong index when querying a view of indexed expressions? - postgresql

Question

When I run the following script with Postgres 9.3 (with enable_seqscan set to off), I expect the final query to make use of the "forms_string" partial index, but instead uses the "forms_int" index, which doesn't make sense.
When I've been testing this with actual code with JSON functions and indexes for more types, it consistently seems to use whatever the last index created was, for every query.
Adding more unrelated rows so that the rows relevant to the partial index are only a small percentage of total rows in the table results in a "bitmap heap scan", but still mentions the same incorrect index after that.
Any idea how I can get it to use the correct index?
CREATE EXTENSION IF NOT EXISTS plv8;
CREATE OR REPLACE FUNCTION
json_string(data json, key text) RETURNS TEXT AS $$
var ret = data,
keys = key.split('.'),
len = keys.length;
for (var i = 0; i < len; ++i) {
if (ret) {
ret = ret[keys[i]]
};
}
if (typeof ret === "undefined") {
ret = null;
} else if (ret) {
ret = ret.toString();
}
return ret;
$$ LANGUAGE plv8 IMMUTABLE STRICT;
CREATE OR REPLACE FUNCTION
json_int(data json, key text) RETURNS INT AS $$
var ret = data,
keys = key.split('.'),
len = keys.length;
for (var i = 0; i < len; ++i) {
if (ret) {
ret = ret[keys[i]]
}
}
if (typeof ret === "undefined") {
ret = null;
} else {
ret = parseInt(ret, 10);
if (isNaN(ret)) {
ret = null;
}
}
return ret;
$$ LANGUAGE plv8 IMMUTABLE STRICT;
CREATE TABLE form_types (
id SERIAL NOT NULL,
name VARCHAR(200),
PRIMARY KEY (id)
);
CREATE TABLE tenants (
id SERIAL NOT NULL,
name VARCHAR(200),
PRIMARY KEY (id)
);
CREATE TABLE forms (
id SERIAL NOT NULL,
tenant_id INTEGER,
type_id INTEGER,
data JSON,
PRIMARY KEY (id),
FOREIGN KEY(tenant_id) REFERENCES tenants (id),
FOREIGN KEY(type_id) REFERENCES form_types (id)
);
CREATE INDEX ix_forms_type_id ON forms (type_id);
CREATE INDEX ix_forms_tenant_id ON forms (tenant_id);
INSERT INTO tenants (name) VALUES ('mike'), ('bob');
INSERT INTO form_types (name) VALUES ('type 1'), ('type 2');
INSERT INTO forms (tenant_id, type_id, data) VALUES
(1, 1, '{"string": "unicorns", "int": 1}'),
(1, 1, '{"string": "pythons", "int": 2}'),
(1, 1, '{"string": "pythons", "int": 8}'),
(1, 1, '{"string": "penguins"}');
CREATE OR REPLACE VIEW foo AS
SELECT forms.id AS forms_id,
json_string(forms.data, 'string') AS "data.string",
json_int(forms.data, 'int') AS "data.int"
FROM forms
WHERE forms.tenant_id = 1 AND forms.type_id = 1;
CREATE INDEX "forms_string" ON forms (json_string(data, 'string'))
WHERE tenant_id = 1 AND type_id = 1;
CREATE INDEX "forms_int" ON forms (json_int(data, 'int'))
WHERE tenant_id = 1 AND type_id = 1;
EXPLAIN ANALYZE VERBOSE SELECT "data.string" from foo;
Outputs:
Index Scan using forms_int on public.forms
(cost=0.13..8.40 rows=1 width=32) (actual time=0.085..0.239 rows=20 loops=1)
Output: json_string(forms.data, 'string'::text)
Total runtime: 0.282 ms
Without enable_seqscan=off:
Seq Scan on public.forms (cost=0.00..1.31 rows=1 width=32) (actual time=0.080..0.277 rows=28 loops=1)
Output: json_string(forms.data, 'string'::text)
Filter: ((forms.tenant_id = 1) AND (forms.type_id = 1))
Total runtime: 0.327 ms
\d forms prints
Table "public.forms"
Column | Type | Modifiers
-----------+---------+----------------------------------------------------
id | integer | not null default nextval('forms_id_seq'::regclass)
tenant_id | integer |
type_id | integer |
data | json |
Indexes:
"forms_pkey" PRIMARY KEY, btree (id)
"forms_int" btree (json_int(data, 'int'::text)) WHERE tenant_id = 1 AND type_id = 1
"forms_string" btree (json_string(data, 'string'::text)) WHERE tenant_id = 1 AND type_id = 1
"ix_forms_tenant_id" btree (tenant_id)
"ix_forms_type_id" btree (type_id)
Foreign-key constraints:
"forms_tenant_id_fkey" FOREIGN KEY (tenant_id) REFERENCES tenants(id)
"forms_type_id_fkey" FOREIGN KEY (type_id) REFERENCES form_types(id)

Answer 1

Index vs seqscan, costs
Looks like your random_page_cost is too high compared to the real performance of your machine. Random I/O is faster (costs less) than Pg thinks it does, so it's choosing a slightly less ideal plan.
That's why the cost estimate for the indexscan is (cost=0.13..8.40 rows=1 width=32) and for the seqscan it's slightly lower at (cost=0.00..1.31 rows=1 width=32).
Lower random_page_cost - try SET random_page_cost = 2 then re-running.
To learn more, read the documentation on PostgreSQL query planning, parameters, and tuning, and the relevant wiki pages.
Index selection
PostgreSQL appears to be picking an index scan on forms_int instead of forms_string because it'll be a more compact, smaller index, and both indexes exactly match the search criteria for the view: tenant_id = 1 AND type_id = 1.
If you disable or drop forms_int it'll probably use forms_string and go slightly slower.
The key thing to understand is that while the index does contain the value of interest, PostgreSQL isn't actually using it. It's scanning the index without an index condition, since every tuple in the index matches, to get tuples from the heap. It's then extracting the value from those heap tuples and outputting them.
This can be demonstrated with an expression-index on a constant:
CREATE INDEX "forms_novalue" ON forms((true)) WHERE tenant_id = 1 AND type_id = 1;
PostgreSQL is quite likely to select this index for the query:
regress=# EXPLAIN ANALYZE VERBOSE SELECT "data.string" from foo;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------
Index Scan using forms_novalue on public.forms (cost=0.13..13.21 rows=4 width=32) (actual time=0.190..0.310 rows=4 loops=1)
Output: json_string(forms.data, 'string'::text)
Total runtime: 0.346 ms
(3 rows)
All the indexes are the same size because they're all so tiny they fit in the minimum allocation:
regress=# SELECT x.idxname, pg_relation_size(x.idxname) FROM (VALUES ('forms_novalue'),('forms_int'),('forms_string')) x(idxname);
idxname | pg_relation_size
---------------+------------------
forms_novalue | 16384
forms_int | 16384
forms_string | 16384
(3 rows)
but the stats for novalue will be somewhat more attractive due to a narrower row width.
Index scan vs index-only scan
It sounds like what you really expect is an index-only scan, where Pg never touches the table's heap and only uses the tuples in the index its self.
I would expect that this query's requirements could be satisfied with forms_string, but can't get Pg to pick an index-only scan plan for it.
It's not immediately clear to me why Pg is not using an index-only scan here, as it should be a candidate, but it doesn't seem to be able to plan one. If I force enable_indexscan = off, it'll pick an inferior bitmap index scan plan instead, and if force disable enable_bitmapscan it'll fall back to a max-cost-estimate seqscan. This is true even after a VACUUM of the table(s) of interest.
That means it must not be being generated as a candidate path in the query planner - Pg doesn't know how to use an index-only scan for this query, or thinks it cannot do so for some reason.
It isn't an issue with view introspection, as an expanded view query is the same.

Answer 2

Your table has insufficient data in it. In short, Postgres won't use an index when the table fits in a single disk page. Ever. When your table contains a few hundred or thousand rows, it'll become too big to fit, and then you'll see Postgres begin to use index scans when relevant.
Another point to consider is you need to analyze your tables after a large import. Without accurate stats on your actual data, Postgres may end up dismissing some index scans as too expensive, when in fact they'd be cheap.
Lastly, there are cases when it is cheaper to not use an index. In essence, whenever Postgres is about to visit most disk pages repeatedly and in a random order to retrieve a large number of rows, it'll seriously consider the cost of visiting most (bitmap index) or all (seq scan) disk pages once sequentially and filtering invalid rows out. The latter wins if you're selecting enough rows.