I have a table with a jsonb column where I store variable data. I would like to search this column and also find fragments (leading or trailing whitespace). I think I know how to do this with text columns but cannot wrap my head around how to achieve this with jsonb columns.
There are two scenarios that I would like to achieve:
Search a specific key inside the jsonb column only (for example
data->>company)
Search the whole jsonb column
For text columns I generate gin indexes using pg_trgm.
Install extension pg_trgm:
CREATE extension if not exists pg_trgm;
Create table & index:
CREATE TABLE tbl (
col_text text,
col_json jsonb
);
CREATE INDEX table_col_trgm_idx ON tbl USING gin (col_text gin_trgm_ops);
Example query:
SELECT * FROM tbl WHERE col_text LIKE '%foo%'; -- leading wildcard
SELECT * FROM tbl WHERE col_text ILIKE '%foo%'; -- works case insensitive as well
Trying the same with the jsonb column fails. If I try to index the whole column
CREATE INDEX table_col_trgm_idx ON tbl USING gin (col_json gin_trgm_ops);
I get the error
ERROR (datatype_mismatch): operator class "gin_trgm_ops" does not accept data type jsonb
(Which makes sense). If I try to index just one key of the jsonb column I also receive an error:
CREATE INDEX table_col_trgm_idx ON tbl USING gin (col_json->>company gin_trgm_ops);
Error:
ERROR (syntax_error): syntax error at or near "->>"
I used this answer by #erwin-brandstetter as a reference. Any help is highly appreciated (and no, I don't want to implement Elasticsearch as of now :) ).
Edit: Creating the index like this actually works:
CREATE INDEX table_col_trgm_idx ON tbl USING gin ((col_json->>'company') gin_trgm_ops);
And querying it also doesn't lead to an error:
SELECT * FROM tbl WHERE col_json->>'company' LIKE '%foo%';
But the result is always empty.
Related
I have a table with a JSONB column that is used to store multiple tags (integer) that have been applied to a task, eg.: '[123, 456, 789]'.
ALTER TABLE "public"."task" ADD COLUMN "tags" jsonb;
I also have a table dedicated to storing all the tags that can be used, and the primary key of each record is used in my JSONB column of the task table.
CREATE TABLE public.tag (
tag_id serial NOT NULL,
label varchar(50) NOT NULL,
);
In this table (tag) I have an index based on the task ID, and I want to use this index in a query that returns the tags labels that were used in a task.
SELECT * FROM task, tag WHERE task.tags #> to_jsonb(tag.tag_id)
Using to_jsonb is really bad as it doesn't use my table's index, but if I change the SQL to something like the example below, the index is used and SQL performance is much better.
SELECT * FROM tag WHERE tag.tag_id IN (123, 456, 789)
How do I convert the jsonb column (task table) to a set of integer values that can be used with the IN condition, as in the example below?
SELECT * FROM task, tag WHERE tag.tag_id IN (task.tags);
You can use PostgreSQL jsonb_array_elements function which convert JSON elements to table records. For example:
SELECT * FROM task, tag WHERE tag.tag_id in (
select jsonb_array_elements('[200, 100, 789]'::jsonb)::int4 as json_data
);
But, for best performance, if you get JSON data from the table fields, so you must index this JSON field not use the standard btree index type. For JSON types PostgreSQL has a different index type as GIN index. This index type will give the best performance. I use this index in my table which has a million records. Very very best performance. Example for creating GIN index:
CREATE INDEX tag_table_json_index ON tag_table USING gin (json_field_name jsonb_path_ops);
I have a table like:
CREATE TABLE IF NOT EXISTS my_table (
id uuid NOT NULL PRIMARY KEY,
duplicate_ids uuid[] DEFAULT NULL,
);
And my query is:
SELECT * FROM my_table WHERE 'some-uuid'=ANY(duplicate_ids)
Using EXPLAIN ANALYZE and trying lots of different indexes, I am unable to get the above to use an index.
Here's what I've tried (Postgres 12):
CREATE INDEX duplicate_ids_idx ON my_table USING GIN (duplicate_ids);
CREATE INDEX duplicate_ids_idx ON my_table USING GIN (duplicate_ids array_ops);
CREATE INDEX duplicate_ids_idx ON my_table USING BTREE (duplicate_ids);
CREATE INDEX duplicate_ids_idx ON my_table USING BTREE (duplicate_ids array_ops);
I've also ran SET enable_seqscan TO off; before these tests to enforce index usage.
Questions I've read:
Can PostgreSQL index array columns?
Doesn't seem to apply for single values
https://dba.stackexchange.com/questions/125413/index-not-used-with-any-but-used-with-in
Seems to be talking about indexing multiple columns and using IN
Thank you very much for your time.
Question was answered by #a_horse_with_no_name
The solution appears to be to use something like:
SELECT * FROM my_table WHERE duplicate_ids && array['some_uuid']::uuid[]
My table (table) has a JSONB field (data) that contains a field with an array where I store tags (tags).
I query that table with an expression like:
SELECT * FROM table WHERE data->'tags' #? '$[*] ? (# like_regex ".*(foo|bar).*" flag "i");
With such use-case is there a way for me to index the data->'tags' array to speed up the query? Or should I rather work on moving the tags array out of the JSONB field and into a TEXT[] field and index that?
I've already tried:
CREATE INDEX foo ON tbl USING GIN ((data->'tags') jsonb_path_ops);
but it doesn't work: https://gist.github.com/vkaracic/a62ac917d34eb6e975c4daeefbd316e8
The index you built can be used (if you set enable_seqscan=off, you will see that it does get used), but it is generally not chosen as it is pretty useless for this query. The only rows it would rule out through the index are the ones that don't have the 'tags' key at all, and even at that is poorly estimated so probably won't be used without drastic measures.
You could try to convert to text[] and the use parray_gin, but probably better would be to convert to a child table with text and then use pg_trgm.
Like the title says, how can I index a JSONB array?
The contents look like...
["some_value", "another_value"]
I can easily access the elements like...
SELECT * FROM table WHERE data->>0 = 'some_value';
I created an index like so...
CREATE INDEX table_data_idx ON table USING gin ((data) jsonb_path_ops);
When I run EXPLAIN, I still see it sequentially scanning...
What am I missing on indexing an array of text elements?
If you want to support that exact query with an index, the index would have to look like this:
CREATE INDEX ON "table" ((data->>0));
If you want to use the index you have, you cannot limit the search to just a specific array element (in your case, the first). You can speed up a search for some_value anywhere in the array:
SELECT * FROM "table"
WHERE data #> '["some_value"]'::jsonb;
I ended up taking a different approach. I am still having problems getting the search to work using a JSONB Type, so I ended up switching my column to a varchar ARRAY
CREATE TABLE table (
data varchar ARRAY NOT NULL
);
CREATE INDEX table_data_idx ON table USING GIN (data);
SELECT * FROM table WHERE data #> '{some_value}';
This works and is using the index.
I think my problem with my JSONB approach is because the element is actually nested much further and being treated as text.
i.e. data->'some_key'->>'array_key'->>0
And everytime I try to search I get all sorts of invalid token errors and other such things.
You may want to create a materialized view that has the primary key (or other unique index of your table) and expands the array field into a text column with the jsonb_array_elements_text function:
CREATE MATERIALIZED VIEW table_mv
AS
SELECT DISTINCT table.id, jsonb_array_elements_text(data->0) AS array_elem FROM table;
You can then create a unique index on this materialized view (primary keys are not supported on materialized views):
CREATE UNIQUE INDEX table_array_idx ON table_mv(id, array_elem);
Then query with a join to the original table on its primary key:
SELECT * FROM table INNER JOIN table_mv ON table.id = table_mv.id WHERE table_mv.array_elem = 'some_value';
This query should use the unique index and then look up the primary key of the original table, both very fast.
The postgresql documentation specifies that a GiST index can have multiple columns, but does not provide an example of what this might look like.
I have a table that tracks assets owned by different customers.
CREATE TABLE asset (
id serial PRIMARY KEY,
description text NOT NULL,
customer_id uuid NOT NULL
);
I'm writing a query that allows a customer to search for an asset based on words in it's description.
SELECT *
FROM asset
WHERE to_tsvector('english', asset.description) ## plainto_tsvector('english', ?)
AND asset.customer_id = ?;
Were this a non-tsvector query, I'd build a simple multicolumn index
CREATE INDEX idx_name ON asset(customer_id, description);
I can create an index only on the tsvector:
CREATE INDEX idx_name ON asset USING gist(to_tsvector('english', asset.description));
However, the query optimizer doesn't use the gist index, because it seems to want to do customer_id filtering first. Is there a way that I can include the non-tsvector field customer_id in the gist index, somehow, or am I out of luck?
You can create a multi column GIN or GiST index in Postrgres using the following:
CREATE INDEX index_name
ON table_name
USING gist (field_one gist_trgm_ops, field_two gist_trgm_ops);