I have a table like:
CREATE TABLE IF NOT EXISTS my_table (
id uuid NOT NULL PRIMARY KEY,
duplicate_ids uuid[] DEFAULT NULL,
);
And my query is:
SELECT * FROM my_table WHERE 'some-uuid'=ANY(duplicate_ids)
Using EXPLAIN ANALYZE and trying lots of different indexes, I am unable to get the above to use an index.
Here's what I've tried (Postgres 12):
CREATE INDEX duplicate_ids_idx ON my_table USING GIN (duplicate_ids);
CREATE INDEX duplicate_ids_idx ON my_table USING GIN (duplicate_ids array_ops);
CREATE INDEX duplicate_ids_idx ON my_table USING BTREE (duplicate_ids);
CREATE INDEX duplicate_ids_idx ON my_table USING BTREE (duplicate_ids array_ops);
I've also ran SET enable_seqscan TO off; before these tests to enforce index usage.
Questions I've read:
Can PostgreSQL index array columns?
Doesn't seem to apply for single values
https://dba.stackexchange.com/questions/125413/index-not-used-with-any-but-used-with-in
Seems to be talking about indexing multiple columns and using IN
Thank you very much for your time.
Question was answered by #a_horse_with_no_name
The solution appears to be to use something like:
SELECT * FROM my_table WHERE duplicate_ids && array['some_uuid']::uuid[]
Related
I have a table with a JSONB column that is used to store multiple tags (integer) that have been applied to a task, eg.: '[123, 456, 789]'.
ALTER TABLE "public"."task" ADD COLUMN "tags" jsonb;
I also have a table dedicated to storing all the tags that can be used, and the primary key of each record is used in my JSONB column of the task table.
CREATE TABLE public.tag (
tag_id serial NOT NULL,
label varchar(50) NOT NULL,
);
In this table (tag) I have an index based on the task ID, and I want to use this index in a query that returns the tags labels that were used in a task.
SELECT * FROM task, tag WHERE task.tags #> to_jsonb(tag.tag_id)
Using to_jsonb is really bad as it doesn't use my table's index, but if I change the SQL to something like the example below, the index is used and SQL performance is much better.
SELECT * FROM tag WHERE tag.tag_id IN (123, 456, 789)
How do I convert the jsonb column (task table) to a set of integer values that can be used with the IN condition, as in the example below?
SELECT * FROM task, tag WHERE tag.tag_id IN (task.tags);
You can use PostgreSQL jsonb_array_elements function which convert JSON elements to table records. For example:
SELECT * FROM task, tag WHERE tag.tag_id in (
select jsonb_array_elements('[200, 100, 789]'::jsonb)::int4 as json_data
);
But, for best performance, if you get JSON data from the table fields, so you must index this JSON field not use the standard btree index type. For JSON types PostgreSQL has a different index type as GIN index. This index type will give the best performance. I use this index in my table which has a million records. Very very best performance. Example for creating GIN index:
CREATE INDEX tag_table_json_index ON tag_table USING gin (json_field_name jsonb_path_ops);
I am solving performance issues on PostgreSQL and I have the following table:
CREATE TABLE main_transaction (
id integer NOT NULL DEFAULT nextval('main_transaction_id_seq'::regclass),
description character varying(255) NOT NULL,
request_no character varying(18),
account character varying(50),
....
)
Above table has 34 columns including 3 FOREIGN KEYs and it has over 1 Million rows data. I have the following conditional SELECT query:
SELECT * FROM main_transaction
WHERE upper(request_no) LIKE upper(concat('%','20080417-0258-0697','%'))
Returning the result in over 2 seconds. I want to decrease working time by using table indexing. So far, I have used btree indexing. However, I didn't notice any fast result. My question is, how can I improve performance for above query?
Your only chance to search for a pattern that begins with % is a trigram index:
CREATE EXTENSION pg_trgm;
CREATE INDEX ON main_transaction
USING gin (upper(request_no) gin_trgm_ops);
I have a table with a jsonb column where I store variable data. I would like to search this column and also find fragments (leading or trailing whitespace). I think I know how to do this with text columns but cannot wrap my head around how to achieve this with jsonb columns.
There are two scenarios that I would like to achieve:
Search a specific key inside the jsonb column only (for example
data->>company)
Search the whole jsonb column
For text columns I generate gin indexes using pg_trgm.
Install extension pg_trgm:
CREATE extension if not exists pg_trgm;
Create table & index:
CREATE TABLE tbl (
col_text text,
col_json jsonb
);
CREATE INDEX table_col_trgm_idx ON tbl USING gin (col_text gin_trgm_ops);
Example query:
SELECT * FROM tbl WHERE col_text LIKE '%foo%'; -- leading wildcard
SELECT * FROM tbl WHERE col_text ILIKE '%foo%'; -- works case insensitive as well
Trying the same with the jsonb column fails. If I try to index the whole column
CREATE INDEX table_col_trgm_idx ON tbl USING gin (col_json gin_trgm_ops);
I get the error
ERROR (datatype_mismatch): operator class "gin_trgm_ops" does not accept data type jsonb
(Which makes sense). If I try to index just one key of the jsonb column I also receive an error:
CREATE INDEX table_col_trgm_idx ON tbl USING gin (col_json->>company gin_trgm_ops);
Error:
ERROR (syntax_error): syntax error at or near "->>"
I used this answer by #erwin-brandstetter as a reference. Any help is highly appreciated (and no, I don't want to implement Elasticsearch as of now :) ).
Edit: Creating the index like this actually works:
CREATE INDEX table_col_trgm_idx ON tbl USING gin ((col_json->>'company') gin_trgm_ops);
And querying it also doesn't lead to an error:
SELECT * FROM tbl WHERE col_json->>'company' LIKE '%foo%';
But the result is always empty.
The postgresql documentation specifies that a GiST index can have multiple columns, but does not provide an example of what this might look like.
I have a table that tracks assets owned by different customers.
CREATE TABLE asset (
id serial PRIMARY KEY,
description text NOT NULL,
customer_id uuid NOT NULL
);
I'm writing a query that allows a customer to search for an asset based on words in it's description.
SELECT *
FROM asset
WHERE to_tsvector('english', asset.description) ## plainto_tsvector('english', ?)
AND asset.customer_id = ?;
Were this a non-tsvector query, I'd build a simple multicolumn index
CREATE INDEX idx_name ON asset(customer_id, description);
I can create an index only on the tsvector:
CREATE INDEX idx_name ON asset USING gist(to_tsvector('english', asset.description));
However, the query optimizer doesn't use the gist index, because it seems to want to do customer_id filtering first. Is there a way that I can include the non-tsvector field customer_id in the gist index, somehow, or am I out of luck?
You can create a multi column GIN or GiST index in Postrgres using the following:
CREATE INDEX index_name
ON table_name
USING gist (field_one gist_trgm_ops, field_two gist_trgm_ops);
I have two tables:
CREATE TABLE soils (
sample_id TEXT PRIMARY KEY,
project_id TEXT,
technician_id TEXT
);
CREATE INDEX soils_idx
ON soils
USING btree
(sample_id COLLATE pg_catalog."default");
CREATE TABLE assays (
sample_id TEXT PRIMARY KEY,
mo_ppm NUMERIC
);
CREATE INDEX assays_idx
ON assays
USING btree
(sample_id COLLATE pg_catalog."default");
Each table contains about a half million records, and, in reality, about 20 additional columns each, of type TEXT (omitted in the DDL posted above to save time here).
When I perform the query:
EXPLAIN SELECT
s.sample_id, s.project_id, s.technician_id, a.mo_ppm
FROM
soils AS s INNER JOIN assays AS a ON s.sample_id = a.sample_id
I get 2 SEQ SCANs, rather than a lookup to the index. Is that expected behaviour?
Since you have no WHERE conditions, you effectively read the whole table. It's cheaper to run sequential scans and not involve any indexes at all.
Try:
EXPLAIN
SELECT s.sample_id, s.project_id, s.technician_id, a.mo_ppm
FROM soils s
JOIN assays a USING (sample_id)
WHERE <some condition that returns few rows>;
... and an index matching the WHERE condition should be used.
You don't need to define an index on a PRIMARY KEY column. A PK constraint is implemented with a unique index automatically. Your additional index is redundant and of no use.
An index on a foreign key column would be a good idea, but there isn't one in your example, which looks odd. Like the two tables could be combined into one. Probably just over-simplification for the test case.
Finally, for big tables, I would consider using a simple integer primary key instead of text, possibly a serial column. That's typically faster.
Yes, that's expected behaviour. On the other hand it depends on your random_page_cost, seq_page_cost and effective_cache_size settings. Your query doesn't have WHERE clause hence it might be faster to read everything sequentially. You can try to penalise sequential scan:
set enable_seqscan = off;
explain analyse <your query>;
and then compare plan/cost/IO wait (it is not possible to disable seq-scan but it gets very high cost -- ~1e7 (or 1e8)).
If you have SSD and WHERE clause in your query then you can lower random_page_cost to 1.5..2.5 and encourage PG to use index.