http://www.postgresql.org/docs/9.2/static/hstore.html states:
hstore has GiST and GIN index support for the #>, ?, ?& and ?| operators
Yet the indexes don't work for the EXIST function, which appears to be equivalent to the ? operator.
What is the difference between operators and functions that makes it harder to index one or the other?
Might future versions of the Hstore extension make these truly equivalent?
Lookup the documentation for "CREATE OPERATOR CLASS" which describes how you can create indexing methods for arbitrary operators. You also need to use "CREATE OPERATOR" to create an operator based on the EXIST function first.
(Caveat: I have no experience with hstore)
http://www.postgresql.org/docs/9.0/static/sql-createoperator.html
http://www.postgresql.org/docs/9.0/static/sql-createopclass.html
Here's your problem: PostgreSQL functions are planner-opaque. The planner has no way of knowing that the operator and the function are semantically equivalent. This comes up a lot.
PostgreSQL does have functional indexes so you can index outputs of immutable functions but this may not quite make things work perfectly well here since you'd probably be able to only index which rows return true for a given call, but this could still be very useful with partial indexes. For example you could always do something like:
CREATE INDEX bar_has_aaa ON foo(exists(bar, 'aaa'));
or
CREATE INDEX bar_has_aaa ON foo(id) where exists (bar, 'aaa');
But I don' see this going exactly where you need it to go. Hopefully it points you in the right direction though.
Edit: The following strikes me as a better workaround. Suppose we have a table foo:
CREATE TABLE foo (
id serial,
bar hstore
);
We can create a table method bar_keys:
CREATE FUNCTION bar_keys(foo) RETURNS text[] IMMUTABLE LANGUAGE SQL AS $$
SELECT akeys($1.bar);
$$;
Then we can index that using GIN:
CREATE INDEX foo_bar_keys_idx ON foo USING gin(bar_keys(foo));
And we can use it in our queries:
SELECT * FROM foo WHERE foo.bar_keys #> array['aaa'];
That should use an index. Note you could just index/use akeys directly, but I think a virtual column leads to cleaner syntax.
Related
In Microsoft SQL Server, it's possible to specify an "accent insensitive" collation (for a database, table or column), which means that it's possible for a query like
SELECT * FROM users WHERE name LIKE 'João'
to find a row with a Joao name.
I know that it's possible to strip accents from strings in PostgreSQL using the unaccent_string contrib function, but I'm wondering if PostgreSQL supports these "accent insensitive" collations so the SELECT above would work.
Update for Postgres 12 or later
Postgres 12 adds nondeterministic ICU collations, enabling case-insensitive and accent-insensitive grouping and ordering. The manual:
ICU locales can only be used if support for ICU was configured when PostgreSQL was built.
If so, this works for you:
CREATE COLLATION ignore_accent (provider = icu, locale = 'und-u-ks-level1-kc-true', deterministic = false);
CREATE INDEX users_name_ignore_accent_idx ON users(name COLLATE ignore_accent);
SELECT * FROM users WHERE name = 'João' COLLATE ignore_accent;
fiddle
Read the manual for details.
This blog post by Laurenz Albe may help to understand.
But ICU collations also have drawbacks. The manual:
[...] they also have some drawbacks. Foremost, their use leads to a
performance penalty. Note, in particular, that B-tree cannot use
deduplication with indexes that use a nondeterministic collation.
Also, certain operations are not possible with nondeterministic
collations, such as pattern matching operations. Therefore, they
should be used only in cases where they are specifically wanted.
My "legacy" solution may still be superior:
For all versions
Use the unaccent module for that - which is completely different from what you are linking to.
unaccent is a text search dictionary that removes accents (diacritic
signs) from lexemes.
Install once per database with:
CREATE EXTENSION unaccent;
If you get an error like:
ERROR: could not open extension control file
"/usr/share/postgresql/<version>/extension/unaccent.control": No such file or directory
Install the contrib package on your database server like instructed in this related answer:
Error when creating unaccent extension on PostgreSQL
Among other things, it provides the function unaccent() you can use with your example (where LIKE seems not needed).
SELECT *
FROM users
WHERE unaccent(name) = unaccent('João');
Index
To use an index for that kind of query, create an index on the expression. However, Postgres only accepts IMMUTABLE functions for indexes. If a function can return a different result for the same input, the index could silently break.
unaccent() only STABLE not IMMUTABLE
Unfortunately, unaccent() is only STABLE, not IMMUTABLE. According to this thread on pgsql-bugs, this is due to three reasons:
It depends on the behavior of a dictionary.
There is no hard-wired connection to this dictionary.
It therefore also depends on the current search_path, which can change easily.
Some tutorials on the web instruct to just alter the function volatility to IMMUTABLE. This brute-force method can break under certain conditions.
Others suggest a simple IMMUTABLE wrapper function (like I did myself in the past).
There is an ongoing debate whether to make the variant with two parameters IMMUTABLE which declares the used dictionary explicitly. Read here or here.
Another alternative would be this module with an IMMUTABLE unaccent() function by Musicbrainz, provided on Github. Haven't tested it myself. I think I have come up with a better idea:
Best for now
This approach is more efficient than other solutions floating around, and safer.
Create an IMMUTABLE SQL wrapper function executing the two-parameter form with hard-wired, schema-qualified function and dictionary.
Since nesting a non-immutable function would disable function inlining, base it on a copy of the C-function, (fake) declared IMMUTABLE as well. Its only purpose is to be used in the SQL function wrapper. Not meant to be used on its own.
The sophistication is needed as there is no way to hard-wire the dictionary in the declaration of the C function. (Would require to hack the C code itself.) The SQL wrapper function does that and allows both function inlining and expression indexes.
CREATE OR REPLACE FUNCTION public.immutable_unaccent(regdictionary, text)
RETURNS text
LANGUAGE c IMMUTABLE PARALLEL SAFE STRICT AS
'$libdir/unaccent', 'unaccent_dict';
Then:
CREATE OR REPLACE FUNCTION public.f_unaccent(text)
RETURNS text
LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT AS
$func$
SELECT public.immutable_unaccent(regdictionary 'public.unaccent', $1)
$func$;
In Postgres 14 or later, an SQL-standard function is slightly cheaper, yet:
CREATE OR REPLACE FUNCTION public.f_unaccent(text)
RETURNS text
LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT
BEGIN ATOMIC
SELECT public.immutable_unaccent(regdictionary 'public.unaccent', $1);
END;
See:
What does BEGIN ATOMIC mean in a PostgreSQL SQL function / procedure?
Drop PARALLEL SAFE from both functions for Postgres 9.5 or older.
public being the schema where you installed the extension (public is the default).
The explicit type declaration (regdictionary) defends against hypothetical attacks with overloaded variants of the function by malicious users.
Previously, I advocated a wrapper function based on the STABLE function unaccent() shipped with the unaccent module. That disabled function inlining. This version executes ten times faster than the simple wrapper function I had here earlier.
And that was already twice as fast as the first version which added SET search_path = public, pg_temp to the function - until I discovered that the dictionary can be schema-qualified, too. Still (Postgres 12) not too obvious from documentation.
If you lack the necessary privileges to create C functions, you are back to the second best implementation: An IMMUTABLE function wrapper around the STABLE unaccent() function provided by the module:
CREATE OR REPLACE FUNCTION public.f_unaccent(text)
RETURNS text
LANGUAGE sql IMMUTABLE PARALLEL SAFE STRICT AS
$func$
SELECT public.unaccent('public.unaccent', $1) -- schema-qualify function and dictionary
$func$;
Finally, the expression index to make queries fast:
CREATE INDEX users_unaccent_name_idx ON users(public.f_unaccent(name));
Remember to recreate indexes involving this function after any change to function or dictionary, like an in-place major release upgrade that would not recreate indexes. Recent major releases all had updates for the unaccent module.
Adapt queries to match the index (so the query planner will use it):
SELECT * FROM users
WHERE f_unaccent(name) = f_unaccent('João');
We don't need the function in the expression to the right of the operator. There we can also supply unaccented strings like 'Joao' directly.
The faster function does not translate to much faster queries using the expression index. Index look-ups operate on pre-computed values and are very fast either way. But index maintenance and queries not using the index benefit. And access methods like bitmap index scans may have to recheck values in the heap (the main relation), which involves executing the underlying function. See:
"Recheck Cond:" line in query plans with a bitmap index scan
Security for client programs has been tightened with Postgres 10.3 / 9.6.8 etc. You need to schema-qualify function and dictionary name as demonstrated when used in any indexes. See:
'text search dictionary “unaccent” does not exist' entries in postgres log, supposedly during automatic analyze
Ligatures
In Postgres 9.5 or older ligatures like 'Œ' or 'ß' have to be expanded manually (if you need that), since unaccent() always substitutes a single letter:
SELECT unaccent('Œ Æ œ æ ß');
unaccent
----------
E A e a S
You will love this update to unaccent in Postgres 9.6:
Extend contrib/unaccent's standard unaccent.rules file to handle all
diacritics known to Unicode, and expand ligatures correctly (Thomas
Munro, Léonard Benedetti)
Bold emphasis mine. Now we get:
SELECT unaccent('Œ Æ œ æ ß');
unaccent
----------
OE AE oe ae ss
Pattern matching
For LIKE or ILIKE with arbitrary patterns, combine this with the module pg_trgm in PostgreSQL 9.1 or later. Create a trigram GIN (typically preferable) or GIST expression index. Example for GIN:
CREATE INDEX users_unaccent_name_trgm_idx ON users
USING gin (f_unaccent(name) gin_trgm_ops);
Can be used for queries like:
SELECT * FROM users
WHERE f_unaccent(name) LIKE ('%' || f_unaccent('João') || '%');
GIN and GIST indexes are more expensive (to maintain) than plain B-tree:
Difference between GiST and GIN index
There are simpler solutions for just left-anchored patterns. More about pattern matching and performance:
Pattern matching with LIKE, SIMILAR TO or regular expressions in PostgreSQL
pg_trgm also provides useful operators for "similarity" (%) and "distance" (<->).
Trigram indexes also support simple regular expressions with ~ et al. and case insensitive pattern matching with ILIKE:
PostgreSQL accent + case insensitive search
No, PostgreSQL does not support collations in that sense
PostgreSQL does not support collations like that (accent insensitive or not) because no comparison can return equal unless things are binary-equal. This is because internally it would introduce a lot of complexities for things like a hash index. For this reason collations in their strictest sense only affect ordering and not equality.
Workarounds
Full-Text-Search Dictionary that Unaccents lexemes.
For FTS, you can define your own dictionary using unaccent,
CREATE EXTENSION unaccent;
CREATE TEXT SEARCH CONFIGURATION mydict ( COPY = simple );
ALTER TEXT SEARCH CONFIGURATION mydict
ALTER MAPPING FOR hword, hword_part, word
WITH unaccent, simple;
Which you can then index with a functional index,
-- Just some sample data...
CREATE TABLE myTable ( myCol )
AS VALUES ('fóó bar baz'),('qux quz');
-- No index required, but feel free to create one
CREATE INDEX ON myTable
USING GIST (to_tsvector('mydict', myCol));
You can now query it very simply
SELECT *
FROM myTable
WHERE to_tsvector('mydict', myCol) ## 'foo & bar'
mycol
-------------
fóó bar baz
(1 row)
See also
Creating a case-insensitive and accent/diacritics insensitive search on a field
Unaccent by itself.
The unaccent module can also be used by itself without FTS-integration, for that check out Erwin's answer
I'm pretty sure PostgreSQL relies on the underlying operating system for collation. It does support creating new collations, and customizing collations. I'm not sure how much work that might be for you, though. (Could be quite a lot.)
Since btree_gin in 9.5 does not support boolean data type, how can I use boolean column as part of multi-column gin index?
Technically, it's possible, but you need to index the (is_read::int::bit) expression (instead of the column directly). But: you would need to use this expression in your WHERE clauses, to make use of this index; i.e.:
WHERE is_read::int::bit = '1'
-- or
WHERE is_read::int::bit = '0'
-- or even
WHERE is_read::int::bit < '1' -- which is just an obfuscated version of "= '0'"
However, this will make your queries less readable. And maybe even slower (see later).
If you ever query for one value (i.e. WHERE is_read or WHERE NOT is_read, but not both), a partial index would be a better fit.
However, dropping the column from the index could make it (somewhat) more compact, which can even fasten your queries in some cases.
I advise you to test each of these methods on your actual data, or show us (in another, follow-up question) your queries you are concerned with.
Here is a comparison for the above cases with some fairly artificial data:
http://rextester.com/OWXUA55980
Trying to create an aggregate function:
create aggregate min (my_type) (
sfunc = least,
stype = my_type
);
ERROR: syntax error at or near "least"
LINE 2: sfunc = least,
^
What am I missing?
Although the manual calls least a function:
The GREATEST and LEAST functions select the largest or smallest value from a list of any number of expressions.
I can not find it:
\dfS least
List of functions
Schema | Name | Result data type | Argument data types | Type
--------+------+------------------+---------------------+------
(0 rows)
Like CASE, COALESCE and NULLIF, GREATEST and LEAST are listed in the chapter Conditional Expressions. These SQL constructs are not implemented as functions .. like #Laurenz provided in the meantime.
The manual advises:
Tip: If your needs go beyond the capabilities of these conditional
expressions, you might want to consider writing a stored procedure in
a more expressive programming language.
The terminology is a bit off here as well, since Postgres does not support true "stored procedures", just functions. (Which is why there is an open TODO item "Implement stored procedures".)
This manual page might be sharpened to avoid confusion ...
#Laurenz also provided an example. I would just use LEAST in the function to get identical functionality:
CREATE FUNCTION f_least(anyelement, anyelement)
RETURNS anyelement LANGUAGE sql IMMUTABLE AS
'SELECT LEAST($1, $2)';
Do not make it STRICT, that would be incorrect. LEAST(1, NULL) returns 1 and not NULL.
Even if STRICT was correct, I would not use it, because it can prevent function inlining.
Note that this function is limited to exactly two parameters while LEAST takes any number of parameters. You might overload the function to cover 3, 4 etc. input parameters. Or you could write a VARIADIC function for up to 100 parameters.
LEAST and GREATEST are not real functions; internally they are parsed as MinMaxExpr (see src/include/nodes/primnodes.h).
You could achieve what you want with a generic function like this:
CREATE FUNCTION my_least(anyelement, anyelement) RETURNS anyelement
LANGUAGE sql IMMUTABLE CALLED ON NULL INPUT
AS 'SELECT LEAST($1, $2)';
(thanks to Erwin Brandstetter for the CALLED ON NULL INPUT and the idea to use LEAST.)
Then you can create your aggregate as
CREATE AGGREGATE min(my_type) (sfunc = my_least, stype = my_type);
This will only work if there are comparison functions for my_type, otherwise you have to come up with a different my_least function.
I am trying to index documents to be searchable on their tag array.
CREATE INDEX doc_search_idx ON documents
USING gin(
to_tsvector('english', array_to_string(tags, ' ')) ||
to_tsvector('english', coalesce(notes, '')))
)
Where tags is a (ci)text[]. However, PG will refuse to index array_to_string because it is not always immutable.
PG::InvalidObjectDefinition: ERROR: functions in index expression must be marked IMMUTABLE
I've tried creating a homebrew array_to_string immutable function, but I feel like playing with fire as I don't know what I'm doing. Any way not to re-implement it?
Looks like I could just repackage the same function and label it immutable, but looks like there are risks when doing that.
How do I index the array for full-text search?
In my initial answer I suggested a plain cast to text: tags::text. However, while most casts to text from basic types are defined IMMUTABLE, this it is not the case for array types. Obviously because (quoting Tom Lane in a post to pgsql-general):
Because it's implemented via array_out/array_in rather than any more
direct method, and those are marked stable because they potentially
invoke non-immutable element I/O functions.
Bold emphasis mine.
We can work with that. The general case cannot be marked as IMMUTABLE. But for the case at hand (cast citext[] or text[] to text) we can safely assume immutability. Create a simple IMMUTABLE SQL function that wraps the function. However, the appeal of my simple solution is mostly gone now. You might as well wrap array_to_string() (like you already pondered) for which similar considerations apply.
For citext[] (create separate functions for text[] if needed):
Either (based on a plain cast to text):
CREATE OR REPLACE FUNCTION f_ciarr2text(citext[])
RETURNS text LANGUAGE sql IMMUTABLE AS 'SELECT $1::text';
This is faster.
Or (using array_to_string() for a result without curly braces):
CREATE OR REPLACE FUNCTION f_ciarr2text(citext[])
RETURNS text LANGUAGE sql IMMUTABLE AS $$SELECT array_to_string($1, ',')$$;
This is a bit more correct.
Then:
CREATE INDEX doc_search_idx ON documents USING gin (
to_tsvector('english', COALESCE(f_ciarr2text(tags), '')
|| ' ' || COALESCE(notes,'')));
I did not use the polymorphic type ANYARRAY like in your answer, because I know text[] or citext[] are safe, but I can't vouch for all other array types.
Tested in Postgres 9.4 and works for me.
I added a space between the two strings to avoid false positive matches across the concatenated strings. There is an example in the manual.
If you sometimes want to search just tags or just notes, consider a multicolumn index instead:
CREATE INDEX doc_search_idx ON documents USING gin (
to_tsvector('english', COALESCE(f_ciarr2text(tags), '')
, to_tsvector('english', COALESCE(notes,''));
The risks you are referring to apply to temporal functions mostly, which are used in the referenced question. If time zones (or just the type timestamptz) are involved, results are not actually immutable. We do not lie about immutability here. Our functions are actually IMMUTABLE. Postgres just can't tell from the general implementation it uses.
Related
Often people think they need text search, while similarity search with trigram indexes would be a better fit:
PostgreSQL LIKE query performance variations
Not relevant in this exact case, but while working with citext, consider this:
Index on column with data type citext not used
Here's my naive solution, to wrap it and call it immutable, as suspected.
CREATE FUNCTION immutable_array_to_string(arr ANYARRAY, sep TEXT)
RETURNS text
AS $$
SELECT array_to_string(arr, sep);
$$
LANGUAGE SQL
IMMUTABLE
;
I read about Exclusion Constraints in PostgreSQL but can't seem to find a way to use bitwise operators on bitstrings.
I have two columns (name text, value bit(8)). And I want to create a constraint that basically says this:
ADD CONSTRAINT route_method_overlap
EXCLUDE USING gist(name WITH =, value WITH &)
But this doesn't work since:
operator &(bit,bit) is not a member of operator family "gist_bit_ops"
I assume this is because the bit_ops operator & doesn't return boolean. But is there a way to do what I'm trying to do? Is there a way to coerce operator & to cast its return value as a boolean?
Currently using Postgres 9.1.4 with the "btree_gist" extension installed, all from the Ubuntu 12.04 repos. But the version doesn't matter. If there's fixes/updates upstream, I can install from the repos. I'm still in the design phase.
You installed the extension btree_gist. Without it, the example would already fail at name WITH =.
CREATE EXTENSION btree_gist;
The operator classes installed by btree_gist cover many operators. Unfortunately, the & operator is not among them. Obviously, because it does not return a boolean which would be expected of an operator to qualify.
Alternative solution
I would use a combination of a b-tree multi-column index (for speed) and a trigger instead. Consider this demo, tested on PostgreSQL 9.1:
CREATE TABLE t (
name text
, value bit(8)
);
INSERT INTO t VALUES ('a', B'10101010');
CREATE INDEX t_name_value_idx ON t (name, value);
CREATE OR REPLACE FUNCTION trg_t_name_value_inversion_prohibited()
RETURNS trigger
LANGUAGE plpgsql AS
$func$
BEGIN
IF EXISTS (
SELECT FROM t
WHERE (name, value) = (NEW.name, ~ NEW.value) -- example: exclude inversion
) THEN
RAISE EXCEPTION 'Your text here!';
END IF;
RETURN NEW;
END
$func$;
CREATE TRIGGER insup_bef_t_name_value_inversion_prohibited
BEFORE INSERT OR UPDATE OF name, value -- only involved columns relevant!
ON t
FOR EACH ROW
EXECUTE FUNCTION trg_t_name_value_inversion_prohibited();
INSERT INTO t VALUES ('a', ~ B'10101010'); -- fails with your error msg.
In Postgres 10 or older use instead:
...
EXECUTE PROCEDURE trg_t_name_value_inversion_prohibited();
See:
Trigger function does not exist, but I am pretty sure it does
~ is the inversion operator.
The extension btree_gist is not required in this scenario.
I restricted the trigger to INSERT OR UPDATE OF relevant columns for efficiency.
A check constraint wouldn't work. I quote the manual on CREATE TABLE:
Currently, CHECK expressions cannot contain subqueries nor refer to
variables other than columns of the current row.
Bold emphasis mine.
Should perform very well, actually better than the exclusion constraint, because maintenance of a b-tree index is cheaper than a GiST index. And the look-up with basic = operators should be faster than hypothetical look-ups with the & operator.
This solution is not as bullet-proof as an exclusion constraint, because triggers can more easily be circumvented - in a subsequent trigger on the same event for instance, or if the trigger is disabled temporarily. Be prepared to run extra checks on the whole table if such conditions apply.
More complex condition
The example trigger only catches the inversion of value. As you clarified in your comment, you actually need a condition like this instead:
IF EXISTS (
SELECT FROM t
WHERE name = NEW.name
AND value & NEW.value <> B'00000000'::bit(8)
) THEN
This condition is slightly more expensive, but can still use an index. The multi-column index from above would work - if you have use for it anyway. Or, more efficiently, a simple index on name:
CREATE INDEX t_name_idx ON t (name);
You commented that there can only be a maximum of 8 distinct rows per name, fewer in practice. So this should still be fast.
Ultimate INSERT performance
If INSERT performance is paramount, especially if many attempted INSERTs fail the condition, you could do more: create a materialized view that pre-aggregated value per name:
CREATE TABLE mv_t AS
SELECT name, bit_or(value) AS value
FROM t
GROUP BY 1
ORDER BY 1;
name is guaranteed to be unique here. I'd use a PRIMARY KEY on name to provide the index we're after:
ALTER TABLE mv_t SET (FILLFACTOR=90);
ALTER TABLE mv_t
ADD CONSTRAINT mv_t_pkey PRIMARY KEY(name);
Then your INSERT could look like this:
WITH i(n,v) AS (SELECT 'a'::text, B'10101010'::bit(8))
INSERT INTO t (name, value)
SELECT n, v
FROM i
LEFT JOIN mv_t m ON m.name = i.n
AND m.value & i.v <> B'00000000'::bit(8)
WHERE m.n IS NULL; -- alternative syntax for EXISTS (...)
The fillfactor is only useful if your table gets a lot of updates.
Update rows in the materialized view in a TRIGGER AFTER INSERT OR UPDATE OF name, value OR DELETE to keep it current. Cost and gain of additional objects have to be weighed carefully. Largely depends on your typical load.