How can I speed up a window query with string prefix matching? - postgresql

Question

I have a table with about 7.5million records and am trying to implement an autocomplete form based on said table, but the performance is pretty bad.
The schema (irrelevant fields omitted) is as follows:
COMPANIES
---------
sid (integer primary key)
world_hq_sid (integer)
name (varchar(64))
marketing_alias (varchar(64))
address_country_code (char(4))
address_state (varchar(64))
sort_order integer
search_weight integer
annual_sales integer
The fields passed in are the optional country_code and state, along with a search term. What I want is for the search term to match (case insensitive) the beginning of either name or marketing_alias. I want the top ten results, with those results that also match country and state at the top, then country only, then no state/country match. After that, I want the results sorted by sort_order.
Also, I only want one match per world_hq_sid. Finally, when I have the top match per world_hq_sid, I want the final results to be sorted by search_weight.
I'm using a window query to achieve the world_hq_sid part. Here is the query:
SELECT * FROM (
SELECT ROW_NUMBER() OVER (PARTITION BY world_hq_sid ORDER BY CASE WHEN address_country_code = 'US' AND address_state = 'CA' THEN 2 WHEN address_country_code = 'US' THEN 1 ELSE 0 END desc, sort_order asc) AS r,
companies.*
FROM companies
WHERE ((upper(name) LIKE upper('co%')) OR (upper(marketing_alias) LIKE upper('co%')))
) x
WHERE x.r = 1
ORDER BY CASE WHEN address_country_code = 'US' AND address_state = 'CA' THEN 2 WHEN address_state = 'CA' THEN 1 ELSE 0 END desc, search_weight asc, annual_sales desc
LIMIT 10;
I have normal btree indexes on address_state, address_country_code, world_hq_sid, sort_order, and search_weight.
I have the following indexes on the name and marketing_alias fields:
CREATE INDEX companies_alias_pattern_upper_idx ON companies(upper(marketing_alias) varchar_pattern_ops);
CREATE INDEX companies_name_pattern_upper_idx ON companies(upper(name) varchar_pattern_ops)
And here is the explain analyze when I pass CA as the state and 'co' as the search term
Limit (cost=676523.01..676523.03 rows=10 width=939) (actual time=18695.686..18695.687 rows=10 loops=1)
-> Sort (cost=676523.01..676526.67 rows=1466 width=939) (actual time=18695.686..18695.687 rows=10 loops=1)
Sort Key: x.search_weight, x.annual_sales
Sort Method: top-N heapsort Memory: 30kB
-> Subquery Scan on x (cost=665492.58..676491.33 rows=1466 width=939) (actual time=18344.715..18546.830 rows=151527 loops=1)
Filter: (x.r = 1)
Rows Removed by Filter: 20672
-> WindowAgg (cost=665492.58..672825.08 rows=293300 width=931) (actual time=18344.710..18511.625 rows=172199 loops=1)
-> Sort (cost=665492.58..666225.83 rows=293300 width=931) (actual time=18344.702..18359.145 rows=172199 loops=1)
Sort Key: companies.world_hq_sid, (CASE WHEN ((companies.address_state)::text = 'CA'::text) THEN 1 ELSE 0 END), companies.sort_order
Sort Method: quicksort Memory: 108613kB
-> Bitmap Heap Scan on companies (cost=17236.64..518555.98 rows=293300 width=931) (actual time=1861.665..17999.806 rows=172199 loops=1)
Recheck Cond: ((upper((name)::text) ~~ 'CO%'::text) OR (upper((marketing_alias)::text) ~~ 'CO%'::text))
Filter: ((upper((name)::text) ~~ 'CO%'::text) OR (upper((marketing_alias)::text) ~~ 'CO%'::text))
-> BitmapOr (cost=17236.64..17236.64 rows=196219 width=0) (actual time=1829.061..1829.061 rows=0 loops=1)
-> Bitmap Index Scan on companies_name_pattern_upper_idx (cost=0.00..8987.98 rows=97772 width=0) (actual time=971.331..971.331 rows=169390 loops=1)
Index Cond: ((upper((name)::text) ~>=~ 'CO'::text) AND (upper((name)::text) ~<~ 'CP'::text))
-> Bitmap Index Scan on companies_alias_pattern_upper_idx (cost=0.00..8102.02 rows=98447 width=0) (actual time=857.728..857.728 rows=170616 loops=1)
Index Cond: ((upper((marketing_alias)::text) ~>=~ 'CO'::text) AND (upper((marketing_alias)::text) ~<~ 'CP'::text))
I've bumped work_mem and shared_buffers to 100M.
As you can see, this query returns in 18 seconds. What is odd is that the results are all over the board for different starting characters, from 400ms (acceptable) to 30 seconds (very not acceptable). Postgres gurus, my question is, am I just expecting too much of postgresql to perform such a query quickly consistently? Is there a way I can speed this up?

Answer 1

select *
from (
select distinct on (world_hq_sid)
world_hq_sid,
(address_country_code = 'US')::int + (address_state = 'CA')::int address_weight,
sort_order,
search_weight, annual_sales,
sid, name, marketing_alias,
address_country_code, address_state
from companies
where
upper(name) LIKE upper('co%')
OR upper(marketing_alias) LIKE upper('co%')
order by 1, 2 desc, 3
) s
order by
address_weight desc,
search_weight,
annual_sales desc
limit 10

Answer 2

For autocomplete it's possible to use trigram search.
pg_trgm module.
CREATE EXTENSION pg_trgm;
ALTER TABLE companies ADD COLUMN name_trgm TEXT NULL;
UPDATE companies SET name_trgm = UPPER(name);
CREATE INDEX companies_name_trgm_gin_idx ON companies USING GIN (name_trgm gin_trgm_ops);