Single Column Table query much faster than with Multi Column Table - postgresql-performance

Question

I have a bunch of strings for which I like to find the best match in a table column. This column contains about 400,000 rows and the strings are no longer than 100 characters.
When I run the query on the whole table (8 columns in total) my query takes about 32 secs. All text columns have a GIN index.
with my_phrases(phrase) as (
values
('ABC'),
('123'),
('XYZ'),
('MNO'),
('KLM'),
('FOO'),
('AYE'),
('OPS')
)
select my_phrases.phrase, best_match.phrase, best_match.similarity
from my_phrases,
lateral (
select opm.phrase, similarity(opm.phrase, my_phrases.phrase) similarity
from my_table opm
where phrase % my_phrases.phrase
order by opm.phrase <-> my_phrases.phrase
limit 1
) best_match
order by my_phrases.phrase
;
Now when I copy the column phrase into a separate table and add a GIN index. The query becomes 400ms.
with my_phrases(phrase) as (
values
('ABC'),
('123'),
('XYZ'),
('MNO'),
('KLM'),
('FOO'),
('AYE'),
('OPS')
)
select my_phrases.phrase, best_match.phrase, best_match.similarity
from my_phrases,
lateral (
select phrase, similarity(phrase, my_phrases.phrase) similarity
from my_one_column_table
where phrase % section.phrase
order by phrase <-> my_phrases.phrase
limit 1
) best_match
order by my_phrases.phrase
;
Here is more info on the server:
select version(); ->
PostgreSQL 14.3 on aarch64-unknown-linux-gnu, compiled by gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-6), 64-bit
The multi-column table was created from a csv as follows:
create table my_table
(
id serial primary key,
p_id uuid,
p_name varchar,
b_id uuid,
b_name varchar,
pt_id uuid,
pt_name varchar,
phrase varchar
);
\copy my_table(p_id, p_name, b_id, b_name, pt_id, pt_name, phrase) from 'XXX.csv' csv header;
create index my_table__p_name__gin on my_table using gin(p_name gin_trgm_ops);
create index my_table__b_name__gin on my_table using gin(b_name gin_trgm_ops);
create index my_table__pt_name__gin on my_table using gin(pt_name gin_trgm_ops);
create index my_table__[hrase]__gin on my_table using gin(phrase gin_trgm_ops);
\d+ my_table
Column │ Type │ Collation │ Nullable │ Default │ Storage │ Compression │ Stats target │ Description
═══════════════════╪═══════════════════╪═══════════╪══════════╪═══════════════════════════════════════════════════════╪══════════╪═════════════╪══════════════╪═════════════
id │ integer │ │ not null │ nextval('my_table_id_seq'::regclass) │ plain │ │ │
p_id │ uuid │ │ │ │ plain │ │ │
p_name │ character varying │ │ │ │ extended │ │ │
b_id │ uuid │ │ │ │ plain │ │ │
b_name │ character varying │ │ │ │ extended │ │ │
pt_id │ uuid │ │ │ │ plain │ │ │
pt_name │ character varying │ │ │ │ extended │ │ │
phrase │ character varying │ │ │ │ extended │ │ │
Indexes:
"my_table_pkey" PRIMARY KEY, btree (id)
"my_table__b_name__gin" gin (b_name gin_trgm_ops)
"my_table__phrase__gin" gin (phrase gin_trgm_ops)
"my_table__phrase__idx" btree (phrase)
"my_table__p_name__gin" gin (p_name gin_trgm_ops)
"my_table__pt_name__gin" gin (pt_name gin_trgm_ops)
Access method: heap
For the slow query, the query plan is here
For the fast query, the query plan is here