Identical query 1000x times slower in production than development - postgresql

Question

I have a query. In production env it takes around 15seconds, while in development (using a almost up to date snapshot of the production database), it takes no longer than 100ms.
The database is not really under heavy load, less than 50 operations/second (probably less)
Production:
Nested Loop Anti Join (cost=999.94..2116.89 rows=1 width=2171) (actual time=16922.305..16922.305 rows=0 loops=1)
Join Filter: (auto_message_events.auto_message_id = a0.id)
Rows Removed by Join Filter: 1683
-> Nested Loop Left Join (cost=225.57..1311.15 rows=1 width=2235) (actual time=37.732..130.746 rows=5049 loops=1)
7.600..114.552 rows=51 loops=1)=202.12..329.72 rows=1 width=2235) (actual time=3--More--
-> Nested Loop (cost=0.28..86.34 rows=3 width=2194) (actual time=0.905..1.088 rows=3 loops=1)
-> Seq Scan on auto_messages a0 (cost=0.00..61.42 rows=3 width=2171) (actual time=0.880..1.009 rows=3 loops=1)
Filter: (((status)::text = 'live'::text) AND ((channel)::text = 'email'::text))
Rows Removed by Filter: 425
-> Index Only Scan using apps_pkey on apps a1 (cost=0.28..8.29 rows=1 width=23) (actual time=0.014..0.017 rows=1 loops=3)
Index Cond: (id = (a0.app_id)::text)
Heap Fetches: 0
-> Hash Join (cost=201.84..215.68 rows=1 width=313) (actual time=37.603..37.783 rows=17 loops=3)
Hash Cond: ((find_matching_visitors.id)::text = (v2.id)::text)
Join Filter: CASE WHEN a0.window_enabled THEN ((date_part('dow'::text, timezone(COALESCE((v2.location ->> 'timezone'::text), 'Europe/Paris'::text), now())) = ANY ((a0.window_days_of_week)::double precision[])) AND (timezone(COALESCE((v2.location ->> 'timezone'::text), 'Europe/Paris'::text), ('now'::cstring)::time with time zone) >= (a0.window_start)::time with time zone) AND (timezone(COALESCE((v2.location ->> 'timezone'::text), 'Europe/Paris'::text), ('now'::cstring)::time with time zone) <= (a0.window_end)::time with time zone)) ELSE true END
Rows Removed by Join Filter: 12
-> Function Scan on find_matching_visitors (cost=0.25..10.25 rows=1000 width=32) (actual time=29.358..29.393 rows=31 loops=3)
-> Hash (cost=201.54..201.54 rows=4 width=313) (actual time=8.037..8.037 rows=881 loops=3)
Buckets: 1024 Batches: 1 Memory Usage: 299kB
-> Index Scan using visitors_app_id_signed_up_index on visitors v2 (cost=0.55..201.54 rows=4 width=313) (actual time=0.036..7.259 rows=881 loops=3)
Index Cond: ((app_id)::text = (a1.id)::text)
Filter: ((NOT merged) AND (email IS NOT NULL) AND (NOT unsubscribed) AND (((type)::text = 'user'::text) OR ((type)::text = 'lead'::text)))
Rows Removed by Filter: 1558
-> Bitmap Heap Scan on auto_message_events a3 (cost=23.46..977.51 rows=392 width=4) (actual time=0.035..0.198 rows=99 loops=51)
Recheck Cond: (auto_message_id = a0.id)
Heap Blocks: exact=2499
-> Bitmap Index Scan on auto_message_events_auto_message_id_visitor_id_event_index (cost=0.00..23.36 rows=392 width=0) (actual time=0.023..0.023 rows=99 loops=51)
Index Cond: (auto_message_id = a0.id)
h=73) (actual time=3.320..3.320 rows=1 loops=5049)ost=774.36..790.02 rows=6 widt--More--
Recheck Cond: ((((visitor_id)::text = (v2.id)::text) AND ((event)::text = 'sent'::text)) OR (((event)::text = 'sent'::text) AND ((visitor_user_id)::text = (v2.user_id)::text)) OR (((event)::text = 'sent'::text) AND ((visitor_email)::text = (v2.email)::text)))
Heap Blocks: exact=5940
-> BitmapOr (cost=774.36..774.36 rows=6 width=0) (actual time=3.316..3.316 rows=0 loops=5049)
-> Bitmap Index Scan on auto_message_events_auto_message_id_visitor_id_event_index (cost=0.00..746.48 rows=1 width=0) (actual time=3.088..3.088 rows=2 loops=5049)
Index Cond: (((visitor_id)::text = (v2.id)::text) AND ((event)::text = 'sent'::text))
-> Bitmap Index Scan on auto_message_id_event_visitor_user_id_idx (cost=0.00..13.42 rows=3 width=0) (actual time=0.113..0.113 rows=2 loops=5049)
Index Cond: (((event)::text = 'sent'::text) AND ((visitor_user_id)::text = (v2.user_id)::text))
-> Bitmap Index Scan on auto_message_id_event_visitor_email_idx (cost=0.00..14.46 rows=3 width=0) (actual time=0.110..0.110 rows=2 loops=5049)
Index Cond: (((event)::text = 'sent'::text) AND ((visitor_email)::text = (v2.email)::text))
Planning time: 4.171 ms
Execution time: 16922.671 ms
Development
QUERY PLAN
Nested Loop Anti Join (cost=32.20..1602.94 rows=1 width=2175) (actual time=57.971..57.971 rows=0 loops=1)
Join Filter: (((auto_message_events.visitor_id)::text = (v2.id)::text) OR ((auto_message_events.visitor_user_id)::text = (v2.user_id)::text) OR ((auto_message_events.visitor_email)::text = (v2.email)::text))
Rows Removed by Join Filter: 4095
-> Nested Loop Left Join (cost=16.29..595.98 rows=1 width=2736) (actual time=37.186..52.374 rows=91 loops=1)
-> Nested Loop (cost=1.22..40.02 rows=1 width=2736) (actual time=37.134..52.245 rows=1 loops=1)
Join Filter: ((v2.id)::text = (find_matching_visitors.id)::text)
Rows Removed by Join Filter: 152
-> Nested Loop (cost=0.97..17.27 rows=1 width=2736) (actual time=0.258..0.621 rows=3 loops=1)
Join Filter: CASE WHEN a0.window_enabled THEN ((date_part('dow'::text, timezone(COALESCE((v2.location ->> 'timezone'::text), 'Europe/Paris'::text), now())) = ANY ((a0.window_days_of_week)::double precision[])) AND (timezone(COALESCE((v2.location ->> 'timezone'::text), 'Europe/Paris'::text), ('now'::cstring)::time with time zone) >= (a0.window_start)::time with time zone) AND (timezone(COALESCE((v2.location ->> 'timezone'::text), 'Europe/Paris'::text), ('now'::cstring)::time with time zone) <= (a0.window_end)::time with time zone)) ELSE true END
Rows Removed by Join Filter: 6
-> Nested Loop (cost=0.70..16.75 rows=1 width=834) (actual time=0.105..0.196 rows=3 loops=1)
-> Index Scan using email_idx on visitors v2 (cost=0.42..8.45 rows=1 width=810) (actual time=0.084..0.127 rows=3 loops=1)
Index Cond: (email IS NOT NULL)
Filter: ((NOT merged) AND (NOT unsubscribed) AND (((type)::text = 'user'::text) OR ((type)::text = 'lead'::text)))
-> Index Only Scan using apps_pkey on apps a1 (cost=0.28..8.29 rows=1 width=24) (actual time=0.019..0.020 rows=1 loops=3)
Index Cond: (id = (v2.app_id)::text)
Heap Fetches: 3
-> Index Scan using auto_messages_app_id_user_id_title_index on auto_messages a0 (cost=0.27..0.44 rows=1 width=2175) (actual time=0.057..0.100 rows=3 loops=3)
Index Cond: ((app_id)::text = (a1.id)::text)
Filter: (((status)::text = 'live'::text) AND ((channel)::text = 'email'::text))
Rows Removed by Filter: 49
-> Function Scan on find_matching_visitors (cost=0.25..10.25 rows=1000 width=32) (actual time=17.191..17.195 rows=51 loops=3)
-> Bitmap Heap Scan on auto_message_events a3 (cost=15.07..552.54 rows=342 width=4) (actual time=0.043..0.080 rows=91 loops=1)
Recheck Cond: (auto_message_id = a0.id)
Heap Blocks: exact=26
-> Bitmap Index Scan on auto_message_events_auto_message_id_visitor_id_event_index (cost=0.00..14.98 rows=342 width=0) (actual time=0.031..0.031 rows=91 loops=1)
Index Cond: (auto_message_id = a0.id)
-> Bitmap Heap Scan on auto_message_events (cost=15.91..508.98 rows=281 width=73) (actual time=0.022..0.040 rows=46 loops=91)
Recheck Cond: ((auto_message_id = a0.id) AND ((event)::text = 'sent'::text))
Heap Blocks: exact=1820
-> Bitmap Index Scan on auto_message_events_auto_message_id_visitor_id_event_index (cost=0.00..15.84 rows=281 width=0) (actual time=0.020..0.020 rows=52 loops=91)
Index Cond: ((auto_message_id = a0.id) AND ((event)::text = 'sent'::text))
Planning time: 4.882 ms
Execution time: 58.174 ms
UPDATE
Dynamic query function used in join:
CREATE OR REPLACE FUNCTION find_matching_visitors(app_id text, default_filters text[], custom_filters text[])
RETURNS TABLE (
id varchar
) AS
$body$
DECLARE
default_filterstring text;
custom_filterstring text;
default_filter_length integer;
custom_filter_length integer;
sql VARCHAR;
BEGIN
default_filter_length := COALESCE(array_length(default_filters, 1), 0);
custom_filter_length := COALESCE(array_length(custom_filters, 1), 0);
default_filterstring := array_to_string(default_filters, ' AND ');
custom_filterstring := array_to_string(custom_filters, ' AND ');
IF custom_filterstring = '' or custom_filterstring is null THEN
custom_filterstring := '1=1';
END IF;
IF default_filterstring = '' or default_filterstring is null THEN
default_filterstring := '1=1';
END IF;
sql := format('
SELECT v.id FROM visitors v
LEFT JOIN trackings t on v.id = t.visitor_id
WHERE v.app_id = ''%s''
group by v.id
having (%s) AND (%s)
', app_id, custom_filterstring, default_filterstring);
RETURN QUERY EXECUTE sql;
END;
$body$
LANGUAGE 'plpgsql';
Any ideas?