PostgreSQL performance difference with datetime comparison

PostgreSQL performance difference with datetime comparison - postgresql

I'm trying to optimize the performance of my PostgreSQL queries. I noticed a big change in the time required to execute my query when I change the datetime in my query by one second. I'm trying to figure out why there's this drastic change in performance with such a small change in the query. I ran an explain(analyze, buffers) and see there's a difference in how they are operating but I don't understand enough to determine what to do about it. Any help?
Here is the first query
SELECT avg(travel_time_all)
FROM tt_data
WHERE date_time >= '2014-01-01 08:00:00' and
date_time < '2014-01-01 8:14:13' and
(tmc = '118P04252' or tmc = '118P04253' or tmc = '118P04254' or tmc = '118P04255' or tmc = '118P04256')
group by tmc order by tmc
If I increase the later date_time by one second to 2014-01-01 8:14:14 and rerun the query, it drastically increases the execution time.
Here are the results of the explain (analyze, buffers) on the two queries. First query:
GroupAggregate (cost=6251.99..6252.01 rows=1 width=14) (actual time=0.829..0.829 rows=1 loops=1)
Buffers: shared hit=506
-> Sort (cost=6251.99..6252.00 rows=1 width=14) (actual time=0.823..0.823 rows=1 loops=1)
Sort Key: tmc
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=506
-> Bitmap Heap Scan on tt_data (cost=36.29..6251.98 rows=1 width=14) (actual time=0.309..0.817 rows=1 loops=1)
Recheck Cond: ((date_time >= '2014-01-01 08:00:00'::timestamp without time zone) AND (date_time < '2014-01-01 08:14:13'::timestamp without time zone))
Filter: ((tmc = '118P04252'::text) OR (tmc = '118P04253'::text) OR (tmc = '118P04254'::text) OR (tmc = '118P04255'::text) OR (tmc = '118P04256'::text))
Rows Removed by Filter: 989
Buffers: shared hit=506
-> Bitmap Index Scan on tt_data_2_date_time_idx (cost=0.00..36.29 rows=1572 width=0) (actual time=0.119..0.119 rows=990 loops=1)
Index Cond: ((date_time >= '2014-01-01 08:00:00'::timestamp without time zone) AND (date_time < '2014-01-01 08:14:13'::timestamp without time zone))
Buffers: shared hit=7
Total runtime: 0.871 ms
Below is the second query:
GroupAggregate (cost=6257.31..6257.34 rows=1 width=14) (actual time=52.444..52.444 rows=1 loops=1)
Buffers: shared hit=2693
-> Sort (cost=6257.31..6257.32 rows=1 width=14) (actual time=52.438..52.438 rows=1 loops=1)
Sort Key: tmc
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=2693
-> Bitmap Heap Scan on tt_data (cost=6253.28..6257.30 rows=1 width=14) (actual time=52.427..52.431 rows=1 loops=1)
Recheck Cond: ((date_time >= '2014-01-01 08:00:00'::timestamp without time zone) AND (date_time < '2014-01-01 08:14:14'::timestamp without time zone) AND ((tmc = '118P04252'::text) OR (tmc = '118P04253'::text) OR (tmc = '118P04254'::text) OR (...)
Rows Removed by Index Recheck: 5
Buffers: shared hit=2693
-> BitmapAnd (cost=6253.28..6253.28 rows=1 width=0) (actual time=52.410..52.410 rows=0 loops=1)
Buffers: shared hit=2689
-> Bitmap Index Scan on tt_data_2_date_time_idx (cost=0.00..36.31 rows=1574 width=0) (actual time=0.132..0.132 rows=990 loops=1)
Index Cond: ((date_time >= '2014-01-01 08:00:00'::timestamp without time zone) AND (date_time < '2014-01-01 08:14:14'::timestamp without time zone))
Buffers: shared hit=7
-> BitmapOr (cost=6216.71..6216.71 rows=271178 width=0) (actual time=52.156..52.156 rows=0 loops=1)
Buffers: shared hit=2682
-> Bitmap Index Scan on tt_data_2_tmc_idx (cost=0.00..1243.34 rows=54236 width=0) (actual time=8.439..8.439 rows=125081 loops=1)
Index Cond: (tmc = '118P04252'::text)
Buffers: shared hit=483
-> Bitmap Index Scan on tt_data_2_tmc_idx (cost=0.00..1243.34 rows=54236 width=0) (actual time=10.257..10.257 rows=156115 loops=1)
Index Cond: (tmc = '118P04253'::text)
Buffers: shared hit=602
-> Bitmap Index Scan on tt_data_2_tmc_idx (cost=0.00..1243.34 rows=54236 width=0) (actual time=6.867..6.867 rows=102318 loops=1)
Index Cond: (tmc = '118P04254'::text)
Buffers: shared hit=396
-> Bitmap Index Scan on tt_data_2_tmc_idx (cost=0.00..1243.34 rows=54236 width=0) (actual time=13.371..13.371 rows=160566 loops=1)
Index Cond: (tmc = '118P04255'::text)
Buffers: shared hit=619
-> Bitmap Index Scan on tt_data_2_tmc_idx (cost=0.00..1243.34 rows=54236 width=0) (actual time=13.218..13.218 rows=150709 loops=1)
Index Cond: (tmc = '118P04256'::text)
Buffers: shared hit=582
Total runtime: 52.507 ms
Any advice on how to make the second query as fast as the first? I'd like to increase this time interval by a greater amount but don't want the performance to decrease.

Related

Bitmap heap scan slow with same condition as index scan

I have a query with joins to rather large tables, but do not understand the slow performance of it.
Especially this part of the query plan seems weird to me (complete plan and query below):
-> Bitmap Heap Scan on order_line (cost=65.45..11521.37 rows=3228 width=20) (actual time=22.555..7764.120 rows=6250 loops=12)
Recheck Cond: (product_id = catalogue_product.id)
Heap Blocks: exact=71735
Buffers: shared hit=55299 read=16686
-> Bitmap Index Scan on order_line_product_id_e620902d (cost=0.00..64.65 rows=3228 width=0) (actual time=21.532..21.532 rows=6269 loops=12)
Index Cond: (product_id = catalogue_product.id)
Buffers: shared hit=143 read=107
Why does it need to recheck product_id = catalogue_product.id which is the same as in index and then take so much time?
As far as i understand recheck is needed if a) only part of the condition can be covered by index or b) bitmap is too big and must be compressed - but then there should be a lossy=x entry, right?
Complete query:
SELECT ("order_order"."date_placed" AT TIME ZONE 'UTC')::date, "partner_partner"."odoo_id", "catalogue_product"."odoo_id", SUM("order_line"."quantity") AS "orders"
FROM "order_line"
INNER JOIN "order_order" ON ("order_line"."order_id" = "order_order"."id")
INNER JOIN "catalogue_product" ON ("order_line"."product_id" = "catalogue_product"."id")
INNER JOIN "partner_stockrecord" ON ("order_line"."stockrecord_id" = "partner_stockrecord"."id")
INNER JOIN "partner_partner" ON ("partner_stockrecord"."partner_id" = "partner_partner"."id")
WHERE (("order_order"."date_placed" AT TIME ZONE 'UTC')::date IN ('2022-11-22'::DATE)
AND "catalogue_product"."odoo_id" IN (6241, 6499, 6500, 49195, 44753, 44754, 53427, 6452, 44755, 44787, 6427, 6428)
AND "partner_partner"."odoo_id" IS NOT NULL AND NOT ("order_order"."status" IN ('Pending', 'PaymentDeclined', 'Canceled')))
GROUP BY ("order_order"."date_placed" AT TIME ZONE 'UTC')::date, "partner_partner"."odoo_id", "catalogue_product"."odoo_id", "order_line"."id"
ORDER BY "order_line"."id" ASC
Complete plan:
GroupAggregate (cost=141002.93..141003.41 rows=16 width=24) (actual time=93629.346..93629.369 rows=52 loops=1)
Group Key: order_line.id, ((timezone('UTC'::text, order_order.date_placed))::date), partner_partner.odoo_id, catalogue_product.odoo_id
Buffers: shared hit=56537 read=16693
-> Sort (cost=141002.93..141002.97 rows=16 width=20) (actual time=93629.331..93629.335 rows=52 loops=1)
Sort Key: order_line.id, partner_partner.odoo_id, catalogue_product.odoo_id
Sort Method: quicksort Memory: 29kB
Buffers: shared hit=56537 read=16693
-> Hash Join (cost=2319.22..141002.61 rows=16 width=20) (actual time=859.917..93629.204 rows=52 loops=1)
Hash Cond: (partner_stockrecord.partner_id = partner_partner.id)
Buffers: shared hit=56537 read=16693
-> Nested Loop (cost=2318.11..141001.34 rows=16 width=24) (actual time=859.853..93628.903 rows=52 loops=1)
Buffers: shared hit=56536 read=16693
-> Hash Join (cost=2317.69..140994.41 rows=16 width=24) (actual time=859.824..93627.791 rows=52 loops=1)
Hash Cond: (order_line.order_id = order_order.id)
Buffers: shared hit=56328 read=16693
-> Nested Loop (cost=108.94..138731.32 rows=20700 width=20) (actual time=1.566..93206.434 rows=74999 loops=1)
Buffers: shared hit=55334 read=16686
-> Bitmap Heap Scan on catalogue_product (cost=43.48..87.52 rows=12 width=8) (actual time=0.080..0.183 rows=12 loops=1)
Recheck Cond: (odoo_id = ANY ('{6241,6499,6500,49195,44753,44754,53427,6452,44755,44787,6427,6428}'::integer[]))
Heap Blocks: exact=11
Buffers: shared hit=35
-> Bitmap Index Scan on catalogue_product_odoo_id_c5e41bad (cost=0.00..43.48 rows=12 width=0) (actual time=0.072..0.072 rows=12 loops=1)
Index Cond: (odoo_id = ANY ('{6241,6499,6500,49195,44753,44754,53427,6452,44755,44787,6427,6428}'::integer[]))
Buffers: shared hit=24
-> Bitmap Heap Scan on order_line (cost=65.45..11521.37 rows=3228 width=20) (actual time=22.555..7764.120 rows=6250 loops=12)
Recheck Cond: (product_id = catalogue_product.id)
Heap Blocks: exact=71735
Buffers: shared hit=55299 read=16686
-> Bitmap Index Scan on order_line_product_id_e620902d (cost=0.00..64.65 rows=3228 width=0) (actual time=21.532..21.532 rows=6269 loops=12)
Index Cond: (product_id = catalogue_product.id)
Buffers: shared hit=143 read=107
-> Hash (cost=2194.42..2194.42 rows=1147 width=12) (actual time=365.766..365.766 rows=1313 loops=1)
Buckets: 2048 Batches: 1 Memory Usage: 73kB
Buffers: shared hit=994 read=7
-> Index Scan using order_date_placed_utc_date_idx on order_order (cost=0.43..2194.42 rows=1147 width=12) (actual time=0.050..365.158 rows=1313 loops=1)
Index Cond: ((timezone('UTC'::text, date_placed))::date = '2022-11-22'::date)
Filter: ((status)::text <> ALL ('{Pending,PaymentDeclined,Canceled}'::text[]))
Rows Removed by Filter: 253
Buffers: shared hit=994 read=7
-> Index Scan using partner_stockrecord_pkey on partner_stockrecord (cost=0.41..0.43 rows=1 width=8) (actual time=0.017..0.017 rows=1 loops=52)
Index Cond: (id = order_line.stockrecord_id)
Buffers: shared hit=208
-> Hash (cost=1.05..1.05 rows=5 width=8) (actual time=0.028..0.028 rows=5 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
Buffers: shared hit=1
-> Seq Scan on partner_partner (cost=0.00..1.05 rows=5 width=8) (actual time=0.013..0.015 rows=5 loops=1)
Filter: (odoo_id IS NOT NULL)
Buffers: shared hit=1
Planning time: 3.275 ms
Execution time: 93629.781 ms

It doesn't have to do any rechecks. That line in the plan comes from the planner, not from the run-time part. (you can tell because if you just do EXPLAIN without ANALYZE, the line still appears.) At planning time, it doesn't know whether any of the bitmap will overflow, so it has to be prepared to do the recheck, even if that turns out not to be necessary to execute it at run time. The slowness almost certainly comes from the time spent reading 16686 random pages, which could be made clear by turning on track_io_timing.

Slow performance and index not taken into consideration in PostgreSQL

I'll post my query plan, view and indexes at the bottom of the page so that I can keep this question as clean as possible.
The issue I have is slow performance for a view that is not using indexes as I would expect them to be used. I have a table with around 7 million rows that I use as source for the view below.
I have added an index on eventdate which is being used as expected, but why is the index on manufacturerkey ignored? Which indexes would be more efficient?
Also, is it maybe this part to_char(fe.eventdate, 'HH24:MI'::text) AS hourminutes that hurts the performance?
Query plan: https://explain.dalibo.com/plan/Pvw
CREATE OR REPLACE VIEW public.v_test
AS SELECT df.facilityname,
dd.date,
dt.military_hour AS hour,
to_char(fe.eventdate, 'HH24:MI'::text) AS hourminutes,
df.tenantid,
df.tenantname,
dev.name AS event_type_name,
dtt.name AS ticket_type_name,
dde.name AS device_type_name,
count(*) AS count,
dl.country,
dl.state,
dl.district,
ds.systemmanufacturer
FROM fact_entriesexits fe
JOIN dim_facility df ON df.key = fe.facilitykey
JOIN dim_date dd ON dd.key = fe.datekey
JOIN dim_time dt ON dt.key = fe.timekey
LEFT JOIN dim_device dde ON dde.key = fe.devicekey
JOIN dim_eventtype dev ON dev.key = fe.eventtypekey
JOIN dim_tickettype dtt ON dtt.key = fe.tickettypekey
JOIN dim_licenseplate dl ON dl.key = fe.licenseplatekey
LEFT JOIN dim_systeminterface ds ON ds.key = fe.systeminterfacekey
WHERE fe.manufacturerkey = ANY (ARRAY[2, 1])
AND fe.eventdate >= '2022-01-01'
GROUP BY df.tenantname, df.tenantid, dl.region, dl.country, dl.state,
dl.district, df.facilityname, dev.name, dtt.name, dde.name,
ds.systemmanufacturer, dd.date, dt.military_hour, (to_char(fe.eventdate, 'HH24:MI'::text)), fe.licenseplatekey;
Here are the indexes the table fact_entriesexits contains:
CREATE INDEX idx_devicetype_fact_entriesexits_202008 ON public.fact_entriesexits_202008 USING btree (devicetype)
CREATE INDEX idx_etlsource_fact_entriesexits_202008 ON public.fact_entriesexits_202008 USING btree (etlsource)
CREATE INDEX idx_eventdate_fact_entriesexits_202008 ON public.fact_entriesexits_202008 USING btree (eventdate)
CREATE INDEX idx_fact_entriesexits_202008 ON public.fact_entriesexits_202008 USING btree (datekey)
CREATE INDEX idx_manufacturerkey_202008 ON public.fact_entriesexits_202008 USING btree (manufacturerkey)
Query plan:
Subquery Scan on v_lpr2 (cost=505358.60..508346.26 rows=17079 width=340) (actual time=85619.542..109797.440 rows=3008065 loops=1)
Buffers: shared hit=91037 read=366546, temp read=83669 written=83694
-> Finalize GroupAggregate (cost=505358.60..508175.47 rows=17079 width=359) (actual time=85619.539..109097.943 rows=3008065 loops=1)
Group Key: df.tenantname, df.tenantid, dl.region, dl.country, dl.state, dl.district, df.facilityname, dev.name, dtt.name, dde.name, ds.systemmanufacturer, dd.date, dt.military_hour, (to_char(fe.eventdate, 'HH24:MI'::text)), fe.licenseplatekey
Buffers: shared hit=91037 read=366546, temp read=83669 written=83694
-> Gather Merge (cost=505358.60..507392.70 rows=14232 width=359) (actual time=85619.507..105395.429 rows=3308717 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=91037 read=366546, temp read=83669 written=83694
-> Partial GroupAggregate (cost=504358.57..504749.95 rows=7116 width=359) (actual time=85169.770..94043.715 rows=1102906 loops=3)
Group Key: df.tenantname, df.tenantid, dl.region, dl.country, dl.state, dl.district, df.facilityname, dev.name, dtt.name, dde.name, ds.systemmanufacturer, dd.date, dt.military_hour, (to_char(fe.eventdate, 'HH24:MI'::text)), fe.licenseplatekey
Buffers: shared hit=91037 read=366546, temp read=83669 written=83694
-> Sort (cost=504358.57..504376.36 rows=7116 width=351) (actual time=85169.748..91995.088 rows=1500405 loops=3)
Sort Key: df.tenantname, df.tenantid, dl.region, dl.country, dl.state, dl.district, df.facilityname, dev.name, dtt.name, dde.name, ds.systemmanufacturer, dd.date, dt.military_hour, (to_char(fe.eventdate, 'HH24:MI'::text)), fe.licenseplatekey
Sort Method: external merge Disk: 218752kB
Buffers: shared hit=91037 read=366546, temp read=83669 written=83694
-> Hash Left Join (cost=3904.49..503903.26 rows=7116 width=351) (actual time=52.894..46338.295 rows=1500405 loops=3)
Hash Cond: (fe.systeminterfacekey = ds.key)
Buffers: shared hit=90979 read=366546
-> Hash Join (cost=3886.89..503848.87 rows=7116 width=321) (actual time=52.458..44551.012 rows=1500405 loops=3)
Hash Cond: (fe.licenseplatekey = dl.key)
Buffers: shared hit=90943 read=366546
-> Hash Left Join (cost=3849.10..503792.31 rows=7116 width=269) (actual time=51.406..43869.673 rows=1503080 loops=3)
Hash Cond: (fe.devicekey = dde.key)
Buffers: shared hit=90870 read=366546
-> Hash Join (cost=3405.99..503330.51 rows=7116 width=255) (actual time=47.077..43258.069 rows=1503080 loops=3)
Hash Cond: (fe.timekey = dt.key)
Buffers: shared hit=90021 read=366546
-> Hash Join (cost=570.97..500476.80 rows=7116 width=257) (actual time=6.869..42345.723 rows=1503080 loops=3)
Hash Cond: (fe.datekey = dd.key)
Buffers: shared hit=87348 read=366546
-> Hash Join (cost=166.75..500053.90 rows=7116 width=257) (actual time=2.203..41799.463 rows=1503080 loops=3)
Hash Cond: (fe.facilitykey = df.key)
Buffers: shared hit=86787 read=366546
-> Hash Join (cost=2.72..499871.14 rows=7116 width=224) (actual time=0.362..41103.372 rows=1503085 loops=3)
Hash Cond: (fe.tickettypekey = dtt.key)
Buffers: shared hit=86427 read=366546
-> Hash Join (cost=1.14..499722.81 rows=54741 width=214) (actual time=0.311..40595.537 rows=1503085 loops=3)
Hash Cond: (fe.eventtypekey = dev.key)
Buffers: shared hit=86424 read=366546
-> Append (cost=0.00..494830.25 rows=1824733 width=40) (actual time=0.266..40015.860 rows=1503085 loops=3)
Buffers: shared hit=86421 read=366546
-> Parallel Seq Scan on fact_entriesexits fe (cost=0.00..0.00 rows=1 width=40) (actual time=0.001..0.001 rows=0 loops=3)
Filter: ((manufacturerkey = ANY ('{2,1}'::integer[])) AND (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone))
-> Parallel Index Scan using idx_eventdate_fact_entriesexits_202101 on fact_entriesexits_202101 fe_25 (cost=0.42..4.28 rows=1 width=40) (actual time=0.005..0.006 rows=0 loops=3)
Index Cond: (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone)
Filter: (manufacturerkey = ANY ('{2,1}'::integer[]))
Buffers: shared hit=3
-> Parallel Index Scan using idx_eventdate_fact_entriesexits_202102 on fact_entriesexits_202102 fe_26 (cost=0.42..4.27 rows=1 width=40) (actual time=0.005..0.006 rows=0 loops=3)
Index Cond: (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone)
Filter: (manufacturerkey = ANY ('{2,1}'::integer[]))
Buffers: shared hit=3
-> Parallel Index Scan using idx_eventdate_fact_entriesexits_202103 on fact_entriesexits_202103 fe_27 (cost=0.42..4.24 rows=1 width=40) (actual time=0.007..0.007 rows=0 loops=3)
Index Cond: (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone)
Filter: (manufacturerkey = ANY ('{2,1}'::integer[]))
Buffers: shared hit=3
-> Parallel Index Scan using idx_eventdate_fact_entriesexits_202104 on fact_entriesexits_202104 fe_28 (cost=0.42..4.05 rows=1 width=40) (actual time=0.006..0.006 rows=0 loops=3)
Index Cond: (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone)
Filter: (manufacturerkey = ANY ('{2,1}'::integer[]))
Buffers: shared hit=3
-> Parallel Index Scan using idx_eventdate_fact_entriesexits_202105 on fact_entriesexits_202105 fe_29 (cost=0.43..4.12 rows=1 width=40) (actual time=0.006..0.006 rows=0 loops=3)
Index Cond: (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone)
Filter: (manufacturerkey = ANY ('{2,1}'::integer[]))
Buffers: shared hit=3
-> Parallel Index Scan using idx_eventdate_fact_entriesexits_202106 on fact_entriesexits_202106 fe_30 (cost=0.43..4.19 rows=1 width=40) (actual time=0.005..0.006 rows=0 loops=3)
Index Cond: (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone)
Filter: (manufacturerkey = ANY ('{2,1}'::integer[]))
Buffers: shared hit=3
-> Parallel Index Scan using idx_eventdate_fact_entriesexits_202107 on fact_entriesexits_202107 fe_31 (cost=0.43..4.28 rows=1 width=40) (actual time=0.005..0.006 rows=0 loops=3)
Index Cond: (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone)
Filter: (manufacturerkey = ANY ('{2,1}'::integer[]))
Buffers: shared hit=3
-> Parallel Index Scan using idx_eventdate_fact_entriesexits_202108 on fact_entriesexits_202108 fe_32 (cost=0.43..3.83 rows=1 width=40) (actual time=0.007..0.007 rows=0 loops=3)
Index Cond: (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone)
Filter: (manufacturerkey = ANY ('{2,1}'::integer[]))
Buffers: shared hit=3
-> Parallel Index Scan using idx_eventdate_fact_entriesexits_202109 on fact_entriesexits_202109 fe_33 (cost=0.43..3.40 rows=1 width=40) (actual time=0.006..0.007 rows=0 loops=3)
Index Cond: (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone)
Filter: (manufacturerkey = ANY ('{2,1}'::integer[]))
Buffers: shared hit=3
-> Parallel Index Scan using idx_eventdate_fact_entriesexits_202110 on fact_entriesexits_202110 fe_34 (cost=0.43..2.77 rows=1 width=40) (actual time=0.005..0.005 rows=0 loops=3)
Index Cond: (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone)
Filter: (manufacturerkey = ANY ('{2,1}'::integer[]))
Buffers: shared hit=3
-> Parallel Index Scan using idx_eventdate_fact_entriesexits_202111 on fact_entriesexits_202111 fe_35 (cost=0.43..3.21 rows=1 width=40) (actual time=0.005..0.005 rows=0 loops=3)
Index Cond: (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone)
Filter: (manufacturerkey = ANY ('{2,1}'::integer[]))
Buffers: shared hit=3
-> Parallel Index Scan using idx_eventdate_fact_entriesexits_202112 on fact_entriesexits_202112 fe_36 (cost=0.43..3.45 rows=1 width=40) (actual time=0.004..0.004 rows=0 loops=3)
Index Cond: (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone)
Filter: (manufacturerkey = ANY ('{2,1}'::integer[]))
Buffers: shared hit=3
-> Parallel Seq Scan on fact_entriesexits_202201 fe_37 (cost=0.00..382550.76 rows=445931 width=40) (actual time=0.032..39090.092 rows=379902 loops=3)
Filter: ((manufacturerkey = ANY ('{2,1}'::integer[])) AND (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 298432
Buffers: shared hit=3286 read=366546
-> Parallel Seq Scan on fact_entriesexits_202204 fe_38 (cost=0.00..39567.99 rows=469653 width=40) (actual time=0.015..242.895 rows=375639 loops=3)
Filter: ((manufacturerkey = ANY ('{2,1}'::integer[])) AND (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 158868
Buffers: shared hit=29546
-> Parallel Seq Scan on fact_entriesexits_202202 fe_39 (cost=0.00..30846.99 rows=437343 width=40) (actual time=0.019..230.952 rows=357451 loops=3)
Filter: ((manufacturerkey = ANY ('{2,1}'::integer[])) AND (eventdate >= '2022-01-01 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 98708
Buffers: shared hit=22294

I think you'll get the most benefit out of creating a composite index for querying with both eventdate and manufacturerkey; e.g.:
CREATE INDEX idx_manufacturerkey_eventdate_202008
ON public.fact_entriesexits_202008 USING btree (manufacturerkey, eventdate)
Since it's a composite index, put whatever column you're more likely to query by alone on the left side. You can remove the other index for that column, since it will be covered by the composite index.
As for the to_char on evendate, while you could make a special index for that calculation, you might be able to get better performance by splitting the query up into a grouped CTE and a join. In other words, limit the group by to the columns that actually define your unique groups, and then join that query with the tables that you need to get the final selection of columns.

Your slowest seq scan step is returning over half the rows of its partition, removing 298432 and returning 379902. (times around 3 each due to parallel workers). An index is unlikely to be helpful when returning so much of the table rows anyway.
Note that that partition also seems to be massively bloated. It is hard to see why else it would be so slow, and require so many buffer reads compared to the number of rows.

Postgres hash join batches explosion

We are having some struggle identifying why Postgres is using too much batches to resolve a join.
Here it is the output of explain analyze of a problematic execution:
https://explain.dalibo.com/plan/xNJ#plan
Limit (cost=20880.87..20882.91 rows=48 width=205) (actual time=10722.953..10723.358 rows=48 loops=1)
-> Unique (cost=20880.87..21718.12 rows=19700 width=205) (actual time=10722.951..10723.356 rows=48 loops=1)
-> Sort (cost=20880.87..20930.12 rows=19700 width=205) (actual time=10722.950..10722.990 rows=312 loops=1)
Sort Key: titlemetadata_titlemetadata.creation_date DESC, titlemetadata_titlemetadata.id, titlemetadata_titlemetadata.title_type, titlemetadata_titlemetadata.original_title, titlemetadata_titlemetadata.alternative_ids, titlemetadata_titlemetadata.metadata,
titlemetadata_titlemetadata.is_adult, titlemetadata_titlemetadata.is_kids, titlemetadata_titlemetadata.last_modified, titlemetadata_titlemetadata.year, titlemetadata_titlemetadata.runtime, titlemetadata_titlemetadata.rating, titlemetadata_titlemetadata.video_provider, tit
lemetadata_titlemetadata.series_id_id, titlemetadata_titlemetadata.season_number, titlemetadata_titlemetadata.episode_number
Sort Method: quicksort Memory: 872kB
-> Hash Right Join (cost=13378.20..19475.68 rows=19700 width=205) (actual time=1926.352..10709.970 rows=2909 loops=1)
Hash Cond: (t4.titlemetadata_id = t3.id)
Filter: ((hashed SubPlan 1) OR (hashed SubPlan 2))
Rows Removed by Filter: 63248
-> Seq Scan on video_provider_offer t4 (cost=0.00..5454.90 rows=66290 width=16) (actual time=0.024..57.893 rows=66390 loops=1)
-> Hash (cost=11314.39..11314.39 rows=22996 width=221) (actual time=489.530..489.530 rows=60096 loops=1)
Buckets: 65536 (originally 32768) Batches: 32768 (originally 1) Memory Usage: 11656kB
-> Hash Right Join (cost=5380.95..11314.39 rows=22996 width=221) (actual time=130.024..225.271 rows=60096 loops=1)
Hash Cond: (video_provider_offer.titlemetadata_id = titlemetadata_titlemetadata.id)
-> Seq Scan on video_provider_offer (cost=0.00..5454.90 rows=66290 width=16) (actual time=0.011..32.950 rows=66390 loops=1)
-> Hash (cost=5129.28..5129.28 rows=20133 width=213) (actual time=129.897..129.897 rows=55793 loops=1)
Buckets: 65536 (originally 32768) Batches: 2 (originally 1) Memory Usage: 7877kB
-> Merge Left Join (cost=1.72..5129.28 rows=20133 width=213) (actual time=0.041..93.057 rows=55793 loops=1)
Merge Cond: (titlemetadata_titlemetadata.id = t3.series_id_id)
-> Index Scan using titlemetadata_titlemetadata_pkey on titlemetadata_titlemetadata (cost=1.30..4130.22 rows=20133 width=205) (actual time=0.028..62.949 rows=43921 loops=1)
Filter: ((NOT is_adult) AND (NOT (hashed SubPlan 3)) AND (((title_type)::text = 'MOV'::text) OR ((title_type)::text = 'TVS'::text) OR ((title_type)::text = 'TVP'::text) OR ((title_type)::text = 'EVT'::text)))
Rows Removed by Filter: 14121
SubPlan 3
-> Seq Scan on cable_operator_cableoperatorexcludedtitle u0_2 (cost=0.00..1.01 rows=1 width=8) (actual time=0.006..0.006 rows=0 loops=1)
Filter: (cable_operator_id = 54)
-> Index Scan using titlemetadata_titlemetadata_series_id_id_73453db4_uniq on titlemetadata_titlemetadata t3 (cost=0.41..3901.36 rows=58037 width=16) (actual time=0.011..9.375 rows=12887 loops=1)
SubPlan 1
-> Hash Join (cost=44.62..885.73 rows=981 width=8) (actual time=0.486..36.806 rows=5757 loops=1)
Hash Cond: (w2.device_id = w3.id)
-> Nested Loop (cost=43.49..866.20 rows=2289 width=16) (actual time=0.441..33.096 rows=20180 loops=1)
-> Nested Loop (cost=43.06..414.98 rows=521 width=8) (actual time=0.426..9.952 rows=2909 loops=1)
Join Filter: (w1.id = w0.video_provider_id)
-> Nested Loop (cost=42.65..54.77 rows=13 width=24) (actual time=0.399..0.532 rows=15 loops=1)
-> HashAggregate (cost=42.50..42.95 rows=45 width=16) (actual time=0.390..0.403 rows=45 loops=1)
Group Key: v0.id
-> Nested Loop (cost=13.34..42.39 rows=45 width=16) (actual time=0.095..0.364 rows=45 loops=1)
-> Hash Semi Join (cost=13.19..32.72 rows=45 width=8) (actual time=0.084..0.229 rows=45 loops=1)
Hash Cond: (v1.id = u0.id)
-> Seq Scan on cable_operator_cableoperatorprovider v1 (cost=0.00..17.36 rows=636 width=16) (actual time=0.010..0.077 rows=636 loops=1)
-> Hash (cost=12.63..12.63 rows=45 width=8) (actual time=0.046..0.046 rows=45 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Index Scan using cable_operator_cableoperatorprovider_4d6e54b3 on cable_operator_cableoperatorprovider u0 (cost=0.28..12.63 rows=45 width=8) (actual time=0.016..0.035 rows=45 loops=1)
Index Cond: (cable_operator_id = 54)
-> Index Only Scan using video_provider_videoprovider_pkey on video_provider_videoprovider v0 (cost=0.15..0.20 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=45)
Index Cond: (id = v1.provider_id)
Heap Fetches: 45
-> Index Scan using video_provider_videoprovider_pkey on video_provider_videoprovider w1 (cost=0.15..0.25 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (id = v0.id)
Filter: ((video_provider_type)::text = 'VOD'::text)
Rows Removed by Filter: 1
-> Index Scan using video_provider_offer_da942d2e on video_provider_offer w0 (cost=0.42..27.22 rows=39 width=16) (actual time=0.026..0.585 rows=194 loops=15)
Index Cond: (video_provider_id = v0.id)
Filter: (((end_date > '2021-09-02 19:23:00-03'::timestamp with time zone) OR (end_date IS NULL)) AND (access_criteria && '{vtv_mas,TBX_LOGIN,urn:spkg:tve:fox-premium,urn:tve:mcp,AMCHD,AMC_CONSORCIO,ANIMAL_PLANET,ASUNTOS_PUBLI
COS,ASUNTOS_PUBLICOS_CONSORCIO,CINECANALLIVE,CINECANAL_CONSORCIO,DISCOVERY,DISCOVERY_KIDS_CONSORCIO,DISCOVERY_KIDS_OD,DISNEY,DISNEY_CH_CONSORCIO,DISNEY_XD,DISNEY_XD_CONSORCIO,EL_CANAL_HD,EL_CANAL_HD_CONSORCIO,EL_GOURMET_CONSORCIO,ESPN,ESPN2_HD_CONSORCIO,ESPN3_HD_CONSORCIO
,ESPNMAS_HD_CONSORCIO,ESPN_BASIC,ESPN_HD_CONSORCIO,ESPN_PLAY,EUROPALIVE,EUROPA_EUROPA,EUROPA_EUROPA_CONSORCIO,FILMANDARTS_DISPOSITIVOS,FILMS_ARTS,FILM_AND_ARTS_CONSORCIO,FOXLIFE,FOX_LIFE_CONSORCIO,FOX_SPORTS_1_DISPOSITIVOS,FOX_SPORTS_2_DISPOSITIVOS,FOX_SPORTS_2_HD_CONSORC
IO,FOX_SPORTS_3_DISPOSITIVOS,FOX_SPORTS_3_HD_CONSORCIO,FOX_SPORTS_HD_CONSORCIO,FRANCE24_DISPOSITIVOS,FRANCE_24_CONSORCIO,GOURMET,GOURMET_DISPOSITIVOS,HOME_HEALTH,INVESTIGATION_DISCOVERY,MAS_CHIC,NATGEOKIDS_DISPOSITIVOS,NATGEO_CONSORCIO,NATGEO_DISPOSITIVOS,NATGEO_KIDS_CONS
ORCIO,PASIONES,PASIONES_CONSORCIO,SVOD_TYC_BASIC,TBX_LOGIN,TCC_2_CONSORCIO,TCC_2_HD,TLC,TVE,TVE_CONSORCIO,TYC_SPORTS_CONSORCIO,VTV_LIVE,clarosports,discoverykids,espnplay_south_alt,urn:spkg:tve:fox-basic,urn:tve:babytv,urn:tve:cinecanal,urn:tve:discoverykids,urn:tve:foxli
fe,urn:tve:fp,urn:tve:fx,urn:tve:natgeo,urn:tve:natgeokids,urn:tve:natgeowild,urn:tve:thefilmzone}'::character varying(50)[]) AND ((((content_type)::text = 'VOD'::text) AND ((start_date < '2021-09-02 19:23:00-03'::timestamp with time zone) OR (start_date IS NULL))) OR ((c
ontent_type)::text = 'LIV'::text)))
Rows Removed by Filter: 5
-> Index Only Scan using video_provider_offer_devices_offer_id_device_id_key on video_provider_offer_devices w2 (cost=0.42..0.81 rows=6 width=16) (actual time=0.004..0.007 rows=7 loops=2909)
Index Cond: (offer_id = w0.id)
Heap Fetches: 17828
-> Hash (cost=1.10..1.10 rows=3 width=8) (actual time=0.029..0.029 rows=2 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on platform_device_device w3 (cost=0.00..1.10 rows=3 width=8) (actual time=0.024..0.027 rows=2 loops=1)
Filter: ((device_code)::text = ANY ('{ANDROID,ott_dual_tcc,ott_k2_tcc}'::text[]))
Rows Removed by Filter: 5
SubPlan 2
-> Hash Join (cost=44.62..885.73 rows=981 width=8) (actual time=0.410..33.580 rows=5757 loops=1)
Hash Cond: (w2_1.device_id = w3_1.id)
-> Nested Loop (cost=43.49..866.20 rows=2289 width=16) (actual time=0.375..29.886 rows=20180 loops=1)
-> Nested Loop (cost=43.06..414.98 rows=521 width=8) (actual time=0.366..9.134 rows=2909 loops=1)
Join Filter: (w1_1.id = w0_1.video_provider_id)
-> Nested Loop (cost=42.65..54.77 rows=13 width=24) (actual time=0.343..0.476 rows=15 loops=1)
-> HashAggregate (cost=42.50..42.95 rows=45 width=16) (actual time=0.333..0.347 rows=45 loops=1)
Group Key: v0_1.id
-> Nested Loop (cost=13.34..42.39 rows=45 width=16) (actual time=0.083..0.311 rows=45 loops=1)
-> Hash Semi Join (cost=13.19..32.72 rows=45 width=8) (actual time=0.076..0.202 rows=45 loops=1)
Hash Cond: (v1_1.id = u0_1.id)
-> Seq Scan on cable_operator_cableoperatorprovider v1_1 (cost=0.00..17.36 rows=636 width=16) (actual time=0.005..0.057 rows=636 loops=1)
-> Hash (cost=12.63..12.63 rows=45 width=8) (actual time=0.038..0.038 rows=45 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Index Scan using cable_operator_cableoperatorprovider_4d6e54b3 on cable_operator_cableoperatorprovider u0_1 (cost=0.28..12.63 rows=45 width=8) (actual time=0.007..0.020 rows=45 loops=1)
Index Cond: (cable_operator_id = 54)
-> Index Only Scan using video_provider_videoprovider_pkey on video_provider_videoprovider v0_1 (cost=0.15..0.20 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=45)
Index Cond: (id = v1_1.provider_id)
Heap Fetches: 45
-> Index Scan using video_provider_videoprovider_pkey on video_provider_videoprovider w1_1 (cost=0.15..0.25 rows=1 width=8) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (id = v0_1.id)
Filter: ((video_provider_type)::text = 'VOD'::text)
Rows Removed by Filter: 1
-> Index Scan using video_provider_offer_da942d2e on video_provider_offer w0_1 (cost=0.42..27.22 rows=39 width=16) (actual time=0.022..0.536 rows=194 loops=15)
Index Cond: (video_provider_id = v0_1.id)
Filter: (((end_date > '2021-09-02 19:23:00-03'::timestamp with time zone) OR (end_date IS NULL)) AND (access_criteria && '{vtv_mas,TBX_LOGIN,urn:spkg:tve:fox-premium,urn:tve:mcp,AMCHD,AMC_CONSORCIO,ANIMAL_PLANET,ASUNTOS_PUBLI
COS,ASUNTOS_PUBLICOS_CONSORCIO,CINECANALLIVE,CINECANAL_CONSORCIO,DISCOVERY,DISCOVERY_KIDS_CONSORCIO,DISCOVERY_KIDS_OD,DISNEY,DISNEY_CH_CONSORCIO,DISNEY_XD,DISNEY_XD_CONSORCIO,EL_CANAL_HD,EL_CANAL_HD_CONSORCIO,EL_GOURMET_CONSORCIO,ESPN,ESPN2_HD_CONSORCIO,ESPN3_HD_CONSORCIO
,ESPNMAS_HD_CONSORCIO,ESPN_BASIC,ESPN_HD_CONSORCIO,ESPN_PLAY,EUROPALIVE,EUROPA_EUROPA,EUROPA_EUROPA_CONSORCIO,FILMANDARTS_DISPOSITIVOS,FILMS_ARTS,FILM_AND_ARTS_CONSORCIO,FOXLIFE,FOX_LIFE_CONSORCIO,FOX_SPORTS_1_DISPOSITIVOS,FOX_SPORTS_2_DISPOSITIVOS,FOX_SPORTS_2_HD_CONSORC
IO,FOX_SPORTS_3_DISPOSITIVOS,FOX_SPORTS_3_HD_CONSORCIO,FOX_SPORTS_HD_CONSORCIO,FRANCE24_DISPOSITIVOS,FRANCE_24_CONSORCIO,GOURMET,GOURMET_DISPOSITIVOS,HOME_HEALTH,INVESTIGATION_DISCOVERY,MAS_CHIC,NATGEOKIDS_DISPOSITIVOS,NATGEO_CONSORCIO,NATGEO_DISPOSITIVOS,NATGEO_KIDS_CONS
ORCIO,PASIONES,PASIONES_CONSORCIO,SVOD_TYC_BASIC,TBX_LOGIN,TCC_2_CONSORCIO,TCC_2_HD,TLC,TVE,TVE_CONSORCIO,TYC_SPORTS_CONSORCIO,VTV_LIVE,clarosports,discoverykids,espnplay_south_alt,urn:spkg:tve:fox-basic,urn:tve:babytv,urn:tve:cinecanal,urn:tve:discoverykids,urn:tve:foxli
fe,urn:tve:fp,urn:tve:fx,urn:tve:natgeo,urn:tve:natgeokids,urn:tve:natgeowild,urn:tve:thefilmzone}'::character varying(50)[]) AND ((((content_type)::text = 'VOD'::text) AND ((start_date < '2021-09-02 19:23:00-03'::timestamp with time zone) OR (start_date IS NULL))) OR ((c
ontent_type)::text = 'LIV'::text)))
Rows Removed by Filter: 5
-> Index Only Scan using video_provider_offer_devices_offer_id_device_id_key on video_provider_offer_devices w2_1 (cost=0.42..0.81 rows=6 width=16) (actual time=0.003..0.006 rows=7 loops=2909)
Index Cond: (offer_id = w0_1.id)
Heap Fetches: 17828
-> Hash (cost=1.10..1.10 rows=3 width=8) (actual time=0.015..0.015 rows=2 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on platform_device_device w3_1 (cost=0.00..1.10 rows=3 width=8) (actual time=0.010..0.011 rows=2 loops=1)
Filter: ((device_code)::text = ANY ('{ANDROID,ott_dual_tcc,ott_k2_tcc}'::text[]))
Rows Removed by Filter: 5
Planning time: 8.255 ms
Execution time: 10723.830 ms
(100 rows)
The weird part is that the same query, sometimes just uses a single batch. Here is an example: https://explain.dalibo.com/plan/zTv#plan
Here is the work_mem being used:
show work_mem;
work_mem
----------
8388kB
(1 row)
I'm not interested in changing the query to be more performant, but in understanding why is the different behavior.
I've found this thread apparently related with this, but I don't quite understand what are they talking about: https://www.postgresql.org/message-id/flat/CA%2BhUKGKWWmf%3DWELLG%3DaUGbcugRaSQbtm0tKYiBut-B2rVKX63g%40mail.gmail.com
Can anyone tell me why is this different behavior? The underlying data is the same in both cases.

If the hash is done in memory, there will only be a single batch.
A difference with the original hash batch numbers is due to Postgres choosing to increase the number of batches in order to reduce memory consumption.
You might find this EXPLAIN glossary useful (disclaimer: I'm one of the authors), here is the page on Hash Batches which also links to the PostgreSQL source code (it's very nicely documented in plain English).
While not a perfect heuristic, you can see that the memory required for the operations with multiple batches are around or above your work_mem setting. They can be lower than it, due to operations on disk generally requiring less memory overall.
I'm not 100% sure why in your exact case one was chosen over the other, but it does look like there are some very slight row estimate differences, which might be a good place to start.
As of PostgreSQL 13 there is also now a hash_mem_multiplier setting that can be used to give more memory to hashes without doing so for other operations (like sorts).

We where able to solve the problem just by doing VACUUM FULL ANALYZE;.
After that, everything started to work as expected (https://explain.depesz.com/s/eoqH#html)
Side note: we where not aware that we should do this on daily basis.

Append Cost very high on Partitioned table

I have a query joining two tables partitioned on timestamp column. Both tables are filtered on current date partition. But query is unusually slow with APPEND Cost of the driving table very high.
Query and Plan : https://explain.dalibo.com/plan/wVA
Nested Loop (cost=0.56..174042.82 rows=16 width=494) (actual time=0.482..20.133 rows=1713 loops=1)
Output: tran.transaction_status, mgwor.apx_transaction_id, org.organisation_name, mgwor.order_status, mgwor.request_date, mgwor.response_date, (date_part('epoch'::text, mgwor.response_date) - date_part('epoch'::text, mgwor.request_date))
Buffers: shared hit=5787 dirtied=3
-> Nested Loop (cost=0.42..166837.32 rows=16 width=337) (actual time=0.459..7.803 rows=1713 loops=1)
Output: mgwor.apx_transaction_id, mgwor.order_status, mgwor.request_date, mgwor.response_date, org.organisation_name
Join Filter: ((account.account_id)::text = (mgwor.account_id)::text)
Rows Removed by Join Filter: 3007
Buffers: shared hit=589
-> Nested Loop (cost=0.27..40.66 rows=4 width=54) (actual time=0.203..0.483 rows=2 loops=1)
Output: account.account_id, org.organisation_name
Join Filter: ((account.organisation_id)::text = (org.organisation_id)::text)
Rows Removed by Join Filter: 289
Buffers: shared hit=27
-> Index Scan using account_pkey on mdm.account (cost=0.27..32.55 rows=285 width=65) (actual time=0.013..0.122 rows=291 loops=1)
Output: account.account_id, account.account_created_at, account.account_name, account.account_status, account.account_valid_until, account.currency_id, account.organisation_id, account.organisation_psp_id, account."account_threeDS_required", account.account_use_webhook, account.account_webhook_url, account.account_webhook_max_attempt, account.reporting_account_id, account.card_type, account.country_id, account.product_id
Buffers: shared hit=24
-> Materialize (cost=0.00..3.84 rows=1 width=55) (actual time=0.000..0.000 rows=1 loops=291)
Output: org.organisation_name, org.organisation_id
Buffers: shared hit=3
-> Seq Scan on mdm.organisation_smd org (cost=0.00..3.84 rows=1 width=55) (actual time=0.017..0.023 rows=1 loops=1)
Output: org.organisation_name, org.organisation_id
Filter: ((org.organisation_name)::text = 'ABC'::text)
Rows Removed by Filter: 67
Buffers: shared hit=3
-> Materialize (cost=0.15..166576.15 rows=3835 width=473) (actual time=0.127..2.826 rows=2360 loops=2)
Output: mgwor.apx_transaction_id, mgwor.order_status, mgwor.request_date, mgwor.response_date, mgwor.account_id
Buffers: shared hit=562
-> Append (cost=0.15..166556.97 rows=3835 width=473) (actual time=0.252..3.661 rows=2360 loops=1)
Buffers: shared hit=562
Subplans Removed: 1460
-> Bitmap Heap Scan on public.mgworderrequest_part_20200612 mgwor (cost=50.98..672.23 rows=2375 width=91) (actual time=0.251..2.726 rows=2360 loops=1)
Output: mgwor.apx_transaction_id, mgwor.order_status, mgwor.request_date, mgwor.response_date, mgwor.account_id
Recheck Cond: ((mgwor.request_type)::text = ANY ('{CARD,CARD_PAYMENT}'::text[]))
Filter: ((mgwor.request_date >= date(now())) AND (mgwor.request_date < (date(now()) + 1)))
Heap Blocks: exact=549
Buffers: shared hit=562
-> Bitmap Index Scan on mgworderrequest_part_20200612_request_type_idx (cost=0.00..50.38 rows=2375 width=0) (actual time=0.191..0.192 rows=2361 loops=1)
Index Cond: ((mgwor.request_type)::text = ANY ('{CARD,CARD_PAYMENT}'::text[]))
Buffers: shared hit=13
-> Append (cost=0.14..435.73 rows=1461 width=316) (actual time=0.005..0.006 rows=1 loops=1713)
Buffers: shared hit=5198 dirtied=3
Subplans Removed: 1460
-> Index Scan using transaction_part_20200612_pkey on public.transaction_part_20200612 tran (cost=0.29..0.87 rows=1 width=42) (actual time=0.004..0.005 rows=1 loops=1713)
Output: tran.transaction_status, tran.transaction_id
Index Cond: (((tran.transaction_id)::text = (mgwor.apx_transaction_id)::text) AND (tran.transaction_created_at >= date(now())) AND (tran.transaction_created_at < (date(now()) + 1)))
Filter: (tran.transaction_status IS NOT NULL)
Buffers: shared hit=5198 dirtied=3
Planning Time: 19535.308 ms
Execution Time: 21.006 ms
Partition pruning is working on both the tables.
Am I missing something obvious here?
Thanks,
VA

I don't know why the cost estimate for the append is so large, but presumably you are really worried about how long this takes, not how large the estimate is. As noted, the actual time is going to planning, not to execution.
A likely explanation is that it was waiting on a lock. Time spent waiting on a table lock for a partition table (but not for the parent table) gets attributed to planning time.

postgres window function trebles query time

I'm using postgres 10, and have the following query
select
count(task.id) over() as _total_ ,
json_agg(u.*) as users,
task.*
from task
left outer join taskuserlink_history tu on (task.id = tu.taskid)
left outer join "user" u on (tu.userId = u.id)
group by task.id offset 10 limit 10;
this query takes approx 800ms to execute
if I remove the count(task.id) over() as _total_ , line, then it executes in 250ms
I have to confess being a complete sql noob, so the query itself may be completely borked
I was wondering if anyone could point to the flaws in the query, and make suggestions on how to speed it up.
The number of tasks is approx 15k, with an average of 5 users per task, linked through taskuserlink
I have looked at the pgadmin "explain" diagram
but to be honest can't really figure it out yet ;)
the table definitions are
task , with id (int) as primary column
taskuserlink_history, with taskId (int) and userId (int) (both as foreign key constraints, indexed)
user, with id (int) as primary column
the query plan is as follows
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=4.74..12.49 rows=10 width=44) (actual time=1178.016..1178.043 rows=10 loops=1)
Buffers: shared hit=3731, temp read=6655 written=6914
-> WindowAgg (cost=4.74..10248.90 rows=13231 width=44) (actual time=1178.014..1178.040 rows=10 loops=1)
Buffers: shared hit=3731, temp read=6655 written=6914
-> GroupAggregate (cost=4.74..10083.51 rows=13231 width=36) (actual time=0.417..1049.294 rows=13255 loops=1)
Group Key: task.id
Buffers: shared hit=3731
-> Nested Loop Left Join (cost=4.74..9586.77 rows=66271 width=36) (actual time=0.103..309.372 rows=66162 loops=1)
Join Filter: (taskuserlink_history.userid = user_archive.id)
Rows Removed by Join Filter: 1182904
Buffers: shared hit=3731
-> Merge Left Join (cost=0.58..5563.22 rows=66271 width=8) (actual time=0.044..73.598 rows=66162 loops=1)
Merge Cond: (task.id = taskuserlink_history.taskid)
Buffers: shared hit=3629
-> Index Only Scan using task_pkey on task (cost=0.29..1938.30 rows=13231 width=4) (actual time=0.026..7.683 rows=13255 loops=1)
Heap Fetches: 13255
Buffers: shared hit=1810
-> Index Scan using taskuserlink_history_task_fk_idx on taskuserlink_history (cost=0.29..2764.46 rows=66271 width=8) (actual time=0.015..40.109 rows=66162 loops=1)
Filter: (timeend IS NULL)
Rows Removed by Filter: 13368
Buffers: shared hit=1819
-> Materialize (cost=4.17..50.46 rows=4 width=36) (actual time=0.000..0.001 rows=19 loops=66162)
Buffers: shared hit=102
-> Bitmap Heap Scan on user_archive (cost=4.17..50.44 rows=4 width=36) (actual time=0.050..0.305 rows=45 loops=1)
Recheck Cond: (archived_at IS NULL)
Heap Blocks: exact=11
Buffers: shared hit=102
-> Bitmap Index Scan on user_unique_username (cost=0.00..4.16 rows=4 width=0) (actual time=0.014..0.014 rows=46 loops=1)
Buffers: shared hit=1
SubPlan 1
-> Aggregate (cost=8.30..8.31 rows=1 width=8) (actual time=0.003..0.003 rows=1 loops=45)
Buffers: shared hit=90
-> Index Scan using task_assignedto_idx on task task_1 (cost=0.29..8.30 rows=1 width=4) (actual time=0.002..0.002 rows=0 loops=45)
Index Cond: (assignedtoid = user_archive.id)
Buffers: shared hit=90
Planning time: 0.989 ms
Execution time: 1191.451 ms
(37 rows)
without the window function it is
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=4.74..12.36 rows=10 width=36) (actual time=0.510..1.763 rows=10 loops=1)
Buffers: shared hit=91
-> GroupAggregate (cost=4.74..10083.51 rows=13231 width=36) (actual time=0.509..1.759 rows=10 loops=1)
Group Key: task.id
Buffers: shared hit=91
-> Nested Loop Left Join (cost=4.74..9586.77 rows=66271 width=36) (actual time=0.073..0.744 rows=50 loops=1)
Join Filter: (taskuserlink_history.userid = user_archive.id)
Rows Removed by Join Filter: 361
Buffers: shared hit=91
-> Merge Left Join (cost=0.58..5563.22 rows=66271 width=8) (actual time=0.029..0.161 rows=50 loops=1)
Merge Cond: (task.id = taskuserlink_history.taskid)
Buffers: shared hit=7
-> Index Only Scan using task_pkey on task (cost=0.29..1938.30 rows=13231 width=4) (actual time=0.016..0.031 rows=11 loops=1)
Heap Fetches: 11
Buffers: shared hit=4
-> Index Scan using taskuserlink_history_task_fk_idx on taskuserlink_history (cost=0.29..2764.46 rows=66271 width=8) (actual time=0.009..0.081 rows=50 loops=1)
Filter: (timeend IS NULL)
Rows Removed by Filter: 11
Buffers: shared hit=3
-> Materialize (cost=4.17..50.46 rows=4 width=36) (actual time=0.001..0.009 rows=8 loops=50)
Buffers: shared hit=84
-> Bitmap Heap Scan on user_archive (cost=4.17..50.44 rows=4 width=36) (actual time=0.040..0.382 rows=38 loops=1)
Recheck Cond: (archived_at IS NULL)
Heap Blocks: exact=7
Buffers: shared hit=84
-> Bitmap Index Scan on user_unique_username (cost=0.00..4.16 rows=4 width=0) (actual time=0.012..0.012 rows=46 loops=1)
Buffers: shared hit=1
SubPlan 1
-> Aggregate (cost=8.30..8.31 rows=1 width=8) (actual time=0.005..0.005 rows=1 loops=38)
Buffers: shared hit=76
-> Index Scan using task_assignedto_idx on task task_1 (cost=0.29..8.30 rows=1 width=4) (actual time=0.003..0.003 rows=0 loops=38)
Index Cond: (assignedtoid = user_archive.id)
Buffers: shared hit=76
Planning time: 0.895 ms
Execution time: 1.890 ms
(35 rows)|

I believe the LIMIT clause is making the difference. LIMIT is limiting the number of rows returned, not neccessarily the work involved:
Your second query can be aborted early after 20 rows have been constructed (10 for OFFSET and 10 for LIMIT).
However, your first query needs to go through the whole set to calculate the count(task.id).
Not what you were asking, but I say it anyway:
"user" is not a table, but a view. That is were both queries actually get slower than they should be (The "Materialize" in the plan).
Using OFFSET for paging calls for trouble because it will get slow when the OFFSET increases
Using OFFSET and LIMIT without an ORDER BY is most likely not what you want. The result sets might not be identical on consecutive calls.