Understanding why my Postgres queries are faster without indexes

Understanding why my Postgres queries are faster without indexes - postgresql

We've inherited a Postgresql driven datawarehouse with some serious performance issues. We've selected a database for one of our customers and are benchmarking queries. We've picked into the queries and found a couple of common tables which are selected from in all queries, which we believe is at the heart of our poor performance. We're concentrating particularly on cold start performance, when no data is loaded in to the shared buffers as this is common scenario for our customers. We took a large query and stripped it down to its slowest part;
explain (analyze, buffers, costs, format json)
select
poi."SKUId" as "SKUId",
poi."ConvertedLineTotal" as "totalrevenue",
poi."TotalDispachCost" as "totaldispachcost",
poi."Quantity" as "quantitysold"
from public.processedorder o
join public.processedorder_item poi on poi."OrderId" = o."OrderId"
WHERE o."ReceivedDate" >= '2020-09-01' and o."ReceivedDate" <= '2021-01-01';
And in fact, stripped this down further to a single table which seems particularly slow;
explain (analyze, buffers, costs, format json)
select o."OrderId", o."ChannelId"
from public.processedorder o
WHERE o."ReceivedDate" >= '2020-09-01' and o."ReceivedDate" <= '2021-01-01'
This processedorders table has around 2.1 million rows with around 35/40k rows being added each month.
The table looks like this- it's fairly wide;
CREATE TABLE public.processedorder (
"OrderId" int4 NOT NULL,
"ChannelId" int4 NOT NULL,
"ShippingId" int4 NOT NULL,
"CountryId" int4 NOT NULL,
"LocationId" int4 NOT NULL,
"PackagingId" int4 NOT NULL,
"ConvertedTotal" numeric(18, 6) NOT NULL,
"ConvertedSubtotal" numeric(18, 6) NOT NULL,
"ConvertedShippingCost" numeric(18, 6) NOT NULL,
"ConvertedShippingTax" numeric(18, 6) NOT NULL,
"ConvertedTax" numeric(18, 6) NOT NULL,
"ConvertedDiscount" numeric(18, 6) NOT NULL,
"ConversionRate" numeric(18, 6) NOT NULL,
"Currency" varchar(3) NOT NULL,
"OriginalTotal" numeric(18, 6) NOT NULL,
"OriginalSubtotal" numeric(18, 6) NOT NULL,
"OriginalShippingCost" numeric(18, 6) NOT NULL,
"OrignalShippingTax" numeric(18, 6) NOT NULL,
"OriginalTax" numeric(18, 6) NOT NULL,
"OriginalDiscount" numeric(18, 6) NOT NULL,
"ReceivedDate" timestamp NOT NULL,
"DispatchByDate" timestamp NOT NULL,
"ProcessedDate" timestamp NOT NULL,
"HoldOrCancel" bool NOT NULL,
"CustomerHash" varchar(100) NOT NULL,
"EmailHash" varchar(100) NOT NULL,
"GetPostalCode" varchar(10) NOT NULL,
"TagId" uuid NOT NULL,
"timestamp" timestamp NOT NULL,
"IsRMA" bool NOT NULL DEFAULT false,
"ConversionType" int4 NOT NULL DEFAULT 0,
"ItemWeight" numeric(18, 6) NULL,
"TotalWeight" numeric(18, 6) NULL,
"PackageWeight" numeric(18, 6) NULL,
"PackageCount" int4 NULL,
CONSTRAINT processedorder_tagid_unique UNIQUE ("TagId")
)
WITH (
fillfactor=50
);
The confusion we have is, on a local copy of the database we run the smallest query with a simple index on receivedDate and it returns the results in 4 seconds-
create INDEX if not exists ix_processedorder_btree_receieveddate ON public.processedorder USING btree ("ReceivedDate" DESC);
execution plan can be seen here https://explain.tensor.ru/archive/explain/639d403ef7bf772f698502ed98ae3f63:0:2021-12-08#explain
Hash Join (cost=166953.18..286705.36 rows=201855 width=19) (actual time=3078.441..4176.251 rows=198552 loops=1)
Hash Cond: (poi."OrderId" = o."OrderId")
Buffers: shared hit=3 read=160623
-> Seq Scan on processedorder_item poi (cost=0.00..108605.28 rows=2434228 width=23) (actual time=0.158..667.435 rows=2434228 loops=1)
Buffers: shared read=84263
-> Hash (cost=164773.85..164773.85 rows=174346 width=4) (actual time=3077.990..3077.991 rows=173668 loops=1)
Buckets: 262144 (originally 262144) Batches: 1 (originally 1) Memory Usage: 8154kB
Buffers: shared read=76360
-> Bitmap Heap Scan on processedorder o (cost=3703.48..164773.85 rows=174346 width=4) (actual time=27.285..3028.721 rows=173668 loops=1)
Recheck Cond: (("ReceivedDate" >= '2020-09-01 00:00:00'::timestamp without time zone) AND ("ReceivedDate" <= '2021-01-01 00:00:00'::timestamp without time zone))
Heap Blocks: exact=75882
Buffers: shared read=76360
-> Bitmap Index Scan on ix_receiveddate (cost=0.00..3659.89 rows=174346 width=0) (actual time=17.815..17.815 rows=173668 loops=1)
Index Cond: (("ReceivedDate" >= '2020-09-01 00:00:00'::timestamp without time zone) AND ("ReceivedDate" <= '2021-01-01 00:00:00'::timestamp without time zone))
Buffers: shared read=478
We then apply this index to our staging server (same copy of the DB) and run the query, but this time it takes 44 seconds;
https://explain.tensor.ru/archive/explain/9610c603972ba89aac4e223072f27575:0:2021-12-08
Gather (cost=112168.21..275565.45 rows=174360 width=19) (actual time=42401.776..44549.996 rows=145082 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=149058 read=50771
-> Hash Join (cost=111168.21..257129.45 rows=72650 width=19) (actual time=42397.903..44518.824 rows=48361 loops=3)
Hash Cond: (poi."OrderId" = o."OrderId")
Inner Unique: true
Buffers: shared hit=445001 read=161024
-> Parallel Seq Scan on processedorder_item poi (cost=0.00..117223.50 rows=880850 width=23) (actual time=0.302..1426.753 rows=702469 loops=3)
Filter: ((NOT "ContainsComposites") AND ("SKUId" <> 0))
Rows Removed by Filter: 108940
Buffers: shared read=84260
-> Hash (cost=105532.45..105532.45 rows=173408 width=4) (actual time=42396.156..42396.156 rows=173668 loops=3)
Buckets: 262144 (originally 262144) Batches: 1 (originally 1) Memory Usage: 8154kB
Buffers: shared hit=444920 read=76764
-> Index Scan using ix_processedorder_receieveddate on processedorder o (cost=0.43..105532.45 rows=173408 width=4) (actual time=0.827..42152.428 rows=173668 loops=3)
Index Cond: (("ReceivedDate" >= '2020-09-01 00:00:00'::timestamp without time zone) AND ("ReceivedDate" <= '2021-01-01 00:00:00'::timestamp without time zone))
Buffers: shared hit=444920 read=76764
Finally, with common sense apparently not working, we just remove the index on the staging server and find it returns the data in 4 seconds (just like our local machine with the index)
https://explain.tensor.ru/archive/explain/486512e19b45d5cbe4b893fdecc434b8:0:2021-12-08
Gather (cost=1000.00..200395.65 rows=177078 width=4) (actual time=2.556..4695.986 rows=173668 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared read=41868
-> Parallel Seq Scan on processedorder o (cost=0.00..181687.85 rows=73782 width=4) (actual time=0.796..4663.511 rows=57889 loops=3)
Filter: (("ReceivedDate" >= '2020-09-01 00:00:00'::timestamp without time zone) AND ("ReceivedDate" <= '2021-01-01 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 642941
Buffers: shared read=151028
Before each query run I execute from SSH:
echo 3 > /proc/sys/vm/drop_caches; service postgresql restart;
We also ran a vacuum full; analyze; before testing.
Is anyone able to explain why this is happening as it makes no sense to us- I would expect the query to perform fastest with the index, given that we are querying a small portion of the data (order records span 9 years and we are selecting just 3 months).
The server itself is Postgres 10.4 running on Amazon AWS E2 i3.2xlarge instance with a couple io2 EBS block store drives running in RAID 0 hosting the psql data.
work_mem is 150MB
shared_buffers is set to 15Gb (60gb server total ram)
effective_io_concurrency = 256
effective_cache_size=45GB
----- Update 1
As per franks suggestion we tried adding a new index which didn't seem to help
Gather (cost=1000.86..218445.54 rows=201063 width=19) (actual time=4.412..61616.676 rows=198552 loops=1)
Output: poi."SKUId", poi."ConvertedLineTotal", poi."TotalDispachCost", poi."Quantity"
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=239331 read=45293
-> Nested Loop (cost=0.86..197339.24 rows=83776 width=19) (actual time=3.214..61548.378 rows=66184 loops=3)
Output: poi."SKUId", poi."ConvertedLineTotal", poi."TotalDispachCost", poi."Quantity"
Buffers: shared hit=748815 read=136548
Worker 0: actual time=2.494..61658.415 rows=65876 loops=1
Buffers: shared hit=247252 read=45606
Worker 1: actual time=3.033..61667.490 rows=69100 loops=1
Buffers: shared hit=262232 read=45649
-> Parallel Index Only Scan using ix_processedorder_btree_receieveddate_orderid on public.processedorder o (cost=0.43..112293.34 rows=72359 width=4) (actual time=1.811..40429.474 rows=57889 loops=3)
Output: o."ReceivedDate", o."OrderId"
Index Cond: ((o."ReceivedDate" >= '2020-09-01 00:00:00'::timestamp without time zone) AND (o."ReceivedDate" <= '2021-01-01 00:00:00'::timestamp without time zone))
Buffers: shared hit=97131 read=76571
Worker 0: actual time=1.195..40625.809 rows=57420 loops=1
Buffers: shared hit=31847 read=25583
Worker 1: actual time=1.850..40463.813 rows=60469 loops=1
Buffers: shared hit=34898 read=25584
-> Index Scan using ix_processedorder_item_orderid on public.processedorder_item poi (cost=0.43..1.12 rows=2 width=23) (actual time=0.316..0.361 rows=1 loops=173668)
Output: poi."OrderItemId", poi."OrderId", poi."SKUId", poi."Quantity", poi."ConvertedLineTotal", poi."ConvertedLineSubtotal", poi."ConvertedLineTax", poi."ConvertedLineDiscount", poi."ConversionRate", poi."OriginalLineTotal", poi."OriginalLineSubtotal", poi."OriginalLineTax", poi."OriginalLineDiscount", poi."Currency", poi."ContainsComposites", poi."TotalDispachCost", poi."TagId", poi."timestamp"
Index Cond: (poi."OrderId" = o."OrderId")
Buffers: shared hit=651684 read=59977
Worker 0: actual time=0.316..0.362 rows=1 loops=57420
Buffers: shared hit=215405 read=20023
Worker 1: actual time=0.303..0.347 rows=1 loops=60469
Buffers: shared hit=227334 read=20065
---- update 2
We've rerun the larger of the two queries
explain (analyze, buffers, verbose, costs, format json)
select
poi."SKUId" as "SKUId",
poi."ConvertedLineTotal" as "totalrevenue",
poi."TotalDispachCost" as "totaldispachcost",
poi."Quantity" as "quantitysold"
from public.processedorder o
join public.processedorder_item poi on poi."OrderId" = o."OrderId"
WHERE o."ReceivedDate" >= '2020-09-01' and o."ReceivedDate" <= '2021-01-01';
this time with track io timings enabled- we ran this twice, once with an index, and again without the index- these are the plans;
With Index:
https://explain.tensor.ru/archive/explain/d763d7e1754c4ddac8bb61e403b135d2:0:2021-12-09
Gather (cost=111153.17..252254.44 rows=201029 width=19) (actual time=36978.100..39083.705 rows=198552 loops=1)
Output: poi."SKUId", poi."ConvertedLineTotal", poi."TotalDispachCost", poi."Quantity"
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=138354 read=60159
I/O Timings: read=16453.227
-> Hash Join (cost=110153.17..231151.54 rows=83762 width=19) (actual time=36974.257..39044.116 rows=66184 loops=3)
Output: poi."SKUId", poi."ConvertedLineTotal", poi."TotalDispachCost", poi."Quantity"
Hash Cond: (poi."OrderId" = o."OrderId")
Buffers: shared hit=444230 read=160640
I/O Timings: read=37254.816
Worker 0: actual time=36972.516..39140.086 rows=79787 loops=1
Buffers: shared hit=155175 read=48507
I/O Timings: read=9382.648
Worker 1: actual time=36972.438..39138.754 rows=77496 loops=1
Buffers: shared hit=150701 read=51974
I/O Timings: read=11418.941
-> Parallel Seq Scan on public.processedorder_item poi (cost=0.00..114682.68 rows=1014089 width=23) (actual time=0.262..1212.102 rows=811409 loops=3)
Output: poi."OrderItemId", poi."OrderId", poi."SKUId", poi."Quantity", poi."ConvertedLineTotal", poi."ConvertedLineSubtotal", poi."ConvertedLineTax", poi."ConvertedLineDiscount", poi."ConversionRate", poi."OriginalLineTotal", poi."OriginalLineSubtotal", poi."OriginalLineTax", poi."OriginalLineDiscount", poi."Currency", poi."ContainsComposites", poi."TotalDispachCost", poi."TagId", poi."timestamp"
Buffers: shared read=84260
I/O Timings: read=1574.316
Worker 0: actual time=0.073..1258.453 rows=870378 loops=1
Buffers: shared read=30133
I/O Timings: read=535.914
Worker 1: actual time=0.021..1256.769 rows=841299 loops=1
Buffers: shared read=29126
I/O Timings: read=538.814
-> Hash (cost=104509.15..104509.15 rows=173662 width=4) (actual time=36972.511..36972.511 rows=173668 loops=3)
Output: o."OrderId"
Buckets: 262144 (originally 262144) Batches: 1 (originally 1) Memory Usage: 8154kB
Buffers: shared hit=444149 read=76380
I/O Timings: read=35680.500
Worker 0: actual time=36970.996..36970.996 rows=173668 loops=1
Buffers: shared hit=155136 read=18374
I/O Timings: read=8846.735
Worker 1: actual time=36970.783..36970.783 rows=173668 loops=1
Buffers: shared hit=150662 read=22848
I/O Timings: read=10880.127
-> Index Scan using ix_processedorder_btree_receieveddate on public.processedorder o (cost=0.43..104509.15 rows=173662 width=4) (actual time=0.617..36736.881 rows=173668 loops=3)
Output: o."OrderId"
Index Cond: ((o."ReceivedDate" >= '2020-09-01 00:00:00'::timestamp without time zone) AND (o."ReceivedDate" <= '2021-01-01 00:00:00'::timestamp without time zone))
Buffers: shared hit=444149 read=76380
I/O Timings: read=35680.500
Worker 0: actual time=0.018..36741.158 rows=173668 loops=1
Buffers: shared hit=155136 read=18374
I/O Timings: read=8846.735
Worker 1: actual time=0.035..36733.684 rows=173668 loops=1
Buffers: shared hit=150662 read=22848
I/O Timings: read=10880.127
And then, the faster run, without indexes;
https://explain.tensor.ru/archive/explain/d0815f4b0c9baf3bdd512bc94051e768:0:2021-12-09
Gather (cost=231259.20..372360.47 rows=201029 width=19) (actual time=4829.302..7920.614 rows=198552 loops=1)
Output: poi."SKUId", poi."ConvertedLineTotal", poi."TotalDispachCost", poi."Quantity"
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=106720 read=69983
I/O Timings: read=2431.673
-> Hash Join (cost=230259.20..351257.57 rows=83762 width=19) (actual time=4825.285..7877.981 rows=66184 loops=3)
Output: poi."SKUId", poi."ConvertedLineTotal", poi."TotalDispachCost", poi."Quantity"
Hash Cond: (poi."OrderId" = o."OrderId")
Buffers: shared hit=302179 read=235288
I/O Timings: read=7545.729
Worker 0: actual time=4823.181..7978.501 rows=81394 loops=1
Buffers: shared hit=87613 read=93424
I/O Timings: read=2556.079
Worker 1: actual time=4823.645..7979.241 rows=76022 loops=1
Buffers: shared hit=107846 read=71881
I/O Timings: read=2557.977
-> Parallel Seq Scan on public.processedorder_item poi (cost=0.00..114682.68 rows=1014089 width=23) (actual time=0.267..2122.529 rows=811409 loops=3)
Output: poi."OrderItemId", poi."OrderId", poi."SKUId", poi."Quantity", poi."ConvertedLineTotal", poi."ConvertedLineSubtotal", poi."ConvertedLineTax", poi."ConvertedLineDiscount", poi."ConversionRate", poi."OriginalLineTotal", poi."OriginalLineSubtotal", poi."OriginalLineTax", poi."OriginalLineDiscount", poi."Currency", poi."ContainsComposites", poi."TotalDispachCost", poi."TagId", poi."timestamp"
Buffers: shared read=84260
I/O Timings: read=4135.860
Worker 0: actual time=0.034..2171.677 rows=865244 loops=1
Buffers: shared read=29949
I/O Timings: read=1394.407
Worker 1: actual time=0.068..2174.408 rows=827170 loops=1
Buffers: shared read=28639
I/O Timings: read=1395.723
-> Hash (cost=224615.18..224615.18 rows=173662 width=4) (actual time=4823.318..4823.318 rows=173668 loops=3)
Output: o."OrderId"
Buckets: 262144 (originally 262144) Batches: 1 (originally 1) Memory Usage: 8154kB
Buffers: shared hit=302056 read=151028
I/O Timings: read=3409.869
Worker 0: actual time=4820.884..4820.884 rows=173668 loops=1
Buffers: shared hit=87553 read=63475
I/O Timings: read=1161.672
Worker 1: actual time=4822.104..4822.104 rows=173668 loops=1
Buffers: shared hit=107786 read=43242
I/O Timings: read=1162.254
-> Seq Scan on public.processedorder o (cost=0.00..224615.18 rows=173662 width=4) (actual time=0.744..4644.291 rows=173668 loops=3)
Output: o."OrderId"
Filter: ((o."ReceivedDate" >= '2020-09-01 00:00:00'::timestamp without time zone) AND (o."ReceivedDate" <= '2021-01-01 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 1928823
Buffers: shared hit=302056 read=151028
I/O Timings: read=3409.869
Worker 0: actual time=0.040..4651.203 rows=173668 loops=1
Buffers: shared hit=87553 read=63475
I/O Timings: read=1161.672
Worker 1: actual time=0.035..4637.415 rows=173668 loops=1
Buffers: shared hit=107786 read=43242
I/O Timings: read=1162.254

From your last two execution plans, I gather the following:
the slow plan using the index reads 22848 blocks from disk for processedorder, which takes 10.88 seconds
the fast plan using the sequential scan reads 43242 blocks from disk for processedorder, which takes 1.162 seconds
So there can be two explanations:
in the fast case, these blocks are actually cached in the kernel cache, so it is a caching effect
Experiment by clearing the kernel cache.
random I/O is more expensive than the optimizer thinks it is
In that case, you could consider raising random_page_cost.

That first query would benefit from an index on the data and the id in a single index:
CREATE INDEX ix_processedorder_btree_receieveddate ON public.processedorder USING btree ("ReceivedDate" DESC, "OrderId");
Does that change the query plan?

Related

Sudden drop in postgreSQL Memory

Version: 13.6
Hi ,
This is a 2 days old issue. Our Postgresql is hosted on AWS RDS Postgresql. Suddenly our free memory dropped from 1300 MB to 23 MB and swap memory increased from 150 mb to 12OO. MB. CPU was little high, read and write iops were high for 2-3 hours , post that low.
Average row size from tbl1 was 65 bytes and from tbl2 was 350 bytes.
In performance insight we had only one query which was having most wait on CPU. Below is explain plan of the query. I wanted to understand why this query lead to so much of free memory drop and swap memory increase?
Once we created a combined index on (hotel_id, checkin) include(booking_id), free memory became stable.
Query:
database1 => EXPLAIN(ANALYZE, VERBOSE, BUFFERS, COSTS, TIMING)
database1-> select l.*, o.* FROM tbl1 o inner join tbl2 l on o.booking_id = l.booking_id where o.checkin = '2022-09-09' and o.hotel_id = 95315 and l.status = 'YES_RM';
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=3014.65..3077.08 rows=1 width=270) (actual time=71.157..71.159 rows=0 loops=1)
Output: l.*,o.*
Inner Unique: true
Buffers: shared hit=3721 read=170 dirtied=4
I/O Timings: read=8.173
-> Bitmap Heap Scan on public.tbl1 o (cost=3014.08..3034.13 rows=5 width=74) (actual time=69.447..70.814 rows=8 loops=1)
Output: o.booking_id, o.checkin, o.created_at, o.hotel_id, o.hotel_time_zone, o.invoice, o.booking_created_at, o.updated_at, o.user_profile_id
Recheck Cond: ((o.hotel_id = 95315) AND (o.checkin = '2022-09-09'::date))
Rows Removed by Index Recheck: 508
Heap Blocks: exact=512
Buffers: shared hit=3683 read=163
I/O Timings: read=8.148
-> BitmapAnd (cost=3014.08..3014.08 rows=5 width=0) (actual time=69.182..69.183 rows=0 loops=1)
Buffers: shared hit=3312 read=19
I/O Timings: read=7.812
-> Bitmap Index Scan on idx33t1pn8h284cj2rg6pa33bxqb (cost=0.00..167.50 rows=6790 width=0) (actual time=8.550..8.550 rows=3909 loops=1)
Index Cond: (o.hotel_id = 95315)
Buffers: shared hit=5 read=19
I/O Timings: read=7.812
-> Bitmap Index Scan on idx6xuah53fdtx4p5afo5cbgul47 (cost=0.00..2846.33 rows=114368 width=0) (actual time=59.391..59.391 rows=297558 loops=1)
Index Cond: (o.checkin = '2022-09-09'::date)
Buffers: shared hit=3307
-> Index Scan using idxicjnq3084849d8vvmnhrornn7 on public. tbl2 l (cost=0.57..8.59 rows=1 width=204) (actual time=0.042..0.042 rows=0 loops=8)
Output: l.*
Index Cond: (l.booking_id = o.booking_id)
Filter: ((l.status)::text = 'YES_RM'::text)
Rows Removed by Filter: 1
Buffers: shared hit=38 read=7 dirtied=4
I/O Timings: read=0.024
Planning:
Buffers: shared hit=335 read=57
I/O Timings: read=0.180
Planning Time: 1.472 ms
Execution Time: 71.254 ms
(34 rows)
After Index creation:
Nested Loop (cost=1.14..65.02 rows=1 width=270) (actual time=6.640..6.640 rows=0 loops=1)
Output: l.*,o.*
Inner Unique: true
Buffers: shared hit=54 read=16 dirtied=4
I/O Timings: read=5.554
-> Index Scan using hotel_id_check_in_index on public.tbl1 (cost=0.57..22.07 rows=5 width=74) (actual time=2.105..5.361 rows=11 loops=1)
Output: o.*
Index Cond: ((o.hotel_id = 95315) AND (o.checkin = '2022-09-09'::date))
Buffers: shared hit=8 read=13 dirtied=4
I/O Timings: read=4.354
-> Index Scan using idxicjnq3084849d8vvmnhrornn7 on public.tbl2 l (cost=0.57..8.59 rows=1 width=204) (actual time=0.116..0.116 rows=0 loops=11)
Output: l.*
Index Cond: (l.booking_id = o.booking_id)
Filter: ((l.tatus)::text = 'YES_RM'::text)
Rows Removed by Filter: 0
Buffers: shared hit=46 read=3
I/O Timings: read=1.199
Planning:
Buffers: shared hit=416
Planning Time: 1.323 ms
Execution Time: 6.667 ms
(21 rows)

Picking random row from query with an exists where clause seem slow

I am trying to pick a random row from a table with what seems like a simple condition which includes it should be present in another table as well having a specific type.
select * from table1 t1 where type='Other' and exists (select 1 from table2 t2 where t2.id = t1.id FETCH FIRST ROW ONLY) order by random() limit 1;
I am trying to cut the selection in the second table short by only picking the first row it finds, I am really just interested if any row in that table exists, so one will do. However the time it took to run didn't change anything really. I've also tried joining the tables instead, that took twice as long it seems. Right now it takes a little over a minute to run, I am trying to cut this down to a few seconds, max 10.
Any ideas? The type could change in the query, it needs to be a generic index, not a specific one for "Other", there are hundreds of types.
Table1 has over +10M rows with unique id's, table2 has +95M.
I have an index on the id's as well as type
CREATE INDEX type_idx ON table1 USING btree (type);
CREATE INDEX id_idx ON table2 USING btree (id);
CREATE UNIQUE INDEX table1_pkey ON table1 USING btree (id);
Here is the explain
Limit (cost=297536.80..297536.80 rows=1 width=51) (actual time=68436.446..68456.389
rows=1 loops=1)
Buffers: shared hit=1503764 read=299217
I/O Timings: read=199577.751
-> Sort (cost=297536.80..297586.31 rows=19807 width=51) (actual
time=68436.444..68456.386 rows=1 loops=1)
Sort Key: (random())
Sort Method: top-N heapsort Memory: 25kB
Buffers: shared hit=1503764 read=299217
I/O Timings: read=199577.751
-> Gather (cost=7051.90..297437.76 rows=19807 width=51) (actual
time=117.271..68418.453 rows=58327 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=1503764 read=299217
I/O Timings: read=199577.751
-> Nested Loop Semi Join (cost=6051.90..294407.54 rows=8253 width=43)
(actual time=84.291..68358.619 rows=19442 loops=3)
Buffers: shared hit=1503764 read=299217
I/O Timings: read=199577.751
-> Parallel Bitmap Heap Scan on table1 t1 (cost=6051.46..135601.49
rows=225539 width=43) (actual time=83.250..24802.725 rows=185267 loops=3)
Recheck Cond: ((type)::text = 'Other'::text)
Rows Removed by Index Recheck: 1119917
Heap Blocks: exact=20174 lossy=11038
Buffers: shared read=94319
I/O Timings: read=72301.594
-> Bitmap Index Scan on type_idx (cost=0.00..5916.13
rows=541293 width=0) (actual time=89.207..89.208 rows=555802 loops=1)
Index Cond: ((type)::text = 'Other'::text)
Buffers: shared read=470
I/O Timings: read=33.209
-> Index Only Scan using id_idx on events (cost=0.44..65.15
rows=257 width=8) (actual time=0.234..0.234 rows=0 loops=555802)
Index Cond: (t2.id = t1.id)
Heap Fetches: 461
Buffers: shared hit=1503764 read=204898
I/O Timings: read=127276.157
Planning:
Buffers: shared hit=8 read=8
I/O Timings: read=3.139
Planning Time: 5.713 ms
Execution Time: 68457.688 ms
Here is the explain plan after I changed the type index to also include the id
Limit (cost=305876.92..305876.92 rows=1 width=51) (actual
time=81055.897..81077.393 rows=1 loops=1)
Buffers: shared hit=1501397 read=303247
I/O Timings: read=237093.600
-> Sort (cost=305876.92..305926.44 rows=19807 width=51) (actual
time=81055.895..81077.390 rows=1 loops=1)
Sort Key: (random())
Sort Method: top-N heapsort Memory: 25kB
Buffers: shared hit=1501397 read=303247
I/O Timings: read=237093.600
-> Gather (cost=15392.02..305777.89 rows=19807 width=51)
(actual time=87.662..81032.107 rows=58327 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=1501397 read=303247
I/O Timings: read=237093.600
-> Nested Loop Semi Join (cost=14392.02..302747.67
rows=8253 width=43) (actual time=73.967..80990.425 rows=19442
loops=3)
Buffers: shared hit=1501397 read=303247
I/O Timings: read=237093.600
-> Parallel Bitmap Heap Scan on table1 t1
(cost=14391.58..143941.61 rows=225539 width=43) (actual
time=73.193..20476.307 rows=185267 loops=3)
Recheck Cond: ((type)::text =
'Other'::text)
Rows Removed by Index Recheck: 1124091
Heap Blocks: exact=20346 lossy=11134
Buffers: shared read=95982
I/O Timings: read=59211.444
-> Bitmap Index Scan on type_idx
(cost=0.00..14256.26 rows=541293 width=0) (actual
time=73.552..73.552 rows=555802 loops=1)
Index Cond: ((type)::text =
'Other'::text)
Buffers: shared read=2133
I/O Timings: read=6.812
-> Index Only Scan using id_idx on table2
(cost=0.44..65.15 rows=257 width=8) (actual time=0.326..0.326
rows=0 loops=555802)
Index Cond: (t2.id = t1.id)
Heap Fetches: 461
Buffers: shared hit=1501397 read=207265
I/O Timings: read=177882.156
Planning:
Buffers: shared hit=29 read=10
I/O Timings: read=4.789
Planning Time: 11.993 ms
Execution Time: 81078.404 ms

Could you try a combination of columns in your index:
CREATE INDEX type_id_idx ON table1 USING btree (type, id);
How does this change the query plan?

Postgresql Limit 1 causing large number of loops

We have a transactions table, approx. 10m rows and growing. Each customer we have specifies many rules which group certain transactions together based on locations, associated products, sale customer, etc. Off of these rules we produce reports each night which allows them to see the price the customer is paying for products vs their purchase prices from different price lists, these lists are changing daily and each date on the transaction we have to either find their set yearly price or the price that was effective at the date of the transaction.
These price lists can change historically and do all the time as do new historic transactions which are added so within each financial year we have to continue to regenerate these reports.
We are having a problem with the two types of price list/price joins we have to do. The first is on the set yearly price list.
I have removed the queries which bring the transactions in and put into a table called transaction_data_6787.
EXPLAIN analyze
SELECT *
FROM transaction_data_6787 t
inner JOIN LATERAL
(
SELECT p."Price"
FROM "Prices" p
INNER JOIN "PriceLists" pl on p."PriceListId" = pl."Id"
WHERE (pl."CustomerId" = 20)
AND (pl."Year" = 2020)
AND (pl."PriceListTypeId" = 2)
AND p."ProductId" = t.product_id
limit 1
) AS prices ON true
Nested Loop (cost=0.70..133877.20 rows=5394 width=165) (actual time=0.521..193.638 rows=5394 loops=1) -> Seq Scan on transaction_data_6787 t (cost=0.00..159.94 rows=5394 width=145) (actual time=0.005..0.593 rows=5394 loops=1) -> Limit (cost=0.70..24.77 rows=1 width=20) (actual time=0.035..0.035 rows=1 loops=5394)
-> Nested Loop (cost=0.70..24.77 rows=1 width=20) (actual time=0.035..0.035 rows=1 loops=5394)
-> Index Scan using ix_prices_covering on "Prices" p (cost=0.42..8.44 rows=1 width=16) (actual time=0.006..0.015 rows=23 loops=5394)
Index Cond: (("ProductId" = t.product_id))
-> Index Scan using ix_pricelists_covering on "PriceLists" pl (cost=0.28..8.30 rows=1 width=12) (actual time=0.001..0.001 rows=0 loops=122443)
Index Cond: (("Id" = p."PriceListId") AND ("CustomerId" = 20) AND ("PriceListTypeId" = 2))
Filter: ("Year" = 2020)
Rows Removed by Filter: 0 Planning Time: 0.307 ms Execution Time: 193.982 ms
If I remove the LIMIT 1, the execution time drops to 3ms and the 122443 loops on ix_pricelists_covering don't happen. The reason we are doing a lateral join is the price query is dynamically built and sometimes when not joining on the annual price list we join on the effective price lists. This looks like the below:
EXPLAIN analyze
SELECT *
FROM transaction_data_6787 t
inner JOIN LATERAL
(
SELECT p."Price"
FROM "Prices" p
INNER JOIN "PriceLists" pl on p."PriceListId" = pl."Id"
WHERE (pl."CustomerId" = 20)
AND (pl."PriceListTypeId" = 1)
AND p."ProductId" = t.product_id
and pl."ValidFromDate" <= t.transaction_date
ORDER BY pl."ValidFromDate" desc
limit 1
) AS prices ON true
This is killing our performance, some queries are taking 20 seconds plus when we don't order by date desc/limit 1 it completes in ms but we could get duplicate prices back.
We are happy to rewrite if a better way of joining the most recent record. We have thousands of price lists and 100k or prices and there could be 100s if not 1000s of effective prices for each transaction and we need to ensure we get the one which was most recently effective for a product at the date of the transaction.
I have found if I denormalise the price lists/prices into a single table and add an index with ValidFromDate DESC it seems to eliminate the loops but I am hesitant to denormalise and have to maintain that data, these reports can be run adhoc as well as batch jobs and we would have to maintain that data in real time.
Updated Explain/Analyze:
I've added below the query which joins on prices which need to get the most recently effective for the transaction date. I see now that when <= date clause and limit 1 is removed it's actually spinning up multiple workers which is why it seems faster.
I am still seeing the slower query doing a large number of loops, 200k+ (when limit 1/<= date is included).
Maybe the better question is what can we do instead of a lateral join which will allow us to join the effective prices for transactions in the most efficient/performant way possible. I am hoping to avoid denormalising and maintainint that data but if it is the only way we'll do it. If there is a way to rewrite this and not denormalise then I'd really appreciate any insight.
Nested Loop (cost=14.21..76965.60 rows=5394 width=10) (actual time=408.948..408.950 rows=0 loops=1)
Output: t.transaction_id, pr."Price"
Buffers: shared hit=688022
-> Seq Scan on public.transaction_data_6787 t (cost=0.00..159.94 rows=5394 width=29) (actual time=0.018..0.682 rows=5394 loops=1)
Output: t.transaction_id
Buffers: shared hit=106
-> Limit (cost=14.21..14.22 rows=1 width=10) (actual time=0.075..0.075 rows=0 loops=5394)
Output: pr."Price", pl."ValidFromDate"
Buffers: shared hit=687916
-> Sort (cost=14.21..14.22 rows=1 width=10) (actual time=0.075..0.075 rows=0 loops=5394)
Output: pr."Price", pl."ValidFromDate"
Sort Key: pl."ValidFromDate" DESC
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=687916
-> Nested Loop (cost=0.70..14.20 rows=1 width=10) (actual time=0.074..0.074 rows=0 loops=5394)
Output: pr."Price", pl."ValidFromDate"
Inner Unique: true
Buffers: shared hit=687916
-> Index Only Scan using ix_prices_covering on public."Prices" pr (cost=0.42..4.44 rows=1 width=10) (actual time=0.007..0.019 rows=51 loops=5394)
Output: pr."ProductId", pr."ValidFromDate", pr."Id", pr."Price", pr."PriceListId"
Index Cond: (pr."ProductId" = t.product_id)
Heap Fetches: 0
Buffers: shared hit=17291
-> Index Scan using ix_pricelists_covering on public."PriceLists" pl (cost=0.28..8.30 rows=1 width=8) (actual time=0.001..0.001 rows=0 loops=273678)
Output: pl."Id", pl."Name", pl."CustomerId", pl."ValidFromDate", pl."PriceListTypeId"
Index Cond: ((pl."Id" = pr."PriceListId") AND (pl."CustomerId" = 20) AND (pl."PriceListTypeId" = 1))
Filter: (pl."ValidFromDate" <= t.transaction_date)
Rows Removed by Filter: 0
Buffers: shared hit=670625
Planning Time: 1.254 ms
Execution Time: 409.088 ms
Gather (cost=6395.67..7011.99 rows=68 width=10) (actual time=92.481..92.554 rows=0 loops=1)
Output: t.transaction_id, pr."Price"
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=1466 read=2
-> Hash Join (cost=5395.67..6005.19 rows=28 width=10) (actual time=75.126..75.129 rows=0 loops=3)
Output: t.transaction_id, pr."Price"
Inner Unique: true
Hash Cond: (pr."PriceListId" = pl."Id")
Join Filter: (pl."ValidFromDate" <= t.transaction_date)
Rows Removed by Join Filter: 41090
Buffers: shared hit=1466 read=2
Worker 0: actual time=64.707..64.709 rows=0 loops=1
Buffers: shared hit=462
Worker 1: actual time=72.545..72.547 rows=0 loops=1
Buffers: shared hit=550 read=1
-> Merge Join (cost=5374.09..5973.85 rows=3712 width=18) (actual time=26.804..61.492 rows=91226 loops=3)
Output: t.transaction_id, t.transaction_date, pr."Price", pr."PriceListId"
Merge Cond: (pr."ProductId" = t.product_id)
Buffers: shared hit=1325 read=2
Worker 0: actual time=17.677..51.590 rows=83365 loops=1
Buffers: shared hit=400
Worker 1: actual time=24.995..59.395 rows=103814 loops=1
Buffers: shared hit=488 read=1
-> Parallel Index Only Scan using ix_prices_covering on public."Prices" pr (cost=0.42..7678.38 rows=79544 width=29) (actual time=0.036..12.136 rows=42281 loops=3)
Output: pr."ProductId", pr."ValidFromDate", pr."Id", pr."Price", pr."PriceListId"
Heap Fetches: 0
Buffers: shared hit=989 read=2
Worker 0: actual time=0.037..9.660 rows=36873 loops=1
Buffers: shared hit=285
Worker 1: actual time=0.058..13.459 rows=47708 loops=1
Buffers: shared hit=373 read=1
-> Sort (cost=494.29..507.78 rows=5394 width=29) (actual time=9.037..14.700 rows=94555 loops=3)
Output: t.transaction_id, t.product_id, t.transaction_date
Sort Key: t.product_id
Sort Method: quicksort Memory: 614kB
Worker 0: Sort Method: quicksort Memory: 614kB
Worker 1: Sort Method: quicksort Memory: 614kB
Buffers: shared hit=336
Worker 0: actual time=6.608..12.034 rows=86577 loops=1
Buffers: shared hit=115
Worker 1: actual time=8.973..14.598 rows=107126 loops=1
Buffers: shared hit=115
-> Seq Scan on public.transaction_data_6787 t (cost=0.00..159.94 rows=5394 width=29) (actual time=0.020..2.948 rows=5394 loops=3)
Output: t.transaction_id, t.product_id, t.transaction_date
Buffers: shared hit=318
Worker 0: actual time=0.017..2.078 rows=5394 loops=1
Buffers: shared hit=106
Worker 1: actual time=0.027..2.976 rows=5394 loops=1
Buffers: shared hit=106
-> Hash (cost=21.21..21.21 rows=30 width=8) (actual time=0.145..0.145 rows=35 loops=3)
Output: pl."Id", pl."ValidFromDate"
Buckets: 1024 Batches: 1 Memory Usage: 10kB
Buffers: shared hit=53
Worker 0: actual time=0.137..0.138 rows=35 loops=1
Buffers: shared hit=18
Worker 1: actual time=0.149..0.150 rows=35 loops=1
Buffers: shared hit=18
-> Bitmap Heap Scan on public."PriceLists" pl (cost=4.59..21.21 rows=30 width=8) (actual time=0.067..0.114 rows=35 loops=3)
Output: pl."Id", pl."ValidFromDate"
Recheck Cond: (pl."CustomerId" = 20)
Filter: (pl."PriceListTypeId" = 1)
Rows Removed by Filter: 6
Heap Blocks: exact=15
Buffers: shared hit=53
Worker 0: actual time=0.068..0.108 rows=35 loops=1
Buffers: shared hit=18
Worker 1: actual time=0.066..0.117 rows=35 loops=1
Buffers: shared hit=18
-> Bitmap Index Scan on "IX_PriceLists_CustomerId" (cost=0.00..4.58 rows=41 width=0) (actual time=0.049..0.049 rows=41 loops=3)
Index Cond: (pl."CustomerId" = 20)
Buffers: shared hit=8
Worker 0: actual time=0.053..0.054 rows=41 loops=1
Buffers: shared hit=3
Worker 1: actual time=0.048..0.048 rows=41 loops=1
Buffers: shared hit=3
Planning Time: 2.236 ms
Execution Time: 92.814 ms

Postgres explain plan High number of Buffers Shared read, for very slow number of rows [duplicate]

Postgres Version : 12
Query:
EXPLAIN (ANALYZE TRUE, VERBOSE TRUE, COSTS TRUE, BUFFERS TRUE, TIMING TRUE)
SELECT MIN("id"), MAX("id") FROM "public"."hotel_slot_inventory" WHERE ( "updated_at" >= '2021-03-02 13:30:03' AND "updated_at" < '2021-03-03 06:15:19.127884' );
Query Plan:
Result (cost=512.44..512.45 rows=1 width=8) (actual time=57839.244..57839.250 rows=1 loops=1)
Output: $0, $1
Buffers: shared hit=1 read=454374 written=185
I/O Timings: read=54564.571 write=2.686
InitPlan 1 (returns $0)
-> Limit (cost=0.57..256.22 rows=1 width=4) (actual time=57599.761..57599.764 rows=1 loops=1)
Output: hotel_slot_inventory.id
Buffers: shared read=453546 written=185
I/O Timings: read=54330.640 write=2.686
-> Index Only Scan using hotel_slot_inventory_id_updated_at_idx on public.hotel_slot_inventory (cost=0.57..3285663.29 rows=12852 width=4) (actual time=57599.758..57599.759 rows=1 loops=1)
Output: hotel_slot_inventory.id
Index Cond: ((hotel_slot_inventory.id IS NOT NULL) AND (hotel_slot_inventory.updated_at >= '2021-03-02 13:30:03'::timestamp without time zone) AND (hotel_slot_inventory.updated_at < '2021-03-03 06:15:19.127884'::timestamp without time zone))
Heap Fetches: 0
Buffers: shared read=453546 written=185
I/O Timings: read=54330.640 write=2.686
InitPlan 2 (returns $1)
-> Limit (cost=0.57..256.22 rows=1 width=4) (actual time=239.468..239.470 rows=1 loops=1)
Output: hotel_slot_inventory_1.id
Buffers: shared hit=1 read=828
I/O Timings: read=233.931
-> Index Only Scan Backward using hotel_slot_inventory_id_updated_at_idx on public.hotel_slot_inventory hotel_slot_inventory_1 (cost=0.57..3285663.29 rows=12852 width=4) (actual time=239.465..239.465 rows=1 loops=1)
Output: hotel_slot_inventory_1.id
Index Cond: ((hotel_slot_inventory_1.id IS NOT NULL) AND (hotel_slot_inventory_1.updated_at >= '2021-03-02 13:30:03'::timestamp without time zone) AND (hotel_slot_inventory_1.updated_at < '2021-03-03 06:15:19.127884'::timestamp without time zone))
Heap Fetches: 0
Buffers: shared hit=1 read=828
I/O Timings: read=233.931
Planning Time: 10.577 ms
Execution Time: 57839.332 ms
(28 rows)
In both the InitPlan, rows=12852 while actual rows=1. why is that? Limit clause has been added separately after Index only scan.
Edit Index Bloat Details:
real_size: 3751411712 = 3.49 GB
extra_size: 470237184 = 448 MB
extra_ratio:12.53
fillfactor: 90
bloat_size: 107053056 = 102 MB
bloat_ratio:2.85
Table Bloat Size:
bloat_size: 475283456 = 453 MB
bloat_ratio: 5.088

This is normal. The expected row count of the "index only scan" line gives the count of rows which is expected to satisfy the index condition, if scanned to completion. The actual rows reflects the fact that the LIMIT made it stop early. Notice the cost estimate of the LIMIT is lower than the cost estimate of the index scan, because that estimate is also given for a complete scan, which is then be "backed off" by the LIMIT.

Looks like an index is bloated out of shape. Try
REINDEX INDEX hotel_slot_inventory_id_updated_at_idx;
or
VACUUM (FULL) hotel_slot_inventory;

Suggestions to reduce memory usage on table partitioning (psql 11)

I have few tables will 20-40million rows, due to which my queries used to take a lot of time for execution. Are there any suggestions to troubleshoot/analyze the queries in details as of where most of the memory is consumed or any more suggestions before going for partitioning?
Also, I have few queries which are used for analysis too, and these queries run over whole range of dates (have to go through whole data).
So I will need an overall solution to keep my basic queries fast and that the analysis queries doesn't fail by going out of memory or crashing the DB.
One table size is nearly 120GB, other tables just have huge number of rows.
I tried to partition the tables with weekly and monthly date basis but then the queries are running out of memory, number of locks increases by a huge factor while having partitions, normal table query took 13 locks and queries on partitioned tables take 250 locks (monthly partition) and 1000 locks (weekly partitions).
I read, there is an overhead that adds up while we have partitions.
Analysis query:
SELECT id
from TABLE1
where id NOT IN (
SELECT DISTINCT id
FROM TABLE2
);
TABLE1 and TABLE2 are partitioned, the first by event_data_timestamp and the second by event_timestamp.
Analysis queries run out of memory and consumes huge number of locks, date based queries are pretty fast though.
QUERY:
EXPLAIN (ANALYZE, BUFFERS) SELECT id FROM Table1_monthly WHERE event_timestamp > '2019-01-01' and id NOT IN (SELECT DISTINCT id FROM Table2_monthly where event_data_timestamp > '2019-01-01');
Append (cost=32731.14..653650.98 rows=4656735 width=16) (actual time=2497.747..15405.447 rows=10121827 loops=1)
Buffers: shared hit=3 read=169100
-> Seq Scan on TABLE1_monthly_2019_01_26 (cost=32731.14..77010.63 rows=683809 width=16) (actual time=2497.746..3489.767 rows=1156382 loops=1)
Filter: ((event_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone) AND (NOT (hashed SubPlan 1)))
Rows Removed by Filter: 462851
Buffers: shared read=44559
SubPlan 1
-> HashAggregate (cost=32728.64..32730.64 rows=200 width=16) (actual time=248.084..791.054 rows=1314570 loops=6)
Group Key: TABLE2_monthly_2019_01_26.cid
Buffers: shared read=24568
-> Append (cost=0.00..32277.49 rows=180458 width=16) (actual time=22.969..766.903 rows=1314570 loops=1)
Buffers: shared read=24568
-> Seq Scan on TABLE2_monthly_2019_01_26 (cost=0.00..5587.05 rows=32135 width=16) (actual time=22.965..123.734 rows=211977 loops=1)
Filter: (event_data_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone)
Rows Removed by Filter: 40282
Buffers: shared read=4382
-> Seq Scan on TABLE2_monthly_2019_02_25 (cost=0.00..5573.02 rows=32054 width=16) (actual time=0.700..121.657 rows=241977 loops=1)
Filter: (event_data_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone)
Buffers: shared read=4371
-> Seq Scan on TABLE2_monthly_2019_03_27 (cost=0.00..5997.60 rows=34496 width=16) (actual time=0.884..123.043 rows=253901 loops=1)
Filter: (event_data_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone)
Buffers: shared read=4704
-> Seq Scan on TABLE2_monthly_2019_04_26 (cost=0.00..6581.55 rows=37855 width=16) (actual time=0.690..129.537 rows=282282 loops=1)
Filter: (event_data_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone)
Buffers: shared read=5162
-> Seq Scan on TABLE2_monthly_2019_05_26 (cost=0.00..6585.38 rows=37877 width=16) (actual time=1.248..122.794 rows=281553 loops=1)
Filter: (event_data_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone)
Buffers: shared read=5165
-> Seq Scan on TABLE2_monthly_2019_06_25 (cost=0.00..999.60 rows=5749 width=16) (actual time=0.750..23.020 rows=42880 loops=1)
Filter: (event_data_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone)
Buffers: shared read=784
-> Seq Scan on TABLE2_monthly_2019_07_25 (cost=0.00..12.75 rows=73 width=16) (actual time=0.007..0.007 rows=0 loops=1)
Filter: (event_data_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone)
-> Seq Scan on TABLE2_monthly_2019_08_24 (cost=0.00..12.75 rows=73 width=16) (actual time=0.003..0.004 rows=0 loops=1)
Filter: (event_data_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone)
-> Seq Scan on TABLE2_monthly_2019_09_23 (cost=0.00..12.75 rows=73 width=16) (actual time=0.003..0.004 rows=0 loops=1)
Filter: (event_data_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone)
-> Seq Scan on TABLE2_monthly_2019_10_23 (cost=0.00..12.75 rows=73 width=16) (actual time=0.007..0.007 rows=0 loops=1)
Filter: (event_data_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone)
-> Seq Scan on TABLE1_monthly_2019_02_25 (cost=32731.14..88679.16 rows=1022968 width=16) (actual time=1008.738..2341.807 rows=1803957 loops=1)
Filter: ((event_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone) AND (NOT (hashed SubPlan 1)))
Rows Removed by Filter: 241978
Buffers: shared hit=1 read=25258
-> Seq Scan on TABLE1_monthly_2019_03_27 (cost=32731.14..97503.58 rows=1184315 width=16) (actual time=1000.795..2474.769 rows=2114729 loops=1)
Filter: ((event_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone) AND (NOT (hashed SubPlan 1)))
Rows Removed by Filter: 253901
Buffers: shared hit=1 read=29242
-> Seq Scan on TABLE1_monthly_2019_04_26 (cost=32731.14..105933.54 rows=1338447 width=16) (actual time=892.820..2405.941 rows=2394619 loops=1)
Filter: ((event_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone) AND (NOT (hashed SubPlan 1)))
Rows Removed by Filter: 282282
Buffers: shared hit=1 read=33048
-> Seq Scan on TABLE1_monthly_2019_05_26 (cost=32731.14..87789.65 rows=249772 width=16) (actual time=918.397..2614.059 rows=2340789 loops=1)
Filter: ((event_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone) AND (NOT (hashed SubPlan 1)))
Rows Removed by Filter: 281553
Buffers: shared read=32579
-> Seq Scan on TABLE1_monthly_2019_06_25 (cost=32731.14..42458.60 rows=177116 width=16) (actual time=923.367..1141.672 rows=311351 loops=1)
Filter: ((event_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone) AND (NOT (hashed SubPlan 1)))
Rows Removed by Filter: 42880
Buffers: shared read=4414
-> Seq Scan on TABLE1_monthly_2019_07_25 (cost=32731.14..32748.04 rows=77 width=16) (actual time=0.008..0.008 rows=0 loops=1)
Filter: ((event_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone) AND (NOT (hashed SubPlan 1)))
-> Seq Scan on TABLE1_monthly_2019_08_24 (cost=32731.14..32748.04 rows=77 width=16) (actual time=0.003..0.003 rows=0 loops=1)
Filter: ((event_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone) AND (NOT (hashed SubPlan 1)))
-> Seq Scan on TABLE1_monthly_2019_09_23 (cost=32731.14..32748.04 rows=77 width=16) (actual time=0.003..0.003 rows=0 loops=1)
Filter: ((event_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone) AND (NOT (hashed SubPlan 1)))
-> Seq Scan on TABLE1_monthly_2019_10_23 (cost=32731.14..32748.04 rows=77 width=16) (actual time=0.003..0.003 rows=0 loops=1)
Filter: ((event_timestamp > '2019-01-01 00:00:00+00'::timestamp with time zone) AND (NOT (hashed SubPlan 1)))
Planning Time: 244.669 ms
Execution Time: 15959.111 ms
(69 rows)

A query that joins two large partitioned tables to produce 10 million rows is going to consume resources, there is no way around that.
You can trade memory consumption for speed by reducing work_mem: smaller vakues will make your queries slower, but consume less memory.
I'd say that the best thing would be to leave work_mem high but reduce max_connections so that you don't run out of memory so fast. Also, putting more RAM into the machine is one of the cheapest hardware tuning techniques.
You can improve the query slighty:
Remove the DISTINCT, which is useless, consumes CPU resources and throws your estimates off.
ANALYZE table2 so that you get better estimates.
About partitioning: if these queries scan all partitions, the query will be slower with partitioned tables.
Whether partitioning is a good idea for you or not depends on the question if you have other queries that benefit from partitioning:
First and foremost, mass deletion, which is painless by dropping partitions.
Sequential scans where the partitioning key is part of the scan filter.
Contrary to popular belief, partitioning is not something you always benefit from if you have large tables: many queries become slower by partitioning.
The locks are your least worry: just increase max_locks_per_transaction.

We Keep Coding

iphone swift flutter scala powershell matlab mongodb postgresql perl eclipse

Understanding why my Postgres queries are faster without indexes - postgresql

That first query would benefit from an index on the data and the id in a single index: CREATE INDEX ix_processedorder_btree_receieveddate ON public.processedorder USING btree ("ReceivedDate" DESC, "OrderId"); Does that change the query plan?

Related

Sudden drop in postgreSQL Memory

Picking random row from query with an exists where clause seem slow

Postgresql Limit 1 causing large number of loops

Postgres explain plan High number of Buffers Shared read, for very slow number of rows [duplicate]

Suggestions to reduce memory usage on table partitioning (psql 11)

Categories

Resources