I have a simple postgres query that counts the number of Record objects for a given Database object:
select count(*) from store_record where database_id='a0b0dffb-4e5a-40d7-8f81-d129ffe35bbc';
In the local db, the Record table has 35 million objects and Database.id="a0b0dffb-4e5a-40d7-8f81-d129ffe35bbc" has 11.7 million objects.
In the cloud db, the Record table has 39 million objects and Database.id="a0b0dffb-4e5a-40d7-8f81-d129ffe35bbc" has ~12 million objects.
The stats are similar but the execution times are very different:
Local: the query takes 3 seconds
Cloud: the query takes ~2 min
Here are the query plans for both queries:
Local
Finalize Aggregate (cost=849219.70..849219.71 rows=1 width=8) (actual time=2905.165..2915.472 rows=1 loops=1)
-> Gather (cost=849219.58..849219.69 rows=1 width=8) (actual time=2904.858..2915.456 rows=2 loops=1)
Workers Planned: 1
Workers Launched: 1
-> Partial Aggregate (cost=848219.58..848219.59 rows=1 width=8) (actual time=2871.121..2871.122 rows=1 loops=2)
-> Parallel Seq Scan on store_record (cost=0.00..831132.87 rows=6834686 width=0) (actual time=55.204..2597.471 rows=5835078 loops=2)
Filter: (database_id = 'a0b0dffb-4e5a-40d7-8f81-d129ffe35bbc'::uuid)
Rows Removed by Filter: 11665094
Planning Time: 0.765 ms
JIT:
Functions: 10
" Options: Inlining true, Optimization true, Expressions true, Deforming true"
" Timing: Generation 3.757 ms, Inlining 52.540 ms, Optimization 36.494 ms, Emission 19.836 ms, Total 112.626 ms"
Execution Time: 2918.714 ms
Cloud
Finalize Aggregate (cost=2736538.10..2736538.10 rows=1 width=8) (actual time=126638.968..126675.865 rows=1 loops=1)
-> Gather (cost=2736538.00..2736538.10 rows=1 width=8) (actual time=126638.828..126675.853 rows=2 loops=1)
Workers Planned: 1
Workers Launched: 1
-> Partial Aggregate (cost=2735538.00..2735538.00 rows=1 width=8) (actual time=126612.325..126612.326 rows=1 loops=2)
-> Parallel Seq Scan on store_record (cost=0.00..2731015.70 rows=9044601 width=0) (actual time=117.924..126072.782 rows=7658330 loops=2)
Filter: (database_id = 'a0b0dffb-4e5a-40d7-8f81-d129ffe35bbc'::uuid)
Rows Removed by Filter: 12357700
Planning Time: 5.079 ms
JIT:
Functions: 10
" Options: Inlining true, Optimization true, Expressions true, Deforming true"
" Timing: Generation 1.249 ms, Inlining 153.195 ms, Optimization 45.848 ms, Emission 34.744 ms, Total 235.036 ms"
Execution Time: 126740.456 ms
What is causing this massive difference? My goal is to reproduce the slow cloud query locally so I can debug / optimize it.
Related
I've got table with 50millions rows. I need to find every row with id from the array, but when I put in "ANY" statement more than 4 values my query lasts 45sec+. 4 and less values takes < 100ms.
What the fastest method to do the same thing:
SELECT * FROM tbl WHERE id = ANY('{1, 12, 41, etc.}');
or how to fix this behaviour?
id is a primary key
upd:
account_id is id from example above
EXPLAIN(ANALYZE, BUFFERS) with 4 values:
Gather (cost=194818.11..14487783.08 rows=8426816 width=195) (actual time=62.011..67.316 rows=0 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=16
-> Parallel Bitmap Heap Scan on player_match (cost=193818.11..13644101.48 rows=3511173 width=195) (actual time=1.080..1.081 rows=0 loops=3)
Recheck Cond: (account_id = ANY ('{4,6322,435,75}'::bigint[]))
Buffers: shared hit=16
-> Bitmap Index Scan on player_match_pkey (cost=0.00..191711.41 rows=8426816 width=0) (actual time=0.041..0.042 rows=0 loops=1)
Index Cond: (account_id = ANY ('{4,6322,435,75}'::bigint[]))
Buffers: shared hit=16
Planning Time: 0.118 ms
JIT:
Functions: 6
Options: Inlining true, Optimization true, Expressions true, Deforming true
Timing: Generation 1.383 ms, Inlining 0.000 ms, Optimization 0.000 ms, Emission 0.000 ms, Total 1.383 ms
Execution Time: 67.925 ms
with 5 values:
Gather (cost=1000.00..14995098.33 rows=10533520 width=195) (actual time=59544.067..59557.119 rows=0 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=10327 read=11077591
-> Parallel Seq Scan on player_match (cost=0.00..13940746.33 rows=4388967 width=195) (actual time=59498.932..59498.933 rows=0 loops=3)
Filter: (account_id = ANY ('{4,6322,435,75,1}'::bigint[]))
Rows Removed by Filter: 140446932
Buffers: shared hit=10327 read=11077591
Planning Time: 0.137 ms
JIT:
Functions: 6
Options: Inlining true, Optimization true, Expressions true, Deforming true
Timing: Generation 1.742 ms, Inlining 203.984 ms, Optimization 25.001 ms, Emission 20.096 ms, Total 250.823 ms
Execution Time: 59557.657 ms
upd2: fixed by VACUUM(FULL, ANALYZE) tbl;
Fixed by
VACUUM(FULL, ANALYZE) tbl
My PostgreSQL Version 13. and below is parallel related parameters.
SELECT name, setting FROM pg_Settings WHERE name LIKE '%parallel%'
name |setting|
--------------------------------+-------+
enable_parallel_append |on |
enable_parallel_hash |on |
force_parallel_mode |off |
max_parallel_maintenance_workers|4 |
max_parallel_workers |96 |
max_parallel_workers_per_gather |2 |
min_parallel_index_scan_size |64 |
min_parallel_table_scan_size |1024 |
parallel_leader_participation |on |
parallel_setup_cost |1000 |
parallel_tuple_cost |0.1 |
When I Run bleow Query, It works good. (only 3 secs)
EXPLAIN (analyze)
SELECT t1_code
,COUNT(1) AS cnt
FROM t1 a
WHERE 1=1
GROUP BY t1_code
Finalize GroupAggregate (cost=620185.13..620185.64 rows=2 width=12) (actual time=2953.797..3186.877 rows=2 loops=1)
Group Key: t1_code
-> Gather Merge (cost=620185.13..620185.60 rows=4 width=12) (actual time=2953.763..3186.835 rows=6 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Sort (cost=619185.11..619185.11 rows=2 width=12) (actual time=2926.805..2926.808 rows=2 loops=3)
Sort Key: t1_code
Sort Method: quicksort Memory: 25kB
Worker 0: Sort Method: quicksort Memory: 25kB
Worker 1: Sort Method: quicksort Memory: 25kB
-> Partial HashAggregate (cost=619185.08..619185.10 rows=2 width=12) (actual time=2926.763..2926.768 rows=2 loops=3)
Group Key: t1_code
Batches: 1 Memory Usage: 24kB
Worker 0: Batches: 1 Memory Usage: 24kB
Worker 1: Batches: 1 Memory Usage: 24kB
-> Parallel Seq Scan on t1 a (cost=0.00..551015.72 rows=13633872 width=4) (actual time=0.017..1412.845 rows=10907098 loops=3)
Planning Time: 1.295 ms
JIT:
Functions: 21
Options: Inlining true, Optimization true, Expressions true, Deforming true
Timing: Generation 2.595 ms, Inlining 156.371 ms, Optimization 112.165 ms, Emission 63.886 ms, Total 335.017 ms
Execution Time: 3243.358 ms
But, Without "EXPLAIN Analyze" That Query not use parallel process, when i see the pg_stat_activity. Only 1 process works. so Elapsed time is double. (6 secs)
T1 Table size is 3GB.
Thany for your help.
<< another test >>
below is result of verbose, buffers option. In the same way before, without analyze, below query use single process. and orce_parallel_mode to ON does not effect.
EXPLAIN(ANALYZE, VERBOSE, BUFFERS)
SELECT COUNT(*)
FROM (
SELECT
ID
FROM T2
WHERE CODE1 <> '003'
AND CODE2 <> 'Y'
AND CODE3 <> 'Y'
GROUP BY ID
) t1 ;
Aggregate (cost=204350.48..204350.49 rows=1 width=8) (actual time=2229.919..2229.997 rows=1 loops=1)
Output: count(*)
Buffers: shared hit=216 read=140248 dirtied=10
I/O Timings: read=1404.532
-> Finalize HashAggregate (cost=202326.98..203226.31 rows=89933 width=14) (actual time=2128.682..2199.811 rows=605244 loops=1)
Output: T2.ID
Group Key: T2.ID
Batches: 1 Memory Usage: 53265kB
Buffers: shared hit=216 read=140248 dirtied=10
I/O Timings: read=1404.532
-> Gather (cost=182991.39..201877.32 rows=179866 width=14) (actual time=1632.564..1817.564 rows=1019855 loops=1)
Output: T2.ID
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=216 read=140248 dirtied=10
I/O Timings: read=1404.532
-> Partial HashAggregate (cost=181991.39..182890.72 rows=89933 width=14) (actual time=1592.762..1643.902 rows=339952 loops=3)
Output: T2.ID
Group Key: T2.ID
Batches: 1 Memory Usage: 32785kB
Buffers: shared hit=216 read=140248 dirtied=10
I/O Timings: read=1404.532
Worker 0: actual time=1572.928..1624.075 rows=327133 loops=1
Batches: 1 Memory Usage: 28689kB
JIT:
Functions: 8
Options: Inlining false, Optimization false, Expressions true, Deforming true
Timing: Generation 1.203 ms, Inlining 0.000 ms, Optimization 0.683 ms, Emission 9.159 ms, Total 11.046 ms
Buffers: shared hit=72 read=43679 dirtied=2
I/O Timings: read=470.405
Worker 1: actual time=1573.005..1619.235 rows=330930 loops=1
Batches: 1 Memory Usage: 28689kB
JIT:
Functions: 8
Options: Inlining false, Optimization false, Expressions true, Deforming true
Timing: Generation 1.207 ms, Inlining 0.000 ms, Optimization 0.673 ms, Emission 9.169 ms, Total 11.049 ms
Buffers: shared hit=63 read=44135 dirtied=6
I/O Timings: read=460.591
-> Parallel Seq Scan on T2 (cost=0.00..176869.37 rows=2048806 width=14) (actual time=10.934..1166.528 rows=1638627 loops=3)
Filter: (((T2.CODE1)::text <> '003'::text) AND ((T2.CODE2)::text <> 'Y'::text) AND ((T2.CODE3)::text <> 'Y'::text))
Rows Removed by Filter: 24943
Buffers: shared hit=216 read=140248 dirtied=10
I/O Timings: read=1404.532
Worker 0: actual time=10.083..1162.319 rows=1533436 loops=1
Buffers: shared hit=72 read=43679 dirtied=2
I/O Timings: read=470.405
Worker 1: actual time=10.083..1161.430 rows=1561181 loops=1
Buffers: shared hit=63 read=44135 dirtied=6
I/O Timings: read=460.591
Planning:
Buffers: shared hit=70
Planning Time: 0.253 ms
JIT:
Functions: 31
Options: Inlining false, Optimization false, Expressions true, Deforming true
Timing: Generation 4.451 ms, Inlining 0.000 ms, Optimization 2.182 ms, Emission 29.515 ms, Total 36.148 ms
Execution Time: 2234.037 ms
I solved the problem. This issue was a DBeaver issue. Using psql, parallel processing is performed, and DBeaver is treated as a single process. In DBeaver, parallel processing is performed by CTRL+ALT+Shift+A. However, when Ctrl+Enter is performed, it is treated as a Single process.
When you run explain analyze you are actually executing the query as if it were typed into psql. Additionally there are no results back to the user the results are piped into /dev/null. When you execute the query in DBeaver it is using the pgjdbc JDBC driver to execute the query and is likely setting the fetch size to a non-zero value which turns off parallel query. See https://www.postgresql.org/docs/current/when-can-parallel-query-be-used.html for details.
I have a table of conversations, and one of messages. I want to return a list of conversations ordered by their
select *
from "conversations"
left outer join "messages" on "conversations"."sender_id" = "messages"."sender_id" and event = 'user'
order by (cast("messages"."parse_data" -> 'transcription' ->> 'confidence' as double precision)) nulls last fetch next 50 rows only
And here is the index created :
create index "messages_parse_data_transcription_index_bt"
on messages using btree (sender_id, event, (cast("messages"."parse_data" -> 'transcription' ->> 'confidence' as double precision)) asc NULLS LAST );
analyse messages;
Unfortunately while it works on a large database (3GB, 7M lines on messages) it takes a very long time. Index creation is already very long (3m), and the query itself is also very very long (2M48)
Limit (cost=523667.88..523673.63 rows=50 width=3052) (actual time=132421.177..142355.335 rows=50 loops=1)
-> Gather Merge (cost=523667.88..560621.87 rows=321339 width=3052) (actual time=131738.458..141672.591 rows=50 loops=1)
Workers Planned: 1
Workers Launched: 1
-> Sort (cost=522667.87..523471.22 rows=321339 width=3052) (actual time=131626.455..131626.606 rows=48 loops=2)
Sort Key: ((((messages.parse_data -> 'transcription'::text) ->> 'confidence'::text))::double precision)
Sort Method: top-N heapsort Memory: 125kB
Worker 0: Sort Method: top-N heapsort Memory: 125kB
-> Parallel Hash Left Join (cost=369055.84..511993.22 rows=321339 width=3052) (actual time=101533.553..131406.086 rows=279394 loops=2)
Hash Cond: ((conversations.sender_id)::text = (messages.sender_id)::text)
-> Parallel Seq Scan on conversations (cost=0.00..2634.32 rows=69832 width=33) (actual time=0.018..22.318 rows=59357 loops=2)
-> Parallel Hash (cost=281743.66..281743.66 rows=227615 width=3011) (actual time=90093.869..90093.870 rows=276748 loops=2)
Buckets: 2048 Batches: 512 Memory Usage: 2352kB
-> Parallel Seq Scan on messages (cost=0.00..281743.66 rows=227615 width=3011) (actual time=404.291..52209.756 rows=276748 loops=2)
Filter: ((event)::text = 'user'::text)
Rows Removed by Filter: 3163558
Planning Time: 20.895 ms
JIT:
Functions: 27
" Options: Inlining true, Optimization true, Expressions true, Deforming true"
" Timing: Generation 18.348 ms, Inlining 174.073 ms, Optimization 889.102 ms, Emission 424.619 ms, Total 1506.142 ms"
Execution Time: 142365.711 ms
How can I speed up my query execution ?
Any suggestions on how to speed up this join?
Activity.joins('JOIN relationships as owner ON owner.person_id = activities.owner_id ')
There's an index on both 'person_id' and 'owner_id'
ANALYZE:
=> EXPLAIN ANALYZE for: SELECT "activities".* FROM "activities" JOIN relationships as owner ON owner.person_id = activities.owner_id
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------
Hash Join (cost=45827.59..598928.22 rows=17869287 width=78) (actual time=376.884..4954.592 rows=2963220 loops=1)
Hash Cond: (activities.owner_id = owner.person_id)
-> Seq Scan on activities (cost=0.00..164430.24 rows=6484724 width=78) (actual time=0.036..646.086 rows=6484724 loops=1)
-> Hash (cost=25685.15..25685.15 rows=1227715 width=4) (actual time=376.635..376.636 rows=1221946 loops=1)
Buckets: 131072 Batches: 32 Memory Usage: 2382kB
-> Seq Scan on relationships owner (cost=0.00..25685.15 rows=1227715 width=4) (actual time=106.584..228.241 rows=1221946 loops=1)
Planning Time: 0.236 ms
JIT:
Functions: 10
Options: Inlining true, Optimization true, Expressions true, Deforming true
Timing: Generation 1.108 ms, Inlining 2.677 ms, Optimization 61.629 ms, Emission 42.139 ms, Total 107.552 ms
Execution Time: 5032.112 ms
(12 rows)
A table raw_data has an index ix_raw_data_timestamp:
CREATE TABLE IF NOT EXISTS public.raw_data
(
ts timestamp without time zone NOT NULL,
log_msg character varying COLLATE pg_catalog."default",
log_image bytea
)
CREATE INDEX IF NOT EXISTS ix_raw_data_timestamp
ON public.raw_data USING btree
(ts ASC NULLS LAST)
TABLESPACE pg_default;
For some reason the index is not used for the following query (and therefore is very slow):
SELECT ts,
log_msg
FROM raw_data
ORDER BY ts ASC
LIMIT 5e6;
The result of EXPLAIN (analyze, buffers, format text) for the query above:
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=9752787.07..10336161.14 rows=5000000 width=50) (actual time=789124.600..859046.614 rows=5000000 loops=1)
Buffers: shared hit=12234 read=888521, temp read=2039471 written=2664654
-> Gather Merge (cost=9752787.07..18421031.89 rows=74294054 width=50) (actual time=789085.442..822547.099 rows=5000000 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=12234 read=888521, temp read=2039471 written=2664654
-> Sort (cost=9751787.05..9844654.62 rows=37147027 width=50) (actual time=788203.880..795491.054 rows=1667070 loops=3)
Sort Key: "ts"
Sort Method: external merge Disk: 1758904kB
Worker 0: Sort Method: external merge Disk: 1762872kB
Worker 1: Sort Method: external merge Disk: 1756216kB
Buffers: shared hit=12234 read=888521, temp read=2039471 written=2664654
-> Parallel Seq Scan on raw_data (cost=0.00..1272131.27 rows=37147027 width=50) (actual time=25.436..119352.861 rows=29717641 loops=3)
Buffers: shared hit=12141 read=888520
Planning Time: 5.240 ms
JIT:
Functions: 7
Options: Inlining true, Optimization true, Expressions true, Deforming true
Timing: Generation 0.578 ms, Inlining 76.678 ms, Optimization 24.578 ms, Emission 13.060 ms, Total 114.894 ms
Execution Time: 877489.531 ms
(20 rows)
But it is used for this one:
SELECT ts,
log_msg
FROM raw_data
ORDER BY ts ASC
LIMIT 4e6;
EXPLAIN (analyze, buffers, format text) of the query above is:
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=0.57..9408157.15 rows=4000000 width=50) (actual time=15.081..44747.127 rows=4000000 loops=1)
Buffers: shared hit=24775 read=61155
-> Index Scan using ix_raw_data_timestamp on raw_data (cost=0.57..209691026.73 rows=89152864 width=50) (actual time=2.218..16077.755 rows=4000000 loops=1)
Buffers: shared hit=24775 read=61155
Planning Time: 1.306 ms
JIT:
Functions: 3
Options: Inlining true, Optimization true, Expressions true, Deforming true
Timing: Generation 0.406 ms, Inlining 1.121 ms, Optimization 7.917 ms, Emission 3.721 ms, Total 13.165 ms
Execution Time: 59028.951 ms
(10 rows)
Needless to say that the aim is to get all queries to use the index no matter the size, but I cannot seem to find a solution.
PS:
There's about 89152922 rows in the database.
Edit:
After increasing the memory to 2G (SET work_mem = '2GB';), the query is a little faster (doesn't use disk anymore) but still nowhere as fast:
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=5592250.54..6175624.61 rows=5000000 width=50) (actual time=215887.445..282393.743 rows=5000000 loops=1)
Buffers: shared hit=12224 read=888531
-> Gather Merge (cost=5592250.54..14260731.75 rows=74296080 width=50) (actual time=215874.072..247030.062 rows=5000000 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=12224 read=888531
-> Sort (cost=5591250.52..5684120.62 rows=37148040 width=50) (actual time=215854.323..221828.921 rows=1667147 loops=3)
Sort Key: "ts"
Sort Method: top-N heapsort Memory: 924472kB
Worker 0: Sort Method: top-N heapsort Memory: 924379kB
Worker 1: Sort Method: top-N heapsort Memory: 924281kB
Buffers: shared hit=12224 read=888531
-> Parallel Seq Scan on raw_data (cost=0.00..1272141.40 rows=37148040 width=50) (actual time=25.899..107034.903 rows=29717641 loops=3)
Buffers: shared hit=12130 read=888531
Planning Time: 0.058 ms
JIT:
Functions: 7
Options: Inlining true, Optimization true, Expressions true, Deforming true
Timing: Generation 0.642 ms, Inlining 53.860 ms, Optimization 23.848 ms, Emission 11.768 ms, Total 90.119 ms
Execution Time: 300281.654 ms
(20 rows)
The problem here is you're going to PARALLEL SEQ SCAN and GATHER_MERGE. The gather merge is taking in 74,294,054 rows to output 5,000,000. Which makes sense, because you're saying there are 89,152,922 rows in the DB, and you have no conditional for which to limit them.
Why would it choose this plan, probably because it is forcing materialization because you're over work_mem. So increase your work_mem. If PostgreSQL thinks it can fit all this in memory and that it doesn't have to do this on disk then it will move massively faster.