Large COUNT DISTINCT performs slowly in postgresql - postgresql

Question

I am running a big count(DISTINCT) query group by query against a table on postgresql 12. The table is roughly 32GB, 300MM rows. It is partitioned by YEAR. The groups are more or less exactly distributed:
EXPLAIN (ANALYZE,BUFFERS)
SELECT
date_trunc('month', condition_start_date::timestamp) as dt,
condition_source_value,
COUNT(DISTINCT person_id)
FROM synpuf5.condition_occurrence_yrpart
GROUP BY date_trunc('month', condition_start_date::timestamp), condition_source_value
ORDER BY COUNT(DISTINCT person_id) DESC LIMIT 10;
Here is the output of the query planner:
QUERY PLAN
Limit (cost=50052765961.82..50052765961.85 rows=10 width=21) (actual time=691022.306..691022.308 rows=10 loops=1)
Buffers: shared hit=3062256 read=222453
-> Sort (cost=50052765961.82..50052777188.87 rows=4490820 width=21) (actual time=690786.364..690786.364 rows=10 loops=1)
Sort Key: (count(DISTINCT condition_occurrence_yrpart_2007.person_id)) DESC
Sort Method: top-N heapsort Memory: 26kB
Buffers: shared hit=3062256 read=222453
-> GroupAggregate (cost=50049709699.80..50052668916.82 rows=4490820 width=21) (actual time=567099.326..690705.612 rows=360849 loops=1)
Group Key: (date_trunc('month'::text, (condition_occurrence_yrpart_2007.condition_start_date)::timestamp without time zone)), condition_occurrence_yrpart_2007.condition_source_value
Buffers: shared hit=3062253 read=222453
-> Sort (cost=50049709699.80..50050432663.48 rows=289185472 width=17) (actual time=567098.345..619461.044 rows=289182385 loops=1)
Sort Key: (date_trunc('month'::text, (condition_occurrence_yrpart_2007.condition_start_date)::timestamp without time zone)), condition_occurrence_yrpart_2007.condition_source_value
Sort Method: quicksort Memory: 30333184kB
Buffers: shared hit=3062246 read=222453
-> Append (cost=10000000000.00..50009068412.44 rows=289185472 width=17) (actual time=0.065..74222.771 rows=289182385 loops=1)
Buffers: shared hit=3062240 read=222453
-> Seq Scan on condition_occurrence_yrpart_2007 (cost=10000000000.00..10000001125.61 rows=42774 width=17) (actual time=0.064..13.756 rows=42774 loops=1)
Buffers: shared read=484
-> Seq Scan on condition_occurrence_yrpart_2008 (cost=10000000000.00..10002732063.72 rows=103678448 width=17) (actual time=0.039..21209.532 rows=103676930 loops=1)
Buffers: shared hit=954918 read=221969
-> Seq Scan on condition_occurrence_yrpart_2009 (cost=10000000000.00..10003024874.44 rows=114743696 width=17) (actual time=0.142..20191.131 rows=114743002 loops=1)
Buffers: shared hit=1303719
-> Seq Scan on condition_occurrence_yrpart_2010 (cost=10000000000.00..10001864406.36 rows=70720224 width=17) (actual time=0.050..12464.117 rows=70719679 loops=1)
Buffers: shared hit=803603
-> Seq Scan on condition_occurrence_yrpart_2011 (cost=10000000000.00..10000000014.95 rows=330 width=17) (actual time=0.022..0.022 rows=0 loops=1)
I have also heavily configured my postgresql to attempt to fit all the data in memory, including:
shared_buffers = 80GB
work_mem = 32GB
max_worker_processes = 32
max_parallel_workers_per_gather = 16
max_parallel_workers = 32
wal_compression = on
max_wal_size = 8GB
enable_seqscan = off
enable_partitionwise_join = on
enable_partitionwise_aggregate = on
parallel_tuple_cost = 0.01
parallel_setup_cost = 100.0
shared_preload_libraries = 'pg_prewarm'
effective_cache_size = 192GB
The VM I am running is quite behemoth. 256 GB ram, 32 cores. SSD which is where the postgres directory is housed...
Several questions here:
Why is it so slow?
Why is it not operating in parallel?
Why does performance not increase when I run again despite pg_prewarm?
Why is memory released when my session ends? I am using prewarm?

Answer 1

Why is it so slow?
Sorting 300 million rows takes a while, even with generous work_mem. More than 9 minutes of the query execution time are spent sorting for the GROUP BY.
Why is it not operating in parallel?
Because sorting cannot be parallelized in PostgreSQL.
Why does performance not increase when I run again despite pg_prewarm?
Because everything is already cached.
Why is memory released when my session ends? I am using prewarm?
The memory that your backend uses certainly will be freed when your session ends. The memory used for shared_buffers will not be freed, because that is the cache shared by all processes in the database. You don't want that memory freed.
This is a heavy query, and it takes some time. I don't think that can be improved.
You don't tell us what the partitioning expression is, but since it probably is not date_trunc('month', condition_start_date::timestamp), you don't get partitionwise aggregation despite enable_partitionwise_aggregate = on. PostgreSQL is not smart enough to infer that it could actually do that (assuming you partition on condition_start_date).

Answer 2

It is slow because doing stuff with 300 million rows takes some time.
It is not operating in parallel I think because COUNT(DISTINCT...) code is very old and has not seen much attention lately. It doesn't know how to use hash aggregation, nor operate in parallel. (In my hands, if I lower parallel_tuple_cost all the way to zero, it does operate in parallel, but the gather is below the massive sort and doesn't do any good. But I'm not working with your real data, so could get different results.)
You can get around the inflexibility of COUNT(DISTINCT...) by doing the DISTINCT and the COUNT in separate steps:
select dt, condition_source_value, count(person_id) from (
SELECT distinct
date_trunc('month', condition_start_date::timestamp) as dt,
condition_source_value,
person_id
FROM condition_occurrence_yrpart
) foo
GROUP BY dt, condition_source_value
ORDER BY COUNT(person_id) DESC LIMIT 10;
It still might not do the parallelization in the right spot, though.

Answer 3

I would rewrite the query like this:
SELECT date_trunc('month', condition_start_date::timestamp) as dt
, condition_source_value
, CNT
FROM (
SELECT CONDITION_START_DATE
, condition_source_value
, COUNT(PERSON_ID) AS CNT
FROM (SELECT CONDITION_START_DATE, CONDITION_SOURCE_VALUE, PERSON_ID
FROM synpuf5.condition_occurrence_yrpart
GROUP BY CONDITION_START_DATE, CONDITION_SOURCE_VALUE, PERSON_ID) A
GROUP BY CONDITION_START_DATE, CONDITION_SOURCE_VALUE
ORDER BY COUNT(PERSON_ID) DESC
LIMIT 10) B;
My answer is almost the same with JJANES' answer. But I think you had better reduce the number of DATE_TRUNC function calls. In the revised query, PostgreSQL would run DATE_TRUNC functioins just 10 times.
The following is the URL explaining the tuning concept of this query.
https://blog.naver.com/naivety1/222371990077