We have a view which does string_agg on a table with 54 million records where string_agg need to process 2 strings per group. View definition using query like following is resulting in postgresql server process to die due to out of memory.
SELECT id, string_agg(msg, “,”) FROM msgs GROUP BY id
When the view query is modified as below, it is working fine.
SELECT id, string_agg(msg, ',') as msgs
FROM ( SELECT id, msg, row_number() over (partition by id) as row_num
FROM msgs) as limited_alerts
WHERE row_num < 5
GROUP BY id
What is the reason? Is it because postgres is able to use temporary files in 2nd case ? Are there any links/articles which explains the details ?
Table msgs
id bigint PRIMARY KEY
service text
msg text
Total number of rows: 54 million
Max num of msgs per id: 2
Looks like the one which crashes is using HashAggregate and other one GroupAggregate
explain select count(*) from (SELECT msgs.id, string_agg(msgs.alerts, ','::text) AS alerts FROM msgs GROUP BY msgs.id) as a;
QUERY PLAN
------------------------------------------------------------------------------------------------------
Aggregate (cost=3032602.58..3032602.59 rows=1 width=0)
-> HashAggregate (cost=3032597.58..3032600.08 rows=200 width=108)
Group Key: msgs.id
-> Append (cost=0.00..2587883.05 rows=88942906 width=108)
-> Seq Scan on msgs (cost=0.00..0.00 rows=1 width=548)
-> Seq Scan on msgs_2016_01_24 (cost=0.00..506305.40 rows=17394640 width=107)
-> Seq Scan on msgs_2016_01_31 (cost=0.00..509979.80 rows=17512480 width=107)
-> Seq Scan on msgs_2016_02_07 (cost=0.00..491910.32 rows=16883332 width=108)
-> Seq Scan on msgs_2016_02_14 (cost=0.00..496443.84 rows=17071384 width=108)
-> Seq Scan on msgs_2016_02_21 (cost=0.00..552162.84 rows=19026084 width=108)
-> Seq Scan on msgs_2016_02_28 (cost=0.00..31038.05 rows=1054705 width=111)
-> Seq Scan on msgs_2016_03_06 (cost=0.00..10.70 rows=70 width=548)
-> Seq Scan on msgs_2016_03_13 (cost=0.00..10.70 rows=70 width=548)
-> Seq Scan on msgs_2016_03_20 (cost=0.00..10.70 rows=70 width=548)
-> Seq Scan on msgs_2016_03_27 (cost=0.00..10.70 rows=70 width=548)
Related
I have this view that is using lateral against another function. The query is running fine and fast but as soon as I add the condition the where clause and order by. It crawls.
CREATE OR REPLACE VIEW public.vw_top_info_v1_0
AS
SELECT pse.symbol,
pse.order_book,
pse.company_name,
pse.logo_url,
pse.display_logo,
pse.base_url,
stats.value::numeric(20,4) AS stock_value,
stats.volume::numeric(20,0) AS volume,
stats.last_trade_price,
stats.stock_date AS last_trade_date
FROM ( SELECT pse_1.symbol,
pse_1.company_name,
pse_1.order_book,
pse_1.display_logo,
pse_1.base_url,
pse_1.logo_url
FROM vw_pse_traded_companies pse_1
WHERE pse_1.group_name::text = 'N'::text) pse,
LATERAL iq_get_stats_security_for_top_data_v1_0(pse.order_book, (( SELECT date(d.added_date) AS date
FROM prod_itchbbo_p_small_message d
ORDER BY d.added_date DESC
LIMIT 1))::timestamp without time zone) stats(value, volume, stock_date, last_trade_price)
WHERE stats.value IS NOT NULL
ORDER BY stats.value DESC;***
Here's the explain output.
Subquery Scan on vw_top_info_v1_0 (cost=161022.59..165450.34 rows=354220 width=192)
-> Sort (cost=161022.59..161908.14 rows=354220 width=200)
Sort Key: stats.value DESC
InitPlan 1 (returns $0)
-> Limit (cost=49734.18..49734.18 rows=1 width=12)
-> Sort (cost=49734.18..51793.06 rows=823553 width=12)
Sort Key: d.added_date DESC
-> Seq Scan on prod_itchbbo_p_small_message d (cost=0.00..45616.41 rows=823553 width=12)
-> Nested Loop (cost=188.59..10837.44 rows=354220 width=200)
-> Sort (cost=188.34..189.23 rows=356 width=2866)
Sort Key: info.order_book, listed.symbol
-> Hash Join (cost=18.19..173.25 rows=356 width=2866)
Hash Cond: ((info.symbol)::text = (listed.symbol)::text)
-> Seq Scan on prod_stock_information info (cost=0.00..151.85 rows=1220 width=12)
Filter: ((group_name)::text = 'N'::text)
-> Hash (cost=13.64..13.64 rows=364 width=128)
-> Seq Scan on prod_pse_listed_companies listed (cost=0.00..13.64 rows=364 width=128)
-> Function Scan on iq_get_stats_security_for_top_data_v1_0 stats (cost=0.25..10.25 rows=995 width=32)
Filter: (value IS NOT NULL)
Is there a way to improve the query?
I don't fully understand what this is doing but there's a significant cost is coming from the seq scan on prod_itchbbo_p_small_message to sort by date to find the max.
You indicated the cost changes when you add the sort, so if you don't have one, I'd add a b-tree index on prod_itchbbo_p_small_message.added_date.
My app employs a multilevel hierarchical structure. There are many PL/pgSQL functions in the app that use the same type of selection: "select entities according to a list and all their child entities". I created a recursive view trying to avoid redundancy. The problem is, if i understand correctly, PostgreSQL (12.3, compiled by Visual C++ build 1914, 64-bit) selects all entities first and then it filters the records.
Here is a simplified example.
drop view if exists v;
drop table if exists t;
create table t
(
id int primary key,
parent_id int
);
insert into t (id, parent_id)
select s, (s - 1) * random()
from generate_series(1, 100000) as s;
create recursive view v (start_id, id, pid) as
select id, id, parent_id
from t
union all
select v.start_id, t.id, t.parent_id
from t
inner join v on v.id = t.parent_id;
explain (analyze)
select *
from v
where start_id = 10
order by start_id, id;
explain (analyze)
select *
from v
where start_id in (10, 11, 12, 20, 100)
order by start_id, id;
Is there a better solution? Any help is greatly appreciated.
Here is the query plan I got on my computer:
Sort (actual time=3809.581..3812.541 rows=29652 loops=1)
" Sort Key: v.start_id, v.id"
Sort Method: quicksort Memory: 2158kB
-> CTE Scan on v (actual time=0.044..3795.424 rows=29652 loops=1)
" Filter: (start_id = ANY ('{10,11,12,20,100}'::integer[]))"
Rows Removed by Filter: 1069171
CTE v
-> Recursive Union (actual time=0.028..3411.325 rows=1098823 loops=1)
-> Seq Scan on t (actual time=0.025..19.465 rows=100000 loops=1)
-> Merge Join (actual time=74.631..127.916 rows=41618 loops=24)
Merge Cond: (t_1.parent_id = v_1.id)
-> Sort (actual time=46.021..59.589 rows=99997 loops=24)
Sort Key: t_1.parent_id
Sort Method: external merge Disk: 1768kB
-> Seq Scan on t t_1 (actual time=0.016..11.797 rows=100000 loops=24)
-> Materialize (actual time=23.542..42.088 rows=65212 loops=24)
-> Sort (actual time=23.188..29.740 rows=45385 loops=24)
Sort Key: v_1.id
Sort Method: quicksort Memory: 25kB
-> WorkTable Scan on v v_1 (actual time=0.017..7.412 rows=45784 loops=24)
Planning Time: 0.260 ms
Execution Time: 3819.152 ms
I have two tables in a Postgres 11 database:
client table
--------
client_id integer
client_name character_varying
file table
--------
file_id integer
client_id integer
file_name character_varying
The client table is not partitioned, the file table is partitioned by client_id (partition by list). When a new client is inserted into the client table, a trigger creates a new partition for the file table.
The file table has a foreign key constraint referencing the client table on client_id.
When I execute this SQL (where c.client_id = 1), everything seems fine:
explain
select *
from client c
join file f using (client_id)
where c.client_id = 1;
Partition pruning is used, only the partition file_p1 is scanned:
Nested Loop (cost=0.00..3685.05 rows=100001 width=82)
-> Seq Scan on client c (cost=0.00..1.02 rows=1 width=29)
Filter: (client_id = 1)
-> Append (cost=0.00..2684.02 rows=100001 width=57)
-> Seq Scan on file_p1 f (cost=0.00..2184.01 rows=100001 width=57)
Filter: (client_id = 1)
But when I use a where clause like "where c.client_name = 'test'", the database scans in all partitions and does not recognize, that client_name "test" is equal to client_id 1:
explain
select *
from client c
join file f using (client_id)
where c.client_name = 'test';
Execution plan:
Hash Join (cost=1.04..6507.57 rows=100001 width=82)
Hash Cond: (f.client_id = c.client_id)
-> Append (cost=0.00..4869.02 rows=200002 width=57)
-> Seq Scan on file_p1 f (cost=0.00..1934.01 rows=100001 width=57)
-> Seq Scan on file_p4 f_1 (cost=0.00..1934.00 rows=100000 width=57)
-> Seq Scan on file_pdefault f_2 (cost=0.00..1.00 rows=1 width=556)
-> Hash (cost=1.02..1.02 rows=1 width=29)
-> Seq Scan on client c (cost=0.00..1.02 rows=1 width=29)
Filter: ((name)::text = 'test'::text)
So for this SQL, alle partitions in the file-table are scanned.
So should every select use the column on which the tables are partitioned by? Is the database not able to deviate the partition pruning criteria?
Edit:
To add some information:
In the past, I have been working with Oracle databases most of the time.
The execution plan there would be something like
Do a full table scan on client table with the client name to find out the client_id.
Do a "PARTITION LIST" access to the file table, where SQL Developer states PARTITION_START = KEY and PARTITION_STOP = KEY to indicate the exact partition is not known when calculating the execution plan, but the access will be done to only a list of paritions, which are calculated on the client_id found in the client table.
This is what I would have expected in Postgresql as well.
The documentation states that dynamic partition pruning is possible
(...) During actual execution of the query plan. Partition pruning may also be performed here to remove partitions using values which are only known during actual query execution. This includes values from subqueries and values from execution-time parameters such as those from parameterized nested loop joins.
If I understand it correctly, it applies to prepared statements or queries with subqueries which provide the partition key value as a parameter. Use explain analyse to see dynamic pruning (my sample data contains a million rows in three partitions):
explain analyze
select *
from file
where client_id = (
select client_id
from client
where client_name = 'test');
Append (cost=25.88..22931.88 rows=1000000 width=14) (actual time=0.091..96.139 rows=333333 loops=1)
InitPlan 1 (returns $0)
-> Seq Scan on client (cost=0.00..25.88 rows=6 width=4) (actual time=0.040..0.042 rows=1 loops=1)
Filter: (client_name = 'test'::text)
Rows Removed by Filter: 2
-> Seq Scan on file_p1 (cost=0.00..5968.66 rows=333333 width=14) (actual time=0.039..70.026 rows=333333 loops=1)
Filter: (client_id = $0)
-> Seq Scan on file_p2 (cost=0.00..5968.68 rows=333334 width=14) (never executed)
Filter: (client_id = $0)
-> Seq Scan on file_p3 (cost=0.00..5968.66 rows=333333 width=14) (never executed)
Filter: (client_id = $0)
Planning Time: 0.423 ms
Execution Time: 109.189 ms
Note that scans for partitions p2 and p3 were never executed.
Answering your exact question, the partition pruning in queries with joins described in the question is not implemented in Postgres (yet?)
I have an accounts table, units table and reports table. An account has many units (foreign key of units is account_id), and a unit has many reports (foreign key of reports is unit_id). I want to select account name, total number of units for that account, and the last report time:
SELECT accounts.name AS account_name,
COUNT(units.id) AS unit_count,
(SELECT reports.time FROM reports INNER JOIN units ON units.id = reports.unit_id ORDER BY time desc LIMIT 1) AS last_reported_time
FROM accounts
INNER JOIN units ON accounts.id = units.account_id
INNER JOIN reports ON units.id = reports.unit_id
GROUP BY account_name, last_reported_time
ORDER BY unit_count desc;
This query has been running forever, and I am not sure it's doing what I expect.
An account has many units and a unit has many reports. I want to display the time of the newest report from all the units associated for each given account. Is this query correct? If not, how can I accomplish my task (if possible without using a scripting language)?
The result of EXPLAIN:
Sort (cost=21466114.58..21466547.03 rows=172980 width=38)
Sort Key: (count(public.units.id))
InitPlan 1 (returns $0)
-> Limit (cost=0.00..12.02 rows=1 width=8)
-> Nested Loop (cost=0.00..928988485.04 rows=77309416 width=8)
-> Index Scan Backward using index_reports_on_time on reports (cost=0.00..296291138.34 rows=77309416 width=12)
-> Index Scan using units_pkey on units (cost=0.00..8.17 rows=1 width=4)
Index Cond: (public.units.id = public.reports.unit_id)
-> GroupAggregate (cost=20807359.99..21446321.09 rows=172980 width=38)
-> Sort (cost=20807359.99..20966559.70 rows=63679885 width=38)
Sort Key: accounts.name, public.units.last_reported_time
-> Hash Join (cost=975.50..3846816.82 rows=63679885 width=38)
Hash Cond: (public.reports.unit_id = public.units.id)
-> Seq Scan on reports (cost=0.00..2919132.16 rows=77309416 width=4)
-> Hash (cost=961.43..961.43 rows=1126 width=38)
-> Hash Join (cost=16.37..961.43 rows=1126 width=38)
Hash Cond: (public.units.account_id = accounts.id)
-> Seq Scan on units (cost=0.00..928.67 rows=1367 width=28)
-> Hash (cost=11.72..11.72 rows=372 width=18)
-> Seq Scan on accounts (cost=0.00..11.72 rows=372 width=18)
About 95% of the cost of the query is here:
-> Sort (cost=20807359.99..20966559.70 rows=63679885 width=38)
Sort Key: accounts.name, public.units.last_reported_time
-> Hash Join (cost=975.50..3846816.82 rows=63679885 width=38)
Hash Cond: (public.reports.unit_id = public.units.id)
-> Seq Scan on reports (cost=0.00..2919132.16 rows=77309416 width=4)
Do you have an index on reports.unit_id? If not, you should definitely add one.
Other than that, the output column unit_count appears to be giving the number of units per account, but calculating it after all the joins and then ordering by it is very wasteful. The sub-query in the select list is something of mystery to me; I presume you want the most recent reporting time for each unit but it will give you only the last time of reporting over all units combined. Try this instead:
SELECT a.account_name, u.unit_count, r.last_reported_time
FROM account a
JOIN (
SELECT account_id, COUNT(*) AS unit_count
FROM units
GROUP BY 1) u ON u.account_id = a.id
LEFT JOIN ( -- allow for units that have not yet submitted a report
SELECT u.account_id, max(r.time) AS last_reported_time
FROM reports r
JOIN units u ON u.id = r.unit_id
GROUP BY 1) r ON r.account_id = a.id
ORDER BY 2 DESC;
I'm using a partitioned postgres table following the documentation using rules, using a partitioning scheme based on date ranges (my date column is an epoch integer)
The problem is that a simple query to select the row with the maximum value of the sharded column is not using indices:
First, some settings to coerce postgres to do what I want:
SET constraint_exclusion = on;
SET enable_seqscan = off;
The query on a single partition works:
explain (SELECT * FROM urls_0 ORDER BY date_created ASC LIMIT 1);
Limit (cost=0.00..0.05 rows=1 width=38)
-> Index Scan using urls_date_created_idx_0 on urls_0 (cost=0.00..436.68 rows=8099 width=38)
However, the same query on the entire table is seq scanning:
explain (SELECT * FROM urls ORDER BY date_created ASC LIMIT 1);
Limit (cost=50000000274.88..50000000274.89 rows=1 width=51)
-> Sort (cost=50000000274.88..50000000302.03 rows=10859 width=51)
Sort Key: public.urls.date_created
-> Result (cost=10000000000.00..50000000220.59 rows=10859 width=51)
-> Append (cost=10000000000.00..50000000220.59 rows=10859 width=51)
-> Seq Scan on urls (cost=10000000000.00..10000000016.90 rows=690 width=88)
-> Seq Scan on urls_15133 urls (cost=10000000000.00..10000000016.90 rows=690 width=88)
-> Seq Scan on urls_15132 urls (cost=10000000000.00..10000000016.90 rows=690 width=88)
-> Seq Scan on urls_15131 urls (cost=10000000000.00..10000000016.90 rows=690 width=88)
-> Seq Scan on urls_0 urls (cost=10000000000.00..10000000152.99 rows=8099 width=38)
Finally, a lookup by date_created does work correctly with contraint exclusions and index scans:
explain (SELECT * FROM urls where date_created = 1212)
Result (cost=10000000000.00..10000000052.75 rows=23 width=45)
-> Append (cost=10000000000.00..10000000052.75 rows=23 width=45)
-> Seq Scan on urls (cost=10000000000.00..10000000018.62 rows=3 width=88)
Filter: (date_created = 1212)
-> Index Scan using urls_date_created_idx_0 on urls_0 urls (cost=0.00..34.12 rows=20 width=38)
Index Cond: (date_created = 1212)
Does anyone know how to use partitioning so that this type of query will use an index scan?
Postgresql 9.1 knows how to optimize this out of the box.
In 9.0 or earlier, you need to decompose the query manually, by unioning each of the subqueries individually with their own order by/limit statement.