Below query is nested loop and runs for 21 mins, after disabling nested loop it works in < 1min. Table stats are up to date and vacuum is run on the tables, any way to figure out why postgres is taking nested loop instead of efficient hash join.
Also to disable a nested loop is it better to set enable_nestloop to off or increase the random_page_cost? I think setting nestloop to off would stop plans from using nested loop if its going to be efficient in some places. What would be a better alternative, please advise.
SELECT DISTINCT ON (three.quebec_delta)
last_value(three.reviewed_by_nm) OVER wnd AS reviewed_by_nm,
last_value(three.reviewer_specialty_nm) OVER wnd AS reviewer_specialty_nm,
last_value(three.kilo) OVER wnd AS kilo,
last_value(three.review_reason_dscr) OVER wnd AS review_reason_dscr,
last_value(three.review_notes) OVER wnd AS review_notes,
last_value(three.seven_uniform_charlie) OVER wnd AS seven_uniform_charlie,
last_value(three.di_audit_source_system_cd) OVER wnd AS di_audit_source_system_cd,
last_value(three.di_audit_update_dtm) OVER wnd AS di_audit_update_dtm,
three.quebec_delta
FROM
ods_authorization.quebec_foxtrot seven_uniform_foxtrot
JOIN ods_authorization.golf echo ON seven_uniform_foxtrot.four = echo.oscar
JOIN ods_authorization.papa three ON echo.five = three.quebec_delta
AND three.xray = '0'::bpchar
WHERE
seven_uniform_foxtrot.two_india >= (zulu () - '2 years'::interval)
AND lima (three.kilo, 'ADVISOR'::character varying)::text = 'ADVISOR'::text
WINDOW wnd AS (PARTITION BY three.quebec_delta ORDER BY three.seven_uniform_charlie DESC ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
Plan running for 21m and taking nested loop
Unique (cost=550047.63..550257.15 rows=5238 width=281) (actual time=1295000.966..1296128.356 rows=319863 loops=1)
-> WindowAgg (cost=550047.63..550244.06 rows=5238 width=281) (actual time=1295000.964..1296013.046 rows=461635 loops=1)
-> Sort (cost=550047.63..550060.73 rows=5238 width=326) (actual time=1295000.929..1295089.796 rows=461635 loops=1)
Sort Key: three.quebec_delta, three.seven_uniform_charlie DESC
Sort Method: quicksort Memory: 197021kB
-> Nested Loop (cost=1001.12..549724.06 rows=5238 width=326) (actual time=8.274..1292470.826 rows=461635 loops=1)
-> Gather (cost=1000.56..527782.84 rows=24896 width=391) (actual time=4.287..12701.687 rows=3484699 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Nested Loop (cost=0.56..524293.24 rows=10373 width=391) (actual time=3.492..400998.923 rows=1161566 loops=3)
-> Parallel Seq Scan on papa three (cost=0.00..436912.84 rows=10373 width=326) (actual time=1.554..2455.626 rows=1161566 loops=3)
Filter: ((xray = 'november'::bpchar) AND ((lima_sierra(kilo, 'two_zulu'::character varying))::text = 'two_zulu'::text))
Rows Removed by Filter: 501723
-> Index Scan using five_tango on golf echo (cost=0.56..8.42 rows=1 width=130) (actual time=0.342..0.342 rows=1 loops=3484699)
Index Cond: (five_hotel = three.quebec_delta)
-> Index Scan using lima_alpha on quebec_foxtrot seven_uniform_foxtrot (cost=0.56..0.88 rows=1 width=65) (actual time=0.366..0.366 rows=0 loops=3484699)
Index Cond: (four = echo.oscar)
Filter: (two_india >= (zulu() - 'two_two'::interval))
Rows Removed by Filter: 1
Planning time: 0.777 ms
Execution time: 1296183.259 ms
Plan after setting enable_nestloop to off and work_mem to 8GB. I get the same plan when increasing random_page_cost to 1000.
Unique (cost=5933437.24..5933646.68 rows=5236 width=281) (actual time=19898.050..20993.124 rows=319980 loops=1)
-> WindowAgg (cost=5933437.24..5933633.59 rows=5236 width=281) (actual time=19898.049..20879.655 rows=461769 loops=1)
-> Sort (cost=5933437.24..5933450.33 rows=5236 width=326) (actual time=19898.022..19978.839 rows=461769 loops=1)
Sort Key: three.quebec_delta, three.seven_uniform_charlie DESC
Sort Method: quicksort Memory: 197056kB
-> Hash Join (cost=1947451.87..5933113.80 rows=5236 width=326) (actual time=11616.323..17931.146 rows=461769 loops=1)
Hash Cond: (echo.oscar = seven_uniform_foxtrot.four)
-> Gather (cost=438059.74..4423656.32 rows=24897 width=391) (actual time=1909.685..7291.289 rows=3484833 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Parallel Hash Join (cost=437059.74..4420166.62 rows=10374 width=391) (actual time=1904.546..7385.948 rows=1161611 loops=3)
Hash Cond: (echo.five = three.quebec_delta)
-> Parallel Seq Scan on golf echo (cost=0.00..3921922.09 rows=8152209 width=130) (actual time=0.003..1756.576 rows=6531668 loops=3)
-> Parallel Hash (cost=436930.07..436930.07 rows=10374 width=326) (actual time=1904.354..1904.354 rows=1161611 loops=3)
Buckets: 4194304 (originally 32768) Batches: 1 (originally 1) Memory Usage: 1135200kB
-> Parallel Seq Scan on papa three (cost=0.00..436930.07 rows=10374 width=326) (actual time=0.009..963.728 rows=1161611 loops=3)
Filter: ((xray = 'november'::bpchar) AND ((lima(kilo, 'two_zulu'::character varying))::text = 'two_zulu'::text))
Rows Removed by Filter: 502246
-> Hash (cost=1476106.74..1476106.74 rows=2662831 width=65) (actual time=9692.517..9692.517 rows=2685656 loops=1)
Buckets: 4194304 Batches: 1 Memory Usage: 287171kB
-> Seq Scan on quebec_foxtrot seven_uniform_foxtrot (cost=0.00..1476106.74 rows=2662831 width=65) (actual time=0.026..8791.556 rows=2685656 loops=1)
Filter: (two_india >= (zulu() - 'two_two'::interval))
Rows Removed by Filter: 9984069
Planning time: 0.742 ms
Execution time: 21218.770 ms
Try an index on papa(lima_sierra(kilo, 'two_zulu'::character varying)) and ANALYZE the table. With that index in place, PostgreSQL collects statistics on the expression, which should improve the estimate, so that you don't get a nested loop join.
If you just replace COALESCE(r_cd, 'ADVISOR') = 'ADVISOR' with
(r_cd = 'ADVISOR' or r_cd IS NULL)
That might use the current table statistics to improve the estimates enough to change the plan.
Related
I am having these results for analyze for a simple query that does not return more than 150 records from tables less than 200 records most of them, as I have a table that stores latest value and the other fields are FK of the data.
Update: see the new results from same query some our later. The site is not public and/or there should be not users right now as it is in development.
explain analyze
SELECT lv.station_id,
s.name AS station_name,
s.latitude,
s.longitude,
s.elevation,
lv.element_id,
e.symbol AS element_symbol,
u.symbol,
e.name AS element_name,
lv.last_datetime AS datetime,
lv.last_value AS valor,
s.basin_id,
s.municipality_id
FROM (((element_station lv /*350 records*/
JOIN stations s ON ((lv.station_id = s.id))) /*40 records*/
JOIN elements e ON ((lv.element_id = e.id))) /*103 records*/
JOIN units u ON ((e.unit_id = u.id))) /* 32 records */
WHERE s.id = lv.station_id AND e.id = lv.element_id AND lv.interval_id = 6 and
lv.last_datetime >= ((now() - '06:00:00'::interval) - '01:00:00'::interval)
I have already tried VACUUM and after that some is saved, but again after some times it goes up. I have implemented an index on the fields.
Nested Loop (cost=0.29..2654.66 rows=1 width=92) (actual time=1219.390..35296.253 rows=157 loops=1)
Join Filter: (e.unit_id = u.id)
Rows Removed by Join Filter: 4867
-> Nested Loop (cost=0.29..2652.93 rows=1 width=92) (actual time=1219.383..35294.083 rows=157 loops=1)
Join Filter: (lv.element_id = e.id)
Rows Removed by Join Filter: 16014
-> Nested Loop (cost=0.29..2648.62 rows=1 width=61) (actual time=1219.301..35132.373 rows=157 loops=1)
-> Seq Scan on element_station lv (cost=0.00..2640.30 rows=1 width=20) (actual time=1219.248..1385.517 rows=157 loops=1)
Filter: ((interval_id = 6) AND (last_datetime >= ((now() - '06:00:00'::interval) - '01:00:00'::interval)))
Rows Removed by Filter: 168
-> Index Scan using stations_pkey on stations s (cost=0.29..8.31 rows=1 width=45) (actual time=3.471..214.941 rows=1 loops=157)
Index Cond: (id = lv.station_id)
-> Seq Scan on elements e (cost=0.00..3.03 rows=103 width=35) (actual time=0.003..0.999 rows=103 loops=157)
-> Seq Scan on units u (cost=0.00..1.32 rows=32 width=8) (actual time=0.002..0.005 rows=32 loops=157)
Planning time: 8.312 ms
Execution time: 35296.427 ms
update, same query running it tonight; no changes:
Sort (cost=601.74..601.88 rows=55 width=92) (actual time=1.822..1.841 rows=172 loops=1)
Sort Key: lv.last_datetime DESC
Sort Method: quicksort Memory: 52kB
-> Nested Loop (cost=11.60..600.15 rows=55 width=92) (actual time=0.287..1.680 rows=172 loops=1)
-> Hash Join (cost=11.31..248.15 rows=55 width=51) (actual time=0.263..0.616 rows=172 loops=1)
Hash Cond: (e.unit_id = u.id)
-> Hash Join (cost=9.59..245.60 rows=75 width=51) (actual time=0.225..0.528 rows=172 loops=1)
Hash Cond: (lv.element_id = e.id)
-> Bitmap Heap Scan on element_station lv (cost=5.27..240.25 rows=75 width=20) (actual time=0.150..0.359 rows=172 loops=1)
Recheck Cond: ((last_datetime >= ((now() - '06:00:00'::interval) - '01:00:00'::interval)) AND (interval_id = 6))
Heap Blocks: exact=22
-> Bitmap Index Scan on element_station_latest (cost=0.00..5.25 rows=75 width=0) (actual time=0.136..0.136 rows=226 loops=1)
Index Cond: ((last_datetime >= ((now() - '06:00:00'::interval) - '01:00:00'::interval)) AND (interval_id = 6))
-> Hash (cost=3.03..3.03 rows=103 width=35) (actual time=0.062..0.062 rows=103 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 15kB
-> Seq Scan on elements e (cost=0.00..3.03 rows=103 width=35) (actual time=0.006..0.031 rows=103 loops=1)
-> Hash (cost=1.32..1.32 rows=32 width=8) (actual time=0.019..0.019 rows=32 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
-> Seq Scan on units u (cost=0.00..1.32 rows=32 width=8) (actual time=0.003..0.005 rows=32 loops=1)
-> Index Scan using stations_pkey on stations s (cost=0.29..6.39 rows=1 width=45) (actual time=0.005..0.006 rows=1 loops=172)
Index Cond: (id = lv.station_id)
Planning time: 2.390 ms
Execution time: 2.009 ms
The problem is the misestimate of the number of rows in the sequential scan on element_station. Either autoanalyze has kicked in and calculated new statistics for the table or the data changed.
The problem is probably that PostgreSQL doesn't know the result of
((now() - '06:00:00'::interval) - '01:00:00'::interval)
at query planning time.
If that is possible for you, do it in two steps: First, calculate the expression above (either in PostgreSQL or on the client side). Then run the query with the result as a constant. That will make it easier for PostgreSQL to estimate the result count.
I need help optimizing this PostgreSQL Query. I've already created some indexes but the only index being used is "cp_campaign_task_ap_index".
EXPLAIN ANALYZE SELECT * FROM campaign_phones
INNER JOIN campaigns ON campaigns.id = campaign_phones.campaign_id
INNER JOIN campaign_ports ON campaign_ports.campaign_id = campaigns.id
WHERE campaign_phones.task_id IS NULL
AND campaign_phones.assigned_port = 2
AND (campaigns.auto_start IS TRUE)
AND (campaigns.starts_at_date::date <= '2018-07-08'
AND campaigns.ends_at_date::date >= '2018-07-08')
AND (campaign_ports.gateway_port_id = 611)
AND (campaign_ports.op_mode != 1)
ORDER BY campaigns.last_sent_at ASC NULLS FIRST LIMIT 1;`
The output of the command is:
Limit (cost=26031.86..26031.87 rows=1 width=475) (actual time=2335.421..2335.421 rows=1 loops=1)
-> Sort (cost=26031.86..26047.26 rows=6158 width=475) (actual time=2335.419..2335.419 rows=1 loops=1)
Sort Key: campaigns.last_sent_at NULLS FIRST
Sort Method: top-N heapsort Memory: 25kB
-> Nested Loop (cost=136.10..26001.07 rows=6158 width=475) (actual time=1.176..1510.276 rows=36666 loops=1)
Join Filter: (campaigns.id = campaign_phones.campaign_id)
-> Nested Loop (cost=0.00..28.28 rows=2 width=218) (actual time=0.163..0.435 rows=4 loops=1)
Join Filter: (campaigns.id = campaign_ports.campaign_id)
Rows Removed by Join Filter: 113
-> Seq Scan on campaign_ports (cost=0.00..21.48 rows=9 width=55) (actual time=0.017..0.318 rows=9 loops=1)
Filter: ((op_mode <> 1) AND (gateway_port_id = 611))
Rows Removed by Filter: 823
-> Materialize (cost=0.00..5.74 rows=8 width=163) (actual time=0.001..0.008 rows=13 loops=9)
-> Seq Scan on campaigns (cost=0.00..5.70 rows=8 width=163) (actual time=0.011..0.050 rows=13 loops=1)
Filter: ((auto_start IS TRUE) AND ((starts_at_date)::date <= '2018-07-08'::date) AND ((ends_at_date)::date >= '2018-07-08'::date))
Rows Removed by Filter: 22
-> Bitmap Heap Scan on campaign_phones (cost=136.10..12931.82 rows=4366 width=249) (actual time=43.079..302.895 rows=9166 loops=4)
Recheck Cond: ((campaign_id = campaign_ports.campaign_id) AND (task_id IS NULL) AND (assigned_port = 2))
Heap Blocks: exact=6686
-> Bitmap Index Scan on cp_campaign_task_ap_index (cost=0.00..135.01 rows=4366 width=0) (actual time=8.884..8.884 rows=9167 loops=4)
Index Cond: ((campaign_id = campaign_ports.campaign_id) AND (task_id IS NULL) AND (assigned_port = 2))
Planning time: 1.115 ms
Execution time: 2335.563 ms
The "campaign_phones" relation could have many rows, perhaps a million.
I don't know where to start optimizing, perhaps creating indexes or changing query structure.
Thanks.
I'm using postgres v9.6.5. I have a query which seems not that complicated and was wondering why is it so "slow" (it's not really that slow, but I don't have a lot of data actually - like a few thousand rows).
Here is the query:
SELECT o0.*
FROM "orders" AS o0
JOIN "balances" AS b1 ON b1."id" = o0."balance_id"
JOIN "users" AS u3 ON u3."id" = b1."user_id"
WHERE (u3."partner_id" = 3)
ORDER BY o0."id" DESC LIMIT 10;
And that's query plan:
Limit (cost=0.43..12.84 rows=10 width=148) (actual time=0.062..53.866 rows=4 loops=1)
-> Nested Loop (cost=0.43..4750.03 rows=3826 width=148) (actual time=0.061..53.864 rows=4 loops=1)
Join Filter: (b1.user_id = u3.id)
Rows Removed by Join Filter: 67404
-> Nested Loop (cost=0.43..3945.32 rows=17856 width=152) (actual time=0.025..38.457 rows=16852 loops=1)
-> Index Scan Backward using orders_pkey on orders o0 (cost=0.29..897.80 rows=17856 width=148) (actual time=0.016..11.558 rows=16852 loops=1)
-> Index Scan using balances_pkey on balances b1 (cost=0.14..0.16 rows=1 width=8) (actual time=0.001..0.001 rows=1 loops=16852)
Index Cond: (id = o0.balance_id)
-> Materialize (cost=0.00..1.19 rows=3 width=4) (actual time=0.000..0.000 rows=4 loops=16852)
-> Seq Scan on users u3 (cost=0.00..1.18 rows=3 width=4) (actual time=0.023..0.030 rows=4 loops=1)
Filter: (partner_id = 3)
Rows Removed by Filter: 12
Planning time: 0.780 ms
Execution time: 54.053 ms
I actually tried without LIMIT and I got quite different plan:
Sort (cost=874.23..883.80 rows=3826 width=148) (actual time=11.361..11.362 rows=4 loops=1)
Sort Key: o0.id DESC
Sort Method: quicksort Memory: 26kB
-> Hash Join (cost=3.77..646.55 rows=3826 width=148) (actual time=11.300..11.346 rows=4 loops=1)
Hash Cond: (o0.balance_id = b1.id)
-> Seq Scan on orders o0 (cost=0.00..537.56 rows=17856 width=148) (actual time=0.012..8.464 rows=16852 loops=1)
-> Hash (cost=3.55..3.55 rows=18 width=4) (actual time=0.125..0.125 rows=24 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Hash Join (cost=1.21..3.55 rows=18 width=4) (actual time=0.046..0.089 rows=24 loops=1)
Hash Cond: (b1.user_id = u3.id)
-> Seq Scan on balances b1 (cost=0.00..1.84 rows=84 width=8) (actual time=0.011..0.029 rows=96 loops=1)
-> Hash (cost=1.18..1.18 rows=3 width=4) (actual time=0.028..0.028 rows=4 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Seq Scan on users u3 (cost=0.00..1.18 rows=3 width=4) (actual time=0.014..0.021 rows=4 loops=1)
Filter: (partner_id = 3)
Rows Removed by Filter: 12
Planning time: 0.569 ms
Execution time: 11.420 ms
And also without WHERE (but with LIMIT):
Limit (cost=0.43..4.74 rows=10 width=148) (actual time=0.023..0.066 rows=10 loops=1)
-> Nested Loop (cost=0.43..7696.26 rows=17856 width=148) (actual time=0.022..0.065 rows=10 loops=1)
Join Filter: (b1.user_id = u3.id)
Rows Removed by Join Filter: 139
-> Nested Loop (cost=0.43..3945.32 rows=17856 width=152) (actual time=0.009..0.029 rows=10 loops=1)
-> Index Scan Backward using orders_pkey on orders o0 (cost=0.29..897.80 rows=17856 width=148) (actual time=0.007..0.015 rows=10 loops=1)
-> Index Scan using balances_pkey on balances b1 (cost=0.14..0.16 rows=1 width=8) (actual time=0.001..0.001 rows=1 loops=10)
Index Cond: (id = o0.balance_id)
-> Materialize (cost=0.00..1.21 rows=14 width=4) (actual time=0.001..0.001 rows=15 loops=10)
-> Seq Scan on users u3 (cost=0.00..1.14 rows=14 width=4) (actual time=0.005..0.007 rows=16 loops=1)
Planning time: 0.286 ms
Execution time: 0.097 ms
As you can see, without WHERE it's much faster. Can someone provide me with some information where can I look for explanations for those plans to better understand them? And also what can I do to make those queries faster (or I shouldn't worry cause with like 100 times more data they will still be fast enough? - 50ms is fine for me tbh)
PostgreSQL thinks that it will be fastest if it scans orders in the correct order until it finds a matching users entry that satisfies the WHERE condition.
However, it seems that the data distribution is such that it has to scan almost 17000 orders before it finds a match.
Since PostgreSQL doesn't know how values correlate across tables, there is nothing much you can do to change that.
You can force PostgreSQL to plan the query without the LIMIT clause like this:
SELECT *
FROM (<your query without ORDER BY and LIMIT> OFFSET 0) q
ORDER BY id DESC LIMIT 10;
With a top-N-sort this should perform better.
I have having a problem where the same query on a table sometimes takes several seconds and most times completes in a few milliseconds. Restart of Postgres seems to resolve the issue and keep it away for a couple of days.
Table - alarm
20 clients reading from the table.
1 client making several inserts and udpates every second. Delete/purge is a nightly cron.
Plenty of RAM still available, only 20% of CPU on the host being used.
After the above conditions persist for a while, the query starts taking 5+ seconds to complete.
Just restart of the clients and Postgres solves this by bring it down to few milliseconds.
explain analyze shows a sequential scan on this table both the times. But it is in the order of milliseconds when Postgres is just started and later it increases to a few seconds.
There are about 25000 records in this table.
Any suggestions, ideas on how to debug when the seq scan is slow? Could it be fragmentation in the shared_buffer or locking of the table causing this. I doubt it is a disk or IO issue because they should cause a problem irrespective of when Postgres is started.
At the time of the issue, I missed using buffers in analyze. The output is like below:
explain analyze SELECT s.severity, s.severityid from SEVERITY s where s.severityid =(select MIN(severityid) from ALARM a, (SELECT * FROM RESOURCEDETAILS) as RESOURCEDETAILS WHERE RESOURCEDETAILS.resourcegroupid = 132 AND RESOURCEDETAILS.resourceid = a.resourceid AND a.severityid 1 AND a.severityid 2 AND a.severityid 7);
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------
Seq Scan on severity s (cost=4701.39..4702.47 rows=1 width=13) (actual time=7057.939..7057.940 rows=1 loops=1)
Filter: (severityid = $0)
Rows Removed by Filter: 6
InitPlan 1 (returns $0)
-> Aggregate (cost=4701.38..4701.39 rows=1 width=8) (actual time=7057.918..7057.918 rows=1 loops=1)
-> Hash Join (cost=533.69..4701.25 rows=51 width=8) (actual time=1849.630..7057.783 rows=223 loops=1)
Hash Cond: (a.resourceid = resourcedetails.resourceid)
-> Seq Scan on alarm a (cost=0.00..4157.70 rows=2491 width=16) (actual time=45.792..6608.120 rows=852 loops=1)
Filter: ((severityid 1) AND (severityid 2) AND (severityid 7))
Rows Removed by Filter: 24376
-> Hash (cost=528.27..528.27 rows=434 width=8) (actual time=448.540..448.540 rows=19 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 1kB
-> Bitmap Heap Scan on resourcedetails (cost=7.65..528.27 rows=434 width=8) (actual time=0.128..448.495 rows=19 loops=1)
Recheck Cond: (resourcegroupid = 132)
-> Bitmap Index Scan on resourcedetails_rg_idx (cost=0.00..7.54 rows=434 width=0) (actual time=0.103..0.103 rows=63 loops=1)
Index Cond: (resourcegroupid = 132)
Total runtime: 7058.156 ms
(17 rows)
After postgres restart, the problem was resolved and here is the output:
(This time with all options of explain)
<pre>
explain (analyze,buffers,costs,timing,verbose) SELECT s.severity, s.severityid from SEVERITY s where s.severityid =(select MIN(severityid) from ALARM a, (SELECT * FROM RESOURCEDETAILS) as RESOURCEDETAILS WHERE RESOURCEDETAILS.resourcegroupid = 132 AND RESOURCEDETAILS.resourceid = a.resourceid AND a.severityid <> 1 AND a.severityid <> 2 AND a.severityid <> 7);
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------
Seq Scan on public.severity s (cost=4516.93..4518.02 rows=1 width=13) (actual time=15.881..15.882 rows=1 loops=1)
Output: s.severity, s.severityid
Filter: (s.severityid = $0)
Rows Removed by Filter: 6
Buffers: shared hit=3167
InitPlan 1 (returns $0)
-> Aggregate (cost=4516.92..4516.93 rows=1 width=8) (actual time=15.874..15.875 rows=1 loops=1)
Output: min(a.severityid)
Buffers: shared hit=3166
-> Hash Join (cost=520.61..4516.91 rows=5 width=8) (actual time=7.043..15.869 rows=2 loops=1)
Output: a.severityid
Hash Cond: (a.resourceid = resourcedetails.resourceid)
Buffers: shared hit=3166
-> Seq Scan on public.alarm a (cost=0.00..3995.37 rows=232 width=16) (actual time=0.142..15.492 rows=108 loops=1)
Output: a.alarmid, a.eventtypeid, a.severityid, a.source, a.resourceid, a.componentid, a.category, a.createdtime, a.updatedtime, a.owner, a.ackstatus, a.acktime, a.clearedtime, a.messa
ge, a.remedy, a.optionalfield, a.version
Filter: ((a.severityid <> 1) AND (a.severityid <> 2) AND (a.severityid <> 7))
Rows Removed by Filter: 26130
Buffers: shared hit=3108
-> Hash (cost=515.39..515.39 rows=418 width=8) (actual time=0.338..0.338 rows=19 loops=1)
Output: resourcedetails.resourceid
Buckets: 1024 Batches: 1 Memory Usage: 1kB
Buffers: shared hit=58
-> Bitmap Heap Scan on public.resourcedetails (cost=7.53..515.39 rows=418 width=8) (actual time=0.033..0.330 rows=19 loops=1)
Output: resourcedetails.resourceid
Recheck Cond: (resourcedetails.resourcegroupid = 132)
Buffers: shared hit=58
-> Bitmap Index Scan on resourcedetails_rg_idx (cost=0.00..7.42 rows=418 width=0) (actual time=0.023..0.023 rows=61 loops=1)
Index Cond: (resourcedetails.resourcegroupid = 132)
Buffers: shared hit=3
Total runtime: 15.937 ms
(30 rows)
After postgresql restart, the query seems to deteriorate steadily...
Please let me know if you need the table definitions. As I'm sure is obvious, I have several tables holding information about a user (partydevicestatus, groupgps) each have a foreign key relationship to the user held in partyrelationship and each row has an identifier of the "group" that user's in.
With this query, I simply want to, for a particular group (in this example, 6) get the user details, position and device info for each user.
I can clearly see with the explain that the Sort is the issue here, due to having 2 columns with a lot of data. However I have an index on both the columns being sorted on and it has yielded no improvement. I'm almost certain this is a terribly optimised query but I am not experienced enough with PostgreSQL to find a better one?
SELECT DISTINCT ON("public".groupgps.groupmember)
"public".groupgps.groupgps,
"public".groupgps.groupmember,
"public".groupgps.messagetype,
"public".groupgps.lat,
"public".groupgps.lon,
"public".groupgps.date_stamp,
"public".partyrelationship.to_party,
"public".partyrelationship.to_name,
"public".partyrelationship.image_url,
"public".partyrelationship.partyrelationship,
"public".partydevicestatus.connection_type,
"public".partydevicestatus.battery_level,
"public".partydevicestatus.charging_state,
"public".partydevicestatus.timestamp
FROM "public".groupgps
INNER JOIN "public".partyrelationship
ON "public".partyrelationship.partyrelationship = "public".groupgps.groupmember
INNER JOIN "public".partysettings
ON "public".partysettings.groupmember = "public".groupgps.groupmember
LEFT JOIN "public".partydevicestatus
ON "public".partydevicestatus.groupmember_id = "public".groupgps.groupmember
WHERE "public".partyrelationship.from_party = 6
AND "public".partysettings.gps_tracking_enabled = true
ORDER BY "public".groupgps.groupmember, "public".groupgps.date_stamp DESC
Explain Result
Unique (cost=1368961.43..1390701.85 rows=25 width=192) (actual time=24622.609..27043.061 rows=4 loops=1)
-> Sort (cost=1368961.43..1379831.64 rows=4348083 width=192) (actual time=24622.601..26604.659 rows=2221853 loops=1)
Sort Key: groupgps.groupmember, groupgps.date_stamp DESC
Sort Method: external merge Disk: 431400kB
-> Hash Left Join (cost=50.64..87013.93 rows=4348083 width=192) (actual time=0.499..3011.806 rows=2221853 loops=1)
Hash Cond: (groupgps.groupmember = partydevicestatus.groupmember_id)
-> Hash Join (cost=31.66..29732.32 rows=77101 width=167) (actual time=0.153..2242.950 rows=109041 loops=1)
Hash Cond: (groupgps.groupmember = partyrelationship.partyrelationship)
-> Seq Scan on groupgps (cost=0.00..24372.00 rows=1217200 width=50) (actual time=0.005..1933.528 rows=1217025 loops=1)
-> Hash (cost=31.48..31.48 rows=14 width=125) (actual time=0.141..0.141 rows=5 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 9kB
-> Hash Join (cost=10.31..31.48 rows=14 width=125) (actual time=0.092..0.138 rows=5 loops=1)
Hash Cond: (partysettings.groupmember = partyrelationship.partyrelationship)
-> Seq Scan on partysettings (cost=0.00..20.75 rows=75 width=8) (actual time=0.003..0.038 rows=75 loops=1)
Filter: gps_tracking_enabled
-> Hash (cost=9.79..9.79 rows=42 width=117) (actual time=0.076..0.076 rows=42 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 11kB
-> Seq Scan on partyrelationship (cost=0.00..9.79 rows=42 width=117) (actual time=0.007..0.058 rows=42 loops=1)
Filter: (from_party = 6)
Rows Removed by Filter: 181
-> Hash (cost=12.88..12.88 rows=488 width=29) (actual time=0.341..0.341 rows=489 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 41kB
-> Seq Scan on partydevicestatus (cost=0.00..12.88 rows=488 width=29) (actual time=0.023..0.163 rows=489 loops=1)
Planning time: 0.878 ms
Execution time: 27218.016 ms