I have a query that's run on a table (TABLE_A) that's partitioned by recTime……
WITH subquery AS (
select count(*) AS cnt, date_trunc('day',recTime) AS recTime
from TABLE_A
WHERE (recTime >= to_timestamp('2018-Nov-03 00:00:00','YYYY-Mon-DD HH24:MI:SS')
AND recTime <= to_timestamp('2018-Nov-03 23:59:59','YYYY-Mon-DD HH24:MI:SS'))
GROUP BY date_trunc('day',recTime)
)
UPDATE sumDly
SET fixes = subquery.cnt
FROM subquery
WHERE
sumDly.Day = subquery.recTime
If I do an explain on the query as shown above it's apparent that the database is doing an index scan on each of the partition in the parent table. The associated cost is high and the elapsed time is ridiculous.
If I explicitly force the use of the partition that actually has the data in by replacing….
from TABLE_A
With….
from TABLE_A_20181103
Then the explain only uses the required partition, and the query takes only a few minutes (and the returned results are the same as before)
QUESTION - Why does the database want to scan all the partitions in the table? I though the whole idea of partitioning was to help the database eliminate vast swathes of unneeded data in the first pass rather than forcing a scan on all the indexes in individual partitions?
UPDATE - I am using version 10.5 of postgres
SET constraint_exclusion = on and make sure to hard-code the constraints into the query. Note that you have indeed hard-coded the constrains in the query:
...
WHERE (recTime >= to_timestamp('2018-Nov-03 00:00:00','YYYY-Mon-DD HH24:MI:SS')
AND recTime <= to_timestamp('2018-Nov-03 23:59:59','YYYY-Mon-DD HH24:MI:SS'))
...
Related
my table contains 1 billion records. It is also partitioned by month.Id and datetime is the primary key for the table. When I select
select col1,col2,..col8
from mytable t
inner join cte on t.Id=cte.id and dtime>'2020-01-01' and dtime<'2020-10-01'
It uses index scan, but takes more than 5 minutes to select.
Please suggest me.
Note: I have set work_mem to 1GB. cte table results comes with in 3 seconds.
Well it's the nature of join and it is usually known as a time consuming operation.
First of all, I recommend to use in rather than join. Of course they have got different meanings, but in some cases technically you can use them interchangeably. Check this question out.
Secondly, according to the relation algebra whenever you use join each rows of mytable table is combined with each rows from the second table, and DBMS needs to make a huge temporary table, and finally igonre unsuitable rows. Undoubtedly all the steps and the result would take much time. Before using the Join opeation, it's better to filter your tables (for example mytable based date) and make them smaller, and then use the join operations.
We're currently running a query that performs a pretty simple join and group by for a row count and at the end a union all.
(select
table_p."name",
table_p.id,
count(table.id),
sum(table.views)
from table
inner join table_p on table_p.id = table.pageid
where table.date BETWEEN '2020-03-01' AND '2020-03-31'
group by table_p.id
order by table_p.id)
union all
(select
table_p."name",
table_p.id,
count(table.id),
sum(table.views)
from table
inner join table_p on table_p.id = table.pageid
where table.date BETWEEN '2020-02-01' AND '2020-02-29'
group by table_p.id
order by table_p.id)
union all ....
We've decided to use a BRIN index due to the count of our table being 360 million records. We do have the option to go with B-Tree if needed.
Now for some reason, we're seeing in the explain analyze that the BRIN Index has "parallel aware" set to false with two workers being listed in the plan outputted? Also we're seeing a linear performance when breaking up the amount that we're querying, i.e. one month in 5 seconds, four months in 20 seconds. I'd assume this means that we're querying asynchronously rather than parallel.
Does anyone have any ideas on what we could potentially be missing in order to get parallel queries going where possible? Does BRIN not work with Parallel Workers?
Edit: Here is the BRIN index on "table":
CREATE INDEX table_brin_idx
ON table USING brin
(date, teamid, id, pageid, devicetypeid, makeid, modelid)
TABLESPACE pg_default;
My postgres version is PostgreSQL 11.6, compiled by Visual C++ build 1800, 64-bit
Here's a link to the explain analyze that's too big to post here.
Information from PostgreSQL documentation: Currently, parallel index scans are supported only for btree indexes.
Source: https://www.postgresql.org/docs/11/parallel-plans.html#PARALLEL-SCANS
My app reports rainfall and streamflow information to whitewater boaters. Postgres is my data store for the gauge readings that come in 15 minute intervals. Over time, these tables get pretty big and the availabity of range partitioning in Postgres 10 inspired me to leave my shared hosting service and build a server from scratch at Linode. My queries on these large tables became way faster after I partitioned the readings into 2 week chunks. Several months down the road, I checked out the query plan and was very surprised to see that using now() in a query caused PG to scan allof the indexes on my partitioned tables. What the heck?!?! Isn't the point of partitiong data is to avoid situations like this?
Here's my set up: my partitioned table
CREATE TABLE public.precip
(
gauge_id smallint,
inches numeric(8, 2),
reading_time timestamp with time zone
) PARTITION BY RANGE (reading_time)
I've created partitions for every two weeks, so I have about 50 partition tables so far. One of my partitions:
CREATE TABLE public.precip_y2017w48 PARTITION OF public.precip
FOR VALUES FROM ('2017-12-03 00:00:00-05') TO ('2017-12-17 00:00:00-05');
Then each partition is indexed on gauge_id and reading_time
I have a lot of queries like
WHERE gauge_id = xxx
AND precip.reading_time > (now() - '01:00:00'::interval)
AND precip.reading_time < now()
As I mentioned postgres scans all of the indexes on reading_time for every 'child' table rather than only querying the child table that has timestamps in the query range. If I enter literal values (e.g., precip.reading_time > '2018-03-01 01:23:00') instead of now(), it only scans the indexes of appropriate child tables(s). I've done some reading and I understand that now() is volatile and that the planner won't know what the value will be when the query executes. I've also read that query planning is expensive so postgres caches plans. I can understand why PG is programmed to do that. However, one counter argument I read was that a re-planned query is probably way less expensive than a query that end up ignoring partitions. I agree - and that's probably the case in my situation.
As a work arounds, I've created this function:
CREATE OR REPLACE FUNCTION public.hours_ago2(i integer)
RETURNS timestamp with time zone
LANGUAGE 'plpgsql'
COST 100
IMMUTABLE
ROWS 0
AS $BODY$
DECLARE X timestamp with time zone;
BEGIN
X:= now() + cast(i || ' hours' as interval);
RETURN X;
END;
$BODY$;
Note the IMMUTABLE statment. Now when issue queries like
select * from stream
where gauge_id = 2142 and reading_time > hours_ago2(-3)
and reading_time < hours_ago2(0)
PG only searches the partition table that stores data for that time frame. This is the goal I was shooting for when I set up the partitions in the first place. Booyah. But is this safe? Will the query planner ever cache the results of hours_ago2(-3) and use it over and over again for hours down the road? It's ok if it's cached for a few minutes. Again, my app reports rain and streamflow information; it doesn't deal with financial transactions or any other 'critical' types of data processing. I've tested simple statements like select hours_ago2(-3) and it returns new values every time. So it seems safe. But is it really?
That is not safe because at planning time you have no idea if the statement will be executed in the same transaction or not.
If you are in a situation where query plans are cached, this will return wrong results. Query plans are cached for named prepared statements and statements in PL/pgSQL functions, so you could end up with an out-of-date value for the duration of the database session.
For example:
CREATE TABLE times(id integer PRIMARY KEY, d timestamptz NOT NULL);
PREPARE x AS SELECT * FROM times WHERE d > hours_ago2(1);
The function is evaluated at planning time, and the result is a constant in the execution plan (for immutable functions that is fine).
EXPLAIN (COSTS off) EXECUTE x;
QUERY PLAN
---------------------------------------------------------------------------
Seq Scan on times
Filter: (d > '2018-03-12 14:25:17.380057+01'::timestamp with time zone)
(2 rows)
SELECT pg_sleep(100);
EXPLAIN (COSTS off) EXECUTE x;
QUERY PLAN
---------------------------------------------------------------------------
Seq Scan on times
Filter: (d > '2018-03-12 14:25:17.380057+01'::timestamp with time zone)
(2 rows)
The second query definitely does not return the result you want.
I think you should evaluate now() (or better an equivalent function on the client side) first, perform your date arithmetic and supply the result as parameter to the query. Inside of PL/pgSQL functions, use dynamic SQL.
Change the queries to use 'now'::timestamptz instead of now(). Also, interval math on timestamptz is not immutable.
Change your query to something like:
WHERE gauge_id = xxx
AND precip.reading_time > ((('now'::timestamptz AT TIME ZONE 'UTC') - '01:00:00'::interval) AT TIME ZONE 'UTC')
AND precip.reading_time < 'now'::timestamptz
I am trying to count all the records created yesterday. There is a created_at column and it is indexed.
If i run
explain
select count(*) from events where created_at::date = current_date - 1;
It says
Aggregate (cost=14365728.05..14365728.06 rows=1 width=0)
-> Index Only Scan using index_events_created_at on events (cost=0.57..14362310.20 rows=1367140 width=0)
Filter: ((created_at)::date = (('now'::cstring)::date - 1))
So it event kind of knows how many rows there are. But the
select count(*) from events where created_at::date = current_date - 1;
query itself keeps running forever. Why is that?
TRY this:
SELECT count(*)
FROM events
WHERE created_at >= current_date - 1
AND created_at < current_date;
So, to start: Why is the explain plan able to provide an estimated row count so much quicker than the query can run?
The optimizer is estimating the row count based on stored statistics and/or extrapolations from stored statistics. As you can see, this isn't necessary very accurate. (Based on comment discussion, the estimate was off by almost 20%.) So the query has to actually count, based on either data in the table or data in the index. So that's more work. But it's not obvious why it's 10 minutes worth of "more work".
One reasonable guess would be lock contention. Depending on your transaction isolation settings, it could be that your query keeps having to wait on inserts or updates to the table to finish. (The optimizer wouldn't have this problem in calculating its estimate, because it will just assume that the effects of concurrent queries are not a big deal for its purposes.) Even though none of the added data would affect your count, table-level locks could still conflict.
One way to test this theory would be to copy the table, so that you have a table with the same data (and same indexes, etc) that nobody's querying, and see if your count runs faster against it.
(As an aside: In general when the stats seem significantly off you could suspect that the optimizer had picked a poor execution plan; but it's hard to see how an index scan could be the wrong solution here.)
I have table
create table big_table (
id serial primary key,
-- other columns here
vote int
);
This table is very big, approximately 70 million rows, I need to query:
SELECT * FROM big_table
ORDER BY vote [ASC|DESC], id [ASC|DESC]
OFFSET x LIMIT n -- I need this for pagination
As you may know, when x is a large number, queries like this are very slow.
For performance optimization I added indexes:
create index vote_order_asc on big_table (vote asc, id asc);
and
create index vote_order_desc on big_table (vote desc, id desc);
EXPLAIN shows that the above SELECT query uses these indexes, but it's very slow anyway with a large offset.
What can I do to optimize queries with OFFSET in big tables? Maybe PostgreSQL 9.5 or even newer versions have some features? I've searched but didn't find anything.
A large OFFSET is always going to be slow. Postgres has to order all rows and count the visible ones up to your offset. To skip all previous rows directly you could add an indexed row_number to the table (or create a MATERIALIZED VIEW including said row_number) and work with WHERE row_number > x instead of OFFSET x.
However, this approach is only sensible for read-only (or mostly) data. Implementing the same for table data that can change concurrently is more challenging. You need to start by defining desired behavior exactly.
I suggest a different approach for pagination:
SELECT *
FROM big_table
WHERE (vote, id) > (vote_x, id_x) -- ROW values
ORDER BY vote, id -- needs to be deterministic
LIMIT n;
Where vote_x and id_x are from the last row of the previous page (for both DESC and ASC). Or from the first if navigating backwards.
Comparing row values is supported by the index you already have - a feature that complies with the ISO SQL standard, but not every RDBMS supports it.
CREATE INDEX vote_order_asc ON big_table (vote, id);
Or for descending order:
SELECT *
FROM big_table
WHERE (vote, id) < (vote_x, id_x) -- ROW values
ORDER BY vote DESC, id DESC
LIMIT n;
Can use the same index.
I suggest you declare your columns NOT NULL or acquaint yourself with the NULLS FIRST|LAST construct:
PostgreSQL sort by datetime asc, null first?
Note two things in particular:
The ROW values in the WHERE clause cannot be replaced with separated member fields. WHERE (vote, id) > (vote_x, id_x) cannot be replaced with:
WHERE vote >= vote_x
AND id > id_x
That would rule out all rows with id <= id_x, while we only want to do that for the same vote and not for the next. The correct translation would be:
WHERE (vote = vote_x AND id > id_x) OR vote > vote_x
... which doesn't play along with indexes as nicely, and gets increasingly complicated for more columns.
Would be simple for a single column, obviously. That's the special case I mentioned at the outset.
The technique does not work for mixed directions in ORDER BY like:
ORDER BY vote ASC, id DESC
At least I can't think of a generic way to implement this as efficiently. If at least one of both columns is a numeric type, you could use a functional index with an inverted value on (vote, (id * -1)) - and use the same expression in ORDER BY:
ORDER BY vote ASC, (id * -1) ASC
Related:
SQL syntax term for 'WHERE (col1, col2) < (val1, val2)'
Improve performance for order by with columns from many tables
Note in particular the presentation by Markus Winand I linked to:
"Pagination done the PostgreSQL way"
Have you tried partioning the table ?
Ease of management, improved scalability and availability, and a
reduction in blocking are common reasons to partition tables.
Improving query performance is not a reason to employ partitioning,
though it can be a beneficial side-effect in some cases. In terms of
performance, it is important to ensure that your implementation plan
includes a review of query performance. Confirm that your indexes
continue to appropriately support your queries after the table is
partitioned, and verify that queries using the clustered and
nonclustered indexes benefit from partition elimination where
applicable.
http://sqlperformance.com/2013/09/sql-indexes/partitioning-benefits