I'm trying to sort out a very weird behavior.
I'm working with:
JBoss AS 7.1.1
EJB 3.0
JPA
XA DataSource
Oracle 11g
In one of the systems fuctionalitites, the user can see the status of each Store. For each Store I fire a query, to sum up all the files that have been processed. The query is something like this:
SELECT
SUM(CASE file.type
WHEN 'TYPE_1' THEN 1
ELSE 0
END)
,
SUM(CASE file.type
WHEN 'TYPE_2' THEN 1
ELSE 0
END)
,
SUM(CASE file.type
WHEN 'TYPE_3' THEN 1
ELSE 0
END)
FROM
File file
WHERE
file.type IN ('TYPE_1', 'TYPE_2', 'TYPE_3')
AND file.status = 'RECEIVED'
AND file.store.id = :storeId
The thing is, the user can select which of the stores he wants to check, and that's where things get weird.
When I check the first store, the result comes blazing fast, but all subsequent queries take significantly more time. Let me exemplify:
User checks store 15 (Blazing fast result) - About 200 ms
User checks store 2 (Very slow result) - About 8000 ms
Now pay attention to this part, it's very important.
User logs out, and logs in again.
User checks store 2 (the one that took 8000ms), and now the result is blazing fast.
This is very odd, the same store that took a while before, is now loading pretty fast.
Whenever I try the queries on SQLDeveloper the results come pretty fast as well.
I annotated my EJB with #TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) but I didn't get any difference on the execution time.
I created a standalone project to run the queries using JDBC and the result was fast again, which leaves me thinking it may be some configuration on my DataSource, persistence.xml or anything like that.
Does anyone have any clues why this happens?
few things :
when the user checks store2 in the second time , the oracle optimizer is probably using it's "cache" , and therefore it is blazing fast.
how much File records does store2 has ? try to perform a group by sentence to see if this File table needs special statistics , if for example the store2 has dramaticly more File records than other stores then try to perform this method :
begin
dbms_stats.gather_table_stats(user,'file' , estimate_percent=>100);
end;
this will ensure the table's statistics are accurate.
you can optimize the query , you don't have to perform 3 times "sum" , you can do something like this :
select f.type , count(*)
from File f
where f.store.id = :storeId
and f.type IN ('TYPE_1', 'TYPE_2', 'TYPE_3')
group by f.type
you may run in a cardinality feedback problem; just look in this blog;
http://orcasoracle.squarespace.com/oracle-rdbms/2012/12/18/when-a-query-runs-slower-on-second-execution-a-possible-side.html
/KR
Related
I'm trying to locate the cause of a slow query that hits 3 tables with records ranging from a few hundred thousand to a several million
tango - 6166101
kilo_golf - 822805
three_romeo - 535782
Version
PostgreSQL 11.10
Current query
select count(*) as aggregate
from "tango"
where "lima" = juliet
and not exists(select 1
from "three_romeo" quebec_seven_oscar
where quebec_seven_oscar.six_two = tango.six_two
and quebec_seven_oscar."romeo" >= six_seven
and quebec_seven_oscar."three_seven" in
('partial_survey', 'survey_completed', 'wrong_number', 'moved'))
and ("mike" <= '2021-02-03 13:26:22' or "mike" is null)
and not exists(select 1
from "kilo_golf" as "delta"
where "delta"."to" = tango.six_two
and "two" = november
and "delta"."romeo" >= '2021-02-05 13:49:15')
and not exists(select 1
from "three_romeo" as "four"
where "four".foxtrot = tango.quebec_seven_victor
and "four"."three_seven" in ('deceased', 'block_calls', 'block_all'))
and "tango"."yankee" is null;
This is the analysis of the query in its current state - https://explain.depesz.com/s/Di51
It feels like the problematic area is in the tango table
tango.lima is equal to 'juliet' in the majority of records (low cardinality), we don't currently have an index on this
The long filter makes me wonder if I should create some sort of composite index?
After reading another post (https://stackoverflow.com/a/50148594/682754) tried removing the or "mike" is null and this helped quite a lot
https://explain.depesz.com/s/XgmB
Should I try and remove the not exists in favour of using joins?
Thanks
I don't think that using explicit joins will help you, since PostgreSQL converts NOT EXISTS into an anti-join anyway.
But you spotted the problem: it is the OR. I would recommend that you use a dynamic query: add the cindition only if mikeis not NULL rather than having a static query with OR.
You are counting about 6 million rows, and that will take some time. The reason that removing or "mike" is null can help so much is that it no longer needs to count the rows where mike is null, which is vast majority of them.
But this is of no use to you if you actually do need to count those rows. So, do you? I'm having a hard time picturing a situation in which you need an exact count of 6 million rows often enough that waiting 4 seconds for it is a problem.
I am writing dynamic sql code and it would be easier to use a generic where column in (<comma-seperated values>) clause, even when the clause might have 1 term (it will never have 0).
So, does this query:
select * from table where column in (value1)
have any different performance than
select * from table where column=value1
?
All my test result in the same execution plans, but if there is some knowledge/documentation that sets it to stone, it would be helpful.
This might not hold true for each and any RDBMS as well as for each an any query with its specific circumstances.
The engine will translate WHERE id IN(1,2,3) to WHERE id=1 OR id=2 OR id=3.
So your two ways to articulate the predicate will (probably) lead to exactly the same interpretation.
As always: We should not really bother about the way the engine "thinks". This was done pretty well by the developers :-) We tell - through a statement - what we want to get and not how we want to get this.
Some more details here, especially the first part.
I Think this will depend on platform you are using (optimizer of the given SQL engine).
I did a little test using MySQL Server and:
When I query select * from table where id = 1; i get 1 total, Query took 0.0043 seconds
When I query select * from table where id IN (1); i get 1 total, Query took 0.0039 seconds
I know this depends on Server and PC and what.. But The results are very close.
But you have to remember that IN is non-sargable (non search argument able), it will not use the index to resolve the query, = is sargable and support the index..
If you want the best one to use, You should test them in your environment because they both work so good!!
I'm using EF Core 3.0 code first with MSSQL database. I have big table that has ~5 million records. I have indexes on ProfileId, EventId and UnitId. This query takes ~25-30 seconds to execute. Is it normal or there is a way to optimize it?
await (from x in _dbContext.EventTable
where x.EventId == request.EventId
group x by new { x.ProfileId, x.UnitId } into grouped
select new
{
ProfileId = grouped.Key.ProfileId,
UnitId = grouped.Key.UnitId,
Sum = grouped.Sum(a => a.Count * a.Price)
}).AsNoTracking().ToListAsync();
I tried to loos through profileIds, adding another WHERE clause and removing ProfileId from grouping parameter, but it worked slower.
Capture the SQL being executed with a profiling tool (SSMS has one, or Express Profiler) then run that within SSMS /w execution plan enabled. This may highlight an indexing improvement. If the execution time in SSMS roughly correlates to what you're seeing in EF then the only real avenue of improvement will be hardware on the SQL box. You are running a query that will touch 5m rows any way you look at it.
Operations like this are not that uncommon, just not something that a user would expect to sit and wait for. This is more of a reporting-type request so when faced with requirements like this I would look at options to have users queue up a request where they can receive a notification when the operation completes to fetch the results. This would be set up to prevent users from repeatedly requesting updates ("not sure if I clicked" type spams) or also considerations to ensure too many requests from multiple users aren't kicked off simultaneously. Ideally this would be a candidate to run off a read-only reporting replica rather than the read-write production DB to avoid locks slowing/interfering with regular operations.
Try to remove ToListAsync(). Or replace it with AsQueryableAsync(). Add ToList slow performance down.
await (from x in _dbContext.EventTable
where x.EventId == request.EventId
group x by new { x.ProfileId, x.UnitId } into grouped
select new
{
ProfileId = grouped.Key.ProfileId,
UnitId = grouped.Key.UnitId,
Sum = grouped.Sum(a => a.Count * a.Price)
});
I'm using FB 2.5.5 and I'm trying to understand why a very simple query does not use an index and thus takes forever to execute. I've read a lot of articles about why existing indices might be ignored by the query optimizer but I'm not understanding how it can happens in my case. I recomputed the selectivity for all my indices within IB Expert, and I've also done a backup/restore of the database to be sure I wasn't missing something.
The index selectivity, as displayed by IB Expert, is approx 0,000024 - which is far from 1 :
CREATE INDEX TVERSIONS_IDX_LASTMODDATE ON TVERSIONS (LASTMODDATE)
The table I'm querying contains approx. 2M records :
SELECT COUNT(ID) FROM TVERSIONS
2479518
I'm trying to fetch all records based on the LASTMODDATE field (TIMETSAMP, indexed by TVERSIONS_IDX_LASTMODDATE). An oversimplified version of the query would be :
SELECT COUNT(ID) FROM TVERSIONS WHERE LASTMODDATE > :TheDate
In this case, the execution plan shows that the index is actually used :
Plan
PLAN (TVERSIONS INDEX (TVERSIONS_IDX_LASTMODDATE))
...and records matching the condition are fetched very quickly :
------ Performance info ------
Prepare time = 172ms
Execute time = 16ms <----
Avg fetch time = 16,00 ms
Current memory = 2 714 672
Max memory = 10 128 480
Memory buffers = 90
Reads from disk to cache = 57
Writes from cache to disk = 0
Fetches from cache = 387
Now, the "real" query fetches the same fields using the same condition on LASTMODDATE but adds a JOIN over 3 tables :
SELECT COUNT(ID) FROM TVERSIONS
JOIN TFILES ON TFILES.ID = TVERSIONS.FILEID
JOIN TROOTS ON TROOTS.ID = TFILES.ROOTID
JOIN TUSERSBACKUPS ON TROOTS.BACKUPID = TUSERSBACKUPS.BACKUPID
WHERE TUSERSBACKUPS.USERID= :UserID
AND TVERSIONS.LASTMODDATE >:TheDate
Now the query plan does not use the index anymore :
Plan
PLAN JOIN (TUSERSBACKUPS INDEX (RDB$FOREIGN4), TROOTS INDEX (RDB$FOREIGN3), TFILES INDEX (RDB$FOREIGN2), TVERSIONS INDEX (RDB$FOREIGN6))
Without any surprise execution time is far more slower (approx. 1 minute):
------ Performance info ------
Prepare time = 329ms
Execute time = 53s 593ms <---
Avg fetch time = 53 593,00 ms
Current memory = 3 044 736
Max memory = 10 128 480
Memory buffers = 90
Reads from disk to cache = 55 732
Writes from cache to disk = 0
Fetches from cache = 6 952 648
In other words, searching the WHOLE table is magnitude faster than searching into a subset of rows returned by JOIN.
I can't understand why the index on the LASTMODDATE field is not used anymore just because I'm adding the join clause. The selectivity of the index is good and the query is very simple. What do I miss ?
It seems Firebird decided to start with condition TUSERSBACKUPS.USERID=:UserID using index RDB$FOREIGN4. Probably it happens because you have here equality, and for condition TVERSIONS.LASTMODDATE >:TheDate you have inequality which could lead to potentially larger set of records (for example if TheDate is a date 200 years ago it will include the whole table).
To force Firebird use a plan which you (but not its optimizer) prefer - use PLAN clause, see http://www.firebirdfaq.org/faq224/
I think I've understood what happened, and... I guess it was my fault.
I forgot that the table I'm querying has been "denormalized" to avoid such long JOINs. The problematic query can indeed by rewritten in a much shorter way :
SELECT COUNT(TVERSIONS.ID) FROM TVERSIONS
JOIN TUSERSBACKUPS ON TUSERSBACKUPS.BACKUPID = TVERSIONS.RD_BACKUPID
WHERE TUSERSBACKUPS.USERID= :UserID
AND TVERSIONS.LASTMODDATE >:TheDate
This one properly uses the indices I set before and has a very short execution time.
I have the impression that when Firebird detects you're deliberately using a sub-optimal path to access records in a table it does not even try to use your indices and let you shoot yourself in the foot...
Anyway, the problem is solved. Thank you all for your suggestions.
A friend wrote a query with the following condition:
AND ( SELECT count(1) FROM users_alerts_status uas
WHERE uas.alert_id = context_alert.alert_id
AND uas.user_id = 18309
AND uas.status = 'read' ) = 0
Seeing this, I suggested we change it to:
AND NOT EXISTS ( SELECT 1 FROM users_alerts_status uas
WHERE uas.alert_id = context_alert.alert_id
AND uas.user_id = 18309
AND uas.status = 'read' )
But in testing, the first version of the query is consistently between 20 and 30ms faster (we tested after restarting the server). Conceptually, what am I missing?
My guess would be that the first one can short circuit; as soon as it sees any rows that match the criteria, it can return the count of 1. The second one needs to check every row (and it returns a row of "1" for every result), so doesn't get the speed benefit of short circuiting.
That being said, doing an EXPLAIN (or whatever your database supports) might give a better insight than my guess.
Conceptually, I'd say that your option is at least as good as the other, at least a little more elegant. I'm not sure if it should be slower or faster - and if those 25ms are relevant.
The definite answer, usually, comes by looking at the EXPLAIN output.
What Postgresql version is that? PG 8.4 is said to have some optimizations regarding NOT EXISTS