We just recently moved our DB from 9i to 10G
(Yes..better late than never and No - moving to 11g is currently not an option :-))
Details of my Oracle 10G DB are :-
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Prod
PL/SQL Release 10.2.0.1.0 - Production
CORE 10.2.0.1.0 Production
I am faced with a very weird problem since that move.
A query that was and still is working fine with 9i just wont work on 10G.
I did search through other SO questions related to rownum but couldnt really find anything similar.
SQL Query is :-
SELECT * FROM
( SELECT field1, field2 , field3, field4, field5, field6, field7, to_char(rownum) field8
FROM
( SELECT
field1,
field2,
field3,
field4,
field5,
field6,
field7,
''
FROM
.......REST OF MY COMPLEX INNER QUERY
)
)
WHERE field8 BETWEEN 21 AND 30;
Basically, the 21 / 30 are numbers that are the index of the records passed to the query for pagination and in 9i, this query works like expected and returns the specified set of data only.
However in 10G, this same query does not work at all - always returns 0 records.
If i comment the rownum related parts of the query:-
to_char(rownum) field8 and
WHERE field8 BETWEEN 21 AND 30;
then i get the entire result set and thats great.
But since my intention is to do pagination using the rownum, the entire purpose is defeated.
Does anyone know of any reason why this query has stopped working with 10G.
I tried looking up any updates to the rownum implementation but havent been able to really come across anything that will help.
EDIT :-
While doing my debugging, i have come across something that to me, is making no sense.
I am putting in the entire query below as i cant explain without it.
SELECT * FROM
( SELECT field1, field2 , field3, field4, field5, field6, field7, to_char(rownum) field8 from
( SELECT PM.POLICY_NO field1
,PM.INSURED_CODE field2
,PM.INSURED_NAME field3
,TO_CHAR(PM.POLICY_EFFECTIVE_DATE,'DD/MM/YYYY') field4
,TO_CHAR(PM.POLICY_EXPIRATION_DATE,'DD/MM/YYYY') field5
,'' field6
,'' field7
,'' field8
FROM POLICY_MAIN PM
,POLICY_ENDORSEMENT_MAIN PEM
,MASTER_UW_LOB_CLASS MAS
WHERE PM.POLICY_NO = PEM.POLICY_NO
AND PM.POLICY_NO LIKE UPPER('%%')
AND PM.INSURED_CODE LIKE UPPER('%%')
AND PM.SOURCE_OF_BUSINESS LIKE UPPER('%%')
AND PM.POLICY_TYPE IS NULL
AND PM.POLICY_STATUS = 'POST'
AND PM.POLICY_LOB = MAS.UW_LOB_CODE
AND MAS.UW_CLASS_CODE LIKE UPPER('AUTO')
AND PEM.POLICY_ENDORSEMENT_NO =
(SELECT MAX(PEM2.POLICY_ENDORSEMENT_NO)
FROM POLICY_ENDORSEMENT_MAIN PEM2
WHERE PEM.POLICY_NO = PEM2.POLICY_NO
***AND PEM.ENDORSEMENT_STATUS = 'POST'***
)
***order by 1 ASC***
)
)
WHERE field8 BETWEEN 21 AND 40
Refer the lines marked between *** in the innermost subquery.
If i comment this line from my query, the query works fine.
AND PEM.ENDORSEMENT_STATUS = 'POST'
If i comment this line from my query and everything else remains unchanged from the original, the query works fine too
order by 1 ASC
The earlier points related to rownum still hold true but commenting these lines individually seems to be making the rownum thing irrelevant and the entire query works fine (except for that fact that the results are logically different now)
I am confused. To say the least!!!
EDIT 2:
Adding the execution plan for the above query
Execution Plan
----------------------------------------------------------
0 SELECT STATEMENT Optimizer=ALL_ROWS (Cost=19 Card=1 Bytes=114)
1 0 VIEW (Cost=19 Card=1 Bytes=114)
2 1 COUNT
3 2 FILTER
4 3 VIEW (Cost=17 Card=1 Bytes=128)
5 4 SORT (ORDER BY) (Cost=17 Card=1 Bytes=130)
6 5 TABLE ACCESS (BY INDEX ROWID) OF 'POLICY_ENDORSEMENT_MAIN' (TABLE) (Cost=2 Card=1 Bytes=39)
7 6 NESTED LOOPS (Cost=16 Card=1 Bytes=130)
8 7 NESTED LOOPS (Cost=14 Card=1 Bytes=91)
9 8 TABLE ACCESS (FULL) OF 'POLICY_MAIN' (TABLE) (Cost=14 Card=1 Bytes=82)
10 8 INDEX (UNIQUE SCAN) OF 'PK_MASTER_UW_LOB_CLASS' (INDEX (UNIQUE)) (Cost=0 Card=1 Bytes=9)
11 7 INDEX (RANGE SCAN) OF 'PK_POLICY_ENDORSEMENT_MAIN' (INDEX (UNIQUE)) (Cost=1 Card=1)
12 3 SORT (AGGREGATE)
13 12 FILTER
14 13 INDEX (RANGE SCAN) OF 'PK_POLICY_ENDORSEMENT_MAIN' (INDEX (UNIQUE)) (Cost=2 Card=2 Bytes=68)
EDIT 3:
Exact same query as above but if i remove the
ORDER BY 1 ASC
clause, then the results are retrieved as expected.
The PLAN for this query without the order by is below
Execution Plan
----------------------------------------------------------
0 SELECT STATEMENT Optimizer=ALL_ROWS (Cost=18 Card=1 Bytes=114)
1 0 VIEW (Cost=18 Card=1 Bytes=114)
2 1 COUNT
3 2 FILTER
4 3 TABLE ACCESS (BY INDEX ROWID) OF 'POLICY_ENDORSEMENT_MAIN' (TABLE) (Cost=2 Card=1 Bytes=39)
5 4 NESTED LOOPS (Cost=16 Card=1 Bytes=130)
6 5 NESTED LOOPS (Cost=14 Card=1 Bytes=91)
7 6 TABLE ACCESS (FULL) OF 'POLICY_MAIN' (TABLE) (Cost=14 Card=1 Bytes=82)
8 6 INDEX (UNIQUE SCAN) OF 'PK_MASTER_UW_LOB_CLASS' (INDEX (UNIQUE)) (Cost=0 Card=1 Bytes=9)
9 5 INDEX (RANGE SCAN) OF 'PK_POLICY_ENDORSEMENT_MAIN' (INDEX (UNIQUE)) (Cost=1 Card=1)
10 3 SORT (AGGREGATE)
11 10 FILTER
12 11 INDEX (RANGE SCAN) OF 'PK_POLICY_ENDORSEMENT_MAIN' (INDEX (UNIQUE)) (Cost=2 Card=2 Bytes=68)
Note that the only real difference between the two plans is that the one that is not working has the following two additional steps after step 3 where as these steps are not present in the query without the order by - which is working fine.
As expected, step 5 is the step where the ordering of the data is being done.
4 3 VIEW (Cost=17 Card=1 Bytes=128)
5 4 SORT (ORDER BY) (Cost=17 Card=1 Bytes=130)
It seems that step 4 is maybe an additional view being created due to the ordering.
WHY this should prevent the rownum logic from working is what i am still trying to grasp.
Any help appreciated!!
EDIT 4 - Original Query plan from 9i environment
Execution Plan
----------------------------------------------------------
0 SELECT STATEMENT Optimizer=CHOOSE
1 0 VIEW
2 1 COUNT
3 2 VIEW
4 3 SORT (ORDER BY)
5 4 FILTER
6 5 TABLE ACCESS (BY INDEX ROWID) OF 'POLICY_MAIN'
7 6 NESTED LOOPS
8 7 NESTED LOOPS
9 8 TABLE ACCESS (FULL) OF 'POLICY_ENDORSEMENT_MAIN'
10 8 INDEX (RANGE SCAN) OF 'PK_MASTER_UW_LOB_CLASS' (UNIQUE)
11 7 INDEX (RANGE SCAN) OF 'PK_POLICY_MAIN' (UNIQUE)
12 5 SORT (AGGREGATE)
13 12 FILTER
14 13 INDEX (RANGE SCAN) OF 'PK_POLICY_ENDORSEMENT_MAIN' (UNIQUE)
As Adam has suggested, the subquery is filtering the results after the sort and ROWNUM are applied.
I think you need to force that subquery to be filtered earlier, by using the PUSH_SUBQ hint:
SELECT * FROM
( SELECT field1, field2 , field3, field4, field5, field6, field7,
ROWNUM field8 from
( SELECT PM.POLICY_NO field1
,PM.INSURED_CODE field2
,PM.INSURED_NAME field3
,TO_CHAR(PM.POLICY_EFFECTIVE_DATE,'DD/MM/YYYY') field4
,TO_CHAR(PM.POLICY_EXPIRATION_DATE,'DD/MM/YYYY') field5
,'' field6
,'' field7
,'' field8
FROM POLICY_MAIN PM
,POLICY_ENDORSEMENT_MAIN PEM
,MASTER_UW_LOB_CLASS MAS
WHERE PM.POLICY_NO = PEM.POLICY_NO
AND PM.POLICY_NO LIKE UPPER('%%')
AND PM.INSURED_CODE LIKE UPPER('%%')
AND PM.SOURCE_OF_BUSINESS LIKE UPPER('%%')
AND PM.POLICY_TYPE IS NULL
AND PM.POLICY_STATUS = 'POST'
AND PM.POLICY_LOB = MAS.UW_LOB_CODE
AND MAS.UW_CLASS_CODE LIKE UPPER('AUTO')
AND PEM.POLICY_ENDORSEMENT_NO =
(SELECT /*+ PUSH_SUBQ*/
MAX(PEM2.POLICY_ENDORSEMENT_NO)
FROM POLICY_ENDORSEMENT_MAIN PEM2
WHERE PEM.POLICY_NO = PEM2.POLICY_NO
AND PEM.ENDORSEMENT_STATUS = 'POST'
)
order by 1 ASC
)
)
WHERE field8 BETWEEN 21 AND 40
I've also removed the TO_CHAR from the ROWNUM - you want to use numbers for that range comparison.
EDIT
Try #2 - use CTE instead:
WITH q AS
( SELECT /*+MATERIALIZE*/
field1, field2 , field3, field4, field5, field6, field7,
ROWNUM field8 from
( SELECT PM.POLICY_NO field1
,PM.INSURED_CODE field2
,PM.INSURED_NAME field3
,TO_CHAR(PM.POLICY_EFFECTIVE_DATE,'DD/MM/YYYY') field4
,TO_CHAR(PM.POLICY_EXPIRATION_DATE,'DD/MM/YYYY') field5
,'' field6
,'' field7
,'' field8
FROM POLICY_MAIN PM
,POLICY_ENDORSEMENT_MAIN PEM
,MASTER_UW_LOB_CLASS MAS
WHERE PM.POLICY_NO = PEM.POLICY_NO
AND PM.POLICY_NO LIKE UPPER('%%')
AND PM.INSURED_CODE LIKE UPPER('%%')
AND PM.SOURCE_OF_BUSINESS LIKE UPPER('%%')
AND PM.POLICY_TYPE IS NULL
AND PM.POLICY_STATUS = 'POST'
AND PM.POLICY_LOB = MAS.UW_LOB_CODE
AND MAS.UW_CLASS_CODE LIKE UPPER('AUTO')
AND PEM.POLICY_ENDORSEMENT_NO =
(SELECT MAX(PEM2.POLICY_ENDORSEMENT_NO)
FROM POLICY_ENDORSEMENT_MAIN PEM2
WHERE PEM.POLICY_NO = PEM2.POLICY_NO
AND PEM.ENDORSEMENT_STATUS = 'POST'
)
order by 1 ASC
)
)
SELECT * from q
WHERE field8 BETWEEN 21 AND 40
It sounds like Oracle is mergeing the inline view into the main query so that field8 (based on ROWNUM) is calculated too late. I haven't seen that happen myself, but if that is what is happening you could try adding a NO_MERGE hint like this:
SELECT /*+ NO_MERGE(vw) */ * FROM
( SELECT field1, field2 , field3, field4, field5, field6, field7, to_char(rownum) field8
FROM
( SELECT
field1,
field2,
field3,
field4,
field5,
field6,
field7,
''
FROM
.......REST OF MY COMPLEX INNER QUERY
)
) vw
WHERE field8 BETWEEN 21 AND 30;
(Incidentally, why the TO_CHAR on ROWNMUM when you are treating it as a number in the WHERE clause anyway?)
Try this:
SELECT field1, field2 , field3, field4, field5, field6, field7, to_char(rn) field8 from
(SELECT PM.POLICY_NO field1
,PM.INSURED_CODE field2
,PM.INSURED_NAME field3
,TO_CHAR(PM.POLICY_EFFECTIVE_DATE,'DD/MM/YYYY') field4
,TO_CHAR(PM.POLICY_EXPIRATION_DATE,'DD/MM/YYYY') field5
,'' field6
,'' field7
,rownum as rn
FROM POLICY_MAIN PM
inner join POLICY_ENDORSEMENT_MAIN PEM
on PM.POLICY_NO = PEM.POLICY_NO
inner join MASTER_UW_LOB_CLASS MAS
on PM.POLICY_LOB = MAS.UW_LOB_CODE
WHERE PM.POLICY_NO LIKE UPPER('%%')
AND PM.INSURED_CODE LIKE UPPER('%%')
AND PM.SOURCE_OF_BUSINESS LIKE UPPER('%%')
AND PM.POLICY_TYPE IS NULL
AND PM.POLICY_STATUS = 'POST'
AND MAS.UW_CLASS_CODE = 'AUTO'
AND PEM.ENDORSEMENT_STATUS = 'POST'
AND PEM.POLICY_ENDORSEMENT_NO =
(SELECT MAX(PEM2.POLICY_ENDORSEMENT_NO)
FROM POLICY_ENDORSEMENT_MAIN PEM2
WHERE PEM.POLICY_NO = PEM2.POLICY_NO
)
order by pm.policy_no ASC)
WHERE rn BETWEEN 21 AND 40
Changes:
Restructured joins to use ANSI syntax to differentiate joins from filters.
Changed LIKE UPPER('AUTO') to = 'AUTO'
Removed unnecessary level of nesting.
Changed order by to use expression vs. positional notaion
Moved filtering criteria PEM.ENDORSEMENT_STATUS = 'POST' from correlated subquery to main query, which may correct wrong results issue.
Changed pagination condition to use a numeric expression rather than a character one, because:
select * from dual where '211' between '21' and '40';
select * from dual where 211 between 21 and 40;
Do not return the same results.
Explain plan should help you identify the problem. As Tony has stated, any merging of the inner query into the outer query will break your query. Any queries where the condition RONUM > 1 unconditionally applies will fail.
Queries such as you are building may require building the entire result set and then filtering out the rows for the page. You may want to consider building a key set for the desired rows in then inner query and then adding additional columns in the outer query. A CARDINALITY hint on the query selecting on rownum may help.
Try using "rownum() over (order by 1) rn" to generate the order. (I assume order is different than 1 at times.) Add a "/*+ FIRST_ROWS(20) */" to the first inner query. http://www.oracle.com/technology/oramag/oracle/07-jan/o17asktom.html for more help.
Related
I have a query, which returns a simple list of numbers:
SELECT unnest(c) FROM t ORDER BY f LIMIT 10;
And it goes like
1
1
3
4
2
3
5
1
5
6
3
2
I want to keep the result unique, but also preserve order:
1
3
2
4
5
6
select distinct(id) from (select ...) as c;
does not work, beacuse it uses HashAggregate, which breaks order (and processes all rows to return just 10?). I tried GROUP BY, it also uses HashAggregate the whole table(?) and then sort and return 10 required rows.
Is it possible to do it effectively on DB size? Or should I just read rows from my first query in my application and do the stream filtering?
with ordinality is your friend to preserve the order.
select val
from unnest('{1,1,3,4,2,3,5,1,5,6,3,2}'::int[]) with ordinality t(val, ord)
group by val
order by min(ord); -- the first time that this item appeared
val
1
3
4
2
5
6
Or it may make sense to define this function:
create function arr_unique(arr anyarray)
returns anyarray language sql immutable as
$$
select array_agg(val order by ord)
from
(
select val, min(ord) ord
from unnest(arr) with ordinality t(val, ord)
group by val
) t;
$$;
select elem
from (
select
elem, elem_no, row_no, row_number() over (partition by elem order by row_no) as occurence_no
from (
select elem, elem_no, row_number() over () as row_no from t, unnest(c) WITH ORDINALITY a(elem, elem_no)
) A
) B
where occurence_no = 1
order by row_no
I am using TSQL, SSMS v.17.9.1 The underlying db is Microsoft SQL Server 2014 SP3
For display purposes, I want to concatenate the results of two queries:
SELECT TOP 1 colA as 'myCol1' FROM tableA
--
SELECT TOP 1 colB as 'myCol2' FROM tableB
and display the results from the queries in one row in SSMS.
(The TOP 1 directive would hopefully guarantee the same number of results from each query, which would assist displaying them together. If this could be generalized to TOP 10 per query that would help also)
This should work for any number of rows, it assumes you want to pair ordered by the values in the column displayed
With
TableA_CTE AS
(
SELECT TOP 1 colA as myCol1
,Row_Number() OVER (ORDER BY ColA DESC) AS RowOrder
FROM tableA
),
TableB_CTE AS
(
SELECT TOP 1 colB as myCol2
,Row_Number() OVER (ORDER BY ColB DESC) AS RowOrder
FROM tableB
)
SELECT A.myCol1, B.MyCol2
FROM TableA_CTE AS A
INNER JOIN TableB_CTE AS B
ON A.RowOrder = B.RowOrder
There are currently two issues with the accepted answer:
I) a missing comma before the line: "Table B As"
II) TSQL seems to find it recursive as written, so I re-wrote it in a non-recursive way:
This is a re-working of the accepted answer that actually works in T-SQL:
USE [Database_1];
With
CTE_A AS
(
SELECT TOP 1 [Col1] as myCol1
,Row_Number() OVER (ORDER BY [Col2] desc) AS RowOrder
FROM [TableA]
)
,
CTE_B AS
(
SELECT TOP 1 [Col2] as myCol2
,Row_Number() OVER (ORDER BY [Col2] desc) AS RowOrder
FROM [TableB]
)
SELECT A.myCol1, B.myCol2
FROM CTE_A AS A
INNER JOIN CTE_B AS B
ON ( A.RowOrder = B.RowOrder)
So, in my queries, I've had the need to restructure some data.
To do this--I have created select statements that select some set into a table that before hand, I checked if it existed and then dropped the table if it did.
These tables are repopulated everytime I have to run an SSIS package and I feel like if I keep going down this path--things will get messy.
Is there a way to select out of a select statement in the middle of a select statement?
Select Field1
,Field2
,Field3
,SELECT DISTINCT R.Field4
FROM
(
SELECT DISTINCT
L.Open_DT
,L.Group_NPI
, bil.*
,Row_Number() over (partition by group_npi order by L.open_dt) AS RNK
FROM tbl_Location L
LEFT OUTER JOIN [MOAD].[dbo].[qry_Location_Address_Billing] bil
ON L.Location_ID = bil.Location_ID
) AS R
WHERE Rnk = 1 AND Location_ID IS NOT NULL AS Field4,
,Field5
,Field6
From Tables
I've edited this question in attempts to clarify.
I want it to come out like so:
Field1, Field2, Field3, Field4 (from that nested query), Field5, Field6
For an example, I would like to select id with max date group by category, the result is: 7, 2, 6
id category date
1 a 2013-01-01
2 b 2013-01-03
3 c 2013-01-02
4 a 2013-01-02
5 b 2013-01-02
6 c 2013-01-03
7 a 2013-01-03
8 b 2013-01-01
9 c 2013-01-01
This is the SQL I think can work:
SELECT * FROM Table1 t1
JOIN
(
SELECT category, MAX(date) AS MAXDATE
FROM Table1
GROUP BY category
) t2
ON T1.category = t2.category
AND t1.date = t2.MAXDATE
But how to translate that into a query on Ecto?
You can use subquery function
subquery = from t in "Table1"
|> select([t], %{categoty: t.category, max_date: max(t.date)})
|> group_by([t], t.category)
from t in "Table1"
|> join(:inner, [u], t in subquery(subquery), t.category == u.category and t.max_date == u.date)
|> Repo.all
An issue with many frameworks is that they cannot capture all the complexities of a SQL SELECT statement. The easiest solution: wrap your complex query in a view:
CREATE VIEW my_complex_view AS
SELECT * FROM Table1 t1
JOIN (
SELECT category, MAX(date) AS maxdate
FROM Table1
GROUP BY category) t2
ON t1.category = t2.category AND t1.date = t2.maxdate;
Now you have a simple query (SELECT * FROM my_complex_view) which any decent framework can easily handle.
tblUserProfile - I have a table which holds all the Profile Info (too many fields)
tblMonthlyProfiles - Another table which has just the ProfileID in it (the idea is that this table holds 2 profileids which sometimes become monthly profiles (on selection))
Now when I need to show monthly profiles, I simply do a select from this tblMonthlyProfiles and Join with tblUserProfile to get all valid info.
If there are no rows in tblMonthlyProfile, then monthly profile section is not displayed.
Now the requirement is to ALWAYS show Monthly Profiles. If there are no rows in monthlyProfiles, it should pick up 2 random profiles from tblUserProfile. If there is only one row in monthlyProfiles, it should pick up only one random row from tblUserProfile.
What is the best way to do all this in one single query ?
I thought something like this
select top 2 * from tblUserProfile P
LEFT OUTER JOIN tblMonthlyProfiles M
on M.profileid = P.profileid
ORder by NEWID()
But this always gives me 2 random rows from tblProfile. How can I solve this ?
Try something like this:
SELECT TOP 2 Field1, Field2, Field3, FinalOrder FROM
(
select top 2 Field1, Field2, Field3, FinalOrder, '1' As FinalOrder from tblUserProfile P JOIN tblMonthlyProfiles M on M.profileid = P.profileid
UNION
select top 2 Field1, Field2, Field3, FinalOrder, '2' AS FinalOrder from tblUserProfile P LEFT OUTER JOIN tblMonthlyProfiles M on M.profileid = P.profileid ORDER BY NEWID()
)
ORDER BY FinalOrder
The idea being to pick two monthly profiles (if that many exist) and then 2 random profiles (as you correctly did) and then UNION them. You'll have between 2 and 4 records at that point. Grab the top two. FinalOrder column is an easy way to make sure that you try and get the monthly's first.
If you have control of the table structure, you might save yourself some trouble by simply adding a boolean field IsMonthlyProfile to the UserProfile table. Then it's a single table query, order by IsBoolean, NewID()
In SQL 2000+ compliant syntax you could do something like:
Select ...
From (
Select TOP 2 ...
From tblUserProfile As UP
Where Not Exists( Select 1 From tblMonthlyProfile As MP1 )
Order By NewId()
) As RandomProfile
Union All
Select MP....
From tblUserProfile As UP
Join tblMonthlyProfile As MP
On MP.ProfileId = UP.ProfileId
Where ( Select Count(*) From tblMonthlyProfile As MP1 ) >= 1
Union All
Select ...
From (
Select TOP 1 ...
From tblUserProfile As UP
Where ( Select Count(*) From tblMonthlyProfile As MP1 ) = 1
Order By NewId()
) As RandomProfile
Using SQL 2005+ CTE you can do:
With
TwoRandomProfiles As
(
Select TOP 2 ..., ROW_NUMBER() OVER ( ORDER BY UP.ProfileID ) As Num
From tblUserProfile As UP
Order By NewId()
)
Select MP.Col1, ...
From tblUserProfile As UP
Join tblMonthlyProfile As MP
On MP.ProfileId = UP.ProfileId
Where ( Select Count(*) From tblMonthlyProfile As MP1 ) >= 1
Union All
Select ...
From TwoRandomProfiles
Where Not Exists( Select 1 From tblMonthlyProfile As MP1 )
Union All
Select ...
From TwoRandomProfiles
Where ( Select Count(*) From tblMonthlyProfile As MP1 ) = 1
And Num = 1
The CTE has the advantage of only querying for the random profiles once and the use of the ROW_NUMBER() column.
Obviously, in all the UNION statements the number and type of the columns must match.