I have a table T as follows with 1 Billion records. Currently, this table has no Primary key or Indexes.
create table T(
day_c date,
str_c varchar2(20),
comm_c varchar2(20),
src_c varchar2(20)
);
some sample data:
insert into T
select to_date('20171011','yyyymmdd') day_c,'st1' str_c,'c1' comm_c,'s1' src_c from dual
union
select to_date('20171012','yyyymmdd'),'st1','c1','s1' from dual
union
select to_date('20171013','yyyymmdd'),'st1','c1','s1' from dual
union
select to_date('20171014','yyyymmdd'),'st1','c1','s2' from dual
union
select to_date('20171015','yyyymmdd'),'st1','c1','s2' from dual
union
select to_date('20171016','yyyymmdd'),'st1','c1','s2' from dual
union
select to_date('20171017','yyyymmdd'),'st1','c1','s1' from dual
union
select to_date('20171018','yyyymmdd'),'st1','c1','s1' from dual
union
select to_date('20171019','yyyymmdd'),'st1','c1','s1' from dual
union
select to_date('20171020','yyyymmdd'),'st1','c1','s1' from dual;
The expected result is to generate the date ranges for the changes in column src_c.
I have the following code snippet which provides the desired result. However, it is slow as the cost of running lag and lead is quite high on the table.
WITH EndsMarked AS (
SELECT
day_c,str_c,comm_c,src_c,
CASE WHEN src_c= LAG(src_c,1) OVER (ORDER BY day_c)
THEN 0 ELSE 1 END AS IS_START,
CASE WHEN src_c= LEAD(src_c,1) OVER (ORDER BY day_c)
THEN 0 ELSE 1 END AS IS_END
FROM T
), GroupsNumbered AS (
SELECT
day_c,str_c,comm_c,
src_c,
IS_START,
IS_END,
COUNT(CASE WHEN IS_START = 1 THEN 1 END)
OVER (ORDER BY day_c) AS GroupNum
FROM EndsMarked
WHERE IS_START=1 OR IS_END=1
)
SELECT
str_c,comm_c,src_c,
MIN(day_c) AS GROUP_START,
MAX(day_c) AS GROUP_END
FROM GroupsNumbered
GROUP BY str_c,comm_c, src_c,GroupNum
ORDER BY groupnum;
Output :
STR_C COMM_C SRC_C GROUP_START GROUP_END
st1 c1 s1 11-OCT-17 13-OCT-17
st1 c1 s2 14-OCT-17 16-OCT-17
st1 c1 s1 17-OCT-17 20-OCT-17
Any suggestion to speed up?
Oracle database :12c.
SGA Memory:20GB
Total CPU:22
Explain plan:
Order by day_c only, or do you need to partition by str_c and comm_c first? It seems so - in which case I am not sure your query is correct, and Sentinel's solution will need to be adjusted accordingly.
Then:
For some reason (which escapes me), it appears that the match_recognize clause (available only since Oracle 12.1) is faster than analytic functions, even when the work done seems to be the same.
In your problem, (1) you must read 1 billion rows from disk, which can't be done faster than the hardware allows (do you REALLY need to do this on all 1 billion rows, or should you archive a large portion of your table, perhaps after performing this identification of GROUP_START and GROUP_END)? (2) you must order the data by day_c no matter what method you use, and that is time consuming.
With that said, the tabibitosan method (see Sentinel's answer) will be faster than the start-of-group method (which is close to, but simpler than what you currently have).
The match_recognize solution, which will probably be faster than any solution based on analytic functions, looks like this:
select str_c, comm_c, src_c, group_start, group_end
from t
match_recognize(
partition by str_c, comm_c
order by day_c
measures x.src_c as src_c,
first(day_c) as group_start,
last(day_c) as group_end
pattern ( x y* )
define y as src_c = x.src_c
)
-- Add ORDER BY clause here, if needed
;
Here is a quick explanation of how this works; for developers who are not familiar with match_recognize, I provided links to a few good tutorials in a Comment below this Answer.
The match_recognize clause partitions the input rows by str_c and comm_c and orders them by day_c. So far this is exactly the same work that analytic functions do.
Then in the PATTERN and DEFINE clauses I declare and define two "classes" of rows, which will be flagged as X and Y, respectively. X is any row (there are no restrictions on it in the DEFINE clause). However, Y is restricted: it must have the same src_c as the last X row preceding it.
So, in each partition, and reading from the earliest row to the latest (within the partition), I am looking for any number of matches, where a match consists of an arbitrary row (marked X), followed by as many Y rows as possible; where Y means "same src_c as the first row in this match. So, this will identify sequences of rows where the src_c did not change.
For each match that is found, the clause will output the src_c value from the X row (which is the same, really, for all the rows in that match), and the first and the last value in the day_c column for that match. That is what we need to put in the SELECT clause of the overall query.
You can eliminate one CTE by using the Tabibito-san (Traveler) method:
with Groups as (
select t.*
, row_number() over (order by day_c)
- row_number() over (partition by str_c
, comm_c
, src_c
order by day_c) GroupNum
from t
)
select str_c
, comm_c
, src_c
, min(day_c) GROUP_START
, max(day_c) GROUP_END
from Groups
group by str_c
, comm_c
, src_c
, GroupNum
I want to create a pivot table view showing month on month sum of bookings for every travel_mode.
Table bookings:
timestamp
, bookings
, provider_id
Table providers:
provider_id
, travel_mode
Pivot table function and crosstab functions are not to be used to do this. So I am trying to use JOIN and CASE. Following is the query:
SELECT b.month,
(CASE WHEN p.travel_mode=train then b.amount end)train,
(CASE WHEN p.travel_mode=bus then b.amount end)bus,
(CASE WHEN p.travel_mode=air then b.amount end)air
FROM
(SELECT to_char(date_,month) as month, travel_mode, sum(bookings) as amount
from bookings as b
join providers as p
on b.provider_id=p.provider_id
group by b.month, p.travel_mode)
group by b.month;
However I am getting an error which says:
subquery in FROM must have an alias LINE 6:
And when I add an alias it throws an error saying:
column p.travel_mode must appear in the GROUP BY clause or be used in an aggregate function
LINE 2:
The final result should be something like this
Month Air Bus Train
01 Amount(air) Amount(Bus) Amount(train)
I have a feeling it is a minor error somewhere but I am unable to figure it out at all.
P.S. I had to remove all quotations in the question as it was not allowing me to post this. But those are being taken care of in the actual query.
Multiple problems. The missing table alias is just one of them. This query should work:
SELECT month
, sum(CASE WHEN travel_mode = 'train' THEN amount END) AS train
, sum(CASE WHEN travel_mode = 'bus' THEN amount END) AS bus
, sum(CASE WHEN travel_mode = 'air' THEN amount END) AS air
FROM (
SELECT to_char(timestamp, 'MM') AS month, travel_mode, sum(bookings) AS amount
FROM bookings b
JOIN providers p USING (provider_id)
GROUP BY month, p.travel_mode
) sub
GROUP BY month;
Missing single quotes for string literals. (You seem to have removed those being under the wrong impression you couldn't post quotations.)
Missing table alias for the subquery - just like the 1st error message says.
In the outer query, table names (or aliases) of underlying tables in the subquery are not visible. Only the table alias of the subquery is. Since there is only one subquery, you don't need table-qualification at all there.
month is an output column name (not in the underlying table), so the table qualification b.month was wrong, too.
You seem to want 2-digit numbers for months. Use the template pattern 'MM' instead of 'month' with to_char().
The aggregation in the outer query does not work like you had it - just like your 2nd error message says. You have to wrap the outer CASE expression in a aggregate function. You might as well use min() or max() in this case, because there are never more than one rows after the subquery.
Still unclear where date_ is coming from? You mean timestamp? (which is not a good identifier).
But you don't need the subquery to begin with and can simplify to:
SELECT to_char(timestamp, 'MM') AS month
, sum(CASE WHEN p.travel_mode = 'train' THEN b.bookings END) AS train
, sum(CASE WHEN p.travel_mode = 'bus' THEN b.bookings END) AS bus
, sum(CASE WHEN p.travel_mode = 'air' THEN b.bookings END) AS air
FROM bookings b
JOIN providers p USING (provider_id)
GROUP BY 1;
For best performance you should still use crosstab(), though:
PostgreSQL Crosstab Query
You have to name the subquery as the error message says:
SELECT b.month,
(CASE WHEN p.travel_mode=train then b.amount end)train,
(CASE WHEN p.travel_mode=bus then b.amount end)bus,
(CASE WHEN p.travel_mode=air then b.amount end)air
FROM
(SELECT to_char(date_,month) as month, travel_mode, sum(bookings) as amount
from bookings as b
join providers as p
on b.provider_id=p.provider_id
group by b.month, p.travel_mode)
**as foo** group by b.month;
Remove the stars to make it work.
I would like to do something like this:
CNT=2;
//[edit]
select avg(price) from (
select first :CNT p.Price
from Price p
order by p.Date desc
);
This does not work, Firebird does not allow :cnt as a parameter to FIRST. I need to average the first CNT newest prices. The number 2 changes so it can not be hard-coded.
This can be broken out into a FOR SELECT loop and break when a count is reached. Is that the best way though? Can this be done in a single SQL statement?
Creating the SQL as a string and running it is not the best fit either. It is important that the database compile my SQL statement.
You don't have to use CTE, you can do it directly:
select avg(price) from (
select first :cnt p.Price
from Price p
order by p.Date desc
);
You can use a CTE (Common Table Expression) (see http://www.firebirdsql.org/refdocs/langrefupd21-select.html#langrefupd21-select-cte) to select data before calculate average.
See example below:
with query1 as (
select first 2 p.Price
from Price p
order by p.Date desc
)
select avg(price) from query1
In a table I have records with id's 2,4,5,8. How can I receive a list with values 1,3,6,7. I have tried in this way
SELECT t1.id + 1
FROM table t1
WHERE NOT EXISTS (
SELECT *
FROM table t2
WHERE t2.id = t1.id + 1
)
but it's not working correctly. It doesn't bring all available positions.
Is it possible without another table?
You can get all the missing ID's from a recursive CTE, like this:
with recursive numbers as (
select 1 number
from rdb$database
union all
select number+1
from rdb$database
join numbers on numbers.number < 1024
)
select n.number
from numbers n
where not exists (select 1
from table t
where t.id = n.number)
the number < 1024 condition in my example limit the query to the max 1024 recursion depth. After that, the query will end with an error. If you need more than 1024 consecutive ID's you have either run the query multiple times adjusting the interval of numbers generated or think in a different query that produces consecutive numbers without reaching that level of recursion, which is not too difficult to write.
Currently I've a query as follows:
-- Query 1
SELECT
acc_code, acc_name, alias, LAmt, coalesce(LAmt,0) AS amt
FROM
(SELECT
acc_code, acc_name, alias,
(SELECT
(SUM(cr_amt)-SUM(dr_amt))
FROM
ledger_mcg l
WHERE
(l.acc_code LIKE a.acc_code + '.%' OR l.acc_code=a.acc_code)
AND
fy_id=1
AND
posted_date BETWEEN '2010-01-01' AND '2011-06-02') AS LAmt
FROM
acc_head_mcg AS a
WHERE
(acc_type='4')) AS T1
WHERE
coalesce(LAmt,0)<>0
Query 2 is same as Query 1 except that acc_type = '5' in Query 2. Query 2 always returns a resultset with a single row. Now, I need the union of the two queries i.e
Query 1
UNION
Query 2
only when the amt returned by Query 2 is less than 0. Else, I don't need a union but only the resulset from Query 1.
The best way I can think of is to create a parameterised scalar function. How best can I do this?
You could store the result of the first query into a temporary table, then, if the table wasn't empty, execute the other query.
IF OBJECT_ID('tempdb..#MultipleQueriesResults') IS NOT NULL
DROP TABLE #MultipleQueriesResults;
SELECT
acc_code, acc_name, alias, LAmt, coalesce(LAmt,0) AS amt
INTO #MultipleQueriesResults
FROM
(SELECT
acc_code, acc_name, alias,
(SELECT
(SUM(cr_amt)-SUM(dr_amt))
FROM
ledger_mcg l
WHERE
(l.acc_code LIKE a.acc_code + '.%' OR l.acc_code=a.acc_code)
AND
fy_id=1
AND
posted_date BETWEEN '2010-01-01' AND '2011-06-02') AS LAmt
FROM
acc_head_mcg AS a
WHERE
(acc_type='4')) AS T1
WHERE
coalesce(LAmt,0)<>0;
IF NOT EXISTS (SELECT * FROM #MultipleQueriesResults)
… /* run Query 2 */