I am using the T-SQL NTILE function like this:
CREATE TABLE #Temp(val int)
GO
INSERT INTO #Temp(val) VALUES(1);
INSERT INTO #Temp(val) VALUES(2);
INSERT INTO #Temp(val) VALUES(3);
SELECT val, Quantile
FROM
(
SELECT val, NTILE(10) OVER (ORDER BY val) AS Quantile
FROM
#Temp
) A
GO
DROP TABLE #Temp
Unfortunately the resut:
val Quantile
1 1
2 2
3 3
is not what I am expecting.
I expect that the row is filled from right to left instead from left to right, when I have less that 10 records. This would make sense, because normally you like to distribute the data this way
This problem is mentioned in the community addon here:
http://msdn.microsoft.com/en-us/library/ms175126.aspx
Do you have any idea how to get rid of this?
If I understand you correctly, you just want NTILE's behaviour, but filling from high to low instead. You can achieve this by subtracting (n+1)-NTILE(n) and order by descending, in your case;
SELECT val, Quantile FROM
(
SELECT val, 11-NTILE(10) OVER (ORDER BY val DESC) AS Quantile
FROM #Temp
) A
> 1 - 8
> 2 - 9
> 3 - 10
An SQLfiddle to test with.
Related
While trying to map some data to a table, I wanted to obtain the ID of a table and its modulo respect the total rows in the same table. For example, given this table:
id
--
1
3
10
12
I would like this result:
id | mod
---+----
1 | 1 <- 1 mod 4
3 | 3 <- 3 mod 4
10 | 2 <- 10 mod 4
12 | 0 <- 12 mod 4
Is there an easy way to achieve this dynamically (as in, not counting the rows on before hand or doing it in an atomic way)?
So far I've tried something like this:
SELECT t1.id, t1.id % COUNT(t1.id) mod FROM tbl t1, tbl t2 GROUP BY t1.id;
This works but you must have the GROUP BY and tbl t2 as otherwise it returns 0 for the mod column which makes sense because I think it works by multiplying the table by itself so each ID gets a full set of the table. I guess for small enough tables this is ok but I can see how this becomes problematic for larger tables.
Edit: Found another hack-ish way:
WITH total AS (
SELECT COUNT(*) cnt FROM tbl
)
SELECT t1.id, t1.id % t2.cnt mod FROM tbl t1, total t2
It similar to the previous query but it "collapses" the multiplication to a single row with the previous count.
You can use COUNT() window function:
SELECT id,
id % COUNT(*) OVER () mod
FROM tbl;
I'm sure that the optimizer is smart enough to calculate the result of the window function only once.
See the demo.
I have paste the details as below,
The detailed requirements are here:
Sno Column A Column B Column C Formula
1 0 -1 Yrs 0.963190184 100000
2 1-4 Yrs 0.992394232 96319 (B1 * C1 (default value 100000)) =C2
3 5 - 9 Yrs 0.994964922 95586 (B2 * C2 ) =C3
4 10 - 14 Yrs 0.998372661 95105 (B3 * C3 ) =C4
5 15 - 19 Yrs 0.994485603 94950
6 20- 24 Yrs 0.992903887 94427
7 25 - 29 Yrs 0.994008987 93757
8 30 - 34 Yrs 0.994041445 93195
9 35 - 39 Yrs 0.991283828 92640
10 40 - 44 Yrs 0.987141228 91832
The above same formula needs to be apply for the remaining columns through a PostgreSQL query.
Since your basic requirement is looking back to the prior row the lag function first comes to mind. However this dos not work as the necessary value is being calculated of the fly and lag operates on the result set (prior to lag operation). So we turn to a recursive query where at each recursion we have available the just calculated on the prior recursion. This of course assumes the other columns are already loaded. We get the desired values with:
with recursive demo( rno, r_colb, r_colc) as
( select 0, 1::numeric, 100000::numeric -- prime the recursion
union all
select sno, column_b, r_colb*r_colc
from test
join demo
on (sno = rno+1) -- next sno
)
select rno, r_colb, round(r_colc) r_colc
from demo ;
That does not get to the final need - updating the table. Fortunately, Postgres supports "update ... from (subquery) where ..." structure and the subquery just basically needs to be a valid query; including the above. So we arrive at:
update test
set column_c = r_colc
from (
with recursive demo( rno, r_colb, r_colc) as
( select 0, 1::numeric, 100000::numeric -- prime the recursion
union all
select sno, column_b, r_colb*r_colc
from test
join demo
on (sno = (select min(sno) -- since I do trust autogenerated ids
from test -- to actually be in perfect sequence
where sno > rno))
)
select rno, r_colb, round(r_colc) r_colc
from demo
) s
where sno = rno;
See demo run in v9.6.
In response to "applied for continuous calculation", yes that is possible and actually only changing the WITH clause to prime the recursion. But first, when speaking on SQL remove IF from your vocabulary - there is no such thing. There are conditionals (where, case, when...) but no IF.
The non-recursive query just gets the last known value then the recursive part looks at each subsequent and calculates column_c from it. To restart the calculation, just do the same. I modified the update query to do just that. See revised demo. It would have been possible to just modify the non-recursive query to select sno 10 (the last known value for column_c). But doing so would require updating the query for subsequent execution. Instead it looks for the last know (not null column_c) selecting that as the primer. This does require all new rows leave column_c null. But it should be good for any number of additional rows.
Note: The term "prime/primer" is my terminology for this query. It is not a general technical term used with a recursive CTE.
update test
set column_c = r_colc
from (
with recursive demo( rno, r_colb, r_colc) as
( select sno, column_b, column_c -- prime the recursion
from test
where sno = (select max(sno)
from test
where column_c is not null
)
union all
select sno, column_b, r_colb*r_colc
from test
join demo
on (sno = (select min(sno) -- since I do trust autogenerated ids
from test -- to actually be in perfect sequence
where sno > rno))
)
select rno, r_colb, round(r_colc) r_colc
from demo
) s
where sno = rno;
I have a table T as follows with 1 Billion records. Currently, this table has no Primary key or Indexes.
create table T(
day_c date,
str_c varchar2(20),
comm_c varchar2(20),
src_c varchar2(20)
);
some sample data:
insert into T
select to_date('20171011','yyyymmdd') day_c,'st1' str_c,'c1' comm_c,'s1' src_c from dual
union
select to_date('20171012','yyyymmdd'),'st1','c1','s1' from dual
union
select to_date('20171013','yyyymmdd'),'st1','c1','s1' from dual
union
select to_date('20171014','yyyymmdd'),'st1','c1','s2' from dual
union
select to_date('20171015','yyyymmdd'),'st1','c1','s2' from dual
union
select to_date('20171016','yyyymmdd'),'st1','c1','s2' from dual
union
select to_date('20171017','yyyymmdd'),'st1','c1','s1' from dual
union
select to_date('20171018','yyyymmdd'),'st1','c1','s1' from dual
union
select to_date('20171019','yyyymmdd'),'st1','c1','s1' from dual
union
select to_date('20171020','yyyymmdd'),'st1','c1','s1' from dual;
The expected result is to generate the date ranges for the changes in column src_c.
I have the following code snippet which provides the desired result. However, it is slow as the cost of running lag and lead is quite high on the table.
WITH EndsMarked AS (
SELECT
day_c,str_c,comm_c,src_c,
CASE WHEN src_c= LAG(src_c,1) OVER (ORDER BY day_c)
THEN 0 ELSE 1 END AS IS_START,
CASE WHEN src_c= LEAD(src_c,1) OVER (ORDER BY day_c)
THEN 0 ELSE 1 END AS IS_END
FROM T
), GroupsNumbered AS (
SELECT
day_c,str_c,comm_c,
src_c,
IS_START,
IS_END,
COUNT(CASE WHEN IS_START = 1 THEN 1 END)
OVER (ORDER BY day_c) AS GroupNum
FROM EndsMarked
WHERE IS_START=1 OR IS_END=1
)
SELECT
str_c,comm_c,src_c,
MIN(day_c) AS GROUP_START,
MAX(day_c) AS GROUP_END
FROM GroupsNumbered
GROUP BY str_c,comm_c, src_c,GroupNum
ORDER BY groupnum;
Output :
STR_C COMM_C SRC_C GROUP_START GROUP_END
st1 c1 s1 11-OCT-17 13-OCT-17
st1 c1 s2 14-OCT-17 16-OCT-17
st1 c1 s1 17-OCT-17 20-OCT-17
Any suggestion to speed up?
Oracle database :12c.
SGA Memory:20GB
Total CPU:22
Explain plan:
Order by day_c only, or do you need to partition by str_c and comm_c first? It seems so - in which case I am not sure your query is correct, and Sentinel's solution will need to be adjusted accordingly.
Then:
For some reason (which escapes me), it appears that the match_recognize clause (available only since Oracle 12.1) is faster than analytic functions, even when the work done seems to be the same.
In your problem, (1) you must read 1 billion rows from disk, which can't be done faster than the hardware allows (do you REALLY need to do this on all 1 billion rows, or should you archive a large portion of your table, perhaps after performing this identification of GROUP_START and GROUP_END)? (2) you must order the data by day_c no matter what method you use, and that is time consuming.
With that said, the tabibitosan method (see Sentinel's answer) will be faster than the start-of-group method (which is close to, but simpler than what you currently have).
The match_recognize solution, which will probably be faster than any solution based on analytic functions, looks like this:
select str_c, comm_c, src_c, group_start, group_end
from t
match_recognize(
partition by str_c, comm_c
order by day_c
measures x.src_c as src_c,
first(day_c) as group_start,
last(day_c) as group_end
pattern ( x y* )
define y as src_c = x.src_c
)
-- Add ORDER BY clause here, if needed
;
Here is a quick explanation of how this works; for developers who are not familiar with match_recognize, I provided links to a few good tutorials in a Comment below this Answer.
The match_recognize clause partitions the input rows by str_c and comm_c and orders them by day_c. So far this is exactly the same work that analytic functions do.
Then in the PATTERN and DEFINE clauses I declare and define two "classes" of rows, which will be flagged as X and Y, respectively. X is any row (there are no restrictions on it in the DEFINE clause). However, Y is restricted: it must have the same src_c as the last X row preceding it.
So, in each partition, and reading from the earliest row to the latest (within the partition), I am looking for any number of matches, where a match consists of an arbitrary row (marked X), followed by as many Y rows as possible; where Y means "same src_c as the first row in this match. So, this will identify sequences of rows where the src_c did not change.
For each match that is found, the clause will output the src_c value from the X row (which is the same, really, for all the rows in that match), and the first and the last value in the day_c column for that match. That is what we need to put in the SELECT clause of the overall query.
You can eliminate one CTE by using the Tabibito-san (Traveler) method:
with Groups as (
select t.*
, row_number() over (order by day_c)
- row_number() over (partition by str_c
, comm_c
, src_c
order by day_c) GroupNum
from t
)
select str_c
, comm_c
, src_c
, min(day_c) GROUP_START
, max(day_c) GROUP_END
from Groups
group by str_c
, comm_c
, src_c
, GroupNum
Database is HP Vertica 7 or PostgreSQL 9.
create table test (
id int,
card_id int,
tran_dt date,
amount int
);
insert into test values (1, 1, '2017-07-06', 10);
insert into test values (2, 1, '2017-06-01', 20);
insert into test values (3, 1, '2017-05-01', 30);
insert into test values (4, 1, '2017-04-01', 40);
insert into test values (5, 2, '2017-07-04', 10);
Of the payment cards used in the last 1 day, what is the maximum amount charged on that card in the last 90 days.
select t.card_id, max(t2.amount) max
from test t
join test t2 on t2.card_id=t.card_id and t2.tran_dt>='2017-04-06'
where t.tran_dt>='2017-07-06'
group by t.card_id
order by t.card_id;
Results are correct
card_id max
------- ---
1 30
I want to rewrite the query into sql window functions.
select card_id, max(amount) over(partition by card_id order by tran_dt range between '60 days' preceding and current row) max
from test
where card_id in (select card_id from test where tran_dt>='2017-07-06')
order by card_id;
But result set does not match, how can this be done?
Test data here:
http://sqlfiddle.com/#!17/db317/1
I can't try PostgreSQL, but in Vertica, you can apply the ANSI standard OLAP window function.
But you'll need to nest two queries: The window function only returns sensible results if it has all rows that need to be evaluated in the result set.
But you only want the row from '2017-07-06' to be displayed.
So you'll have to filter for that date in an outer query:
WITH olap_output AS (
SELECT
card_id
, tran_dt
, MAX(amount) OVER (
PARTITION BY card_id
ORDER BY tran_dt
RANGE BETWEEN '90 DAYS' PRECEDING AND CURRENT ROW
) AS the_max
FROM test
)
SELECT
card_id
, the_max
FROM olap_output
WHERE tran_dt='2017-07-06'
;
card_id|the_max
1| 30
As far as I know, PostgreSQL Window function doesn't support bounded range preceding thus range between '90 days' preceding won't work. It does support bounded rows preceding such as rows between 90 preceding, but then you would need to assemble a time-series query similar to the following for the Window function to operate on the time-based rows:
SELECT c.card_id, t.amount, g.d as d_series
FROM generate_series(
'2017-04-06'::timestamp, '2017-07-06'::timestamp, '1 day'::interval
) g(d)
CROSS JOIN ( SELECT distinct card_id from test ) c
LEFT JOIN test t ON t.card_id = c.card_id and t.tran_dt = g.d
ORDER BY c.card_id, d_series
For what you need (based on your question description), I would stick to using group by.
I have been banging my head trying to come up with the correct logic (SQL Server 2012) needed to achieve something I would imagine would be fairly routine but I have been unable to find any examples of this anywhere. Basically, I have 3 columns in a table: product, flag, value. It is possible for a product to be listed multiple times within the table but only once with a unique flag (i.e. product1 can have flag1 or flag2 with different/identical but there will never be 2 records with product1 and flag1 and different/identical values).
The flag represents a pre-defined value (1,2,3,4) and the intention behind this field is to be able to assign a unique mathematical equation based on the value of the flag. The end result would yield a single product, the unique flag, and a new cumulative total based on the mathematical equation output. For instance, let's say product1 was listed 4 times with flag values of flag1, flag2, flag3, flag4 (see below):
Product-----Flag-----Value
Product1----Flag1----1.00
Product1----Flag2----3.00
Product1----Flag3----5.00
Product1----Flag4----7.00
Product-----Flag-----Value
Product1----Flag1----1.00 (flag1 value)
Product1----Flag2----4.00 (flag1+flag2 value)
Product1----Flag3----6.00 (flag1+flag3 value)
Product1----Flag4----10.00 (flag2+flag4 value)
Flag1 is defined as add flag1 only. Flag2 is defined as add flag1 and flag2. Flag 3 is defined as add flag1 and flag 3. Flag 4 is defined as add flag2 and flag4. the new output would be product1 listed four times with flag values of flag1, flag2, flag3, flag4 but new values as flag1, flag1_flag2, flag1+flag3, flag2+flag4.
I have tried to apply the logic via a case statement but I can't figure out how to traverse all the products for each condition and I have tried to go with a running totals solution but I am not sure how to incorporate the flag condition into it so it only performs a running total for when those conditions are true. Any assistance and/or article to help get me going down the right path would be greatly appreciated.
While I'm not sure I fully understand your question I think this might be what you want. For this to work it assumes flag1 is always present when flags 1 through 3 are and that flag2 is present when flag4 is.
;with cte as (
select
product,
max(case when flag = 'Flag1' then Value end) as f1Value,
max(case when flag = 'Flag2' then Value end) as f2Value,
max(case when flag = 'Flag3' then Value end) as f3Value,
max(case when flag = 'Flag4' then Value end) as f4Value
from flags group by Product
)
select
flags.Product,
flags.Flag,
flags.Value as "Org. value",
case flag
when 'Flag1' then f1Value
when 'Flag2' then f1Value + f2Value
when 'Flag3' then f1Value + f3Value
when 'Flag4' then f2Value + f4Value
else flags.Value -- take the present value when flag is not Flag1-4
end as "New value"
from flags
inner join cte on flags.Product = cte.Product
Take a look at this Sample SQL Fiddle to see it in action.
You can join a table to itself, and pick the conditions appropriately:
SELECT p1.product,p1.Flag,p1.Value + COALESCE(p2.Value,0)
FROM
Products p1
left join
Products p2
on
p1.Product = p2.Product and
p2.Flag = CASE p1.Flag
--1 doesn't need a previous value
WHEN 2 THEN 1
WHEN 3 THEN 1
WHEN 4 THEN 2
END
I assumed and tried on Range values.
CREATE TABLE #tmp (Product VARCHAR(10), flag VARCHAR(10),value numeric(13,2))
GO
INSERT INTO #tmp
SELECT 'Product1' , 'Flag1',1
UNION
SELECT 'Product1' , 'Flag2',3
UNION
SELECT 'Product1' , 'Flag3',5
UNION
SELECT 'Product1' , 'Flag4',7
GO
;WITH cte
AS
(
SELECT row_number () OVER(
ORDER BY flag) 'row',*
FROM #tmp
)
SELECT *,value 'RT'
FROM cte
WHERE row = 1
UNION
SELECT * ,(
SELECT cte.value
FROM cte
WHERE row = 1
) + value 'RT'
FROM cte
WHERE row BETWEEN 2
AND 3
UNION
SELECT * ,(
SELECT cte.value
FROM cte
WHERE row =2
) + value 'RT'
FROM cte
WHERE row >3
GO
DROP TABLE #tmp