I am recording the connection and disconnection time of a device in a table.
The final purpose is to define the maximum disconnection time.
My table looks like this
Before proceeding with the maximum DATEDIFF calculation I tried to attempt a DATEDIFF between each disconnection-connection rows (see the picture below):
The query is obviously wrong because it is making a difference between all of the "Disconnected" and "Connected" rows.
What am I missing?
Thank you
Try this, Note that negative value implies that the connection is still active
;WITH cte_Connectivity(DeviseID, EventTime, EventType) AS
(
SELECT 1,'2017-3-24 00:01:00.000', 'On' UNION ALL
SELECT 1,'2017-3-24 00:02:00.000', 'Off' UNION ALL
SELECT 2,'2017-3-24 00:01:00.000', 'On' UNION ALL
SELECT 2,'2017-3-24 00:04:00.000', 'Off' UNION ALL
SELECT 3,'2017-3-24 00:01:00.000', 'On' UNION ALL
SELECT 4,'2017-3-24 00:01:00.000', 'On' UNION ALL
SELECT 4,'2017-3-24 00:06:00.000', 'Off' UNION ALL
SELECT 5,'2017-3-24 00:01:00.000', 'On' UNION ALL
SELECT 5,'2017-3-24 00:10:00.000', 'Off'
)
SELECT a.DeviseID,
a.EventTime,
DATEDIFF(MINUTE, a.EventTime, ISNULL(b.EventTime, DATEADD(minute, - 1, a.EventTime))) AS TimeConnected
FROM cte_Connectivity a
LEFT JOIN cte_Connectivity b
ON CASE
WHEN a.EventType = 'On'
THEN 1
ELSE NULL
END = CASE
WHEN b.EventType = 'Off'
THEN 1
ELSE NULL
END
AND a.DeviseID = b.DeviseID
WHERE a.EventType = 'ON'
I then easily solved the problem:
created two views (one with only on, starting from the thrid row, and the other with only off rows)
joined both tables and added another column with a DATEDIFF
then create a new view that looks for the maximum datediff
... nothing special, thank you again folks!
Related
I have a table called user_activity in Redshift that has department, user_id, activity_type, activity_id, activity_date.
I'd like to query a daily report of how many days since the last event (of any type). Using CROSS APPLY (SQL Server) or LATERAL JOIN (Postgres 9+), I'd do something like...
SELECT d.date, a.last_activity_date
FROM date_table d
CROSS JOIN (
SELECT DISTINCT user_id FROM activity_table
) u
CROSS APPLY (
SELECT TOP 1 activity_date as last_activity_date
FROM activity_table
WHERE user_id = u.user_id AND activity_date <= d.date
ORDER BY activity_date DESC
) a
For now, I write it similar to the below, but it is a bit slow and I am afraid it'll only get slower.
with user_activity as (
select distinct activity_date, user_id from activity_table
)
select
d.date, u.user_id,
max(u.activity_date) as last_activity_date
from date_table d
inner join user_activity u on u.activity_date <= d.date
where d.date between '2020-01-01' and current_date
group by 1, 2
Can someone suggest a good alternative for my needs or for CROSS APPLY / LATERAL JOIN.
As you are seeing cross joining and inequality joining will slow down as you data grows and are generally not the approach you want in Redshift. This is due to the data size increase that comes with this type of action when applied to large data tables that are typical in Redshift.
You want to use window functions to perform this type of analysis. But you will need to step back and rethink how you will structure the SQL. A MAX(activity_date) window function, partitioned by user_id and ordered by date and with a frame clause of all preceding rows, will find the most recent activity to any activity.
Now this will produce only rows for user_ids and dates that exist in the data table and it looks like you want 1 row for each date for each user_id, right? To do this you need to UNION in a frame of data that has 1 row for each date for each user_id ahead of the window function. You will need NULLs in for the other columns so that the data widths match. You will also want the dates in a separate column from activity_date. Now all dates for all user ids will be in the source and the window function will give you the result you want.
You also ask ‘how is this better than the joins?’ Well in the joins you are replicating all the data records by the number of dates which can get really big. In this approach you just have the original data records plus one row per user_id per date (which is the size of your output) and as the number of records per user_id grows this approach doesn’t.
——— Request to modify asker’s code per comments made to their approach ———
Your code is definitely on the right track as you have removed the massive inequality join of your original. I made 2 comments about it. The first is that I believe you need GROUP BY user_id, date to prevent multiple rows per user_id per date that would result if there are records for the same user_id on a single date with differing activity_types. This is a simple oversight.
The second is to state that I intended for you to use UNION ALL, not LEFT JOIN, in combining the actual data and the user_id/date framework. Your approach works fine but I have found that unioning with very large amounts of data is generally faster than joining but you do need to make sure the columns match up. Either way we end up with a data segment with 3 columns - 2 date columns, one with NULLs for framework rows, and 1 user_id. Your approach is fine and the difference in performance is likely very small unless you have huge tables.
Since you asked for a rewrite, here it is with both changes. (NOTE: my laptop is in the shop so I don’t have ready access to Redshift at the moment and this SQL is untested. If the intent is not clear from this and you need me to debug it will be delayed by a few days. I’m keeping your setup methods and SQL structure.)
with date_table as (
select '2000-01-01'::date as date
union all
select '2000-01-02'::date
union all
select '2000-01-03'::date
union all
select '2000-01-04'::date
union all
select '2000-01-05'::date
union all
select '2000-01-06'::date
),
users as (
select 1 as user_id
union all
select 2
union all
select 3
),
user_activity as (
select 1 as user_id, '2000-01-01'::date as activity_date
union all
select 1 as user_id, '2000-01-04'::date as activity_date
union all
select 3 as user_id, '2000-01-03'::date as activity_date
union all
select 1 as user_id, '2000-01-05'::date as activity_date
union all
select 1 as user_id, '2000-01-06'::date as activity_date
),
user_dates as (
select d.date, u.user_id
from date_table d
cross join users u
),
user_date_activity as (
select cal_date, user_id,
lag(max(activity_date), 1) ignore nulls over (partition by user_id order by date) as last_activity_date
from (
Select user_id, date as cal_date, NULL as activity_date from user_dates
Union all
Select user_id, activity_date as cal_date, activity_date from user_activity
)
Group by user_id, cal_date
)
select * from user_date_activity
order by user_id, cal_date```
This was my query based on Bill's answer.
with date_table as (
select '2000-01-01'::date as date
union all
select '2000-01-02'::date
union all
select '2000-01-03'::date
union all
select '2000-01-04'::date
union all
select '2000-01-05'::date
union all
select '2000-01-06'::date
),
users as (
select 1 as user_id
union all
select 2
union all
select 3
),
user_activity as (
select 1 as user_id, '2000-01-01'::date as activity_date
union all
select 1 as user_id, '2000-01-04'::date as activity_date
union all
select 3 as user_id, '2000-01-03'::date as activity_date
union all
select 1 as user_id, '2000-01-05'::date as activity_date
union all
select 1 as user_id, '2000-01-06'::date as activity_date
),
user_dates as (
select d.date, u.user_id
from date_table d
cross join users u
),
user_date_activity as (
select ud.date, ud.user_id,
lag(ua.activity_date, 1) ignore nulls over (partition by ud.user_id order by ud.date) as last_activity_date
from user_dates ud
left join user_activity ua on ud.date = ua.activity_date and ud.user_id = ua.user_id
)
select * from user_date_activity
order by user_id, date
I am sure this could be very silly question, but I was not able to find answer from the internet.
I am trying to select rows with this conditon.
Select all rows if Side has both 'Right' and 'Left'.
In this case, it will select both row of Id 3 and 4 (yellow highlighted ones).
I know bottom query is not going to work.
select * from "Table" where ("Side" = 'Right') or ("Side" = 'Left')
How do I go about doing this?
You need a join. If your table name is processes
select
p.*
from processes p
inner join processes p2
on p.pid = p2.pid
and p.id <> p2.id
and p.side <> p2.side
You may filter it using a WHERE & HAVING clause in a GROUP BY
SELECT *
FROM t
WHERE pid IN (
SELECT pid
FROM t
WHERE side IN (
'Right',
'Left' )
GROUP BY pid
HAVING COUNT(DISTINCT side) = 2 )
Demo
I have a table MyTable that is populated by BizTalk from two outside agencies (AGC1 and AGC2) every day. I have recently run into a problem where AGC2 has been sending a blank file, so no records get updated.
I need to run a query like below, that would return
AGC1 01/28/2016
AGC2 NULL
But since I have no records for AGC2 I only get
AGC1 01/28/2016
How can I get AGC2 NULL as in the first resultset?
SELECT Agency, MAX(CONVERT(nvarchar(30), DATEENTERED, 101)), Case when MAX(CONVERT(nvarchar(30), DATEENTERED, 101)) = CONVERT(nvarchar(30), GETDATE(), 101) THEN 'True' ELSE 'False' end
FROM DBO.MYTABLE
GROUP BY AGENCY
Have another table containing:
NameTable
Name
-----
AGC1
AGC2
Then use this query:
Select n.Name, max(m.dateentered)
from MyTable m right join NameTable n
on m.Agency = n.Name
group by n.Name
A RIGHT JOIN with a table containing all agencies should do the trick. This version uses a derived table with VALUES:
SELECT
ag.Agency,
MAX(CONVERT(nvarchar(30), mt.DATEENTERED, 101)),
Case when MAX(CONVERT(nvarchar(30), mt.DATEENTERED, 101)) = CONVERT(nvarchar(30), GETDATE(), 101) THEN 'True' ELSE 'False' end
FROM
DBO.MYTABLE AS mt
RIGHT JOIN ( VALUES('AGC1'),('AGC2') ) AS ag(Agency) ON
ag.Agency=mt.Agency
GROUP BY
ag.AGENCY
Is there a way to use the lead function such that I can get the next row where something has changed, as opposed it where it is the same?
In this example, the RowType can be 'in' or 'out', for each 'in' I need to know the next RowNumber where it has become 'out'. I have been playing with the lead function as it is really fast, however I haven't been able to get it working. I just need to do the following really, which is partition by a RowType which isn't the one in the current row.
select
RowNumber
,RowType --In this case I am only interested in RowType = 'In'
, Lead(RowNumber)
OVER (partition by "RowType = out" --This is the bit I am stuck on--
order by RowNumber ASC) as NextOutFlow
from table
order by RowNumber asc
Thanks in advance for any help
Rather than using lead() I would use an outer apply that returns the next row with type out for all rows with type in:
select RowNumber, RowType, nextOut
from your_table t
outer apply (
select min(RowNumber) as nextOut
from your_table
where RowNumber > t.RowNumber and RowType='Out'
) oa
where RowType = 'In'
order by RowNumber asc
Given sample data like:
RowNumber RowType
1 in
2 out
3 in
4 in
5 out
6 in
This would return:
RowNumber RowType nextOut
1 in 2
3 in 5
4 in 5
6 in NULL
I think this will work
If you would use a bit field for in out you would get better performance
;with cte1 as
(
SELECT [inden], [OnOff]
, lag([OnOff]) over (order by [inden]) as [lagOnOff]
FROM [OnOff]
), cte2 as
(
select [inden], [OnOff], [lagOnOff]
, lead([inden]) over (order by [inden]) as [Leadinden]
from cte1
where [OnOff] <> [lagOnOff]
or [lagOnOff] is null
)
select [inden], [OnOff], [lagOnOff], [Leadinden]
from cte2
where [OnOff] = 'true'
probably slower but if you have the right indexes may work
select t1.rowNum as 'rowNumIn', min(t2.rownum) as 'nextRowNumOut'
from tabel t1
join table t2
on t1.rowType = 'In'
and t2.rowType = 'Out'
and t2.rowNum > t1.rowNum
and t2.rowNum < t1.rowNum + 1000 -- if you can constrain it
group by t1.rowNum
I have been banging my head trying to come up with the correct logic (SQL Server 2012) needed to achieve something I would imagine would be fairly routine but I have been unable to find any examples of this anywhere. Basically, I have 3 columns in a table: product, flag, value. It is possible for a product to be listed multiple times within the table but only once with a unique flag (i.e. product1 can have flag1 or flag2 with different/identical but there will never be 2 records with product1 and flag1 and different/identical values).
The flag represents a pre-defined value (1,2,3,4) and the intention behind this field is to be able to assign a unique mathematical equation based on the value of the flag. The end result would yield a single product, the unique flag, and a new cumulative total based on the mathematical equation output. For instance, let's say product1 was listed 4 times with flag values of flag1, flag2, flag3, flag4 (see below):
Product-----Flag-----Value
Product1----Flag1----1.00
Product1----Flag2----3.00
Product1----Flag3----5.00
Product1----Flag4----7.00
Product-----Flag-----Value
Product1----Flag1----1.00 (flag1 value)
Product1----Flag2----4.00 (flag1+flag2 value)
Product1----Flag3----6.00 (flag1+flag3 value)
Product1----Flag4----10.00 (flag2+flag4 value)
Flag1 is defined as add flag1 only. Flag2 is defined as add flag1 and flag2. Flag 3 is defined as add flag1 and flag 3. Flag 4 is defined as add flag2 and flag4. the new output would be product1 listed four times with flag values of flag1, flag2, flag3, flag4 but new values as flag1, flag1_flag2, flag1+flag3, flag2+flag4.
I have tried to apply the logic via a case statement but I can't figure out how to traverse all the products for each condition and I have tried to go with a running totals solution but I am not sure how to incorporate the flag condition into it so it only performs a running total for when those conditions are true. Any assistance and/or article to help get me going down the right path would be greatly appreciated.
While I'm not sure I fully understand your question I think this might be what you want. For this to work it assumes flag1 is always present when flags 1 through 3 are and that flag2 is present when flag4 is.
;with cte as (
select
product,
max(case when flag = 'Flag1' then Value end) as f1Value,
max(case when flag = 'Flag2' then Value end) as f2Value,
max(case when flag = 'Flag3' then Value end) as f3Value,
max(case when flag = 'Flag4' then Value end) as f4Value
from flags group by Product
)
select
flags.Product,
flags.Flag,
flags.Value as "Org. value",
case flag
when 'Flag1' then f1Value
when 'Flag2' then f1Value + f2Value
when 'Flag3' then f1Value + f3Value
when 'Flag4' then f2Value + f4Value
else flags.Value -- take the present value when flag is not Flag1-4
end as "New value"
from flags
inner join cte on flags.Product = cte.Product
Take a look at this Sample SQL Fiddle to see it in action.
You can join a table to itself, and pick the conditions appropriately:
SELECT p1.product,p1.Flag,p1.Value + COALESCE(p2.Value,0)
FROM
Products p1
left join
Products p2
on
p1.Product = p2.Product and
p2.Flag = CASE p1.Flag
--1 doesn't need a previous value
WHEN 2 THEN 1
WHEN 3 THEN 1
WHEN 4 THEN 2
END
I assumed and tried on Range values.
CREATE TABLE #tmp (Product VARCHAR(10), flag VARCHAR(10),value numeric(13,2))
GO
INSERT INTO #tmp
SELECT 'Product1' , 'Flag1',1
UNION
SELECT 'Product1' , 'Flag2',3
UNION
SELECT 'Product1' , 'Flag3',5
UNION
SELECT 'Product1' , 'Flag4',7
GO
;WITH cte
AS
(
SELECT row_number () OVER(
ORDER BY flag) 'row',*
FROM #tmp
)
SELECT *,value 'RT'
FROM cte
WHERE row = 1
UNION
SELECT * ,(
SELECT cte.value
FROM cte
WHERE row = 1
) + value 'RT'
FROM cte
WHERE row BETWEEN 2
AND 3
UNION
SELECT * ,(
SELECT cte.value
FROM cte
WHERE row =2
) + value 'RT'
FROM cte
WHERE row >3
GO
DROP TABLE #tmp