Finding groups of clustered data points - tsql

Question

I have a table which records events:
create table #events
( intRowId int identity(1,1),
intItemId int,
intUserId int,
datEvent datetime)
It's a big table with many millions of rows, recording events against several thousand items and tens of thousands of users.
There's a select group of ten itemIDs I want to look for, but only when they occur in a certain pattern: I'm trying to find rows where all ten of these items have events registered against them for the same userID and close together in time, say 5 minutes.
I have absolutely NO IDEA how to go about this. One would assume partitioning is involved somewhere, but help, even just somewhere to get started, would be much appreciated.
Cheers,
Matt

Answer 1

Lets say you want statistics about items id: 1, 2, ... , 10.
First create a table EventByItems:
CREATE TABLE EventByItems
(
intRowId int identity(1,1),
intUserId int,
datEvent datetime,
intItem1 int,
intItem2 int,
intItem3 int,
...
intItem10 int
)
Then use query to populate this table:
SELECT intUserId, datEvent,
SUM(pvt.[1]), SUM(pvt.[2]), SUM(pvt.[3]), ... , SUM(pvt.[10])
FROM #events
PIVOT
(
COUNT(intItemId)
FOR intItemId IN ([1], [2], [3], ... , [10])
) AS pvt
GROUP BY intUserId, datEvent
Now we can do some work with that table. For example we can update it to fill gaps according your logic. Or we can do queries like that:
SELECT
intRowId,
intUserId,
datEvent
FROM
EventByItems AS E
WHERE
((intItem1 > 0) OR EXISTS(SELECT *
FROM EventByItems
WHERE intUserId = E.intUserId
AND intItem1 > 0
AND DATEDIFF(MINUTE, datEvent, E.datEvent) <= 5
AND intRowId != E.intRowId ))
AND
...
AND
((intItem10 > 0) OR EXISTS(SELECT *
FROM EventByItems
WHERE intUserId = E.intUserId
AND intItem10 > 0
AND DATEDIFF(MINUTE, datEvent, E.datEvent) <= 5
AND intRowId != E.intRowId ))

Answer 2

Ok, so below you'll find a working example which does what you want. I'm assuming that events don't have to appear in tens.
But the solution is very crude, and will run slow, especially if you increase number of items/users.
Temp table with pre-selected events would help with performance in my solution, but what you really need are windowed functions like in Oracle..
DROP TABLE #events
GO
create table #events
( intRowId int identity(1,1),
intItemId int,
intUserId int,
datEvent datetime)
GO
insert into #events (intUserId,intItemId, datEvent)
select '1','1','2013-05-01 10:25' union all --group1
select '1','2','2013-05-01 10:25' union all --group1
select '1','3','2013-05-01 10:26' union all --group1
select '1','7','2013-05-01 10:25' union all
select '1','8','2013-05-01 10:25' union all
select '1','9','2013-05-01 10:26' union all
select '1','1','2013-05-01 10:50' union all --group2
select '1','2','2013-05-01 10:52' union all --group2
select '1','3','2013-05-01 10:59' union all
select '1','1','2013-05-01 11:10' union all --group3
select '1','1','2013-05-01 11:12' union all --group3
select '1','3','2013-05-01 11:17' union all --group3
select '1','2','2013-05-01 11:25' union all
select '1','1','2013-05-01 11:31' union all
select '1','7','2013-05-01 11:32' union all
select '1','2','2013-05-01 11:50' union all --group4
select '1','2','2013-05-01 11:50' union all --group4
select '1','3','2013-05-01 11:50' union all --group4
select '1','1','2013-05-01 11:56'
GO
DROP TABLE #temp
GO
select
e1.intRowId as intRowId_1, e1.intItemId as intItemId_1, e1.intUserId as intUserId_1, e1.datEvent as datEvent_1
,e2.intRowId as intRowId_2, e2.intItemId as intItemId_2, e2.intUserId as intUserId_2, e2.datEvent as datEvent_2
into #temp
from #events e1
join #events e2
on e1.intUserId=e2.intUserId
and e1.datEvent<=e2.datEvent
and e1.intRowId<>e2.intRowId
where 1=1
and e1.intUserId=1
and e2.intUserId=1
and e1.intItemId in (1,2,3)
and e2.intItemId in (1,2,3)
and datediff(minute,e1.datevent,e2.datevent)<6
order by
e1.intRowId, e2.intRowId
GO
select distinct
*
from (
select
intRowId_1 as intRowId, intItemId_1 as intItemId, intUserId_1 as intUserId, datEvent_1 as datEvent
from #temp
UNION ALL
select
intRowId_2 as intRowId, intItemId_2 as intItemId, intUserId_2 as intUserId, datEvent_2 as datEvent
from #temp
) x
order by
datEvent, intRowId