postgresql: merge rows keeping some information, without loops - postgresql

Question

I have a list of calls per every user sometimes separated by minutes. Users can buy something in these calls or not.
When a user makes a call within 45 minutes after the last call, I need to consider that it was the same call as the first one.
I need to get the final number of calls ( aggregating the calls separated by less than 45 minutes)
and the number of calls in which they bought something, per user.
So for example, I have a list like this:
buyer timestamp bougth_flag
tom 20150201 9:15 1
anna 20150201 9:25 0
tom 20150201 10:15 0
tom 20150201 10:45 1
tom 20150201 10:48 1
anna 20150201 11:50 0
tom 20150201 11:52 0
anna 20150201 11:54 0
The final table would be:
buyer time_started calls articles_bought
tom 20150201 9:15 1 1
anna 20150201 9:25 1 0
tom 20150201 10:15 3 2
anna 20150201 10:50 2 0
tom 20150201 11:52 1 0
So, I need to merge rows separated by less than 45 minutes, and separate still per user.
This is very easy to do with a loop but I don't have loops or functions/procedures in the postgresql I am using.
Any ideas about how to do it?
Thank you

Answer 1

Since you do not know beforehand how long a "call" is going to be (you could have a call from some buyer every 30 minutes for the full day - see comment to question), you can only solve this with a recursive CTE. (Note that I changed your column 'timestamp' to 'ts'. Never use a keyword as a table or column name.)
WITH conversations AS (
WITH RECURSIVE calls AS (
SELECT buyer, ts, bought_flag, row_number() OVER (ORDER BY ts) AS conversation, 1::int AS calls
FROM (
SELECT buyer, ts, lag(ts) OVER (PARTITION BY buyer ORDER BY ts) AS lag, bought_flag
FROM list) sub
WHERE lag IS NULL OR ts - lag > interval '45 minutes'
UNION ALL
SELECT l.buyer, l.ts, l.bought_flag, c.conversation, c.calls + 1
FROM list l
JOIN calls c ON c.buyer = l.buyer AND l.ts > c.ts
WHERE l.ts - c.ts < interval '45 minutes'
)
SELECT buyer, ts, bought_flag, conversation, max(calls) AS calls
FROM calls
GROUP BY buyer, ts, bought_flag, conversation
order by conversation, ts
)
SELECT buyer, min(ts) AS time_started, max(calls) AS calls, sum(bought_flag) AS articles_bought
FROM conversations
GROUP BY buyer, conversation
ORDER BY time_started
A few words of explanation:
The starting term of the inner recursive CTE has a sub-query that gets the basic data from the table for every call, together with the time of the previous call. The main query in the starting term of the inner CTE keeps only those rows where there is no previous call (lag IS NULL) or where the previous call is more than 45 minutes away. These are therefore the initial calls in what I term here a "conversation". The conversation gets a column and an id which is just the row number from the query, and another column to track the number of calls in the conversation "calls".
In the recursive term successive calls in the same conversation are added, with the "calls" counter incremented.
When calls are very close together (such as 10:45 and 10:48 after 10:15) then the later calls may be included multiple times, those duplicates (10:48) are dropped in the outer CTE by selecting the earliest call in the sequence for each conversation.
In the main query, finally, the 'bought_flag' column is summed for every conversation of every buyer.

Answer 2

The big problem is that you need to group your results per 45 minutes which makes it tricky. This query is a nice starting point but it's not completely correct. It should help you get going though:
SELECT a.buyer,
MIN(a.timestamp),
COUNT(a),
COUNT(b),
SUM(a.bougth_flag),
SUM(b.bougth_flag)
FROM calls a
LEFT JOIN calls b ON (a.buyer = b.buyer
AND a.timestamp != b.timestamp
AND a.timestamp < b.timestamp
AND a.timestamp + '45 minutes'::INTERVAL > b.timestamp)
GROUP BY a.buyer,
DATE_TRUNC('hour', a.timestamp) ;
Results:
┌───────┬─────────────────────┬───────┬───────┬─────┬─────┐
│ buyer │ min │ count │ count │ sum │ sum │
├───────┼─────────────────────┼───────┼───────┼─────┼─────┤
│ tom │ 2015-02-01 11:52:00 │ 1 │ 0 │ 0 │ Ø │
│ anna │ 2015-02-01 11:50:00 │ 2 │ 1 │ 0 │ 0 │
│ anna │ 2015-02-01 09:25:00 │ 1 │ 0 │ 0 │ Ø │
│ tom │ 2015-02-01 09:15:00 │ 1 │ 0 │ 1 │ Ø │
│ tom │ 2015-02-01 10:15:00 │ 4 │ 3 │ 2 │ 3 │
└───────┴─────────────────────┴───────┴───────┴─────┴─────┘

Answer 3

Thanks Patrick for notice about original version.
You are defently need WINDOW functions here, but CTE is optional here.
with start_points as(
select tmp.*,
--calculate distance between start points
(lead(ts) OVER w)-ts AS start_point_lead from( select t.*, ts - (lag(ts) OVER w) AS lag from test t window w as (PARTITION BY buyer ORDER BY ts)
) tmp where lag is null or lag>interval '45 minutes'
window w as (PARTITION BY buyer ORDER BY ts) order by ts
)
select s.buyer, s.ts, count(*), sum(t.bougth_flag) from start_points s join test t
on t.buyer=s.buyer and (t.ts-s.ts<s.start_point_lead or s.start_point_lead is null)and t.ts>=s.ts
group by s.buyer, s.ts order by s.ts