MongoDB count concurrent - mongodb

Question

I have a collection that has a start_time and end_time denoting a session
I need to count max concurrent sessions in a given hour.
something like aggregate and group by the hour.
what's the most efficient way to do this?

Answer 1

Your Query to do so will be something like:
db.collection_name.aggregate(
[ { $group : { _id : $hour, no_of_sessions : { $sum:1 } } } ]
)
Here: $hour is your time variable (assuming you are just storing the hour, if not you can apply (hour: { $hour: "$date" }) function to get it from date).
If hours are like 1:01 to 2:59 then you will need to define _id as compound key. something like: _id: {start_time: $start_time , end_time : $end_time}.
To get a more specific answer, please give the exact case.
Cheers!

Answer 2

The problems with this type of aggregation come in that a "session" with a "start_time" and an "end_time" can actually therefore "emit" hours that cross each grouped hour, so it is present in more that one hourly time period until the session ends. This can potentially span hours
The other main problem here is that the session may indeeed "start" before the time period you want to look at, or even end "after" the specified range, such as a day. Here you need to consider that you are generally looking for a "start_time" that is less than the end of the day you are looking at, and that the "end_time" is greater than the start of the day you are looking at.
Even so there are other considerations, such as does something have an "end_time" at all at the time it is analysed? Usually the best way to deal with this is consider a reasonable "session life" value, and factor that in to the basic query selection.
So with a few variables at play, we basically come of with the "base criteria" for selection:
var startDay = new Date("2015-08-30"),
endDay = new Date("2015-08-31"),
oneHour = 1000*60*60,
sessionTime = 3*oneHour;
var query = {
"start_time": {
"$gte": new Date(startDay.valueOf()-sessionTime),
"$lt": endDay
},
"$or": [
{ "end_time": { "$exists": false } },
{ "end_time": null },
{ "end_time": {
"$lt": new Date(endDay.valueOf()+sessionTime),
"$gte": startDay
}}
]
};
Working with a 3 hour window here for example to also include found dates outside of the current day to include in "possible" output.
Next consider some data to work with as a sample:
{ "_id": 1, "start_time": new Date("2015-08-29T23:30"), "end_time": new Date("2015-08-29T23:45") },
{ "_id": 2, "start_time": new Date("2015-08-29T23:30"), "end_time": new Date("2015-08-30T00:45") },
{ "_id": 3, "start_time": new Date("2015-08-30T00:30"), "end_time": new Date("2015-08-30T01:30") },
{ "_id": 4, "start_time": new Date("2015-08-30T01:30"), "end_time": new Date("2015-08-30T01:45") },
{ "_id": 5, "start_time": new Date("2015-08-30T01:30"), "end_time": new Date("2015-08-30T03:45") },
{ "_id": 6, "start_time": new Date("2015-08-30T01:45"), "end_time": new Date("2015-08-30T02:30") },
{ "_id": 7, "start_time": new Date("2015-08-30T23:30"), "end_time": null },
{ "_id": 8, "start_time": new Date("2015-08-30T23:30") },
{ "_id": 9, "start_time": new Date("2015-08-31T01:30") }
If we look at the criteria for the date range and the general query selection, then you can expect that records 2 through 8 would be considered in the day we are looking at as they either "ended" within the day or "started" within the day. The "session window" is mainly because some data does not have an "end_time", being either null or not present. That "window" helps filter out other irrelevant data that may be from more recent dates than what is being looked at, and keeps the size reasonable.
A quick visual scan should tell you that the counts per hour should be this:
0: 2
1: 4,
2: 2,
3: 1
23: 2
The actual process is better handled with mapReduce than any other aggregation medium. This is because the conditional logic required allows a "single document" to be "emitted" as a value valid for multiple periods. So there is an inherrent "looping" required here
db.sessions.mapReduce(
function() {
var oneHour = 1000*60*60,
start = (this.start_time > startDay)
? ( this.start_time.valueOf() - ( this.start_time.valueOf() % oneHour ))
: startDay,
end = (this.hasOwnProperty("end_time") && this.end_time != null)
? ( this.end_time.valueOf() - ( this.end_time.valueOf() % oneHour ))
: endDay;
// Uncomment to Emit blank values for each hour on first iteration
/*
if ( count == 0 ) {
for ( var x = 1; x <= 24; x++ ) {
emit(x,0);
}
count++;
}
*/
for ( var y = start; y <= end && (y-startDay)/oneHour < 24; y+= oneHour) {
emit(
(y-startDay ==0) ? 0 : ((y-startDay)/oneHour)
,1
);
}
},
function(key,values) {
return Array.sum(values);
},
{
"out": { "inline": 1 },
"scope": {
"startDay": startDay.valueOf(),
"endDay": endDay.valueOf(),
"count": 0
},
"query": query
}
)
Combined with the variable set earlier, this will correcty count how many sessions are currently running within each hour:
"results" : [
{
"_id" : 0,
"value" : 2
},
{
"_id" : 1,
"value" : 4
},
{
"_id" : 2,
"value" : 2
},
{
"_id" : 3,
"value" : 1
},
{
"_id" : 23,
"value" : 2
}
],
The basic actions for each record are:
Round out the start and end time each to 1 hour
Replace each value with either the startDay for the day being looked at or the endDay where the start was before the current day or the end_time is not present
From the start time, loop with one hour increments until the end time is reached or one day difference is reached. Each emission is a "count" for the hours difference from the startDay.
Reduce to sum the totals per each hour
There is an optional section which will also emit 0 values for every hour of the day, so that if no data is recorded then at least there is output for that hour as 0.