MongoDB aggregation group by month in big collection - optimize pipeline - mongodb

Question

I'm aware that this question has been asked before at SO - but I can't seem to find how to handle aggregation grouping in bigger collections. I have a set of +10 million records, and I just can't get any speed to it.
Running MongoDB v 3.2.
Having a field __createDateUtc (ISODate) in the schema, I'm trying the following pipeline:
db.transactions.aggregate([
{
$project: {
__createDateUtc: 1
}
},
{
$group: {
'_id': { $year: '$__createDateUtc' },
'count': {$sum: 1},
}
},
{
$limit: 10
},
])
This runs at +20 seconds. Could it be made faster? This is a fairly simple pipeline - so really - is there any other strategy that might help in this situation?

Answer 1

I did some bench marking with four different ways of getting the results that I wanted. The results are a discouraging.
Again, with a schema looking like:
{
"_id" : ObjectId("5d665491fd5852755236a5dc"),
...
"__createDateUtc" : ISODate("2019-08-28T10:16:49Z"),
"__createDate" : {
"year" : 2019,
"month" : 8,
"day" : 28,
"yearMonth" : 201908,
"yearMonthDay" : 20190829
}
}
The results:
// Group by __createDate.yearMonth
db.transactions.aggregate([
{ $group: {
'_id': '$__createDate.yearMonth',
'count': {$sum: 1},
} },
{ $limit: 10 },
{ $sort: {'_id': -1 } }
])
// 20 169 ms
// Group by year and month
db.transactions.aggregate([
{$group: {
'_id': {year: '$__createDate.year', month: '$__createDate.month' },
'count': {$sum: 1},
}},
{ $limit: 10 },
{ $sort: {'_id': -1 } }
])
// 23 777 ms
// Group by calculating year and month from ISODate
db.transactions.aggregate([
{$group: {
'_id': {year: { $year: '$__createDateUtc' }, month: { $month: '$__createDateUtc' } },
'count': {$sum: 1},
}},
{ $limit: 10 },
{ $sort: {'_id': -1 } }
])
// 16 444 ms
// Last stupid method to just run many queries with count
var years = [2017, 2018, 2019];
var results = {}
years.forEach(year => {
results[year] = {};
for(var i = 1; i < 13; i++) {
var count = db.transactions.find({'__createDate.year': year, '__createDate.month': i}).count();
if(count > 0) results[year][i] = count;
}
})
// 10 701 ms
As you can see the last method of just running multiple counts is by far the fastest. Especially since I'm actually fetching a lot more data compared to the three other methods.
This just seems stupid to me. I know MongoDB is no search engine, but still. Aggregation is just not fast at all. Makes me wanna sync data to elastic search and try to aggregate within ES instead.