MongoDb MapReduce on child array - mongodb

Question

I've searched the internet long and hard but can't find a solution to this problem. Whilst there are lots of Map reduce examples, i'm getting confused because my document has a property which is an array of objects.
I'm pretty sure this should be easy for someone with experience but i'm a noob at the minute.
I have a document which looks roughly like this
{
_id:guid,
clientId:guid,
reference:'abc123'
items:
[
{ _id:guid, category:'A', length:100, active:true },
{ _id:guid, category:'B', length:150, active:true },
{ _id:guid, category:'A', length:10, active:false },
{ _id:guid, category:'A', length:111, active:true },
]
}
and I want to produce this output
dateFromIdGuid(day) category countOfItems countOfActive sumOfLength
I'd like to keep the data in this format to reduce the number of write operations (there are already over 1000 writes to this collection per second and rising)
This is driving me insane so any help would be very much appreciated.
Thanks.

Answer 1

If you are talking about extracting a timestamp and reducing that to a discrete day from a GUID, then MongoDB is not going to be of much help to you there. You would need an external language implementation that would support such a function and implement an external mapReduce process such as with Hadoop.
It makes me wonder though if we are in fact talking about a GUID or whether you actually mean an ObjectID which would be the default value for the _id field of your document unless this has been specifically overridden to have a GUID in there.
Even if that is not true, you would be helped by adding a "timestamp" field of some sort to your document and using the correct BSON Date object type as shown below:
{
_id:guid,
"timestamp": ISODate("2014-05-27T00:00:00Z")
"clientId":guid,
"reference":'abc123'
"items":
[
{ _id:guid, category:'A', length:100, active:true },
{ _id:guid, category:'B', length:150, active:true },
{ _id:guid, category:'A', length:10, active:false },
{ _id:guid, category:'A', length:111, active:true },
]
}
This allows you to use the MongoDB aggregation framework as it can operate on Date objects of this type in order to break down the results to discrete days:
db.collection.aggregate([
{ "$unwind": "$items" },
{ "$group": {
"_id": {
"day": { "$dayOfYear": "$timestamp" },
"category": "$items.category"
},
"countOfItems": { "$sum": 1 },
"countOfActive": {
"$sum": {
"$cond": [
"$items.active",
1,
0
]
}
},
"sumOfLength": { "$sum": "$items.length" }
}}
])
That not only gives you the results in the fastest way MongoDB can do it but that "timestamp" value is also useful for filtering queries within date ranges which is something you cannot easily do from other values.
Also there is a way in the JavaScript available to MongoDB mapReduce that allows you to get the date from an ObejctId. This runs slower than the aggregation framework though:
db.collection.mapReduce(
function() {
var date = this._id.getTimestamp();
items.forEach(function(item) {
var day =
"" + date.getFullyear() +
"" + ( date.getMonth() + 1 ) +
"" + date.getDate();
emit(
{
day: day,
category: item.category
},
{
countOfItems: 1,
countOfActive: ( item.active ) ? 1 : 0,
sumOfLength: item.length
}
);
});
},
function( key, values ) {
var reduced = {
countOfItems: 0,
countOfActive: 0,
sumOfLength: 0
};
values.forEach(function(value) {
for ( var k in value ) {
reduced[k] += value[k];
}
});
return reduced;
},
{
"out": { "inline": 1 }
}
)
That basically does the same thing where the mapper breaks apart the array and provides grouping keys while the reducer just sums up the values from the mapper. So even if you had to extract from GUID's that gives you a basic layout for a mapper and reducer in a language such as Java when using Hadoop.
Take a look at the aggregate and mapReduce manual pages for more information on options you can apply.