MapReduce sub-document - mongodb

Question

I am trying to graph email activity that I am recording in a Mongo db. Whenever I send out an email I create a record, then, when there is activity on the email (open, click, mark as spam) I update the document by adding to it's history.
Here is a sample document:
{
"_id" : new BinData(3, "wbbS0lRI0ESx5DyStKq9pA=="),
"MemberId" : null,
"NewsletterId" : 4,
"NewsletterTypeId" : null,
"Contents" : "[message goes here]",
"History" : [{
"EmailActionType" : "spam",
"DateAdded" : new Date("Sat, 10 Dec 2011 04:17:26 GMT -08:00")
}, {
"EmailActionType" : "processed",
"DateAdded" : new Date("Sun, 11 Dec 2011 04:17:26 GMT -08:00")
}, {
"EmailActionType" : "deffered",
"DateAdded" : new Date("Mon, 12 Dec 2011 04:17:26 GMT -08:00")
}],
"DateAdded" : new Date("Mon, 01 Jan 0001 00:00:00 GMT -08:00")
}
What I would like to do is query the database for a specific history date range. The end result should be a list with an item for each day where there is an activity and a total for each activity type:
date: "20111210", spam: 1, processed: 0, deffered: 0
date: "20111211", spam: 0, processed: 1, deffered: 0
date: "20111212", spam: 0, processed: 0, deffered: 1
Here is what I currently have:
db.runCommand({ mapreduce: Email,
map : function Map() {
var key = this.NewsletterId;
emit(
key,
{ "history" : this.History }
);
}
reduce : function Reduce(key, history) {
var from = new Date (2011, 1, 1, 0, 0, 0, 0);
var to = new Date (2013, 05, 15, 23, 59, 59, 0);
// \/ determine # days in the date range \/
var ONE_DAY = 1000 * 60 * 60 * 24; // The number of milliseconds in one day
var from_ms = from.getTime(); // Convert both date1 to milliseconds
var to_ms = to.getTime(); // Convert both date1 to milliseconds
var difference_ms = Math.abs(from_ms - to_ms); // Calculate the difference in milliseconds
var numDays = Math.round(difference_ms/ONE_DAY); // Convert back to days and return
// /\ determine # days between the two days /\
var results = new Array(numDays); //array where we will store the results. We will have an entry for each day in the date range.
//initialize array that will contain our results for each type of emailActivity
for(var i=0; i < numDays; i++){
results[i] = {
numSpam: 0,
numProcessed: 0,
numDeffered: 0
}
}
//traverse the history records and count each type of event
for (var i = 0; i < history.length; i++){
var to_ms2 = history[i].DateAdded.getTime(); // Convert both date1 to milliseconds
var difference_ms2 = Math.abs(from_ms - to_ms2); // Calculate the difference in milliseconds
var resultsIndex = Math.round(difference_ms2/ONE_DAY); //determine which row in the results array this date corresponds to
switch(history[i].EmailActionType)
{
case 'spam':
results[resultsIndex].numSpam = ++results[resultsIndex].numSpam;
break;
case 'processed':
results[resultsIndex].numProcessed = ++results[resultsIndex].numProcessed;
break;
case 'deffered':
results[resultsIndex].numDeffered = ++results[resultsIndex].numDeffered;
break;
}
}
return results;
}
finalize : function Finalize(key, reduced) {
return {
"numSpam": reduced.numSpam,
"numProcessed": reduced.numProcessed,
"numDeffered": reduced.numDeffered,
};
}
out : { inline : 1 }
});
When I run it, I don't get anything, but I'm also not getting any errors, so not really sure where to look.

Answer 1

Your problem is definitely in your Map / Reduce functions. There is a disconnect between your emit and your expected output.
Your expected output:
date: "20111210", spam: 1, processed: 0, deffered: 0
Map / Reduce always outputs in terms of a key and a value. So your output would look like this:
_id: "20111220", value: { spam: 1, processed: 0, deferred: 0 }
Here is the basic premise. Your emit needs to output data of the correct format. So if you emit(key, value), then you should have:
var key='20111220'
var value={spam:1, processed:0, deferred:0}
In your case, you are emitting several times per document as you loop through History. This is normal.
The reduce function is only run if there are multiple values for the same key. So if you have this:
_id: "20111220", value: { spam: 1, processed: 0, deferred: 0 }
_id: "20111220", value: { spam: 1, processed: 2, deferred: 0 }
Then reduce will pull those together and give you this:
_id: "20111220", value: { spam: **2**, processed: **2**, deferred: 0 }
Here is a quick stab at the answer:
map = function() {
for(var i in this.History) {
var key = get_date(this.History[i].DateAdded);
var value = {spam: 0, processed: 0, deffered: 0};
if(this.History[i].EmailActionType == "Spam") { value.spam++; }
else if(....)
...
emit(key, value);
}
}
reduce = function(key, values) {
// values is an array of these things {spam: 0, processed: 0, deffered: 0}
var returnValue = { spam: 1, processed: 0, deffered: 0 };
for(var i in values) {
returnValue.spam += values[i].spam;
returnValue.processed += values[i].processed;
returnValue.deffered += values[i].deffered;
}
return returnValue;
}
Just remember that the structure of emit has to match the structure of your final values.