MapReduce in MongoDB doesn't reduce all the k-v pairs with the same key in one go - mongodb

Question

I have imported a db from a csv with has info about:
country
region
commodity
price
date
(This is the csv: https://www.kaggle.com/jboysen/global-food-prices)
the strings in the csv are ordered in this way:
country 1, region 1.1, commodity X, price, dateA
country 1, region 1.1, commodity X, price, dateB
country 1, region 1.1, commodity Y, price, dateA
country 1, region 1.1, commodity Y, price, dateB
...
country 1, region 1.2, commodity X, price, dateA
country 1, region 1.2, commodity X, price, dateB
country 1, region 1.2, commodity Y, price, dateA
country 1, region 1.2, commodity Y, price, dateB
...
country 2, region 2.1, commodity X, price, dateA
...
I need to show, for each country, for each product, the biggest price.
I wrote:
1) a map with key country+commodity and value price
var map = function() {
emit({country: this.country_name, commodity: this.commodity_name}, {price: this.price});
};
2) a reduce that scans the prices related to a key and check what's the highest price
var reduce = function(key, values) {
var maxPrice = 0.0;
values.forEach(function(doc) {
var thisPrice = parseFloat(doc.price);
if( typeof doc.price != "undefined") {
if (thisPrice > maxPrice) {
maxPrice = thisPrice;
}
}
});
return {max_price: maxPrice};
};
3) I send the output of a map reduce to a collection "mr"
db.prices.mapReduce(map, reduce, {out: "mr"});
PROBLEM:
For example, if I open the csv and manually order by:
country (increasing order)
commodity (increasing order)
price (decreasing order)
I can check that (to give an example of data) in Afghanistan the highest price for the commodity Bread is 65.25
When I check the M-R though, it results 0 for max price of Bread in Afghanistan.
WHAT HAPPENS:
There are 10 regions in the csv in which Bread is logged for Afghanistan.
I've added, on the last line of the reduce:
print("reduce with key: " + key.country + ", " + key.commodity + "; max price: " + maxPrice);
Theoretically, if I search in mongodb log, I should only find ONE entrance with "reduce with key: Afghanistan, Bread; max price: ???".
Instead I see TEN lines (same numbers of the regions), each one with a different max price.
The last one has "max price 0".
MY HYPOTESIS:
It seems that, after the emit, when the reduce is called, instead of looking for ALL k-v pairs with the same key, it considers sub-groups that are in promixity.
So, recalling my starting example on the csv structure:
until the reduce scans emit outputs related to "afghanista, region 1, bread", it does a reduce on themm
then it does a reduce on the outputs related to "afghanistan, region 1, commodityX"
then it does another reduce on the outputs related to "afghanistan, region 2, bread" (instead of reducing ALL the k-v pairs with afghanistan+bread in a single reduce)
Do I have to do a re-reduce to work on all the partial reduce jobs?

Answer 1

I've managed to solve this.
MongoDB doesn't necessarily do the reducing of all k-v pairs with the same key in one go.
It can happen that (as in this case) MongoDB will perform a reduce on a subset of k-v pairs related to a specific key, and then it will send the output of this first reduce when it will do a second reduce on another subset related to the same key.
My code didn't work because:
MongoDB performed a reduce on a subset of k-v pairs related to the key "Afghanistan, Bread", with a variable in output named "maxPrice"
MongoDB would proceed to reduce other subsets
MongoDB, when faced with another subset of "Afghanistan, Bread", would take the output of the first reduce, and use it as a value
The output of a reduce is named "maxPrice", but the other values are named "price"
Since I ask for the value "doc.price", when I scan the doc that contains "maxPrice", it gets ignored
There are 2 approaches to solve this:
1) You use the same name for the reduce output variable as the emit output value
2) You index the properties chosen as key, and you use the "sort" option on mapReduce() so that all k-v pairs related to a key get reduced in one go
The second approach is if you don't want to give up using a different name for the name of the reduce output (plus it has better performance since it only does a single reduce per key).