Mongodb Map/Reduce - Multiple Group By - mongodb

I am trying to run a map/reduce function in mongodb where I group by 3 different fields contained in objects in my collection. I can get the map/reduce function to run, but all the emitted fields run together in the output collection. I'm not sure this is normal or not, but outputting the data for analysis takes more work to clean up. Is there a way to separate them, then use mongoexport?
Let me show you what I mean:
The fields I am trying to group by are the day, user ID (or uid) and destination.
I run these functions:
map = function() {
day = (this.created_at.getFullYear() + "-" + (this.created_at.getMonth()+1) + "-" + this.created_at.getDate());
emit({day: day, uid: this.uid, destination: this.destination}, {count:1});
}
/* Reduce Function */
reduce = function(key, values) {
var count = 0;
values.forEach(function(v) {
count += v['count'];
}
);
return {count: count};
}
/* Output Function */
db.events.mapReduce(map, reduce, {query: {destination: {$ne:null}}, out: "TMP"});
The output looks like this:
{ "_id" : { "day" : "2012-4-9", "uid" : "1234456", "destination" : "Home" }, "value" : { "count" : 1 } }
{ "_id" : { "day" : "2012-4-9", "uid" : "2345678", "destination" : "Home" }, "value" : { "count" : 1 } }
{ "_id" : { "day" : "2012-4-9", "uid" : "3456789", "destination" : "Login" }, "value" : { "count" : 1 } }
{ "_id" : { "day" : "2012-4-9", "uid" : "4567890", "destination" : "Contact" }, "value" : { "count" : 1 } }
{ "_id" : { "day" : "2012-4-9", "uid" : "5678901", "destination" : "Help" }, "value" : { "count" : 1 } }
When I attempt to use mongoexport, I can not separate day, uid, or destination by columns because the map combines the fields together.
What I would like to have would look like this:
{ { "day" : "2012-4-9" }, { "uid" : "1234456" }, { "destination" : "Home"}, { "count" : 1 } }
Is this even possible?
As an aside - I was able to make the output work by applying sed to the file and cleaning up the CSV. More work, but it worked. It would be ideal if I could get it out of mongodb in the correct format.

MapReduce only returns documents of the form {_id:some_id, value:some_value}
see: How to change the structure of MongoDB's map-reduce results?

Related

Insert document into mongodb from existing table

I am trying to write a query in mongo that will create a new table, loop through my data set, and insert the TopExecutiveTitle into the new table. I also would like it to keep count of each position and only insert a position into the table when it is new.
This is what I have so far. This code loops through my table and inserts the TopExectuiveTitle into a new table. However, it does not group them together and keep count. How do I write my query so that it will?
db.car.find().forEach( function (x) {
db.TopExecutiveTable.insert({Topexecutivetitle: x.Topexecutivetitle})
});
Here is a sample of a document in my database.
{
"_id" : ObjectId("5a22c8e562c2e489c5df70fa"),
"2016rank" : 1,
"Dealershipgroupname" : "AutoNation Inc.?",
"Address" : "200 S.W. 1st Ave.",
"City/State/Zip" : "Fort Lauderdale, FL 33301",
"Phone" : "(954) 769-7000",
"Companywebsite" : "www.autonation.com",
"Topexecutive" : "Mike Jackson",
"Topexecutivetitle" : "chairman & CEO",
"Totalnewretailunits" : "337,622",
"Totalusedunits" : "225,713",
"Totalfleetunits" : 3,
"Totalwholesaleunits" : "82,342",
"Total_units" : "649,415",
"Total_number_of _dealerships" : 260,
"Grouprevenuealldepartments*" : "$21,609,000,000",
"2015rank" : 1
}
The result I would like is something like this
"Topexecutivetitle" : "chairman & CEO"
"Count" : 3
"Topexecutivetitle" : "president"
"Count" : 7
}
To do this you need to use the aggregate function of mongo, something like this:
db.car.aggregate([
{
$group:{
_id:"$Topexecutivetitle",
count:{$sum:1}
}
},
{
$project:{
Topexecutivetitle:"$_id",
count:1,
_id:0
}
},
{
$out:"result"
}])
This will give you your desired output and store it into a new collection "result":
{
"_id" : "president",
"count" : 1.0
},
{
"_id" : "chairman & CEO",
"count" : 3.0
}

Mongo query to return distinct count, large documents

I need to be able to get a count of distinct 'transactions' the problem I'm having is that using .distinct() comes back with an error because the documents too large.
I'm not familiar with aggregation either.
I need to be able to group it by 'agencyID' as you see below there are 2 different agencyID's
I need to be able to count transactions where the agencyID is 01721487 etc
db.myCollection.distinct("bookings.transactions").length
this doesn't work as I need to be able to group by agencyID and if there are too many results I get an error saying it's too large.
{
"_id" : ObjectId("5624a610a6e6b53b158b4744"),
"agencyID" : "01721487",
"paxID" : "-530189664",
"bookings" : [
{
"bookingID" : "24232",
"transactions" : [
{
"tranID" : "001",
"invoices" : [
{
"invNum" : "1312",
"type" : "r",
"inv_date" : "20150723",
"inv_time" : "0953",
"inv_val" : -300
}
],
"tranType" : "Fee",
"tranDate" : "20150723",
"tranTime" : "0952",
"opCode" : "admin",
"udf_1" : "j s"
}
],
"acctID" : "acct11",
"agt_id" : "xy"
}
],
"title" : "",
"firstname" : "",
"surname" : "f bar"
}
I've also tried this but it didn't work for me.
thank you for text data -
this is something you could play with:
db.kieron.aggregate([{
$unwind : "$bookings"
}, {
$match : {
"bookings.transactions" : {
$exists : true,
$not : {
$size : 0
}
}
}
}, {
$group : {
_id : "$agencyID",
count : {
$sum : {
$size : "$bookings.transactions"
}
}
}
}
])
as there is nested array we need to unwind it first, and then we can check size of inner array.
Happy reporting!

substract two date end return value

I need help to build a query to substract two dates in mongodb.
I have some documents like above :
{"_id" : "32472034809", "center": "102030", dateArq : 141010, inDate : "ISODate("2014-06-06T02:57:19.000-03:00)", biDate : ISODate("2014-06-07T02:57:19.000-03:00)"}
And Im trying to write a query
db.teste.aggregation([{$match : {dateArq : 141010}},{$project : {$subtract : ["$biDate" "$inDate"]}}])
In fact, I want to do : for each _id I want to result biDate - inDate , because I need to see if dateArq keep in a line constante.
In Oracle I did
select dateArq, (biDate - inDate) diff from teste where dateArq = 141010
Tks for help
The document and aggregation pipeline provided had syntax problems, and you needed to put a field name for the result of the $subtract, but otherwise your pipeline works for me:
> db.test.findOne()
{
"_id" : "32472034809",
"center" : "102030",
"dateArq" : 141010,
"inDate" : ISODate("2014-06-06T05:57:19Z"),
"biDate" : ISODate("2014-06-07T05:57:19Z")
}
> db.test.findOnedb.test.aggregate([
{ "$match" : { "dateArq" : 141010 } },
{ "$project" : { "dateDiff" : { "$subtract" : ["$biDate", "$inDate"] } } }
])
{ "_id" : "32472034809", "dateDiff" : NumberLong(86400000) }

MongoDB MapReduce producing different results for each document

This is a follow-up from this question, where I tried to solve this problem with the aggregation framework. Unfortunately, I have to wait before being able to update this particular mongodb installation to a version that includes the aggregation framework, so have had to use MapReduce for this fairly simple pivot operation.
I have input data in the format below, with multiple daily dumps:
"_id" : "daily_dump_2013-05-23",
"authors_who_sold_books" : [
{
"id" : "Charles Dickens",
"original_stock" : 253,
"customers" : [
{
"time_bought" : 1368627290,
"customer_id" : 9715923
}
]
},
{
"id" : "JRR Tolkien",
"original_stock" : 24,
"customers" : [
{
"date_bought" : 1368540890,
"customer_id" : 9872345
},
{
"date_bought" : 1368537290,
"customer_id" : 9163893
}
]
}
]
}
I'm after output in the following format, that aggregates across all instances of each (unique) author across all daily dumps:
{
"_id" : "Charles Dickens",
"original_stock" : 253,
"customers" : [
{
"date_bought" : 1368627290,
"customer_id" : 9715923
},
{
"date_bought" : 1368622358,
"customer_id" : 9876234
},
etc...
]
}
I have written this map function...
function map() {
for (var i in this.authors_who_sold_books)
{
author = this.authors_who_sold_books[i];
emit(author.id, {customers: author.customers, original_stock: author.original_stock, num_sold: 1});
}
}
...and this reduce function.
function reduce(key, values) {
sum = 0
for (i in values)
{
sum += values[i].customers.length
}
return {num_sold : sum};
}
However, this gives me the following output:
{
"_id" : "Charles Dickens",
"value" : {
"customers" : [
{
"date_bought" : 1368627290,
"customer_id" : 9715923
},
{
"date_bought" : 1368622358,
"customer_id" : 9876234
},
],
"original_stock" : 253,
"num_sold" : 1
}
}
{ "_id" : "JRR Tolkien", "value" : { "num_sold" : 3 } }
{
"_id" : "JK Rowling",
"value" : {
"customers" : [
{
"date_bought" : 1368627290,
"customer_id" : 9715923
},
{
"date_bought" : 1368622358,
"customer_id" : 9876234
},
],
"original_stock" : 183,
"num_sold" : 1
}
}
{ "_id" : "John Grisham", "value" : { "num_sold" : 2 } }
The even indexed documents have the customers and original_stock listed, but an incorrect sum of num_sold.
The odd indexed documents only have the num_sold listed, but it is the correct number.
Could anyone tell me what it is I'm missing, please?
Your problem is due to the fact that the format of the output of the reduce function should be identical to the format of the map function (see requirements for the reduce function for an explanation).
You need to change the code to something like the following to fix the problem, :
function map() {
for (var i in this.authors_who_sold_books)
{
author = this.authors_who_sold_books[i];
emit(author.id, {customers: author.customers, original_stock: author.original_stock, num_sold: author.customers.length});
}
}
function reduce(key, values) {
var result = {customers:[] , num_sold:0, original_stock: (values.length ? values[0].original_stock : 0)};
for (i in values)
{
result.num_sold += values[i].num_sold;
result.customers = result.customers.concat(values[i].customers);
}
return result;
}
I hope that helps.
Note : the change num_sold: author.customers.length in the map function. I think that's what you want

Mongodb map reduce trivial query

I have a below map:
var mapFunction = function() {
if(this.url.match(/http:\/\/test.com\/category\/.*?\/checkout/)) {
var key=this.em;
var value = {
url : 'checkout',
count : 1,
account_id:this.accId
}emit(key,value); };
if(this.url.match(/http:\/\/test.com\/landing/)) {
var key=this.em;
var value = {
url : 'landing',
count : 1,
account_id:this.accId
}emit(key,value); };
}
Then I have defined reduce something like below:
var reduceFunction = function (keys, values) {
var reducedValue = {count_checkout:0, count_landing:0};
for (var idx = 0; idx < values.length; idx++) {
if(values[idx].url=='checkout'){
reducedValue.count_checkout++;
}
else {
reducedValue.count_landing++;
}
}
return reducedValue;
}
Now, lets say I have only 1 record:
{
"_id" : ObjectId("516a7cff6dad5949ddf3f7b6"),
"ip" : "1.2.3.4",
"accId" : 123,
"em" : "testing#test.com",
"pgLdTs" : ISODate("2013-04-11T18:30:00Z"),
"url" : "http://test.com/category/prr/checkout",
"domain" : "www.test.com",
"pgUdTs" : ISODate("2013-04-14T09:55:11.682Z"),
"title" : "Test",
"ua" : "Mozilla",
"res" : "1024*768",
"rfr" : "www.google.com"
}
Now if I fire my map reduce like below:
db.test_views.mapReduce(mapFunction,reduceFunction,{out:{inline:1}})
The I get below result returned:
{
"_id" : "testing#test.com",
"value" : {
"url" : "checkout",
"count" : 1,
"account_id" : 123
}
}
So, its basically returning me the map. Now, if I go a add another document for this email id. Finally it becomes something like below.
{
"_id" : ObjectId("516a7cff6dad5949ddf3f7b6"),
"ip" : "1.2.3.4",
"accId" : 123,
"em" : "testing#test.com",
"pgLdTs" : ISODate("2013-04-11T18:30:00Z"),
"url" : "http://test.com/category/prr/checkout",
"domain" : "www.test.com",
"pgUdTs" : ISODate("2013-04-14T09:55:11.682Z"),
"title" : "Test",
"ua" : "Mozilla",
"res" : "1024*768",
"rfr" : "www.google.com"
}
{
"_id" : ObjectId("516a7e1b6dad5949ddf3f7b7"),
"ip" : "1.2.3.4",
"accId" : 123,
"em" : "testing#test.com",
"pgLdTs" : ISODate("2013-04-11T18:30:00Z"),
"url" : "http://test.com/category/prr/checkout",
"domain" : "www.test.com",
"pgUdTs" : ISODate("2013-04-14T09:59:55.326Z"),
"title" : "Test",
"ua" : "Mozilla",
"res" : "1024*768",
"rfr" : "www.google.com"
}
Then, I go again and fire the map reduce, it gives me proper results
{
"_id" : "testing#test.com",
"value" : {
"count_checkout" : 2,
"count_landing" : 0
}
}
Can anyone please help me out in understanding why it returns me a map for single document and doesn't do the counting in reduce.
Thanks for help.
-Lalit
Can anyone please help me out in understanding why it returns me a map for single document and doesn't do the counting in reduce.
The Reduce step combines documents with the same key into a single result document. If you only have one key in the data emitted by your Map function, the data is already "reduced" and the reduce() will not be called.
This is the expected behaviour of the MapReduce algorithm.
The reduce function should return the same type of value objects as the map function emits.
Like you've experienced, when there's a single value associated with a key - the reduce function will not be called at all .
From the MongoDB MapReduce Documentation:
Requirements for the reduce Function:
...
the type of the return object must be identical to the type of the value emitted by the map function to ensure that the following operations is true:
reduce(key, [ C, reduce(key, [ A, B ]) ] ) == reduce( key, [ C, A, B ] )