Map reduce to count the unique count - mongodb

I want a map reduce function to draw the below output from the below input collection which satisfy the below condition.
Input collection:
[{
a:1,
b:'test',
indices:[1,2,4,5]
}, {
a:2,
b:'test',
indices:[2, 3, 5]
}, {
a:2,
b:'test',
indices:[1, 2, 4]
}, {
a:3,
b:'apple',
indices:[1, 2]
}, {
a:4,
b:'apple',
indices:[1, 3, 5]
}, {
a:5,
b:'orange',
indices:[232]
}, {
a:5,
b:'dummy',
indices:[2]
}, {
a:6,
b:'dummy',
indices:[11, 2, 4]
}, {
a:6,
b:'dummy',
indices:[11, 3, 2]
}, {
a:6,
b:'dummy',
indices:[1, 2, 3, 4, 5]
}]
Conditions are:
select only which has indices array has 2. This can be send as
query. i.e, query:{indices:{$in:2}}
Group by b
If there are duplicates a, then it should be considered as 1 eg: Document having a=2 are present in two times satisfying the condition indices
has 2.
My input collection always satisfies the condition of if a
prsents in "test", it will not present in dummy/apple/etc. but a
can be duplicate.
Here is what I tried:
db.x.mapReduce(function(){
emit(this.b, 1);
}, function(key, reducable){
return Array.sum(reducable);
}, {
out: {inline: 1},
query:{
'indices':{$in:2}
}
});
Output:
[
{
"_id" : test",
"value" : {
"count" : 3 -> It should be 2
}
},{
"_id" : apple",
"value" : {
"count" : 2
}
},{
"_id" : dummy",
"value" : {
"count" : 4 -> It should be 2
}
}]
Expected Output:
[{
"_id" : test",
"value" : {
"count" : 2
}
},{
"_id" : apple",
"value" : {
"count" : 2
}
},{
"_id" : dummy",
"value" : {
"count" : 2
}
}]

No need for map/reduce. Use aggregation:
> db.crawler_status.aggregate([
{ "$match" : { "indices" : 2 } },
{ "$group" : { "_id" : { "b" : "$b", "a" : "$a" } } },
{ "$group" : { "_id" : "$_id.b", "count" : { "$sum" : 1 } } }
])
{ "_id" : "test", "count" : 2 }
{ "_id" : "apple", "count" : 1 } // your sample output was mistaken
{ "_id" : "dummy", "count" : 2 }

Related

Mongo aggregation - Sorting using a field value from previous pipeline as the sort field

I have produced the below output using mongodb aggregation (including $group pipeline inside levelsCount field) :
{
"_id" : "1",
"name" : "First",
"levelsCount" : [
{ "_id" : "level_One", "levelNum" : 1, "count" : 1 },
{ "_id" : "level_Three", "levelNum" : 3, "count" : 1 },
{ "_id" : "level_Four", "levelNum" : 4, "count" : 8 }
]
}
{
"_id" : "2",
"name" : "Second",
"levelsCount" : [
{ "_id" : "level_One", "levelNum" : 1, "count" : 5 },
{ "_id" : "level_Two", "levelNum" : 2, "count" : 2 },
{ "_id" : "level_Three", "levelNum" : 3, "count" : 1 },
{ "_id" : "level_Four", "levelNum" : 4, "count" : 3 }
]
}
{
"_id" : "3",
"name" : "Third",
"levelsCount" : [
{ "_id" : "level_One", "levelNum" : 1, "count" : 1 },
{ "_id" : "level_Two", "levelNum" : 2, "count" : 3 },
{ "_id" : "level_Three", "levelNum" : 3, "count" : 2 },
{ "_id" : "level_Four", "levelNum" : 4, "count" : 3 }
]
}
Now, I need to sort these documents based on the levelNum and count fields of levelsCount array elements. I.e. If two documents both had the count 5 forlevelNum: 1 (level_One), then the sort goes to compare the count of levelNum: 2 (level_Two) field and so on.
I see how $sort pipeline would work on multiple fields (Something like { $sort : { level_One : 1, level_Two: 1 } }), But the problem is how to access those values of levelNum of each array element and set that value as a field name to do sorting on that. (I couldn't handle it even after $unwinding the levelsCount array).
P.s: The initial order of levelsCount array's elements may differ on each document and is not important.
Edit:
The expected output of the above structure would be:
// Sorted result:
{
"_id" : "2",
"name" : "Second",
"levelsCount" : [
{ "_id" : "level_One", "levelNum" : 1, "count" : 5 }, // "level_One's count: 5" is greater than "level_One's count: 1" in two other documents, regardless of other level_* fields. Therefore this whole document with "name: Second" is ordered first.
{ "_id" : "level_Two", "levelNum" : 2, "count" : 2 },
{ "_id" : "level_Three", "levelNum" : 3, "count" : 1 },
{ "_id" : "level_Four", "levelNum" : 4, "count" : 3 }
]
}
{
"_id" : "3",
"name" : "Third",
"levelsCount" : [
{ "_id" : "level_One", "levelNum" : 1, "count" : 1 },
{ "_id" : "level_Two", "levelNum" : 2, "count" : 3 }, // "level_Two's count" in this document exists with value (3) while the "level_Two" doesn't exist in the below document which mean (0) value for count. So this document with "name: Third" is ordered higher than the below document.
{ "_id" : "level_Three", "levelNum" : 3, "count" : 2 },
{ "_id" : "level_Four", "levelNum" : 4, "count" : 3 }
]
}
{
"_id" : "1",
"name" : "First",
"levelsCount" : [
{ "_id" : "level_One", "levelNum" : 1, "count" : 1 },
{ "_id" : "level_Three", "levelNum" : 3, "count" : 1 },
{ "_id" : "level_Four", "levelNum" : 4, "count" : 8 }
]
}
Of course, I'd prefer to have an output document in the below format, But the first problem is to sort all docs:
{
"_id" : "1",
"name" : "First",
"levelsCount" : [
{ "level_One" : 1 },
{ "level_Three" : 1 },
{ "level_Four" : 8 }
]
}
You can sort by levelNum as descending order and count as ascending order,
db.collection.aggregate([
{
$sort: {
"levelsCount.levelNum": -1,
"levelsCount.count": 1
}
}
])
Playground
For key-value format result of levelsCount array,
$map to iterate loop of levelsCount array
prepare key-value pair array and convert to object using $arrayToObject
{
$addFields: {
levelsCount: {
$map: {
input: "$levelsCount",
in: {
$arrayToObject: [
[{ k: "$$this._id", v: "$$this.levelNum" }]
]
}
}
}
}
}
Playground

mongodb aggregate sum item as nested data

Here is my some sample data in collection sale
[
{group:2, item:a, qty:3 },
{group:2, item:b, qty:3 },
{group:2, item:b, qty:2 },
{group:1, item:a, qty:3 },
{group:1, item:a, qty:5 },
{group:1, item:b, qty:5 }
]
and I want to query data like below and sort the popular group to the top
[
{ group:1, items:[{name:'a',total_qty:8},{name:'b',total_qty:5} ],total_qty:13 },
{ group:2, items:[{name:'a',total_qty:3},{name:'b',total_qty:5} ],total_qty:8 },
]
Actually we can loop in server script( php, nodejs ...) but the problem is pagination. I cannot use skip to get the right result.
The following query can get us the expected output:
db.collection.aggregate([
{
$group:{
"_id":{
"group":"$group",
"item":"$item"
},
"group":{
$first:"$group"
},
"item":{
$first:"$item"
},
"total_qty":{
$sum:"$qty"
}
}
},
{
$group:{
"_id":"$group",
"group":{
$first:"$group"
},
"items":{
$push:{
"name":"$item",
"total_qty":"$total_qty"
}
},
"total_qty":{
$sum:"$total_qty"
}
}
},
{
$project:{
"_id":0
}
}
]).pretty()
Data set:
{
"_id" : ObjectId("5d84a37febcbd560107c54a7"),
"group" : 2,
"item" : "a",
"qty" : 3
}
{
"_id" : ObjectId("5d84a37febcbd560107c54a8"),
"group" : 2,
"item" : "b",
"qty" : 3
}
{
"_id" : ObjectId("5d84a37febcbd560107c54a9"),
"group" : 2,
"item" : "b",
"qty" : 2
}
{
"_id" : ObjectId("5d84a37febcbd560107c54aa"),
"group" : 1,
"item" : "a",
"qty" : 3
}
{
"_id" : ObjectId("5d84a37febcbd560107c54ab"),
"group" : 1,
"item" : "a",
"qty" : 5
}
{
"_id" : ObjectId("5d84a37febcbd560107c54ac"),
"group" : 1,
"item" : "b",
"qty" : 5
}
Output:
{
"group" : 2,
"items" : [
{
"name" : "b",
"total_qty" : 5
},
{
"name" : "a",
"total_qty" : 3
}
],
"total_qty" : 8
}
{
"group" : 1,
"items" : [
{
"name" : "b",
"total_qty" : 5
},
{
"name" : "a",
"total_qty" : 8
}
],
"total_qty" : 13
}
You need to use $group aggregation with $sum and $push accumulator
db.collection.aggregate([
{ "$group": {
"_id": "$group",
"items": { "$push": "$$ROOT" },
"total_qty": { "$sum": "$qty" }
}},
{ "$sort": { "total_qty": -1 }}
])

MongoDB: Aggregation Group operation over a dynamic key-value pair

I have a simple document with one field as a key-value pair. I want to just perform a group operation in Aggregation over those keys and add their values. But the keys in the pair are not fixed and can be anything.
Here is a sample document.
{
_id: 349587843,
matchPair: {
3 : 21,
9 : 4,
7 : 32
}
},
{
_id: 349587478,
matchPair: {
7 : 11,
54 : 32,
9 : 7,
2 : 19
}
}
And I want a result something like the following.
{
_id : 3,
count : 21
},
{
_id : 9,
count : 11
},
{
_id : 7,
count : 43
},
{
_id : 54,
count : 32
},
{
_id : 2,
count : 19
}
I have the following query in mind and tried using $unwindoperation but it doesn't work probably because "matchPair" isn't an array and I don't know what to specify for the $sumoperation.
db.MatchPairs.aggregate([
{ "$unwind" : "$matchPair" },
{ "$group" : {
_id: "$matchPair",
count : { $sum : $matchPair }
} }
]);
I could also try Map-Reduce but for that too I need to emit() keys and values by name.
I'm sure there's a simple solution to this but I can't figure it out.
:
You could start by projecting and reshaping the matchPair field with $objectToArray
New in version 3.4.4.
{
$project: {
matchPair: { $objectToArray: '$matchPair' }
}
}
which would give
{
matchPair: [{ k: 3, v: 21 }, { k: 9, v: 4 }, ...]
}
Then $unwind based on matchPair
{
$unwind: '$matchPair'
}
which would give
{
matchPair: { k: 3, v: 21 }
}
Then $project
{
$project: {
_id: '$matchPair.k',
count: '$matchPair.v'
}
}
That should give the output you want. Altogether would be
.aggregate([
{
$project: {
matchPair: { $objectToArray: '$matchPair' }
}
},
{ $unwind: '$matchPair' },
{
$project: {
_id: '$matchPair.k',
count: '$matchPair.v'
}
}
])
In the mongoDb documentation for $unwind:
Deconstructs an array field from the input documents to output a
document for each element.
So you have to change your schema for something like:
{
"_id" : ObjectId("5880b57b039a3c89c1db145a"),
"matchPair" : [
{
"_id" : "3",
"count" : 21
},
{
"_id" : "9",
"count" : 4
},
{
"_id" : "7",
"count" : 32
}
]
},
{
"_id" : ObjectId("5880b58c039a3c89c1db145b"),
"matchPair" : [
{
"_id" : "7",
"count" : 11
},
{
"_id" : "54",
"count" : 32
},
{
"_id" : "9",
"count" : 7
},
{
"_id" : "2",
"count" : 19
}
]
}
Then doing:
db.MatchPairs.aggregate([
{ $unwind : "$matchPair" }
]);
will return:
{
"_id" : ObjectId("5880b57b039a3c89c1db145a"),
"matchPair" : {
"_id" : "3",
"count" : 21
}
},
{
"_id" : ObjectId("5880b57b039a3c89c1db145a"),
"matchPair" : {
"_id" : "9",
"count" : 4
}
},
{
"_id" : ObjectId("5880b57b039a3c89c1db145a"),
"matchPair" : {
"_id" : "7",
"count" : 32
}
},
{
"_id" : ObjectId("5880b58c039a3c89c1db145b"),
"matchPair" : {
"_id" : "7",
"count" : 11
}
},
{
"_id" : ObjectId("5880b58c039a3c89c1db145b"),
"matchPair" : {
"_id" : "54",
"count" : 32
}
},
{
"_id" : ObjectId("5880b58c039a3c89c1db145b"),
"matchPair" : {
"_id" : "9",
"count" : 7
}
},
{
"_id" : ObjectId("5880b58c039a3c89c1db145b"),
"matchPair" : {
"_id" : "2",
"count" : 19
}
}
Then you just have to do your grouping.

Use $ and $elemMatch to group entities

Considering the following document in my mongo DB instance :
{
"_id": 1,
"people": [
{"id": 1, "name": "foo"},
{"id": 2, "name": "bar"},
/.../
],
"stats": [
{"peopleId": 1, "workHours": 24},
{"peopleId": 2, "workHours": 36},
/.../
}
Each element in my collection represent the work of every employee in my company, each weeks. As an important note, peopleId may change from one week to another !
I would like to get all weeks where foo worked more than 24 hours. As you can see, the format is kinda annoying since the people name and the work hours are separated in my database. A simple $and is not enough.
I wonder if, using some $ and $elemMatch I can achieve doing this query.
Can I use this to group the "people" entities with "stats" entities ?
Query to get foo worked more than 24 hours.
db.collection.aggregate([
{$unwind: { path : "$people"}},
{$unwind: { path : "$stats"}},
{$match: { "people.name" : "foo"}},
{$group: {
_id: "$_id",
peopleIdMoreThan24: { $addToSet: {
$cond : { if : { $and : [ {"$eq" : ["$people.id", "$stats.peopleId" ] },
{"$gt" : ["$stats.workHours", 24] }]} , then : "$people.id", else: "Not satisfying the condition"}}}
}
},
{$unwind: { path : "$peopleIdMoreThan24" }},
{$match: { "peopleIdMoreThan24" : {$nin : [ "Not satisfying the condition"]}}},
]);
Data in collection:-
/* 1 */
{
"_id" : 1,
"people" : [
{
"id" : 1,
"name" : "foo"
},
{
"id" : 2,
"name" : "bar"
}
],
"stats" : [
{
"peopleId" : 1,
"workHours" : 24
},
{
"peopleId" : 2,
"workHours" : 36
}
]
}
/* 2 */
{
"_id" : 2,
"people" : [
{
"id" : 1,
"name" : "foo"
},
{
"id" : 2,
"name" : "bar"
}
],
"stats" : [
{
"peopleId" : 1,
"workHours" : 25
},
{
"peopleId" : 2,
"workHours" : 36
}
]
}
/* 3 */
{
"_id" : 3,
"people" : [
{
"id" : 1,
"name" : "foo"
},
{
"id" : 2,
"name" : "bar"
}
],
"stats" : [
{
"peopleId" : 1,
"workHours" : 25
},
{
"peopleId" : 2,
"workHours" : 36
}
]
}
Output:-
The output has document id and people id of foo worked more than 24 hours.
/* 1 */
{
"_id" : 3,
"peopleIdMoreThan24" : 1
}
/* 2 */
{
"_id" : 2,
"peopleIdMoreThan24" : 1
}

Use field value as key

I am doing this query
db.analytics.aggregate([
{
$match: {"event":"USER_SENTIMENT"}
},
{ $group: {
_id: {brand:"$data.brandId",sentiment:"$data.sentiment"},
count: {$sum : 1}
}
},
{ $group: {
_id: "$_id.brand",
sentiments: {$addToSet : {sentiment:"$_id.sentiment", count:"$count"}}
}
}
])
Which generates that :
{
"result" : [
{
"_id" : 57,
"sentiments" : [
{
"sentiment" : "Meh",
"count" : 4
}
]
},
{
"_id" : 376,
"sentiments" : [
{
"sentiment" : "Meh",
"count" : 1
},
{
"sentiment" : "Happy",
"count" : 1
},
{
"sentiment" : "Confused",
"count" : 1
}
]
}
],
"ok" : 1
}
But What I want is that :
[
{
"_id" : 57,
"Meh" : 4
},
{
"_id" : 376,
"Meh" : 1,
"Happy" : 1,
"Confused" : 1
}
]
Any idea on how to transform that? The blocking point for me is to transform a value into a key.