Count by elements in list and by field - mongodb

I have a MongoDB collection that looks like this:
{ "_id" : 1, "owner" : "Alice", airline: "RSAirlines", "content" : ["shoes", "pants", "sockets"]}
{ "_id" : 2, "owner" : "Bob", airline: "RSAirlines", "content" : ["phone", "pants"]}
{ "_id" : 3, "owner" : "Charlie", airline: "RSAirlines", "content" : ["shoes", "pants", "bag"]}
{ "_id" : 4, "owner" : "Mary" ,airline: "AirES" "content" : ["sandals", "coins", "sockets"]}
{ "_id" : 5, "owner" : "Olivia", airline: "AirES", "content" : ["gloves", "pants", "sockets"]}
{ "_id" : 6, "owner" : "Dan", airline: "AirES", "content" : ["sockets", "wallet"]}
{ "_id" : 7, "owner" : "Erin", airline: "AirES", "content" : ["pants", "sockets", "dress"]}
I would like to aggregate them to get the following results:
{ "_id": "RSAirlines", "counts": {
"shoes": 2,
"pants": 3,
"sockets": 1,
"phone": 1,
"bag": 1
}}
{ "_id": "AirES", "counts": {
"sandals": 1,
"coins": 1,
"sockets": 4,
"wallet": 1,
"dress": 1,
"pants": 2
}}
Previous I saw this answer for counting the elements, but now I would like to count them by airline.

$unwind deconstruct content array
$group by airline and content and get the total count
$group by the only airline and construct counts array key-value format
$arrayToObject convert key-value array to object
db.collection.aggregate([
{ $unwind: "$content" },
{
$group: {
_id: {
airline: "$airline",
content: "$content"
},
count: { $sum: 1 }
}
},
{
$group: {
_id: "$_id.airline",
counts: {
$push: {
k: "$_id.content",
v: "$count"
}
}
}
},
{ $project: { counts: { $arrayToObject: "$counts" } } }
])
Playground

Related

Filter by nested arrays/objects values (on different levels) and $push by multiple level - MongoDB Aggregate

I have a document with multiple level of embedded subdocument each has some nested array. Using $unwind and sort, do sorting based on day in descending and using push to combine each row records into single array. This Push is working only at one level means it allows only one push. If want to do the same things on the nested level and retains the top level data, got "errmsg" : "Unrecognized expression '$push'".
{
"_id" : ObjectId("5f5638d0ff25e01482432803"),
"name" : "XXXX",
"mobileNo" : 323232323,
"payroll" : [
{
"_id" : ObjectId("5f5638d0ff25e01482432801"),
"month" : "Jan",
"salary" : 18200,
"payrollDetails" : [
{
"day" : "1",
"salary" : 200,
},
{
"day" : "2",
"salary" : 201,
}
]
},
{
"_id" : ObjectId("5f5638d0ff25e01482432802"),
"month" : "Feb",
"salary" : 8300,
"payrollDetails" : [
{
"day" : "1",
"salary" : 300,
},
{
"day" : "2",
"salary" : 400,
}
]
}
],
}
Expected Result:
{
"_id" : ObjectId("5f5638d0ff25e01482432803"),
"name" : "XXXX",
"mobileNo" : 323232323,
"payroll" : [
{
"_id" : ObjectId("5f5638d0ff25e01482432801"),
"month" : "Jan",
"salary" : 18200,
"payrollDetails" : [
{
"day" : "2",
"salary" : 201
},
{
"day" : "1",
"salary" : 200
}
]
},
{
"_id" : ObjectId("5f5638d0ff25e01482432802"),
"month" : "Feb",
"salary" : 8300,
"payrollDetails" : [
{
"day" : "2",
"salary" : 400
},
{
"day" : "1",
"salary" : 300
}
]
}
],
}
Just day will be sorted and remaining things are same
I have tried but it got unrecognized expression '$push'
db.employee.aggregate([
{$unwind: '$payroll'},
{$unwind: '$payroll.payrollDetails'},
{$sort: {'payroll.payrollDetails.day': -1}},
{$group: {_id: '$_id', payroll: {$push: {payrollDetails:{$push:
'$payroll.payrollDetails'} }}}}])
It requires two time $group, you can't use $push operator two times in a field,
$group by main id and payroll id, construct payrollDetails array
$sort by payroll id (you can skip if not required)
$group by main id and construct payroll array
db.employee.aggregate([
{ $unwind: "$payroll" },
{ $unwind: "$payroll.payrollDetails" },
{ $sort: { "payroll.payrollDetails.day": -1 } },
{
$group: {
_id: {
_id: "$_id",
pid: "$payroll._id"
},
name: { $first: "$name" },
mobileNo: { $first: "$mobileNo" },
payrollDetails: { $push: "$payroll.payrollDetails" },
month: { $first: "$payroll.month" },
salary: { $first: "$payroll.salary" }
}
},
{ $sort: { "payroll._id": -1 } },
{
$group: {
_id: "$_id._id",
name: { $first: "$name" },
mobileNo: { $first: "$mobileNo" },
payroll: {
$push: {
_id: "$_id.pid",
month: "$month",
salary: "$salary",
payrollDetails: "$payrollDetails"
}
}
}
}
])
Playground

Is there a way to group results from multiple documents when performing aggregation

I am new to mongo and trying to perform aggregation query to calculate min/max of timestamps for a given document.
Sample documents are below -
{
"_id" : ObjectId("5c9cd93adddca9ebb2b3fcba"),
"frequency" : 5,
"s_id" : "30081993",
"timestamp" : NumberLong(1546300800000),
"date" : ISODate("2019-01-01T00:00:00.000Z"),
"values" : {
"1547439900000" : {
"number_of_values" : 3,
"min_value" : 32.13,
"max_value" : 81.42
},
"1547440200000" : {
"number_of_values" : 3,
"min_value" : 48.08,
"max_value" : 84.52
},
"1547440500000" : {
"number_of_values" : 2,
"min_value" : 27.39,
"max_value" : 94.64
}
}
}
{
"_id" : ObjectId("5c9cd851dddca9ebb2b3f2ac"),
"frequency" : 5,
"s_id" : "27061995",
"timestamp" : NumberLong(1546300800000),
"date" : ISODate("2019-01-01T00:00:00.000Z"),
"values" : {
"1547539900000" : {
"number_of_values" : 31,
"min_value" : 322.13,
"max_value" : 831.42
},
"1547540200000" : {
"number_of_values" : 3,
"min_value" : 418.08,
"max_value" : 8114.52
},
"1547740500000" : {
"number_of_values" : 2,
"min_value" : 207.39,
"max_value" : 940.64
}
}
}
I have come up with the following query which works for a single document.
db.testdb.aggregate([
{
$match: {
"s_id": "30081993",
"frequency": 5,
}
},
{
$project: {
_id: 1,
valuesarray: {
$objectToArray: "$values"
}
}
},
{
$unwind: "$valuesarray"
},
{
$group: {
"_id": "",
"min_timestamp": {
$min: "$valuesarray.k"
},
"max_timestamp": {
$max: "$valuesarray.k"
}
}
}
]);
The output is below
{
"_id" : "",
"min_timestamp" : "1547439900000",
"max_timestamp" : "1547440500000"
}
I want an aggregation query which can calculate the max/min of timestamps but for multiple documents i.e I want to use a $in operator during the $match stage and get min/max of all s_id. Is this possible?
Expected :
{
"_id" : "30081993",
"min_timestamp" : "1547439900000",
"max_timestamp" : "1547440500000"
}
{
"_id" : "27061995",
"min_timestamp" : "1547539900000",
"max_timestamp" : "1547740500000"
}
Yes, only small changes are required to make this work for multiple documents.
In $match stage, specify your $in query:
$match: {
"s_id": { $in : [ "30081993", "27061995" ] },
"frequency": 5,
}
In $project stage, rename s_id to _id, to ensure we keep the s_id associated with each document:
$project: {
_id: "$s_id",
valuesarray: {
$objectToArray: "$values"
}
}
In $group stage, group by _id (originally s_id), to ensure we correctly group the timestamps together before calculating $min/$max:
$group: {
"_id": "$_id",
"min_timestamp": {
$min: "$valuesarray.k"
},
"max_timestamp": {
$max: "$valuesarray.k"
}
}
Whole pipeline:
db.testdb.aggregate([
{
$match: {
"s_id": { $in : [ "30081993", "27061995" ] },
"frequency": 5,
}
},
{
$project: {
_id: "$s_id",
valuesarray: {
$objectToArray: "$values"
}
}
},
{
$unwind: "$valuesarray"
},
{
$group: {
"_id": "$_id",
"min_timestamp": {
$min: "$valuesarray.k"
},
"max_timestamp": {
$max: "$valuesarray.k"
}
}
}
]);

MongoDB aggregate array of objects together by object id and count occurences

I'm trying to figure out what I'm doing wrong, I have collected the following, "Subset of data", "Desired output"
This is how my data objects look
[{
"survey_answers": [
{
"id": "9ca01568e8dbb247", // As they are, this is the key to groupBy
"option_answer": 5, // Represent the index of the choosen option
"type": "OPINION_SCALE" // Opinion scales are 0-10 (meaning elleven options)
},
{
"id": "ba37125ec32b2a99",
"option_answer": 3,
"type": "LABELED_QUESTIONS" // Labeled questions are 0-x (they can change it from survey to survey)
}
],
"survey_id": "test"
},
{
"survey_answers": [
{
"id": "9ca01568e8dbb247",
"option_answer": 0,
"type": "OPINION_SCALE"
},
{
"id": "ba37125ec32b2a99",
"option_answer": 3,
"type": "LABELED_QUESTIONS"
}
],
"survey_id": "test"
}]
My desired output is:
[
{
id: '9ca01568e8dbb247'
results: [
{ _id: 5, count: 1 },
{ _id: 0, count: 1 }
]
},
{
id: 'ba37125ec32b2a99'
results: [
{ _id: 3, count: 2 }
]
}
]
Active query
Model.aggregate([
{
$match: {
'survey_id': survey_id
}
},
{
$unwind: "$survey_answers"
},
{
$group: {
_id: "$survey_answers.option_answer",
count: {
$sum: 1
}
}
}
])
Current output
[
{
"_id": 0,
"count": 1
},
{
"_id": 3,
"count": 2
},
{
"_id": 5,
"count": 1
}
]
I added your records to my db. Post that I tried your commands one by one.
$unwind results you similar to -
> db.survey.aggregate({$unwind: "$survey_answers"})
{ "_id" : ObjectId("5c3859e459875873b5e6ee3c"), "survey_answers" : { "id" : "9ca01568e8dbb247", "option_answer" : 5, "type" : "OPINION_SCALE" }, "survey_id" : "test" }
{ "_id" : ObjectId("5c3859e459875873b5e6ee3c"), "survey_answers" : { "id" : "ba37125ec32b2a99", "option_answer" : 3, "type" : "LABELED_QUESTIONS" }, "survey_id" : "test" }
{ "_id" : ObjectId("5c3859e459875873b5e6ee3d"), "survey_answers" : { "id" : "9ca01568e8dbb247", "option_answer" : 0, "type" : "OPINION_SCALE" }, "survey_id" : "test" }
{ "_id" : ObjectId("5c3859e459875873b5e6ee3d"), "survey_answers" : { "id" : "ba37125ec32b2a99", "option_answer" : 3, "type" : "LABELED_QUESTIONS" }, "survey_id" : "test" }
I am not adding code for match since that is okay in your query as well
The grouping would be -
> db.survey.aggregate({$unwind: "$survey_answers"},{$group: { _id: { 'optionAnswer': "$survey_answers.option_answer", 'id':"$survey_answers.id"}, count: { $sum: 1}}})
{ "_id" : { "optionAnswer" : 0, "id" : "9ca01568e8dbb247" }, "count" : 1 }
{ "_id" : { "optionAnswer" : 3, "id" : "ba37125ec32b2a99" }, "count" : 2 }
{ "_id" : { "optionAnswer" : 5, "id" : "9ca01568e8dbb247" }, "count" : 1 }
You can group on $survey_answers.id to bring it into projection.
The projection is what you're missing in your query -
> db.survey.aggregate({$unwind: "$survey_answers"},{$group: { _id: { 'optionAnswer': "$survey_answers.option_answer", 'id':'$survey_answers.id'}, count: { $sum: 1}}}, {$project : {answer: '$_id.optionAnswer', id: '$_id.id', count: '$count', _id:0}})
{ "answer" : 0, "id" : "9ca01568e8dbb247", "count" : 1 }
{ "answer" : 3, "id" : "ba37125ec32b2a99", "count" : 2 }
{ "answer" : 5, "id" : "9ca01568e8dbb247", "count" : 1 }
Further you can add a group on id and add results to a set. And your final query would be -
db.survey.aggregate(
{$unwind: "$survey_answers"},
{$group: {
_id: { 'optionAnswer': "$survey_answers.option_answer", 'id':'$survey_answers.id'},
count: { $sum: 1}
}},
{$project : {
answer: '$_id.optionAnswer',
id: '$_id.id',
count: '$count',
_id:0
}},
{$group: {
_id:{id:"$id"},
results: { $addToSet: {answer: "$answer", count: '$count'} }
}},
{$project : {
id: '$_id.id',
answer: '$results',
_id:0
}})
Hope this helps.

Count of MongoDB aggregation match results

I'm working with a MongoDB collection that has a lot of duplicate keys. I regularly do aggregation queries to find out what those duplicates are, so that I can dig in and find out what is and isn't different about them.
Unfortunately the database is huge and duplicates are often intentional. What I'd like to do is to find the count of keys that have duplicates, instead of printing a result with thousands of lines of output. Is this possible?
(Side Note: I do all of my querying through the shell, so solutions that don't require external tools or a lot of code would be preferred, but I understand that's not always possible.)
Example Records:
{ "_id" : 1, "type" : "example", "key" : "111111", "value" : "abc" }
{ "_id" : 2, "type" : "example", "key" : "222222", "value" : "def" }
{ "_id" : 3, "type" : "example", "key" : "222222", "value" : "ghi" }
{ "_id" : 4, "type" : "example", "key" : "333333", "value" : "jkl" }
{ "_id" : 5, "type" : "example", "key" : "333333", "value" : "mno" }
{ "_id" : 6, "type" : "example", "key" : "333333", "value" : "pqr" }
{ "_id" : 7, "type" : "example", "key" : "444444", "value" : "stu" }
{ "_id" : 8, "type" : "example", "key" : "444444", "value" : "vwx" }
{ "_id" : 9, "type" : "example", "key" : "444444", "value" : "yz1" }
{ "_id" : 10, "type" : "example", "key" : "444444", "value" : "234" }
Here is the query that I've been using to find duplicates based on key:
db.collection.aggregate([
{
$match: {
type: "example"
}
},
{
$group: {
_id: "$key",
count: {
$sum: 1
}
}
},
{
$match: {
count: {
$gt: 1
}
}
}
])
Which gives me an output of:
{
"_id": "222222",
"count": 2
},
{
"_id": "333333",
"count": 3
},
{
"_id": "444444",
"count": 4
}
The result I want to get instead:
3
You are almost there, just missing the last $count:
db.collection.aggregate([
{
$match: {
type: "example"
}
},
{
$group: {
_id: "$key",
count: {
$sum: 1
}
}
},
{
$match: {
count: {
$gt: 1
}
}
},
{
$count: "count"
}
])
Akrion's answer seems to be correct, but I can't test it because we're on an older version of MongoDB. A coworker gave me an alternative solution that works on 3.2 (not sure about other versions).
Adding .toArray() will convert the results to an array, and you can then get the size of the array using .length.
db.collection.aggregate([
{
$match: {
type: "example"
}
},
{
$group: {
_id: "$key",
count: {
$sum: 1
}
}
},
{
$match: {
count: {
$gt: 1
}
}
}
]).toArray().length

Group by reduced field depending on variable in mongodb

I have the following collection for messages:
{
"_id" : ObjectId("56214d5632001bae07a6e6b3"),
"sender_id" : 8,
"receiver_id" : 2,
"content" : "fdgfd",
"state" : 1,
"timestamp" : 1445023062899.0000000000000000
},
{
"_id" : ObjectId("56214d5c32001bae07a6e6b4"),
"sender_id" : 2,
"receiver_id" : 8,
"content" : "fasfa",
"state" : 1,
"timestamp" : 1445023068443.0000000000000000
},
{
"_id" : ObjectId("56214d8032001bae07a6e6b5"),
"sender_id" : 2,
"receiver_id" : 8,
"content" : "dfdsfds",
"state" : 1,
"timestamp" : 1445023104363.0000000000000000
},
{
"_id" : ObjectId("56214d8032001bae07a6e6b6"),
"sender_id" : 2,
"receiver_id" : 8,
"content" : "fdsf",
"state" : 1,
"timestamp" : 1445023104825.0000000000000000
},
{
"_id" : ObjectId("56214d8132001bae07a6e6b7"),
"sender_id" : 2,
"receiver_id" : 8,
"content" : "sfsdfs",
"state" : 1,
"timestamp" : 1445023105436.0000000000000000
},
{
"_id" : ObjectId("56214d8132001bae07a6e6b8"),
"sender_id" : 2,
"receiver_id" : 8,
"content" : "f",
"state" : 1,
"timestamp" : 1445023105963.0000000000000000
},
{
"_id" : ObjectId("56214d8432001bae07a6e6b9"),
"sender_id" : 2,
"receiver_id" : 8,
"content" : "qwqwqwq",
"state" : 1,
"timestamp" : 1445023108202.0000000000000000
},
{
"_id" : ObjectId("56214db032001bae07a6e6ba"),
"sender_id" : 9902,
"receiver_id" : 2,
"content" : "fsafa",
"state" : 1,
"timestamp" : 1445023152297.0000000000000000
}
I'm trying to get all unique users ids that had been messaging with user 2, along with the last content message. So the result should be:
[ { user: 8, lastContent: "qwqwqwq" }, { user: 9902, lastContent: "fsafa" } ]
By now, I have the following code:
db.getCollection('messenger').group({
keyf: function(doc) {
return { user: doc.user };
},
cond: {
$or : [
{ sender_id : 2 },
{ receiver_id : 2 }
]
},
reduce: function( curr, result ) {
result.user = (curr.sender_id == 2 ? curr.receiver_id : curr.sender_id);
result.content = curr.content;
},
initial: { } })
But I only get the last id. The result:
{
"0" : {
"user" : 9902.0000000000000000,
"content" : "fsafa"
} }
Can anyone help me with this?
You need to use the .aggregate() method. You need to reduce the size of documents in the pipeline using the $match operator which filter out all documents where the receiver_id is not equal to 2. After that you need to $sort your document by timestamp in descending order this will help us get the content of last message sent. Now comes the $group stage where you group your documents and use the $addToSet operator which returns array of distinct sender_id and distinct receiver_id and the $last operator to get the last message content. Now to get the user_ids we need union of distinct sender_id and receiver_id which we can get after $projection using the $setUnion operator.
db.messenger.aggregate([
{ "$match": {
"$or": [
{ "sender_id": 2 },
{ "receiver_id": 2 }
]
}},
{ "$sort": { "timestamp": 1 } },
{ "$group": {
"_id": null,
"receiver_id": {
"$addToSet": { "$receiver_id" }
},
"sender_id": {
"$addToSet": { "$sender_id" }
},
"lastContent": { "$last": "$content" }
}},
{ "$project": {
"_id": 0,
"lastContent": 1,
"user_ids": {
"$setUnion": [
"$sender_id",
"$receiver_id"
]
}
}}
])
Which returns:
{ "lastContent" : "fsafa", "user_ids" : [ 9902, 2, 8 ] }
Now if what you want is distinct user alongside their last content message with user 2 then here it is:
db.messenger.aggregate([
{ "$match": {
"$or": [
{ "sender_id": 2 },
{ "receiver_id": 2 }
]
}},
{ "$sort": { "timestamp": 1 } },
{ "$group": {
"_id": {
"sender": "$sender_id",
"receiver": "$receiver_id"
},
"lastContent": {
"$last": "$content"
},
"timestamp": { "$last": "$timestamp" },
"sender": { "$addToSet": "$sender_id" },
"receiver": { "$addToSet": "$receiver_id" }
}},
{ "$project": {
"_id": 0,
"user": {
"$setDifference": [
{ "$setUnion": [ "$sender", "$receiver" ] },
[ 2 ]
]
},
"lastContent": 1,
"timestamp": 1
}},
{ "$unwind": "$user" },
{ "$sort": { "timestamp": 1 } },
{ "$group": {
"_id": "$user",
"lastContent": { "$last": "$lastContent" }
} }
])
Which yields:
{ "_id" : 9902, "lastContent" : "fsafa" }
{ "_id" : 8, "lastContent" : "qwqwqwq" }