Mongodb grouby by and sum and get media - mongodb

I have this collection in my database:
{ "IdUser" : "1", "IdItem" : "1" },
{ "IdUser" : "1", "IdItem" : "2" },
{ "IdUser" : "1", "IdItem" : "3" },
{ "IdUser" : "2", "IdItem" : "4" },
{ "IdUser" : "2", "IdItem" : "5" },
{ "IdUser" : "4", "IdItem" : "6" },
{ "IdUser" : "5", "IdItem" : "7" }
How can I obtain this result:
Users with one item: 2
Users with two items: 1
Users with three items: 1

You need to first $group your documents by IdUser then count the number of time each IdUser appear in your collection using the $sum accumulator operator. This allows you in the next stage to group your documents by "count" and return the count for "user" with same number of "items".
db.items.aggregate([
{ "$group": {
"_id": "$IdUser",
"count": { "$sum": 1 }
}},
{ "$group": {
"_id": "$count",
"nUsers": { "$sum": 1 }
}}
])

Using the aggregate() method will give you the desired result though the output documents would be different in that instead of having a key/value pair, you have two different fields with values that show the number of users and their respective
item count. The following aggregation pipeline explains this:
db.collection.aggregate([
{
"$group": {
"_id": "$IdUser",
"count": {
"$sum": { "$cond": [{ "$gt": [ "$IdItem", null ] }, 1, 0 ] }
}
}
},
{
"$group": {
"_id": "$count",
"users": { "$push": "$_id" }
}
},
{
"$project": {
"_id": 0,
"number_of_items": "$_id",
"number_of_users": { "$size": "$users" }
}
}
])
In the above, you $group all the documents by the user key to get their counts, taking into consideration documents that may not have the item field, which would be discounted in the aggregate. A further $group operation is necessary to then get the users per count, in the form of an array.
The last step in the pipeline $project serves to reshape the final output so that you get the following output (with the sample documents supplied with the question):
{ "number_of_items" : 1, "number_of_users" : 2 }
{ "number_of_items" : 3, "number_of_users" : 1 }
{ "number_of_items" : 2, "number_of_users" : 1 }

Related

Sum array number values across multiple documents

I have Mongo documents which have array number values in order (it's by day) and I want to sum the same values across multiple documents for each position grouped by field outside of the array.
{"_id" : "1",
"group" : "A",
"value_list" : [1,2,3,4,5,6,7]
},
{"_id" : "2",
"group" : "B",
"value_list" : [10,20,30,40,50,60,70]
},
{"_id" : "3",
"group" : "A",
"value_list" : [1,2,3,4,5,6,7]
},
{"_id" : "4",
"group" : "B",
"value_list" : [10,20,30,40,50,60,70]
}
So the results I'm after is listed below.
There are two group A documents above and at position 1 of the value_list array, both documents have the value of 1. so 1+1=2. Position 2 the value is 2 in both documents so 2+2=4, etc.
There are two group B documents above and at position 1 of the value_list array, both documents have the value of 10. so 10+10=20. Position 2 the value is 20 in both documents so 20+20=40, etc.
{"_id" : "30",
"group" : "A",
"value_list" : [2,4,6,8,10,12,14]
},
{"_id" : "30",
"group" : "A",
"value_list" : [20,40,60,80,100,120,140]
}
How would I do this using Mongo Script? Thanks, Matt
Certainly the most "scalable" way is to use the includeArrayIndex option of $unwind in order to track the positions and then $sum the "unwound" combinations, before adding back into array format:
db.getCollection('test').aggregate([
{ "$unwind": { "path": "$value_list", "includeArrayIndex": "index" } },
{ "$group": {
"_id": {
"group": "$group",
"index": "$index"
},
"value_list": { "$sum": "$value_list" }
}},
{ "$sort": { "_id": 1 } },
{ "$group": {
"_id": "$_id.group",
"value_list": { "$push": "$value_list" }
}},
{ "$sort": { "_id": 1 } }
])
Note you need to $sort after the first $group in order to maintain the array positions.
If you can get away with it, you could also apply all arrays into $reduce:
db.getCollection('test').aggregate([
{ "$group": {
"_id": "$group",
"value_list": { "$push": "$value_list" }
}},
{ "$addFields": {
"value_list": {
"$reduce": {
"input": "$value_list",
"initialValue": [],
"in": {
"$map": {
"input": {
"$zip": {
"inputs": ["$$this", "$$value"],
"useLongestLength": true,
}
},
"in": { "$sum": "$$this"}
}
}
}
}
}},
{ "$sort": { "_id": 1 } }
])
Essentially you create an "array of arrays" using the initial $push, which you process with $reduce. The $zip does a "pairwise" assignment per element, which are then added together at each position during $map using $sum.
While a bit more efficient, it's not really practical for large data as you would probably break the BSON limit by adding all grouped "arrays" into a single array on the grouping, before you "reduce" it.
Either method produces the same result:
/* 1 */
{
"_id" : "A",
"value_list" : [
2.0,
4.0,
6.0,
8.0,
10.0,
12.0,
14.0
]
}
/* 2 */
{
"_id" : "B",
"value_list" : [
20.0,
40.0,
60.0,
80.0,
100.0,
120.0,
140.0
]
}

Find Total based on group by of two mongo field

i have collection data like this -
{
"user_id" : "1",
"branch_id" : "1",
"total" : 100,
},
{
"user_id" : "1",
"branch_id" : "1",
"total" : 200
},
{
"user_id" : "1",
"branch_id" : "3",
"total" : 1400
},
{
"user_id" : "2",
"branch_id" : "1",
"total" : 100
},
{
"user_id" : "2",
"branch_id" : "1",
"total" : 100
},
I am looking to get output in the below format -
[
{
"user_id":"1",
"branch_id":"1",
"grand_total":"300"
},
{
"user_id":"1",
"branch_id":"3",
"grand_total":"1400"
},
{
"user_id":"2",
"branch_id":"1",
"grand_total":"200"
}
]
I have tried a mongo aggregate query, but the query gives output as undefined.
Basically I need to get per user wise per branch wise the total points he has earned.
Here is what I have tried but not working -
Collection.aggregate(
{
"$group": {
"_id": "$user_id",
"nameCount": { "$sum": 1 },
"branch_id": {
"$sum": {
"$cond": [ {"$branch_id":{"$ne":null}} ]
}
}
}
},
{
"$project": {
"_id": 0,
"name": "$_id",
"nameCount": 1,
"branch_id":1
}
}
);
Please help.
Your aggregation pipeline has to look like this:
{
"$group": {
"_id": {
user_id: "$user_id",
branch_id: "$branch_id"
},
"grand_total": {
"$sum": "$total"
},
}
}, {
"$project": {
"_id": 0,
"user_id": "$_id.user_id",
"branch_id": "$_id.branch_id",
"total": "$grand_total"
}
}
Inside your _id field in your "$group" pipeline you add the fields that you want to group your documents by. If you only want to group by one field you can write it as follows:
{"$group": {
"_id": "$user_id"
}
}
If you have multiple fields you want to group by (like it seems in your case) then you write it as follows:
{"$group": {
"_id": {
user_id: "$user_id",
branch_id: "$branch_id"
}
}
}
Every aggregation pipeline changes your document. So, in your $group if you call the sum of all totals like that "grand_total"
"grand_total": {
"$sum": "$total"
}
then in your $project pipeline that field total doesn't exist anymore. But instead we created a new field (grand_total) that is the sum.

Mongodb group by and sum and get media with Map

I have these collections in my database:
Items:
{ "IdUser" : "1", "IdItem" : "1" },
{ "IdUser" : "1", "IdItem" : "2" },
{ "IdUser" : "1", "IdItem" : "3" },
{ "IdUser" : "2", "IdItem" : "4" },
{ "IdUser" : "2", "IdItem" : "5" },
{ "IdUser" : "4", "IdItem" : "6" },
{ "IdUser" : "5", "IdItem" : "7" }
Users
{ "_id" : "1", "DateRegister" : ISODate("2016-03-29T22:00:38.764+0000") },
{ "_id" : "2", "DateRegister" : ISODate("2014-03-29T22:00:38.764+0000") },
{ "_id" : "2", "DateRegister" : ISODate("2015-02-29T22:00:38.764+0000") },
{ "_id" : "4", "DateRegister" : ISODate("2013-01-29T22:00:38.764+0000") },
{ "_id" : "5", "DateRegister" : ISODate("2016-04-29T22:00:38.764+0000") }
How can I obtain this result but FILTERED with users registered after 2015:
Users with one item: 2
Users with two items: 1
Users with three items: 1
I have tried with that, but I don't know how to filter... Thanks!
db.collection.aggregate([
{
"$group": {
"_id": "$IdUser",
"count": {
"$sum": { "$cond": [{ "$gt": [ "$IdItem", null ] }, 1, 0 ] }
}
}
},
{
"$group": {
"_id": "$count",
"users": { "$push": "$_id" }
}
},
{
"$project": {
"_id": 0,
"number_of_items": "$_id",
"number_of_users": { "$size": "$users" }
}
}
])
You may want to utilize the $lookup operator to perform a join of the items collection with the users collection and then do a $match filter on the DateRegistered field before piping the main grouping operations.
Following this example + the links herein to the documentation will give you an idea:
db.items.aggregate([
{
"$lookup": {
"from": "users",
"localField": "IdUser",
"foreignField": "_id",
"as": "user"
}
},
{ "$match": { "user.DateRegister": { "$gt": new Date(2015, 11, 31) } } },
{
"$group": {
"_id": "$IdUser",
"count": {
"$sum": { "$cond": [{ "$gt": [ "$IdItem", null ] }, 1, 0 ] }
}
}
},
{
"$group": {
"_id": "$count",
"users": { "$push": "$_id" }
}
},
{
"$project": {
"_id": 0,
"number_of_items": "$_id",
"number_of_users": { "$size": "$users" }
}
}
])
In the event that your MongoDB server does not support the $lookup operator, you will then need a workaround where you split the operations on the different collections i.e.
get a list of user id's that match the given date range criteria, this could be done with the distinct() method on the users collection with the date query option.
use that list in the items collection aggregation pipeline within the $match operator initial step.
The following demonstrates this:
// use distinct to get the user id's list
var userIds = db.users.distinct("_id", { "DateRegister": { "$gt": new Date(2015, 11, 31) } })
// perform your aggregation with a filtered collection using the list from the above operations
db.items.aggregate([
{ "$match": { "IdUser": { "$in": userIds } } },
{
"$group": {
"_id": "$IdUser",
"count": {
"$sum": { "$cond": [{ "$gt": [ "$IdItem", null ] }, 1, 0 ] }
}
}
},
{
"$group": {
"_id": "$count",
"users": { "$push": "$_id" }
}
},
{
"$project": {
"_id": 0,
"number_of_items": "$_id",
"number_of_users": { "$size": "$users" }
}
}
])

How to group by different fields

I want to find all users named 'Hans' and aggregate their 'age' and number of 'childs' by grouping them.
Assuming I have following in my database 'users'.
{
"_id" : "01",
"user" : "Hans",
"age" : "50"
"childs" : "2"
}
{
"_id" : "02",
"user" : "Hans",
"age" : "40"
"childs" : "2"
}
{
"_id" : "03",
"user" : "Fritz",
"age" : "40"
"childs" : "2"
}
{
"_id" : "04",
"user" : "Hans",
"age" : "40"
"childs" : "1"
}
The result should be something like this:
"result" :
[
{
"age" :
[
{
"value" : "50",
"count" : "1"
},
{
"value" : "40",
"count" : "2"
}
]
},
{
"childs" :
[
{
"value" : "2",
"count" : "2"
},
{
"value" : "1",
"count" : "1"
}
]
}
]
How can I achieve this?
This should almost be a MongoDB FAQ, mostly because it is a real example concept of how you should be altering your thinking from SQL processing and embracing what engines like MongoDB do.
The basic principle here is "MongoDB does not do joins". Any way of "envisioning" how you would construct SQL to do this essentially requires a "join" operation. The typical form is "UNION" which is in fact a "join".
So how to do it under a different paradigm? Well first, let's approach how not to do it and understand the reasons why. Even if of course it will work for your very small sample:
The Hard Way
db.docs.aggregate([
{ "$group": {
"_id": null,
"age": { "$push": "$age" },
"childs": { "$push": "$childs" }
}},
{ "$unwind": "$age" },
{ "$group": {
"_id": "$age",
"count": { "$sum": 1 },
"childs": { "$first": "$childs" }
}},
{ "$sort": { "_id": -1 } },
{ "$group": {
"_id": null,
"age": { "$push": {
"value": "$_id",
"count": "$count"
}},
"childs": { "$first": "$childs" }
}},
{ "$unwind": "$childs" },
{ "$group": {
"_id": "$childs",
"count": { "$sum": 1 },
"age": { "$first": "$age" }
}},
{ "$sort": { "_id": -1 } },
{ "$group": {
"_id": null,
"age": { "$first": "$age" },
"childs": { "$push": {
"value": "$_id",
"count": "$count"
}}
}}
])
That will give you a result like this:
{
"_id" : null,
"age" : [
{
"value" : "50",
"count" : 1
},
{
"value" : "40",
"count" : 3
}
],
"childs" : [
{
"value" : "2",
"count" : 3
},
{
"value" : "1",
"count" : 1
}
]
}
So why is this bad? The main problem should be apparent in the very first pipeline stage:
{ "$group": {
"_id": null,
"age": { "$push": "$age" },
"childs": { "$push": "$childs" }
}},
What we asked to do here is group up everything in the collection for the values we want and $push those results into an array. When things are small then this works, but real world collections would result in this "single document" in the pipeline that exceeds the 16MB BSON limit that is allowed. That is what is bad.
The rest of the logic follows the natural course by working with each array. But of course real world scenarios would almost always make this untenable.
You could avoid this somewhat, by doing things like "duplicating" the documents to be of "type" "age or "child" and grouping documents individually by type. But it's all a bit to "over complex" and not a solid way of doing things.
The natural response is "what about a UNION?", but since MongoDB does not do the "join" then how to approach that?
A Better Way ( aka A New Hope )
Your best approach here both architecturally and performance wise is to simply submit "both" queries ( yes two ) in "parallel" to the server via your client API. As the results are received you then "combine" them into a single response you can then send back as a source of data to your eventual "client" application.
Different languages have different approaches to this, but the general case is to look for an "asynchronous processing" API that allows you to do this in tandem.
My example purpose here uses node.js as the "asynchronous" side is basically "built in" and reasonably intuitive to follow. The "combination" side of things can be any type of "hash/map/dict" table implementation, just doing it the simple way for example only:
var async = require('async'),
MongoClient = require('mongodb');
MongoClient.connect('mongodb://localhost/test',function(err,db) {
var collection = db.collection('docs');
async.parallel(
[
function(callback) {
collection.aggregate(
[
{ "$group": {
"_id": "$age",
"type": { "$first": { "$literal": "age" } },
"count": { "$sum": 1 }
}},
{ "$sort": { "_id": -1 } }
],
callback
);
},
function(callback) {
collection.aggregate(
[
{ "$group": {
"_id": "$childs",
"type": { "$first": { "$literal": "childs" } },
"count": { "$sum": 1 }
}},
{ "$sort": { "_id": -1 } }
],
callback
);
}
],
function(err,results) {
if (err) throw err;
var response = {};
results.forEach(function(res) {
res.forEach(function(doc) {
if ( !response.hasOwnProperty(doc.type) )
response[doc.type] = [];
response[doc.type].push({
"value": doc._id,
"count": doc.count
});
});
});
console.log( JSON.stringify( response, null, 2 ) );
}
);
});
Which gives the cute result:
{
"age": [
{
"value": "50",
"count": 1
},
{
"value": "40",
"count": 3
}
],
"childs": [
{
"value": "2",
"count": 3
},
{
"value": "1",
"count": 1
}
]
}
So the key thing to note here is that the "separate" aggregation statements themselves are actually quite simple. The only thing you face is combining those in your final result. There are many approaches to "combining", particularly to deal with large results from each of the queries, but this is the basic example of the execution model.
Key points here.
Shuffling data in the aggregation pipeline is possible but not performant for large data sets.
Use a language implementation and API that support "parallel" and "asynchronous" execution so you can "load up" all or "most" of your operations at once.
The API should support some method of "combination" or otherwise allow a separate "stream" write to process each result set received into one.
Forget about the SQL way. The NoSQL way delegates the processing of such things as "joins" to your "data logic layer", which is what contains the code as shown here. It does it this way because it is scalable to very large datasets. It is rather the job of your "data logic" handling nodes in large applications to deliver this to the end API.
This is fast compared to any other form of "wrangling" I could possibly describe. Part of "NoSQL" thinking is to "Unlearn what you have learned" and look at things a different way. And if that way doesn't perform better, then stick with the SQL approach for storage and query.
That's why alternatives exist.
That was a tough one!
First, the bare solution:
db.test.aggregate([
{ "$match": { "user": "Hans" } },
// duplicate each document: one for "age", the other for "childs"
{ $project: { age: "$age", childs: "$childs",
data: {$literal: ["age", "childs"]}}},
{ $unwind: "$data" },
// pivot data to something like { data: "age", value: "40" }
{ $project: { data: "$data",
value: {$cond: [{$eq: ["$data", "age"]},
"$age",
"$childs"]} }},
// Group by data type, and count
{ $group: { _id: {data: "$data", value: "$value" },
count: { $sum: 1 },
value: {$first: "$value"} }},
// aggregate values in an array for each independant (type,value) pair
{ $group: { _id: "$_id.data", values: { $push: { count: "$count", value: "$value" }} }} ,
// project value to the correctly name field
{ $project: { result: {$cond: [{$eq: ["$_id", "age"]},
{age: "$values" },
{childs: "$values"}]} }},
// group all data in the result array, and remove unneeded `_id` field
{ $group: { _id: null, result: { $push: "$result" }}},
{ $project: { _id: 0, result: 1}}
])
Producing:
{
"result" : [
{
"age" : [
{
"count" : 3,
"value" : "40"
},
{
"count" : 1,
"value" : "50"
}
]
},
{
"childs" : [
{
"count" : 1,
"value" : "1"
},
{
"count" : 3,
"value" : "2"
}
]
}
]
}
And now, for some explanations:
One of the major issues here is that each incoming document has to be part of two different sums. I solved that by adding a literal array ["age", "childs"] to your documents, and then unwinding them by that array. That way, each document will be presented twice in the later stage.
Once that done, to ease processing, I change the data representation to something much more manageable like { data: "age", value: "40" }
The following steps will perform the data aggregation per-se. Up to the third $project step that will map the value fields to the corresponding age or childs field.
The final two steps will simply wrap the two documents in one, removing the unneeded _id field.
Pfff!

Aggregate Query in Mongodb returns specific field

Document Sample:
{
"_id" : ObjectId("53329dfgg43771e49538b4567"),
"u" : {
"_id" : ObjectId("532a435gs4c771edb168c1bd7"),
"n" : "Salman khan",
"e" : "salman#gmail.com"
},
"ps" : 0,
"os" : 1,
"rs" : 0,
"cd" : 1395685800,
"ud" : 0
}
Query:
db.collectiontmp.aggregate([
{$match: {os:1}},
{$project : { name:{$toUpper:"$u.e"} , _id:0 } },
{$group: { _id: "$u._id",total: {$sum:1} }},
{$sort: {total: -1}}, { $limit: 10 }
]);
I need following things from the above query:
Group by u._id
Returns total number of records and email from the record, as shown below:
{
"result":
[
{
"email": "",
"total": ""
},
{
"email": "",
"total": ""
}
],
"ok":
1
}
The first thing you are doing wrong here is not understanding how $project is intended to work. Pipeline stages such as $project and $group will only output the fields that are "explicitly" identified. So only the fields you say to output will be available to the following pipeline stages.
Specifically here you "project" only part of the "u" field in your document and you therefore removed the other data from being available. The only present field here now is "name", which is the one you "projected".
Perhaps it was really your intention to do something like this:
db.collectiontmp.aggregate([
{ "$group": {
"_id": {
"_id": "$u._id",
"email": { "$toUpper": "$u.e" }
},
"total": { "$sum": 1 },
}},
{ "$project": {
"_id": 0,
"email": "$_id.email",
"total": 1
}},
{ "$sort": { "total": -1 } },
{ "$limit": 10 }
])
Or even:
db.collectiontmp.aggregate([
{ "$group": {
"_id": "$u._id",
"email": { "$first": { "$toUpper": "$u.e" } }
"total": { "$sum": 1 },
}},
{ "$project": {
"_id": 0,
"email": 1,
"total": 1
}},
{ "$sort": { "total": -1 } },
{ "$limit": 10 }
])
That gives you the sort of output you are looking for.
Remember that as this is a "pipeline", then only the "output" from a prior stage is available to the "next" stage. There is no "global" concept of the document as this is not a declarative statement such as in SQL, but a "pipeline".
So think Unix pipe "|" command, or otherwise look that up. Then your thinking will fall into place.