I'm wondering how to perform a kind of union in an aggregate in MongoDB. Let's imaging the following document in a collection (the structure is for the sake of the example) :
{
linkedIn: {
people : [
{
name : 'Fred'
},
{
name : 'Matilda'
}
]
},
twitter: {
people : [
{
name : 'Hanna'
},
{
name : 'Walter'
}
]
}
}
How to make an aggregate that returns the union of the people in twitter and linkedIn ?
{
{ name :'Fred', source : 'LinkedIn'},
{ name :'Matilda', source : 'LinkedIn'},
{ name :'Hanna', source : 'Twitter'},
{ name :'Walter', source : 'Twitter'},
}
There are a couple of approaches to this that you can use the aggregate method for
db.collection.aggregate([
// Assign an array of constants to each document
{ "$project": {
"linkedIn": 1,
"twitter": 1,
"source": { "$cond": [1, ["linkedIn", "twitter"],0 ] }
}},
// Unwind the array
{ "$unwind": "$source" },
// Conditionally push the fields based on the matching constant
{ "$group": {
"_id": "$_id",
"data": { "$push": {
"$cond": [
{ "$eq": [ "$source", "linkedIn" ] },
{ "source": "$source", "people": "$linkedIn.people" },
{ "source": "$source", "people": "$twitter.people" }
]
}}
}},
// Unwind that array
{ "$unwind": "$data" },
// Unwind the underlying people array
{ "$unwind": "$data.people" },
// Project the required fields
{ "$project": {
"_id": 0,
"name": "$data.people.name",
"source": "$data.source"
}}
])
Or with a different approach using some operators from MongoDB 2.6:
db.people.aggregate([
// Unwind the "linkedIn" people
{ "$unwind": "$linkedIn.people" },
// Tag their source and re-group the array
{ "$group": {
"_id": "$_id",
"linkedIn": { "$push": {
"name": "$linkedIn.people.name",
"source": { "$literal": "linkedIn" }
}},
"twitter": { "$first": "$twitter" }
}},
// Unwind the "twitter" people
{ "$unwind": "$twitter.people" },
// Tag their source and re-group the array
{ "$group": {
"_id": "$_id",
"linkedIn": { "$first": "$linkedIn" },
"twitter": { "$push": {
"name": "$twitter.people.name",
"source": { "$literal": "twitter" }
}}
}},
// Merge the sets with "$setUnion"
{ "$project": {
"data": { "$setUnion": [ "$twitter", "$linkedIn" ] }
}},
// Unwind the union array
{ "$unwind": "$data" },
// Project the fields
{ "$project": {
"_id": 0,
"name": "$data.name",
"source": "$data.source"
}}
])
And of course if you simply did not care what the source was:
db.collection.aggregate([
// Union the two arrays
{ "$project": {
"data": { "$setUnion": [
"$linkedIn.people",
"$twitter.people"
]}
}},
// Unwind the union array
{ "$unwind": "$data" },
// Project the fields
{ "$project": {
"_id": 0,
"name": "$data.name",
}}
])
Not sure if using aggregate is recommended over a map-reduce for that kind of operation but the following is doing what you're asking for (dunno if $const can be used with no issue at all in the .aggregate() function) :
aggregate([
{ $project: { linkedIn: '$linkedIn', twitter: '$twitter', idx: { $const: [0,1] }}},
{ $unwind: '$idx' },
{ $group: { _id : '$_id', data: { $push: { $cond:[ {$eq:['$idx', 0]}, { source: {$const: 'LinkedIn'}, people: '$linkedIn.people' } , { source: {$const: 'Twitter'}, people: '$twitter.people' } ] }}}},
{ $unwind: '$data'},
{ $unwind: '$data.people'},
{ $project: { _id: 0, name: '$data.people.name', source: '$data.source' }}
])
Related
I have a collection of users where each document has following structure:
{
"_id": "<id>",
"login": "xxx",
"solved": [
{
"problem": "<problemID>",
"points": 10
},
...
]
}
The field solved may be empty or contain arbitrary many subdocuments. My goal is to get a list of users together with the total score (sum of points) where users that haven't solved any problem yet will be assigned total score of 0. Is this possible to do this with a single query (ideally using aggregation framework)?
I was trying to use following query in aggregation framework:
{ "$group": {
"_id": "$_id",
"login": { "$first": "$login" },
"solved": { "$addToSet": { "points": 0 } }
} }
{ "$unwind": "$solved" }
{ "$group": {
"_id": "$_id",
"login": { "$first": "$login" },
"solved": { "$sum": "$solved.points" }
} }
However I am getting following error:
exception: The top-level _id field is the only field currently supported for exclusion
Thank you in advance
With MongoDB 3.2 version and newer, the $unwind operator now has some options where in particular the preserveNullAndEmptyArrays option will solve this.
If this option is set to true and if the path is null, missing, or an empty array, $unwind outputs the document. If false, $unwind does not output a document if the path is null, missing, or an empty array. In your case, set it to true:
db.collection.aggregate([
{ "$unwind": {
"path": "$solved",
"preserveNullAndEmptyArrays": true
} },
{ "$group": {
"_id": "$_id",
"login": { "$first": "$login" },
"solved": { "$sum": "$solved.points" }
} }
])
Here is the solution - it assumes that the field "solved" is either absent, is equal to null or has an array of problems and scores solved. The case it does not handle is "solved" being an empty array - although that would be a simple additional adjustment you could add.
project = {$project : {
"s" : {
"$ifNull" : [
"$solved",
[
{
"points" : 0
}
]
]
},
"login" : 1
}
};
unwind={$unwind:"$s"};
group= { "$group" : {
"_id" : "$_id",
"login" : {
"$first" : "$login"
},
"score" : {
"$sum" : "$s.points"
}
}
}
db.students.aggregate( [ project, unwind, group ] );
$lookup then $unwind inside look up array and that could be empty
let posts = await Post.aggregate<ActivityDoc>([
{
$match: {
_id: new mongoose.Types.ObjectId(req.params.id),
},
},
{
$lookup: {
from: 'users',
localField: 'user',
foreignField: '_id',
as: 'user',
},
},
{
$unwind: '$user',
},
{
$unwind: {
path: '$user.follower',
preserveNullAndEmptyArrays: true,
},
},
{
$match: {
$or: [
{
$and: [
{
'privacy.mode': {
$eq: PrivacyMode.EveryOne,
},
},
],
},
{
$and: [
{
'privacy.mode': {
$eq: PrivacyMode.MyCircle,
},
},
{
'user.follower.id': {
$eq: req.currentUser?.id,
},
},
],
},
],
},
},
]);
This is locations collection data.
{
_id: "1",
location: "loc1",
sublocations: [
{
_id: 2,
sublocation: "subloc1",
},
{
_id: 3,
sublocation: "subloc2",
}
]
},
{
_id: "4",
location: "loc2",
sublocations: [
{
_id: 5,
sublocation: "subloc1",
},
{
_id: 6,
sublocation: "subloc2",
}
]
}
This is products collection data
{
_id: "1",
product: "product1",
prices: [
{
_id: 2,
sublocationid: 2, //ObjectId of object in sublocations array
price: 500
},
{
_id: 3,
sublocationid: 5, //ObjectId of object in sublocations array
price: 200
}
]
}
Now I need to get the sublocation in product schema in the prices array. Expected result is as below.
{
_id: "1",
product: "product1",
prices: [
{
_id: 2,
sublocationid: 3,
sublocation: "subloc2",
price: 500
},
{
_id: 3,
sublocationid: 5,
sublocation: "subloc1"
price: 200
}
]
}
To achieve it, I did it like in the following way.
First, performing aggregation on locations collection - $unwind the sublocations array and store the $out in the new collection.
Second, perform aggregation on 'products' collection - $unwind the prices, $lookup the sublocationid from the new collection and $group them.
Third, after getting data delete the data of new collection.
Is there any other simplified way? Please let me know if there is any.
If you want to stick with 3.4 version, you can try this query:
db.products.aggregate([
{
$unwind: {
"path": "$prices"
}
},
{
$lookup: {
"from": "locations",
"localField": "prices.sublocationid",
"foreignField": "sublocations._id",
"as": "locations"
}
},
{
$unwind: {
"path": "$locations"
}
},
{
$unwind: {
"path": "$locations.sublocations"
}
},
{
$addFields: {
"keep": {
"$eq": [
"$prices.sublocationid",
"$locations.sublocations._id"
]
}
}
},
{
$match: {
"keep": true
}
},
{
$addFields: {
"price": {
"_id": "$prices._id",
"sublocationid": "$prices.sublocationid",
"sublocation": "$locations.sublocations.sublocation",
"price": "$prices.price"
}
}
},
{
$group: {
"_id": "$_id",
"product": { "$first": "$product" },
"prices": { "$addToSet": "$price" }
}
}
]);
It's not as nice as 3.6 version though, because of a higher memory consumption.
You can try below aggregation query in 3.6 version.
Since both local field and foreign field are array you have to $unwind both to do equality comparison.
For this you will have to use new $lookup syntax.
$match with $expr provides comparsion between document fields to look up the location's sublocation document for each product's sublocation id.
$project to project the matching sublocation doc.
$addFields with $arrayElemAt to convert the looked up sublocation array into a document.
$group to push all prices with matching sublocation's document for each product.
db.products.aggregate[
{
"$unwind": "$prices"
},
{
"$lookup": {
"from": "locations",
"let": {
"prices": "$prices"
},
"pipeline": [
{
"$unwind": "$sublocations"
},
{
"$match": {
"$expr": [
"$$prices.sublocationid",
"$sublocations._id"
]
}
},
{
"$project": {
"sublocations": 1,
"_id": 0
}
}
],
"as": "prices.sublocations"
}
},
{
"$addFields": {
"prices.sublocations": {
"$arrayElemAt": [
"$prices.sublocations",
0
]
}
}
},
{
"$group": {
"_id": "$_id",
"product": {
"$first": "$product"
},
"prices": {
"$push": "$prices"
}
}
}
])
I have a collections with documents structured like below:
{
carrier: "abc",
flightNumber: 123,
dates: [
ISODate("2015-01-01T00:00:00Z"),
ISODate("2015-01-02T00:00:00Z"),
ISODate("2015-01-03T00:00:00Z")
]
}
I would like to search the collection to see if there are any documents with the same carrier and flightNumber that also have dates in the dates array that over lap. For example:
{
carrier: "abc",
flightNumber: 123,
dates: [
ISODate("2015-01-01T00:00:00Z"),
ISODate("2015-01-02T00:00:00Z"),
ISODate("2015-01-03T00:00:00Z")
]
},
{
carrier: "abc",
flightNumber: 123,
dates: [
ISODate("2015-01-03T00:00:00Z"),
ISODate("2015-01-04T00:00:00Z"),
ISODate("2015-01-05T00:00:00Z")
]
}
If the above records were present in the collection I would like to return them because they both have carrier: abc, flightNumber: 123 and they also have the date ISODate("2015-01-03T00:00:00Z") in the dates array. If this date were not present in the second document then neither should be returned.
Typically I would do this by grouping and counting like below:
db.flights.aggregate([
{
$group: {
_id: { carrier: "$carrier", flightNumber: "$flightNumber" },
uniqueIds: { $addToSet: "$_id" },
count: { $sum: 1 }
}
},
{
$match: {
count: { $gt: 1 }
}
}
])
But I'm not sure how I could modify this to look for array overlap. Can anyone suggest how to achieve this?
You $unwind the array if you want to look at the contents as "grouped" within them:
db.flights.aggregate([
{ "$unwind": "$dates" },
{ "$group": {
"_id": { "carrier": "$carrier", "flightnumber": "$flightnumber", "date": "$dates" },
"count": { "$sum": 1 },
"_ids": { "$addToSet": "$_id" }
}},
{ "$match": { "count": { "$gt": 1 } } },
{ "$unwind": "$_ids" },
{ "$group": { "_id": "$_ids" } }
])
That does in fact tell you which documents where the "overlap" resides, because the "same dates" along with the other same grouping key values that you are concerned about have a "count" which occurs more than once. Indicating the overlap.
Anything after the $match is really just for "presentation" as there is no point reporting the same _id value for multiple overlaps if you just want to see the overlaps. In fact if you want to see them together it would probably be best to leave the "grouped set" alone.
Now you could add a $lookup to that if retrieving the actual documents was important to you:
db.flights.aggregate([
{ "$unwind": "$dates" },
{ "$group": {
"_id": { "carrier": "$carrier", "flightnumber": "$flightnumber", "date": "$dates" },
"count": { "$sum": 1 },
"_ids": { "$addToSet": "$_id" }
}},
{ "$match": { "count": { "$gt": 1 } } },
{ "$unwind": "$_ids" },
{ "$group": { "_id": "$_ids" } },
}},
{ "$lookup": {
"from": "flights",
"localField": "_id",
"foreignField": "_id",
"as": "_ids"
}},
{ "$unwind": "$_ids" },
{ "$replaceRoot": {
"newRoot": "$_ids"
}}
])
And even do a $replaceRoot or $project to make it return the whole document. Or you could have even done $addToSet with $$ROOT if it was not a problem for size.
But the overall point is covered in the first three pipeline stages, or mostly in just the "first". If you want to work with arrays "across documents", then the primary operator is still $unwind.
Alternately for a more "reporting" like format:
db.flights.aggregate([
{ "$addFields": { "copy": "$$ROOT" } },
{ "$unwind": "$dates" },
{ "$group": {
"_id": {
"carrier": "$carrier",
"flightNumber": "$flightNumber",
"dates": "$dates"
},
"count": { "$sum": 1 },
"_docs": { "$addToSet": "$copy" }
}},
{ "$match": { "count": { "$gt": 1 } } },
{ "$group": {
"_id": {
"carrier": "$_id.carrier",
"flightNumber": "$_id.flightNumber",
},
"overlaps": {
"$push": {
"date": "$_id.dates",
"_docs": "$_docs"
}
}
}}
])
Which would report the overlapped dates within each group and tell you which documents contained the overlap:
{
"_id" : {
"carrier" : "abc",
"flightNumber" : 123.0
},
"overlaps" : [
{
"date" : ISODate("2015-01-03T00:00:00.000Z"),
"_docs" : [
{
"_id" : ObjectId("5977f9187dcd6a5f6a9b4b97"),
"carrier" : "abc",
"flightNumber" : 123.0,
"dates" : [
ISODate("2015-01-03T00:00:00.000Z"),
ISODate("2015-01-04T00:00:00.000Z"),
ISODate("2015-01-05T00:00:00.000Z")
]
},
{
"_id" : ObjectId("5977f9187dcd6a5f6a9b4b96"),
"carrier" : "abc",
"flightNumber" : 123.0,
"dates" : [
ISODate("2015-01-01T00:00:00.000Z"),
ISODate("2015-01-02T00:00:00.000Z"),
ISODate("2015-01-03T00:00:00.000Z")
]
}
]
}
]
}
I have following json structure in mongo collection-
{
"students":[
{
"name":"ABC",
"fee":1233
},
{
"name":"PQR",
"fee":345
}
],
"studentDept":[
{
"name":"ABC",
"dept":"A"
},
{
"name":"XYZ",
"dept":"X"
}
]
},
{
"students":[
{
"name":"XYZ",
"fee":133
},
{
"name":"LMN",
"fee":56
}
],
"studentDept":[
{
"name":"XYZ",
"dept":"X"
},
{
"name":"LMN",
"dept":"Y"
},
{
"name":"ABC",
"dept":"P"
}
]
}
Now I want to calculate following output.
if students.name = studentDept.name
so my result should be as below
{
"name":"ABC",
"fee":1233,
"dept":"A",
},
{
"name":"XYZ",
"fee":133,
"dept":"X"
}
{
"name":"LMN",
"fee":56,
"dept":"Y"
}
Do I need to use mongo aggregation or is it possible to get above given output without using aggregation???
What you are really asking here is how to make MongoDB return something that is actually quite different from the form in which you store it in your collection. The standard query operations do allow a "limitted" form of "projection", but even as the title on the page shared in that link suggests, this is really only about "limiting" the fields to display in results based on what is present in your document already.
So any form of "alteration" requires some form of aggregation, which with both the aggregate and mapReduce operations allow to "re-shape" the document results into a form that is different from the input. Perhaps also the main thing people miss with the aggregation framework in particular, is that it is not just all about "aggregating", and in fact the "re-shaping" concept is core to it's implementation.
So in order to get results how you want, you can take an approach like this, which should be suitable for most cases:
db.collection.aggregate([
{ "$unwind": "$students" },
{ "$unwind": "$studentDept" },
{ "$group": {
"_id": "$students.name",
"tfee": { "$first": "$students.fee" },
"tdept": {
"$min": {
"$cond": [
{ "$eq": [
"$students.name",
"$studentDept.name"
]},
"$studentDept.dept",
false
]
}
}
}},
{ "$match": { "tdept": { "$ne": false } } },
{ "$sort": { "_id": 1 } },
{ "$project": {
"_id": 0,
"name": "$_id",
"fee": "$tfee",
"dept": "$tdept"
}}
])
Or alternately just "filter out" the cases where the two "name" fields do not match and then just project the content with the fields you want, if crossing content between documents is not important to you:
db.collection.aggregate([
{ "$unwind": "$students" },
{ "$unwind": "$studentDept" },
{ "$project": {
"_id": 0,
"name": "$students.name",
"fee": "$students.fee",
"dept": "$studentDept.dept",
"same": { "$eq": [ "$students.name", "$studentDept.name" ] }
}},
{ "$match": { "same": true } },
{ "$project": {
"name": 1,
"fee": 1,
"dept": 1
}}
])
From MongoDB 2.6 and upwards you can even do the same thing "inline" to the document between the two arrays. You still want to reshape that array content in your final output though, but possible done a little faster:
db.collection.aggregate([
// Compares entries in each array within the document
{ "$project": {
"students": {
"$map": {
"input": "$students",
"as": "stu",
"in": {
"$setDifference": [
{ "$map": {
"input": "$studentDept",
"as": "dept",
"in": {
"$cond": [
{ "$eq": [ "$$stu.name", "$$dept.name" ] },
{
"name": "$$stu.name",
"fee": "$$stu.fee",
"dept": "$$dept.dept"
},
false
]
}
}},
[false]
]
}
}
}
}},
// Students is now an array of arrays. So unwind it twice
{ "$unwind": "$students" },
{ "$unwind": "$students" },
// Rename the fields and exclude
{ "$project": {
"_id": 0,
"name": "$students.name",
"fee": "$students.fee",
"dept": "$students.dept"
}},
])
So where you want to essentially "alter" the structure of the output then you need to use one of the aggregation tools to do. And you can, even if you are not really aggregating anything.
I have aggregation pipeline stage:
$project: {
'school': {
'id': '$_id',
'name': '$name',
'manager': '$manager'
},
'students': '$groups.students',
'teachers': '$groups.teachers'
}
Need something like this:
{
'users': // manager + students + teachers
}
Tried:
{
'users': {
$push: {
$each: ['$school.manager', '$students', '$teachers']
}
}
}
I'm presuming that "students" and "teachers" are both arrays here and located under a common sub-document heading like so:
{
"_id": 123,
"name": "This school",
"manager": "Bill"
"groups": {
"teachers": ["Ted"],
"students": ["Missy"]
}
}
So in order to get all of those in a singular array such as "users" then it depends on your MongoDB version and the "uniqueness" of your data. For true "sets" and where you have MongoDB 2.6 or greater available, there is the $setUnion operator, albeit with an additional level of $group to make "manager" and array:
db.collection.aggregate([
{ "$group": {
"_id": { "_id": "$_id", "name": "$name" },
"manager": { "$push": "$manager" },
"groups": { "$first": "$groups" }
}},
{ "$project": {
"users": {
"$setUnion": [ "$manager", "$groups.teachers", "$groups.students" ]
}
}}
])
Or otherwise where that operator is not available or there is a "unique" problem then there is this way to handle "combining":
db.collection.aggregate([
{ "$group": {
"_id": { "_id": "_id", "name": "$name" },
"manager": { "$push": "$manager" },
"teachers": { "$first": "$groups.teachers" },
"students": { "$first": "$groups.students" },
"type": { "$first": { "$const": ["M","T","S"] } }
}},
{ "$unwind": "$type" },
{ "$project": {
"users": {
"$cond": [
{ "$eq": [ "$type", "M" ] },
"$manager",
{ "$cond": [
{ "$eq": [ "$type", "T" ] },
"$teachers",
"$students"
]}
]
}
}},
{ "$unwind": "$users" },
{ "$group": {
"_id": "$_id",
"users": { "$push": "$users" }
}}
])
This essentially "tags" each field by a "type" for which the document is copied in the pipeline. Then placed into a single "users" field depending on which "type" matched. The single array then from the resulting three documents from each original can then be safely "unwound" and combined in a final $group operation.
So "sets" are your fastest option where available or where not available or not unique you can use the later technique in order to combine these to a single list.