Mongo how to join two collection and add condition on second collection - mongodb

I have two collections
users{id, name} and files{id, userId, name} I want to find all the files whose file name "abc.xyz", I tried to write a code using $lookup but getting all the files belong to user and not filtering it by name "abc.xyz", I have written following query.
db.user.aggregate([
{"$lookup":
{
"from": "files",
"localField": "id",
"foreignField": "userId",
"as": "fileList"
}
},
{"$project": { "filList":{
"$filter": {
"input":"$fileList",
"as":"file"
"cond": {"$eq": ["$file.name","abc.xyz"]}
}
}
}
}
])
Thank you

I want to find all the files whose file name "abc.xyz" … but getting all the files belong to user and not filtering it by name "abc.xyz"
Based on your question above, it could also be interpreted as "Find all files with name abc.xyz along with its owner information".
In which case, it would be better to filter the files collection first using $match to filter file name equal to abc.xyz. This would limit the number of documents to look-up into users collection, instead of perfoming lookup for both collections then perform filtering.
For example:
db.files.aggregate([
{"$match": {"name":"abc.xyz"}},
{"$lookup": {
"from": "users",
"localField": "userId",
"foreignField": "_id",
"as": "users"
}
}]);
Please note that the collection is now reversed, from files looking up into users. An example result would be:
"result": [
{
"_id": 111,
"userId": 999,
"name": "abc.xyz",
"owner": [
{
"_id": 999,
"name": "xmejx"
}
]
},
{
"_id": 222,
"userId": 998,
"name": "abc.xyz",
"owner": [
{
"_id": 998,
"name": "kalwb"
}
]
}
]
I would also recommend to check out:
Data Model Examples with Patterns
Aggregation Pipeline Optimisation

Related

Aggregate Budgets by Customers

I have two collections: customers and budgets.
I need to get all customers with the related budgets inside an array.
My problem is, I need to start the aggregate from the budgets collection.
Also, I need to return customers who don't have any budgets related.
I need a list with something like this:
customer: {
Id: Guid,
Name: string,
CpfCnpj: number,
AccountantId: Guid
Budgets: []
}
How can I do that?
Here the example
You may want to "$project"/etc. the results somewhat differently, but here's one way to output a document for each customer with an array of their budgets (or an empty array if there is no budgets document for them).
db.budgets.aggregate([
{ // get customer docs with possible budgets
"$unionWith": {
"coll": "customers",
"pipeline": [
{
"$lookup": {
"from": "budgets",
"localField": "_id",
"foreignField": "CustomerId",
"as": "budgets"
}
}
]
}
},
{ // only keep budgets with a customer
"$match": {
"name": {"$exists": true}
}
},
{ // "budgets" set to empty array if missing
"$set": {
"budgets": {
"$ifNull": ["$budgets", [] ]
}
}
}
])
Try it on mongoplayground.net.
If "$unionWith" (introduced in MongoDB version 4.4) is unavailable, here's another way to do it by "transforming" (first four stages below) the original queried collection (here, budgets) into the desired collection (here, customers). The remainder of the pipeline is a simple "$lookup" to get the desired info.
db.budgets.aggregate([
{"$limit": 1},
{
"$lookup": {
"from": "customers",
"pipeline": [],
"as": "customers"
}
},
{"$unwind": "$customers"},
{"$replaceWith": "$customers"},
{
"$lookup": {
"from": "budgets",
"localField": "_id",
"foreignField": "CustomerId",
"as": "budgets"
}
}
])
Try it on mongoplayground.net.

multi-stage aggregation pipeline matching data based on fields retrieved through $lookup

I'm trying to build a complex, nested aggregation pipeline in MongoDB (4.4.9 Community Edition, using the pymongo driver for Python 3.10).
There are relevant data points in different collections which I want to aggregate into one, NEW (ideally) view (or, if that doesn't work) collection.
The collections, and the relevant fields therein follow a hierarchy. There is members, which contains the top-level key on which other data is to be merged,
membershipNumber.
> members.find_one()
{'_id': ObjectId('61153299af6122XXXXXXXXXXXXX'), 'membershipNumber': 'N03XXXXXX'}
Then, there's a different collection, which contains membershipNumber, but also a different, linked field, an_user_id. an_user_id is used in other collections to denote records/fields in arrays that pertain to that particular user.
I 'join' members and an_users like so:
result = members.aggregate([
{
'$lookup': {
'from': 'an_users',
'localField': 'membershipNumber',
'foreignField': 'memref',
'as': 'an_users'
}
},
{ '$unwind' : '$an_users' },
{
'$project' : {
'_id' : 1,
'membershipNumber' : 1,
'an_user_id' : '$an_users.user_id'
}
}
]);
So far so good, this returns the desired, aggregated record:
{'_id': ObjectId('61153253aBBBBBBBBBBBB'),
'membershipNumber': 'N0XXXXXXXX',
'an_user_id': '48XXXXXX'}
Now, I have a third collection, which contains the an_user_id as a string in arrays, denoting wherever that user clicked a given email, whereby a record is an email (and the an_user_ids in the clicks array are users that clicked a link in that email.
{'_id': ObjectId('blah'),
'email_id': '407XXX',
'actions_count': 17,
'administrative_title': 'test',
'bounce': ['3440XXXX'],
'click': ['38294CCC',
'418FFFF',
'48XXXXXX',
'38eGGGG'}
I want to count the number occurences of a given an_user_id (which I've attained from aggregating) in arrays (e.g. clicks, bounces, opens) in the emails collection, and include it in the .aggregate call, to retrieve something like this:
{'_id': ObjectId('61153253aBBBBBBBBBBBB'),
'membershipNumber': 'N0XXXXXXXX',
'an_user_id': '48XXXXXX',
'n_email_clicks' : 412,
'n_email_bounces' : 12
}
Further, I might want to also attach counts of an_user_id in other collections in my DB.
Consider, e.g., this collection called events:
{
"_id": "617ffa96ee11844e143a63dd",
"id": "12345",
"administrative_title": "my_event",
"created_at": {
"$date": "2020-01-15T16:28:50.000Z"
},
"event_creator_id": "123456",
"event_title": "my_event",
"group_id": "123456",
"permalink": "event_id",
"rsvp_count": 54,
"rsvps": [{
"rsvp_id": "56789",
"display_name": "John Doe",
"rsvp_user_id": "48XXXXXX",
"rsvp_created_at": {
"$date": "2020-01-28T15:38:50.000Z"
},
"rsvp_updated_at": {
"$date": "2020-01-28T15:38:50.000Z"
},
"first_name": "John",
"last_name": "Doe",
}, {
"rsvp_id": "543895",
"display_name": "James Appleslice",
"rsvp_user_id": "N03XXXXXX",
"rsvp_created_at": {
"$date": "2020-02-05T13:15:14.000Z"
},
"rsvp_updated_at": {
"$date": "2020-02-05T13:15:14.000Z"
},
"first_name": "James",
"last_name": "Appleslice"}
]
}
So, the end-product would look something like this:
{'_id': ObjectId('61153253aBBBBBBBBBBBB'),
'membershipNumber': 'N0XXXXXXXX',
'an_user_id': '48XXXXXX',
'n_email_clicks' : 412,
'n_email_bounces' : 12,
'n_rsvps' : 12
}
My idea was to use the $lookup parameter -- however, I only know how to use this for matching on fields that I have in the parent collection that I'm performing the aggregation on, but not on fields that have been generated in the process of the aggregation.
Any help would be hugely appreciated!
You could use $lookup pipeline. First you would $lookup the user id followed by another $lookup to verify if the user id exists in email. Lastly few more stages to collect the results and format per your need. Furthermore, you can add $out stage if you would like to write the results into another collection.
db.members.aggregate([{
$lookup: {
from: "an_users",
let: {
membershipNumber: "$membershipNumber"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$memref",
"$$membershipNumber"
]
},
}
},
{
"$lookup": {
"from": "emails",
"localField": "user_id",
"foreignField": "click",
"as": "clicks"
}
},
{
"$project": {
"_id": 1,
"membershipNumber": 1,
"an_user_id": "$user_id",
"n_email_clicks": {
$size: "$clicks"
}
}
}
],
as: "details"
}
},
{
$replaceRoot: {
newRoot: {
$mergeObjects: [
{
$arrayElemAt: [
"$details",
0
]
},
"$$ROOT"
]
}
}
},
{
$project: {
details: 0
}
}])
Working example - https://mongoplayground.net/p/yrFsNp44hpi

MongoDB Aggregation Lookup with Pipeline Doesn't Work

I have two collections. I am trying to add the documents of Collection 2 to Collection 1, if number 1 and number 2 in Collection 2 is within a certain range as specified in Collection 1. FYI ObjectId in Collection 1 and ObjectId in Collection 2 refer to two different items/products, hence I cannot join the two collections on this id.
Example Document from Collection 1:
{'_id': ObjectId('4321'),
'number1_lb': 61.205672407820025,
'number1_ub': 61.24170844385606,
'number2_lb': -149.75074963516136,
'number2_ub': -149.71471359912533}
Example Document from Collection 2:
{'_id': ObjectId('1234'),
'number1': 1.282298,
'number2': 103.8475}
I want the output:
{'_id': ObjectId('4321'),
'number1_lb': 61.205672407820025,
'number1_ub': 61.24170844385606,
'number2_lb': -149.75074963516136,
'number2_ub': -149.71471359912533,
'recs': [ObjectId('3456'), ObjectId('4567'),...]
I thought that a lookup stage with pipeline would work. My code is currently as follows:
{"$lookup":{
"from": "Collection 2",
"let":{
"number1_lb":"$number1_lb",
"number1_ub":"$number1_ub",
"number2_lb":"$number2_lb",
"number2_ub":"$number2_ub"
},
"pipeline": [
{"$match":
{"$expr":
{"$and":[
{"$gte":["$number1","$$number1_lb"]},
{"$gte":["$number2","$$number2_lb"]},
{"$lte":["$number1","$$number1_ub"]},
{"$lte":["$number2","$$number2_ub"]}
]}}}
],
"as": "recs"
}}
But running the above gives me no output. Am I doing something wrong??
I ran it and it seems to work fine; but I had to tweak your input data in coll1 as it didn't meet the $match the criteria.
from pymongo import MongoClient
from bson.json_util import dumps
db = MongoClient()["testdatabase"]
# Data Setup
db.coll1.replace_one({"_id": "4321"}, {"_id": "4321", "number1_lb": -61.205672407820025, "number1_ub": 61.24170844385606, "number2_lb": -149.75074963516136, "number2_ub": 149.71471359912533}, upsert=True)
db.coll2.replace_one({"_id": "1234"}, {"_id": "1234", "number1": 1.282298, "number2": 103.8475}, upsert=True)
# Run the aggregation
results = db.coll1.aggregate([
{"$lookup": {
"from": "coll2",
"let": {
"number1_lb": "$number1_lb",
"number1_ub": "$number1_ub",
"number2_lb": "$number2_lb",
"number2_ub": "$number2_ub"
},
"pipeline": [
{"$match":
{"$expr":
{"$and": [
{"$gte": ["$number1", "$$number1_lb"]},
{"$gte": ["$number2", "$$number2_lb"]},
{"$lte": ["$number1", "$$number1_ub"]},
{"$lte": ["$number2", "$$number2_ub"]}
]}}}
],
"as": "recs"
}}
])
# pretty up the results
print(dumps(results, indent=4))
gives:
[
{
"_id": "4321",
"number1_lb": -61.205672407820025,
"number1_ub": 61.24170844385606,
"number2_lb": -149.75074963516136,
"number2_ub": 149.71471359912533,
"recs": [
{
"_id": "1234",
"number1": 1.282298,
"number2": 103.8475
}
]
}
]
You are looking to use a $lookup and a $project :
{
$lookup: {
from: "Collection2",
localField: [Foreign Field of the Collection1],
foreignField: [Principal field of the foreign collection here Collection2],
as: "nameJoint"
}
},
{$project: {
"newFieldName":
}},
But to make a joint between 2 document there as to be an commun field between those 2 documents. I am not sure there is one in this situation or I misunderstand it.
(A $lookup is bassicaly a SQL joint in noSQL )

How to perform operations in pipeline in mongo $lookup [duplicate]

I have the following collections:
venue collection
{ "_id" : ObjectId("5acdb8f65ea63a27c1facf86"),
"name" : "ASA College - Manhattan Campus",
"addedBy" : ObjectId("5ac8ba3582c2345af70d4658"),
"reviews" : [
ObjectId("5acdb8f65ea63a27c1facf8b"),
ObjectId("5ad8288ccdd9241781dce698")
]
}
reviews collection
{ "_id" : ObjectId("5acdb8f65ea63a27c1facf8b"),
"createdAt" : ISODate("2018-04-07T12:31:49.503Z"),
"venue" : ObjectId("5acdb8f65ea63a27c1facf86"),
"author" : ObjectId("5ac8ba3582c2345af70d4658"),
"content" : "nice place",
"comments" : [
ObjectId("5ad87113882d445c5cbc92c8")
]
}
comment collection
{ "_id" : ObjectId("5ad87113882d445c5cbc92c8"),
"author" : ObjectId("5ac8ba3582c2345af70d4658"),
"comment" : "dcfdsfdcfdsfdcfdsfdcfdsfdcfdsfdcfdsfdcfdsfdcfdsf",
"review" : ObjectId("5acdb8f65ea63a27c1facf8b"),
"__v" : 0
}
author collection
{ "_id" : ObjectId("5ac8ba3582c2345af70d4658"),
"firstName" : "Bruce",
"lastName" : "Wayne",
"email" : "bruce#linkites.com",
"followers" : [ObjectId("5ac8b91482c2345af70d4650")]
}
Now the following populate query works fine
const venues = await Venue.findOne({ _id: id.id })
.populate({
path: 'reviews',
options: { sort: { createdAt: -1 } },
populate: [
{ path: 'author' },
{ path: 'comments', populate: [{ path: 'author' }] }
]
})
However, I want to achieve it with $lookup query, but it splits the venue when I am doing '$unwind' to the reviews... I want reviews in same array (like populate) and in same order...
I want to achieve following query with $lookup because author have followers field so I need to send field isFollow by doing $project which cannot be done using populate...
$project: {
isFollow: { $in: [mongoose.Types.ObjectId(req.user.id), '$followers'] }
}
There are a couple of approaches of course depending on your available MongoDB version. These vary from different usages of $lookup through to enabling object manipulation on the .populate() result via .lean().
I do ask that you read the sections carefully, and be aware that all may not be as it seems when considering your implementation solution.
MongoDB 3.6, "nested" $lookup
With MongoDB 3.6 the $lookup operator gets the additional ability to include a pipeline expression as opposed to simply joining a "local" to "foreign" key value, what this means is you can essentially do each $lookup as "nested" within these pipeline expressions
Venue.aggregate([
{ "$match": { "_id": mongoose.Types.ObjectId(id.id) } },
{ "$lookup": {
"from": Review.collection.name,
"let": { "reviews": "$reviews" },
"pipeline": [
{ "$match": { "$expr": { "$in": [ "$_id", "$$reviews" ] } } },
{ "$lookup": {
"from": Comment.collection.name,
"let": { "comments": "$comments" },
"pipeline": [
{ "$match": { "$expr": { "$in": [ "$_id", "$$comments" ] } } },
{ "$lookup": {
"from": Author.collection.name,
"let": { "author": "$author" },
"pipeline": [
{ "$match": { "$expr": { "$eq": [ "$_id", "$$author" ] } } },
{ "$addFields": {
"isFollower": {
"$in": [
mongoose.Types.ObjectId(req.user.id),
"$followers"
]
}
}}
],
"as": "author"
}},
{ "$addFields": {
"author": { "$arrayElemAt": [ "$author", 0 ] }
}}
],
"as": "comments"
}},
{ "$sort": { "createdAt": -1 } }
],
"as": "reviews"
}},
])
This can be really quite powerful, as you see from the perspective of the original pipeline, it really only knows about adding content to the "reviews" array and then each subsequent "nested" pipeline expression also only ever sees it's "inner" elements from the join.
It is powerful and in some respects it may be a bit clearer as all field paths are relative to the nesting level, but it does start that indentation creep in the BSON structure, and you do need to be aware of whether you are matching to arrays or singular values in traversing the structure.
Note we can also do things here like "flattening the author property" as seen within the "comments" array entries. All $lookup target output may be an "array", but within a "sub-pipeline" we can re-shape that single element array into just a single value.
Standard MongoDB $lookup
Still keeping the "join on the server" you can actually do it with $lookup, but it just takes intermediate processing. This is the long standing approach with deconstructing an array with $unwind and the using $group stages to rebuild arrays:
Venue.aggregate([
{ "$match": { "_id": mongoose.Types.ObjectId(id.id) } },
{ "$lookup": {
"from": Review.collection.name,
"localField": "reviews",
"foreignField": "_id",
"as": "reviews"
}},
{ "$unwind": "$reviews" },
{ "$lookup": {
"from": Comment.collection.name,
"localField": "reviews.comments",
"foreignField": "_id",
"as": "reviews.comments",
}},
{ "$unwind": "$reviews.comments" },
{ "$lookup": {
"from": Author.collection.name,
"localField": "reviews.comments.author",
"foreignField": "_id",
"as": "reviews.comments.author"
}},
{ "$unwind": "$reviews.comments.author" },
{ "$addFields": {
"reviews.comments.author.isFollower": {
"$in": [
mongoose.Types.ObjectId(req.user.id),
"$reviews.comments.author.followers"
]
}
}},
{ "$group": {
"_id": {
"_id": "$_id",
"reviewId": "$review._id"
},
"name": { "$first": "$name" },
"addedBy": { "$first": "$addedBy" },
"review": {
"$first": {
"_id": "$review._id",
"createdAt": "$review.createdAt",
"venue": "$review.venue",
"author": "$review.author",
"content": "$review.content"
}
},
"comments": { "$push": "$reviews.comments" }
}},
{ "$sort": { "_id._id": 1, "review.createdAt": -1 } },
{ "$group": {
"_id": "$_id._id",
"name": { "$first": "$name" },
"addedBy": { "$first": "$addedBy" },
"reviews": {
"$push": {
"_id": "$review._id",
"venue": "$review.venue",
"author": "$review.author",
"content": "$review.content",
"comments": "$comments"
}
}
}}
])
This really is not as daunting as you might think at first and follows a simple pattern of $lookup and $unwind as you progress through each array.
The "author" detail of course is singular, so once that is "unwound" you simply want to leave it that way, make the field addition and start the process of "rolling back" into the arrays.
There are only two levels to reconstruct back to the original Venue document, so the first detail level is by Review to rebuild the "comments" array. All you need to is to $push the path of "$reviews.comments" in order to collect these, and as long as the "$reviews._id" field is in the "grouping _id" the only other things you need to keep are all the other fields. You can put all of these into the _id as well, or you can use $first.
With that done there is only one more $group stage in order to get back to Venue itself. This time the grouping key is "$_id" of course, with all properties of the venue itself using $first and the remaining "$review" details going back into an array with $push. Of course the "$comments" output from the previous $group becomes the "review.comments" path.
Working on a single document and it's relations, this is not really so bad. The $unwind pipeline operator can generally be a performance issue, but in the context of this usage it should not really cause that much of an impact.
Since the data is still being "joined on the server" there is still far less traffic than the other remaining alternative.
JavaScript Manipulation
Of course the other case here is that instead of changing data on the server itself, you actually manipulate the result. In most cases I would be in favor of this approach since any "additions" to the data are probably best handled on the client.
The problem of course with using populate() is that whilst it may 'look like' a much more simplified process, it is in fact NOT A JOIN in any way. All populate() actually does is "hide" the underlying process of submitting multiple queries to the database, and then awaiting the results through async handling.
So the "appearance" of a join is actually the result of multiple requests to the server and then doing "client side manipulation" of the data to embed the details within arrays.
So aside from that clear warning that the performance characteristics are nowhere close to being on par with a server $lookup, the other caveat is of course that the "mongoose Documents" in the result are not actually plain JavaScript objects subject to further manipulation.
So in order to take this approach, you need to add the .lean() method to the query before execution, in order to instruct mongoose to return "plain JavaScript objects" instead of Document types which are cast with schema methods attached to the model. Noting of course that the resulting data no longer has access to any "instance methods" that would otherwise be associated with the related models themselves:
let venue = await Venue.findOne({ _id: id.id })
.populate({
path: 'reviews',
options: { sort: { createdAt: -1 } },
populate: [
{ path: 'comments', populate: [{ path: 'author' }] }
]
})
.lean();
Now venue is a plain object, we can simply process and adjust as needed:
venue.reviews = venue.reviews.map( r =>
({
...r,
comments: r.comments.map( c =>
({
...c,
author: {
...c.author,
isAuthor: c.author.followers.map( f => f.toString() ).indexOf(req.user.id) != -1
}
})
)
})
);
So it's really just a matter of cycling through each of the inner arrays down until the level where you can see the followers array within the author details. The comparison then can be made against the ObjectId values stored in that array after first using .map() to return the "string" values for comparison against the req.user.id which is also a string (if it is not, then also add .toString() on that ), since it is easier in general to compare these values in this way via JavaScript code.
Again though I need to stress that it "looks simple" but it is in fact the sort of thing you really want to avoid for system performance, as those additional queries and the transfer between the server and the client cost a lot in time of processing and even due to the request overhead this adds up to real costs in transport between hosting providers.
Summary
Those are basically your approaches you can take, short of "rolling your own" where you actually perform the "multiple queries" to the database yourself instead of using the helper that .populate() is.
Using the populate output, you can then simply manipulate the data in result just like any other data structure, as long as you apply .lean() to the query to convert or otherwise extract the plain object data from the mongoose documents returned.
Whilst the aggregate approaches look far more involved, there are "a lot" more advantages to doing this work on the server. Larger result sets can be sorted, calculations can be done for further filtering, and of course you get a "single response" to a "single request" made to the server, all with no additional overhead.
It is totally arguable that the pipelines themselves could simply be constructed based on attributes already stored on the schema. So writing your own method to perform this "construction" based on the attached schema should not be too difficult.
In the longer term of course $lookup is the better solution, but you'll probably need to put a little more work into the initial coding, if of course you don't just simply copy from what is listed here ;)

Inner Join on two Fields

I have the following schemas
var User = mongoose.Schema({
email:{type: String, trim: true, index: true, unique: true, sparse: true},
password: String,
name:{type: String, trim: true, index: true, unique: true, sparse: true},
gender: String,
});
var Song = Schema({
track: { type: Schema.Types.ObjectId, ref: 'track' },//Track can be deleted
author: { type: Schema.Types.ObjectId, ref: 'user' },
url: String,
title: String,
photo: String,
publishDate: Date,
views: [{ type: Schema.Types.ObjectId, ref: 'user' }],
likes: [{ type: Schema.Types.ObjectId, ref: 'user' }],
collaborators: [{ type: Schema.Types.ObjectId, ref: 'user' }],
});
I want to select all users (without the password value) , but I want each user will have all the songs where he is the author or one of the collaborators and the was published in the last 2 weeks.
What is the best strategy perform this action (binding between the user.id and song .collaborators) ? Can it be done in one select?
It's very possible in one request, and the basic tool for this with MongoDB is $lookup.
I would think this actually makes more sense to query from the Song collection instead, since your criteria is that they must be listed in one of two properties on that collection.
Optimal INNER Join - Reversed
Presuming the actual "model" names are what is listed above:
var today = new Date.now(),
oneDay = 1000 * 60 * 60 * 24,
twoWeeksAgo = new Date(today - ( oneDay * 14 ));
var userIds; // Should be assigned as an 'Array`, even if only one
Song.aggregate([
{ "$match": {
"$or": [
{ "author": { "$in": userIds } },
{ "collaborators": { "$in": userIds } }
],
"publishedDate": { "$gt": twoWeeksAgo }
}},
{ "$addFields": {
"users": {
"$setIntersection": [
userIds,
{ "$setUnion": [ ["$author"], "$collaborators" ] }
]
}
}},
{ "$lookup": {
"from": User.collection.name,
"localField": "users",
"foreignField": "_id",
"as": "users"
}},
{ "$unwind": "$users" },
{ "$group": {
"_id": "$users._id",
"email": { "$first": "$users.email" },
"name": { "$first": "$users.name" },
"gender": { "$first": "$users.gender" },
"songs": {
"$push": {
"_id": "$_id",
"track": "$track",
"author": "$author",
"url": "$url",
"title": "$title",
"photo": "$photo",
"publishedDate": "$publishedDate",
"views": "$views",
"likes": "$likes",
"collaborators": "$collaborators"
}
}
}}
])
That to me is the most logical course as long as it's an "INNER JOIN" you want from the results, meaning that "all users MUST have a mention on at least one song" in the two properties involved.
The $setUnion takes the "unique list" ( ObjectId is unique anyway ) of combining those two. So if an "author" is also a "collaborator" then they are only listed once for that song.
The $setIntersection "filters" the list from that combined list to only those that were specified in the query condition. This removes any other "collaborator" entries that would not have been in the selection.
The $lookup does the "join" on that combined data to get the users, and the $unwind is done because you want the User to be the main detail. So we basically reverse the "array of users" into "array of songs" in the result.
Also, since the main criteria is from Song, then it makes sense to query from that collection as the direction.
Optional LEFT Join
Doing this the other way around is where the "LEFT JOIN" is wanted, being "ALL Users" regardless if there are any associated songs or not:
User.aggregate([
{ "$lookup": {
"from": Song.collection.name,
"localField": "_id",
"foreignField": "author",
"as": "authors"
}},
{ "$lookup": {
"from": Song.collection.name,
"localField": "_id",
"foreignField": "collaborators",
"as": "collaborators"
}},
{ "$project": {
"email": 1,
"name": 1,
"gender": 1,
"songs": { "$setUnion": [ "$authors", "$collaborators" ] }
}}
])
So the listing of the statement "looks" shorter, but it is forcing "two" $lookup stages in order to obtain results for possible "authors" and "collaborators" rather than one. So the actual "join" operations can be costly in execution time.
The rest is pretty straightforward in applying the same $setUnion but this time the the "result arrays" rather than the original source of the data.
If you wanted similar "query" conditions to above on the "filter" for the "songs" and not the actual User documents returned, then for LEFT Join you actually $filter the array content "post" $lookup:
User.aggregate([
{ "$lookup": {
"from": Song.collection.name,
"localField": "_id",
"foreignField": "author",
"as": "authors"
}},
{ "$lookup": {
"from": Song.collection.name,
"localField": "_id",
"foreignField": "collaborators",
"as": "collaborators"
}},
{ "$project": {
"email": 1,
"name": 1,
"gender": 1,
"songs": {
"$filter": {
"input": { "$setUnion": [ "$authors", "$collaborators" ] },
"as": "s",
"cond": {
"$and": [
{ "$setIsSubset": [
userIds
{ "$setUnion": [ ["$$s.author"], "$$s.collaborators" ] }
]},
{ "$gte": [ "$$s.publishedDate", oneWeekAgo ] }
]
}
}
}
}}
])
Which would mean that by LEFT JOIN Conditions, ALL User documents are returned but the only ones which will contain any "songs" will be those that met the "filter" conditions as being part of the supplied userIds. And even those users which were contained in the list will only show those "songs" within the required range for publishedDate.
The main addition within the $filter is the $setIsSubset operator, which is a short way of comparing the supplied list in userIds to the "combined" list from the two fields present in the document. Noting here the the "current user" already had to be "related" due to the earlier conditions of each $lookup.
MongoDB 3.6 Preview
A new "sub-pipeline" syntax available for $lookup from the MongoDB 3.6 release means that rather than "two" $lookup stages as shown for the LEFT Join variant, you can in fact structure this as a "sub-pipeline", which also optimally filters content before returning results:
User.aggregate([
{ "$lookup": {
"from": Song.collection.name,
"let": {
"user": "$_id"
},
"pipeline": [
{ "$match": {
"$or": [
{ "author": { "$in": userIds } },
{ "collaborators": { "$in": userIds } }
],
"publishedDate": { "$gt": twoWeeksAgo },
"$expr": {
"$or": [
{ "$eq": [ "$$user", "$author" ] },
{ "$setIsSubset": [ ["$$user"], "$collaborators" ]
]
}
}}
],
"as": "songs"
}}
])
And that is all there is to it in that case, since $expr allows usage of the $$user variable declared in "let" to be compared with each entry in the song collection to select only those that are matching in addition to the other query criteria. The result being only those matching songs per user or an empty array. Thus making the whole "sub-pipeline" simply a $match expression, which is pretty much the same as additional logic as opposed to fixed local and foreign keys.
So you could even add a stage to the pipeline following $lookup to filter out any "empty" array results, making the overall result an INNER Join.
So personally I would go for the first approach when you can and only use the second approach where you need to.
NOTE: There are a couple of options here that don't really apply as well. The first being a special case of $lookup + $unwind + $match coalescence in which whilst the basic case applies to the initial INNER Join example it cannot be applied with the LEFT Join Case.
This is because in order for a LEFT Join to be obtained, the usage of $unwind must be implemented with preserveNullAndEmptyArrays: true, and this breaks the rule of application in that the unwinding and matching cannot be "rolled up" within the $lookup and applied to the foreign collection "before" returning results.
Hence why it is not applied in the sample and we use $filter on the returned array instead, since there is no optimal action that can be applied to the foreign collection "before" the results are returned, and nothing stopping all results for songs matching on simply the foreign key from returning. INNER Joins are of course different.
The other case is .populate() with mongoose. The most important distinction being that .populate() is not a single request, but just a programming "shorthand" for actually issuing multiple queries. So at any rate, there would actually be multiple queries issued and always requiring ALL results in order to apply any filtering.
Which leads to the limitation on where the filtering is actually applied, and generally means that you cannot really implement "paging" concepts when you utilize "client side joins" that require conditions to be applied on the foreign collection.
There are some more details on this on Querying after populate in Mongoose, and an actual demonstration of how the basic functionality can be wired in as a custom method in mongoose schema's anyway, but actually using the $lookup pipeline processing underneath.