I have 3 documents that are structured like this: (I add some annotation to highlight what data I am using for the different lookup stages)
Client {
_id,
companyId -> initial match
}
Project {
_id,
clientId, -> stage 1 lookup (client._id)
createdBy
}
User {
_id, -> stage 2 lookup (project.createdBy)
firstName,
lastName
}
I have an aggregate pipeline setup like this that returns an array of clients with an array of projects for each client.
Client.aggregate([
{
$match: {
companyId: mongoose.Types.ObjectId(req.user.companyId)
}
},
{
$lookup: {
from: 'projects',
localField: '_id',
foreignField: 'clientId',
as: 'projects'
}
},
])
.sort({ createdAt: -1 })
Now within the projects array is an object of the project key/values and I want to do another $lookup to get user information based off the createdBy field. I made a few attempts at solutions while reading through the aggregation pipeline docs and wasn't able to come to a working solution. One issue I'm having is that I'm not able to $unwind the first lookup as I need an array of projects. I think it would be possible if my projects were unwinded and looked something like this:
projects: {
"projectId": {
...project
}
}
But that's not currently how things are. Is there a way to accomplish this with lookup stages? My desired output is this:
clients: [
{
_id: 0,
projects: [
{
_id: 0,
createdBy: {
firstName: "John",
lastName: "Doe"
},
}
]
}
]
// ideal 2nd stage lookup
{
lookup: {
from 'users',
localField: 'projects.project.createdBy',
foreignField: '_id',
as: 'createdBy'
}
},
{ $unwind: '$createdBy' }
Related
I apologize for the vague question description, but I have quite a complex question regarding filtration in MongoDB aggregations. Please, see my data schema to understand the question better:
Company {
_id: ObjectId
name: string
}
License {
_id: ObjectId
companyId: ObjectId
userId: ObjectId
}
User {
_id: ObjectId
companyId: ObjectId
email: string
}
The goal:
I would like to query all non-licensed users. In order to do this, you would need these plain MongoDB queries:
const licenses = db.licenses.find({ companyId }); // Get all licenses for specific company
const userIds = licenses.toArray().map(l => l.userId); // Collect all licensed user ids
const nonLicensedUsers = db.users.find({ _id: { $nin: userIds } }); // Query all users that don't hold a license
The problem:
The code above works perfectly fine. However, in our system, companies may have hundreds of thousands of users. Therefore, the first and the last step become exceptionally expensive. I'll elaborate on this. First things first, you need to fetch a big number of documents from DB and transmit them via the network, which is fairly expensive. Then, we need to pass a huge $nin query to MongoDB over the network again, which doubles overhead costs.
So, I would like to perform all the mentioned operations on the MongoDB end and return a small slice of non-licensed users to avoid network transmission costs. Are there ideas on how to achieve this?
I was able to come pretty close using the following aggregation (pseudo-code):
db.company.aggregate([
{ $match: { _id: id } }, // Step 1. Find the company entity by id
{ $lookup: {...} }, // Step 2. Joins 'users' collection by `companyId` field
{ $lookup: {...} }, // Step 3. Joins 'licenses' collection by `companyId` field
{
$project: {
licensesMap: // Step 4. Convert 'licenses' array to the map with the shape { 'user-id': true }. Could be done with $arrayToObject operator
}
},
{
$project: {
unlicensedUsers: {
$filter: {...} // And this is the place, where I stopped
}
}
}
]);
Let's have a closer look at the final stage of the above aggregation. I tried to utilize the $filter aggregation in the following manner:
{
$filter: {
input: "$users"
as: "user",
cond: {
$neq: ["$licensesMap[$$user._id]", true]
}
}
}
But, unfortunately, that didn't work. It seemed like MongoDB didn't apply interpolation and just tried to compare a raw "$licensesMap[$$user._id]" string with true boolean value.
Note #1:
Unfortunately, we're not in a position to change the current data schema. It would be costly for us.
Note #2:
I didn't include this in the aggregation example above, but I did convert Mongo object ids to strings to be able to create the licensesMap. And also, I stringified the ids of the users list to be able to access licensesMap properly.
Sample data:
Companies collection:
[
{ _id: "1", name: "Acme" }
]
Licenses collection
[
{ _id: "1", companyId: "1", userId: "1" },
{ _id: "2", companyId: "1", userId: "2" }
]
Users collection:
[
{ _id: "1", companyId: "1" },
{ _id: "2", companyId: "1" },
{ _id: "3", companyId: "1" },
{ _id: "4", companyId: "1" },
]
The expected result is:
[
_id: "1", // company id
name: "Acme",
unlicensedUsers: [
{ _id: "3", companyId: "1" },
{ _id: "4", companyId: "1" },
]
]
Explanation: unlicensedUsers list contains the third and the fourth users because they don't have corresponding entries in the licenses collection.
How about something simple like:
db.usersCollection.aggregate([
{
$lookup: {
from: "licensesCollection",
localField: "_id",
foreignField: "userId",
as: "licensedUsers"
}
},
{$match: {"licensedUsers.0": {$exists: false}}},
{
$group: {
_id: "$companyId",
unlicensedUsers: {$push: {_id: "$_id", companyId: "$companyId"}}
}
},
{
$lookup: {
from: "companiesCollection",
localField: "_id",
foreignField: "_id",
as: "company"
}
},
{$project: {unlicensedUsers: 1, company: {$arrayElemAt: ["$company", 0]}}},
{$project: {unlicensedUsers: 1, name: "$company.name"}}
])
playground example
users collection and licenses collection, both have anything you need on the users so after the first $lookup that "merges" them, and a simple $match to keep only the unlicensed users, all that left is just formatting to the format you request.
Bonus: This solution can work with any type of id. For example playground
If you're facing a similar situation. Bear in mind that the above solution will work fast only with the hashed index.
I have two table
master_document_table document_table
id | title id | master_document_id | userId
1 | Profile
select * from master_document_table as md
left join document_table as d on md.id = d.master_document_id and d.userId = 2
Result:
id | title | master_document_id | userId
1 | Profile null null
how can this be achieved using mongodb i have tried & also did some research from stack overflow and did not got the expected result.
From mongo v3.2, it is possible to aggregate two or more than two collections.
But here one thing, you need to know is, you won't get the same structure as sql. You will get your data on nested document.
Your tables will look something like this in mongodb:
master_document_table (collection / table)
{
_id: ObjectId("5f28d53c00613e6ada45f702"),
title: "Profile",
}
document_table (collection / table)
{
_id: ObjectId("5f28d53c00613e6ada45e790"),
master_document_id: ObjectId("5f28d53c00613e6ada45f702"), // master_doc_table's ref id.
userId: ObjectId("5f28d53c00613e6ada327f231"), // user's ref id.
}
Your query will be:
db.document_table.aggregate([
{$match: {title: "Profile"}}, // you can give any condition here.
{
$lookup:
{
from: "master_document_table",
localField: "master_document_id",
foreignField: "_id",
as: "master_document_id"
}
}
])
Your query result will look something like:
[{
_id: ObjectId("5f28d53c00613e6ada45e790"),
master_document_id: {
_id: ObjectId("5f28d53c00613e6ada45f702"),
title: "Profile"
},
userId: ObjectId("5f28d53c00613e6ada327f231"),
}]
Now you can restructure it based on your requirement.
I would like to recommend you
https://docs.mongodb.com/manual/reference/operator/aggregation/match/
&
https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/
article from mongodb docs.
Hope this will help to resolve your problem.
Inorder to join two tables you can use the $lookup aggregation pipeline operator in version 3.6 and newer. Since you need the results like LEFT OUTER JOIN, you have to $unwind the array to objects and filter out the desired results using $match conditions (ie document_table.userId = 2). Dont forget to use preserveNullAndEmptyArrays to true when unwinding.
Sample code for reference:
db.master_document_table.aggregate([
{ "$lookup": {
"from": "document_table",
"localField": "id",
"foreignField": "master_document_id",
"as": "document_table"
}},
{ $unwind: { path: "$document_table", preserveNullAndEmptyArrays: true } },
{
$match:{
$or:[
{ "document_table": { $exists: false } },
{"document_table.userId" : 2 }
]
}
}
])
I some how managed to get desire result thanks for the help
db.master_document_table.aggregate([
{
$lookup: {
from: 'document_table',
localField: 'id',
foreignField: 'master_document_id',
as: 'users'
}
},
{
$project: {
title: 1,
users: {
$filter: {
input: '$users',
as: 'user',
cond: {
$eq: ['$$user.userId', 2]
}
}
}
}
}
]);
Suppose, In MongoDB i have two collections. one is "Students" and the another is "Course".
Student have the document such as
{"id":"1","name":"Alex"},..
and Course has the document such as
{"course_id":"111","course_name":"React"},..
and there is a third collection named "students-courses" where i have kept student's id with their corresponding course id. Like this
{"student_id":"1","course_id":"111"}
i want to make a query with student's id so that it gives the output with his/her enrolled course. like this
{
"id": "1",
"name":"Alex",
"taken_courses": [
{"course_id":"111","course_name":"React"},
{"course_id":"112","course_name":"Vue"}
]
}
it will be many to many relationship in MongoDB without using ORM. How can i make this query?
Need to use $loopup with pipeline,
First $group by student_id because we are going to get courses of students, $push all course_id in course_ids for next step - lookup purpose
db.StudentCourses.aggregate([
{
$group: {
_id: "$student_id",
course_ids: {
$push: "$course_id"
}
}
},
$lookup with Student Collection and get the student details in student
$unwind student because its an array and we need only one from group of same student record
$project required fields
{
$lookup: {
from: "Student",
localField: "_id",
foreignField: "id",
as: "student"
}
},
{
$unwind: "$student"
},
{
$project: {
id: "$_id",
name: "$student.name",
course_ids: 1
}
},
$lookup Course Collection and get all courses that contains course_ids, that we have prepared in above $group
$project the required fields
course details will store in taken_courses
{
$lookup: {
from: "Course",
let: {
cId: "$course_ids"
},
pipeline: [
{
$match: {
$expr: {
$in: [
"$course_id",
"$$cId"
]
}
}
},
{
$project: {
_id: 0
}
}
],
as: "taken_courses"
}
},
$project details, removed not required fields
{
$project: {
_id: 0,
course_ids: 0
}
}
])
Working Playground: https://mongoplayground.net/p/FMZgkyKHPEe
For more details related syntax and usage, check aggregation
I'm trying to get a department by it's _id from collection in Mongodb using $lookup and $match with the _id by receiving the _id from the request parameters.
However I get an empty array as my result, and when I change the request.params.id with static value like 2 or 1 it returns the data with no problem .. any solution ?
DepartmentRouter.get("/departments/:id", (request, response) => {
departmentSchema.aggregate([
{
$lookup: {
from: "students",
localField: "_id",
foreignField: "Department",
as: "studentsObject"
}
},
{
$match: {
_id: request.params.id
}
}
]).then(data => {
response.send(data);
}).catch(error => {
response.send(error);
});
});
request.params.id comes as string (assuming you don't use any specific middleware to re-cast it).
All you have to do is cast is to an ObjectId type. like so:
var ObjectID = require('mongodb').ObjectID
{
$match: {
_id: new ObjectId(request.params.id)
}
}
Also a performance tip, Move the $match stage to be before the $lookup stage so you don't perform a lookup (which is expensive) to all the redundant documents.
guys i found the solution , i had to make a new number object using Number
$match: {
_id: Number(request.params.id)
}
I'm fairly new to MongoDB and need help doing a select, or perhaps some sort of left join, on one collection based on another collection's data.
I have two collections, animals and meals, and I want to get the animal(s) that has had it's last registered meal after a certain date (let's say 20171001) to determine if the animal is still active.
collection animals:
{
name: mr floof,
id: 12345,
lastMeal: abcdef
},
{
name: pluto,
id: 6789,
lastMeal: ghijkl
}
collection meals:
{
id: abcdef,
created: 20171008,
name: carrots
},
{
id: ghijkl,
created: 20170918,
name: lettuce
}
So the expected output of the query in this case would be:
{
name: mr floof,
id: 12345,
lastMeal: abcdef
}
As Mr Floof has had his last meal 20171008, i.e. after 20171001.
Hope I was clear enough, but if not, don't hesitate to ask.
You can try below aggregation query.
db.animals.aggregate([ [
{
"$lookup": {
"from": "meals",
"localField": "lastMeal",
"foreignField": "id",
"as": "last_meal"
}
},
{
"$unwind": "$last_meal"
},
{
"$match": {
"last_meal.created": {
"$gt": 20171001
}
}
}
])
More info here.
You can use $project with exclusion after $match stage to format the response to exclude joined fields. Something like { $project: {"last_meal":0} }
MongoDB supports joins with $lookup , In your case you can use query like:-
db.animals.aggregate([
{
$lookup:
{
from: "meals",
localField: "lastMeal",
foreignField: "id",
as: "last_meal"
}
},
{
$match: {
"created" : {
$gt: "date" //your date format
}
}
}
])
thanks !