Aggregate $lookup Array of Objects - mongodb

I have collection schools with field groups and I am attempting to $lookup documents from the users collection. I am getting empty results however and an extra schools document.
Schools Schema
const SchoolSchema = new Schema({
groups: [
{
name: { type: String },
color: { type: String },
userDrivenName: { type: String },
},
]
});
module.exports = School = mongoose.model("School", SchoolSchema);
User Schema
const UserSchema = new Schema({
name: {
type: String,
required: true,
},
groups: [
{
groupId: { type: String },
name: { type: String },
color: { type: String },
userDrivenName: { type: String },
},
]
});
Query
db.schools.aggregate([
{
$match: {
_id: ObjectId("5d836e584a24e20e6090fd7b")
}
},
{
$project: {
groups: 1
}
},
{
$unwind: "$groups"
},
{
$lookup: {
from: "users",
let: {
groupId: "$groups._id"
},
pipeline: [
{
$match: {
"groups.groupId": "$$groupId"
}
}
],
as: "groups",
},
},
])
Results:
[
{
"_id": "5d836e584a24e20e6090fd7b",
"groups": []
},
{
"_id": "5d836e584a24e20e6090fd7b",
"groups": []
}
]
Expected Results:
[
{
"_id":"5d836e584a24e20e6090fd7b",
"groups":[
{
"_id":"5ec01fdc1dfb0a4f08316dfe",
"name":"GROUP 1",
"users":[
{
"name":"Luke Skywalker"
}
]
}
]
}
]
MongoPlayground

Two things:
There's a type mismatch between groupId and groups.groupId so you need to use $toString (based on your Mongo Playground example),
$lookup with custom pipelines allows only expression when you use $match so you need $in and $expr:
{
$lookup: {
from: "users",
let: { groupId: { $toString: "$groups._id" } },
pipeline: [
{
$match: {
$expr: {
$in: ["$$groupId","$groups.groupId"]
}
}
}
],
as: "groups"
}
}
Mongo Playground

Related

How to join two Mongo DB Collections together, with one being an Array of Objects inside the Other

I have two collections, one being Companies and the others being Projects. I am trying to write an aggregation function that first grabs all Companies with the status of "Client", then from there write a pipeline that will return all filtered Companies where the company._id === project.companyId, as an Array of Objects. An example of the shortened Collections are below:
Companies
{
_id: ObjectId('2341908342'),
companyName: "Meta",
address: "123 Facebook Lane",
status: "Client"
}
Projects
{
_id: ObjectId('234123840'),
companyId: '2341908342',
name: "Test Project",
price: 97450,
}
{
_id: ObjectId('23413456'),
companyId: '2341908342',
name: "Test Project 2",
price: 100000,
}
My desired outcome after the Aggregation:
Companies
{
_id: ObjectId('2341908342'),
companyName: "Meta",
address: "123 Facebook Lane",
projects: [ [Project1], [Project2],
}
The projects field does not currently exist on the Companies collection, so I imagine we would have to add it. I also begun writing a $match function to filter by clients, but I am not sure if this is correct. I am trying to use $lookup for this but can not figure out the pipeline. Can anyone help me?
Where I'm currently stuck:
try {
const allClientsWithProjects = await companyCollection
.aggregate([
{
$match: {
orgId: {
$in: [new ObjectId(req.user.orgId)],
},
status: { $in: ["Client"] },
},
},
{
$addFields: {
projects: [{}],
},
},
{
$lookup: { from: "projects", (I am stuck here) },
},
])
.toArray()
Thank you for any help anyone can provide.
UPDATE*
I am seemingly so close I feel like... This is what I have currently, and it is returning everything but Projects is still an empty array.
try {
const allClients = await companyCollection
.aggregate([
{
$match: {
orgId: {
$in: [new ObjectId(req.user.orgId)],
},
status: {
$in: ["Client"],
},
},
},
{
$lookup: {
from: "projects",
let: {
companyId: {
$toString: [req.user.companyId],
},
},
pipeline: [
{
$match: {
$expr: {
$eq: ["$companyId", "$$companyId"],
},
},
},
],
as: "projects",
},
},
])
.toArray()
All of my company information is being returned correctly for multiple companies, but that projects Array is still []. Any help would be appreciated, and I will still be troubleshooting this.
One option is using a $lookup with a pipeline:
db.company.aggregate([
{
$match: {
_id: {
$in: [
ObjectId("5a934e000102030405000000")
],
},
status: {
$in: [
"Client"
]
},
},
},
{
$lookup: {
from: "Projects",
let: {
companyId: {
$toString: "$_id"
}
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$companyId",
"$$companyId"
]
}
}
}
],
as: "projects"
}
}
])
See how it works on the playground example
Final answer for my question:
try {
const allClientsAndProjects = await companyCollection
.aggregate([
{
$match: {
orgId: {
$in: [new ObjectId(req.user.orgId)],
},
status: {
$in: ["Client"],
},
},
},
{
$lookup: {
from: "projects",
let: {
companyId: {
$toString: "$_id",
},
},
pipeline: [
{
$match: {
$expr: {
$eq: ["$companyId", "$$companyId"],
},
},
},
],
as: "projects",
},
},
])
.toArray()

MongoDB Aggregation: Filter array with _id as string by ObjectId

I have the following collections:
const movieSchema = new Schema({
title: String
...
})
const userSchema = new Schema({
firstName: String,
lastName: String,
movies: [
movie: {
type: Schema.Types.ObjectId,
ref: 'Movie'
},
status: String,
feeling: String
]
...
})
I am trying to match up the movie (with all its details) with the user status and feeling for that movie, with the aggregation:
Movie.aggregate([
{ $match: { _id: ObjectId(movieId) } },
{
$lookup: {
from: 'users',
as: 'user_status',
pipeline: [
{ $match: { _id: ObjectId(userId) } },
{
$project: {
_id: 0,
movies: 1
}
},
{ $unwind: '$movies' }
]
}
},
])
Which returns:
[
{
_id: 610b678702500b0646925542,
title: 'The Shawshank Redemption',
user_status: [
{
"movies": {
"_id": "610b678702500b0646925542",
"status": "watched",
"feeling": "love"
}
},
{
"movies": {
"_id": "610b678502500b0646923627",
"status": "watched",
"feeling": "like"
}
},
{
"movies": {
"_id": "610b678502500b0646923637",
"status": "watched",
"feeling": "like"
}
},
]
}
]
My desired result is to match the first movie in user_status to get the eventual final result:
[
{
_id: 610b678702500b0646925542,
title: 'The Shawshank Redemption',
status: "watched",
feeling: "love"
}
]
I thought the next step in my pipeline would be:
{
$addFields: {
user_status: {
$filter: {
input: '$user_status',
cond: {
$eq: ['$$this.movies._id', '$_id']
}
}
}
}
}
But it doesn't work - Not sure if this $addFields is correct, and one problem I know is that my first _id is an ObjectId and the second appears to be a string.
If I understand correctly, you can $filter the user in the already existing $lookup pipeline, which will make things more simple later:
db.movies.aggregate([
{$match: {_id: ObjectId(movieId)}},
{
$lookup: {
from: "users",
as: "user_status",
pipeline: [
{$match: {_id: ObjectId(userId)}},
{$project: {
movies: {
$first: {
$filter: {
input: "$movies",
cond: {$eq: ["$$this.movie", ObjectId(movieId)]}
}
}
}
}
}
]
}
},
{
$project: {
title: 1,
feeling: {$first: "$user_status.movies.feeling"},
status: {$first: "$user_status.movies.status"}
}
}
])
See how it works on the playground example

Mongoose: Problem populating nested array with aggregate

I have these models:
const UserSchema = new Schema({
profile: {
type: Schema.Types.ObjectId,
ref: "profiles",
},
});
And this model:
const ProfileSchema = new Schema({
user: {
type: Schema.Types.ObjectId,
ref: "users",
},
education: [
{
institution: {
type: Schema.Types.ObjectId,
ref: "institutions",
},
major: {
type: Schema.Types.ObjectId,
ref: "majors",
},
},
],
date: {
type: Date,
default: Date.now,
},
});
I've been trying to populate the user.profile.education array using aggregate.
Particularly, the fields institution and major.
So the expected result is the array of education to have its education elements populated.
So the expected result should be something like this:
[
// user 1
{
profile: {
education: [
{ institution: "institution_1_data", major: "major_1_data" },
{ institution: "institution_2_data", major: "major_2_data" },
{ institution: "institution_3_data", major: "major_3_data" },
],
},
},
// user 2
{
profile: {
education: [
{ institution: "institution_1_data", major: "major_1_data" },
{ institution: "institution_2_data", major: "major_2_data" },
],
},
},
];
This is the query that I wrote:
const getUsersWithPopulatedMajorAndInstitution = async () => {
const unwind_education_stage = {
$unwind: "$education",
};
const populate_education_stage = {
$lookup: {
from: "majors",
let: { major: "$education.major" },
pipeline: [{ $match: { $expr: { $eq: ["$_id", "$$major"] } } }],
as: "education.major",
},
$lookup: {
from: "institutions",
let: { institution: "$education.institution" },
pipeline: [{ $match: { $expr: { $eq: ["$_id", "$$institution"] } } }],
as: "education.institution",
},
};
const populate_profile_stage = {
$lookup: {
from: "profiles",
let: { profile_id: "$profile" },
pipeline: [
{
$match: {
$expr: { $eq: ["$_id", "$$profile_id"] },
},
},
unwind_education_stage,
populate_education_stage,
{
$project: {
education: "$education",
},
},
],
as: "profile",
},
};
let users = await User.aggregate([populate_profile_stage]);
return users;
};
There are two problems with this query.
PROBLEM 1:
It only populates institution because the institution $lookup stage was added after the major $lookup stage.
This makes no sense to me, as I've been using aggregate for a while and would expect both major and institution to be populated.
PROBLEM 2:
Using $unwind means education field would be unwinded.
So if the education array contains more than 1 education element (like the examples above), three "copies" of the user will be created and the end result is something like this:
[
// user 1
{
profile: {
education: [{ institution: "institution_1_data", major: "major_1_data" }],
},
},
{
profile: {
education: [{ institution: "institution_2_data", major: "major_2_data" }],
},
},
{
profile: {
education: [{ institution: "institution_3_data", major: "major_3_data" }],
},
},
// user 2
{
profile: {
education: [{ institution: "institution_1_data", major: "major_1_data" }],
},
},
{
profile: {
education: [{ institution: "institution_2_data", major: "major_2_data" }],
},
},
];
But, that's not the expected result as I mentioned above.
What should I change/add in the query?
Solved this this way:
const getUsersWithPopulatedMajorAndInstitution = async (
user_name_surname_input_value
) => {
const unwind_education_stage = {
$unwind: "$education",
};
const look_up_institution_stage = {
$lookup: {
from: "institutions",
localField: "education.institution",
foreignField: "_id",
as: "education.institution",
},
};
const look_up_major_stage = {
$lookup: {
from: "majors",
localField: "education.major",
foreignField: "_id",
as: "education.major",
},
};
const populate_stage = {
$lookup: {
from: "profiles",
let: { profile_id: "$profile" },
pipeline: [
{
$match: {
$expr: { $eq: ["$_id", "$$profile_id"] },
},
},
unwind_education_stage,
look_up_major_stage,
look_up_institution_stage,
// This is necessary to regroup education elements after unwinding them
{
$group: {
_id: null,
education: {
$push: "$education",
},
},
},
{
$project: {
education: "$education",
},
},
],
as: "profile",
},
};
let filtered_users = await User.aggregate([
populate_stage,
]);
return filtered_users;
};

use lookup and group different collection mongodb

Hello I have the following collections
const TransactionSchema = mongoose.Schema({
schedule: {
type: mongoose.Schema.ObjectId,
required: true,
ref: "Schedule"
},
uniqueCode: {
type: String,
required: true
},
created: {
type: Date,
default: Date.now
},
status: {
type: String,
required: false
},
})
const ScheduleSchema = mongoose.Schema({
start: {
type: Date,
required: true,
},
end: {
type: Date,
required: false,
},
location: {
type: mongoose.Schema.ObjectId,
required: true,
ref: "Location"
},
})
and I want to return how many times the schedule appear in transaction ( where the status is equal to 'Active') and group it based on its location Id and then lookup the location collection to show the name.
For example I have the following data.
transaction
[
{
"_id":"identifier",
"schedule":identifier1,
"uniqueCode":"312312312312",
"created":"Date",
"status": 'Active'
},
{
"_id":"identifier",
"schedule":identifier1,
"uniqueCode":"1213123123",
"created":"Date",
"status": "Deleted"
}
]
schedule
[
{
"_id":identifier1,
"start":"date",
"end":"date",
"location": id1
},
{
"_id":identifier2,
"start":"date",
"end":"date",
"location": id2
}
]
and I want to get the following result and limit the result by 10 and sort it based on its total value:
[
{
"locationName":id1 name,
"total":1
},
{
"locationName":id2 name,
"total":0
}
]
thank you. Sorry for my bad english.
A bit complex and long query.
$lookup - schedule collection joins with transaction collection by matching:
_id (schedule) with schedule (transaction)
status is Active
and return a transactions array.
$lookup - schedule collection joins with location collection to return location array.
$set - Take the first document in location array so this field would be a document field instead of an array. [This is needed to help further stage]
$group - Group by location._id. And need the fields such as location and total.
$sort - Sort by total DESC.
$limit - Limit to 10 documents to be returned.
$project - Decorate the output documents.
db.schedule.aggregate([
{
$lookup: {
from: "transaction",
let: {
scheduleId: "$_id"
},
pipeline: [
{
$match: {
$expr: {
$and: [
{
$eq: [
"$schedule",
"$$scheduleId"
]
},
{
$eq: [
"$status",
"Active"
]
}
]
}
}
}
],
as: "transactions"
}
},
{
$lookup: {
from: "location",
localField: "location",
foreignField: "_id",
as: "location"
}
},
{
$set: {
location: {
$first: "$location"
}
}
},
{
$group: {
_id: "$location._id",
location: {
$first: "$location"
},
total: {
$sum: {
$size: "$transactions"
}
}
}
},
{
$sort: {
"total": -1
}
},
{
$limit: 10
},
{
$project: {
_id: 0,
locationName: "$location.name",
total: 1
}
}
])
Sample Mongo Playground

aggregation lookup and match a nested array

Hello i am trying to join two collections...
#COLLECTION 1
const valuesSchema= new Schema({
value: { type: String },
})
const categoriesSchema = new Schema({
name: { type: String },
values: [valuesSchema]
})
mongoose.model('categories', categoriesSchema )
#COLLECTION 2
const productsSchema = new Schema({
name: { type: String },
description: { type: String },
categories: [{
type: mongoose.Schema.Types.ObjectId,
ref: 'categories',
}]
})
mongoose.model('productos', productsSchema )
Now, what i pretend to do is join these collections and have an output like this.
#Example Product Document
{
name: 'My laptop',
description: 'Very ugly laptop',
categories: ['5f55949054f3f31db0491b5c','5f55949054f3f31db0491b5b'] // these are _id of valuesSchema
}
#Expected Output
{
name: 'My laptop',
description: 'Very ugly laptop',
categories: [{value: 'Laptop'}, {value: 'PC'}]
}
This is what i tried.
{
$lookup: {
from: "categories",
let: { "categories": "$categories" },
as: "categories",
pipeline: [
{
$match: {
$expr: {
$in: [ '$values._id','$$categories']
},
}
},
]
}
}
but this query is not matching... Any help please?
You can try,
$lookup with categories
$unwind deconstruct values array
$match categories id with value id
$project to show required field
db.products.aggregate([
{
$lookup: {
from: "categories",
let: { cat: "$categories" },
as: "categories",
pipeline: [
{ $unwind: "$values" },
{ $match: { $expr: { $in: ["$values._id", "$$cat"] } } },
{
$project: {
_id: 0,
value: "$values.value"
}
}
]
}
}
])
Playground
Since you try to use the non-co-related queries, I appreciate it, you can easily achieve with $unwind to flat the array and then $match. To regroup the array we use $group. The $reduce helps to move on each arrays and store some particular values.
[
{
$lookup: {
from: "categories",
let: {
"categories": "$categories"
},
as: "categories",
pipeline: [
{
$unwind: "$values"
},
{
$match: {
$expr: {
$in: [
"$values._id",
"$$categories"
]
},
}
},
{
$group: {
_id: "$_id",
values: {
$addToSet: "$values"
}
}
}
]
}
},
{
$project: {
categories: {
$reduce: {
input: "$categories",
initialValue: [],
in: {
$concatArrays: [
"$$this.values",
"$$value"
]
}
}
}
}
}
]
Working Mongo template