Mongodb aggregation framework and nested $lookup - mongodb

I'm trying to do a nested population by using the mongodb's aggregation framework and I have some troubles with it.
This is the initial data that I have:
[
{
_id: ObjectId('123'),
profileId: ObjectId('456')
},
// ...other documents
]
I apply this aggregation pipeline to it so I can populate the profile field by using the profileId field:
MyModel.aggregate([
{
$lookup: {
from: 'profiles',
localField: 'profileId',
foreignField: '_id',
as: 'profile'
}
},
{
$project: {
profile: {
$arrayElemAt: [ '$profile', 0 ]
},
profileId: 1,
}
},
]
And this is the result:
[
{
_id: ObjectId('123'),
profileId: ObjectId('456'),
profile: {
grades: [
ObjectId('...'),
ObjectId('...')
]
// ... other props
}
}
// ...other documents
]
Now I expand the pipeline by adding a $lookup to populate the grades inside each profile document:
MyModel.aggregate([
{
$lookup: {
from: 'profiles',
localField: 'profileId',
foreignField: '_id',
as: 'profile'
}
},
{
$project: {
profile: {
$arrayElemAt: [ '$profile', 0 ]
},
profileId: 1,
}
},
{
$lookup: {
from: 'profile-grades',
localField: 'profile._id',
foreignField: 'profile',
as: 'profile.grades'
}
}
]
This is the result so far:
[
{
_id: ObjectId('123'),
profileId: ObjectId('456'),
profile: {
grades: [
{
studyLocationId: ObjectId('789'),
// ... other props
},
// ... other grades
]
}
}
// ...other documents
]
The final step that I'm not able to achieve is the population of each studyLocation inside the grades array.
Wanted result:
[
{
_id: ObjectId('123'),
profileId: ObjectId('456'),
profile: {
grades: [
{
studyLocationId: ObjectId('789'),
studyLocation: {
_id: ObjectId('...'),
// other props
},
},
// ... other grades
]
}
}
// ...
]
I've tried by adding another $lookup stage but without luck:
{
$lookup: {
from: 'study-locations',
localField: 'profile.grades.studyLocationId',
foreignField: '_id',
as: 'profile.grades.studyLocation'
}
}
This simply returns:
grades: {
studyLocation: []
}
How should I do it? Is this even possible?
Thank you!

If you are using mongodb3.4 this will work, for previous versions you need unwind operator on grades.

Related

Mongodb can't group properly after a chain of lookup/unwined stages

I have a complex query requiring a chain of nested unwinds and grouping them in order.
here are relations between models [policy, asset, assetType, field, fieldType]
policy has many asset
asset has one assetType
asset has many fields
field has one fieldType
example object would be something like, where
{
policy: {
..., // policy fields
assets: [
{
..., // asset fields
assetType: {},
fields: [
{
..., // field fields
fieldType: {},
},
],
},
],
},
}
Now I'm trying to do a pipeline to get the nested date with the same structure above
this is the far I get to
mongoose.model('policy').aggregate([
{
$lookup: {
from: 'assets',
localField: 'assets',
foreignField: '_id',
as: 'assets',
},
},
{
$lookup: {
from: 'assettypes',
let: {
id: '$assets._id',
fields: '$assets.fields',
name: '$assets.displayName',
atId: '$assets.assetType',
},
pipeline: [
{
$match: {
$expr: {
$eq: ['$_id', '$$atId'],
},
},
},
{
$project: {
_id: '$$id',
assetId: '$$id',
assetDisplayName: '$$name',
assetFields: '$$fields',
type: 1,
name: 1,
},
},
],
as: 'assets',
},
},
{
$unwind: {
path: '$assets',
},
},
{
$unwind: {
path: '$assets.fields',
},
},
{
$lookup: {
from: 'fieldtypes',
let: {
ftId: '$assets.fields.fieldType',
value: '$assets.fields.value',
ref: '$assets._id',
},
pipeline: [
{
$match: {
$expr: {
$eq: ['$_id', '$$ftId'],
},
},
},
{
$addFields: {
value: '$$value',
assetId: '$$ref',
},
},
],
as: 'assets.fields',
},
},
])
and now I'm stuck with grouping the results to get the optimal object I described above.
Can you help, please?
UPDATE: here is Sample data
If I understand you correctly, you want something like this:
Get all the relevant assets from the policies and unwind them (I guess you only want it for few selected policies, otherwise, if you want to use all assets, you may as well start from their collection and in the end group them by policy)
Get all the wanted data from other collections. Create a fieldtypes array in each document
In order to match each item in fields with its fieldtype use $map with $mergeObjects (this is the more complicated part).
Group by policy
db.policies.aggregate([
{$lookup: {
from: "assets",
localField: "assets",
foreignField: "_id",
as: "assets"
}},
{$unwind: "$assets"},
{$lookup: {
from: "fields",
localField: "assets.fields",
foreignField: "_id",
as: "assets.fields"
}},
{$lookup: {
from: "assettypes",
localField: "assets.assetType",
foreignField: "_id",
as: "assets.assetType"
}},
{$lookup: {
from: "fieldtypes",
localField: "assets.fields.fieldType",
foreignField: "_id",
as: "assets.fieldtypes"
}},
{$set: {
"assets.assetType": {$first: "$assets.assetType"},
"assets.fields": {
$map: {
input: "$assets.fields",
in: {
$mergeObjects: [
"$$this",
{fieldType: {
$getField: {
input: {
$arrayElemAt: [
"$assets.fieldtypes",
{$indexOfArray: ["$assets.fieldtypes._id", "$$this.fieldType"]}
]
},
field: "key"
}
}
}
]
}
}
},
"assets.fieldtypes": "$$REMOVE"
}
},
{$group: {_id: "$_id", assets: {$push: "$assets"}}}
])
See how it works on the playground example

Aggregate multiple lookups return no data

I have documents like this in DB. And I need to grab the data of each item based on the itemType from their own collection.
{ listId: 2, itemType: 'book', itemId: 5364 },
{ listId: 2, itemType: 'car', itemId: 354 },
{ listId: 2, itemType: 'laptop', itemId: 228 }
Based on MongoDB docs and some search, I figured out that I need to use let and $expr in lookup, to make some condition.
ListItemsModel.aggregate([
{ $match: { listId: 2 } },
{ $lookup:
{
from: 'books',
localField: 'itemId',
foreignField: '_id',
let: { "itemType": "$itemType" },
pipeline: [
{ $project: { _id: 1, title: 1 }},
{ $match: { $expr: { $eq: ["$$itemType", "book"] } }}
],
as: 'data'
}
},
{ $lookup:
{
from: 'cars',
localField: 'itemId',
foreignField: '_id',
let: { "itemType": "$itemType" },
pipeline: [
{ $project: { _id: 1, title: 1 }},
{ $match: { $expr: { $eq: ["$$itemType", "car"] } }}
],
as: 'data'
}
},
{ $lookup:
{
from: 'laptops',
localField: 'itemId',
foreignField: '_id',
let: { "itemType": "$itemType" },
pipeline: [
{ $project: { _id: 1, title: 1 }},
{ $match: { $expr: { $eq: ["$$itemType", "laptop"] } }}
],
as: 'data'
}
}
]);
The problem is, in the result all data fields are empty as data: [].
The syntax seems correct to me. What's wrong?
Any subsequent reassignment of field values will eliminate any previous value.
So, for your aggregation pipeline, you need to assign different values to each "$lookup" "as" field.
For example:
// ...
{ $lookup:
{
from: 'books',
// ...
as: 'booksData'
}
},
{ $lookup:
{
from: 'cars',
// ...
as: 'carsData'
}
},
{ $lookup:
{
from: 'laptops',
// ...
as: 'laptopsData'
}
},
// ...

mongodb: how can i use $lookup and $match and $elemMatch in one query

I'm trying to do a lookup from collection1 to collection2 using attachments.collection2_keys (an array), and at the same time filter collection1 where collection2.type is typeA, for any key in attachments.collection2_keys.
I believe this query should work, but it returns no results:
db.getCollection('collection1').aggregate([{ $lookup: { from: "collection2", localField: "attachments.collection2_keys", foreignField: "collection2_key", as: "attachments.collection2_items" } }, {$match: {'attachments.collection2_items': {$elemMatch: {$type: 'typeA'}}}}])
Based on other stackoverflow questions, I tried this query, but it filters attachments.collection2_keys not collection1:
db.getCollection('collection1').aggregate([{ $lookup: { from: "collection2", localField: "attachments.collection2_keys", foreignField: "collection2_key", as: "attachments.collection2_items", pipeline: [{$match: {type: 'typeA'}}] } }])
collection1 example:
[
{ attachments: { collection2_keys: [ 'x' ] } },
{ attachments: { collection2_keys: [ 'y' ] } }
]
collection2 example:
[
{
collection2_key: 'x',
type: 'typeA'
},
{
collection2_key: 'y',
type: 'typeB'
}
]
desired result:
[
{
attachments: {
collection2_keys: [ 'x' ],
collection2_items: [
{
collection2_key: 'x',
type: 'typeA'
}
]
}
}
]
try using lookup along with match, addfields and project.
db.getCollection('collection1').aggregate([
{
'$lookup': {
'from': 'collection2',
'localField': 'attachments.collection2_keys',
'foreignField': 'collection2_key',
'as': 'collection2_items'
}
}, {
'$match': {
'collection2_items.type': 'typeA'
}
}, {
'$addFields': {
'attachments.collection2_items': '$collection2_items'
}
}, {
'$project': {
'_id': 1,
'attachments': 1
}
}
])
try this
db.getCollection('collection1').aggregate([
{
'$lookup': {
'from': 'collection2',
'let': { 'keys': '$attachments.collection2_keys' },
'pipeline' : [{
$match : {
$expr: { $in: ["$collection2_key", "$$keys"] },
}
}],
'as': 'attachments.collection2_items'
}
},
{
$match : {
"attachments.collection2_items.type": 'typeA'
}
}
])

mongo replace ObjectId in array with data

I have model like this:
{
name: 'John Doe',
items: [
{ count: 5, item: ObjectId('xxx1') },
{ count: 2, item: ObjectId('xxx2') }
]
}
items field is not required and contains number field and reference to other entity. I would like to replace item inside items array with data from entity like so:
{
name: 'John Doe',
items: [
{ count: 5, item: { more: 'data', from: 'other entity' } },
{ count: 2, item: { more: 'data 2', from: 'other entity 2' } }
]
}
I tried using mongo's aggregate and lookup:
Model.aggregate([
{
$lookup: {
from: 'items',
localField: 'users.items',
foreignField: '_id',
as: 'users.items'
}
}
]);
but that's replacing everything inside items array (loosing field count). How can I fix it?
Try this one:
db.collection.aggregate([
{ $unwind: { path: "$items", preserveNullAndEmptyArrays: true } },
{
$lookup: {
from: 'items',
localField: 'items.item',
foreignField: '_id',
as: 'items.item'
}
},
{ $set: { "items.item": { $arrayElemAt: ["$items.item", 0] } } },
{
$group: {
_id: { _id: "$_id", name: "$name" },
items: { $push: "$$ROOT.items" }
}
},
{ $replaceRoot: { newRoot: { $mergeObjects: ["$$ROOT", "$_id"] } } }
])

$lookup when foreignField is in nested array

I have two collections :
Student
{
_id: ObjectId("657..."),
name:'abc'
},
{
_id: ObjectId("593..."),
name:'xyz'
}
Library
{
_id: ObjectId("987..."),
book_name:'book1',
issued_to: [
{
student: ObjectId("657...")
},
{
student: ObjectId("658...")
}
]
},
{
_id: ObjectId("898..."),
book_name:'book2',
issued_to: [
{
student: ObjectId("593...")
},
{
student: ObjectId("594...")
}
]
}
I want to make a Join to Student collection that exists in issued_to array of object field in Library collection.
I would like to make a query to student collection to get the student data as well as in library collection, that will check in issued_to array if the student exists or not if exists then get the library document otherwise not.
I have tried $lookup of mongo 3.6 but I didn`t succeed.
db.student.aggregate([{$match:{_id: ObjectId("593...")}}, $lookup: {from: 'library', let: {stu_id:'$_id'}, pipeline:[$match:{$expr: {$and:[{"$hotlist.clientEngagement": "$$stu_id"]}}]}])
But it thorws error please help me in regard of this. I also looked at other questions asked at stackoverflow like. question on stackoverflow,
question2 on stackoverflow but these are comapring simple fields not array of objects. please help me
I am not sure I understand your question entirely but this should help you:
db.student.aggregate([{
$match: { _id: ObjectId("657...") }
}, {
$lookup: {
from: 'library',
localField: '_id' ,
foreignField: 'issued_to.student',
as: 'result'
}
}])
If you want to only get the all book_names for each student you can do this:
db.student.aggregate([{
$match: { _id: ObjectId("657657657657657657657657") }
}, {
$lookup: {
from: 'library',
let: { 'stu_id': '$_id' },
pipeline: [{
$unwind: '$issued_to' // $expr cannot digest arrays so we need to unwind which hurts performance...
}, {
$match: { $expr: { $eq: [ '$issued_to.student', '$$stu_id' ] } }
}, {
$project: { _id: 0, "book_name": 1 } // only include the book_name field
}],
as: 'result'
}
}])
This might not be a very good answer, but if you can change your schema of Library to:
{
_id: ObjectId("987..."),
book_name:'book1'
issued_to: [
ObjectId("657..."),
ObjectId("658...")
]
},
{
_id: "ObjectId("898...")",
book_name:'book2'
issued_to: [
ObjectId("593...")
ObjectId("594...")
]
}
Then when you do:
{
$lookup: {
from: 'student',
localField: 'issued_to',
foreignField: '_id',
as: 'issued_to_students', // this creates a new field without overwriting your original 'issued_to'
}
},
You should get, based on your example above:
{
_id: ObjectId("987..."),
book_name:'book1'
issued_to_students: [
{ _id: ObjectId("657..."), name: 'abc', ... },
{ _id: ObjectId("658..."), name: <name of this _id>, ... }
]
},
{
_id: "ObjectId("898...")",
book_name:'book2'
issued_to: [
{ _id: ObjectId("593..."), name: 'xyz', ... },
{ _id: ObjectId("594..."), name: <name of this _id>, ... }
]
}
You need to $unwind the issued_to from library collection to match the issued_to.student with _id
db.student.aggregate([
{ "$match": { "_id": mongoose.Types.ObjectId(id) } },
{ "$lookup": {
"from": Library.collection.name,
"let": { "studentId": "$_id" },
"pipeline": [
{ "$unwind": "$issued_to" },
{ "$match": { "$expr": { "$eq": [ "$issued_to.student", "$$studentId" ] } } }
],
"as": "issued_to"
}}
])