Aggregate multiple lookups return no data - mongodb

I have documents like this in DB. And I need to grab the data of each item based on the itemType from their own collection.
{ listId: 2, itemType: 'book', itemId: 5364 },
{ listId: 2, itemType: 'car', itemId: 354 },
{ listId: 2, itemType: 'laptop', itemId: 228 }
Based on MongoDB docs and some search, I figured out that I need to use let and $expr in lookup, to make some condition.
ListItemsModel.aggregate([
{ $match: { listId: 2 } },
{ $lookup:
{
from: 'books',
localField: 'itemId',
foreignField: '_id',
let: { "itemType": "$itemType" },
pipeline: [
{ $project: { _id: 1, title: 1 }},
{ $match: { $expr: { $eq: ["$$itemType", "book"] } }}
],
as: 'data'
}
},
{ $lookup:
{
from: 'cars',
localField: 'itemId',
foreignField: '_id',
let: { "itemType": "$itemType" },
pipeline: [
{ $project: { _id: 1, title: 1 }},
{ $match: { $expr: { $eq: ["$$itemType", "car"] } }}
],
as: 'data'
}
},
{ $lookup:
{
from: 'laptops',
localField: 'itemId',
foreignField: '_id',
let: { "itemType": "$itemType" },
pipeline: [
{ $project: { _id: 1, title: 1 }},
{ $match: { $expr: { $eq: ["$$itemType", "laptop"] } }}
],
as: 'data'
}
}
]);
The problem is, in the result all data fields are empty as data: [].
The syntax seems correct to me. What's wrong?

Any subsequent reassignment of field values will eliminate any previous value.
So, for your aggregation pipeline, you need to assign different values to each "$lookup" "as" field.
For example:
// ...
{ $lookup:
{
from: 'books',
// ...
as: 'booksData'
}
},
{ $lookup:
{
from: 'cars',
// ...
as: 'carsData'
}
},
{ $lookup:
{
from: 'laptops',
// ...
as: 'laptopsData'
}
},
// ...

Related

Mongodb can't group properly after a chain of lookup/unwined stages

I have a complex query requiring a chain of nested unwinds and grouping them in order.
here are relations between models [policy, asset, assetType, field, fieldType]
policy has many asset
asset has one assetType
asset has many fields
field has one fieldType
example object would be something like, where
{
policy: {
..., // policy fields
assets: [
{
..., // asset fields
assetType: {},
fields: [
{
..., // field fields
fieldType: {},
},
],
},
],
},
}
Now I'm trying to do a pipeline to get the nested date with the same structure above
this is the far I get to
mongoose.model('policy').aggregate([
{
$lookup: {
from: 'assets',
localField: 'assets',
foreignField: '_id',
as: 'assets',
},
},
{
$lookup: {
from: 'assettypes',
let: {
id: '$assets._id',
fields: '$assets.fields',
name: '$assets.displayName',
atId: '$assets.assetType',
},
pipeline: [
{
$match: {
$expr: {
$eq: ['$_id', '$$atId'],
},
},
},
{
$project: {
_id: '$$id',
assetId: '$$id',
assetDisplayName: '$$name',
assetFields: '$$fields',
type: 1,
name: 1,
},
},
],
as: 'assets',
},
},
{
$unwind: {
path: '$assets',
},
},
{
$unwind: {
path: '$assets.fields',
},
},
{
$lookup: {
from: 'fieldtypes',
let: {
ftId: '$assets.fields.fieldType',
value: '$assets.fields.value',
ref: '$assets._id',
},
pipeline: [
{
$match: {
$expr: {
$eq: ['$_id', '$$ftId'],
},
},
},
{
$addFields: {
value: '$$value',
assetId: '$$ref',
},
},
],
as: 'assets.fields',
},
},
])
and now I'm stuck with grouping the results to get the optimal object I described above.
Can you help, please?
UPDATE: here is Sample data
If I understand you correctly, you want something like this:
Get all the relevant assets from the policies and unwind them (I guess you only want it for few selected policies, otherwise, if you want to use all assets, you may as well start from their collection and in the end group them by policy)
Get all the wanted data from other collections. Create a fieldtypes array in each document
In order to match each item in fields with its fieldtype use $map with $mergeObjects (this is the more complicated part).
Group by policy
db.policies.aggregate([
{$lookup: {
from: "assets",
localField: "assets",
foreignField: "_id",
as: "assets"
}},
{$unwind: "$assets"},
{$lookup: {
from: "fields",
localField: "assets.fields",
foreignField: "_id",
as: "assets.fields"
}},
{$lookup: {
from: "assettypes",
localField: "assets.assetType",
foreignField: "_id",
as: "assets.assetType"
}},
{$lookup: {
from: "fieldtypes",
localField: "assets.fields.fieldType",
foreignField: "_id",
as: "assets.fieldtypes"
}},
{$set: {
"assets.assetType": {$first: "$assets.assetType"},
"assets.fields": {
$map: {
input: "$assets.fields",
in: {
$mergeObjects: [
"$$this",
{fieldType: {
$getField: {
input: {
$arrayElemAt: [
"$assets.fieldtypes",
{$indexOfArray: ["$assets.fieldtypes._id", "$$this.fieldType"]}
]
},
field: "key"
}
}
}
]
}
}
},
"assets.fieldtypes": "$$REMOVE"
}
},
{$group: {_id: "$_id", assets: {$push: "$assets"}}}
])
See how it works on the playground example

MongoDB conditional $lookup with aggregration framework

I'm trying to do a conditional lookup with aggregration framework. In my local and foreign collections I have two fields: fieldA and fieldB. If fieldA != 0 my $lookup should be:
{ from: 'collectionA', localField: 'fieldA', foreignField: 'fieldA', as: 'agg' }
Otherwise, if fieldA = 0 my $lookup should be:
{ from: 'collectionA', localField: 'fieldB', foreignField: 'fieldB', as: 'agg' }
Is it possible to combine these conditions with a single $lookup?
It's not really possible OOB, but you can work around this in multiple ways.
for example add a new "temporary" field based on this condition:
db.colleciton.aggregate([
{
$lookup: {
from: 'collectionA',
let: {
fieldA: "$fieldA", fieldB: "$fieldB"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
{
$cond: [
{
$eq: [
'$$fieldA',
0,
],
},
'$$fieldB',
'$$fieldA',
],
},
{
$cond: [
{
$eq: [
'$$fieldA',
0,
],
},
'$fieldB',
'$fieldA',
],
}
],
},
},
},
],
as: 'agg',
},
}
])
The issue with this approach is that indexes won't be utilized for the lookup for older Mongo versions, which in some cases can be crucial.
You can work around for performance purposes like so:
db.collection.aggregate([
{
$facet: {
one: [
{
$match: {
fieldA: { $ne: 0 },
},
},
{
$lookup: { from: 'collectionA', localField: 'fieldA', foreignField: 'fieldA', as: 'agg' },
},
{
$match: {
'agg.0': { $exists: true },
},
},
],
two: [
{
$match: {
fieldA: { $eq: 0 },
},
},
{
$lookup: { from: 'collectionA', localField: 'fieldB', foreignField: 'fieldB', as: 'agg' },
},
{
$match: {
'agg.0': { $exists: true },
},
},
],
},
},
{
$addFieldS: {
combined: {
$concatArrays: [
'$one',
'$two',
],
},
},
},
{
$unwind: '$combined',
},
{
$replaceRoot: {
newRoot: "$combined"
},
},
]);
While there is some overhead here it will still work faster than an unindexed lookup.

Populate fields lookup of an obj array

Inside Element I have an array of associations, these associations are a collection.
This collection is obj made up of many keys, among these keys there are elements that refer to other collections (obj, obj2) I would like to populate these two collections.
But I'm not able to understand how it can do, even to try to do as few operations as possible.
From what I understand maybe you need to use $unwind.
Element:
{
_id: "",
element: "",
associations: [],
updatedAt: "",
createdAt: ""
}
Association:
{
_id: "",
code: "",
obj: "5s5s55s5d555dff", //populate - Schema.Types.ObjectId - ref: 'objGroup'
obj2: "5f5e5e5e5d5d5d5", //populate - Schema.Types.ObjectId - ref: 'obj2Group'
updatedAt: "",
createdAt: ""
}
In element:
aggregate.push({
$lookup: {
from: 'associations',
let: { cId: '$_id' },
pipeline: [
{ $match: { $expr: { $eq: ['$code', '$$cId'] } } },
{ $match: { $or: getTimeRange(from, to) } }
],
as: 'associations'
}
})
You can use nested lookup inside lookup pipeline,
db.element.aggregate([
{
$lookup: {
from: "associations",
let: { cId: "$_id" },
pipeline: [
{ $match: { $expr: { $eq: ["$code", "$$cId"] } } },
{ $match: { $or: getTimeRange(from, to) } },
{
$lookup: {
from: "objCollection",
localField: "obj",
foreignField: "_id",
as: "obj"
}
},
{ $unwind: "$obj" },
{
$lookup: {
from: "obj2Collection",
localField: "obj2",
foreignField: "_id",
as: "obj2"
}
},
{ $unwind: "$obj2" }
],
as: "associations"
}
}
])

Add two extra filter of 1 collection using mongo

I have 4 collections:
users
users: { id: '123123123', name: 'MrMins' }
matches
{ id: 1, team1: 23, team2: 24, date: '6/14', matchday: 1, locked: false, score1: null, score2: null }
{ id: 2, team1: 9, team2: 32, date: '6/15', matchday: 1, locked: false, score1: null, score2: null }
countries
{id: 23, country: "Russia", pais: "Rusia", group: 'A' }
{id: 24, country: "Saudi Arabia", pais: "Arabia Saudita", group: 'A' }
{id: 9, country: "Egypt", pais: "Egipto", group: 'A' }
{id: 32, country: "Uruguay", pais: "Uruguay", group: 'A' }
forecast
{ matchid: 1, score1: 3, score2: 4, userid: '123123123' }
{ matchid: 2, score1: 3, score2: 0, userid: '123123123' }
My query:
db.collection('matches').aggregate([
{
$lookup: {
from: 'countries',
localField: 'team1',
foreignField: 'id',
as: 'team1'
}
},{
$lookup: {
from: 'countries',
localField: 'team2',
foreignField: 'id',
as: 'team2'
}
}
]).toArray(function(err, res) {
callback(err, res);
});
Just now, I have the relation between matches, and countries (two times).
How I can add the additional filter to forecast doing a relation with matchid and userid ?
It will be quite simple using the mongodb 3.6 $lookup version with using nested pipeline
db.matches.aggregate([
{
$lookup: {
from: 'countries',
let: { 'team1': '$team1' },
pipeline: [
{ $match: { $expr: { $eq: [ '$id', '$$team1' ] } }}
],
as: 'team1'
}
},{
$lookup: {
from: 'countries',
let: { 'team2': '$team2' },
pipeline: [
{ $match: { $expr: { $eq: [ '$id', '$$team2' ] } }}
],
as: 'team2'
}
}, {
$lookup: {
from: 'forecast',
let: { "match_id": "$id" },
pipeline: [
{ $match: { $expr: { $eq: [ '$matchid', '$$match_id' ] } }},
{ $lookup: {
from: 'users',
let: { 'userid': '$userid' },
pipeline: [
{ $match: { $expr: { $eq: [ '$id', '$$userid' ] } }}
],
as: 'user'
}}
],
as: 'forecast'
}
}
])
$lookup works as left outer join but it inserts "joined" results as an array in your document. So every field you add (using as option) will be an array.
To get data both from forecasts and users you have to transform that array into an object and you can do it using $arrayElemAt operator:
db.matches.aggregate([
{
$lookup: {
from: 'countries',
localField: 'team1',
foreignField: 'id',
as: 'team1'
}
},{
$lookup: {
from: 'countries',
localField: 'team2',
foreignField: 'id',
as: 'team2'
}
},{
$lookup: {
from: 'forecast',
localField: 'id',
foreignField: 'matchid',
as: 'forecast'
}
}, {
$addFields: {
forecast: { $arrayElemAt: [ "$forecast", 0 ] }
}
},{
$lookup: {
from: 'users',
localField: 'forecast.userid',
foreignField: 'id',
as: 'forecast.user'
}
}
])

Mongodb aggregation framework and nested $lookup

I'm trying to do a nested population by using the mongodb's aggregation framework and I have some troubles with it.
This is the initial data that I have:
[
{
_id: ObjectId('123'),
profileId: ObjectId('456')
},
// ...other documents
]
I apply this aggregation pipeline to it so I can populate the profile field by using the profileId field:
MyModel.aggregate([
{
$lookup: {
from: 'profiles',
localField: 'profileId',
foreignField: '_id',
as: 'profile'
}
},
{
$project: {
profile: {
$arrayElemAt: [ '$profile', 0 ]
},
profileId: 1,
}
},
]
And this is the result:
[
{
_id: ObjectId('123'),
profileId: ObjectId('456'),
profile: {
grades: [
ObjectId('...'),
ObjectId('...')
]
// ... other props
}
}
// ...other documents
]
Now I expand the pipeline by adding a $lookup to populate the grades inside each profile document:
MyModel.aggregate([
{
$lookup: {
from: 'profiles',
localField: 'profileId',
foreignField: '_id',
as: 'profile'
}
},
{
$project: {
profile: {
$arrayElemAt: [ '$profile', 0 ]
},
profileId: 1,
}
},
{
$lookup: {
from: 'profile-grades',
localField: 'profile._id',
foreignField: 'profile',
as: 'profile.grades'
}
}
]
This is the result so far:
[
{
_id: ObjectId('123'),
profileId: ObjectId('456'),
profile: {
grades: [
{
studyLocationId: ObjectId('789'),
// ... other props
},
// ... other grades
]
}
}
// ...other documents
]
The final step that I'm not able to achieve is the population of each studyLocation inside the grades array.
Wanted result:
[
{
_id: ObjectId('123'),
profileId: ObjectId('456'),
profile: {
grades: [
{
studyLocationId: ObjectId('789'),
studyLocation: {
_id: ObjectId('...'),
// other props
},
},
// ... other grades
]
}
}
// ...
]
I've tried by adding another $lookup stage but without luck:
{
$lookup: {
from: 'study-locations',
localField: 'profile.grades.studyLocationId',
foreignField: '_id',
as: 'profile.grades.studyLocation'
}
}
This simply returns:
grades: {
studyLocation: []
}
How should I do it? Is this even possible?
Thank you!
If you are using mongodb3.4 this will work, for previous versions you need unwind operator on grades.