I'm trying to make a lookup, where the foreignField is dynamic:
{
$merge: {
_id: ObjectId('61e56339b528bf009feca149')
}
},
{
$lookup: {
from: 'computer',
localField: '_id',
foreignField: 'configs.?.refId',
as: 'computers'
}
}
I know that the foreignField always starts with configs and ends with refId, but the string between the two is dynamic.
Here is an example of what a document looks like:
'_id': ObjectId('6319bd1540b41d1a35717a16'),
'name': 'MyComputer',
'configs': {
'ybe': {
'refId': ObjectId('61e56339b528bf009feca149')
'name': 'Ybe Config'
},
'test': {
'refId': ObjectId('61f3d7ec47805d1443f14540')
'name': 'TestConfig'
},
...
}
As you can see the configs property contains different objects with different names ('ybe', 'test', etc...). I want to lookup based on the refId inside of all of those objects.
How do I achieve that?
Using dynamic value as a field name is considered an anti-pattern and introduces unnecessary complexity to querying. However, you can achieve your behaviour with $objectToArray by converting the object into array of k-v pairs and perform the $match in a sub-pipeline.
db.coll.aggregate([
{
"$lookup": {
"from": "computer",
"let": {
id: "$_id"
},
"pipeline": [
{
$set: {
configs: {
"$objectToArray": "$configs"
}
}
},
{
"$unwind": "$configs"
},
{
$match: {
$expr: {
$eq: [
"$$id",
"$configs.v.refId"
]
}
}
}
],
"as": "computers"
}
}
])
MongoPlayground
Related
I have this aggregation:
db.getCollection("users").aggregate([
{
"$match": {
"_id": "5a708a38e6a4078bd49f01d5"
}
},
{
"$lookup": {
"from": "user-locations",
"localField": "locations",
"as": "locations",
"foreignField": "_id"
}
}
])
It works well, but there is one small thing that I don't understand and I can't fix.
In the query output, the locations array is reordered by ObjectId and I really need to keep the original order of data.
Here is how the locations array from the users collection looks like
'locations' : [
ObjectId("5b55e9820b720a1a7cd19633"),
ObjectId("5a708a38e6a4078bd49ef13f")
],
And here is the result after the aggregation:
'locations' : [
{
'_id' : ObjectId("5a708a38e6a4078bd49ef13f"),
'name': 'Location 2'
},
{
'_id' : ObjectId("5b55e9820b720a1a7cd19633"),
'name': 'Location 1'
}
],
What am I missing here? I really have no idea how to proceed with this issue.
Could you give me a push?
$lookup does not guarantee order of result documents, you can try a approach to manage natural order of document,
$unwind deconstruct locations array and add auto index number will start from 0,
$lookup with locations
$set to select first element from locations
$sort by index field in ascending order
$group by _id and reconstruct locations array
db.users.aggregate([
{ $match: { _id: "5a708a38e6a4078bd49f01d5" } },
{
$unwind: {
path: "$locations",
includeArrayIndex: "index"
}
},
{
$lookup: {
from: "user-locations",
localField: "locations",
foreignField: "_id",
as: "locations"
}
},
{ $set: { locations: { $arrayElemAt: ["$locations", 0] } } },
{ $sort: { index: 1 } },
{
$group: {
_id: "$_id",
locations: { $push: "$locations" }
}
}
])
Playground
From this closed bug report:
When using $lookup, the order of the documents returned is not guaranteed. The documents are returned in "natural order" - as they are encountered in the database. The only way to get a guaranteed consistent order is to add a $sort stage to the query.
Basically the way any Mongo query/pipeline works is that it returns documents in the order they were matched, meaning the "right" order is not guaranteed especially if there's indes usage involved.
What you should do is add a $sort stage as suggested, like so:
db.collection.aggregate([
{
"$match": {
"_id": "5a708a38e6a4078bd49f01d5"
}
},
{
"$lookup": {
"from": "user-locations",
"let": {
"locations": "$locations"
},
"pipeline": [
{
"$match": {
"$expr": {
"$setIsSubset": [
[
"$_id"
],
"$$locations"
]
}
}
},
{
$sort: {
_id: 1 // any other sort field you want.
}
}
],
"as": "locations",
}
}
])
You can also keep the original $lookup syntax you're using and just $unwind, $sort and then $group to restore the structure.
I have these entities:
// collectionA
{
key: "value",
ref: SOME-OBJECT-ID
}
// collectionB
{
_id: SOME-OBJECT-ID
key1: "value1"
}
I want that if ref exists in the collectionA entity, it will lookup for it on the collectionB and bring its data.
If the ref key is missing or it doesn't missing but the entity in collectionB is missing I get empty result from all of the aggregate query.
This is the aggregate query:
{ $match },
{
$lookup: {
from: "collectionB",
let: {
ref: "$ref"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$_id", "$$ref"
]
}
}
},
{
$project: {
key1: 1
}
}
],
as: "someData"
}
}
How can I avoid this or add any conditional $lookup?
One way of doing that is adding another match at the beginning to skip from source
To skip from B, you can omit at the end.
{$match:{ ref:{$exists:true}}}
It will consider only ref existing docs.
play
db.A.aggregate([
{
"$match": {
ref: {
$exists: true
}
}
},
{
"$lookup": {
"from": "B",
"localField": "ref",
"foreignField": "_id",
"as": "output"
}
}
])
But you don't need to do this if you don't have specific use case, as it will not impact much.
I have found it. The document was not selected because I have used the $unwind - and it won't return the document if we are trying to do it on an empty array. So this is the fix:
{
$unwind: {
path: "$ref",
preserveNullAndEmptyArrays: true
}
}
Instead of:
{
$unwind: "$ref"
}
I found the preserveNullAndEmptyArrays from this answer How to get all result if unwind field does not exist in mongodb
I have those collection schemas
Schema.users = {
name : "string",
username : "string",
[...]
}
Schema.rooms = {
name : "string",
hidden: "boolean",
user: "string",
sqmt: "number",
service: "string"
}
Schema.room_price = {
morning : "string",
afternoon: "string",
day: "string",
room:'string'
}
I need to aggregate the users with the rooms and foreach room the specific room prices.
the expected result would be
[{
_id:"xXXXXX",
name:"xyz",
username:"xyz",
rooms:[
{
_id: 1111,
name:'room1',
sqmt: '123x',
service:'ppp',
room_prices: [{morning: 123, afternoon: 321}]
}
]}]
The first part of the aggregate could be
db.collection('users').aggregate([
{$match: cond},
{$lookup: {
from: 'rooms',
let: {"user_id", "$_id"},
pipeline: [{$match:{expr: {$eq: ["$user", "$$user_id"]}}}],
as: "rooms"
}}])
but I can't figure out how to get the room prices within the same aggregate
Presuming that room from the room_prices collection has the matching data from the name of the rooms collection, then that would the expression to match on for the "inner" pipeline of the $lookup expression with yet another $lookup:
db.collection('users').aggregate([
{ $match: cond },
{ $lookup: {
from: 'rooms',
let: { "user_id": "$_id" },
pipeline: [
{ $match:{ $expr: { $eq: ["$user", "$$user_id"] } } },
{ $lookup: {
from: 'room_prices',
let: { 'name': '$name' },
pipeline: [
{ $match: { $expr: { $eq: [ '$room', '$$name'] } } },
{ $project: { _id: 0, morning: 1, afternoon: 1 } }
],
as: 'room_prices'
}}
],
as: "rooms"
}}
])
That's also adding a $project in there to select only the fields you want from the prices. When using the expressive form of $lookup you actually do get to express a "pipeline", which can be any aggregation pipeline combination. This allows for complex manipulation and such "nested lookups".
Note that using mongoose you can also get the collection name from the model object using something like:
from: RoomPrice.collection.name
This is generally future proofing against possible model configuration changes which might possibly change the name of the underlying collection.
You can also do pretty much the same with the "legacy" form of $lookup prior to the sub-pipeline syntax available from MongoDB 3.6 and upwards. It's just a bit more processing and reconstruction:
db.collection('users').aggregate([
{ $match: cond },
// in legacy form
{ $lookup: {
from: 'rooms',
localField: 'user_id',
foreignField: 'user',
as: 'rooms'
}},
// unwind the output array
{ $unwind: '$rooms' },
// lookup for the second collection
{ $lookup: {
from: 'room_prices',
localField: 'name',
foreignField: 'room',
as: 'rooms.room_prices'
}},
// Select array fields with $map
{ $addFields: {
'rooms': {
'room_prices': {
$map: {
input: '$rooms.room_prices',
in: {
morning: '$this.morning',
afternoon: '$this.afternoon'
}
}
}
}
}},
// now group back to 'users' data
{ $group: {
_id: '$_id',
name: { $first: '$name' },
username: { $first: '$username' },
// same for any other fields, then $push 'rooms'
rooms: { $push: '$rooms' }
}}
])
That's a bit more overhead mostly from usage of $unwind and also noting that the "field selection" does actually mean you did return the "whole documents" from room_prices "first", and only after that was complete can you select the fields.
So there are advantages to the newer syntax, but it still could be done with earlier versions if you wanted to.
I have two collections :
Student
{
_id: ObjectId("657..."),
name:'abc'
},
{
_id: ObjectId("593..."),
name:'xyz'
}
Library
{
_id: ObjectId("987..."),
book_name:'book1',
issued_to: [
{
student: ObjectId("657...")
},
{
student: ObjectId("658...")
}
]
},
{
_id: ObjectId("898..."),
book_name:'book2',
issued_to: [
{
student: ObjectId("593...")
},
{
student: ObjectId("594...")
}
]
}
I want to make a Join to Student collection that exists in issued_to array of object field in Library collection.
I would like to make a query to student collection to get the student data as well as in library collection, that will check in issued_to array if the student exists or not if exists then get the library document otherwise not.
I have tried $lookup of mongo 3.6 but I didn`t succeed.
db.student.aggregate([{$match:{_id: ObjectId("593...")}}, $lookup: {from: 'library', let: {stu_id:'$_id'}, pipeline:[$match:{$expr: {$and:[{"$hotlist.clientEngagement": "$$stu_id"]}}]}])
But it thorws error please help me in regard of this. I also looked at other questions asked at stackoverflow like. question on stackoverflow,
question2 on stackoverflow but these are comapring simple fields not array of objects. please help me
I am not sure I understand your question entirely but this should help you:
db.student.aggregate([{
$match: { _id: ObjectId("657...") }
}, {
$lookup: {
from: 'library',
localField: '_id' ,
foreignField: 'issued_to.student',
as: 'result'
}
}])
If you want to only get the all book_names for each student you can do this:
db.student.aggregate([{
$match: { _id: ObjectId("657657657657657657657657") }
}, {
$lookup: {
from: 'library',
let: { 'stu_id': '$_id' },
pipeline: [{
$unwind: '$issued_to' // $expr cannot digest arrays so we need to unwind which hurts performance...
}, {
$match: { $expr: { $eq: [ '$issued_to.student', '$$stu_id' ] } }
}, {
$project: { _id: 0, "book_name": 1 } // only include the book_name field
}],
as: 'result'
}
}])
This might not be a very good answer, but if you can change your schema of Library to:
{
_id: ObjectId("987..."),
book_name:'book1'
issued_to: [
ObjectId("657..."),
ObjectId("658...")
]
},
{
_id: "ObjectId("898...")",
book_name:'book2'
issued_to: [
ObjectId("593...")
ObjectId("594...")
]
}
Then when you do:
{
$lookup: {
from: 'student',
localField: 'issued_to',
foreignField: '_id',
as: 'issued_to_students', // this creates a new field without overwriting your original 'issued_to'
}
},
You should get, based on your example above:
{
_id: ObjectId("987..."),
book_name:'book1'
issued_to_students: [
{ _id: ObjectId("657..."), name: 'abc', ... },
{ _id: ObjectId("658..."), name: <name of this _id>, ... }
]
},
{
_id: "ObjectId("898...")",
book_name:'book2'
issued_to: [
{ _id: ObjectId("593..."), name: 'xyz', ... },
{ _id: ObjectId("594..."), name: <name of this _id>, ... }
]
}
You need to $unwind the issued_to from library collection to match the issued_to.student with _id
db.student.aggregate([
{ "$match": { "_id": mongoose.Types.ObjectId(id) } },
{ "$lookup": {
"from": Library.collection.name,
"let": { "studentId": "$_id" },
"pipeline": [
{ "$unwind": "$issued_to" },
{ "$match": { "$expr": { "$eq": [ "$issued_to.student", "$$studentId" ] } } }
],
"as": "issued_to"
}}
])
I have next mongo code:
db.users.aggregate([
{
$match: {
$and: [
{ UserName: { $eq: 'administrator' } },
{ 'Company.CompanyName': { $eq: 'test' } }
]
}
},
{
$lookup: {
from: "companies",
localField: "CompanyID",
foreignField: "CompanyID",
as: "Company"
}
},
])
The $lookup part of the code working great. I got next result:
But if I add $match to the code, it brings nothing.
I found that the problem is in the second match: { 'Company.CompanyName': { $eq: 'test' } }, but I can not realize what is wrong with it.
Any ideas?
UPDATE:
I had also tried $unwind on the $lookup result, but no luck:
db.users.aggregate([
{
$match: {
$and: [
{ UserName: { $eq: 'administrator' } },
{ 'Company.CompanyName': { $eq: 'edt5' } }
]
}
},
{ unwind: '$Company' },
{
$lookup: {
from: 'companies',
localField: 'CompanyID',
foreignField: 'CompanyID',
as: 'Company'
}
},
])
With MongoDB 3.4, you can run an aggregation pipeline that uses the $addFields pipeline and a $filter operator to only return the Company array with elements that match the given condition. You can then wrap the $filter expression with the $arrayElemAt operator to return a single document which in essence incorporates the $unwind functionality by flattening the array.
Follow this example to understand the above concept:
db.users.aggregate([
{ "$match": { "UserName": "administrator" } },
{
"$lookup": {
"from": 'companies',
"localField": 'CompanyID',
"foreignField": 'CompanyID',
"as": 'Company'
}
},
{
"$addFields": {
"Company": {
"$arrayElemAt": [
{
"$filter": {
"input": "$Company",
"as": "comp",
"cond": {
"$eq": [ "$$comp.CompanyName", "edt5" ]
}
}
}, 0
]
}
}
}
])
Below answer is for mongoDB 3.6 or later.
Given that:
You have a collection users with a field CompanyID and a collection of companies with a field CompanyID
you want to lookup Companies on Users by matching CompanyID, where additionally:
each User must match condition: User.UserName equals administrator
each Company on User must match condition: CompanyName equals edt5
The following query will work for you:
db.users.aggregate([
{ $match: { UserName: 'administrator' } },
{
$lookup: {
from: 'companies',
as: 'Company',
let: { CompanyID: '$CompanyID' },
pipeline: [
{
$match: {
$expr: {
$and: [
{ $eq: ['$CompanyID', '$$CompanyID'] },
{ $eq: ['$CompanyName', 'edt5'] },
]
}
}
}
]
}
},
])
Explanation:
This is the way to perform left join queries with conditions more complex than simple foreign / local field equality match.
Instead of using localField and foreignField, you use:
let option where you can map local fields to variables,
pipeline option where you can specify aggregation Array.
In pipeline you can use $match filter, with $expr, where you can reuse variables defined earlier in let.
More info on $lookup
Nice tutorial
here is code for fitering array inside lookup.
const userId = req.userData.userId;
const limit = parseInt(req.params.limit);
const page = parseInt(req.params.page);
Collection.aggregate([
{ $match: {} },
{ $sort: { count: -1 } },
{ $skip: limit * page },
{ $limit: limit },
{
$lookup: {
from: Preference.collection.name,
let: { keywordId: "$_id" },
pipeline: [
{
$match: {
$expr: {
$and: [
{ $eq: ["$keyword", "$$keywordId"] },
{
$eq: ["$user", mongoose.Types.ObjectId(userId)],
},
],
},
},
},
],
as: "keywordData",
},
},
{
$project: {
_id: 0,
id: "$_id",
count: 1,
for: 1,
against: 1,
created_at: 1,
updated_at: 1,
keyword: 1,
selected: {
$cond: {
if: {
$eq: [{ $size: "$keywordData" }, 0],
},
then: false,
else: true,
},
},
},
}])