MongoDB Lookup in array - mongodb

Good Morning.
I'm trying to query mongoDB by joining two collections. But I'm not able to do that when the primary identifier is inside an array in the secondary collection.
See my code and my collection
Pipe Colletion
{ _id: 1, deal_id: 25698}
{ _id: 2, deal_id: 45879}
{ _id: 3, deal_id: 54142}
Leads Colletion
{ _id: 1, name:"Teste A", deals_id:[25698,45879]}
{ _id: 2, name:"Teste B", deals_id:[54142]}
Desired result when searching for deal_id from the Pipe collection:
Results
{ _id: 1, deal_id: 25698, name:"Teste A"}
{ _id: 2, deal_id: 45879, name:"Teste A"}
{ _id: 3, deal_id: 54142, name:"Teste B"}
My Code:
db.pipedrive.aggregate([
{
$lookup: {
from: "leads",
'let': {
deal_id: '$deal_id'
},
pipeline: [
{
$match: {
$expr: {
$and: [
{$eq: ["$$deal_id", "$deals_id"]}
]
}
}
}
],
as: "Results"
}
}
]);
As I am currently doing, the analytics line is always returning me empty.
Can you help me?

With your original approach, exchanging $eq with $in should work, and since you only have one condition $and is not necessary. Note that $deals_id should exists in foreign document and must be an array, as $in requires the second parameter to be an array. So in case it doesn't exist we will have to wrap it with $ifNull
db.pipedrive.aggregate([{
$lookup: {
from: "leads",
let: {
deal_id: '$deal_id'
},
pipeline: [
{
$match: {
$expr: { $eq: ["$$deal_id", { $ifNull: ["$deals_id", []] ] }
}
}
],
as: "Results"
}
}
// optional other stages to output as desired shape
]);
A Better alternative would be just to use regular $lookup, since mongodb is able to compare the with the elements of the foreign field array. Just like with find query. This will also account for missing fields or non array fields.
db.pipedrive.aggregate([{
$lookup: {
from: "leads",
localField: "deal_id",
foreignField: "deals_id",
as: "Results"
}
}
// optional other stages to output as desired shape
{
$unwind: "$Results"
},
{
$project: {
deal_id: true,
name: "$Results.name"
}
}
]);
Mongo playground

You can use this query to get desired results. $lookup will take care of array as well.
db.pipe.aggregate([
{
$lookup: {
from: "leads",
localField: "deal_id",
foreignField: "deals_id",
as: "results"
}
},
{
$unwind: "$results"
},
{
$project: {
_id: 1,
deal_id: 1,
name: "$results.name"
}
}
])
Here is the link to check results,
https://mongoplayground.net/p/mhF6zHD9d26

Related

$lookup from nested array without overwriting array

My collection looks like:
collectionName: {
_id: ObjectId("..."),
thing1: 'whatever',
thing2: 100,
arrayOfThings: [{
item1: 'something'
item2: 200,
other_id: ObjectId("..."),
]}
}
Essentially I want to be able to find this entry by its _id, then for each of the items in the arrayOfThings I want to add an "other" field which is the entry in my other collection with the _id in "other_id".
Resulting in:
collectionName: {
_id: ObjectId("..."),
thing1: 'whatever',
thing2: 100,
arrayOfThings: [{
item1: 'something'
item2: 200,
other_id: ObjectId("..."),
other: {
otherField1: 'random data',
otherField2: 3000
]}
}
Everything I've tried either overwrites the entire arrayOfThings array with the array that is returned from the other collection or returns several objects, each with only one entry in the arrayOfThings array by doing something like this:
aggregate([
{ $match: { _id: req.params._id }},
{ $unwind: "$arrayOfThings" },
{ $lookup: { from: "otherCollection", localField: "arrayOfThings.other_id", foreignField: "_id", as: "arrayOfThings.other" }},
]);
Any help is appreciated, thanks.
$match, $unwind and $lookup stages remain same
$addFields to get first element from lookup result
$group by _id and reconstruct arrayOfThings, and get other fields using $first
db.col1.aggregate([
// $match, $unwind and $lookup skipping,
{
$addFields: {
"arrayOfThings.other": { $arrayElemAt: ["$arrayOfThings.other", 0] }
}
},
{
$group: {
_id: "$_id",
arrayOfThings: { $push: "$arrayOfThings" },
thing1: { $first: "$thing1" },
thing2: { $first: "$thing2" }
}
}
])
Playground

Mongodb $lookup aggregation returns all documents from foreign index

I have a users index
[{
username: "foo#bar,
roleIds: [ Types.ObjectId("1234") ]
},
{
username: "foo#moo,
}]
With a roles
{
"_id" : ObjectId("60465768f768621ec5828b68"),
"name" : "admin",
"permissionIds" : ObjectId("604657e8e715ss1f2d78b945")
}
and permissions
{
"_id" : ObjectId("604657e8e715ss1f2d78b945"),
"name" : "view-user",
}
When I get the user by username, I want to hydrate the role information. Not all users have roleIds so I need to be able to still return the user regardless of whether they have roleIds.
At the moment the lookup for the roles always returns every item in the roles index!
My idea is that I lookup the roles joining on the index roles by the array roleIds to _ids
Then I pipeline within that lookup to grab the permission information from within the role.
db.getCollection('users').aggregate([
{
$match: {
'username':'foo#bar',
}
},
{
$lookup: {
from: 'roles',
let: {'roleIds': '_id'},
as: 'roles',
pipeline: [{
$lookup: {
from: 'permissions',
let: {'permissionIds': "_id"},
as: 'permissions',
pipeline: [
{
$project: {
name: 1
}
}
]
}
},{
$project: {
name: 1,
permissions: 1
}
}
]
}
}
])
This route just seems to return all documents within the roles index regardless of whether it is actually a join.
Is there something I am immediately doing wrong??
Following things are wrong in your query:
You are passing wrong variables to pipeline inside let in both the lookups.
Missing $match stage inside pipeline which performs the actual join operation.
Initialize empty roleIds with [] using ifNull (since all users don't have that field).
Try this query:
db.getCollection('users').aggregate([
{
$match: {
'username': 'foo#bar',
}
},
{
$lookup: {
from: 'roles',
let: { 'roleIds': { $ifNull: ["$roleIds", []] } },
as: 'roles',
pipeline: [
{
$match: {
$expr: { $in: ["$_id", "$$roleIds"] }
}
},
{
$lookup: {
from: 'permissions',
let: { 'permissionId': "$permissionIds" },
pipeline: [
{
$match: {
$expr: { $eq: ["$_id", "$$permissionId"] }
}
},
{
$project: {
name: 1
}
}
],
as: 'permissions'
}
},
{
$project: {
name: 1,
permissions: 1
}
}
]
}
}
]);

mongodb 2 level aggregate lookup

I have those collection schemas
Schema.users = {
name : "string",
username : "string",
[...]
}
Schema.rooms = {
name : "string",
hidden: "boolean",
user: "string",
sqmt: "number",
service: "string"
}
Schema.room_price = {
morning : "string",
afternoon: "string",
day: "string",
room:'string'
}
I need to aggregate the users with the rooms and foreach room the specific room prices.
the expected result would be
[{
_id:"xXXXXX",
name:"xyz",
username:"xyz",
rooms:[
{
_id: 1111,
name:'room1',
sqmt: '123x',
service:'ppp',
room_prices: [{morning: 123, afternoon: 321}]
}
]}]
The first part of the aggregate could be
db.collection('users').aggregate([
{$match: cond},
{$lookup: {
from: 'rooms',
let: {"user_id", "$_id"},
pipeline: [{$match:{expr: {$eq: ["$user", "$$user_id"]}}}],
as: "rooms"
}}])
but I can't figure out how to get the room prices within the same aggregate
Presuming that room from the room_prices collection has the matching data from the name of the rooms collection, then that would the expression to match on for the "inner" pipeline of the $lookup expression with yet another $lookup:
db.collection('users').aggregate([
{ $match: cond },
{ $lookup: {
from: 'rooms',
let: { "user_id": "$_id" },
pipeline: [
{ $match:{ $expr: { $eq: ["$user", "$$user_id"] } } },
{ $lookup: {
from: 'room_prices',
let: { 'name': '$name' },
pipeline: [
{ $match: { $expr: { $eq: [ '$room', '$$name'] } } },
{ $project: { _id: 0, morning: 1, afternoon: 1 } }
],
as: 'room_prices'
}}
],
as: "rooms"
}}
])
That's also adding a $project in there to select only the fields you want from the prices. When using the expressive form of $lookup you actually do get to express a "pipeline", which can be any aggregation pipeline combination. This allows for complex manipulation and such "nested lookups".
Note that using mongoose you can also get the collection name from the model object using something like:
from: RoomPrice.collection.name
This is generally future proofing against possible model configuration changes which might possibly change the name of the underlying collection.
You can also do pretty much the same with the "legacy" form of $lookup prior to the sub-pipeline syntax available from MongoDB 3.6 and upwards. It's just a bit more processing and reconstruction:
db.collection('users').aggregate([
{ $match: cond },
// in legacy form
{ $lookup: {
from: 'rooms',
localField: 'user_id',
foreignField: 'user',
as: 'rooms'
}},
// unwind the output array
{ $unwind: '$rooms' },
// lookup for the second collection
{ $lookup: {
from: 'room_prices',
localField: 'name',
foreignField: 'room',
as: 'rooms.room_prices'
}},
// Select array fields with $map
{ $addFields: {
'rooms': {
'room_prices': {
$map: {
input: '$rooms.room_prices',
in: {
morning: '$this.morning',
afternoon: '$this.afternoon'
}
}
}
}
}},
// now group back to 'users' data
{ $group: {
_id: '$_id',
name: { $first: '$name' },
username: { $first: '$username' },
// same for any other fields, then $push 'rooms'
rooms: { $push: '$rooms' }
}}
])
That's a bit more overhead mostly from usage of $unwind and also noting that the "field selection" does actually mean you did return the "whole documents" from room_prices "first", and only after that was complete can you select the fields.
So there are advantages to the newer syntax, but it still could be done with earlier versions if you wanted to.

$lookup when foreignField is in nested array

I have two collections :
Student
{
_id: ObjectId("657..."),
name:'abc'
},
{
_id: ObjectId("593..."),
name:'xyz'
}
Library
{
_id: ObjectId("987..."),
book_name:'book1',
issued_to: [
{
student: ObjectId("657...")
},
{
student: ObjectId("658...")
}
]
},
{
_id: ObjectId("898..."),
book_name:'book2',
issued_to: [
{
student: ObjectId("593...")
},
{
student: ObjectId("594...")
}
]
}
I want to make a Join to Student collection that exists in issued_to array of object field in Library collection.
I would like to make a query to student collection to get the student data as well as in library collection, that will check in issued_to array if the student exists or not if exists then get the library document otherwise not.
I have tried $lookup of mongo 3.6 but I didn`t succeed.
db.student.aggregate([{$match:{_id: ObjectId("593...")}}, $lookup: {from: 'library', let: {stu_id:'$_id'}, pipeline:[$match:{$expr: {$and:[{"$hotlist.clientEngagement": "$$stu_id"]}}]}])
But it thorws error please help me in regard of this. I also looked at other questions asked at stackoverflow like. question on stackoverflow,
question2 on stackoverflow but these are comapring simple fields not array of objects. please help me
I am not sure I understand your question entirely but this should help you:
db.student.aggregate([{
$match: { _id: ObjectId("657...") }
}, {
$lookup: {
from: 'library',
localField: '_id' ,
foreignField: 'issued_to.student',
as: 'result'
}
}])
If you want to only get the all book_names for each student you can do this:
db.student.aggregate([{
$match: { _id: ObjectId("657657657657657657657657") }
}, {
$lookup: {
from: 'library',
let: { 'stu_id': '$_id' },
pipeline: [{
$unwind: '$issued_to' // $expr cannot digest arrays so we need to unwind which hurts performance...
}, {
$match: { $expr: { $eq: [ '$issued_to.student', '$$stu_id' ] } }
}, {
$project: { _id: 0, "book_name": 1 } // only include the book_name field
}],
as: 'result'
}
}])
This might not be a very good answer, but if you can change your schema of Library to:
{
_id: ObjectId("987..."),
book_name:'book1'
issued_to: [
ObjectId("657..."),
ObjectId("658...")
]
},
{
_id: "ObjectId("898...")",
book_name:'book2'
issued_to: [
ObjectId("593...")
ObjectId("594...")
]
}
Then when you do:
{
$lookup: {
from: 'student',
localField: 'issued_to',
foreignField: '_id',
as: 'issued_to_students', // this creates a new field without overwriting your original 'issued_to'
}
},
You should get, based on your example above:
{
_id: ObjectId("987..."),
book_name:'book1'
issued_to_students: [
{ _id: ObjectId("657..."), name: 'abc', ... },
{ _id: ObjectId("658..."), name: <name of this _id>, ... }
]
},
{
_id: "ObjectId("898...")",
book_name:'book2'
issued_to: [
{ _id: ObjectId("593..."), name: 'xyz', ... },
{ _id: ObjectId("594..."), name: <name of this _id>, ... }
]
}
You need to $unwind the issued_to from library collection to match the issued_to.student with _id
db.student.aggregate([
{ "$match": { "_id": mongoose.Types.ObjectId(id) } },
{ "$lookup": {
"from": Library.collection.name,
"let": { "studentId": "$_id" },
"pipeline": [
{ "$unwind": "$issued_to" },
{ "$match": { "$expr": { "$eq": [ "$issued_to.student", "$$studentId" ] } } }
],
"as": "issued_to"
}}
])

Complex aggregation query with in clause from document array

Below is the sample MongoDB Data Model for a user collection:
{
"_id": ObjectId('58842568c706f50f5c1de662'),
"userId": "123455",
"user_name":"Bob"
"interestedTags": [
"music",
"cricket",
"hiking",
"F1",
"Mobile",
"racing"
],
"listFriends": [
"123456",
"123457",
"123458"
]
}
listFriends is an array of userId for other users
For a particular userId I need to extract the listFriends (userId's) and for those userId's I need to aggregate the interestedTags and their count.
I would be able to achieve this by splitting the query into two parts:
1.) Extract the listFriends for a particular userId,
2.) Use this list in an aggregate() function, something like this
db.user.aggregate([
{ $match: { userId: { $in: [ "123456","123457","123458" ] } } },
{ $unwind: '$interestedTags' },
{ $group: { _id: '$interestedTags', countTags: { $sum : 1 } } }
])
I am trying to solve the question: Is there a way to achieve the above functionality (both steps 1 and 2) in a single aggregate function?
You could use $lookup to look for friend documents. This stage is usually used to join two different collection, but it can also do join upon one single collection, in your case I think it should be fine:
db.user.aggregate([{
$match: {
_id: 'user1',
}
}, {
$unwind: '$listFriends',
}, {
$lookup: {
from: 'user',
localField: 'listFriends',
foreignField: '_id',
as: 'friend',
}
}, {
$project: {
friend: {
$arrayElemAt: ['$friend', 0]
}
}
}, {
$unwind: '$friend.interestedTags'
}, {
$group: {
_id: '$friend.interestedTags',
count: {
$sum: 1
}
}
}]);
Note: I use $lookup and $arrayElemAt which are only available in Mongo 3.2 or newer version, so check your Mongo version before using this pipeline.