In a simplified data model, I have three types of documents: items, users and assignments. Users and items are stored in their own collections, while assignments are embedded in items. A sample item might look like this:
{
"_id" : ObjectId("xxx"),
"name" : "yyy",
"assignments" : [
{
"assignmentDate" : ISODate("2018-01-11T10:05:20.125Z"),
"user" : ObjectId("zzz"),
},
{
"assignmentDate" : ISODate("2018-01-12T10:05:20.125Z"),
"user" : ObjectId("iii"),
}
]
}
I would like to query all items that are currently assigned to a given user. This aggregation pipeline does the job:
db.Item.aggregate([
{
$addFields: {
currentAssignment: { $arrayElemAt: ['$assignments', -1] }
}
},
{
$lookup: {
from: 'User',
localField: 'currentAssignment.user',
foreignField: '_id',
as: 'currentUser'
}
},
{
$match: {
'currentUser.name': { $eq: 'admin' }
}
}
]);
How can I build this with the Doctrine ODM Aggregation Builder? The Stage::lookup method accepts only a from parameter. If I use it on a computed field from the aggregation pipeline (currentAssignment in this case), it results in:
arguments to $lookup must be strings, localField: null is type null
Other solutions (if possible even without aggregation?) for retrieving the described dataset are also welcome.
The Lookup stage has more methods, one of which is localField which sets the localField in the aggregation stage.
Related
I have two collections:
cats
balls
"cats" collection has documents with key "ballId" of type string
"balls" collection has documents with key "_id" of type ObjectId
An $lookup inside an aggregation is able to retrieve results if the join is done on keys with the same data type. However in my case, "ballId" and "_id" are of different types. This code retrieves the cats but doesn't retrieve the related balls:
collection('cats').aggregate([
{ $match:{} },
{
$lookup: {
from: "balls",
localField: "ballId",
foreignField: "_id",
as: "balls"
}
}
]);
How can I use $lookup with lossy data type?
Use $lookup with pipeline stage.
Join both collections by converting balls' _id to string ($toString) and next compare both values as string ($eq).
db.cats.aggregate([
{
$match: {}
},
{
$lookup: {
from: "balls",
let: {
ballId: "$ballId"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
{
"$toString": "$_id"
},
"$$ballId"
]
},
}
}
],
as: "balls"
}
}
])
Sample Mongo Playground
Suppose, In MongoDB i have two collections. one is "Students" and the another is "Course".
Student have the document such as
{"id":"1","name":"Alex"},..
and Course has the document such as
{"course_id":"111","course_name":"React"},..
and there is a third collection named "students-courses" where i have kept student's id with their corresponding course id. Like this
{"student_id":"1","course_id":"111"}
i want to make a query with student's id so that it gives the output with his/her enrolled course. like this
{
"id": "1",
"name":"Alex",
"taken_courses": [
{"course_id":"111","course_name":"React"},
{"course_id":"112","course_name":"Vue"}
]
}
it will be many to many relationship in MongoDB without using ORM. How can i make this query?
Need to use $loopup with pipeline,
First $group by student_id because we are going to get courses of students, $push all course_id in course_ids for next step - lookup purpose
db.StudentCourses.aggregate([
{
$group: {
_id: "$student_id",
course_ids: {
$push: "$course_id"
}
}
},
$lookup with Student Collection and get the student details in student
$unwind student because its an array and we need only one from group of same student record
$project required fields
{
$lookup: {
from: "Student",
localField: "_id",
foreignField: "id",
as: "student"
}
},
{
$unwind: "$student"
},
{
$project: {
id: "$_id",
name: "$student.name",
course_ids: 1
}
},
$lookup Course Collection and get all courses that contains course_ids, that we have prepared in above $group
$project the required fields
course details will store in taken_courses
{
$lookup: {
from: "Course",
let: {
cId: "$course_ids"
},
pipeline: [
{
$match: {
$expr: {
$in: [
"$course_id",
"$$cId"
]
}
}
},
{
$project: {
_id: 0
}
}
],
as: "taken_courses"
}
},
$project details, removed not required fields
{
$project: {
_id: 0,
course_ids: 0
}
}
])
Working Playground: https://mongoplayground.net/p/FMZgkyKHPEe
For more details related syntax and usage, check aggregation
I have two collections in MongoDb and want to compare those and get the difference documents.
for example Collection A has below 5 documents
{
"Number" : "0000A95B"
}
{
"Number" : "0001385B"
}
{
"Number" : "0002195B"
}
{
"Number" : "0002E85B"
}
{
"Number" : "0002FC5B"
}
Collection B has below 3 documents:
{
"Number" : "0000A95B"
}
{
"Number" : "0001385B"
}
{
"Number" : "0002195B"
}
I need a query to get the documents which are present in A but not in B
Could use an aggregation query with a $lookup.
db.getCollection("collection_a").aggregate([{
$lookup: {
from: "collection_b",
localField: "Number",
foreignField: "Number",
as: "b_docs"
}
},{
$match: {
b_docs: {
$size: 0
}
}
}])
The first $lookup stage should perform a "join" of sorts on collection_a and collection_b wherein the docs with the matching value of number from b will be added in the b_docs property as an array. If no document is found in collection_b, b_docs should be an empty array, so, just add a $match pipeline to filter the results where the size of the b_docs array is 0.
I have not tested the above query, you might want to try it out.
I'm new to mongodb and currently I'm facing this problem,
db.medical_records.aggregate([
{
"$group": {
"_id": {
"disease_id": "$disease_id" //a string
}, "count": { "$sum": 1 }
}
},
{
"$addFields": {
"disease_id": { "$toObjectId": "$disease_id" }
// i tried to change it into objectID so i could $lookup it
}
},
{
"$lookup": {
"from": "diseases",
"localField": "disease_id",
"foreignField": "_id",
"as": "disease"
}
}
])
this is an example of my medical record collection
{
"_id" : ObjectId("5989c8f13f3958120800682e"),
"disease_id" : "5989c8f13f3958120800682f",
"patient_id" : "5989c8f13f3958120800681f"
}
disease collection
{
"_id" : ObjectId("5989c8f13f3958120800682f"),
"name" : "Culpa autem officia.",
"code" : "Est aperiam."
}
and the result I expect is kind of,
{
"_id" : {disease_id: 5989c8f13f3958120800682f},
"count" : 20,
"disease" : {
"_id" : ObjectId("5989c8f13f3958120800682f"),
"name" : "Culpa autem officia.",
"code" : "Est aperiam."
}
}
I need to join my medical record collection to disease collection as queried above.
When I tried to lookup it to disease collection it failed as foreignField is not the same type as the localField. I've been trying for some time to find a workaround on this problem. And the query above returned another error,
Unrecognized expression '$toObjectId'
This problem might have been asked several times, but I really need a workaround on this problem, please help
New in 4.0: https://docs.mongodb.com/manual/reference/operator/aggregation/toObjectId/
// Define stage to add convertedId field with converted _id value
idConversionStage = {
$addFields: {
convertedId: { $toObjectId: "$_id" }
}
};
// Define stage to sort documents by the converted qty values
sortStage = {
$sort: { "convertedId": -1 }
};
db.orders.aggregate( [
idConversionStage,
sortStage
])
Whew... After going through all the docs and stackoverflow answers, I will like to recommend a simple fix.
When saving your users_id or any document Object id in another collection make sure the dataType remains ObjectId or cast it to ObjectId before saving the document.
Then using the documentation on Mongodb alone without trying to cast or bind.
db.orders.aggregate([
{
$lookup:
{
from: "users",
localField: "user_id",
foreignField: "_id",
as: "inventory_docs"
}
}
])
This case user_id is coming from your current model then _id is coming from your users collections which both are already ObjectId by Origin.
When saving a ref to a new collection for example. user_id in orders collection. make sure the dataType is ObjectID. then you can use the ObjectID without having to convert. For example vehicles and drivers collection. by default aggregate will match dataType to dataType this also maintain the speed of your query.
await Vehicle.aggregate([{
$lookup: {
from: 'users',
localField: "driver_id",
foreignField: "_id",
as: "users"
}
}]).then(vehicles => {
data.status = 200;
data.message = "Vehicles retreived!";
data.data = vehicles;
}).catch(err => {
// console log value
console.log(err);
data.status = 500;
data.message = "Error retreiving vehicle";
data.data = JSON.stringify(err);
});
Answer only valid for versions < 4 of MongoDB:
This cannot be done with you current data structure.
Also see here: Mongoose $lookup where localField is a string of an ObjectId in foreignField
And here: Is it possible to compare string with ObjectId via $lookup
However, why don't you change the data in your medical record collection to this:
{
"_id" : ObjectId("5989c8f13f3958120800682e"),
"disease_id" : ObjectId("5989c8f13f3958120800682f"), // note the ObjectId
"patient_id" : ObjectId("5989c8f13f3958120800681f") // note the ObjectId
}
Given this format you can get what you want using the following query:
db.medical_records.aggregate([
{
"$group": {
"_id": {
"disease_id": "$disease_id" //a string
}, "count": { "$sum": 1 }
}
},
{
"$lookup": {
"from": "diseases",
"localField": "_id.disease_id",
"foreignField": "_id",
"as": "disease"
}
}
])
EDIT based on your comment below:
There is no $toObjectId operator in MongoDB. Try searching the documentation and you'll find nothing! So there simply is no way to achieve your goal without changing your data. I do not know the "eloquent laravel-mongodb" framework you're mentioning but based on its documentation I am thinking your models might need some tuning. How do you model your relationship right now?
Is it possible to do SQL inner joins kind of stuff in MongoDB?
I know there is the $lookup attribute in an aggregation pipeline and it is equivalent to outer joins in SQL, but I want to do something similar to inner joins.
I have three collections which need to merge together:
// User Collection
db.User.find({});
// Output:
{
ID : 1,
USER_NAME : "John",
password : "pass"
}
{
ID : 2,
USER_NAME : "Andrew",
PASSWORD : "andrew"
}
// Role Collection
db.ROLE.find({});
// Output:
{
ID : 1,
ROLE_NAME : "admin"
},
{
ID : 2,
ROLE_NAME : "staff"
}
// USER_ROLE Collection
db.USER_ROLE.find({});
// Output:
{
ID : 1,
USER_ID : 1,
ROLE_ID : 1
}
I have the above collections and I want to extract only the documents matched with users and their respective roles, not all the documents. How can I manage it in MongoDB?
I found answer my self it was
$unwind
done the trick to me following query worked for me
db.USER.aggregate([{
$lookup: {
from: "USER_ROLE",
localField: "ID",
foreignField: "USER_ID",
as: "userRole"
}
}, {
$unwind: {
path: "$userRole",
preserveNullAndEmptyArrays: false
}
}, {
$lookup: {
from: "ROLE",
localField: "userRole.ROLE_ID",
foreignField: "ID",
as: "role"
}
}, {
$unwind: {
path: "$role",
preserveNullAndEmptyArrays: false
}
}, {
$match: {
"role.ROLE_NAME": "staff"
}, {
$project: {
USER_NAME: 1,
_id: 0
}
}
]).pretty()
Anyway thanks for the answers
As Tiramisu wrote this looks like schema issue.
You can make a manual inner join, by removing documents where $lookup returned empty array.
....
{$lookup... as myArray},
{$match: {"myArray":{$ne:[]}}},
{$lookup... as myArray2},
{$match: {"myArray2":{$ne:[]}}},
schema change
I personally will go for schema update, like this:
db.User.find({})
{
ID : 1,
USER_NAME : "John",
password : "pass"
roles:[{ID : 1, ROLE_NAME : "admin"}]
}
db.ROLE.find({})
{
ID : 1,
ROLE_NAME : "admin"
},
Will this help
const RolesSchema = new Schema({
....
});
const Roles = mongoose.model('Roles', RolesSchema);
const UserSchema = new Schema({
...
roles: [{ type: mongoose.Schema.Types.ObjectId, ref: "Roles" }]
});
using the populate on userschema you can also reduce the redundancy
Well you are correct, $lookup attribute is equivalent to outer joins in SQL, but in mongoDB, you need additional aggregation stages so as to perform a similar INNER JOIN in mongo. Here's an example for joining User and ROLE collections based on the ID and displaying the results based on USER_NAME and ROLE_NAME
db.User.aggregate([{
$lookup: {
from: "ROLE",
localField: "ID",
foreignField: "ID",
as: "result"
}
},{
$match: {
result: {
$ne: []
}
}
},{
$addFields: {
result: {
$arrayElemAt: ["$result",0]
}
}
},{
$project: {
USER_NAME: "$USER_NAME",
ROLE_NAME: "$result.ROLE_NAME"
}
}])
Hope it helps!!
MongoDB $lookup aggregation is the most formal and the best-optimized method for this question. However, If you are using Node.js as the server-side, Then you can use a little hack as follows.
CollectionOne.find().then((data0) => {
if (data0.length > 0) {
let array = [];
for (let i = 0; i < data0.length; i++) {
let x = data0[i]
let y = x.yourForeignKey;
array.push({_id: y});
}
CollectionTwo.find(
{$or: array}
).then((data1) => {
res.status(200).json(data1);
}).catch((error1) => {
return error1;
})
}
}).catch((error0) => {
return error0;
});
I used the Array Push() method and the $or operator of MongoDB. You can use the $nor operator instead of the $or to find outer join documents. And also you can change the finding algorithm by using $ne, $nor, $or, $and and etc.
$lookup aggregation
Performs a left outer join to a collection in the same database to filter in documents from the "joined" collection for processing. The
$lookup
stage adds a new array field to each input document. The new array field contains the matching documents from the "joined" collection. The
$lookup
stage passes these reshaped documents to the next stage.
Starting in MongoDB 5.1,
$lookup
works across sharded collections.
To combine elements from two different collections, use the
$unionWith
pipeline stage.
Syntax
The $lookup stage has the following syntaxes:
Equality Match with a Single Join Condition
To perform an equality match between a field from the input documents with a field from the documents of the "joined" collection, the
$lookup
stage has this syntax:
{
$lookup:
{
from: <collection to join>,
localField: <field from the input documents>,
foreignField: <field from the documents of the "from" collection>,
as: <output array field>
}
}
More details: https://www.mongodb.com/docs/manual/reference/operator/aggregation/lookup/
> show dbs
admin 0.000GB
config 0.000GB
local 0.002GB
> use local
switched to db local
> show collections
startup_log
test1
test2
> db.test2.aggregate([{
... $lookup: {
... from: "test1",
... localField: "id",
... foreignField: "id",
... as: "aggTest"
... }
... }])