Find documents matching ObjectIDs in a foreign array - mongodb

I have a collection Users:
{
_id: "5cds8f8rfdshfd"
name: "Ted"
attending: [ObjectId("2cd9fjdkfsld")]
}
I have another collection Events:
{
_id: "2cd9fjdkfsld"
title: "Some Event Attended"
},
{
_id: "34dshfj29jg"
title: "Some Event NOT Attended"
}
I would like to return a list of all events being attended by a given user. However, I need to do this query from the Events collection as this is part of a larger query.
I have gone through the following questions:
$lookup on ObjectId's in an array - This question has the array as a local field; mine is foreign
MongoDB lookup when foreign field is an array of objects - The array is of objects themselves
MongoDB lookup when foreign field is an array
I have tried various ways of modifying the above answers to fit my situation but have been unsuccessful. The second answer from the third question gets me closest but I would like to filter out unmatching results rather than have them returned with a value of 0.
My desired output:
[
{
_id: "2cd9fjdkfsld"
title: "Some Event Attended"
},
]

One option would be like this:
db.getCollection('Events').aggregate({
$lookup: // join
{
from: "Users", // on Users collection
let: { eId: "$_id" }, // keep a local variable "eId" that points to the currently looked at event's "_id"
pipeline: [{
$match: { // filter where
"_id": ObjectId("5c6efc937ef75175b2b8e7a4"), // a specific user
$expr: { $in: [ "$$eId", "$attending" ] } // attends the event we're looking at
}
}],
as: "users" // push all matched users into the "users" array
}
}, {
$match: { // remove events that the user does not attend
"users": { $ne: [] }
}
})
You could obviously get rid of the users field by adding another projection if needed.

Related

How to update fields in a MongoDB collection if certain conditions met between two collections?

What am I doing?
So I am trying to update two fields in my MongoDB collection. The collection name is mydata and looks like this:
{
id: 123,
name: "John",
class: "A-100",
class_id: "", <-- Need to update this field,
class_type: "", <-- Need to update this field
}
What do I want to do?
I have another collection that is older, but it contains two fields that I need that I do not have in my current collection. But they both have the id field that corresponds. This is how it looks like the other collection:
{
id: 123,
name: "John",
class: "A-100",
class_id: 235, <-- Field That I need,
class_type: "Math" <-- Field That I need
}
What have I done so far?
I started an aggregate function that starts with a $lookup then $unwind then $match then $project. Looks like this:
db.mydata.aggregate([
{
$lookup: {
from: "old_collection",
localField: "id",
foreignField: "id",
as: "newData"
}
},
{
$unwind: "newData"
},
{
$match: {"class": "A-100"}
},
{
$project: {
_id: 0,
"id": "$newData.id",
"class_id": "$newData.class_id",
"class_type": "$newData.class_type"
}
},
Need help here to update mydata collection in the
fields that I pointed in the top
])
In summary
What I am trying to do is: If two objects from different collections have the same Id then pick the keys from the second object and update the keys in the first object.
Is there a way to do that in MongoDB?
Thanks.

Can't remove object in array using Mongoose

This has been extensively covered here, but none of the solutions seems to be working for me. I'm attempting to remove an object from an array using that object's id. Currently, my Schema is:
const scheduleSchema = new Schema({
//unrelated
_id: ObjectId
shifts: [
{
_id: Types.ObjectId,
name: String,
shift_start: Date,
shift_end: Date,
},
],
});
I've tried almost every variation of something like this:
.findOneAndUpdate(
{ _id: req.params.id },
{
$pull: {
shifts: { _id: new Types.ObjectId(req.params.id) },
},
}
);
Database:
Database Format
Within these variations, the usual response I've gotten has been either an empty array or null.
I was able slightly find a way around this and accomplish the deletion by utilizing the main _id of the Schema (instead of the nested one:
.findOneAndUpdate(
{ _id: <main _id> },
{ $pull: { shifts: { _id: new Types.ObjectId(<nested _id>) } } },
{ new: true }
);
But I was hoping to figure out a way to do this by just using the nested _id. Any suggestions?
The problem you are having currently is you are using the same _id.
Using mongo, update method allows three objects: query, update and options.
query object is the object into collection which will be updated.
update is the action to do into the object (add, change value...).
options different options to add.
Then, assuming you have this collection:
[
{
"_id": 1,
"shifts": [
{
"_id": 2
},
{
"_id": 3
}
]
}
]
If you try to look for a document which _id is 2, obviously response will be empty (example).
Then, if none document has been found, none document will be updated.
What happens if we look for a document using shifts._id:2?
This tells mongo "search a document where shifts field has an object with _id equals to 2". This query works ok (example) but be careful, this returns the WHOLE document, not only the array which match the _id.
This not return:
[
{
"_id": 1,
"shifts": [
{
"_id": 2
}
]
}
]
Using this query mongo returns the ENTIRE document where exists a field called shifts that contains an object with an _id with value 2. This also include the whole array.
So, with tat, you know why find object works. Now adding this to an update query you can create the query:
This one to remove all shifts._id which are equal to 2.
db.collection.update({
"shifts._id": 2
},
{
$pull: {
shifts: {
_id: 2
}
}
})
Example
Or this one to remove shifts._id if parent _id is equal to 1
db.collection.update({
"_id": 1
},
{
$pull: {
shifts: {
_id: 2
}
}
})
Example

I need to extract data in the form of key-value pairs from a collection of records and merge them into their parent record in mongoDB

Below I have a structure for supporting custom picklist fields (in this example) within my sails.js application. The general idea is we support a collection of custom picklist values on any model within the app and the customer can have total control of the configuration of the custom field.
I went with this relationship model as using a simple json field lacks robustness when it comes to updating each individual custom picklist value. If I allow a customer to change "Internal" to "External" I need to update all records that have the value "Internal" recorded against that custom picklist with the new value.
This way - when I update the "value" field of CustomPicklistValue wherever that record is referenced via ID it will use the new value.
Now the problem comes when I need to integrate this model into my existing report engine...
rawCollection
.aggregate(
[
{
$match: {
createdAt: {
$gte: rangeEndDate,
$lte: rangeStartDate
},
...$match
}
},
{
$project: {
...$project,
total: $projectAggregation
}
},
{
$group: {
_id: {
...$groupKey
},
total: {
[`$${aggrAttrFunc}`]: "$total"
}
}
}
],
{
cursor: {
batchSize: 100
}
}
)
Here is the main part of a method for retrieving and aggregating any models stored in my mongodb instance. A user can specify a whole range of things including but not limited to the model, field specific date ranges and filters such as "where certificate status equals expired" etc.
So I'm now presented with this data structure:
{
id: '5e5fb732a9422a001146509f',
customPicklistValues: [
{
id: '5e4e904f16ab94bff1a324a0',
value: 'Internal',
fieldName: 'Business Group',
customPicklist: '109c7a1a9d00b664f2ee7827'
},
{
id: '5e4e904f16ab94bff1a324a4',
value: 'Slack',
fieldName: 'Application',
customPicklist: '109c5a1a9d00b664f2ee7827'
}
],
}
And for the life of me can't work out if there's any way I can essentially pull out fieldName and value for each of the populated records as key-value pairs and add each to the parent record before running my match clause...
I think I need to use lookup to initially populate the customPicklistValues and then merge them somehow?
Any help appreciated.
EDIT:
#whoami has suggested I use $addFields. There was a fair amount I needed to do before $addFields to populate the linked records (due to how Waterline via sails.js handles saving Mongo ObjectIDs in related collections as strings), you can see my steps in compass here:
Final step would be to edit this or add a stage to it to actually be able to support a key:value pair like Business Group: "Finance" in this example.
You can try these stages after your $lookup stage :
db.collection.aggregate([
{
$addFields: {
customPicklistValues:
{
$arrayToObject: {
$map: {
input: '$customPicklistValues',
in: { k: '$$this.fieldName', v: '$$this.value' }
}
}
}
}
},
{ $replaceRoot: { newRoot: { $mergeObjects: ['$customPicklistValues', '$$ROOT'] } } },
{ $project: { customPicklistValues: 0 } }
])
Test : MongoDB-Playground

Mongo aggregation framework on big data

Could you please help me with mongoDB aggregation. Here is what I would like to do next:
I have collection A. A document from A represents an object like:
{
nameA: 'first',
items: [
'item1',
'item2',
'item3',
'item4'
]
}
And I have the Collection B with documents like:
[
{
item: 'item3',
info: 'info1'
},
{
item: 'item3',
info: 'info2'
},
{
item: 'item3',
info: 'info3'
}
]
I work with big data, so it would be better to do it in one query. Imagine that we already have all data from collection A. I would like to build a query on collection B to get next structure result:
{
'first'/*nameA*/: ['info1', 'info2', 'info3'],
....
}
How do I achieve the desired result with MongoDB aggregation?
As Rahul Kumar mentioned in his comment, your design is more leaning towards a relational database schema design, and it makes it quite difficult to design efficient MongoDB it.
However, it is still possible to achieve the functionality you are looking for by leveraging the $lookup stage of the aggregation framework, as follows:
db.A.aggregate([
{
$unwind: {
path: "$items"
}
},
{
$lookup: {
from: "B",
localField: "items",
foreignField: "item",
as: "item_info"
}
},
{
$unwind: {
path: "$item_info"
}
},
{
$group: {
_id: "$nameA",
item_info: { $addToSet: "$item_info.info" }
}
}
]);
In the first $unwind stage you normalize the items array on
collection A in order to be able to pass its output to the next
stage
In the $lookup stage you make a left join between two collections
that are part of the same database, in this case used to get the
item information from collection B
In the second $unwind stage you normalize the data you extracted
from collection B in order to flatten the array containing the
objects from collection B that were mapped to the corresponding
items in collection A
Finally, in the $group stage you group all the entries of the
result set by nameA and create an array of unique item information
values. If you would like to have all the duplicate occurrences of
the item information values, you can replace the $addToSet
accumulator with $push.
Below is the result of running the above aggregation pipeline on the collections that you provided:
{ "_id" : "second", "item_info" : [ "info3", "info2", "info1" ] }
{ "_id" : "first", "item_info" : [ "info3", "info2", "info1" ] }

Can I use populate before aggregate in mongoose?

I have two models, one is user
userSchema = new Schema({
userID: String,
age: Number
});
and the other is the score recorded several times everyday for all users
ScoreSchema = new Schema({
userID: {type: String, ref: 'User'},
score: Number,
created_date = Date,
....
})
I would like to do some query/calculation on the score for some users meeting specific requirement, say I would like to calculate the average of score for all users greater than 20 day by day.
My thought is that firstly do the populate on Scores to populate user's ages and then do the aggregate after that.
Something like
Score.
populate('userID','age').
aggregate([
{$match: {'userID.age': {$gt: 20}}},
{$group: ...},
{$group: ...}
], function(err, data){});
Is it Ok to use populate before aggregate? Or I first find all the userID meeting the requirement and save them in a array and then use $in to match the score document?
No you cannot call .populate() before .aggregate(), and there is a very good reason why you cannot. But there are different approaches you can take.
The .populate() method works "client side" where the underlying code actually performs additional queries ( or more accurately an $in query ) to "lookup" the specified element(s) from the referenced collection.
In contrast .aggregate() is a "server side" operation, so you basically cannot manipulate content "client side", and then have that data available to the aggregation pipeline stages later. It all needs to be present in the collection you are operating on.
A better approach here is available with MongoDB 3.2 and later, via the $lookup aggregation pipeline operation. Also probably best to handle from the User collection in this case in order to narrow down the selection:
User.aggregate(
[
// Filter first
{ "$match": {
"age": { "$gt": 20 }
}},
// Then join
{ "$lookup": {
"from": "scores",
"localField": "userID",
"foriegnField": "userID",
"as": "score"
}},
// More stages
],
function(err,results) {
}
)
This is basically going to include a new field "score" within the User object as an "array" of items that matched on "lookup" to the other collection:
{
"userID": "abc",
"age": 21,
"score": [{
"userID": "abc",
"score": 42,
// other fields
}]
}
The result is always an array, as the general expected usage is a "left join" of a possible "one to many" relationship. If no result is matched then it is just an empty array.
To use the content, just work with an array in any way. For instance, you can use the $arrayElemAt operator in order to just get the single first element of the array in any future operations. And then you can just use the content like any normal embedded field:
{ "$project": {
"userID": 1,
"age": 1,
"score": { "$arrayElemAt": [ "$score", 0 ] }
}}
If you don't have MongoDB 3.2 available, then your other option to process a query limited by the relations of another collection is to first get the results from that collection and then use $in to filter on the second:
// Match the user collection
User.find({ "age": { "$gt": 20 } },function(err,users) {
// Get id list
userList = users.map(function(user) {
return user.userID;
});
Score.aggregate(
[
// use the id list to select items
{ "$match": {
"userId": { "$in": userList }
}},
// more stages
],
function(err,results) {
}
);
});
So by getting the list of valid users from the other collection to the client and then feeding that to the other collection in a query is the onyl way to get this to happen in earlier releases.