Automatic data exclusion in aggregations with Mongoose - mongodb

In my Mongoose models, I've set sensitive data to select: false so it isn't returned with queries by default:
const userSchema = new mongoose.Schema<User, UserModelType, {}, UserQueryHelpers>({
username: {...},
email: { type: String, required: true, unique: true, validate: [validator.isEmail, 'invalid email'], select: false },
password: { type: String, required: true, select: false },
profilePicUrl: { type: String },
emailVerified: { type: Boolean, required: true, default: false, select: false },
[...]
});
To my shock, I realized that this doesn't work in all scenarios. Namely lookup queries in aggregations don't respect select: false.
const result = await KarmaLogEntry.aggregate()
.match({
createdAt: { $gte: startOfCurrentMonth }
})
.group({
_id: "$user",
totalPoints: { $sum: "$points" }
})
.sort({ 'totalPoints': -1 })
.limit(10)
.lookup({
from: "users",
localField: "_id",
foreignField: "_id",
as: "user",
})
.unwind('$user')
.exec();
This returns all the sensitive fields!
I know that I can exclude fields with an additional $project stage:
.lookup({
from: "users",
localField: "_id",
foreignField: "_id",
as: "user",
pipeline: [
{
$project: {
username: '$username',
displayName: '$displayName',
profilePicUrl: '$profilePicUrl',
}
}
]
})
However, if I ever forget to add this for an endpoint, I will leak sensitive user data. Is there a way to apply select: false to all forms of MongoDB queries, including aggregation and lookups?

Related

Perform share calculations in MongoDB aggregation

I have an Order schema, like so:
const orderSchema = new mongoose.Schema({
order_items: [{
type: mongoose.Schema.Types.ObjectId,
ref: 'OrderItem',
required: true
}],
user: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User',
required: true
}
total_price: {
type: Number
}
});
And the OrderItems contains purchased products, like so:
const orderItemSchema = new mongoose.Schema({
product_id: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Product',
required: true
},
quantity: {
type: Number,
required: true
}
});
And the Product schema like so:
const productSchema = new mongoose.Schema({
name: {
type: Map,
of: String,
required: true
},
thumbnail: {
type: String,
default: ''
},
unit_price: {
type: Number,
required: true
}
});
I'm trying to get the share of each purchased product from the total price of the order.
I tried the following:
const totalSales = await Order.aggregate([
{
$lookup: {
from: "orderitems",
localField: "order_items",
foreignField: "_id",
as: "order_items"
}
},
{
$lookup: {
from: "products",
localField: "order_items.product_id",
foreignField: "_id",
as: "products",
pipeline: []
}
},
{
$project: {
order_items: 0,
products: { $divide: ['$products.unit_price', '$total_price'] }
}
}
]);
But I got the following error in postman:
Invalid $project :: caused by :: Cannot use expression other than
$meta in exclusion projection
So, how can I get the desired output?
Thanks
Edit:
I removed order_items: 0 from the project, and now I got this error message:
PlanExecutor error during aggregation :: caused by :: $divide only
supports numeric types, not array and int
From the documentation,
If you specify the exclusion of a field other than _id, you cannot
employ any other $project specification forms.
You should remove order_items: 0, because it is anyway not included in the output.

Mongoose query on multiple populated fields

I have three collections i.e Events, News and FuneralNews.
I have another Notifications collection that is just a combination of all three and contains the reference Id of either/one of the three collections.
I want to fetch only those Notifcations whose eventId OR newsId OR funeralNewsId field isActive is true
EventsSchema:
var EventsSchema = new Schema({
...
isActive: Boolean
});
FuneralNewsSchema:
var FuneralNewsSchema = new Schema({
...
isActive: Boolean
});
NewsSchema:
var NewsSchema = new Schema({
...
isActive: Boolean
});
NotificationSchema:
var NotificationSchema = new Schema({
type: {
type: String,
required: true
},
creationDate: {
type: Date,
default: Date.now
},
eventId: {type: Schema.Types.ObjectId, ref: 'Events'},
newsId: {type: Schema.Types.ObjectId, ref: 'News'},
funeralNewsId: {type: Schema.Types.ObjectId, ref: 'FuneralNews'},
organisationId: {type: Schema.Types.ObjectId, ref: 'Organization'}
});
This was my query before I need a check on isActive property of referenced collection:
let totalItems;
const query = { organisationId: organisationId };
Notification.find(query)
.countDocuments()
.then((count) => {
totalItems = count;
return Notification.find(query, null, { lean: true })
.skip(page * limit)
.limit(limit)
.sort({ creationDate: -1 })
.populate("eventId")
.populate("newsId")
.populate("funeralNewsId")
.exec();
})
.then((notifications, err) => {
if (err) throw new Error(err);
res.status(200).json({ notifications, totalItems });
})
.catch((err) => {
next(err);
});
Now I don't know how to check on isActive field of three populated collections prior population.
I have seen other questions like this and this but being a newbie can't edit it according to my use-case. Any help would be highly appreciated
use $lookup for each objectId refrence
then group by _id of null to get data and add myCount as total number put original data to array
and use unwind to destruct array and use addField
model
.aggregate([
{
$lookup: {
from: "Events", // events collection name
localField: "eventId",
foreignField: "_id",
as: "events",
},
},
{
$lookup: {
from: "FuneralNews", //FuneralNews collection name
localField: "funeralNewsId",
foreignField: "_id",
as: "funeralnews",
},
},
{
$lookup: {
from: "News", // news collection name
localField: "newsId",
foreignField: "_id",
as: "news",
},
},
{
$match: {
$or: [
{ "news.isActive": true },
{ "events.isActive": true },
{ "funeralnews.isActive": true },
],
},
},
{
$group: {
_id: null,
myCount: {
$sum: 1,
},
root: {
$push: "$$ROOT",
},
},
},
{
$unwind: {
path: "$root",
},
},
{
$addFields: {
"root.total": "$myCount",
},
},
{
$replaceRoot: {
newRoot: "$root",
},
},
{
$sort: {
creationDate: -1,
},
},
])
.skip(page * limit)
.limit(limit);

How can I use the $match operator twice in a single mongoDB query?

I have two models:
const ClientRequest = new mongoose.Schema({
sourceLanguage: {
type: String,
default: '',
trim: true
},
type: {
type: String,
default: '',
trim: true
},
customer: {
type: Schema.Types.ObjectId, ref: 'Client'
}
}
and
const Client = new mongoose.Schema({
name: {
type: String,
default: '',
trim: true
},
web: {
type: String,
default: '',
trim: true
},
country: {
type: String,
default: '',
trim: true
}
}
And I need to find all requests filtered by sourceLanguage and name.
I'm using this query:
const requests = await ClientRequest.aggregate([
{$match: {
"sourceLanguage.symbol": "EN-GB"}
},
{
$lookup: {
from: "clients",
localField: "customer",
foreignField: "_id",
as: "clients"
}
},
{
$match: {
"clients.name": filters.clientFilter,
}
}
])
But it returns empty array. If I remove one of the $match it works. But how can I use both of the filters at the same time in a single query?
const requests = await ClientRequest.aggregate([
{$match: {
"sourceLanguage": "EN-GB",
"customer": ObjectId("5d933c4b8dd2942a17fca425")
}
},
{
$lookup: {
from: "clients",
localField: "customer",
foreignField: "_id",
as: "clients"
}
},
])
I tried different approaches, but as sometimes it happens, the simpliest way worked out:
const requests = await ClientRequest.aggregate([
{
$lookup: {
from: "clients",
localField: "customer",
foreignField: "_id",
as: "customer" // I used the same name to replace the Id with the unwinded object
}
},
{
$match: {
"customer.name": filters.clientFilter,
"sourceLanguage.symbol": "EN-GB" // or any other filter
}
},
{$unwind: "$customer"} // here I just unwind from array to an object
])

cross collection query in mongoose with $lookup

I am building a question-answers database. I have three schemas, User, Question and Replies(answers). My problem begins where I want to query just the questions that the user has not already answered. but when I run this query I get the question with an empty replies array. should I populate it somehow? here's where I got:
Query:
let getQuestions = (req, res)=>{
Question.aggregate([
{
$match: {isGlobal: true}
},
{
$lookup: {
from: 'Reply',
localField: '_id',
foreignField: 'questionID',
as: 'replies'
}
},
// {
// $match: {}
// }
]).then(foundQuestions=> console.log(foundQuestions))
};
User Schema (simplified):
const mongoose = require('mongoose');
const userSchema = mongoose.Schema({
firstName: String,
lastName: String,
email: String,
questionReplies: [
{
type: mongoose.Schema.Types.ObjectId,
ref: 'Reply'
}
],
}, {timestamps: true});
module.exports = mongoose.model('User', userSchema);
Question Schema:
const mongoose = require('mongoose');
const questionSchema = mongoose.Schema({
title: String,
description: String,
isGlobal: Boolean,
options: [
{
number: Number,
title: String,
}
]
}, {timestamps: true});
module.exports = mongoose.model('Question', questionSchema);
Reply Schema:
const mongoose = require('mongoose');
const replySchema = mongoose.Schema({
questionID: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Question',
required: true
},
email: {type: String, required: true},
answerID: {type: String, required: true},
number: Number,
title: String,
}, {timestamps: true});
module.exports = mongoose.model('Reply', replySchema);
my replies collection has this document inside:
"questionID" : ObjectId("5c6f6867cbff9c2a9004eb6d")
and I have a question with this ID:
"_id" : ObjectId("5c6f6867cbff9c2a9004eb6d"),
(any opnions on improving database design are welcome too).
Try below:
let getQuestions = (req, res)=>{
Question.aggregate([
{
$match: {isGlobal: true}
},
{
$lookup: {
from: 'Reply',
localField: '_id',
foreignField: 'questionID',
as: 'replies'
}
},
{
$match: { "replies": { $eq: [] } }
}
]).then(foundQuestions=> console.log(foundQuestions))
};
So you will get question which does not have any replies.
I solved my problem and I post here for future references.
the from field used in $lookup refers to the collection name created by mongodb, not the Model name used in mongoose. So the correct query is:
$lookup: {
from: 'replies',
localField: '_id',
foreignField: 'questionID',
as: 'replies'
}
and then I added
{
$match: { "replies": { $eq: [] } }
}
for finding just the questions with no answers.

MongoDB aggregrate two collections on foreign key

I am attempting to pull data from my Mongo instance that will show all users for a given account Id. Users can be part of more than one account so I have currently got this structure for my Mongo models:
UserModel:
username: {
type: String,
required: true,
unique: true,
lowercase: true
},
name: {
type: String,
required: true
},
password: {
type: String,
required: true
},
email: {
type: String,
required: true,
unique: true
},
UsersToAccountModel:
{
user: {
type: Schema.Types.ObjectId,
ref: 'users'
},
userGroup: {
type: Schema.Types.ObjectId,
ref: 'userGroups'
},
account: {
type: Schema.Types.ObjectId,
ref: 'accounts'
},
default: {
type: Boolean,
default: null
}
}
The UsersToAccount model collection holds the ObjectId of the user, account, and userGroup in it's fields so is acting as a link collection as such.
For every userId that matches the given account ID in the UsersToAccount collection I want to take that ID and query the users table and return it. In MYSQL the query would be:
SELECT * FROM userToAccounts u2a LEFT JOIN users u ON u.id = u2a.userId WHERE u2a.account = $accountId
Can anyone help me here?I have tired aggregation but I am not getting very far.
Here is my attempt so far which isn't working:
const users = await this.userToAccountModel.aggregate(
{
$match: { account: requestVars.account },
},
{
$lookup : { from: "users", localField: "_id", as: "userData", foreignField: "user"}
}
);
Thanks
Firstly, the $lookup stage has the following syntax:
{
$lookup: {
from: <collection to join>,
localField: <field from the input documents>,
foreignField: <field from the documents of the "from" collection>,
as: <output array field>
}
}
So I think you need to swap the localField and foreignField values.
Secondly, aggregate() expect an array.
const users = await this.userToAccountModel.aggregate([
{
$match: { account: requestVars.account },
},
{
$lookup : { from: "users", localField: "user", foreignField: "_id", as: "userData" }
}
]);