Aggregate data and populate in one request - mongodb

I am a bit puzzled by populate in MongoDB.
I've got a Schema:
import { Schema, Document, model } from "mongoose";
export interface ProductGroupType {
id: Schema.Types.ObjectId,
title: String,
name: String,
description: String,
}
const ProductGroupSchema: Schema<Document<ProductGroupType>> = new Schema({
title: { type: String, trim: true },
name: { type: String, trim: true },
description: { type: String, trim: true },
}, { collection: "productGroups", timestamps: true });
export const ProductGroupModel = model('ProductGroup', ProductGroupSchema);
and products
import { Schema, Document, model } from "mongoose";
import { plugin as autocomplete } from 'mongoose-auto-increment';
const ProductSchema: Schema<Document<IProduct>> = new Schema({
article: Number,
name: String,
category: { type: Schema.Types.ObjectId, ref: 'ProductCategory' },
group: { type: Schema.Types.ObjectId, ref: 'ProductGroup' },
price: { type: Number, default: 0 },
discount: { type: Number, default: 0 },
stock: {
available: { type: Number, default: 0 },
reserved: { type: Number, default: 0 },
},
images: [Object],
description: String,
productDetails: Object,
}, { collection: "products", timestamps: true });
ProductSchema.plugin(autocomplete, {
model: 'Product',
field: 'article',
startAt: 10000,
});
export const ProductModel = model('Product', ProductSchema);
I need to make a request and group on the MongoDB side data by the field 'group'.
I can make this like this:
await ProductModel.aggregate([
{ $match: { category: Types.ObjectId(queryCategory.id) } },
{
$group: {
_id: '$group',
products: {
$push: {
id: '$_id',
name: '$name',
article: '$article',
price: '$price',
discount: '$discount',
description: '$description',
group: '$groupName',
}
},
count: { $sum: 1 },
}
},
]);
but the output here is:
[
{ _id: 61969583ad32e113f87d0e99, products: [ [Object] ], count: 1 },
{
_id: 61993fff452631090bfff750,
products: [ [Object], [Object] ],
count: 2
}
]
almost what I need but I've been playing around with population and I cannot make it work with Aggregation framework.
I already tried to use the 'lookup' operator but it returns an empty array and doesn't want to work.
That's how I wanted to make it work:
const products: Array<IProduct> = await ProductModel.aggregate([
{ $match: { category: Types.ObjectId(queryCategory.id) } },
{
$group: {
_id: '$group',
products: {
$push: {
id: '$_id',
name: '$name',
article: '$article',
price: '$price',
discount: '$discount',
description: '$description',
group: '$groupName',
}
},
count: { $sum: 1 },
}
},
{
$lookup: {
"from": "productGroups",
"localField": "group",
"foreignField": "_id",
"as": "groupName"
},
},
]);
Is it possible to get the same result as I've got now but populate in the same query group field?
So far the only way I've managed to populate it like this as the second request:
await ProductGroupModel.populate( products.map( (product: any) => {
return {
_id: new ProductGroupModel(product),
products: product.products,
count: product.count,
}
} ), { "path": "_id" } )

In a MongoDB aggregation pipeline, the $group stage passes along only those field explicitly declared in the stage.
In the same pipeline you show, the documents passed along by the $group stage would contain the fields:
_id
products
count
When the exector arrives a the $lookup stage, none of the documents contain a field named group.
However, the value previously contained in the group field still exists, in the _id field.
In the $lookup stage, use
"localField": "_id",
to find documents based on that value.

Related

How can I populate nested array objects in mongoDB or mongoose?

I have an orders collection where each order has the following shape:
{
"_id": "5252875356f64d6d28000001",
"lineItems": [
{ productId: 'prod_007', quantity: 3 },
{ productId: 'prod_003', quantity: 2 }
]
// other fields omitted
}
I also have a products collection, where each product contains a unique productId field.
How can I populate each lineItem.productId with a matching product from the products collection? Thanks! :)
EDIT: orderSchema and productSchema:
const orderSchema = new Schema({
checkoutId: {
type: String,
required: true,
},
customerId: {
type: String,
required: true,
},
lineItems: {
type: [itemSubSchema],
required: true,
},
});
const itemSubSchema = new Schema(
{
productId: {
type: String,
required: true,
},
quantity: {
type: Number,
required: true,
},
},
{ _id: false }
);
const productSchema = new Schema({
productId: {
type: String,
required: true,
},
name: {
type: String,
required: true,
},
imageURL: {
type: String,
required: true,
},
price: {
type: Number,
default: 0,
},
});
I don't know the exact output you want but I think this is what you are looking for:
The trick here is to use $lookup in an aggregation stage.
First $unwind to deconstruct the array and can merge each id with the other collection.
Then the $lookup itself. This is like a join in SQL. It merges the desired objects with same ids.
Then recreate the population using $mergeObjects to get properties from both collections.
And last re-group objects to get the array again.
db.orders.aggregate([
{
"$unwind": "$lineItems"
},
{
"$lookup": {
"from": "products",
"localField": "lineItems.productId",
"foreignField": "_id",
"as": "result"
}
},
{
"$set": {
"lineItems": {
"$mergeObjects": [
"$lineItems",
{
"$first": "$result"
}
]
}
}
},
{
"$group": {
"_id": "$_id",
"lineItems": {
"$push": "$lineItems"
}
}
}
])
Example here
With this query you have the same intial data but "filled" with the values from the other collection.
Edit: You can also avoid one stage, maybe it is clear with the $set stage but this example do the same as it merge the objects in the $group stage while pushing to the array.
You can use the Mongoose populate method either when you query your documents or as middleware. However, Mongoose only allows normal population on the _id field.
const itemSubSchema = new Schema({
product: {
type: mongoose.Schema.Types.ObjectId,
ref: 'productSchema',
}
});
const order = await orderSchema.find().populate('lineItems.$*.product');
// special populate syntax necessary for nested documents
Using middleware you would still need to reconfigure your item schema to save the _id from products. But this method would automatically call populate each time you query items:
itemSubSchema.pre('find', function(){
this.populate('product');
});
You could also declare your item schema within your order schema to reduce one layer of joining data:
const orderSchema = new Schema({
lineItems: [{
type: {
quantity: {type: Number, required: true},
product: {
type: mongoose.Schema.Types.ObjectId,
required: true,
ref: 'productSchema',
}
},
required: true,
}]
});
const orders = orderSchema.find().populate('lineItems');

Problem using aggregation in mongodb retrieving data from two collections

i am strugling with a query that i don't know how to perform... I have two collections,
Tarifas Collection
tarifaConfig = new Schema({
producto: { type: String },
titulo: { type: String },
bloqueo: { type: Boolean },
margen: { type: Number },
precioVenta: { type: Number },
precioVentaIva: { type: Number },
})
const tarifaSchema = new Schema({
codigo: { type: String },
titulo: { type: String },
margen: { type: Number },
estado: { type: Boolean },
bloqueo: { type: Boolean },
configs: [tarifaConfig]
})
Producto Collection
const productosSchema = new Schema({
ref: { type: String },
nombre: { type: String },
precioCompra: { type: Number },
precioCompraIva: { type: Number },
precioVenta: { type: Number },
precioVentaIva: { type: Number },
iva: { type: Number },
})
Now i am using an Aggregation method to retrieve both collection in a response
productosModel.aggregate([
{
$match: { _id: ObjectId(req.params.id) }
},
{
$lookup: {
from: "tarifas",
as: "tarifas",
pipeline: []
}
}
]).then((producto) => {
res.json(producto);
})
This is working and gives me both collections in the response... but..
In tarifa's collection i have a propertie called 'configs' that is an array with lot of sub collections... this sub collections are a config of each product that i have,
So what i need to do is, retrieve all tarifas that has a configs for the product, and if the configs does not contain retrieve the tarifa with a empty array.
Expected result
{
ref: 'rbe34',
nombre: 'bike',
precioCompra: 10,
precioCompraIva: 12.1,
precioVenta: "",
precioVentaIva: "",
iva: 21,
tarifas:[
{
codigo: 'NOR',
titulo: 'Normal tarifa',
margen: 33,
estado: true,
bloqueo: true,
configs: [], ///HERE I NEED A EMPTY ARRAY IF THERE IS NOT ANY CONFIG THAT MATCH WITH THE PRODUCT ID,
}
]
}
i tried to add $match in my aggregation pipeline.
productosModel.aggregate([
{
$match: { _id: ObjectId(req.params.id) }
},
{
$lookup: {
from: "tarifas",
as: "tarifas",
pipeline: [
{ $match: { 'configs.producto': req.params.id } }
]
}
}
])
But if there is not any config that match the product it doesn't retrieve the rest of Tarifa's collection
It seems you are trying to $filter the array after you retrieve it.
This pipeline will return only the configs for which the producto field from the config matches the ref field from the product.
[
{
$match: { _id: ObjectId(req.params.id) }
},
{
$lookup: {
from: "tarifas",
as: "tarifas",
pipeline: [
{
$addFields: {
"tarifas.configs":{ $filter:{
input: "$tarifas.configs",
cond: {$eq:["$$this.producto","$ref"]}
} }
}
}
]
}
},
]
Change the fields in the $eq array to the ones you need to match.

Mongoose querying a reference collection data using aggrigate/populate

I am a newbie in MongoDB and Express JS
I have a Product model looks like
const productSchema = mongoose.Schema({
name: String,
title: String,
brand_id: { type: mongoose.Schema.Types.ObjectId, ref: 'brand' },
varient:[{
country_id: { type: mongoose.Schema.Types.ObjectId,ref: 'countries'},
max_retail_price : Number,
profit_margin : Number
}],
and Order model
const orderTransactionSchema = mongoose.Schema({
shop_id: { type: mongoose.Schema.Types.ObjectId, ref: 'shop' },
brand_id:{ type: mongoose.Schema.Types.ObjectId, ref: 'brand' },
product_id: { type: mongoose.Schema.Types.ObjectId, ref: 'product' },
product_varient_id:{ type: mongoose.Schema.Types.ObjectId, ref: 'product.varient._id' },
transaction_id:{ type: mongoose.Schema.Types.ObjectId, ref: 'transaction' }
})
module.exports = mongoose.model('order', orderTransactionSchema );
In my Product collection, each product can have multiple variants.
But in order collection, a user can order only one product variant under a product.
I am trying to display orders with particular product details and variant details, But the problem is when I try to display it using either populate/Aggregate I am getting all the variants in the response array. actually, I want only one product and variant details as in the order collection.
This is what I tried
order.aggregate([{ $match :{} },
{
$lookup: {
from: "products",
localField: "product_id",
foreignField: "_id",
as: "product_data"
}
},
]).exec(function(err,result){
console.log(result);
});
and I am getting the output as
{ _id: 5c8a010b8feeb875abc1b066,
shop_id: 5c7d194ca10ea45c0c03a0ee,
brand_id: 5c41a8c34272c61a176b7639,
product_varient_id: 5c41a9f3f8e1e71aa75b4f32,
transaction_id: 5c6670d5b6c63d0762c6cc77,
product_id: 5c41aac4d45a731af564c433,
product_data:
[ { _id: 5c41aac4d45a731af564c433,
brand_id: 5c41a8c34272c61a176b7639,
image: 'test.jpg',
varient: //getting all the varients here
[ { _id: 5c4ee531bc20b27948b3aa98,
sell_rate_local: 66,
country_id: 5c01149d3c20440a6b2e4928 },
{ _id: 5c4ee53bbc20b27948b3aa99,
sell_rate_local: 66,
country_id: 5c00e1697dd7a23f08bdae68 } ],
__v: 0 } ] } ]
In the Order table, there is porduct_id and product_varient_id is there
I want to populate only the product with product_varient_id.
I also tried something with Populate
order.find().
populate([
{ path: 'shop_id', model: 'shop',select : 'name accounts' }, //it works
{ path: 'transaction_id', model: 'transaction' }, //it wrks
{ path: 'product_varient_id', model: 'product', select:'product.varient.name'},
]).then(result =>
{
// console.log(result);
}).catch(err =>{
// console.log(err);
});
These are the sample product and order document
Order Document Sample :
{
"_id":"5c77a025d65a892f6acf1803",
"shop_id":"5c7d194ca10ea45c0c03a0ee",
"brand_id":"5c41a8b44272c61a176b7638",
"product_varient_id":"5c41a9f3f8e1e71aa75b4f32",
"buy_rate":10,
"buy_rate_after_discount":20,
"product_mrp":30,
"sell_rate":40,
"customer_mobile":123456789,
"status":true,
"transaction_id":"5c6670c9b6c63d0762c6cc76",
"product_id":"5c41a95ff8e1e71aa75b4f30",
"createdAt":"2019-02-28T08:47:33.097Z",
"updatedAt":"2019-02-28T08:47:33.097Z",
"__v":0
}
Product document Sample :
{
"_id":"5c41aac4d45a731af564c433",
"recharge_type":[
"5c00d9cf7dd7a23f08bdae5e"
],
"name":"25 OC - Product 1",
"title":"First installation recharge",
"description":"0.1 credit for first time installation",
"offer_message":"Hi.. You got 0.1 credits..!!",
"brand_id":"5c41a8c34272c61a176b7639",
"buy_rate":20,
"profit_margin":80,
"image":"test.jpg",
"varient":[
{
"_id":"5c4ee531bc20b27948b3aa98",
"display_name":"testlia",
"profit_margin":66,
"max_retail_price":66,
"sell_rate":66,
"sell_rate_local":66,
"country_id":"5c01149d3c20440a6b2e4928"
},
{
"_id":"5c4ee53bbc20b27948b3aa99",
"display_name":"testrinu",
"profit_margin":66,
"max_retail_price":66,
"sell_rate":66,
"sell_rate_local":66,
"country_id":"5c00e1697dd7a23f08bdae68"
}
],
"createdAt":"2019-01-18T10:30:28.991Z",
"updatedAt":"2019-01-28T11:19:23.662Z",
"__v":0
}
MongoDB 3.6 or above have new lookup syntax
db.orders.aggregate([{
$lookup: {
from: "products",
let: {
"productId": "$product_id",
"productVarientId": "$product_varient_id"
},
pipeline: [
{ $match: {
$expr: { $eq: [ "$_id", "$$productId" ]}
}
},
{ $addFields: {
varient: {
$filter: {
input: "$varient",
as: "varient",
cond: { $eq: [ "$$productVarientId", "$$varient._id" ] }
}
}
}
}
],
as: "product_data"
}
}])
You can check datasample Here

MongoDB Query from multiple models / schemas and return in one field

I am using Nodejs and MongoDB, mongoose and expressjs, creating a Blog API having users, articles, likes & comments schema. Below are schemas that I use.
const UsersSchema = new mongoose.Schema({
username: { type: String },
email: { type: String },
date_created: { type: Date },
last_modified: { type: Date }
});
const ArticleSchema = new mongoose.Schema({
id: { type: String, required: true },
text: { type: String, required: true },
posted_by: { type: Schema.Types.ObjectId, ref: 'User', required: true },
images: [{ type: String }],
date_created: { type: Date },
last_modified: { type: Date }
});
const CommentSchema = new mongoose.Schema({
id: { type: String, required: true },
commented_by: { type: Schema.Types.ObjectId, ref: 'User', required: true },
article: { type: Schema.Types.ObjectId, ref: 'Article' },
text: { type: String, required: true },
date_created: { type: Date },
last_modified: { type: Date }
});
What I actually need is when I * get collection of articles * I also want to get the number of comments together for each articles. How do I query mongo?
Since you need to query more than one collection, you can use MongoDB's aggregation.
Here: https://docs.mongodb.com/manual/aggregation/
Example:
Article
.aggregate(
{
$lookup: {
from: '<your comments collection name',
localField: '_id',
foreignField: 'article',
as: 'comments'
}
},
{
$project: {
comments: '$comments.commented_by',
text: 1,
posted_by: 1,
images: 1,
date_created: 1,
last_modified: 1
}
},
{
$project: {
hasCommented: {
$cond: {
if: { $in: [ '$comments', '<user object id>' ] },
then: true,
else: false
}
},
commentsCount: { $size: '$comments' },
text: 1,
posted_by: 1,
images: 1,
date_created: 1,
last_modified: 1
}
}
)
The aggregation got a little big but let me try to explain:
First we need to filter the comments after the $lookup. So we $unwind them, making each article contain just one comment object, so we can filter using $match(that's the filter stage, it works just as the <Model>.find(). After filtering the desired's user comments, we $group everything again, $sum: 1 for each comment, using as the grouper _id, the article's _id. And we get the $first result for $text, $images and etc. Later, we $project everything, but now we add hasCommented with a $cond, simply doing: if the $comments is greater than 0(the user has commented, so this will be true, else, false.
MongoDB's Aggregation framework it's awesome and you can do almost whatever you want with your data using it. But be aware that somethings may cost more than others, always read the reference.

How to Check current user's vote before votes are grouped and sumed in same aggregate function

var PostSchema = new mongoose.Schema({
item: {
type: mongoose.Schema.ObjectId,
ref: 'item',
required: true
},
user: {
type: mongoose.Schema.ObjectId,
ref: 'User',
required: true
},
vote: {
type: Number,
default: 0
},
total: {
type: Number,
default: 0
},
awsPostKey: {type: String},
picture: {type: String, required: true}
});
var data = function(){
return Post
.find({})
.then(function(post){
return post;
})
};
var userId = //mongo objectId for current user
//postVote schema:
var PostVoteSchema = new mongoose.Schema({
post: {
type: mongoose.Schema.ObjectId,
ref: 'Post',
required: true
},
user: {
type: mongoose.Schema.ObjectId,
ref: 'User',
required: true
},
vote: {
type: Number,
default: 0
}
});
//pass data from Post query to PostVote sum function:
PostVoteSchema.statics.sum = function (data, userId) {
var postIds = data.map(function (a) {
return a._id;
});
return PostVote
.aggregate(
[
{ $match: { 'post': {$in: postIds}}},
{ $group: { _id:'$post' ,vote:{$sum:'$vote'}}}
])
.execAsync()
.then(function(votes){
return votes;
//desired output to client, _id is for specific post
{_id: 5802ea4bc00cb0beca1972cc, vote: 3, currentUserVote: -1}
});
};
I'm successfully able to get the total sum of all votes with the same postId.
Now, I"m wanting to see if the current user (userId) has placed a vote for the given post as well, then to return how they voted (+1 or -1) along with the sum of all votes for the specific post.
Is it possible to do this, or will I have to do this outside of my aggregate pipeline -- within a second query? It just seems potentially taxing to have to query the collection again.
Yes, that's possible. Within the $group pipeline, you can use the $cond operator as the logic for feeding the $sum accumulator operator. For example:
return PostVote.aggregate([
{ "$match": { "post": { "$in": postIds } } },
{
"$group": {
"_id": "$post",
"votes": { "$sum": "$vote" },
"userVotes": {
"$sum": {
"$cond": [
{ "$eq": ["$user", userId] },
"$vote",
0
]
}
}
}
}
]).execAsync().then(function(votes){
return votes;
});