mongoDB, mongoose - aggregation an array of objects - mongodb

I have 3 collections to aggregate.
1st is colors collection
{
{
_id: 1, <- mongoose objectId
name: red
},
{
_id: 2, <- mongoose objectId
name: green
}
}
2nd is products
{
{
_id: Id777, <- mongoose objectId
productName: test prod 777
},
{
_id: Id888, <- mongoose objectId
productName: test prod 888
}
}
and 3rd it move collection
{
....other fields here
items: [
{
_id: an mongoose id,
itemId: Id777 <- in products collection,
itemColor: 1 <- id in colors collection,
coutn: 7,
....other fields
},
{
_id: an mongoose id,
itemId: Id888 <- in products collection,
itemColor: 2 <- id in colors collection
cout: 10
....other fields
}
]
}
I need to have an output like this:
{
////information from collection
items: [
{
itemId: test prod 777, itemColor: red, count: 7
},
{
itemId: test prod 888, itemColor: green, count: 10
}
]
}
My code is:
const moves = await ProductMoves.aggregate([
{ $match: query }, // this is my query
{
$lookup: {
from: 'products',
localField: 'items.itemId',
foreignField: '_id',
as: 'productName'
}
},
{
$unwind: { path: "$productName" , preserveNullAndEmptyArrays: true }
},
{
$lookup: {
from: 'colors',
localField: 'items.itemColor',
foreignField: '_id',
as: 'cName'
}
},
{
$unwind: { path: "$cName" , preserveNullAndEmptyArrays: true }
},
{
$addFields: {
mItems: {
prName: "$productName.productName",
prColor: "$cName.colorName"
},
productName: 0,
cName: 0
}
}
])
.sort({addedDate: -1})
.skip(+req.query.offset)
.limit(+req.query.limit)
but it returns only 1 element from the object array. probably I need something like a for loop, but i couldn't do it.
thank you for your responses, and have a good day!

$unwind deconstruct items array
$lookup with products collection
$lookup with colors collection
$addFields, $arrayElemAt to get first element from lookup result
$group by _id and reconstruct items array and pass other fields as well
there is no external methods in an aggregate function, you have to use stages for sort, skip and limit like below
$sort by addedDate in descending order
$skip and $limit result
const moves = await ProductMoves.aggregate([
{ $match: query }, // this is my query
{ $unwind: "$items" },
{
$lookup: {
from: "products",
localField: "items.itemId",
foreignField: "_id",
as: "itemId"
}
},
{
$lookup: {
from: "colors",
localField: "items.itemColor",
foreignField: "_id",
as: "itemColor"
}
},
{
$addFields: {
"items.itemId": { $arrayElemAt: ["$itemId.productName", 0] },
"items.itemColor": { $arrayElemAt: ["$itemColor.name", 0] }
}
},
{
$group: {
_id: "$_id",
items: { $push: "$items" },
addedDate: { $first: "$addedDate" }
// add other fields that you want in result like "addedDate"
}
},
{ $sort: { addedDate: -1 } },
{ $skip: +req.query.offset },
{ $limit: +req.query.limit }
])
Playground

Related

Aggregate between referenced collections in MongoDB

Couldn't find an answer here
I got a collection of segments, where an example segment looks like this:
{
_id: {
$oid: "62e5778e34362ad54db6b3f9"
},
name: "Good-Natured Developers"
}
And a collection of users, where an example user looks like this:
{
_id: {
$oid: "62e5225dfdc41ba4f7effefa"
},
age: {
$numberInt: "43"
},
segment_ids: [
{
$oid: "62e577a034362ad54db6b40a"
},
{
$oid: "62e5782f34362ad54db6b497"
},
]
}
How can I query how many users are there in every segment?
I tried this aggregation:
const usersInSegment = await userCollection
.aggregate([
{ $unwind: '$users'},
{ $project: { "_id": 1} },
{
$lookup: {
from: 'segments',
localField: 'segment_ids',
foreignField: '_id',
as: 'joined'
}
}
])
.limit(25)
.toArray()
But I get an empty array.

MongoDB Aggregation - How to keep only docs that has related value in foreign collection

In the lookup part in the aggregate method, how can I keep only documents that have a value in the foreign collection?
For instance, I have this collection users:
[
{ _id: 1, name: 'John', basketId: 4 },
{ _id: 2, name: 'mari', basketId: 9 },
{ _id: 3, name: 'tedd', basketId: 32 },
{ _id: 4, name: 'sara', basketId: 14 },
{ _id: 5, name: 'jane', basketId: 3 },
.
.
.
]
And another collection named baskets
[
{ _id: 1, items: 0 },
{ _id: 2, items: 2 },
{ _id: 3, items: 0 },
{ _id: 4, items: 0 },
{ _id: 5, items: 7 },
.
.
.
]
Now if I want to get users with basket items greater than 0, I use aggregate and lookup:
UserModel.aggregate([
{ $lookup:
{
from: 'baskets',
localField: 'basketId',
foreignField: '_id',
pipeline: [{ $match: { items: { $gt: 0 } } }],
as: 'basket'
}
}
])
It brings up ALL users with their basket data. For those users whose basket items are 0, it shows basket: [].
But I need to get ONLY users that have basket items greater than 0. How can it be done?
You shouldn't place the $match stage in the pipeline of $lookup. As what it did is filter the documents to be returned in the basket array.
Instead, you need a $match stage to filter the documents by comparing the first document's items value in the basket array.
UserModel.aggregate([
{
$lookup: {
from: "baskets",
localField: "basketId",
foreignField: "_id",
as: "basket"
}
},
{
$match: {
$expr: {
$gt: [
{
$first: "$basket.items"
},
0
]
}
}
}
])
Demo 1 # Mongo Playground
The question is ambiguous. You may look for the below query as well (but would return the same result as Demo 1):
UserModel.aggregate([
{
$lookup: {
from: "baskets",
localField: "basketId",
foreignField: "_id",
pipeline: [
{
$match: {
items: {
$gt: 0
}
}
}
],
as: "basket"
}
},
{
$match: {
$expr: {
$gt: [
{
$size: "$basket"
},
0
]
}
}
}
])
Or check is not an empty array
{
$ne: [
"$basket",
[]
]
}
Demo 2 # Mongo Playground

$Lookup inside an array with more properties MongoDB

I am trying $lookup in Array but can't merge it in my object after $lookup.
Collection
_id: '5f7a1053477c8a1ae88e22cf',
name: 'Demo'
price: 423,
related: [
{
idProduct: '61140763ab806726a8ab7aea'
quantity: 2
},
{
idProduct: '61140763ab806726a8ab7aeb'
quantity: 6
},
]
Expected Output:
_id: '5f7a1053477c8a1ae88e22cf',
name: 'Demo',
price: 423,
related: [
{
idProduct: {
_id: '61140763ab806726a8ab7aea',
name: 'related1',
price: 22
},
quantity: 2
},
{
idProduct: {
_id: '61140763ab806726a8ab7aeb',
name: 'related2',
price: 53
},
quantity: 6
},
]
I need $lookup, idProduct and keep that data structure.
Any help? Thanks in advance
$unwind deconstruct related array
$lookup with collection 2, pass related.idProduct as local field and _id as foreign field and set in related.idProduct
$group by _id and reconstruct related array and get first required firlds
db.col1.aggregate([
{ $unwind: "$related" },
{
$lookup: {
from: "col2",
localField: "related.idProduct",
foreignField: "_id",
as: "related.idProduct"
}
},
{
$group: {
_id: "$_id",
name: { $first: "$name" },
price: { $first: "$price" },
related: { $push: "$related" }
}
}
])
Playground
The second approach without using $unwind stage,
$lookup with collection 2,
$map to iterate loop of related array
$filter to iterate loop of col2 array by idProduct field
$arrayElemAt to get first element from above filtered result
$mergeObjects to merge current object with updated idProduct field
db.col1.aggregate([
{
$lookup: {
from: "col2",
localField: "related.idProduct",
foreignField: "_id",
as: "col2"
}
},
{
$addFields: {
col2: "$$REMOVE",
related: {
$map: {
input: "$related",
as: "r",
in: {
$mergeObjects: [
"$$r",
{
idProduct: {
$arrayElemAt: [
{
$filter: {
input: "$col2",
cond: { $eq: ["$$r.idProduct", "$$this._id"] }
}
},
0
]
}
}
]
}
}
}
}
}
])
Playground

MongoDB aggregation of array in single document

I have 3 collections to aggregate.
1st is colors collection with colors details, name value etc
{
{
_id: 1, <- mongoose objectId
name: red
},
{
_id: 2, <- mongoose objectId
name: green
}
}
2nd is products (products details)
{
{
_id: Id777, <- mongoose objectId
productName: test prod 777
},
{
_id: Id888, <- mongoose objectId
productName: test prod 888
}
}
and 3rd it move collection. here is the array of object to aggregation
{
....other fields here
items: [
{
_id: an mongoose id,
itemId: Id777 <- in products collection,
itemColor: 1 <- id in colors collection,
coutn: 7,
....other fields
},
{
_id: an mongoose id,
itemId: Id888 <- in products collection,
itemColor: 2 <- id in colors collection
cout: 10
....other fields
}
]
}
this code works good if I get many documents from collection.
but when I try to get only 1 document by _id, its didn't working
const query = {
$and: [
{appId: req.user.appId},
{_id: req.params._id} <- here is mongodb id
],
const moves = await ProductMoves.aggregate([
{ $match: query }, // this is my query
{ $unwind: "$items" },
{
$lookup: {
from: "products",
localField: "items.itemId",
foreignField: "_id",
as: "itemId"
}
},
{
$lookup: {
from: "colors",
localField: "items.itemColor",
foreignField: "_id",
as: "itemColor"
}
},
{
$addFields: {
"items.itemId": { $arrayElemAt: ["$itemId.productName", 0] },
"items.itemColor": { $arrayElemAt: ["$itemColor.name", 0] }
}
},
{
$group: {
_id: "$_id",
items: { $push: "$items" },
addedDate: { $first: "$addedDate" }
}
}
])
So, how I can get only 1 document by _id and aggregate it? Thank you!

Operation timeout for a MongoDB aggregation pipeline

I have a MongodDB database on MongoDB Atlas.
It has an "orders", "products", "itemTypes" and "brands".
"orders" only keep track of product id ordered.
"products" only keep track of brand id and itemType id
"itemTypes" keep track of item type name
"brands" keep track of brand name.
If I aggregate orders + products + itemTypes it is ok:
[{
$unwind: {
path: '$orders'
}
}, {
$lookup: {
from: 'products',
localField: 'orders.productId',
foreignField: 'productId',
as: 'products'
}
}, {
$lookup: {
from: 'itemTypes',
localField: 'products.typeId',
foreignField: 'typeId',
as: 'itemTypes'
}
}, {
$set: {
'orders.price': {
$arrayElemAt: ['$products.price', 0]
},
'orders.brandId': {
$arrayElemAt: ['$products.brandId', 0]
},
'orders.typeId': {
$arrayElemAt: ['$products.typeId', 0]
},
'orders.typeName': {
$arrayElemAt: ['$itemTypes.name', 0]
}
}
}, {
$group: {
_id: '$_id',
createdAt: {
$first: '$createdAt'
},
status: {
$first: '$status'
},
retailerId: {
$first: '$retailerId'
},
retailerName: {
$first: '$retailerName'
},
orderId: {
$first: '$orderId'
},
orders: {
$push: '$orders'
}
}
}]
If I aggregate orders + products + itemTypes + brands, either Mongo Compass or the web UI of Mongo Atlas aggregation builder will give operation timeout error.
[{
$unwind: {
path: '$orders'
}
}, {
$lookup: {
from: 'products',
localField: 'orders.productId',
foreignField: 'productId',
as: 'products'
}
}, {
$lookup: {
from: 'itemTypes',
localField: 'products.typeId',
foreignField: 'typeId',
as: 'itemTypes'
}
}, {
$lookup: {
from: 'brands',
localField: 'products.brandId',
foreignField: 'brandId',
as: 'brands'
}
}, {
$set: {
'orders.price': {
$arrayElemAt: ['$products.price', 0]
},
'orders.brandId': {
$arrayElemAt: ['$products.brandId', 0]
},
'orders.typeId': {
$arrayElemAt: ['$products.typeId', 0]
},
'orders.typeName': {
$arrayElemAt: ['$itemTypes.name', 0]
},
'orders.brandName': {
$arrayElemAt: ['$brands.name', 0]
}
}
}, {
$group: {
_id: '$_id',
createdAt: {
$first: '$createdAt'
},
status: {
$first: '$status'
},
retailerId: {
$first: '$retailerId'
},
retailerName: {
$first: '$retailerName'
},
orderId: {
$first: '$orderId'
},
orders: {
$push: '$orders'
}
}
}]
This is a demo of the aggregation that timed out:
https://mongoplayground.net/p/Jj6EhSl58MS
We have approximately 50k orders, 14k products, 200 brands, 89 item types.
Is there anyway to optimise this aggregation so that it won't timeout?
P/s: My ultimate goal is to visualise popular brands and item types ordered using beautiful chart in the Mongodb Charts function.
If you are on Mongo Atlas, you can use Triggers to run the aggregation query in the background - either when the database is updated or as a scheduled trigger (https://docs.mongodb.com/realm/triggers/).
When the trigger runs, you can save the result of the aggregation pipeline in a new collection using the "$merge" operation.
exports = function() {
const mongodb = context.services.get(CLUSTER_NAME);
const orders = mongodb.db(DATABASE_NAME).collection("orders");
const ordersSummary = mongodb.db(DATABASE_NAME).collection("orders.summary");
const pipeline = [
{
YOUR_PIPELINE
},
{ $merge: { into: "orders.summary", on: "_id", whenMatched: "replace", whenNotMatched: "insert" } }
];
orders.aggregate(pipeline);
};
This way, your charts will be very fast, since they only have to do a simple query from the new collection.
Do you have index on the collections you $lookup from:
products (productId) + itemTypes (typeId) + brands (brandId).
Otherwise, the lookups can take a long time to complete.