I am evaluating MongoDB for an application and I am trying to learn how to use it.
I don't know whether what I want to achieve is possible, so I am prepared for "No" as an answer.
Suppose the three collections described below (in mongoplayground-style format)
db={
packages: [
{
_id: 10,
name: "small box"
},
{
_id: 20,
name: "big box"
}
],
shipments: [
{
customer: "bob",
items: [
{
_id: 12312,
package_id: 20,
weight: 9.99,
},
{
_id: 65489,
package_id: 10,
weight: 1.5
}
]
}
]
}
I want to use the aggregation framework to produce a document a shipment, where for each item a new property is added that contains the information regarding the package (example given below)
{
customer: "bob",
items: [
{
_id: 12312,
package_id: 20,
weight: 9.99,
package: {
_id: 20,
name: "big box"
}
},
{
_id: 65489,
package_id: 10,
weight: 1.5,
package: {
_id: 10,
name: "small box"
}
}
]
}
I have tried using $lookup without a pipeline, but I can only get up to a point where I replace each item document with the corresponding package document (which of course is not what I want to achieve), and I am lost with using $lookup with a pipeline (which if I were a betting man, I'd bet is the way to achieve what I want). I kind of got somewhere by $unwind'ing the items array, followed by a $lookup and a $group / $push but then I am not sure how to retrieve the other fields of a shipment document other than the items (and I have a feeling this is not the proper way to achieve the desired result).
I can post more code if needed (what I have tried so far), but I am trying to keep the question within a reasonable length
Any help would be appreciated, as I am sure I could be trying for days to produce the sample I want and I am not even sure it is possible.
db.shipments.aggregate([
{
$lookup: {
from: "packages",
localField: "items.package_id",
foreignField: "_id",
as: "packages"
}
},
{
$project: {
customer: 1,
items: {
$map: {
input: "$items",
as: "item",
in: {
"$mergeObjects": [
{
"_id": "$$item._id",
"package_id": "$$item.package_id",
"weight": "$$item.weight"
},
{
"package": {
"$arrayElemAt": [
{
"$filter": {
"input": "$packages",
"as": "package",
"cond": {
"$eq": [
"$$item.package_id",
"$$package._id"
]
}
}
},
0
]
}
}
]
}
}
}
}
}
])
mongoplayground
Try this:
db.shipments.aggregate([
{ $unwind: "$items" },
{
$lookup: {
from: "packages",
let: { package_id: "$items.package_id" },
pipeline: [
{
$match: {
$expr: {
$eq: ["$_id", "$$package_id"]
}
}
}
],
as: "items.package"
}
},
{ $unwind: "$items.package" },
{
$group: {
_id: "$_id",
customer: { $first: "$customer" },
items: { $push: "$items" }
}
}
])
Solution in Playground
Related
How can I count the number of completed houses designed by a specific architect in MongoDB?
I have the next two collections, "plans" and "houses".
Where the only relationship between houses and plans is that houses have the id of a given plan.
Is there a way to do this in MongoDB with just one query?
plans
{
_id: ObjectId("6388024d0dfd27246fb47a5f")
"hight": 10,
"arquitec": "Aneesa Wade",
},
{
_id: ObjectId("1188024d0dfd27246fb4711f")
"hight": 50,
"arquitec": "Smith Stone",
}
houses
{
_id: ObjectId
"plansId": "6388024d0dfd27246fb47a5f" -> string,
"status": "under construction",
},
{
_id: ObjectId
"plansId": "6388024d0dfd27246fb47a5f" -> string,
"status": "completed",
}
What I tried was to use mongo aggregations while using $match and $lookup.
The "idea" with clear errors would be something like this.
db.houses.aggregate([
{"$match": {"status": "completed"}},
{
"$lookup": {
"from": "plans",
"pipeline": [
{
"$match": {
"$expr": {
"$and": [
{ "$eq": [ "houses.plansId", { "$toString": "$plans._id" }]},
{ "plans.arquitec" : "Smith Stone" },
]
}
}
},
],
}
}
If it's a single join condition, simply do a project to object ID to avoid any complicated lookup pipelines.
Example playground - https://mongoplayground.net/p/gaqxZ7SzDTg
db.houses.aggregate([
{
$match: {
status: "completed"
}
},
{
$project: {
_id: 1,
plansId: 1,
status: 1,
plans_id: {
$toObjectId: "$plansId"
}
}
},
{
$lookup: {
from: "plans",
localField: "plans_id",
foreignField: "_id",
as: "plan"
}
},
{
$project: {
_id: 1,
plansId: 1,
status: 1,
plan: {
$first: "$plan"
}
}
},
{
$match: {
"plan.arquitec": "Some One"
}
}
])
Update: As per OP comment, added additional match stage for filtering the final result based on the lookup response.
I have two collections, one being Companies and the others being Projects. I am trying to write an aggregation function that first grabs all Companies with the status of "Client", then from there write a pipeline that will return all filtered Companies where the company._id === project.companyId, as an Array of Objects. An example of the shortened Collections are below:
Companies
{
_id: ObjectId('2341908342'),
companyName: "Meta",
address: "123 Facebook Lane",
status: "Client"
}
Projects
{
_id: ObjectId('234123840'),
companyId: '2341908342',
name: "Test Project",
price: 97450,
}
{
_id: ObjectId('23413456'),
companyId: '2341908342',
name: "Test Project 2",
price: 100000,
}
My desired outcome after the Aggregation:
Companies
{
_id: ObjectId('2341908342'),
companyName: "Meta",
address: "123 Facebook Lane",
projects: [ [Project1], [Project2],
}
The projects field does not currently exist on the Companies collection, so I imagine we would have to add it. I also begun writing a $match function to filter by clients, but I am not sure if this is correct. I am trying to use $lookup for this but can not figure out the pipeline. Can anyone help me?
Where I'm currently stuck:
try {
const allClientsWithProjects = await companyCollection
.aggregate([
{
$match: {
orgId: {
$in: [new ObjectId(req.user.orgId)],
},
status: { $in: ["Client"] },
},
},
{
$addFields: {
projects: [{}],
},
},
{
$lookup: { from: "projects", (I am stuck here) },
},
])
.toArray()
Thank you for any help anyone can provide.
UPDATE*
I am seemingly so close I feel like... This is what I have currently, and it is returning everything but Projects is still an empty array.
try {
const allClients = await companyCollection
.aggregate([
{
$match: {
orgId: {
$in: [new ObjectId(req.user.orgId)],
},
status: {
$in: ["Client"],
},
},
},
{
$lookup: {
from: "projects",
let: {
companyId: {
$toString: [req.user.companyId],
},
},
pipeline: [
{
$match: {
$expr: {
$eq: ["$companyId", "$$companyId"],
},
},
},
],
as: "projects",
},
},
])
.toArray()
All of my company information is being returned correctly for multiple companies, but that projects Array is still []. Any help would be appreciated, and I will still be troubleshooting this.
One option is using a $lookup with a pipeline:
db.company.aggregate([
{
$match: {
_id: {
$in: [
ObjectId("5a934e000102030405000000")
],
},
status: {
$in: [
"Client"
]
},
},
},
{
$lookup: {
from: "Projects",
let: {
companyId: {
$toString: "$_id"
}
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$companyId",
"$$companyId"
]
}
}
}
],
as: "projects"
}
}
])
See how it works on the playground example
Final answer for my question:
try {
const allClientsAndProjects = await companyCollection
.aggregate([
{
$match: {
orgId: {
$in: [new ObjectId(req.user.orgId)],
},
status: {
$in: ["Client"],
},
},
},
{
$lookup: {
from: "projects",
let: {
companyId: {
$toString: "$_id",
},
},
pipeline: [
{
$match: {
$expr: {
$eq: ["$companyId", "$$companyId"],
},
},
},
],
as: "projects",
},
},
])
.toArray()
Here is a hypothetical case of orders and products.
'products' collection
[
{
"_id": "61c53eb76eb2dc65de621bd0",
"name": "Product 1",
"price": 80
},
{
"_id": "61c53efca0a306c3f1160754",
"name": "Product 2",
"price": 10
},
... // truncated
]
'orders' collection:
[
{
"_id": "61c53fb7dca0579de038cea8", // order id
"products": [
{
"_id": "61c53eb76eb2dc65de621bd0", // references products._id
"quantity": 1
},
{
"_id": "61c53efca0a306c3f1160754",
"quantity": 2
},
]
}
]
As you can see, an order owns a list of product ids. When I pull an order's details I also need the product details combined like so:
{
_id: ObjectId("61c53fb7dca0579de038cea8"),
products: [
{
_id: ObjectId("61c53eb76eb2dc65de621bd0"),
quantity: 1,
name: 'Product 1',
price: 80
},
{
_id: ObjectId("61c53efca0a306c3f1160754"),
quantity: 2,
name: 'Product 2',
price: 10
},
... // truncated
]
}
Here is the aggregation pipleline I came up with:
db.orders.aggregate([
{
$match: {_id: ObjectId('61c53fb7dca0579de038cea8')}
},
{
$unwind: {
path: "$products"
}
},
{
$lookup: {
from: 'products',
localField: 'products._id',
foreignField: '_id',
as: 'productDetail'
}
},
{
$unwind: {
path: "$productDetail"
}
},
{
$group: {
_id: "$_id",
products: {
$push: {$mergeObjects: ["$products", "$productDetail"]}
}
}
}
])
Given how the data is organized I'm doubting if the pipeline stages are optimal and could do better (possibility of reducing the number of stages, etc.). Any suggestions?
As already mentioned in comments the design is poor. You can avoid multiple $unwind and $group, usually the performance should be better with this:
db.orders.aggregate([
{ $match: { _id: "61c53fb7dca0579de038cea8" } },
{
$lookup: {
from: "products",
localField: "products._id",
foreignField: "_id",
as: "productDetail"
}
},
{
$project: {
products: {
$map: {
input: "$products",
as: "product",
in: {
$mergeObjects: [
"$$product",
{
$first: {
$filter: {
input: "$productDetail",
cond: { $eq: [ "$$this._id", "$$product._id" ] }
}
}
}
]
}
}
}
}
}
])
Mongo Playground
I have these Schemas:
const chatbots = new Schema({
name: String,
campaigns: [{
name: String,
channels: [{
_id: String,
name: String,
budget: Number
}]
}]
});
const chatbotusers = new Schema({
name: String,
campaign_channel: String
})
And I need to get a list of Campaigns where, for each Channel, I have the total of ChatbotUsers. Something like this:
[
{
"name": "Campaign #1",
"channels": {
"_id": "eyRyZ1gD0",
"name": "Channel #1",
"users": 10
}
},
{
"name": "Campaign #1",
"channels": {
"_id": "tsKH7WxE",
"name": "Channel #2",
"users": 4
}
}
]
Any ideas?
The furthest I got was something like this:
{
$lookup: {
from: "chatbotusers",
localField: "channels._id",
foreignField: "campaign_channel",
as: "users",
}
},
{
$project: {
name: "$name",
channels: {
$map: {
input: "$channels",
as: "channel",
in: {
_id: "$$channel._id",
name: "$$channel.name",
users: { $size: "$users" },
}
}
}
}
}
But it sums the users for the Campaign, not the Channel.
(Sorry if the question title is not appropriate, I didn't even know how to ask this properly)
You can try this query :
db.chatbots.aggregate([
{
$lookup: {
from: "chatbotusers",
localField: "campaigns.channels._id",
foreignField: "campaign_channel",
as: "users"
}
},
{
$addFields: {
campaigns: {
$map: {
input: "$campaigns",
as: "eachCampaign",
in: {
$mergeObjects: ['$$eachCampaign', {
channels:
{
$reduce: {
input: "$$eachCampaign.channels",
initialValue: [],
in: {
$concatArrays: [
"$$value",
[
{
$mergeObjects: [
"$$this",
{
user: {
$size: {
$filter: {
input: "$users",
as: "e",
cond: {
$eq: [
"$$e.campaign_channel",
"$$this._id"
]
}
}
}
}
}
]
}
]
]
}
}
}
}]
}
}
}
}
},
{
$project: {
users: 0
}
}
])
Note : There can be multiple ways to do this, but this way we're working on same no.of docs from the chatbots collection rather than exploding docs by doing $unwind which may be helpful when you've huge dataset.
Test : MongoDB-Playground
This above query should get you what is needed, but in any case if it's slow or you think to enhance it then here :
{
user: {
$size: {
$filter: {
input: "$users", as: "e",
cond: {
$eq: [
"$$e.campaign_channel",
"$$this._id"
]
}
}
}
}
}
Where We're iterating thru users array for every channel in every campaign, So instead of iterating every time, right after lookup - You can iterate over users for once using reduce to get count of each unique campaign_channel replace this data as users array, that way you can get count of users directly. In general main intention of above query is to preserve original document structure with less stages being used.
Alternatively you can use this query, which doesn't preserve original doc structure (also no.of docs in output can be more than what you've in collection) but can do what you needed :
db.chatbots.aggregate([
{
$unwind: "$campaigns"
},
{
$unwind: "$campaigns.channels"
},
{
$lookup: {
from: "chatbotusers",
localField: "campaigns.channels._id",
foreignField: "campaign_channel",
as: "users"
}
},
{
$addFields: {
"channels": "$campaigns.channels",
campaigns: "$campaigns.name"
}
},
{
$addFields: {
"channels.users": {
$size: "$users"
}
}
},
{
$project: {
users: 0
}
}
])
Test : MongoDB-Playground
I have a complicated structure I am trying to "join".
The best way to describe it is that I have "Favorite Teams" stored with a user, as an array of name/IDs - however they are stored in a nested object. I want to return the users Favorite Teams Players WITH the team.
Here are the data models
PLAYERS
{
_id:
team_id:
name:
position:
}
TEAMS
{
_id:
name:
}
USER
{
_id:
name:
favs: {
mascots: [{
_id:
name:
}],
teams: [{
_id:
name:
}],
}
}
I have an array of Team IDs from the user.favs.teams - and what I want back is the players with their team name.
This is the current aggregation I am using - it is returning the players but not the teams...I am pretty sure I need to unwind, or similar.
players.aggregate([
{
$match: {
team_id: {
$in: [--array of team ID's--]
}
}
},
{
$lookup: {
from: 'teams',
localField: 'team_id',
foreignField: '_id',
as: 'players_team'
}
},
{
$project: {
_id: 1,
name: 1,
position: 1,
'players_team[0].name': 1
}
}
])
What I am getting back...
_id: 5c1b37b6fd15241940b11111
name:"Bob"
position:"Test"
team_id:5c1b37b6fd15241940b441dd
player_team:[
_id:5c1b37b6fd15241940b441dd
name:"Team A"
...other fields...
]
What I WANT to get back...
_id: 5c1b37b6fd15241940b11111
name:"Bob"
position:"Test"
team_id:5c1b37b6fd15241940b441dd
player_team: "Team A"
Use Below $lookup (Aggregation)
db.players.aggregate([
{
$lookup: {
from: "teams",
let: { teamId: "$team_id" },
pipeline: [
{
$match: { $expr: { $eq: [ "$_id", "$$teamId" ] } }
},
{
$project: { _id: 0 }
}
],
as: "players_team"
}
},
{
"$replaceRoot": {
"newRoot": {
"$mergeObjects": [
{
"_id": "$_id",
"name": "$name",
"position": "$position",
"team_id": "$team_id"
},
{
player_team: { $arrayElemAt: [ "$players_team.name", 0 ] }
}
]
}
}
}
])
Sorry If your MongoDB version is less then 3.6. Because of new changes in MongoDB 3.6.