MongoDB aggregate lookup with nested array - mongodb

I have a complicated structure I am trying to "join".
The best way to describe it is that I have "Favorite Teams" stored with a user, as an array of name/IDs - however they are stored in a nested object. I want to return the users Favorite Teams Players WITH the team.
Here are the data models
PLAYERS
{
_id:
team_id:
name:
position:
}
TEAMS
{
_id:
name:
}
USER
{
_id:
name:
favs: {
mascots: [{
_id:
name:
}],
teams: [{
_id:
name:
}],
}
}
I have an array of Team IDs from the user.favs.teams - and what I want back is the players with their team name.
This is the current aggregation I am using - it is returning the players but not the teams...I am pretty sure I need to unwind, or similar.
players.aggregate([
{
$match: {
team_id: {
$in: [--array of team ID's--]
}
}
},
{
$lookup: {
from: 'teams',
localField: 'team_id',
foreignField: '_id',
as: 'players_team'
}
},
{
$project: {
_id: 1,
name: 1,
position: 1,
'players_team[0].name': 1
}
}
])
What I am getting back...
_id: 5c1b37b6fd15241940b11111
name:"Bob"
position:"Test"
team_id:5c1b37b6fd15241940b441dd
player_team:[
_id:5c1b37b6fd15241940b441dd
name:"Team A"
...other fields...
]
What I WANT to get back...
_id: 5c1b37b6fd15241940b11111
name:"Bob"
position:"Test"
team_id:5c1b37b6fd15241940b441dd
player_team: "Team A"

Use Below $lookup (Aggregation)
db.players.aggregate([
{
$lookup: {
from: "teams",
let: { teamId: "$team_id" },
pipeline: [
{
$match: { $expr: { $eq: [ "$_id", "$$teamId" ] } }
},
{
$project: { _id: 0 }
}
],
as: "players_team"
}
},
{
"$replaceRoot": {
"newRoot": {
"$mergeObjects": [
{
"_id": "$_id",
"name": "$name",
"position": "$position",
"team_id": "$team_id"
},
{
player_team: { $arrayElemAt: [ "$players_team.name", 0 ] }
}
]
}
}
}
])
Sorry If your MongoDB version is less then 3.6. Because of new changes in MongoDB 3.6.

Related

MongoDB: Optimal joining of one to many relationship

Here is a hypothetical case of orders and products.
'products' collection
[
{
"_id": "61c53eb76eb2dc65de621bd0",
"name": "Product 1",
"price": 80
},
{
"_id": "61c53efca0a306c3f1160754",
"name": "Product 2",
"price": 10
},
... // truncated
]
'orders' collection:
[
{
"_id": "61c53fb7dca0579de038cea8", // order id
"products": [
{
"_id": "61c53eb76eb2dc65de621bd0", // references products._id
"quantity": 1
},
{
"_id": "61c53efca0a306c3f1160754",
"quantity": 2
},
]
}
]
As you can see, an order owns a list of product ids. When I pull an order's details I also need the product details combined like so:
{
_id: ObjectId("61c53fb7dca0579de038cea8"),
products: [
{
_id: ObjectId("61c53eb76eb2dc65de621bd0"),
quantity: 1,
name: 'Product 1',
price: 80
},
{
_id: ObjectId("61c53efca0a306c3f1160754"),
quantity: 2,
name: 'Product 2',
price: 10
},
... // truncated
]
}
Here is the aggregation pipleline I came up with:
db.orders.aggregate([
{
$match: {_id: ObjectId('61c53fb7dca0579de038cea8')}
},
{
$unwind: {
path: "$products"
}
},
{
$lookup: {
from: 'products',
localField: 'products._id',
foreignField: '_id',
as: 'productDetail'
}
},
{
$unwind: {
path: "$productDetail"
}
},
{
$group: {
_id: "$_id",
products: {
$push: {$mergeObjects: ["$products", "$productDetail"]}
}
}
}
])
Given how the data is organized I'm doubting if the pipeline stages are optimal and could do better (possibility of reducing the number of stages, etc.). Any suggestions?
As already mentioned in comments the design is poor. You can avoid multiple $unwind and $group, usually the performance should be better with this:
db.orders.aggregate([
{ $match: { _id: "61c53fb7dca0579de038cea8" } },
{
$lookup: {
from: "products",
localField: "products._id",
foreignField: "_id",
as: "productDetail"
}
},
{
$project: {
products: {
$map: {
input: "$products",
as: "product",
in: {
$mergeObjects: [
"$$product",
{
$first: {
$filter: {
input: "$productDetail",
cond: { $eq: [ "$$this._id", "$$product._id" ] }
}
}
}
]
}
}
}
}
}
])
Mongo Playground

MongoDB aggregation query feasibility

I am evaluating MongoDB for an application and I am trying to learn how to use it.
I don't know whether what I want to achieve is possible, so I am prepared for "No" as an answer.
Suppose the three collections described below (in mongoplayground-style format)
db={
packages: [
{
_id: 10,
name: "small box"
},
{
_id: 20,
name: "big box"
}
],
shipments: [
{
customer: "bob",
items: [
{
_id: 12312,
package_id: 20,
weight: 9.99,
},
{
_id: 65489,
package_id: 10,
weight: 1.5
}
]
}
]
}
I want to use the aggregation framework to produce a document a shipment, where for each item a new property is added that contains the information regarding the package (example given below)
{
customer: "bob",
items: [
{
_id: 12312,
package_id: 20,
weight: 9.99,
package: {
_id: 20,
name: "big box"
}
},
{
_id: 65489,
package_id: 10,
weight: 1.5,
package: {
_id: 10,
name: "small box"
}
}
]
}
I have tried using $lookup without a pipeline, but I can only get up to a point where I replace each item document with the corresponding package document (which of course is not what I want to achieve), and I am lost with using $lookup with a pipeline (which if I were a betting man, I'd bet is the way to achieve what I want). I kind of got somewhere by $unwind'ing the items array, followed by a $lookup and a $group / $push but then I am not sure how to retrieve the other fields of a shipment document other than the items (and I have a feeling this is not the proper way to achieve the desired result).
I can post more code if needed (what I have tried so far), but I am trying to keep the question within a reasonable length
Any help would be appreciated, as I am sure I could be trying for days to produce the sample I want and I am not even sure it is possible.
db.shipments.aggregate([
{
$lookup: {
from: "packages",
localField: "items.package_id",
foreignField: "_id",
as: "packages"
}
},
{
$project: {
customer: 1,
items: {
$map: {
input: "$items",
as: "item",
in: {
"$mergeObjects": [
{
"_id": "$$item._id",
"package_id": "$$item.package_id",
"weight": "$$item.weight"
},
{
"package": {
"$arrayElemAt": [
{
"$filter": {
"input": "$packages",
"as": "package",
"cond": {
"$eq": [
"$$item.package_id",
"$$package._id"
]
}
}
},
0
]
}
}
]
}
}
}
}
}
])
mongoplayground
Try this:
db.shipments.aggregate([
{ $unwind: "$items" },
{
$lookup: {
from: "packages",
let: { package_id: "$items.package_id" },
pipeline: [
{
$match: {
$expr: {
$eq: ["$_id", "$$package_id"]
}
}
}
],
as: "items.package"
}
},
{ $unwind: "$items.package" },
{
$group: {
_id: "$_id",
customer: { $first: "$customer" },
items: { $push: "$items" }
}
}
])
Solution in Playground

$match in aggregate don't return data in mongodb

I have three tables below is the structure like below
I'm looking to get a result like below
"type1": [ -- type from Accounts collection
{
"_id": "5e97e9a224f62f93d5x3zz46", -- _id from Accounts collection
"locs": "sampleLocks 1", -- field from Accounts collection
"solutions": "sample solutions 1", -- field from Accounts collection
"Clause": "clause 1" -- field from AccountsDesc collection
},
{
"_id": "5e97e9a884f62f93d5x3zz46",
"locs": "sampleLocks2",
"solutions": "sample solutions2",
"Clause": "clause2"
}
],
"type2": [
// same data construction as of type1 above
]
_id, locks, solution to be coming from Accounts collection
Clause field to be coming from AccountsDesc collection
accounts_id is kind of a foreign key in AccountsDesc coming from Account
competitor_id is kind of a foreign key in AccountsDesc coming from Competitor
Below is what my query looks like
db.accountDesc.aggregate([
{
$match : {accounts_Id : "123456"}, active: true}
},
{
$lookup: {
from: 'accounts',
pipeline: [{ $match: { type: { $in: ["type1, type2, type3"] } } }],
as: 'accountsData'
}
},
{
$group: {
_id: "$accountsData.type",
data: {
$push: {_id: "$accountsData._id", clause: "$clause", locs: "$type.locs", solutions: "$type.solutions"}
}
}
},
{
$group: {
_id: null,
data: {
$push: {
k: {
$toString: '$_id'
},
v: '$data'
}
}
}
},
{
$replaceRoot: {
newRoot: {
$arrayToObject: '$data'
}
}
}
])
Issues related with the query -
$match : {accountId : "123456"}, active: true} -- No data is returned if i use match on AccountsDesc collection
cant set localField, foriegnField if im using pipeline, then how the mapping will happen like a LEFT join.
clause: "$clause" don't get the value of this field in the response
As we discussed in chat, you want RIGHT OUTER JOIN for your aggregation.
Try the query below:
db.User_Promo_Map.aggregate([
{
$match: {
user_Id: ObjectId("5e8c1180d59de1704ce68112")
}
},
{
$lookup: {
from: "promo",
pipeline: [
{
$match: {
active: true,
platform: {
$in: [
"twitch",
"youtube",
"facebook"
]
}
}
}
],
as: "accountsData"
}
},
{
$unwind: "$accountsData"
},
{
$group: {
_id: "$accountsData.platform",
data2: {
$addToSet: {
amount: "$amount",
promo_Id: "$promo_Id"
}
},
data: {
$addToSet: {
_id: "$accountsData._id",
format: "$accountsData.format",
description: "$accountsData.description"
}
}
}
},
{
$addFields: {
data: {
$map: {
input: "$data",
as: "data",
in: {
"_id": "$$data._id",
"description": "$$data.description",
"format": "$$data.format",
amount: {
$reduce: {
input: "$data2",
initialValue: "$$REMOVE",
in: {
$cond: [
{
$eq: [
"$$this.promo_Id",
"$$data._id"
]
},
"$$this.amount",
"$$value"
]
}
}
}
}
}
}
}
},
{
$group: {
_id: null,
data: {
$push: {
k: {
$toString: "$_id"
},
v: "$data"
}
}
}
},
{
$replaceRoot: {
newRoot: {
$arrayToObject: "$data"
}
}
}
])
MongoPlayground

How to use $lookup on array of subdocuments

I have these Schemas:
const chatbots = new Schema({
name: String,
campaigns: [{
name: String,
channels: [{
_id: String,
name: String,
budget: Number
}]
}]
});
const chatbotusers = new Schema({
name: String,
campaign_channel: String
})
And I need to get a list of Campaigns where, for each Channel, I have the total of ChatbotUsers. Something like this:
[
{
"name": "Campaign #1",
"channels": {
"_id": "eyRyZ1gD0",
"name": "Channel #1",
"users": 10
}
},
{
"name": "Campaign #1",
"channels": {
"_id": "tsKH7WxE",
"name": "Channel #2",
"users": 4
}
}
]
Any ideas?
The furthest I got was something like this:
{
$lookup: {
from: "chatbotusers",
localField: "channels._id",
foreignField: "campaign_channel",
as: "users",
}
},
{
$project: {
name: "$name",
channels: {
$map: {
input: "$channels",
as: "channel",
in: {
_id: "$$channel._id",
name: "$$channel.name",
users: { $size: "$users" },
}
}
}
}
}
But it sums the users for the Campaign, not the Channel.
(Sorry if the question title is not appropriate, I didn't even know how to ask this properly)
You can try this query :
db.chatbots.aggregate([
{
$lookup: {
from: "chatbotusers",
localField: "campaigns.channels._id",
foreignField: "campaign_channel",
as: "users"
}
},
{
$addFields: {
campaigns: {
$map: {
input: "$campaigns",
as: "eachCampaign",
in: {
$mergeObjects: ['$$eachCampaign', {
channels:
{
$reduce: {
input: "$$eachCampaign.channels",
initialValue: [],
in: {
$concatArrays: [
"$$value",
[
{
$mergeObjects: [
"$$this",
{
user: {
$size: {
$filter: {
input: "$users",
as: "e",
cond: {
$eq: [
"$$e.campaign_channel",
"$$this._id"
]
}
}
}
}
}
]
}
]
]
}
}
}
}]
}
}
}
}
},
{
$project: {
users: 0
}
}
])
Note : There can be multiple ways to do this, but this way we're working on same no.of docs from the chatbots collection rather than exploding docs by doing $unwind which may be helpful when you've huge dataset.
Test : MongoDB-Playground
This above query should get you what is needed, but in any case if it's slow or you think to enhance it then here :
{
user: {
$size: {
$filter: {
input: "$users", as: "e",
cond: {
$eq: [
"$$e.campaign_channel",
"$$this._id"
]
}
}
}
}
}
Where We're iterating thru users array for every channel in every campaign, So instead of iterating every time, right after lookup - You can iterate over users for once using reduce to get count of each unique campaign_channel replace this data as users array, that way you can get count of users directly. In general main intention of above query is to preserve original document structure with less stages being used.
Alternatively you can use this query, which doesn't preserve original doc structure (also no.of docs in output can be more than what you've in collection) but can do what you needed :
db.chatbots.aggregate([
{
$unwind: "$campaigns"
},
{
$unwind: "$campaigns.channels"
},
{
$lookup: {
from: "chatbotusers",
localField: "campaigns.channels._id",
foreignField: "campaign_channel",
as: "users"
}
},
{
$addFields: {
"channels": "$campaigns.channels",
campaigns: "$campaigns.name"
}
},
{
$addFields: {
"channels.users": {
$size: "$users"
}
}
},
{
$project: {
users: 0
}
}
])
Test : MongoDB-Playground

$lookup when foreignField is in nested array

I have two collections :
Student
{
_id: ObjectId("657..."),
name:'abc'
},
{
_id: ObjectId("593..."),
name:'xyz'
}
Library
{
_id: ObjectId("987..."),
book_name:'book1',
issued_to: [
{
student: ObjectId("657...")
},
{
student: ObjectId("658...")
}
]
},
{
_id: ObjectId("898..."),
book_name:'book2',
issued_to: [
{
student: ObjectId("593...")
},
{
student: ObjectId("594...")
}
]
}
I want to make a Join to Student collection that exists in issued_to array of object field in Library collection.
I would like to make a query to student collection to get the student data as well as in library collection, that will check in issued_to array if the student exists or not if exists then get the library document otherwise not.
I have tried $lookup of mongo 3.6 but I didn`t succeed.
db.student.aggregate([{$match:{_id: ObjectId("593...")}}, $lookup: {from: 'library', let: {stu_id:'$_id'}, pipeline:[$match:{$expr: {$and:[{"$hotlist.clientEngagement": "$$stu_id"]}}]}])
But it thorws error please help me in regard of this. I also looked at other questions asked at stackoverflow like. question on stackoverflow,
question2 on stackoverflow but these are comapring simple fields not array of objects. please help me
I am not sure I understand your question entirely but this should help you:
db.student.aggregate([{
$match: { _id: ObjectId("657...") }
}, {
$lookup: {
from: 'library',
localField: '_id' ,
foreignField: 'issued_to.student',
as: 'result'
}
}])
If you want to only get the all book_names for each student you can do this:
db.student.aggregate([{
$match: { _id: ObjectId("657657657657657657657657") }
}, {
$lookup: {
from: 'library',
let: { 'stu_id': '$_id' },
pipeline: [{
$unwind: '$issued_to' // $expr cannot digest arrays so we need to unwind which hurts performance...
}, {
$match: { $expr: { $eq: [ '$issued_to.student', '$$stu_id' ] } }
}, {
$project: { _id: 0, "book_name": 1 } // only include the book_name field
}],
as: 'result'
}
}])
This might not be a very good answer, but if you can change your schema of Library to:
{
_id: ObjectId("987..."),
book_name:'book1'
issued_to: [
ObjectId("657..."),
ObjectId("658...")
]
},
{
_id: "ObjectId("898...")",
book_name:'book2'
issued_to: [
ObjectId("593...")
ObjectId("594...")
]
}
Then when you do:
{
$lookup: {
from: 'student',
localField: 'issued_to',
foreignField: '_id',
as: 'issued_to_students', // this creates a new field without overwriting your original 'issued_to'
}
},
You should get, based on your example above:
{
_id: ObjectId("987..."),
book_name:'book1'
issued_to_students: [
{ _id: ObjectId("657..."), name: 'abc', ... },
{ _id: ObjectId("658..."), name: <name of this _id>, ... }
]
},
{
_id: "ObjectId("898...")",
book_name:'book2'
issued_to: [
{ _id: ObjectId("593..."), name: 'xyz', ... },
{ _id: ObjectId("594..."), name: <name of this _id>, ... }
]
}
You need to $unwind the issued_to from library collection to match the issued_to.student with _id
db.student.aggregate([
{ "$match": { "_id": mongoose.Types.ObjectId(id) } },
{ "$lookup": {
"from": Library.collection.name,
"let": { "studentId": "$_id" },
"pipeline": [
{ "$unwind": "$issued_to" },
{ "$match": { "$expr": { "$eq": [ "$issued_to.student", "$$studentId" ] } } }
],
"as": "issued_to"
}}
])