How to aggregate nested lookup array in mongoose? - mongodb

I have a problem with how to lookup nested array, for example i have 4 collections.
User Collection
"user": [
{
"_id": "1234",
"name": "Tony",
"language": [
{
"_id": "111",
"language_id": "919",
"level": "Expert"
},
{
"_id": "111",
"language_id": "920",
"level": "Basic"
}
]
}
]
Language Collection
"language": [
{
"_id": "919",
"name": "English"
},
{
"_id": "920",
"name": "Chinese"
}
]
Job
"job": [
{
"_id": "10",
"title": "Programmer",
"location": "New York"
}
],
CvSubmit Collection
"cvsubmit": [
{
"_id": "11",
"id_user": "1234",
"id_job": "11"
}
]
And my query aggregation is:
db.cvsubmit.aggregate([
{
$lookup: {
from: "user",
localField: "id_user",
foreignField: "_id",
as: "id_user"
}
},
{
$lookup: {
from: "language",
localField: "id_user.language.language_id",
foreignField: "_id",
as: "id_user.language.language_id"
}
},
])
But the result is:
[
{
"_id": "11",
"id_job": "11",
"id_user": {
"language": {
"language_id": [
{
"_id": "919",
"name": "English"
},
{
"_id": "920",
"name": "Chinese"
}
]
}
}
}
]
I want the result like this, also showing all user data detail like name:
[
{
"_id": "11",
"id_job": "11",
"id_user": {
"_id": "1234",
"name": "Tony"
"language": [
{
"_id": "919",
"name": "English",
"Level": "Expert"
},
{
"_id": "920",
"name": "Chinese",
"level": "Basic"
}
]
}
}
]
Mongo Playground link https://mongoplayground.net/p/i0yCucjruey
Thanks before.

$lookup with user collection
$unwind deconstruct id_user array
$lookup with language collection and return in languages field
$map to iterate look of id_user.language array
$reduce to iterate loop of languages array returned from collection, check condition if language_id match then return name
db.cvsubmit.aggregate([
{
$lookup: {
from: "user",
localField: "id_user",
foreignField: "_id",
as: "id_user"
}
},
{ $unwind: "$id_user" },
{
$lookup: {
from: "language",
localField: "id_user.language.language_id",
foreignField: "_id",
as: "languages"
}
},
{
$addFields: {
languages: "$$REMOVE",
"id_user.language": {
$map: {
input: "$id_user.language",
as: "l",
in: {
_id: "$$l._id",
level: "$$l.level",
name: {
$reduce: {
input: "$languages",
initialValue: "",
in: {
$cond: [
{ $eq: ["$$this._id", "$$l.language_id"] },
"$$this.name",
"$$value"
]
}
}
}
}
}
}
}
}
])
Playground
You database structure is not accurate as per NoSQL, there should be max 2 collections, loot of join using $lookup and $unwind will cause performance issues.

Related

mongodb $lookup - suppress non-matching documents

I only want to see matching documents, i.e. only T3 in the example below. I can find the matching documents between lotterytickets (many documents) and lotterydrawing (only a few documents).
How can I filter out the non-matching documents? Basically, I'd not like to see documents with the condition drawnticket==[], but I haven't found the conditional code to apply.
Any help would be appreciated. Thank you in advance
Configuration:
db={
"lotteryticket": [
{
"_id": ObjectId("6021ce0cb4d2c2b4f24c3a2e"),
"ticket": "T1",
"player": "Alice"
},
{
"_id": ObjectId("6021ce0cb4d2c2b4f24c3a2f"),
"ticket": "T2",
"player": "Bob"
},
{
"_id": ObjectId("6021ce0cb4d2c2b4f24c3a33"),
"ticket": "T3",
"player": "Charles"
}
],
"lotterydrawing": [
{
"_id": ObjectId("63309480b749b733c087b758"),
"ticket": "T3"
},
{
"_id": ObjectId("63309480b749b733c087b759"),
"ticket": "T9"
},
{
"_id": ObjectId("63309480b749b733c087b75a"),
"ticket": "T77"
}
]
}
Query:
db.lotteryticket.aggregate([
{
$lookup: {
from: "lotterydrawing",
localField: "ticket",
foreignField: "ticket",
as: "drawnticket",
}
}
])
Result:
[
{
"_id": ObjectId("6021ce0cb4d2c2b4f24c3a2e"),
"drawnticket": [],
"player": "Alice",
"ticket": "T1"
},
{
"_id": ObjectId("6021ce0cb4d2c2b4f24c3a2f"),
"drawnticket": [],
"player": "Bob",
"ticket": "T2"
},
{
"_id": ObjectId("6021ce0cb4d2c2b4f24c3a33"),
"drawnticket": [
{
"_id": ObjectId("63309480b749b733c087b758"),
"ticket": "T3"
}
],
"player": "Charles",
"ticket": "T3"
}
]
https://mongoplayground.net/p/bYcLEzrF5QT
Add a match stage, to filter stages with the empty drawn tickets. Like this:
db.lotteryticket.aggregate([
{
$lookup: {
from: "lotterydrawing",
localField: "ticket",
foreignField: "ticket",
as: "drawnticket",
}
},
{
"$match": {
$expr: {
"$gt": [
{
$size: "$drawnticket"
},
0
]
}
}
}
])
Playground.
try this query
db.lotteryticket.aggregate([
{
$lookup: {
from: "lotterydrawing",
localField: "ticket",
foreignField: "ticket",
as: "drawnticket"
}
},
{
"$match": {
drawnticket: {
$exists: true,
$ne: []
}
}
}
])
Playground.

MongoDB $lookup on array of objects with reference objectId

I have Orders collection and iam getting the data from it as shown below:
[
{
"_id": "628216b7b30bb8aa80c8fd1a",
"promotionsDetails": {
"companyTotalPrice": 27,
"promotionsData": [
{
"_id": "621de063bb5f9f0bf510897f",
"price": 27,
"companyId": "621dd85eb45ca2ae292d9a36"
},
{
"_id": "621de063bb5f9f0bf510897d",
"price": 19,
"companyId": "621dd85eb45ca2ae292d9a32"
}
]
}
},
{
"_id": "628214fcb30bb8aa80c8fd18",
"promotionsDetails": {
"companyTotalPrice": 46,
"promotionsData": [
{
"_id": "621de063bb5f9f0bf510897f",
"price": 46,
"companyId": "621dd85eb45ca2ae292d9a32",
}
]
},
}
]
what I am trying to do is to get the company details from the companies collection using the companyId objectId in each object in the array, like below:
[
{
"_id": "628216b7b30bb8aa80c8fd1a",
"promotionsDetails": {
"companyTotalPrice": 27,
"promotionsData": [
{
"_id": "621de063bb5f9f0bf510897f",
"price": 27,
"companyId": "621dd85eb45ca2ae292d9a36",
"companyData": { "title": "..." }
},
{
"_id": "621de063bb5f9f0bf510897d",
"price": 19,
"companyId": "621dd85eb45ca2ae292d9a32",
"companyData": { "title": "..." }
}
]
}
},
{
"_id": "628214fcb30bb8aa80c8fd18",
"promotionsDetails": {
"companyTotalPrice": 46,
"promotionsData": [
{
"_id": "621de063bb5f9f0bf510897f",
"price": 46,
"companyId": "621dd85eb45ca2ae292d9a32",
"companyData": { "title": "..." }
}
]
}
}
]
i have tried to use lookup and pipeline, but I'm not getting the desired result, thanks!
Actuality $lookup supports arrays, so there is no need to $unwind and change the structure. This will return your expected results:
db.Orders.aggregate([
{
$lookup: {
from: "Company",
localField: "promotionsDetails.promotionsData.companyId",
foreignField: "_id",
as: "companyfullData"
}
},
{
$set: {
"promotionsDetails.promotionsData": {
$map: {
input: "$promotionsDetails.promotionsData",
in: {
$mergeObjects: [
"$$this",
{
companyData: {
$arrayElemAt: [
"$companyfullData",
{$indexOfArray: ["$companyfullData.id", "$$this.id"]}
]
}
}
]
}
}
}
}
},
{$unset: "companyfullData"}
])
Playground
Here, to make it more clear take a look at Mongo playground
db.Orders.aggregate([
{
$unwind: "$promotionsDetails.promotionsData"
},
{
"$lookup": {
"from": "Company",
"localField": "promotionsDetails.promotionsData.companyId",
"foreignField": "_id",
"as": "promotionsDetails.promotionsData.companyData"
}
},
])

How to add embedded field with matching documents

I'm using Python with pymongo to query from the database.
I have 3 different collections:
1st:
# Projects collection
{
"_id": "A",
},
{
"_id": "B",
},
{
"_id": "C"
},
..
2nd:
# Episodes collection
{
"_id": "A/Episode01",
"project": "A",
"name": "Episode01"
},
{
"_id": "A/Episode02",
"project": "A",
"name": "Episode02"
},
{
"_id": "B/Episode01",
"project": "B",
"name": "Episode01"
},
..
3rd:
# Sequences collection
{
"_id": "A/Episode01/Sequence01",
"project": "A",
"episode": "Episode01",
"name": "Sequence01"
},
{
"_id": "A/Episode02/Sequence02",
"project": "A",
"episode": "Episode02",
"name": "Sequence02"
},
{
"_id": "B/Episode01/Sequence01",
"project": "B",
"episode": "Episode01",
"name": "Sequence01"
},
..
I want to use aggregate to query project A and get all of its corresponding episodes and sequences like this:
{
"_id": "A",
"episodes":
[
{
"_id": "A/Episode01",
"project": "A",
"name": "Episode01",
"sequences":
[
{
"_id": "A/Episode01/Sequence01",
"project": "A",
"episode": "Episode01",
"name": "Sequence01"
},
]
},
{
"_id": "A/Episode02",
"project": "A",
"name": "Episode02",
"sequences":
[
{
"_id": "A/Episode02/Sequence02",
"project": "A",
"episode": "Episode02",
"name": "Sequence02"
},
]
},
]
}
I can get as far as getting the proper episodes, but I'm not sure how to add an embed field for any matching sequences. Is it possible to do this all in a single pipeline query?
Right now my query is looking like this:
[
{"$match": {
"_id": "A"}
},
{"$lookup": {
"from": "episodes",
"localField": "_id",
"foreignField": "project",
"as": "episodes"}
},
{"$group": {
"_id": {
"_id": "$_id",
"episodes": "$episodes"}
}}
]
You can do like following
use $match to match the document
use uncorrelated queries to join two collection. But normal joining also possible as you have written. This is easier when we get some complex situations.
Mongo script is given below
[
{
"$match": {
"_id": "A"
}
},
{
$lookup: {
from: "Episodes",
let: {
id: "$_id"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$project",
"$$id"
]
}
}
},
{
$lookup: {
from: "Sequences",
let: {
epi: "$name"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$episode",
"$$epi"
]
}
}
}
],
as: "sequences"
}
}
],
as: "episodes"
}
}
]
Working Mongo playground
Update 01
Using standard lookup
[
{
"$match": {
"_id": "A"
}
},
{
"$lookup": {
"from": "Episodes",
"localField": "_id",
"foreignField": "project",
"as": "episodes"
}
},
{
$unwind: "$episodes"
},
{
"$lookup": {
"from": "Sequences",
"localField": "episodes.name",
"foreignField": "episode",
"as": "episodes.sequences"
}
},
{
$group: {
_id: "$episodes._id",
episodes: {
$addToSet: "$episodes"
}
}
}
]
Working Mongo playground

how to transform data with 3 collection and $looup

I have three collection wanted to fetch records bases on their reference id, I got the result but wanted to transform the data how to make it.
db.post.aggregate([
{
$lookup: {
from: "users",
localField: "created_by",
foreignField: "_id",
as: "users"
}
},
{
$lookup: {
from: "comments",
let: {
p_id: "$_id"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$post_id",
"$$p_id"
]
}
}
}
],
as: "comments"
}
},
{
$lookup: {
from: "users",
localField: "comments.sender_id",
foreignField: "_id",
as: "commented_user"
}
},
])
I am getting this result but what I want merge the commented_user into comments mean who has comments fetch the record.
[
{
"_id": ObjectId("5eeb02881982961ada625c7d"),
"commented_user": [
{
"_id": ObjectId("5e4d0973babf2b74ca868f4d"),
"first_name": "James",
"last_name": "Smith",
"timestamp": 1.582106995137e+12
}
],
"comments": [
{
"_id": ObjectId("5eeb08e26fb7f270e4077617"),
"date": 1.592461538923e+12,
"post_id": ObjectId("5eeb02881982961ada625c7d"),
"sender_id": ObjectId("5e4d0973babf2b74ca868f4d"),
"text": "Nice One "
}
],
"created_by": ObjectId("5e4e74eb380054797d9db623"),
"created_users": [],
"date": 1.589441206774e+12,
"title": "Covid19"
}
]
Would better if I get each commented detail under comments like below ?
"comments": [
{
"_id": ObjectId("5eeb08e26fb7f270e4077617"),
"date": 1.592461538923e+12,
"post_id": ObjectId("5eeb02881982961ada625c7d"),
"sender_id": ObjectId("5e4d0973babf2b74ca868f4d"),
"text": "Nice One ",
"commented_user": [
{
"_id": ObjectId("5e4d0973babf2b74ca868f4d"),
"first_name": "James",
"last_name": "Smith",
"timestamp": 1.582106995137e+12
}
],
}
],
Here is mongoplayground https://mongoplayground.net/p/uw0kMTChFa0
You could use $set and $unset or $project to format the outputs
Example with unset or Example with project
Updated Example. You need to unwind the comments array then perform lookup for each comment to get the commented user.
db.post.aggregate([
{
$lookup: {
from: "users",
localField: "created_by",
foreignField: "_id",
as: "created_users"
}
},
{
$lookup: {
from: "comments",
let: {
p_id: "$_id"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$post_id",
"$$p_id"
]
}
}
}
],
as: "comments"
}
},
{
$unwind: "$comments"
},
{
$lookup: {
from: "users",
localField: "comments.sender_id",
foreignField: "_id",
as: "commented_user"
}
},
{
$set: {
"comments.commented_user": "$commented_user",
}
},
{
$unset: "commented_user"
}
])

MongoDb aggregation $lookup with foreign _ids in arrays

I'm a MongoDb novice. I'm getting pretty good, but no expert yet. I'm trying setup my collections in a way that makes sense. I'd like to keep some links to foreign docs inside arrays of just _ids and also arrays of objects that have _ids.
I created a JSON doc with notes that I think fully shows what I'm trying to do...
// ( item ) Character Inventory/Items collection
[
{
"_id": "1234",
"name": "Sword",
"descr": "Long sword, well worn, light rust",
"encumber": 2,
"del": false
},
{
"_id": "1271",
"name": "Pouch",
"descr": "Small leather waist pouch, suitable for coins",
"encumber": 0,
"del": false
}
]
// ( charnpcclass ) Character Classes collection
[
{ "_id": "2", "name": "Thief", "del": false },
{ "_id": "3", "name": "Cleric", "del": false }
]
// ( charnpcalign ) Character Alignments collection
[
{ "_id": "3", "name": "Lawful Good", "del": false },
{ "_id": "4", "name": "Neutral", "del": false }
]
// ( character ) Characters collection
[
{
"_id": "3345",
"name": "Offut 'Dead Dog' Dubro",
"description": "Halfling, scruffy, looks homeless",
"align": ObjectId("4"),
"classes": [
ObjectId("2"),
ObjectId("3")
],
"carrying": [
{ "itemId": ObjectId("1271"), "qty":1, "where": "Sheath inside vest", "visible": false }
{ "itemId": ObjectId("1234"), "qty":1, "where": "Sword scabbard at waist", "visible": true }
],
"del": false
}
]
// ------------------------------------------------------------
// This is my MongoDb aggregation in the REST api routes
var linkedModels = [
{
"$match": { "del": false }
}, {
"$lookup": {
from: "charnpcclass",
localField: "classes",
foreignField: "_id",
as: "linked_classes"
}
}, {
"$lookup": {
from: "charnpcalign",
localField: "alignId",
foreignField: "_id",
as: "linked_align"
}
}, {
"$lookup": {
from: "item",
localField: "carrying.itemId",
foreignField: "_id",
as: "linked_carrying"
}
}
];
db.collection('character').aggregate(linkedModels).toArray(function (err, docs) {
res.json(201, docs);
next();
});
// Query for Character, return items carrying with data from items collection
// ------------------------------------------------------------
// WHAT I *WANT* IN RESPONSE...
{
"id": "3345",
"name": "Offut 'Dead Dog' Dubro",
"description": "Halfling, scruffy, looks homeless",
"align": "4",
"classes": [
"2",
"3"
],
"carrying": [
{ "itemId": "1271", "qty":1, "where": "Sheath inside vest", "visible": false }
{ "itemId": "1234", "qty":1, "where": "Sword scabbard at waist", "visible": true }
],
"linked_align": [
{ "_id": "4", "name": "Neutral" },
],
"linked_classes": [
{ "_id": "2", "name": "Thief" },
{ "_id": "3", "name": "Cleric" }
],
"linked_carrying": [
{ "_id": "1271", "name": "Dagger", "encumber": 0 },
{ "_id": "1234", "name": "Sword", "encumber": 2 }
]
}
// ------------------------------------------------------------
// WHAT I ACTUALLY GET IN RESPONSE
{
"id": "3345",
"name": "Offut 'Dead Dog' Dubro",
"description": "Halfling, scruffy, looks homeless",
"align": "4",
"classes": [
"2",
"3"
],
"carrying": [
{ "itemId": "1271", "qty":1, "where": "Sheath inside vest", "visible": false }
{ "itemId": "1234", "qty":1, "where": "Sword scabbard at waist", "visible": true }
],
"linked_align": [
{ "_id": "4", "name": "Neutral" },
],
"linked_classes": [],
"linked_carrying": []
}
The problem that I hope you noticed is just above, at bottom of JSON response example. My linked arrays are empty and I'm not sure how to solve this.
I would greatly appreciate your expert MongoDb querying advice :-)
You have to $unwind to flatten the both scalar and sub document foreign _ids and add $group stage at the end of the pipeline to get back the original structure.
$first accumulator to keep the fields and $push with $arrayElemAt to accumulate the array values to adjust for $unwind
var linkedModels = [
{
"$match": { "del": false }
},
{
"$lookup": {
from: "charnpcalign",
localField: "align",
foreignField: "_id",
as: "linked_align"
}
},
{
"$unwind":"$classes"
},
{
"$lookup": {
from: "charnpcclass",
localField: "classes",
foreignField: "_id",
as: "linked_classes"
}
},
{
"$group": {
"_id": "$_id",
"name": {"$first":"$name"},
"align": {"$first":"$align"},
"classes":{"$push":"$classes"},
"carrying":{"$first":"$carrying"},
"linked_align":{"$first":"$linked_align"},
"linked_classes":{"$push":{"$arrayElemAt":["$linked_classes",0]}}
}
},
{
"$unwind":"$carrying"
},
{
"$lookup": {
from: "item",
localField: "carrying.itemId",
foreignField: "_id",
as: "linked_carrying"
}
},
{
"$group": {
"_id": "$_id",
"name": {"$first":"$name"},
"align": {"$first":"$align"},
"classes":{"$first":"$classes"},
"linked_align":{"$first":"$linked_align"},
"carrying":{"$push":"$carrying"},
"linked_carrying":{"$push":{"$arrayElemAt":["$linked_carrying",0]}}
}
}
]
You don't need the $unwind on the scalar array (classes) in 3.4 version and you can replace the {"classes":{"$push":"$classes"}} & {"linked_classes":{"$push":{$arrayElemAt:["$linked_classes",0]}}} with {"classes":{"$first":"$classes"}} & {"linked_classes":{"$first":"$linked_classes"}} respectively.