Use $lookup on a double nested array - mongodb

Question

I'm trying to use an _id saved in a double nested array to find and attach data from a different document. It essentially looks like this:
doc1 = {
_id: ObjectId,
...other stuff...,
firstArray: [
...other stuff...,
secondArray: [
other_id: ObjectId
]
]
}
doc2 = {
_id: ObjectId,
...the stuff I want...
}
Not every entry in secondArray is going to contain an other_id field, only some. This has gotten me close to the result I want except that the firstArray field contains an entry for the total number of entries in the secondArray field. I think I'm just missing one step to essentially undo the one of the two $unwinds.
doc1.aggregate([
{ $match: { _id: req.params._id }},
{ $unwind: "$firstArray" },
{ $unwind: "$firstArray.secondArray" },
{ $lookup: {
from: "doc2",
localField: "firstArray.secondArray.other_id",
foreignField: "_id",
as: "firstArray.secondArray.other",
}},
{ $addFields: {
"firstArray.secondArray.other": {
$arrayElemAt: ["$firstArray.secondArray.other", 0]
}
}},
{ $group: {
_id: "$_id",
...other stuff...,
firstArray: { $push: "$firstArray" },
}},
]);

Answer 1

You need to add another $group stage to reconstruct the "secondArray" structure, in general it's easy to remember that for each $unwind action you do, you'll need to counter that with another $group in order to restore the initial structure.
In order to do that you'll need some kind of a unique identifier for firstArray so you can group by it, if it doesn't exist you can add an _id that is the index of the element in the array like so:
{
"$addFields": {
"firstArray": {
$map: {
input: {
$zip: {
inputs: [
"$firstArray",
{
$range: [
0,
{
$size: "$firstArray"
}
]
}
]
}
},
as: "item",
in: {
"$mergeObjects": [
{
"$arrayElemAt": [
"$$item",
0
]
},
{
_id: {
"$arrayElemAt": [
"$$item",
1
]
}
}
]
}
}
}
}
}
Now firstArray has a unique field _id that we can later use to group on like so:
... the pipeline ...
{
$group: {
_id: {
id: "$_id",
first: "$firstArray._id"
},
secondArray: {
$push: "$firstArray.secondArray"
}
}
},
{
$group: {
_id: "$id._id",
firstArray: {
$push: {
secondArray: "$secondArray"
}
},
}
}
You will need to add support for all the "other stuff" you defined in your own pipeline, but this pipeline flow is the way to go.
Mongo Playground