Add a new field from an object nested inside an array - mongodb

I have two collections:
Assets collection:
{
"_id": { "$oid": "620bc49c937d954ed4acf813" },
"opID": { "$oid": "620bc497937d954ed4acf810" },
"issues": [{
"otherFields": {...}
}],
"__v": 7
}
Operations collection:
{
"_id": { "$oid": "620bc49c937d954ed4acf813" },
"name": "Test",
"otherFields": [...],
"__v": 7
}
I want to add the operation name and _id to all the issue Objects inside the Assets issue field.
My actual pipeline to do it is:
$lookup
{
from: "operations",
localField: "opID",
foreignField: "_id",
as: "operationsJoin",
}
$unwind
{ path: "$issues" }
$addFields
"issues.operationId": "$operationsJoin._id"
$group
{
_id: "$_id",
issues: {
$push: {issues:"$issues"}
}
}
My first issue is at the $addFields stage. The operationsJoin field is an array and when the operationId field it's setted on the issue object it's created as an array as this:
issues: {
operationId: [
"5f3b8ff2906e402014832d33"
]
}
instead of:
issues: {
operationId: "5f3b8ff2906e402014832d33"
}
and i cannot do:
"issues.operationId": "$operationsJoin[0]._id"
And the second issue it's that a doesn't know to add other field to the $group stage apart the issues array that i'm creating.
Anyone has an idea ?? Thanks for the help in advance
Edited:
I let an example of the pipeline here, for now i only want the operationName and operationId fields to be strings, not arrays:
Example Pipeline

Related

multi-stage aggregation pipeline matching data based on fields retrieved through $lookup

I'm trying to build a complex, nested aggregation pipeline in MongoDB (4.4.9 Community Edition, using the pymongo driver for Python 3.10).
There are relevant data points in different collections which I want to aggregate into one, NEW (ideally) view (or, if that doesn't work) collection.
The collections, and the relevant fields therein follow a hierarchy. There is members, which contains the top-level key on which other data is to be merged,
membershipNumber.
> members.find_one()
{'_id': ObjectId('61153299af6122XXXXXXXXXXXXX'), 'membershipNumber': 'N03XXXXXX'}
Then, there's a different collection, which contains membershipNumber, but also a different, linked field, an_user_id. an_user_id is used in other collections to denote records/fields in arrays that pertain to that particular user.
I 'join' members and an_users like so:
result = members.aggregate([
{
'$lookup': {
'from': 'an_users',
'localField': 'membershipNumber',
'foreignField': 'memref',
'as': 'an_users'
}
},
{ '$unwind' : '$an_users' },
{
'$project' : {
'_id' : 1,
'membershipNumber' : 1,
'an_user_id' : '$an_users.user_id'
}
}
]);
So far so good, this returns the desired, aggregated record:
{'_id': ObjectId('61153253aBBBBBBBBBBBB'),
'membershipNumber': 'N0XXXXXXXX',
'an_user_id': '48XXXXXX'}
Now, I have a third collection, which contains the an_user_id as a string in arrays, denoting wherever that user clicked a given email, whereby a record is an email (and the an_user_ids in the clicks array are users that clicked a link in that email.
{'_id': ObjectId('blah'),
'email_id': '407XXX',
'actions_count': 17,
'administrative_title': 'test',
'bounce': ['3440XXXX'],
'click': ['38294CCC',
'418FFFF',
'48XXXXXX',
'38eGGGG'}
I want to count the number occurences of a given an_user_id (which I've attained from aggregating) in arrays (e.g. clicks, bounces, opens) in the emails collection, and include it in the .aggregate call, to retrieve something like this:
{'_id': ObjectId('61153253aBBBBBBBBBBBB'),
'membershipNumber': 'N0XXXXXXXX',
'an_user_id': '48XXXXXX',
'n_email_clicks' : 412,
'n_email_bounces' : 12
}
Further, I might want to also attach counts of an_user_id in other collections in my DB.
Consider, e.g., this collection called events:
{
"_id": "617ffa96ee11844e143a63dd",
"id": "12345",
"administrative_title": "my_event",
"created_at": {
"$date": "2020-01-15T16:28:50.000Z"
},
"event_creator_id": "123456",
"event_title": "my_event",
"group_id": "123456",
"permalink": "event_id",
"rsvp_count": 54,
"rsvps": [{
"rsvp_id": "56789",
"display_name": "John Doe",
"rsvp_user_id": "48XXXXXX",
"rsvp_created_at": {
"$date": "2020-01-28T15:38:50.000Z"
},
"rsvp_updated_at": {
"$date": "2020-01-28T15:38:50.000Z"
},
"first_name": "John",
"last_name": "Doe",
}, {
"rsvp_id": "543895",
"display_name": "James Appleslice",
"rsvp_user_id": "N03XXXXXX",
"rsvp_created_at": {
"$date": "2020-02-05T13:15:14.000Z"
},
"rsvp_updated_at": {
"$date": "2020-02-05T13:15:14.000Z"
},
"first_name": "James",
"last_name": "Appleslice"}
]
}
So, the end-product would look something like this:
{'_id': ObjectId('61153253aBBBBBBBBBBBB'),
'membershipNumber': 'N0XXXXXXXX',
'an_user_id': '48XXXXXX',
'n_email_clicks' : 412,
'n_email_bounces' : 12,
'n_rsvps' : 12
}
My idea was to use the $lookup parameter -- however, I only know how to use this for matching on fields that I have in the parent collection that I'm performing the aggregation on, but not on fields that have been generated in the process of the aggregation.
Any help would be hugely appreciated!
You could use $lookup pipeline. First you would $lookup the user id followed by another $lookup to verify if the user id exists in email. Lastly few more stages to collect the results and format per your need. Furthermore, you can add $out stage if you would like to write the results into another collection.
db.members.aggregate([{
$lookup: {
from: "an_users",
let: {
membershipNumber: "$membershipNumber"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$memref",
"$$membershipNumber"
]
},
}
},
{
"$lookup": {
"from": "emails",
"localField": "user_id",
"foreignField": "click",
"as": "clicks"
}
},
{
"$project": {
"_id": 1,
"membershipNumber": 1,
"an_user_id": "$user_id",
"n_email_clicks": {
$size: "$clicks"
}
}
}
],
as: "details"
}
},
{
$replaceRoot: {
newRoot: {
$mergeObjects: [
{
$arrayElemAt: [
"$details",
0
]
},
"$$ROOT"
]
}
}
},
{
$project: {
details: 0
}
}])
Working example - https://mongoplayground.net/p/yrFsNp44hpi

Getting concrete elements by element field in mongoDB

I know that by the title it is not very clear what is my problem, so let me explain it with an example.
Let's suppose I have a collection in a mongo database called tweets whose elements look like this:
{
"id": "tweet_id",
"text": "this is the tweet's text",
"user": {
"id": "user_id",
"name": "user_name",
}
}
Let's suppose that we have 100 documents that look like that, and 10 users with diferent ids and names.
How would look a query to know see the different user_id s that exist in the collection and their names?
The result I want would look like this:
{"user_id1": "user_name1"}
{"user_id2": "user_name2"}
...
{"user_id10": "user_name10"}
Thank you for your help
You can use this aggregation query:
First $group by user.id to get all differents user ids with the name.
And then use $replaceRoot with $arrayToObject to get the desired output format.
db.collection.aggregate([
{
"$group": {
"_id": "$user.id",
"name": {
"$first": "$user.name"
}
}
},
{
"$replaceRoot": {
"newRoot": {
"$arrayToObject": [
[
{
"k": "$_id",
"v": "$name"
}
]
]
}
}
}
])
Example here

$elemMatch doesn't work on nested documents in MongoDB

Stack Overflow!
I have a very strange problem with using $elemMatch in MongoDB. I added multiple documents to a collection. Some of these documents were added using import feature in MongoDB Compass (Add Data -> Import File -> JSON) and some of them were added using insertMany().
Here is an example structure of a single document:
{
"id": "1234567890",
"date": "YYYY-MM-DD",
"contents": {
"0": {
"content": {
"id": "1111111111",
"name": "Name 1"
}
},
"1": {
"content": {
"id": "2222222222",
"name": "Name 2"
}
},
"2": {
"content": {
"id": "3333333333",
"name": "Name 3"
}
}
}
}
The thing is, when I use the following find query using this filter:
{date: "<some_date_here>", "contents": {
$elemMatch: {
"content.id": <some_id_here>
}
}}
ONLY documents that were imported from MongoDB Compass are showing up. Documents that were added by Mongosh or by NodeJS driver (doesn't matter), do NOT show up.
Am I missing something obvious here? What should I do in order to make all documents in a collection (that matches filter) to show up?
Simple filters that do not include $elemMatch work well and all documents that match the filtering rules show up. Problem seems to be with $elemMatch.
I tried adding the same batch of documents using different methods but only direct importing a JSON file in MongoDB Compass make them appear using a filter mentioned above.
Thank you for your help!
$elemMatch if for matching array , and in this case you don't have array
first you should convert contents object to array and then check the query for example id with filter and use match to find all doc that have specific data and size of new filters array
db.collection.aggregate([
{
"$addFields": {
"newField": {
"$objectToArray": "$contents"
}
}
},
{
"$addFields": {
"newField": {
"$filter": {
"input": "$newField",
"as": "z",
"cond": {
$eq: [
"$$z.v.content.id",
"1111111111"
]
}
}
}
}
},
{
"$addFields": {
"newField": {
$size: "$newField"
}
}
},
{
$match: {$and:[ {newField: {
$gt: 0
}},{date:{$gt:Date}}]}
},
{$project:{
contents:1,
date:1,
id:1,
}}
])
https://mongoplayground.net/p/pue4QPp1dYR
in mongoplayground I don't add filter of date

How to lookup a field in an array of subdocuments in mongoose?

I have an array of review objects like this :
"reviews": {
"author": "5e9167c5303a530023bcae42",
"rate": 5,
"spoiler": false,
"content": "This is a comment This is a comment This is a comment.",
"createdAt": "2020-04-12T16:08:34.966Z",
"updatedAt": "2020-04-12T16:08:34.966Z"
},
What I want to achieve is to lookup the author field and get the user data, but the problem is that the lookup I am trying to use only returns this to me:
Code :
.lookup({
from: 'users',
localField: 'reviews.author',
foreignField: '_id',
as: 'reviews.author',
})
Response :
Any way to get the author's data in that field? That's where the author's Id is.
Try to execute below query on your database :
db.reviews.aggregate([
/** unwind in general is not needed for `$lookup` for if you wanted to match lookup result with specific elem in array is needed */
{
$unwind: { path: "$reviews", preserveNullAndEmptyArrays: true },
},
{
$lookup: {
from: "users",
localField: "reviews.author",
foreignField: "_id",
as: "author", // Pull lookup result into 'author' field
},
},
/** Update 'reviews.author' field in 'reviews' object by checking if 'author' field got a match from 'users' collection.
* If Yes - As lookup returns an array get first elem & assign(As there will be only one element returned -uniques),
* If No - keep 'reviews.author' as is */
{
$addFields: {
"reviews.author": {
$cond: [
{ $ne: ["$author", []] },
{ $arrayElemAt: ["$author", 0] },
"$reviews.author",
],
},
},
},
/** Group back the documents based on '_id' field & push back all individual 'reviews' objects to 'reviews' array */
{
$group: {
_id: "$_id",
reviews: { $push: "$reviews" },
},
},
]);
Test : MongoDB-Playground
Note : Just in case if you've other fields in document along with reviews that needs to be preserved in output then starting at $group use these stages :
{
$group: {
_id: "$_id",
data: {
$first: "$$ROOT"
},
reviews: {
$push: "$reviews"
}
}
},
{
$addFields: {
"data.reviews": "$reviews"
}
},
{
$project: {
"data.author": 0
}
},
{
$replaceRoot: {
newRoot: "$data"
}
}
Test : MongoDB-Playground
Note : Try to keep queries to run on lesser datasets maybe by adding $match as first stage to filter documents & also have proper indexes.
you should use populate('author') method of mongoose on the request to the server which gets the id of that author and adds the user data to the response of mongoose
and dont forget to set your schema in a way that these two collections are connected
in your review schema you should add ref to the schema which the author user is saved
author: { type: Schema.Types.ObjectId, ref: 'users' },
You can follow this code
$lookup:{
from:'users',
localField:'reviews.author',
foreignField:'_id',
as:'reviews.author'
}
**OR**
> When You find the doc then use populate
> reviews.find().populate("author")

How to $lookup/populate an embedded document that is inside an array?

How to $lookup/populate an embedded document that is inside an array?
Below is how my schema is looking like.
const CommentSchema = new mongoose.Schema({
commentText:{
type:String,
required: true
},
arrayOfReplies: [{
replyText:{
type:String,
required: true
},
replier: [{
type: mongoose.Schema.Types.ObjectId,
ref: 'User',
required: true,
}],
}],
});
How can I get query results that look like below:
[
{
commentText: 'comment text',
arrayOfReplies: [
{
replyText: 'replyText',
replier: {
username:"username"
bio: 'bio'
}
}
]
}
]
I am trying to populate the replier field inside the array arrayOfReplies. I have tried several variations of the aggregation query below. The ones that have come close to what I am trying to achieve have one short-coming. The comments that do not have replies have an arrayOfReplies array that has an empty object. I.e arrayOfReplies: [{}], essentially meaning that the array is not empty.
I have tried using add fields, $mergeObjects among other pipeline operators but to no avail.
How to $lookup/populate the replier document that is inside the arrayOfReplies array?
Below is a template of the main part of my aggregation query, minus trying populate the replier document.
Comment.aggregate([
{$unwind: {"path": '$arrayOfReplies', "preserveNullAndEmptyArrays": true }},
{$lookup:{from:"users",localField:"$arrayOfReplies.replier",foreignField:"_id",as:"replier"}},
{$unwind: {"path": "$replier", "preserveNullAndEmptyArrays": true }},
{$group: {
_id : '$_id',
commentText:{$first: '$commentText'},
userWhoPostedThisComment:{$first: '$userWhoPostedThisComment'},
arrayOfReplies: {$push: '$arrayOfReplies' },
}},
After your lookup stage, each document will have
{
commentText: "text",
arrayOfReplies: <single reply, with replier ID>
replier: [<looked up replier data>]
}
Use an $addFields stage to move that replier data inside the reply object before the group, like:
{$addFields: {"arrayOfReplies.replier":"$replier"}}
Then your group stage will rebuild arrayOfReplies like you want.
You can use the following aggregate:
Playground
Comment.aggregate([
{
$unwind: {
"path": "$arrayOfReplies",
"preserveNullAndEmptyArrays": true
}
},
{
$lookup: {
from: "users",
localField: "arrayOfReplies.replier",
foreignField: "_id",
as: "replier"
}
},
{
$addFields: {
"arrayOfReplies.replier": {
$arrayElemAt: [
"$replier",
0
]
}
}
},
{
$project: {
"replier": 0
}
},
{
$group: {
_id: "$_id",
"arrayOfReplies": {
"$push": "$arrayOfReplies"
},
commentText: {
"$first": "$commentText"
}
}
}
]);
All the answers provided did not solve this issue as stated in the question.
I am trying to populate the replier field inside the array
arrayOfReplies. I have tried several variations of the aggregation
query below. The ones that have come close to what I am trying to
achieve have one short-coming. The comments that do not have replies
have an arrayOfReplies array that has an empty object. I.e
arrayOfReplies: [{}], essentially meaning that the array is not empty.
I wanted an aggregation that returns an empty array (not an array with an empty object) when the array is empty.
I was able to achieve what I wanted by using the code below:
arrayOfReplies:
{$cond:{
if: { $eq: ['$arrayOfReplies', {} ] },
then: "$$REMOVE",
else: {
_id : '$arrayOfReplies._id',
replyText:'$arrayOfReplies.replyText',
}
}}
If you combine the code above with #SuleymanSah's answer you get the full working code.