Aggregate and reduce a nested array based upon an ObjectId - mongodb

I have an Event document structured like so and I'm trying to query against the employeeResponses array to gather all responses (which may or may not exist) for a single employee:
[
{
...
eventDate: 2019-10-08T03:30:15.000+00:00,
employeeResponses: [
{
_id:"5d978d372f263f41cc624727",
response: "Available to work.",
notes: ""
},
...etc
];
}
];
My current mongoose aggregation is:
const eventResponses = await Event.aggregate([
{
// find all events for a selected month
$match: {
eventDate: {
$gte: startOfMonth,
$lte: endOfMonth,
},
},
},
{
// unwind the employeeResponses array
$unwind: {
path: "$employeeResponses",
preserveNullAndEmptyArrays: true,
},
},
{
$group: {
_id: null,
responses: {
$push: {
// if a response id matches the employee's id, then
// include their response; otherwise, it's a "No response."
$cond: [
{ $eq: ["$employeeResponses._id", existingMember._id] },
"$employeeResponses.response",
"No response.",
],
},
},
},
},
{ $project: { _id: 0, responses: 1 } },
]);
As you'll no doubt notice, the query above won't work after more than 1 employee records a response because it treats each individual response as a T/F condition, instead of all of the responses within the employeeResponses array as a single T/F condition.
As a result, I had remove all subsequent queries after the initial $match and do a manual reduce:
const responses = eventResponses.reduce((acc, { employeeResponses }) => {
const foundResponse = employeeResponses.find(response => response._id.equals(existingMember._id));
return [...acc, foundResponse ? foundResponse.response : "No response."];
}, []);
I was wondering if it's possible to achieve the same reduce result above, but perhaps using mongo's $reduce function? Or refactor the aggregation query above to treat all responses within the employeeResponses as a single T/F condition?
The ultimate goal of this aggregation is extract any previously recorded employee's responses and/or lack of a response from each found Event within a current month and place their responses into a single array:
["I want to work.", "Available to work.", "Not available to work.", "No response.", "No response." ...etc]

You can use $filter with $map to reshape your data and filter by _id. Then you can keep using $push with $ifNull to provide default value if an array is empty:
db.collection.aggregate([
{
$addFields: {
employeeResponses: {
$map: {
input: {
$filter: {
input: "$employeeResponses",
cond: {
$eq: [ "$$this._id", "5d978d372f263f41cc624727"]
}
}
},
in: "$$this.response"
}
}
}
},
{
$group: {
_id: null,
responses: { $push: { $ifNull: [ { $arrayElemAt: [ "$employeeResponses", 0 ] }, "No response" ] } }
}
}
])
Mongo Playground

Related

Mongoose - filter matched documents and assign the resultant length to a field

I have this collection(some irrelevant fields were omitted for brevity):
clients: {
userId: ObjectId,
clientSalesValue: Number,
currentDebt: Number,
}
Then I have this query that matches all the clients for a specific user, then calculates the sum of all debts and sales and put those results in a separate field each of them:
await clientsCollection.aggregate([
{
$match: { userId: new ObjectId(userId) }
},
{
$group: {
_id: null,
totalSalesValue: { $sum: '$clientSalesValue' },
totalDebts: { $sum: '$currentDebt' },
}
},
{
$unset: ['_id']
}
]).exec();
This works as expected, it returns an array with only one item which is an object, but now I need to also include in that resultant object a field for the amount of debtors, that is for the amount of clients that have currentDebt > 0, how can I do that is the same query? is it possible?
PD: I cannot modify the $match condition, it need to always return all the clients for the corresponding users.
To include a count of how many matching documents have a positive currentDebt, you can use the $sum and $cond operators like so:
await clientsCollection.aggregate([
{
$match: { userId: new ObjectId(userId) }
},
{
$group: {
_id: null,
totalSalesValue: { $sum: '$clientSalesValue' },
totalDebts: { $sum: '$currentDebt' },
numDebtors: {
$sum: {
$cond: [{ $gt: ['$currentDebt', 0] }, 1, 0]
}
},
}
},
{
$unset: ['_id']
}
]).exec();

after aggregation how to check two fields are equal inside a document in mongodb

{
id: 1,
name: "sree",
userId: "001",
paymentData: {
user_Id: "001",
amount: 200
}
},
{
id: 1,
name: "sree",
userId: "001",
paymentData: {
user_Id: "002",
amount: 200
}
}
I got this result after unwind in aggregation any way to check user_Id equal to userId
Are you looking to only retrieve the results when they are equal (meaning you want to filter out documents where the values are not the same) or are you looking to add a field indicating whether the two are equal?
In either case, you append subsequent stage(s) to the aggregation pipeline to achieve your desired result. If you want to filter the documents, the new stage may be:
{
$match: {
$expr: {
$eq: [
"$userId",
"$paymentData.user_Id"
]
}
}
}
See how it works in this playground example.
If instead you want to add a field that compares the two values, then this stage may be what you are looking for:
{
$addFields: {
isEqual: {
$eq: [
"$userId",
"$paymentData.user_Id"
]
}
}
}
See how it works in this playground example.
You could also combine the two as in:
{
$addFields: {
isEqual: {
$eq: [
"$userId",
"$paymentData.user_Id"
]
}
}
},
{
$match: {
isEqual: true
}
}
Playground demonstration here

Conditionally update/upsert embedded array with findOneAndUpdate in MongoDB

I have a collection in the following format:
[
{
"postId": ObjectId("62dffd0acb17483cf015375f"),
"userId": ObjectId("62dff9584f5b702d61c81c3c"),
"state": [
{
"id": ObjectId("62dffc49cb17483cf0153220"),
"notes": "these are my custom notes!",
"lvl": 3,
},
{
"id": ObjectId("62dffc49cb17483cf0153221"),
"notes": "hello again",
"lvl": 0,
},
]
},
]
My goal is to be able to update and add an element in this array in the following situation:
If the ID of the new element is not in the state array, push the new element in the array
If the ID of the new element is in the state array and its lvl field is 0, update that element with the new information
If the ID of the new element exists in the array, and its lvl field is not 0, then nothing should happen. I will throw an error by seeing that no documents were matched.
Basically, to accomplish this I was thinking about using findOneAndUpdate with upsert, but I am not sure how to tell the query to update the state if lvl is 0 or don't do anything if it is bigger than 0 when the match is found.
For solving (1) this is what I was able to come up with:
db.collection.findOneAndUpdate(
{
"postId": ObjectId("62dffd0acb17483cf015375f"),
"userId": ObjectId("62dff9584f5b702d61c81c3c"),
"state.id": {
"$ne": ObjectId("62dffc49cb17483cf0153222"),
},
},
{
"$push": {"state": {"id": ObjectId("62dffc49cb17483cf0153222"), "lvl": 1}}
},
{
"new": true,
"upsert": true,
}
)
What is the correct way to approach this issue? Should I just split the query into multiple ones?
Edit: as of now I have done this in more than one query (one to fetch the document, then I iterate over its state array to check if the ID exists in it, and then I perform (1), (2) and (3) in a normal if-else clause)
If the ID of the new element exists in the array, and its lvl field is not 0, then nothing should happen. I will throw an error by seeing that no documents where matched.
First thing FYI,
upsert is not possible in the nested array
upsert will not add new elements to the array
upsert can add a new document with the new element
if you want to throw an error if the record does not present then you don't need upsert
Second thing, you can achieve this in one query by using an update with aggregation pipeline in MongoDB 4.2,
Note: Here i must inform you, this query will respond updated document but there will be no flag or any clue if this query fulfilled your first situation or second situation, or the third situation out of 3, you have to check in your client-side code through query response.
check conditions for postId and userId fields only
we are going to update state field under $set stage
check the condition if the provided id is present in state's id?
true, $map to iterate loop of state array
check conditions for id and lvl: 0?
true, $mergeObjects to merge current object with the new information
false, it will not do anything
false, then add that new element in state array, by $concatArrays operator
db.collection.findOneAndUpdate(
{
postId: ObjectId("62dffd0acb17483cf015375f"),
userId: ObjectId("62dff9584f5b702d61c81c3c")
},
[{
$set: {
state: {
$cond: [
{ $in: [ObjectId("62dffc49cb17483cf0153221"), "$state.id"] },
{
$map: {
input: "$state",
in: {
$cond: [
{
$and: [
{ $eq: ["$$this.id", ObjectId("62dffc49cb17483cf0153221")] },
{ $eq: ["$$this.lvl", 0] }
]
},
{
$mergeObjects: [
"$$this",
{
// update your new fields here
"notes": "new note"
}
]
},
"$$this"
]
}
}
},
{
$concatArrays: [
"$state",
[
// add new element
{
"id": ObjectId("62dffc49cb17483cf0153221"),
"lvl": 1
}
]
]
}
]
}
}
}],
{ returnNewDocument: true }
)
Playrgound
Third thing, you can execute 2 update queries,
The first query, for the case: element does not present and it will push a new element in state
let response = db.collection.findOneAndUpdate({
postId: ObjectId("62dffd0acb17483cf015375f"),
userId: ObjectId("62dff9584f5b702d61c81c3c"),
"state.id": { $ne: ObjectId("62dffc49cb17483cf0153221") }
},
{
$push: {
state: {
id: ObjectId("62dffc49cb17483cf0153221"),
lvl: 1
}
}
},
{
returnNewDocument: true
})
The second query on the base of if the response of the above query is null then this query will execute,
This will check state id and lvl: 0 conditions if conditions are fulfilled then execute the update fields operation, it will return null if the document is not found
You can throw if this will return null otherwise do stuff with response data and response success
if (response == null) {
response = db.collection.findOneAndUpdate({
postId: ObjectId("62dffd0acb17483cf015375f"),
userId: ObjectId("62dff9584f5b702d61c81c3c"),
state: {
$elemMatch: {
id: ObjectId("62dffc49cb17483cf0153221"),
lvl: 0
}
}
},
{
$set: {
// add your update fields
"state.$.notes": "new note"
}
},
{
returnNewDocument: true
});
// not found and throw an error
if (response == null) {
return {
// throw error;
};
}
}
// do stuff with "response" data and return result
return {
// success;
};
Note: As per the above options, I would recommend you that I explained in the Third thing that you can execute 2 update queries.
What you're trying became possible with the introduction pipelined updates, here is how I would do it by using $concatArrays to concat the exists state array with the new input and $ifNull in case of an upsert to init the empty value, like so:
const inputObj = {
"id": ObjectId("62dffc49cb17483cf0153222"),
"lvl": 1
};
db.collection.findOneAndUpdate({
"postId": ObjectId("62dffd0acb17483cf015375f"),
"userId": ObjectId("62dff9584f5b702d61c81c3c")
},
[
{
$set: {
state: {
$ifNull: [
"$state",
[]
]
},
}
},
{
$set: {
state: {
$concatArrays: [
{
$map: {
input: "$state",
in: {
$mergeObjects: [
{
$cond: [
{
$and: [
{
$in: [
inputObj.id,
"$state.id"
]
},
{
$eq: [
inputObj.lvl,
0
]
}
]
},
inputObj,
{},
]
},
"$$this"
]
}
}
},
{
$cond: [
{
$not: {
$in: [
inputObj.id,
"$state.id"
]
}
},
[
],
[]
]
}
]
}
}
}
],
{
"new": true,
"upsert": true
})
Mongo Playground
Prior to version 4.2 and the introduction of this feature what you're trying to do was not possible using the naive update syntax, If you are using an older version then you'd have to split this into 2 separate calls, first a findOne to see if the document exists, and only then an update based on that. obviously this can cause stability issue's if you have high update volume.

MongoDB set of values with a limit size

I am updating a list of transactions by saving the transaction into the database list, I do not want to have duplicate entries in the list so I use $addtoset
this is because the request can be fired multiple times and we want to make sure that any changes are idempotent to the database. the only catch now is that we want to only store the latest 20 transactions
this could be done with a $push $sort $slice but I need to make sure duplicate entries are not available. there was a feature request to mongo back in 2015 for this to be added to the $addtoset feature, but they declined this due to 'sets' not being in an order...
which is what the $sort function would have been
I thought I could simply append an empty push update to the update object, but from what I understand, each update is potentially threaded and can lead to undesirable edits if the push/slice fires before the $addtoset
right now, the values are an aggregated string with the following formula
timestamp:value but I can easily change the structure to an object
{ts:timestamp, value:value}
Update:
current code, not sure if it will work as intended as each operation maybe independent
await historyDB
.updateOne(
{ trxnId: txid },
{
$addToSet: {
history: {
ts: time,
bid: bid.value,
txid: trxn.txid,
}
},
$push: {
history: {
$each: [{ts:-1}],
$sort: { ts: 1 },
$slice: -10,
},
},
},
{ upsert: true },
).exec();
Your query doesn't work, as you are trying to update history multiple times, which is not allowed in simple update document and raises error Updating the path 'history' would create a conflict at 'history'.
You can however subsequently update history field multiple times with aggregation pipeline.
await historyDB.updateOne(
{ trxnId: txid},
[{
$set: {
history: {
$let: {
vars: {
historyObj: {
ts: time,
bid: bid.value,
txid: trxn.txid,
},
historySafe: { $ifNull: ["$history", []] }
},
in: {
$cond: {
if: { $in: ["$$historyObj", "$$historySafe"] },
then: "$history",
else: { $concatArrays: [ "$$historySafe", ["$$historyObj"] ] }
}
}
}
}
},
},
{
$set: {
history: {
$function: {
body: function(entries) {
entries.sort((a, b) => a.ts - b.ts);
return entries;
},
args: [{ $ifNull: ["$history", []] }],
lang: "js"
}
}
},
},
{
$set: {
history: {
$slice: [ "$history", -10 ]
}
}
}],
{ upsert: true },
).exec()
As of MongoDB 6.0, the second $set stage, which provides sorting, can be replaced with $sortArray operator (see here).

MongoDB aggregation: How to get the index of a document in a collection depending sorted by a document property

Assume I have a collection with millions of documents. Below is a sample of how the documents look like
[
{ _id:"1a1", points:[2,3,5,6] },
{ _id:"1a2", points:[2,6] },
{ _id:"1a3", points:[3,5,6] },
{ _id:"1b1", points:[1,5,6] },
{ _id:"1c1", points:[5,6] },
// ... more documents
]
I want to query a document by _id and return a document that looks like below:
{
_id:"1a1",
totalPoints: 16,
rank: 29
}
I know I can query the whole document, sort by descending order then get the index of the document I want by _id and add one to get its rank. But I have worries about this method.
If the documents are in millions won't this be 'overdoing' it. Querying a whole collection just to get one document? Is there a way to achieve what I want to achieve without querying the whole collection? Or the whole collection has to be involved because of the ranking?
I cannot save them ranked because the points keep on changing. The actual code is more complex but the take away is that I cannot save them ranked.
Total points is the sum of the points in the points array. The rank is calculated by sorting all documents in descending order. The first document becomes rank 1 and so on.
an aggregation pipeline like the following can get the result you want. but how it operates on a collection of millions of documents remains to be seen.
db.collection.aggregate(
[
{
$group: {
_id: null,
docs: {
$push: { _id: '$_id', totalPoints: { $sum: '$points' } }
}
}
},
{
$unwind: '$docs'
},
{
$replaceWith: '$docs'
},
{
$sort: { totalPoints: -1 }
},
{
$group: {
_id: null,
docs: { $push: '$$ROOT' }
}
},
{
$set: {
docs: {
$map: {
input: {
$filter: {
input: '$docs',
as: 'x',
cond: { $eq: ['$$x._id', '1a3'] }
}
},
as: 'xx',
in: {
_id: '$$xx._id',
totalPoints: '$$xx.totalPoints',
rank: {
$add: [{ $indexOfArray: ['$docs._id', '1a3'] }, 1]
}
}
}
}
}
},
{
$unwind: '$docs'
},
{
$replaceWith: '$docs'
}
])