MongoDB $match stage after $replaceRoot - mongodb

Consider the following.
I have documents in which I save versions of fields.
Fields in the original document can null and I only save fields that actually changed.
For example:
{
_id : abc,
state : 'NEW',
field2 : 'WILL NOT CHANGE',
field3 : null,
lastChange : 1236547
changes : [
{
state : 'WORKING',
field3 : 'SOMETHING'
},
{
state : 'DONE'
}
]
}
In my aggregation pipeline I build the latest version of the document by merging all versions with the document root. This means I avoid complex querys in which I search the changes array and find the latest version of every field I want to find and I benefit from the side effect that the pipeline will return the latest document version as a document. The pipeline below ends at the point where my problem is. In the application there are more stages following that do counting and grouping etc.
Mongoose: meldungs.aggregate([
{
'$match': {
'$and': [{
lastChange: {
'$ne': null
}
},
{
lastChange: {
'$gte': ISODate('2017-03-01T23:00:00.000Z')
}
},
{
lastChange: {
'$lt': ISODate('2017-03-02T22:59:59.999Z')
}
}]
}
},
{
'$replaceRoot': {
newRoot: {
'$arrayToObject': {
'$reduce': {
input: '$changes',
initialValue: {
'$filter': {
input: {
'$objectToArray': '$$ROOT'
},
as: 'field',
cond: {
'$and': [
{ $ne : [ '$$field.v', null ] },
{ $ne : [ '$$field.k', '_id' ] }
]
}
}
},
in: {
'$concatArrays': ['$$value',
{
'$filter': {
input: {
'$objectToArray': '$$this.changes'
},
as: 'field',
cond: {
$ne : [ '$$field.v', null ]
}
}
}]
}
}
}
}
}
},
{
'$match': {
state: { '$eq': 'DONE' }
}
],
{})
This is working great and the new document looks as expected:
{
state : 'DONE',
field2 : 'WILL NOT CHANGE',
field3 : 'SOMETHING',
lastChange : 1236547
}
My problem ist that the matching stage that follows the merge/reduce fails to 'query' the new document. Instead the old field values seem to be used.
state: { '$eq': 'DONE' } // should return the document, but doesn't
state: { '$eq': 'NEW' } // should NOT return the document but does
My best guess is that this has something to do with pipeline optimization? Meaning the optimizer moves the $match statement into the first stage, or something like that.
Does anybody had this problem as well and knows a solution?
Udpate 1
After more testing it seems that using $$ROOT as the initialValue of the reduce is part of the problem. If I use an empty array as the initialValue I can $match the reduced fields correctly.

Related

Merge arrays by matching similar values in mongodb

This is an extension of the below question.
Filter arrays in mongodb
I have a collection where each document contains 2 arrays as below.
{
users:[
{
id:1,
name:"A"
},
{
id:2,
name:"B"
},
{
id:3,
name:"C"
}
]
priv_users:[
{
name:"X12/A",
priv:"foobar"
},
{
name:"Y34.B",
priv:"foo"
}
]
}
From the linked question, I learnt to use $map to merge 2 document arrays. But I can't figure out to match users.name to priv_users.name to get below output.
{
users:[
{
id:1,
name:"A",
priv:"foobar"
},
{
id:2,
name:"B",
priv:"foo"
},
{
id:3,
name:"C"
}
]
}
users.name and priv_users.name don't have a consistent pattern, but users.name exists within priv_users.name.
MongoDB version is 4.0
This may not be as generic but will push you in the right direction. Consider using the operators $mergeObjects to merge the filtered document from the priv_users array with the document in users.
Filtering takes the $substr of the priv_users name field and compares it with the users name field. The resulting pipeline will be as follows
db.collection.aggregate([
{ '$addFields': {
'users': {
'$map': {
'input': '$users',
'in': {
'$mergeObjects': [
{
'$arrayElemAt': [
{
'$filter': {
'input': '$priv_users',
'as': 'usr',
'cond': {
'$eq': [
'$$this.name',
{ '$substr': [
'$$usr.name', 4, -1
] }
]
}
}
},
0
]
},
'$$this'
]
}
}
}
} }
])
If using MongoDB 4.2 and newer versions, consider using $regexMatch operator for matching the priv_users name field with the users name field as the regex pattern. Your $cond operator now becomes:
'cond': {
'$regexMatch': {
'input': '$$usr.name',
'regex': '$$this.name',
'options': "i"
}
}

find and update with pull and get the pulled document in return

Here is my collection
[
{_id:1,
persons:[{name:"Jack",age:12},{name:"Ma",age:13}]
}
]
I want to remove {name:"Jack",age:12} in persons by pull but I also want after pulling is completed I will be returned the pulled {name:"Jack",age:12}. How can I do this?
I want to do it like this
db.getCollection('test').findOneAndUpdate({_id:1},{$pull:{"persons":{name:"Jack"}}},
{projection:{"person":{
$filter : {
input: "$persons",
as : "ele",
cond : {$eq : ["$$ele.name","Jack"]}
}
}}})
You can use $reduce, because $filter will return array, also aggregation array operators will support from MongoDB 4.4,
db.getCollection('test').findOneAndUpdate({_id:1},
{ $pull: { "persons": { name: "Jack" } } },
{
projection: {
"person":{
$reduce: {
input: "$persons",
initialValue: {},
in: { $cond: [{$eq: ["$$this.name","Jack"]}, "$$this", "$$value"] }
}
}
}
}
)
Playground

How to conditionally project fields during aggregate in mongodb

I have a user document like:
{
_id: "s0m3Id",
_skills: ["skill1", "skill2"],
}
Now I want to unwind this document by the _skills field and add a score for each skill. So my aggregate looks like:
{
'$unwind': {'path': '$_skills', 'preserveNullAndEmptyArrays': true},
},
{
'$project': {
'_skills':
'label': '$_skills',
'skill_score': 1
},
}
},
Sometimes the _skills field can be empty, however in this case I still want the user document to flow through the aggregation - hence the preserveNullAndEmptyArrays parameter. However, the problem I'm having is that it will project a skill_score (though with no label) onto documents which had empty _skills array fields. Thus, when I go to $group the documents later on, those documents now have a non-empty _skills array, containing a single object, namely {skill_score: 1}. This is not what I want - I want documents which had empty (or non-existent) _skills fields to not have any skill_score projected onto them.
So how can I conditionally project a field based on the existence of another field? Using $exists does not help, because that is intended for querying, not for boolean expressions.
Updated
This aggregation will set the value of skill_score to 0 if _skills does not exist, then use $redact to remove the subdocument having skill_score equals to 0:
db.project_if.aggregate([
{
$unwind: {
path: '$_skills',
preserveNullAndEmptyArrays: true,
}
},
{
$project: {
_skills: {
label: '$_skills',
skill_score: {
$cond: {
if: {
$eq: ['$_skills', undefined]
},
then: 0,
else: 1,
}
}
}
}
},
{
$redact: {
$cond: {
if: { $eq: [ "$skill_score", 0 ] },
then: '$$PRUNE',
else: '$$DESCEND'
}
}
}
]);
Result would be like:
[
{ "_id" : '', "_skills" : { "label" : "skill1", "skill_score" : 1 } },
{ "_id" : '', "_skills" : { "label" : "skill2", "skill_score" : 1 } },
{ "_id" : '' },
]

"dotted field names are only allowed at the top level" when doing $cond

I am doing an aggregation with a project with a cond. Something like this...
Assume Document of:
{outter:{
inner1:"blah",
innerOther:"somethingelse"
}}
Aggregation:
db.collection.aggregate([{
$project : {
"outter.inner1":1,
"outter.inner2":{
$cond : if:{"outter.innerOther":{$exists:true}},
then:{"blah":"blah"},
else:{"blah":"blah"}
}
}
}])
When I run this I get an error: exception: dotted field names are only allowed at the top level on the if condition statement (I tried replacing the if with if:true and the aggregation works).
Is this a bug? Is there a workaround without doing a whole other project to simply get the field available without a dot?
EDIT
So I found a workaround but would still like to see if this is expected behavior. The workaround is to use a variable in the projection via $let. Also, it appears that $exists is not a valid expression so had to also make the following change.
db.collection.aggregate([{
$project : {
"outter.inner1":1,
"outter.inner2":{
$let :{
vars:{innerOther:{$ifNull:["$outter.innerOther", "absent"]}},
}
$cond : if:{$eq:["$$innerOther","absent"]},
then:{"blah":"blah"},
else:{"blah":"blah"}
}
}
}])
Firstly you can't use the $exists because it's only valid in expression with the $match operator or within the .find() method. That being said you can use the "dot notation" in the if condition but you need to prefix it with the $ sign which is missing in your first query. Also you can't return a literal object key/value pairs as value of or expression because it is not a valid aggregation expressions you need to use the "dot notation" instead.
Now one way to do this as you mention is using the $let operation and $projection. Of course the $ifNull conditional operator returns the present value of the field if it exists or the replacement expression.
db.collection.aggregate([
{ $project: {
"outter.inner1": 1,
"outter.inner2.blah": {
$let: {
vars: {
"outterinner2": {
$ifNull: [ "$outter.innerOther", "absent" ]
}
},
in: {
$cond: [
{ $eq: [ "$$outterinner2", "absent" ] },
"blah",
"blah"
]
}
}
}
}}
])
Another way of doing this is using two $projection stages.
db.collection.aggregate([
{ $project: {
"inner1": "$outter.inner1",
"inner2": {
$ifNull: [ "$outter.innerOther", "absent" ]
}
}},
{ $project: {
"outter.inner1": "$inner1",
"outter.inner2.blah": {
$cond: [
{ $eq: ["$inner2", "absent"] },
"blah",
"blah"
]
}
}}
])
Both queries yield something like this:
{
"_id" : ObjectId("5632713c8b13e9fb9cc474f2"),
"outter" : {
"inner1" : "blah",
"inner2" : {
"blah" : "blah"
}
}
}
EDIT:
"dot notation" is allowed in the $cond operator. For example the following query use the "dot notation".
db.collection.aggregate([
{ $project: {
"outter.inner1": 1,
"outter.inner2.blah": {
$cond: [
{ $eq: [ "$outter.innerOther", "somethingelse" ] },
"anotherThing",
"nothing"
]
}
}}
])
and yields:
{
"_id" : ObjectId("56379b020d97b83cd1506650"),
"outter" : {
"inner1" : "blah",
"inner2" : {
"blah" : "anotherThing"
}
}
}
hmmm.... and by this:
Model.aggregate([
{
$project: {
name: 1,
"indexes.name":{
$cond: {
if: {$eq:["$indexes.name",'стафилококки']},
then: "$indexes.name",
else: null
}
},
"indexes.units":1
}
}
], callback)
always return "null" for name.indexes
[
{ _id: 564c6a5991b08dd017c2bd23,
name: ' воздух ',
indexes: [ [Object] ] },
{ _id: 564c6cf091b08dd017c2bd24,
name: 'repa ',
indexes: [ [Object] ] },
]
where indexes are:
[ { units: 'fvf', name: null } ]
[ { units: 'КОЕ', name: null } ]

way to update multiple documents with different values

I have the following documents:
[{
"_id":1,
"name":"john",
"position":1
},
{"_id":2,
"name":"bob",
"position":2
},
{"_id":3,
"name":"tom",
"position":3
}]
In the UI a user can change position of items(eg moving Bob to first position, john gets position 2, tom - position 3).
Is there any way to update all positions in all documents at once?
You can not update two documents at once with a MongoDB query. You will always have to do that in two queries. You can of course set a value of a field to the same value, or increment with the same number, but you can not do two distinct updates in MongoDB with the same query.
You can use db.collection.bulkWrite() to perform multiple operations in bulk. It has been available since 3.2.
It is possible to perform operations out of order to increase performance.
From mongodb 4.2 you can do using pipeline in update using $set operator
there are many ways possible now due to many operators in aggregation pipeline though I am providing one of them
exports.updateDisplayOrder = async keyValPairArr => {
try {
let data = await ContestModel.collection.update(
{ _id: { $in: keyValPairArr.map(o => o.id) } },
[{
$set: {
displayOrder: {
$let: {
vars: { obj: { $arrayElemAt: [{ $filter: { input: keyValPairArr, as: "kvpa", cond: { $eq: ["$$kvpa.id", "$_id"] } } }, 0] } },
in:"$$obj.displayOrder"
}
}
}
}],
{ runValidators: true, multi: true }
)
return data;
} catch (error) {
throw error;
}
}
example key val pair is: [{"id":"5e7643d436963c21f14582ee","displayOrder":9}, {"id":"5e7643e736963c21f14582ef","displayOrder":4}]
Since MongoDB 4.2 update can accept aggregation pipeline as second argument, allowing modification of multiple documents based on their data.
See https://docs.mongodb.com/manual/reference/method/db.collection.update/#modify-a-field-using-the-values-of-the-other-fields-in-the-document
Excerpt from documentation:
Modify a Field Using the Values of the Other Fields in the Document
Create a members collection with the following documents:
db.members.insertMany([
{ "_id" : 1, "member" : "abc123", "status" : "A", "points" : 2, "misc1" : "note to self: confirm status", "misc2" : "Need to activate", "lastUpdate" : ISODate("2019-01-01T00:00:00Z") },
{ "_id" : 2, "member" : "xyz123", "status" : "A", "points" : 60, "misc1" : "reminder: ping me at 100pts", "misc2" : "Some random comment", "lastUpdate" : ISODate("2019-01-01T00:00:00Z") }
])
Assume that instead of separate misc1 and misc2 fields, you want to gather these into a new comments field. The following update operation uses an aggregation pipeline to:
add the new comments field and set the lastUpdate field.
remove the misc1 and misc2 fields for all documents in the collection.
db.members.update(
{ },
[
{ $set: { status: "Modified", comments: [ "$misc1", "$misc2" ], lastUpdate: "$$NOW" } },
{ $unset: [ "misc1", "misc2" ] }
],
{ multi: true }
)
Suppose after updating your position your array will looks like
const objectToUpdate = [{
"_id":1,
"name":"john",
"position":2
},
{
"_id":2,
"name":"bob",
"position":1
},
{
"_id":3,
"name":"tom",
"position":3
}].map( eachObj => {
return {
updateOne: {
filter: { _id: eachObj._id },
update: { name: eachObj.name, position: eachObj.position }
}
}
})
YourModelName.bulkWrite(objectToUpdate,
{ ordered: false }
).then((result) => {
console.log(result);
}).catch(err=>{
console.log(err.result.result.writeErrors[0].err.op.q);
})
It will update all position with different value.
Note : I have used here ordered : false for better performance.