The $match expression on mongodb's stitch application does not work properly.
I am trying to set up a simple update trigger that will only work on one field in a collection.
The trigger setup provides a $match aggregation which seems simple enough to set up.
For example if I want the trigger to only fire when the field "online" in a specified collection gets set to "true" I would do:
{"updateDescription.updatedFields":{"online":"true"}}
which for a stitch trigger is the same as:
{$match:{{updateDescription.updatedFields:{online:"true"}}}
The problem is when i try to match an update on a field that is an object.(for example hours:{online:40,offline:120}
For some reason $exists or $in does not work
So doing:
{"updateDescription.updatedFields":{"hours":{"$exists":true}}
does not work,neither does something like:
{"updateDescription.updatedFields":{"hours.online":{"$exists":true}}
The $match for the trigger is supposed to work exactly like a normal mongo $match.
They just provide one example :
{
"updateDescription.updatedFields": {
"status": "blocked"
}
}
The example is from here:
https://docs.mongodb.com/stitch/triggers/database-triggers/
I tried 100's of variations but i can't seem to get it
The trigger is working fine if the match is a specific value like:
{"updateDescription.updatedFields":{"hours.online":{"$numberInt\":"20"}}
and then i set the hours.online to 20 in the database.
I was able to have it match items by using an explicit $expr operator or declare it as a single field not an embedded object. ie. "updateDescription.updatedFields.statue": "blocked"
I struggled with this myself, trying to get a trigger to fire when a certain nested field was updated to any value (rather than just one specific one).
The issue appears to have to do with how change streams report updated fields.
With credit and thanks to MongoDB support, I can finally offer this as a potential solution, at least for simpler cases:
{
"$expr": {
"$not": {
"$cmp": [{
"$let": {
"vars": { "updated": { "$objectToArray": "$updateDescription.updatedFields" } },
"in": { "$arrayElemAt": [ "$$updated.k", 0 ] }
}},
"example.subdocument.nested_field"
]
}
}
}
Of course replace example.subdocument.nested_field with your own dot-notation field path.
Related
MongoDB has recently added an option to perform an update operation by providing an aggregation pipeline rather than the standard modifier object. Check MongoDB's docs on this topic.
The ability to use aggregation pipeline, whose statements can refer to existing document properties, can be extremely useful in situations when certain fields needs to be evaluated based on other fields, e.g. during data migration.
Moreover, most of the standard update operators like $set, $push, $inc, etc. can be successfully replicated with the aggregation expression language so in some sense this new functionality generalizes the good old modifiers technique. Though, I must admit the pipeline can become quite verbose if one tries to do things like $addToSet. This of course brings up a whole bunch of performance related questions, but let's ignore them for now.
So far, there's been just one thing which I haven't been able to fully replicate with the aggregation pipeline update, namely the $setOnInsert operator. Let's assume that I want to perform an upsert:
db.test.update(selector, pipeline, { upsert: true });
My initial intuition was that the $$ROOT variable (which I can use in the pipeline) will equal null unless there exists a document that matches selector. Unfortunately, but probably for a good reason, MongoDB developers decided that $$ROOT should be derived from selector by default. It makes sense when you think about how normal $setOnInsert works, but it also makes it practically impossible to distinguish between an update and an insert within pipeline.
I know what you're thinking. You can look at $$ROOT._id. This is a good idea, though if _id is part of the selector it doesn't work anymore. I have figured out that this can be bypassed by tricking MongoDB a little bit and doing things like:
selector = {
_id: { $in: [value, 'fake'] },
}
instead if the simpler { _id: value }, but this doesn't look clean. Please note that if $in only contains one element, then Mongo is actually clever enough to figure out what the identifier should be and it populates $$ROOT accordingly (sic!).
I am wondering if anyone has a better idea how to approach this. Maybe there's some hidden variable that I could potentially use inside the pipeline itself to distinguish between update and insert (e.g. in $merge stage there's $$new variable which serves a similar purpose)?
If there is no matching documents, $$ROOT will have only _id field. So you can transform $$ROOT to array by its key/value pairs and check if the size of that array is equal to 1. If it is then create a new document, and if it is not then do nothing.
$objectToArray and $size to convert $$ROOT to an array by its key/value pairs and to get the size of that array
$cond to check if the size of the array above is equal to 1. If it is then merge current $$ROOT (which is only _id field) with the update object. If it is not, return the current $$ROOT. In both scenarios, put result in result feild.
$mergeObjects to merge $$ROOT and the update that you are sending, and put that in the result field
$replaceRoot to replace root to the result field from previous stage
db.collection.update({
_id: 1
},
[
{
$set: {
result: {
$cond: {
if: {
"$eq": [
{
$size: {
$objectToArray: "$$ROOT"
},
},
1
]
},
then: {
$mergeObjects: [
"$$ROOT",
{
key: 3
}
]
},
else: "$$ROOT"
},
}
}
},
{
$replaceRoot: {
newRoot: "$result"
}
}
],
{
upsert: true
})
Working example
Update: I use "$match expression" to describe this but I don't actually use the $match operator. According to the docs, the selector should conform with $match's syntax, though the $match keyword is apparently not necessary in the actual expression.
Update 2: In the actual collection, outerField represents message, fieldA represents fansNo, and fieldB represents sharedNo. So outerField.fieldA represents message.fansNo and outerField.fieldB represents message.sharedNo. This is a stringified representation of the updateDescription field when the trigger fires (i.e. when I only specify updateDescription.updatedField in the match expression):
"updateDescription: {\"removedFields\":[],\"updatedFields\":{\"someOtherField\":310,\"message.fansNo\":1,\"updatedAt\":\"2020-06-22T13:29:08.829Z\"}}"
================================================================
Original post:
So I can't understand why it fails to trigger when I specify message.fansNo and message.sharedNo in the match expression.
I am setting up a database trigger on updates to a collection, but I'm not able to get my $match expression to work in filtering the change events that cause the trigger to fire. I want to fire the trigger only if one or both of 2 nested fields are present, say fieldA and fieldB. These 2 fields are nested inside an object, and the object is the value of a field in each document. Something like this:
// CollectionA schema
{
_id: ...,
outerField: {
fieldA: 1 // or any number
fieldB: 2 // or any number
},
...
}
I have tried using this $match expression below, but the trigger doesn't fire:
{
"$or": [
{
"updateDescription.updatedFields.outerField.fieldA": {"$exists":true}
},
{
"updateDescription.updatedFields.outerField.fieldB":{"$exists":true}
}
]
}
If I remove outerField.<field>, it works. That is:
{
"$or": [
{
"updateDescription.updatedFields": {"$exists":true}
},
{
"updateDescription.updatedFields":{"$exists":true}
}
]
}
But of course that's not useful to me because the trigger will fire on any update at all.
I would provide a demo but I'm not sure how to create a sample that has database triggers configured.
Any help will be appreciated, thanks!
So I was able to get around this problem by changing the query to watch for a field that gets updated at the same time but isn't nested. I think the problem with checking for a nested field is that the ChangeEvent's updateDescription property doesn't contain the actual nested object that has changed; instead it contains the dot-notation representation of the change. So if you look at Update 2 in my post you'll see that updatedFields has this value: {\"someOtherField\":310,\"message.fansNo\":1... instead of {\"someOtherField\":310,\"message\":{\"fansNo\":1.... By using message.fansNo in the $match query, Mongo will look for this object shape: {\"message\":{\"fansNo\":1..., which doesn't match in this case. A "real" solution here could be to escape the . in message.fansNo in my match expression, but I couldn't get that to work (see this thread).
So the "solution" that worked for me is really just a workaround that works for my specific use-case: it so happens that someOtherField is always updated along with message.fansNo, and someOtherField isn't nested. So I can match someOtherField without worrying about nesting. Basically this match expression gives me the results I want:
{
"$or": [
{
"updateDescription.updatedFields.someOtherField": {"$exists":true}
},
{
"updateDescription.updatedFields.someOtherField":{"$exists":true}
}
]
}
Hope this helps someone else!
I am using Mongo 3.2.14
I have a mongo collection that looks like this:
{
'_id':...
'field1':...
'field2':...
'field3':...
etc...
}
I want to aggregate this way:
db.collection.aggregate{
'$match':{},
'$project':{
'field1':1,
'field2':1,
'field3':1,
etc...(all fields)
}
}
Is there a way to include all fields in the project without listing each field one by one ? (I have around 30 fields, and growing...)
I have found info on this here:
MongoDB $project: Retain previous pipeline fields
Include all existing fields and add new fields to document
how to not write every field one by one in project
However, I'm using mongo 3.2.14 and I don't need to create a new field, so, I think I cannot use $addFields. But, if I could, can someone show me how to use it?
Basically, if you want all attributes of your documents to be passed to the next pipeline you can skip the $project pipeline. but if you want all the attributes except the "_id" value then you can pass
{ $project: { _id: 0 } }
which will return every value except the _id.
And if by any chance you have embedded lists or nests that you want to flatten, you can use the $unwind pipeline
you can use $replaceRoot
db.collection.aggregate{
"$match": {},
"$project": {
"document": "$$ROOT"
},
{
"$replaceRoot": { "newRoot": "$document" }
}
this way you can get the exact document returned with all the fields in the document...you don't need to add each field one by one in the $project...try using $replaceRoot at the end of the pipeline.
I have this particular scenario where I have to update certain value in MongoDB depending on different attributes present in same Document. So I am trying to use findAndUpdate with where operator which will be passed a JavaScript function and I will also be using one of the attribute as find criteria. But it has been mentioned in MongoDB documentation that, one should not use where operator until it can not be avoided because of performance issue.
Now lets say I have 3 attributes id, counter1, counter2 in my document and I am updating counter1 by 1 only when counter1 + counter2 = 2. So I will be writing something like
db.mydb.findAndUpdate({"_id" : id, $where : function() {
this.counter1 + this.counter2 == 2 ;}},
{$inc : {counter1 : 1}})
Now my question is:
Will this particular approach create any performance issue? as I am using id as another nonWhere operator criteria to search for a document.
Or I should be having another attribute in mydb collection something called say sumCounter which will store the values of counter1 and counter2.
So the main catch with $where evaluation is that the conditional logic cannot process an "index" in order to filter out matches. In addition, it is JavaScript logic afterall, and needs to be compiled as well as there needs to be "object translation" from the native forms into something that will work with the evaluation in the JavacScript engine.
So it's use should be "very sparingly" and only when "absolutely" required, as in there is no other practical way. In your case this is an "update" operation, therefore if you need that logic then fine. If it where just a "query", then I would say to use $redact in the aggregation framework instead:
db.mydb.aggregate([
{ "$match": { "_id": id } } },
{ "$redact": {
"$cond": {
"if": {
"$eq": [
{ "$add": [ "$counter1", "$counter2" ] },
2
]
},
"then": "$$KEEP",
"else": "$$PRUNE"
}
}}
])
As that is at least all in native operators and therefore going to work faster than JavaScript.
As for "performance", then it is all relative. But however in your case where _id is a "unique" lookup, then the actual performance "hit" should be negligible as the "exact match" was already done on the "index" for the primary key.
This is the general advice for $where conditions. In that you "use them" generally in conjuction with other native query operators that do the "bulk" of the filtering. Then if it takes a few more CPU cycles to apply the conditions in your JavaScript logic ( and it is absolutely needed since there is no other way ), then so be it.
But if however your JavaScript based condition needs to scan many documents without the assistance of other filtering, then that is bad indeed.
Say I have an object with field state, I want to update this field, while keeping the previous value of state in previous_state field. First, I have tried to make an update with unset-rename-set:
collection.update(query, {$unset: {previous_state: ""}, $rename: {state: "previous_state"}, $set: {state: value}})
no surprise it did not work. After reading:
Update MongoDB field using value of another field
MongoDB update: Generate new field based on existing field, or update in place
Update field with another field's value in the document
I am nearly convinced that I do not have a solution to perform this in a single query. So the question is what is the best practice to do it?
There are various ways to do it, depending on the version of MongoDB, and they are described in this answer from another thread: https://stackoverflow.com/a/37280419/5538923 .
For MongoDB 3.4+, for example, there is this query that can be put in MongoSH:
db.collection.aggregate(
[
{ "$addFields": {
"previous_state": { "$concat": [ "$state" ] },
"state": { "$concat": [ "$state", " customly modified" ] }
}},
{ "$out": "collection" }
])
Also note that this query works only when the MongoDB instance is not sharded. When sharded (e.g., often the case in Microsoft Azure CosmosDB), the method described in that answer for MongoDB 3.2+ works, or alternatively put a new collection as destination (in the $out close), and then import the data in the original collection, after removing all the data there.
One solution (if you've got onlty one writer) could be to trigger your update in two steps:
> var previousvalue = collection.findOne(query).state;
> collection.update(query, {$set: {"previous_state": previousvalue, "state": newstatevalue}});