mongodb $addToSet to a non-array field when update on upsert - mongodb

My recent project encountered the same problem as this one: the question
db.test.update(
{name:"abc123", "config.a":1 },
{$addToSet:{ config:{a:1,b:2} } },
true
)
Will produce such error:
Cannot apply $addToSet to a non-array field
But after changed to:
db.test.update(
{name:"abc123", "config.a":{$in:[1]} },
{$addToSet:{ config:{a:1,b:2} } },
true
)
It works fine.
Also referenced this link: Answer
Can Any one explain what's going on? "config.a":1 will turn config to be an object? Where "config.a":{$in:[1]} won't?

What you are trying to do here is add a new item to an array only where the item does not exist and also create a new document where it does not exist. You choose $addToSet because you want the items to be unique, but in fact you really want them to be unique by "a" only.
So $addToset will not do that, and you rather need to "test" the element being present. But the real problem here is that it is not possible to both do that and "upsert" at the same time. The logic cannot work as a new document will be created whenever the array element was not found, rather than append to the array element like you want.
The current operation errors by design as $addToSet cannot be used to "create" an array, but only to "add" members to an existing array. But as stated already, you have other problems with achieving the logic.
What you need here is a sequence of update operations that each "try" to perform their expected action. This can only be done with multiple statements:
// attempt "upsert" where document does not exist
// do not alter the document if this is an update
db.test.update(
{ "name": "abc" },
{ "$setOnInsert": { "config": [{ "a": 1, "b": 2 }] }},
{ "upsert": true }
)
// $push the element where "a": 1 does not exist
db.test.update(
{ "name": "abc", "config.a": { "$ne": 1 } },
{ "$push": { "config": { "a": 1, "b": 2 } }}
)
// $set the element where "a": 1 does exist
db.test.update(
{ "name": "abc", "config.a": 1 },
{ "$set": { "config.$.b": 2 } }
)
On a first iteration the first statement will "upsert" the document and create the array with items. The second statement will not match the document because the "a" element has the value that was specified. The third statement will match the document but it will not alter it in a write operation because the values have not changed.
If you now change the input to "b": 3 you get different responses but the desired result:
db.test.update(
{ "name": "abc" },
{ "$setOnInsert": { "config": [{ "a": 1, "b": 3 }] }},
{ "upsert": true }
)
db.test.update(
{ "name": "abc", "config.a": { "$ne": 1 } },
{ "$push": { "config": { "a": 1, "b": 3 } }}
)
db.test.update(
{ "name": "abc", "config.a": 1 },
{ "$set": { "config.$.b": 3 } }
)
So now the first statement matches a document with "name": "abc" but does not do anything since the only valid operations are on "insert". The second statement does not match because "a" matches the condition. The third statment matches the value of "a" and changes "b" in the matched element to the desired value.
Subsequently changing "a" to another value that does not exist in the array allows both 1 and 3 to do nothing but the second statement adds another member to the array keeping the content unique by their "a" keys.
Also submitting a statement with no changes from existing data will of course result in a response that says nothing is changed on all accounts.
That's how you do your operations. You can do this with "ordered" Bulk operations so that there is only a single request and response from the server with the valid response to modified or created.

As explained in this issue on the MongoDB JIRA (https://jira.mongodb.org/browse/SERVER-3946), this can be solved in a single query:
The following update fails because we use $addToSet on a field which we have also included in the first argument (the field which accepts the fields and values to query for). As far as I understand it, you can't use upsert: true in this scenario where we $addToSet to the same field we query with.
db.foo.update({x : "a"}, {$addToSet: {x: "b"}} , {upsert: true}); // FAILS
The solution is to use $elemMatch: {$eq: field: value}
db.foo.update({x: {$elemMatch: {$eq: "a"}}}, {$addToSet: {x: "b"}}, {upsert: true});

Related

Update an array item of Mongodb with $and query

Hi I am trying to increment the count of the matching requirement in an array. My sample collection looks like the following:
{
"_id": ObjectId("60760ba2e870fa518f2ae48b"),
"userId": "6075f7289822d94dca8066b4",
"requirements": [
{
"searchText": "zee5",
"planType": "basic",
"mode": "PRIVATE",
"count": 32.0
},
{
"searchText": "sony",
"planType": "standard",
"mode": "PUBLIC",
"count": 12.0
},
{
"searchText": "prime",
"planType": "premium",
"mode": "PRIVATE",
"count": 2
}
]
}
If a user searches for prime, with filter premium and PRIVATE, then the count of the last requirement should be updated. If he searches for prime, with filter standard and PRIVATE, then the new requirement will be inserted with count 1.
I am doing in two steps. First I fire an update with the following query and then if no update, I fire a push query with count 1:
db.getCollection('userProfile').update({ "$and" : [{ "requirements.searchText" : {$eq:"prime"}}, {"requirements.mode" : {$eq: "PUBLIC"}}, {"requirements.planType": {$eq: "standard"}}, { "userId" : "6075f7289822d94dca8066b4"}]}, {$inc: {"requirements.$.count" : 1}})
I was expecting that the above query will not update any requirement, since there is no exact match. Interestingly, it increments the count of the second requirement with (sony, standard, public). What is wrong with the query? How can I get it right?
Demo - with Update - https://mongoplayground.net/p/-ISXaAayxxv
Demo No update - https://mongoplayground.net/p/88bTj3lz7U_
Use $elemMatch to make sure all properties are present in the same object inside the array
The $elemMatch operator matches documents that contain an array field with at least one element that matches all the specified query criteria.
db.collection.update(
{
"requirements": {
$elemMatch: { "searchText": "prime","mode": "PUBLIC", "planType": "standard" }
},
"userId": "6075f7289822d94dca8066b4"
},
{ $inc: { "requirements.$.count": 1 } }
)
Problem -
Your current query will match any document with all these fields in
requirements array in any object, if they match 1 property in 1 index of the array and another match in the next index query will find the document valid.
"searchText": "prime",
"mode": "PUBLIC",
"planType": "standard"

Update Array Children Sorted Order

I have a collection containing objects with the following structure
{
"dep_id": "some_id",
"departament": "dep name",
"employees": [{
"name": "emp1",
"age": 31
},{
"name": "emp2",
"age": 35
}]
}
I would like to sort and save the array of employees for the object with id "some_id", by employees.age, descending. The best outcome would be to do this atomically using mongodb's query language. Is this possible?
If not, how can I rearrange the subdocuments without affecting the parent's other data or the data of the subdocuments? In case I have to download the data from the database and save back the sorted array of children, what would happen if something else performs an update to one of the children or children are added or removed in the meantime?
In the end, the data should be persisted to the database like this:
{
"dep_id": "some_id",
"departament": "dep name",
"employees": [{
"name": "emp2",
"age": 35
},{
"name": "emp1",
"age": 31
}]
}
The best way to do this is to actually apply the $sort modifier as you add items to the array. As you say in your comment "My actual objects have a "rank" and 'created_at'", which means that you really should have asked that in your question instead of writing a "contrived" case ( don't know why people do that ).
So for "sorting" by multiple properties, the following reference would adjust like this:
db.collection.update(
{ },
{ "$push": { "employees": { "$each": [], "$sort": { "rank": -1, "created_at": -1 } } } },
{ "multi": true }
)
But to update all the data you presently have "as is shown in the question", then you would sort on "age" with:
db.collection.update(
{ },
{ "$push": { "employees": { "$each": [], "$sort": { "age": -1 } } } },
{ "multi": true }
)
Which oddly uses $push to actually "modify" an array? Yes it's true, since the $each modifier says we are not actually adding anything new yet the $sort modifier is actually going to apply to the array in place and "re-order" it.
Of course this would then explain how "new" updates to the array should be written in order to apply that $sort and ensure that the "largest age" is always "first" in the array:
db.collection.update(
{ "dep_id": "some_id" },
{ "$push": {
"employees": {
"$each": [{ "name": "emp": 3, "age": 32 }],
"$sort": { "age": -1 }
}
}}
)
So what happens here is as you add the new entry to the array on update, the $sort modifier is applied and re-positions the new element between the two existing ones since that is where it would sort to.
This is a common pattern with MongoDB and is typically used in combination with the $slice modifier in order to keep arrays at a "maximum" length as new items are added, yet retain "ordered" results. And quite often "ranking" is the exact usage.
So overall, you can actually "update" your existing data and re-order it with "one simple atomic statement". No looping or collection renaming required. Furthermore, you now have a simple atomic method to "update" the data and maintain that order as you add new array items, or remove them.
In order to get what you want you can use the following query:
db.collection.aggregate({
$unwind: "$employees" // flatten employees array
}, {
$sort: {
"employees.name": -1 // sort all documents by employee name (descending)
}
}, {
$group: { // restore the previous structure
_id: "$_id",
"dep_id": {
$first: "$dep_id"
},
"departament": {
$first: "$departament"
},
"employees": {
$push: "$employees"
},
}
}, {
$out: "output" // write everything out to a separate collection
})
After this step you would want to drop your source table and rename the "output" collection to match your source table name.
This solution will, however, not deal with the concurrency issue. So you should remove write access from the collection first so nobody modifies it during the process and then restore it once you're done with the migration.
You could alternatively query all data first, then sort the employees array on the client side and then use either single update queries or - faster but more complicated - a bulk write operation with all the individual update calls in order to update the existing documents. Here, you could use the entire document that you've initially read as a filter for the update operation. So if an individual update does not modify any document you'd know straight away, that some other change must have modified the document you read before. Those cases you'd need to retry later (or straight away until the update does actually modify a document).

MongoDB: Sort by field existing and then alphabetically

In my database I have a field of name. In some records it is an empty string, in others it has a name in it.
In my query, I'm currently doing:
db.users.find({}).sort({'name': 1})
However, this returns results with an empty name field first, then alphabetically returns results. As expected, doing .sort({'name': -1}) returns results with a name and then results with an empty string, but it's in reverse-alphabetical order.
Is there an elegant way to achieve this type of sorting?
How about:
db.users.find({ "name": { "$exists": true } }).sort({'name': 1})
Because after all when a field you want to sort on is not actually present then the returned value is null and therefor "lower" in the order than any positive result. So it makes sense to exclude those results if you really are only looking for something with a matching value.
If you really want all the results in there and regarless of a null content, then I suggest you "weight" them via .aggregate():
db.users.aggregate([
{ "$project": {
"name": 1,
"score": {
"$cond": [
{ "$ifNull": [ "$name", false ] },
1,
10
]
}
}},
{ "$sort": { "score": 1, "name": 1 } }
])
And that moves all null results to the "end of the chain" by assigning a value as such.
If you want to filter out documents with an empty "name" field, change your query: db.users.find({"name": {"$ne": ""}}).sort({"name": 1})

Update an Element if Position is Unknown with Upsert

It looks like you(/I ) cannot have both upsert and an array element update operation.
If you do (python):
findDct = {
"_id": ObjectId("535e3ab9c36b4417d031402f"),
'events.ids': '176976332'
}
print col.update(findDct, {"$set" : {"events.$.foo": "bar"} }, upsert=True)
It will throw:
pymongo.errors.DuplicateKeyError: insertDocument :: caused by :: 11000 E11000
duplicate key error index: test.col.$_id_ dup key: { : ObjectId('535e3ab9c36b4417d031402f') }
This happens because "_id" is of course an index and mongo tries to insert the document as a new since the find query fails on its 'events.ids': '176976332' part (cheat).
Is it possible to update an unknown element in array with upsert True/how?
Yes it is, but you are going about it in the wrong way. Rather than make "finding" the element that you are not sure whether it exists or not, then try to apply the $addToSet operator instead:
db.collection.update(
{ "_id": ObjectId("535e3ab9c36b4417d031402f" },
{
"$addToSet": { "events": { "foo": "bar" } }
},
{ "upsert": true }
)
Please also note from the positional $ operator documentation that you should not use the $ operator with "upserts" as this will result in the field name being interpreted as a "literal" ( which includes the value as in "events.$.foo" ) and that will be the actual field inserted into the document.
Try to make sure that your array "insert/upsert" operations specify the whole array content in order to make this work.
Another adaptation is with the "bulk" methods, the pymongo driver already has a nice API for this, but this is a general form:
db.runCommand({
"update": "collection",
"updates": [
{
"q": { "_id": ObjectId("535e3ab9c36b4417d031402f" } },
"u": {
"$addToSet": {
"events": {
"foo": "bar", "bar": "baz"
}
}
},
"upsert": true
},
{
"q": { "_id": ObjectId("535e3ab9c36b4417d031402f" } },
"u": {
"$set": { "events.foo": "bar" }
}
}
]
})
But still being very careful that you are not producing duplicates in your sub-document array if you can clearly see the case there. But it is a method, as each update will cascade down even if the first form failed to add anything. Not the best case example, but I hope you see the point.

MongoDB update all fields of array error

Im tring to set 0 the items.qty of a document obtains by a id query.
db.warehouses.update(
// query
{
_id:ObjectId('5322f07e139cdd7e31178b78')
},
// update
{
$set:{"items.$.qty":0}
},
// options
{
"multi" : true, // update only one document
"upsert" : true // insert a new document, if no existing document match the query
}
);
Return:
Cannot apply the positional operator without a corresponding query field containing an array.
This is the document that i want to set all items.qty to 0
{
"_id": { "$oid" : "5322f07e139cdd7e31178b78" },
"items": [
{
"_id": { "$oid" : "531ed4cae604d3d30df8e2ca" },
"brand": "BJFE",
"color": "GDRNCCD",
"hand": 1,
"model": 0,
"price": 500,
"qty": 0,
"type": 0
},
{
"brand": "BJFE",
"color": "GDRNCCD",
"hand": 1,
"id": "23",
"model": 0,
"price": 500,
"qty": 4,
"type": 0
},
{
"brand": "BJFE",
"color": "GDRNCCD",
"hand": 1,
"id": "3344",
"model": 0,
"price": 500,
"qty": 6,
"type": 0
}
],
"name": "a"
}
EDIT
The detail missing from the question was that the required field to update was actually in a sub-document. This changes the answer considerably:
This is a constraint of what you can possibly do with updating array elements. And this is clearly explained in the documentation. Mostly in this paragraph:
The positional $ operator acts as a placeholder for the first element that matches the query document
So here is the thing. Trying to update all of the array elements in a single statement like this will not work. In order to do this you must to the following.
db.warehouses.find({ "items.qty": { "$gt": 0 } }).forEach(function(doc) {
doc.items.forEach(function(item) {
item.qty = 0;
});
db.warehouses.update({ "_id": doc._id }, doc );
})
Which is basically the way to update every array element.
The multi setting in .update() means across multiple "documents". It cannot be applied to multiple elements of an array. So presently the best option is to replace the whole thing. Or in this case we may just as well replace the whole document since we need to do that anyway.
For real bulk data, use db.eval(). But please read the documentation first:
db.eval(function() {
db.warehouses.find({ "items.qty": { "$gt": 0 } }).forEach(function(doc) {
doc.items.forEach(function(item) {
item.qty = 0;
});
db.warehouses.update({ "_id": doc._id }, doc );
});
})
Updating all the elements in an array across the whole collection is not simple.
Original
Pretty much exactly what the error says. In order to use a positional operator you need to match something first. As in:
db.warehouses.update(
// query
{
_id:ObjectId('5322f07e139cdd7e31178b78'),
"items.qty": { "$gt": 0 }
},
// update
{
$set:{"items.$.qty":0}
},
// options
{
"multi" : true,
"upsert" : true
}
);
So where the match condition fins the position of the items that are less than 0 then that index is passed to the positional operator.
P.S : When muti is true it means it updates every document. Leave it false if you only mean one. Which is the default.
You can use the $ positional operator only when you specify an array in the first argument (i.e., the query part used to identify the document you want to update).
The positional $ operator identifies an element in an array field to update without explicitly specifying the position of the element in the array.