update specific element from nested document array mongodb where has two matches - mongodb

I need to update or create if not exist, specific obj,set score.b1 =50 and total=100 where object match curse=5 block=2
{ "_id":"sad445"
"year":2020,
"grade":4,
"seccion":"A",
"id": 100,
"name": "pedro",
"notes":[{"curse":5,
"block":1,
"score":{ "a1": 5,"a2": 10, "a3": 15},
"total" : 50
},{
"curse":5,
"block":2,
"score":{ "b1": 10,"b2": 20, "b3": 30},
"total" : 20
}
]
}
I can update all obj but I need to update or create specific elem from the score and not all. and/or create objs "notes":[{curse, block and score}] if notes is empty notes:[]
notas.UpdateMany(
{"$and":[{"_id":"sad445"},{"notes":{"$elemMatch":{"curse":5,"block":3}}}]},
{"$set":{"updated_at":{"$date":{"$numberLong":"1620322881360"}},
"notes.$.score":{"vvkzo":15,"i2z4i":2,"i2z4i|pm":5},
"notes.$.total":100}},
{"multiple":false})

Demo - https://mongoplayground.net/p/VaE28ujeOPx
Use $ (update)
The positional $ operator identifies an element in an array to update without explicitly specifying the position of the element in the array.
the positional $ operator acts as a placeholder for the first element that matches the query document, and
the array field must appear as part of the query document.
db.collection.update({
"notes": {
"$elemMatch": { "block": 2, "curse": 5 }
}
},
{
$set: { "notes.$.score.b4": 40 }
})
Read upsert: true
Optional. When true, update() either:
Creates a new document if no documents match the query. For more
details see upsert behavior. Updates a single document that matches
the query. If both upsert and multi are true and no documents match
the query, the update operation inserts only a single document.
To avoid multiple upserts, ensure that the query field(s) are uniquely
indexed. See Upsert with Unique Index for an example.
Defaults to false, which does not insert a new document when no match
is found.
Update
Demo - https://mongoplayground.net/p/iQQDyjG2a_B
Use $function
db.collection.update(
{ "_id": "sad445" },
[
{
$set: {
notes: {
$function: {
body: function(notes) {
var record = { curse:5, block:2, score:{ b4:40 } };
if(!notes || !notes.length) { return [record]; } // create new record and return in case of no notes
var updated = false;
for (var i=0; i < notes.length; i++) {
if (notes[i].block == 2 && notes[i].curse == 5) { // check condition for update
updated = true;
notes[i].score.b4=40; break; // update here
}
}
if (!updated) notes.push(record); // if no update push the record in notes array
return notes;
},
args: [
"$notes"
],
lang: "js"
}
}
}
}
]
)

Try to add upsert: true.
Creates a new document if no documents match the query. Updates a single document that matches
the query.
notas.UpdateMany(
{"$and":[{"_id":"sad445"},{"notes":{"$elemMatch":{"curse":5,"block":3}}}]},
{"$set":{"updated_at":{"$date":{"$numberLong":"1620322881360"}},
"notes.$.score":{"vvkzo":15,"i2z4i":2,"i2z4i|pm":5},
"notes.$.total":100}},
{"multiple":false, "upsert":true})

Related

MongoDB - Update One Field or Another

I'm pretty new to MongoDB so this might be my inexperience with it. I'm trying to do an upsert that when a record is found it will update multiple fields based on multiple conditions.
I have the following record in a collection:
{
modelId: "5e68c7eaa0887971ea6ef54c",
versionId: 999,
timeStamp: "1/1/2020",
oldValue: 'Blue',
newValue: 'Red'
}
I'm trying to satisfy the following conditions with a single upsert statement in order to avoid making multiple trips to the DB (based on the query that a document matching the modelId and versionId is found:
If timeStamp of new record is before (lt) the existing document then update oldValue
If timeStamp of new record is after (gt) the existing document then update newValue
If matching records is not found insert the new record.
In psuedo code terms I'm trying to do this with the upsert statement:
existingRecord = item in collection matching modelId and versionId
if(existingRecord = null)
{
//insert newRecord
}
if(newRecord.timeStamp < existingRecord.timeStamp)
{
existingRecord.oldValue = newRecord.oldValue
existingRecord.timeStamp = newRecord.timeStamp
}
else if(newRecord.timeStamp > existingRecord.timeStamp)
{
existingRecord.newValue = newRecord.newValue
existingRecord.timeStamp = newRecord.timeStamp
}
I've seen the possibility to do an upsert based on the condition of a date, something like:
db.collection.update( { id:o.id, date: { $lt:o.date } }, {$set : { o }}, {upsert:true} );
I don't know how to expand that to be able to update either the oldValue or the newValue based on the timeStamp value.
I'm planning on having a good amount of records inserted into the collection every day, estimate around 1MM, I'd hate to have to do a find() and then an update() for each record.
I'm using Mongo 4.0 and would appreciate any advice.
Thanks!
Well, in version 4.0, you are not allowed to use the conditions in the update query. Hence, you end up firing two queries instead.
db.collection.update({condition}, { $set: { o } }, { multi: true ,upsert:true });
db.collection.update({!condition}, { $set: { n } }, { multi: true ,upsert:true });
However, in version 4.2, added db.collection.update pipeline, in which the aggregation is allowed.
And, it contains only the following aggregation stages:
$addFields and its alias $set
$project and its alias $unset
$replaceRoot and its alias $replaceWith.
Hope this will help :)
Update
I have added the $set stage to update the document. It will update the if timestamp condition is true else it will not update. and applies the same for other condition.
I have used the long value of timestamp you can use according to you case.
db.collection.update(
{
modelId: "5e68c7eaa0887971ea6ef54c",
versionId: 999,
},
[
{
$set:{
"oldValue":{
$cond:[
{
$lt:[
"timestamp",
1598598257000
]
},
"green",
"$oldValue"
]
}
}
},
{
$set:{
"newValue":{
$cond:[
{
$gt:[
"timestamp",
1518598257000
]
},
"pink",
"$newValue"
]
}
}
}
]
)

Remove empty field's name from document

I've been able to add a large volume of empty keys in a MongoDB collection. As empty keys is not allowed in MongoDB I'm having a hard time to unset the key using regular methods in MongoDB. Is there a workaround for this kind of problems using mongo shell?
{
"foo": "bar",
"": "I should not exist, I have no key..."
}
We can delete the empty string field using bulk operations.
We need to iterate over the cursor snapshot then use the bulkWrite method method to bulk update the document. Note that we need to replace the document because we can't $unset the empty field or rename it. So an update operation is not possible here.
let requests = [];
db.coll.find( { "": { "$exists": true } } ).snapshot().forEach( document => {
delete document[""];
requests.push( {
"replaceOne": {
"filter": { "_id": document._id },
"replacement": document
}
});
if ( requests.length === 1000 ) {
// Execute per 1000 operations and re-init
db.coll.bulkWrite(requests);
requests = [];
}
});
// Clean up queues
if ( requests.length > 0 ) {
db.coll.bulkWrite(requests);
}
Give this a throw....
db.collection.find().forEach(function(doc) {
delete doc[''];
db.collection.save(doc);
});
There are some abilities to manage documents that have errors in them, such as empty keys.
If you are on version 5.0 or later, you can use $setField expression with expressive pipeline update:
db.c.updateMany({ "": { "$exists": true } },
[ {$replaceWith:{$setField:{field:"",input:"$$ROOT",value:"$$REMOVE"}}}]
)
If you are on 4.2 through 4.4 (pre-5.0) you can do it with different pipeline update:
db.c.updateMany({ "": { "$exists": true } },
[ {$replaceWith:{$arrayToObject:{$filter:{
input:{$objectToArray:"$$ROOT"},
cond:{$ne:["$$this.k",""]}
}}}}]
)

Replace a word from a string

I have mongodb documents with a field like this:
Image : http://static14.com/p/Inc.5-Black-Sandals-5131-2713231-7-zoom.jpg
How can I replace the zoom part in the string value with some other text in order to get:
Image : http://static14.com/p/Inc.5-Black-Sandals-5131-2713231-7-product2.jpg
You could use mongo's forEach() cursor method to do an atomic update with the $set operator :
db.collection.find({}).snapshot().forEach(function(doc) {
var updated_url = doc.Image.replace('zoom', 'product2');
db.collection.update(
{"_id": doc._id},
{ "$set": { "Image": updated_url } }
);
});
Given a very large collection to update, you could speed up things a little bit with bulkWrite and restructure your update operations to be sent in bulk as:
var ops = [];
db.collection.find({}).snapshot().forEach(function(doc) {
ops.push({
"updateOne": {
"filter": { "_id": doc._id },
"update": { "$set": { "Image": doc.Image.replace('zoom', 'product2') } }
}
});
if ( ops.length === 500 ) {
db.collection.bulkWrite(ops);
ops = [];
}
})
if ( ops.length > 0 )
db.collection.bulkWrite(ops);
db.myCollection.update({image: 'http://static14.com/p/Inc.5-Black-Sandals-5131-2713231-7-zoom.jpg'}, {$set: {image : 'http://static14.com/p/Inc.5-Black-Sandals-5131-2713231-7-product2.jpg'}})
If you need to do this multiple times to multiple documents, you need to iterate them with a function. See here: MongoDB: Updating documents using data from the same document
Nowadays,
starting Mongo 4.2, db.collection.updateMany (alias of db.collection.update) can accept an aggregation pipeline, finally allowing the update of a field based on its own value.
starting Mongo 4.4, the new aggregation operator $replaceOne makes it very easy to replace part of a string.
// { "Image" : "http://static14.com/p/Inc.5-Black-Sandals-5131-2713231-7-zoom.jpg" }
// { "Image" : "http://static14.com/p/Inc.5-Black-Sandals-5131-2713231-7-boom.jpg" }
db.collection.updateMany(
{ "Image": { $regex: /zoom/ } },
[{
$set: { "Image": {
$replaceOne: { input: "$Image", find: "zoom", replacement: "product2" }
}}
}]
)
// { "Image" : "http://static14.com/p/Inc.5-Black-Sandals-5131-2713231-7-product2.jpg" }
// { "Image" : "http://static14.com/p/Inc.5-Black-Sandals-5131-2713231-7-boom.jpg" }
The first part ({ "Image": { $regex: /zoom/ } }) is just there to make the query faster by filtering which documents to update (the ones containing "zoom")
The second part ($set: { "Image": {...) is the update aggregation pipeline (note the squared brackets signifying the use of an aggregation pipeline):
$set is a new aggregation operator (Mongo 4.2) which in this case replaces the value of a field.
The new value is computed with the new $replaceOne operator. Note how Image is modified directly based on the its own value ($Image).

MongoDB conditionally $addToSet sub-document in array by specific field

Is there a way to conditionally $addToSet based on a specific key field in a subdocument on an array?
Here's an example of what I mean - given the collection produced by the following sample bootstrap;
cls
db.so.remove();
db.so.insert({
"Name": "fruitBowl",
"pfms" : [
{
"n" : "apples"
}
]
});
n defines a unique document key. I only want one entry with the same n value in the array at any one time. So I want to be able to update the pfms array using n so that I end up with just this;
{
"Name": "fruitBowl",
"pfms" : [
{
"n" : "apples",
"mState": 1111234
}
]
}
Here's where I am at the moment;
db.so.update({
"Name": "fruitBowl",
},{
// not allowed to do this of course
// "$pull": {
// "pfms": { n: "apples" },
// },
"$addToSet": {
"pfms": {
"$each": [
{
"n": "apples",
"mState": 1111234
}
]
}
}
}
)
Unfortunately, this adds another array element;
db.so.find().toArray();
[
{
"Name" : "fruitBowl",
"_id" : ObjectId("53ecfef5baca2b1079b0f97c"),
"pfms" : [
{
"n" : "apples"
},
{
"n" : "apples",
"mState" : 1111234
}
]
}
]
I need to effectively upsert the apples document matching on n as the unique identifier and just set mState whether or not an entry already exists. It's a shame I can't do a $pull and $addToSet in the same document (I tried).
What I really need here is dictionary semantics, but that's not an option right now, nor is breaking out the document - can anyone come up with another way?
FWIW - the existing format is a result of language/driver serialization, I didn't choose it exactly.
further
I've gotten a little further in the case where I know the array element already exists I can do this;
db.so.update({
"Name": "fruitBowl",
"pfms.n": "apples",
},{
$set: {
"pfms.$.mState": 1111234,
},
}
)
But of course that only works;
for a single array element
as long as I know it exists
The first limitation isn't a disaster, but if I can't effectively upsert or combine $addToSet with the previous $set (which of course I can't) then it the only workarounds I can think of for now mean two DB round-trips.
The $addToSet operator of course requires that the "whole" document being "added to the set" is in fact unique, so you cannot change "part" of the document or otherwise consider it to be a "partial match".
You stumbled on to your best approach using $pull to remove any element with the "key" field that would result in "duplicates", but of course you cannot modify the same path in different update operators like that.
So the closest thing you will get is issuing separate operations but also doing that with the "Bulk Operations API" which is introduced with MongoDB 2.6. This allows both to be sent to the server at the same time for the closest thing to a "contiguous" operations list you will get:
var bulk = db.so.initializeOrderedBulkOp();
bulk.find({ "Name": "fruitBowl", "pfms.n": "apples": }).updateOne({
"$pull": { "pfms": { "n": "apples" } }
});
bulk.find({ "Name": "fruitBowl" }).updateOne({
"$push": { "pfms": { "n": "apples", "state": 1111234 } }
})
bulk.execute();
That pretty much is your best approach if it is not possible or practical to move the elements to another collection and rely on "upserts" and $set in order to have the same functionality but on a collection rather than array.
I have faced the exact same scenario. I was inserting and removing likes from a post.
What I did is, using mongoose findOneAndUpdate function (which is similar to update or findAndModify function in mongodb).
The key concept is
Insert when the field is not present
Delete when the field is present
The insert is
findOneAndUpdate({ _id: theId, 'likes.userId': { $ne: theUserId }},
{ $push: { likes: { userId: theUserId, createdAt: new Date() }}},
{ 'new': true }, function(err, post) { // do the needful });
The delete is
findOneAndUpdate({ _id: theId, 'likes.userId': theUserId},
{ $pull: { likes: { userId: theUserId }}},
{ 'new': true }, function(err, post) { // do the needful });
This makes the whole operation atomic and there are no duplicates with respect to the userId field.
I hope this helpes. If you have any query, feel free to ask.
As far as I know MongoDB now (from v 4.2) allows to use aggregation pipelines for updates.
More or less elegant way to make it work (according to the question) looks like the following:
db.runCommand({
update: "your-collection-name",
updates: [
{
q: {},
u: {
$set: {
"pfms.$[elem]": {
"n":"apples",
"mState": NumberInt(1111234)
}
}
},
arrayFilters: [
{
"elem.n": {
$eq: "apples"
}
}
],
multi: true
}
]
})
In my scenario, The data need to be init when not existed, and update the field If existed, and the data will not be deleted. If the datas have these states, you might want to try the following method.
// Mongoose, but mostly same as mongodb
// Update the tag to user, If there existed one.
const user = await UserModel.findOneAndUpdate(
{
user: userId,
'tags.name': tag_name,
},
{
$set: {
'tags.$.description': tag_description,
},
}
)
.lean()
.exec();
// Add a default tag to user
if (user == null) {
await UserModel.findOneAndUpdate(
{
user: userId,
},
{
$push: {
tags: new Tag({
name: tag_name,
description: tag_description,
}),
},
}
);
}
This is the most clean and fast method in the scenario.
As a business analyst , I had the same problem and hopefully I have a solution to this after hours of investigation.
// The customer document:
{
"id" : "1212",
"customerCodes" : [
{
"code" : "I"
},
{
"code" : "YK"
}
]
}
// The problem : I want to insert dateField "01.01.2016" to customer documents where customerCodes subdocument has a document with code "YK" but does not have dateField. The final document must be as follows :
{
"id" : "1212",
"customerCodes" : [
{
"code" : "I"
},
{
"code" : "YK" ,
"dateField" : "01.01.2016"
}
]
}
// The solution : the solution code is in three steps :
// PART 1 - Find the customers with customerCodes "YK" but without dateField
// PART 2 - Find the index of the subdocument with "YK" in customerCodes list.
// PART 3 - Insert the value into the document
// Here is the code
// PART 1
var myCursor = db.customers.find({ customerCodes:{$elemMatch:{code:"YK", dateField:{ $exists:false} }}});
// PART 2
myCursor.forEach(function(customer){
if(customer.customerCodes != null )
{
var size = customer.customerCodes.length;
if( size > 0 )
{
var iFoundTheIndexOfSubDocument= -1;
var index = 0;
customer.customerCodes.forEach( function(clazz)
{
if( clazz.code == "YK" && clazz.changeDate == null )
{
iFoundTheIndexOfSubDocument = index;
}
index++;
})
// PART 3
// What happens here is : If i found the indice of the
// "YK" subdocument, I create "updates" document which
// corresponds to the new data to be inserted`
//
if( iFoundTheIndexOfSubDocument != -1 )
{
var toSet = "customerCodes."+ iFoundTheIndexOfSubDocument +".dateField";
var updates = {};
updates[toSet] = "01.01.2016";
db.customers.update({ "id" : customer.id } , { $set: updates });
// This statement is actually interpreted like this :
// db.customers.update({ "id" : "1212" } ,{ $set: customerCodes.0.dateField : "01.01.2016" });
}
}
}
});
Have a nice day !

Remove all fields that are null

How can I remove all fields that are null from all documents of a given collection?
I have a collection of documents such as:
{
'property1': 'value1',
'property2': 'value2',
...
}
but each document may have a null entry instead of a value entry.
I would like to save disk space by removing all null entries. The existence of the null entries does not contain any information in my case because I know the format of the JSON document a priori.
Starting Mongo 4.2, db.collection.update() can accept an aggregation pipeline, finally allowing the removal of a field based on its value:
// { _id: ObjectId("5d0e8...d2"), property1: "value1", property2: "value2" }
// { _id: ObjectId("5d0e8...d3"), property1: "value1", property2: null, property3: "value3" }
db.collection.update(
{},
[{ $replaceWith: {
$arrayToObject: {
$filter: {
input: { $objectToArray: "$$ROOT" },
as: "item",
cond: { $ne: ["$$item.v", null] }
}
}
}}],
{ multi: true }
)
// { _id: ObjectId("5d0e8...d2"), property1: "value1", property2: "value2" }
// { _id: ObjectId("5d0e8...d3"), property1: "value1", property3: "value3" }
In details:
The first part {} is the match query, filtering which documents to update (in our case all documents).
The second part [{ $replaceWith: { ... }] is the update aggregation pipeline (note the squared brackets signifying the use of an aggregation pipeline):
With $objectToArray, we first transform the document to an array of key/values such as [{ k: "property1", v: "value1" }, { k: "property2", v: null }, ...].
With $filter, we filter this array of key/values by removing items for which v is null.
We then transform back the filtered array of key/values to an object using $arrayToObject.
Finally, we replace the whole document by the modified one with $replaceWith.
Don't forget { multi: true }, otherwise only the first matching document will be updated.
// run in mongo shell
var coll = db.getCollection("collectionName");
var cursor = coll.find();
while (cursor.hasNext()) {
var doc = cursor.next();
var keys = {};
var hasNull = false;
for ( var x in doc) {
if (x != "_id" && doc[x] == null) {
keys[x] = 1;
hasNull = true;
}
}
if (hasNull) {
coll.update({_id: doc._id}, {$unset:keys});
}
}
This is an important question since mongodb cannot index null values (i.e. do not query for nulls or you will be waiting for a long time), so it is best to entirely avoid nulls and set default values using setOnInsert.
Here is a recursive solution to removing nulls:
/**
* RETRIEVES A LIST OF ALL THE KEYS IN A DOCUMENT, WHERE THE VALUE IS 'NULL' OR 'UNDEFINED'
*
* #param doc
* #param keyName
* #param nullKeys
*/
function getNullKeysRecursively(doc, keyName, nullKeys)
{
for (var item_property in doc)
{
// SKIP BASE-CLASS STUFF
if (!doc.hasOwnProperty(item_property))
continue;
// SKIP ID FIELD
if (item_property === "_id")
continue;
// FULL KEY NAME (FOR SUB-DOCUMENTS)
var fullKeyName;
if (keyName)
fullKeyName = keyName + "." + item_property;
else
fullKeyName = item_property;
// DEBUGGING
// print("fullKeyName: " + fullKeyName);
// NULL FIELDS - MODIFY THIS BLOCK TO ADD CONSTRAINTS
if (doc[item_property] === null || doc[item_property] === undefined)
nullKeys[fullKeyName] = 1;
// RECURSE OBJECTS / ARRAYS
else if (doc[item_property] instanceof Object || doc[item_property] instanceof Array)
getNullKeysRecursively(doc[item_property], fullKeyName, nullKeys);
}
}
/**
* REMOVES ALL PROPERTIES WITH A VALUE OF 'NULL' OR 'UNDEFINED'.
* TUNE THE 'LIMIT' VARIABLE TO YOUR MEMORY AVAILABILITY.
* ONLY CLEANS DOCUMENTS THAT REQUIRE CLEANING, FOR EFFICIENCY.
* USES bulkWrite FOR EFFICIENCY.
*
* #param collectionName
*/
function removeNulls(collectionName)
{
var coll = db.getCollection(collectionName);
var lastId = ObjectId("000000000000000000000000");
var LIMIT = 10000;
while (true)
{
// GET THE NEXT PAGE OF DOCUMENTS
var page = coll.find({ _id: { $gt: lastId } }).limit(LIMIT);
if (! page.hasNext())
break;
// BUILD BULK OPERATION
var arrBulkOps = [];
page.forEach(function(item_doc)
{
lastId = item_doc._id;
var nullKeys = {};
getNullKeysRecursively(item_doc, null, nullKeys);
// ONLY UPDATE MODIFIED DOCUMENTS
if (Object.keys(nullKeys).length > 0)
// UNSET INDIVIDUAL FIELDS, RATHER THAN REWRITE THE ENTIRE DOC
arrBulkOps.push(
{ updateOne: {
"filter": { _id: item_doc._id },
"update": { $unset: nullKeys }
} }
);
});
coll.bulkWrite(arrBulkOps, { ordered: false } );
}
}
// GO GO GO
removeNulls('my_collection');
document before:
{
"_id": ObjectId("5a53ed8f6f7c4d95579cb87c"),
"first_name": null,
"last_name": "smith",
"features": {
"first": {
"a": 1,
"b": 2,
"c": null
},
"second": null,
"third" : {},
"fourth" : []
},
"other": [
null,
123,
{
"a": 1,
"b": "hey",
"c": null
}
]
}
document after:
{
"_id" : ObjectId("5a53ed8f6f7c4d95579cb87c"),
"last_name" : "smith",
"features" : {
"first" : {
"a" : 1,
"b" : 2
}
},
"other" : [
null,
123,
{
"a" : 1,
"b" : "hey"
}
]
}
As you can see, it removes null, undefined, empty objects and empty arrays. If you need it to be more/less aggressive, it is a matter of modifying the block "NULL FIELDS - MODIFY THIS BLOCK TO ADD CONSTRAINTS".
edits welcome, especially #stennie
You can use the mongo updateMany functionality, but you must do this by specifying the parameter you are going to update, such as the year parameter:
db.collection.updateMany({year: null}, { $unset : { year : 1 }})
Like this question mentioned (mongodb query without field name):
Unfortunately, MongoDB does not support any method of querying all fields with a particular value.
So, you can either iterate the document (like Wizard's example) or do it in non-mongodb way.
If this is a JSON file, remove all the lines with null in sed might works:
sed '/null/d' ./mydata.json
Update for 2022:
If you delete keys with values Null, [], "", {} from the DB, that won't reduce it's size on disk.
You need to do that before you upload data into the collection.
Tested it myself. I had 6.000.000 documents in collection. Ran script of Xavier Guihot. Before script it was 7.8GB, after the script it became 7.9GB.
I confirm, that script do the job and remove keys, it's just that it doesn't reduce the size of the DB space allocation.
Then I deleted completely the collection, and imported .json dumps, that had already been formatted (removed all keys with values Null, [], "", {}). After collection size was 6.1GB That's minus 22% of the original size.
Here is python script I used to remove all empty keys from json dumps:
import fileinput
import json
for line in fileinput.input(inplace=1):
j = {k:v for k, v in json.loads(line).items() if v}
print(line.replace(line, json.dumps(j)))
Just run the script with file name as argument, for example: python3 main.py dump-00001
ps: take into account, that you need to wait ~200 seconds after changes to DB, because WiredTiger keep history backup of data for consistency after you make changes. That's mean, that only after 200 sec you will see real storage allocation of DB. 200 sec is default value for that action.