I'm pretty new to MongoDB so this might be my inexperience with it. I'm trying to do an upsert that when a record is found it will update multiple fields based on multiple conditions.
I have the following record in a collection:
{
modelId: "5e68c7eaa0887971ea6ef54c",
versionId: 999,
timeStamp: "1/1/2020",
oldValue: 'Blue',
newValue: 'Red'
}
I'm trying to satisfy the following conditions with a single upsert statement in order to avoid making multiple trips to the DB (based on the query that a document matching the modelId and versionId is found:
If timeStamp of new record is before (lt) the existing document then update oldValue
If timeStamp of new record is after (gt) the existing document then update newValue
If matching records is not found insert the new record.
In psuedo code terms I'm trying to do this with the upsert statement:
existingRecord = item in collection matching modelId and versionId
if(existingRecord = null)
{
//insert newRecord
}
if(newRecord.timeStamp < existingRecord.timeStamp)
{
existingRecord.oldValue = newRecord.oldValue
existingRecord.timeStamp = newRecord.timeStamp
}
else if(newRecord.timeStamp > existingRecord.timeStamp)
{
existingRecord.newValue = newRecord.newValue
existingRecord.timeStamp = newRecord.timeStamp
}
I've seen the possibility to do an upsert based on the condition of a date, something like:
db.collection.update( { id:o.id, date: { $lt:o.date } }, {$set : { o }}, {upsert:true} );
I don't know how to expand that to be able to update either the oldValue or the newValue based on the timeStamp value.
I'm planning on having a good amount of records inserted into the collection every day, estimate around 1MM, I'd hate to have to do a find() and then an update() for each record.
I'm using Mongo 4.0 and would appreciate any advice.
Thanks!
Well, in version 4.0, you are not allowed to use the conditions in the update query. Hence, you end up firing two queries instead.
db.collection.update({condition}, { $set: { o } }, { multi: true ,upsert:true });
db.collection.update({!condition}, { $set: { n } }, { multi: true ,upsert:true });
However, in version 4.2, added db.collection.update pipeline, in which the aggregation is allowed.
And, it contains only the following aggregation stages:
$addFields and its alias $set
$project and its alias $unset
$replaceRoot and its alias $replaceWith.
Hope this will help :)
Update
I have added the $set stage to update the document. It will update the if timestamp condition is true else it will not update. and applies the same for other condition.
I have used the long value of timestamp you can use according to you case.
db.collection.update(
{
modelId: "5e68c7eaa0887971ea6ef54c",
versionId: 999,
},
[
{
$set:{
"oldValue":{
$cond:[
{
$lt:[
"timestamp",
1598598257000
]
},
"green",
"$oldValue"
]
}
}
},
{
$set:{
"newValue":{
$cond:[
{
$gt:[
"timestamp",
1518598257000
]
},
"pink",
"$newValue"
]
}
}
}
]
)
Related
I need to update or create if not exist, specific obj,set score.b1 =50 and total=100 where object match curse=5 block=2
{ "_id":"sad445"
"year":2020,
"grade":4,
"seccion":"A",
"id": 100,
"name": "pedro",
"notes":[{"curse":5,
"block":1,
"score":{ "a1": 5,"a2": 10, "a3": 15},
"total" : 50
},{
"curse":5,
"block":2,
"score":{ "b1": 10,"b2": 20, "b3": 30},
"total" : 20
}
]
}
I can update all obj but I need to update or create specific elem from the score and not all. and/or create objs "notes":[{curse, block and score}] if notes is empty notes:[]
notas.UpdateMany(
{"$and":[{"_id":"sad445"},{"notes":{"$elemMatch":{"curse":5,"block":3}}}]},
{"$set":{"updated_at":{"$date":{"$numberLong":"1620322881360"}},
"notes.$.score":{"vvkzo":15,"i2z4i":2,"i2z4i|pm":5},
"notes.$.total":100}},
{"multiple":false})
Demo - https://mongoplayground.net/p/VaE28ujeOPx
Use $ (update)
The positional $ operator identifies an element in an array to update without explicitly specifying the position of the element in the array.
the positional $ operator acts as a placeholder for the first element that matches the query document, and
the array field must appear as part of the query document.
db.collection.update({
"notes": {
"$elemMatch": { "block": 2, "curse": 5 }
}
},
{
$set: { "notes.$.score.b4": 40 }
})
Read upsert: true
Optional. When true, update() either:
Creates a new document if no documents match the query. For more
details see upsert behavior. Updates a single document that matches
the query. If both upsert and multi are true and no documents match
the query, the update operation inserts only a single document.
To avoid multiple upserts, ensure that the query field(s) are uniquely
indexed. See Upsert with Unique Index for an example.
Defaults to false, which does not insert a new document when no match
is found.
Update
Demo - https://mongoplayground.net/p/iQQDyjG2a_B
Use $function
db.collection.update(
{ "_id": "sad445" },
[
{
$set: {
notes: {
$function: {
body: function(notes) {
var record = { curse:5, block:2, score:{ b4:40 } };
if(!notes || !notes.length) { return [record]; } // create new record and return in case of no notes
var updated = false;
for (var i=0; i < notes.length; i++) {
if (notes[i].block == 2 && notes[i].curse == 5) { // check condition for update
updated = true;
notes[i].score.b4=40; break; // update here
}
}
if (!updated) notes.push(record); // if no update push the record in notes array
return notes;
},
args: [
"$notes"
],
lang: "js"
}
}
}
}
]
)
Try to add upsert: true.
Creates a new document if no documents match the query. Updates a single document that matches
the query.
notas.UpdateMany(
{"$and":[{"_id":"sad445"},{"notes":{"$elemMatch":{"curse":5,"block":3}}}]},
{"$set":{"updated_at":{"$date":{"$numberLong":"1620322881360"}},
"notes.$.score":{"vvkzo":15,"i2z4i":2,"i2z4i|pm":5},
"notes.$.total":100}},
{"multiple":false, "upsert":true})
I have a below aggregate query and I wanted to insert/update this result into another collection.
db.coll1.aggregate([
{
$group:
{
_id: "$collId",
name: {$first:"$name"} ,
type: {$first:"$Type"} ,
startDate: {$first:"$startDate"} ,
endDate: {$first:"$endDate"}
}
}
])
I have another collection coll2, which has fields in the above query and some additional fields too.
I may already have the document created in Coll2 matching the _id:collId in the above query. If this Id matches, I want to update the document with the field values of the result and keep the values of other fields in the document.
If the Id does not exists, it should just create a new document in Coll2.
Is there a way to do it in the MongoDB query. I want to implement this in my Spring application.
We can use $merge to do that. It's supported from Mongo v4.2
db.collection.aggregate([
{
$group:{
"_id":"$collId",
"name": {
$first:"$name"
},
"type":{
$first:"$Type"
},
"startDate":{
$first:"$startDate"
},
"endDate":{
$first:"$endDate"
}
}
},
{
$merge:{
"into":"collection2",
"on":"_id",
"whenMatched":"replace",
"whenNotMatched":"insert"
}
}
])
This error happens when I tried to update upsert item:
Updating the path 'x' would create a conflict at 'x'
Field should appear either in $set, or in $setOnInsert. Not in both.
I had the same problem while performing an update query using PyMongo.
I was trying to do:
> db.people.update( {'name':'lmn'}, { $inc : { 'key1' : 2 }, $set: { 'key1' : 5 }})
Notice that here I'm trying to update the value of key1 from two MongoDB Update Operators.
This basically happens when you try to update the value of a same key with more than one MongoDB Update Operators within the same query.
You can find a list of Update Operators over here
If you pass the same key in $set and in $unset when updating an item, you will get that error.
For example:
const body = {
_id: '47b82d36f33ad21b90'
name: 'John',
lastName: 'Smith'
}
MyModel.findByIdAndUpdate(body._id, { $set: body, $unset: {name: 1}})
// Updating the path 'name' would create a conflict at 'name'
You cannot have the same path referenced more than once in an update. For example, even though the below would result in something logical, MongoDB will not allow it.
db.getCollection("user").updateOne(
{_id: ...},
{$set: {'address': {state: 'CA'}, 'address.city' : 'San Diego'}}
)
You would get the following error:
Updating the path 'address.city' would create a conflict at 'address'
db.products.update(
{ _id: 1 },
{
$set: { item: "apple" },
$setOnInsert: { defaultQty: 100 }
},
{ upsert: true }
)
Below is the key explanation to the issue:
MongoDB creates a new document with _id equal to 1 from the
condition, and then applies the $set AND $setOnInsert operations to
this document.
If you want a field value is set or updated regardless of insertion or update, use it in $set. If you want it to be set only on insertion, use it in $setOnInsert.
Here is the example: https://docs.mongodb.com/manual/reference/operator/update/setOnInsert/#example
Starting from MongoDB 4.2 you can use aggregate pipelines in update:
db.your_collection.update({
_id: 1
},
[{
$set:{
x_field: {
$cond: {
if: {$eq:[{$type:"$_id"} , "missing"]},
then: 'upsert value', // it's the upsert case
else: '$x_field' // it's the update case
}
}
}
}],
{
upsert: true
})
db.collection.bulkWrite() also supports it
With the Ruby library at least, it's possible to get this error if you have the same key twice, once as a symbol and once as a string:
db.getCollection("user").updateOne(
{_id: ...},
{$set: {'name': "Horse", name: "Horse"}}
)
I recently had the same issue while using the query below.
TextContainer.findOneAndUpdate({ blockId: req.params.blockId, 'content._id': req.params.noteId }, { $set: { 'content.note': req.body.note } }, { upsert: true, new: true })
When i have changed 'content.note' to 'content.$.note' it has been fixed. So my final query is :
TextContainer.findOneAndUpdate({ blockId: req.params.blockId, 'content._id': req.params.noteId }, { $set: { 'content.$.note': req.body.note } }, { upsert: true, new: true })
In MongoDB, I want to change the structure of my documents from:
{
discount: 10,
discountType: "AMOUNT"
}
to:
{
discount: {
value: 10,
type: "AMOUNT"
}
}
so I tried following query in mongo shell:
db.discounts.update({},
{
$rename: {
discount: "discount.value",
discountType: "discount.type"
}
},
{multi: true}
)
but it throws an error:
"writeError" : {
"code" : 2,
"errmsg" : "The source and target field for $rename must not be on the same path: discount: \"discount.value\""
}
A workaround that comes to my mind is to do it in 2 steps: first assign the new structure to a new field (let's say discount2) and then rename it to discount. But maybe there is a way to do it one step?
The simplest way is to do it in two steps as you allude to in your question; initially renaming discount to a temporary field name so that it can be reused in the second step:
db.discounts.update({}, {$rename: {discount: 'temp'}}, {multi: true})
db.discounts.update({},
{$rename: {temp: 'discount.value', discountType: 'discount.type'}},
{multi: true})
The reason you are getting this error is because as mentioned in the documentation:
The $rename operator logically performs an $unset of both the old name and the new name, and then performs a $set operation with the new name. As such, the operation may not preserve the order of the fields in the document; i.e. the renamed field may move within the document.
And the problem with this is that you can't $set and $unset same field at the same time in MongoDB.
The solution will be to use bulk operations to update your documents in order to change their structure, and even in that case you need to use a field's name that doesn't exist in your collection. Of course the best way to do all this is using "Bulk" operations for maximum efficiency
MongoDB 3.2 or newer
MongoDB 3.2 deprecates Bulk() and its associated methods. You need to use the .bulkWrite() method.
var operations = [];
db.discounts.find().forEach(function(doc) {
var discount = doc.discount;
var discountType = doc.discountType;
var operation = { 'updateOne': {
'filter': { '_id': doc._id },
'update': {
'$unset': { 'discount': '', 'discountType': '' },
'$set': { 'discounts.value': discount, 'discounts.type': discountType }
}
}};
operations.push(operation);
});
operations.push( {
ordered: true,
writeConcern: { w: "majority", wtimeout: 5000 }
});
db.discounts.bulkWrite(operations);
Which yields:
{
"_id" : ObjectId("56682a02e6a2321d88f6d078"),
"discounts" : {
"value" : 10,
"type" : "AMOUNT"
}
}
MongoDB 2.6
Prior to MongoDB 3.2 and using MongoDB version 2.6 or newer you can use the "Bulk" API.
var bulk = db.discounts.initializeOrderedBulkOp();
var count = 0;
db.discounts.find().forEach(function(doc) {
var discount = doc.discount;
var discountType = doc.discountType;
bulk.find( { '_id': doc._id } ).updateOne( {
'$unset': { 'discount': '', 'discountType': '' },
'$set': { 'discounts.value': discount, 'discounts.type': discountType } });
count++;
if (count % 500 === 0) {
bulk.execute();
bulk = db.discounts.initializeOrderedBulkOp();
}
})
if (count > 0)
bulk.execute();
This query yields same result as previous one.
Thanks to answers from Update MongoDB field using value of another field I figured out following solution:
db.discounts.find().snapshot().forEach(
function(elem) {
elem.discount = {
value: elem.discount,
type: elem.discountType
}
delete elem.discountType;
db.discounts.save(elem);
}
)
Which I quite like because the source code reads nicely but performance sucks for large amount of documents.
How to implement somethings similar to db.collection.find().limit(10) but while updating documents?
Now I'm using something really crappy like getting documents with db.collection.find().limit() and then updating them.
In general I wanna to return given number of records and change one field in each of them.
Thanks.
You can use:
db.collection.find().limit(NUMBER_OF_ITEMS_YOU_WANT_TO_UPDATE).forEach(
function (e) {
e.fieldToChange = "blah";
....
db.collection.save(e);
}
);
(Credits for forEach code: MongoDB: Updating documents using data from the same document)
What this will do is only change the number of entries you specify. So if you want to add a field called "newField" with value 1 to only half of your entries inside "collection", for example, you can put in
db.collection.find().limit(db.collection.count() / 2).forEach(
function (e) {
e.newField = 1;
db.collection.save(e);
}
);
If you then want to make the other half also have "newField" but with value 2, you can do an update with the condition that newField doesn't exist:
db.collection.update( { newField : { $exists : false } }, { $set : { newField : 2 } }, {multi : true} );
Using forEach to individually update each document is slow. You can update the documents in bulk using
ids = db.collection.find(<condition>).limit(<limit>).map(
function(doc) {
return doc._id;
}
);
db.collection.updateMany({_id: {$in: ids}}, <update>})
The solutions that iterate over all objects then update them individually are very slow.
Retrieving them all then updating simultaneously using $in is more efficient.
ids = People.where(firstname: 'Pablo').limit(10000).only(:_id).to_a.map(&:id)
People.in(_id: ids).update_all(lastname: 'Cantero')
The query is written using Mongoid, but can be easily rewritten in Mongo Shell as well.
Unfortunately the workaround you have is the only way to do it AFAIK. There is a boolean flag multi which will either update all the matches (when true) or update the 1st match (when false).
As the answer states there is still no way to limit the number of documents to update (or delete) to a value > 1. A workaround to use something like:
db.collection.find(<condition>).limit(<limit>).forEach(function(doc){db.collection.update({_id:doc._id},{<your update>})})
If your id is a sequence number and not an ObjectId you can do this in a for loop:
let batchSize= 10;
for (let i = 0; i <= 1000000; i += batchSize) {
db.collection.update({$and :[{"_id": {$lte: i+batchSize}}, {"_id": {$gt: i}}]}),{<your update>})
}
let fetchStandby = await db.model.distinct("key",{});
fetchStandby = fetchStandby.slice(0, no_of_docs_to_be_updated)
let fetch = await db.model.updateMany({
key: { $in: fetchStandby }
}, {
$set:{"qc.status": "pending"}
})
I also recently wanted something like this. I think querying for a long list of _id just to update in an $in is perhaps slow too, so I tried to use an aggregation+merge
while (true) {
const record = db.records.findOne({ isArchived: false }, {_id: 1})
if (!record) {
print("No more records")
break
}
db.records.aggregate([
{ $match: { isArchived: false } },
{ $limit: 100 },
{
$project: {
_id: 1,
isArchived: {
$literal: true
},
updatedAt: {
$literal: new Date()
}
}
},
{
$merge: {
into: "records",
on: "_id",
whenMatched: "merge"
}
}
])
print("Done update")
}
But feel free to comment if this is better or worse that a bulk update with $in.