This is perhaps one of the most common and basic database operations you can perform. However, how you perform this operation in MongoDb, is elusive.
There's a UpdateManyAsync method in the C# driver. But all I can find for it is examples like;
The following operation updates all documents where violations are greater than 4 and $set a flag for review:
try {
db.restaurant.updateMany(
{ violations: { $gt: 4 } },
{ $set: { "Review" : true } }
);
} catch (e) {
print(e);
}
Frankly I don't know how you guys build your applications but our organization will never embed domain logic like how many violations constitutes a review in our database querying logic. Like all the examples I can find for this operation do.
It seems at best that our option is to use some sort of bulk api for this, which seems unnecessary for such a simple operation. Making the bulk api perform a thousand separate replaces seems ineffective to me, but I might be wrong here?
In other storage technologies and more mature API's all the code necessary to perform this operation is (example Entity framework);
db.UpdateRange(stocks);
await db.SaveChangesAsync();
Is this tracked as a feature request for the Mongo project somewhere?
Bulk API can handle it see code below:
var updates = new List<WriteModel<CosmosDoc>>();
foreach (var doc in docs)
{
updates.Add(new ReplaceOneModel<CosmosDoc>(doc.Id, doc));
}
await MongoDb.DocsCollection.BulkWriteAsync(updates, new BulkWriteOptions() { IsOrdered = false });
Related
I'd like to view the results of a bulk operation, specifically to know the IDs of the documents that were updated. I understand that this information is made available through the Bulk.getOperations() method. However, it doesn't appear that this method is available through the MongoDB NodeJS library (at least, the one I'm using).
Could you please let me know if there's something I'm doing wrong here:
const bulk = db.collection('companies').initializeOrderedBulkOp()
const results = getLatestFinanicialResults() // from remote API
results.forEach(result =>
bulk.find({
name: result.companyName,
report: { $ne: result.report }
}).updateOne([
{ $unset: 'prevReport' },
{ $set: { prevReport: '$report' } },
{ $unset: 'report' },
{ $set: { report: result.report } }
]))
await bulk.execute()
await bulk.getOperations() // <-- fails, undefined in Typescript library
I get a static IDE error:
Uncaught TypeError: bulk.getOperations is not a function
I'd like to view the results of a bulk operation, specifically to know the IDs of the documents that were updated
As of currently (MongoDB server v6.x) There is no methods to return IDs for updated documents from a bulk operations (only insert and upsert operations). However, there may be a work around depending on your use case.
The manual that you linked for Bulk.getOperations() is for mongosh, which is a MongoDB Shell application. If you look into the source code for getOperations() in mongosh, it's just a convenient wrapper for batches'. The method batches` returns a list of operations sent for the bulk execution.
As you are utilising ordered bulk operations, MongoDB executes the operations serially. If an error occurs during the processing of one of the write operations, MongoDB will return without processing any remaining write operations in the list.
Depending on the use case, if you modify the bulk.find() part to contain a search for _id for example:
bulk.find({"_id": result._id}).updateOne({$set:{prevReport:"$report"}});
You should be able to see the _id value of the operation in the batches, i.e.
await batch.execute();
console.log(JSON.stringify(batch.batches));
Example output:
{
"originalZeroIndex":0,
"currentIndex":0,
"originalIndexes":[0],
"batchType":2,
"operations":[{"q":{"_id":"634354787d080d3a1e3da51f"},
"u":{"$set":{"prevReport":"$report"}}],
"size":0,
"sizeBytes":0
}
For additional information, you could also retrieve the BulkWriteResult. For example, the getLastOp to retrieve the last operation (in case of a failure)
I have an app that allows to make donations.
I want to listen to insertion of donation records and save the total amount donated to some projection document.
Currently I do it this:
Donation.watch(
[
{
$match: {
operationType: 'insert',
},
},
]
).on('change', async data => {
const projection = await Projection.findById(someId)
projection.total = projection.total + data.fullDoucment.amount
Projection.save();
});
It kinda works, but there's an obvious problem: if two donations come almost at the same time, then the projection would still refer to the outdated version of the projection, so only one of donations would effectively be accounted for.
Is it possible to somehow process the change stream synchronously, waiting for the processing of one record to finish before starting with the next one?
Do I need to hand-roll some sort of synchronous processing queue for that? Any tips which direction to take here?
edit: apparently it's possible to use $inc to build up a sum projection. But what if I needed to do something more complicated e.g. projection.total = await someAsyncCall(projection.total, data.fullDoucment.amount)?
For the initial question $inc works like a charm:
await Projection.findOneAndUpdate(
{
_id: someId,
},
{
$inc: { total: data.fullDoucment.amount},
}
);
For more complicated processing I'd need a serial queue or relying on mongodb versioning constraints.
In my app the users can subscribe to an event. There is a max number of participants allowed on each event. On booking an event i add the id of the user to an array within the event document. When the length of the array equals the max number of participants the user should instead get added to a second array with stand by participants.
My concern is that if there are many concurrent bookings the event could get overbooked if the write of one booking happens after another booking has read the actual size of the array and then itself adds the user to the event.
As i'm a newbee im unsure on how to implement this with mongodb. The best thing i guess would be if i could lock the document when the reading starts and unlock it when the writing has finished.
I searched for locking a document on mongodb but found no examples. Perhaps i used the wrong terms.
I also found findAndModify but no hint if this method would solve my problem.
...
const module = await Modules.findOne({ _id: args.moduleId });
if (!module.participantsIds ||
module.participantsIds.length < module.maxParticipants) {
updateAction = {
$addToSet: { "participantsIds": userId }
};
} else {
updateAction = {
$addToSet: {
"standByParticipants": {
"participantId": userId, "timeStamp": new Date()
}
}
};
}
await Modules.update(
//query
{
_id: args.moduleId
},
//update
updateAction,
//options
{
"multi": false,
"upsert": false
}
);
...
It would be great if someone could lead me in the right direction or give a hint on how to go about this problem.
Update:
the answer of krishna Prasad below clarified to me the use of findAndModify and seems to solve my problem.
Just out of curiosity i'd appreciate if someone could give an example with a lock on the document, if this is even possible.
Best
Markus
As per the official documentation:
When modifying a single document, both findAndModify and the update() method atomically update the document. See Atomicity and Transactions for more details about interactions and order of operations of these methods.
Link: Just before below link: https://docs.mongodb.com/manual/reference/command/findAndModify/#transactions
And also have a look at the stackoverflow ans Is mongoDB's findAndModify "transaction-save"
MongoError: $sample stage could not find a non-duplicate document after 100 while using a random cursor. This is likely a sporadic failure, please try again.
Using MongoDB version 3.2.8 and just upgraded to 3.2.10.
I'm following the example set forth by the MongoDB documentation.. There is very little documentation. The only thing I can find doing a search is a bug in an issue queue which doesn't solve the problem for me.
My implementation is very simple.
function createBattle(done) {
// Need to make sure that the yachts are different.
async.doWhilst(
function(callback) {
Yacht.aggregate(
{
$sample: { size: 2 }
},
function (err, results) {
console.log(err.toString());
if (err) return callback(err);
callback(null, results);
}
)
},
function(results) {
return results.length !== 2 || _.isEqual(results[0]._id, results[1]._id);
},
function(err, results) {
if (err) return done(err);
done(null, results);
}
)
}
I'm not sure what is going on. Any ideas?
I am using Mongoose. It is in development. My guess is that something occurred when I changed the schema of the document. It works with the test mock data after creating 100000 fake documents or just 10 documents. So I decided to try a fresh start on the collection on the development instance of the database. Regardless, I exported the collection into a clean JSON file. I deleted the collection in the database. Then I imported the JSON dump. It works fine.
It's a bug.
This is probably the last straw with MongoDB for me. I'll probably go back to SQL, use Redis, and try PostGRES.
I'm using MongoDB and Mongoose with Express to store tweets that I retrieve via the Twitter API.
I want to avoid saving duplicate tweets. I am doing something like that :
TweetsModel.find({tweet_id: tweet.tweet_id}, function (err, tweets) {
if(tweets.length > 0){
cb('Tweet already exists',null);
} else{
tweet.save(function(err){
cb(err,user);
});
}
});
My question is : for performance reason, is there a way using Mongoose to avoid doing two requests ? One find and one save ?
Knowing that I don't want to update the tweet if it already exists either.
Thank you
You can use an update call with the upsert option to do this:
TweetsModel.update(
{tweet_id: tweet.tweet_id},
{$setOnInsert: tweet},
{upsert: true},
function(err, numAffected) { .. }
);
If a doc already exists with that tweet id, then this is a no-op. Otherwise it will add the doc.
$setOnInsert requires v2.4+ of MongoDB. If your version is less than 2.4, things get more complicated.