I have a requirement that when a date attribute field is passed, that we would like to trigger two things:
to move the record to be deleted to another table.
to call a function to do other actions.
I understand TTL is only to delete a record when the date field is tripped. Can I hook extra logic to it?
Thanks!
Depending on the requirements there could be quite a few ways to do this.
One way is to execute a script periodically, and run a query to filter documents that have passed certain date value. For each of the documents, perform a document migration to another table and extra actions.
Alternatively is to use MongoDB Change Streams. The trick however, is that delete events from change stream do not return the document itself (because it's already been deleted).
Instead if you were to update a field for documents that have passed certain date value you could listen for the update events. For example, sets a field value to expired:true.
Worth mentioning that if you're going down the route of change streams update events, you could utilise MongoDB Stitch Triggers (relying on change streams). MongoDB Stitch database triggers allow you to automatically execute Stitch functions in response to changes in your MongoDB database.
I suggest write a function and call it via scheduler. That will be the better option to do it.
Related
I have an application that runs multiple instances and that needs to perform this:
Read the document
Check a value of the timestamp field
If an incoming document is newer, then update (including the timestamp field).
Even if I use transactions, two parallel operations would be able to perform 1 and 2 simultaneously and then both will write to the database, potentially writing the "older" document last (since it will check the timestamp of the original document, rather than a "new" one)
So what I am looking for is some kind of a read lock on the document or some other mechanism that will be able to solve this.
We have one field which gets updated on user action, admin action, and in cron at the same time then what should we do in order to handle this kind of scenario in MongoDB.
e.g There is a "balance" field in the user's collection when the cron is running user's balance is decreased, now at the same time if the user is recharging and admin is refunding then the balance is not getting updated.
So please suggest any solution for this problem.
If possible, use update operations. They are atomic at the document level, so this should not be a problem.
If you are using a recent version of mongodb, you can use transactions for read-update-writes.
If you cannot do any of these, you can emulate an optimistic locking scheme using versioning to prevent unintended overwrites. There are several ways this can be done, but it generally goes like this:
Read the document. Document has a version field (which can be an integer, or a unique ObjectId. Don't use timestamp)
Make modifications in memory and update the version (increment the integer, or generate a new ObjectId)
Update the document with query containing (version: oldVersion)
This will fail if someone updated the document after you read it but before you updated it. If it fails, retry.
I am using MongoDB in my web API. MongoDB is being updated/inserted by other sources.
How do I query mongodb to get only newly inserted or updated documents ?
I can get sorted documents by below query but this doesn't solve my purpose
db.collectionName.findOne({}, {sort:{$natural:-1}})
Is there any log or other way like in SQL there is INSERTED and UPDATED
What are these newly inserted/updated documents in your context?
Assuming that you need to fetch newly/inserted documents relative to a time, you need to have a field in your collection that holds the time stamp (for example, inserted, and lastUpdated fields), you have an operator to help with updating. But this needs application changes.
Or you can use change streams, for a trigger like functionality. You can listen for changes and take actions as changes are made.
I heared, that mongo can do it, but I can't find how.
Can mongo create collections, which will be autoremove in future, from time, which i can setup? Or Mongo can't do this magic?
mongodb cannot auto remove collections but it can auto remove BSON records. You just need to set ttl(Time to live) index on a date field that exists in BSON record .
You can read more here MongoDb: Expire Data from Collections by Setting TTL
Collections are auto created on the first write operation (insert, upsert, index creation). So this magic is covered.
If your removal is based on time, you could use cron or at to run this little script
mongo yourDBserver/yourDB --eval 'db.yourCollection.drop()’
As Sammaye pointed out, creating indices is a costly operation. I would assume there is something wrong with your data model. For semantically distinguishing documents, I'd rather create a field on them which does that and set an expiration date or a creation date and a time frame in which the documents are valid and use TTL indices to remove all of those documents.
For using an expiration date, you have to set a field to an ISODate and create a TTL index without a duration:
db.yourColl.ensureIndex({"yourExpirationDateField":1},{expireAfterSeconds:0})
In the case you want the documents to be valid for let's say a week after they are created, you would use the following:
db.yourColl.ensureIndex({"yourCreationDate":1},{expireAfterSeconds:604800})
Either way, here is what happens: Once every minute a background thread called TTLMonitor wakes up, gets all TTL indices for the server and starts processing them. It scans the TTL index, looking for the date values, adds the value given for "expireAfterSeconds" and deletes all documents which it determined to be invalid by now. This process takes some time, so don't expect the documents to be deleted on the very second they expire.
The big advantage of that approach: you don't need any triggering logic to be maintained, the deletes are done automagically in the background and you don't put any load on your application. Plus, using an expiration date, your have very granular control over when a document expires.
The drawback is ... ... Well, if I want to find one it would be that you have to insert a creation date for every document or calculate and insert an expiration date. And you have to send an administrative command to the mongod/mongos once in the application lifetime...
I have a collection with documents which contains array fields with even triggers and when there are no more triggers left i want to remove this document. As i understand mongo doesn't have triggers support. Is there any way i can delegate this job to Mongo?
You are correct, there is no triggers in mongo. So there is no normal way to do this with mogno. You have to use application logic to achieve this. One way would be to do cleaning every n minutes. Where you remove documents which have array of size zero. Another way (which I like a more) is after each update to the document, remove it if it has empty array.
The only feature I know MongoDB provides to expire data is by using an expiration index.
Expire data