I have a collections of documents in mongodb, with the expireAfterSeconds property set on a date-type index.
For the sake of argument, the documents are set to expire after one hour.
When I update a document in this collection, which one of the following will happen?
a) The document will expire one hour after the original creation time.
b) The document will expire one hour after the update time.
c) The document will expire one hour after the indexed variable's time, whatever that may be.
d) None of the above
I think that it's c, but cannot find the reference to confirm it. Am I correct? Where is this documented?
[edit]: To clarify, the situation is that I'm storing password reset codes (that should expire.) and I want the old codes to stop working if a new code is requested. It's not very relevant though, since I can ensure the behaviour I want is always respected by simply deleting the old transaction. This question is not about my current problem, but about Mongo's behaviour.
The correct answer is c)
The expireAfterSeconds property always requires an index on a field which contains a BSON date, because the content of this date field is used to select entries for removal.
When you want an update of a document to reset the time-to-live, also update the indexed date field to the current time.
When you want an update to not affect the TTL, just don't update the date.
However, keep in mind that expireAfterSeconds doesn't guarantee immediate deletion of the document. The deletions are done by a background job which runs every minute. This job is low-priority and can be postponed by MongoDB when the current load is high. So when it's important for your use-case that the expire times are respected accurately to the second, you should add an additional check on the application level.
This feature is documented here: http://docs.mongodb.org/manual/tutorial/expire-data/
If you don't want to rely on mongo demon process for expiring the collection, then better to create an additional createdOn field on collection and compare it with the current timestamp to decide whether to use that document or not.
Related
I'm using Nodejs and MongoDB driver.
A two part question
I have a collection which is called openOffers which I want to expire when it hits the time closeOfferAt. I know that MongoDB offers a TTL, expireAt and expireAfterSeconds. But, when I use these, the same TTL is applied to all the documents in a particular collection. I'm not sure if I'm doing this correctly. I want document level custom expiry. Any syntax might be very useful!
Docs in openOffers
{
"_id":"12345",
"data": {...},
"closeOfferAt" : "2022-02-21T23:22:34.023Z"
}
I want to push these expired documents to another collection openOfferLog. This is so that I can retain the document for later analysis.
Current approach:
I haven't figured a way to have a customized TTL on each doc in openOffers. But, I currently insert docs into both, openOffers and openOffersLog together, and any data change in openOffers has to be separately propagated to openOffersLog to ensure consistency. There has to be a better scalable approach I suppose.
EDIT-1:
I'm looking for some syntax logic that I can use for the above use case. If not possible with the current MongoDB driver, I'm looking for an alternative solution with NodeJS code I can experiment with. I'm new to both NodeJS and MongoDB -- so any reasoning supporting the solution would be super useful as well.
There are two ways to implement TTL indexes.
Delete after a certain amount of time - this is you have already implemented
Delete at a specific clock time - for the detailed answers you can visit the MongoDB Docs
So the second option fulfills your expectation,
Just set 0 (zero) in expireAfterSeconds field while creating an index,
db.collection.createIndex({ "closeOfferAt": 1 }, { expireAfterSeconds: 0 })
just set the expiration date in closeOfferAt while inserting the document, this will remove the document at a particular timestamp.
db.collection.insert({
"_id":"12345",
"data": {...},
"closeOfferAt" : ISODate("2022-02-23T06:14:15.840Z")
})
Do not make your application logic depend on TTL indexes.
In your app you should have some scheduler that runs periodic tasks, some of them would move the finished offers to other collection and delete from the original even in bulk and no need to set a TTL index.
To keep consistency nothing better than a single source of truth, so if you can, avoid deleting and only change some status flag and timestamps.
A good use of a TTL index is to automatically clear old data after a relative long time, like one month or more. This keeps collection/indexes size in check.
I know that the TTL index in MongoDB set a time to live for its documents. When they live for the specified time then those documents gets deleted. So what if I want to archive those documents without losing them. is there any way to do it?
I am sorry If I asked it in the wrong place.
You can use a change stream to capture documents inserted and archive them somewhere. See also this answer on stackoverflow.
At the start you have a document with "column" (timestamp) that is in the TTL index. To prevent its deletion, you can move that timestamp to the future, or remove that "column" from the document.
We have one field which gets updated on user action, admin action, and in cron at the same time then what should we do in order to handle this kind of scenario in MongoDB.
e.g There is a "balance" field in the user's collection when the cron is running user's balance is decreased, now at the same time if the user is recharging and admin is refunding then the balance is not getting updated.
So please suggest any solution for this problem.
If possible, use update operations. They are atomic at the document level, so this should not be a problem.
If you are using a recent version of mongodb, you can use transactions for read-update-writes.
If you cannot do any of these, you can emulate an optimistic locking scheme using versioning to prevent unintended overwrites. There are several ways this can be done, but it generally goes like this:
Read the document. Document has a version field (which can be an integer, or a unique ObjectId. Don't use timestamp)
Make modifications in memory and update the version (increment the integer, or generate a new ObjectId)
Update the document with query containing (version: oldVersion)
This will fail if someone updated the document after you read it but before you updated it. If it fails, retry.
I heared, that mongo can do it, but I can't find how.
Can mongo create collections, which will be autoremove in future, from time, which i can setup? Or Mongo can't do this magic?
mongodb cannot auto remove collections but it can auto remove BSON records. You just need to set ttl(Time to live) index on a date field that exists in BSON record .
You can read more here MongoDb: Expire Data from Collections by Setting TTL
Collections are auto created on the first write operation (insert, upsert, index creation). So this magic is covered.
If your removal is based on time, you could use cron or at to run this little script
mongo yourDBserver/yourDB --eval 'db.yourCollection.drop()’
As Sammaye pointed out, creating indices is a costly operation. I would assume there is something wrong with your data model. For semantically distinguishing documents, I'd rather create a field on them which does that and set an expiration date or a creation date and a time frame in which the documents are valid and use TTL indices to remove all of those documents.
For using an expiration date, you have to set a field to an ISODate and create a TTL index without a duration:
db.yourColl.ensureIndex({"yourExpirationDateField":1},{expireAfterSeconds:0})
In the case you want the documents to be valid for let's say a week after they are created, you would use the following:
db.yourColl.ensureIndex({"yourCreationDate":1},{expireAfterSeconds:604800})
Either way, here is what happens: Once every minute a background thread called TTLMonitor wakes up, gets all TTL indices for the server and starts processing them. It scans the TTL index, looking for the date values, adds the value given for "expireAfterSeconds" and deletes all documents which it determined to be invalid by now. This process takes some time, so don't expect the documents to be deleted on the very second they expire.
The big advantage of that approach: you don't need any triggering logic to be maintained, the deletes are done automagically in the background and you don't put any load on your application. Plus, using an expiration date, your have very granular control over when a document expires.
The drawback is ... ... Well, if I want to find one it would be that you have to insert a creation date for every document or calculate and insert an expiration date. And you have to send an administrative command to the mongod/mongos once in the application lifetime...
I am new to Mongo. I wanted to atomically upsert a document based on multiple criteria. The document looks like the following:
{_id:..., hourOfTime:..., total:..., max:..., min:..., last:...}
This is basically hourly aggregation for number of clicks for an item identified by _id. The clicks for each item is is flushed from the application to MongoDB every five seconds. So, the document need to be updated every five seconds.
Here is the case. Lets say at t=t0, we have {_id:"nike", total:123, max:10, min:3, last:9} then at t=t1, I get message {_id:"nike", count: 12}. Now, for _id="nike", I need to do the following,
Increment total by 12
if max < 12, update max=12
if min > 12, update min=12
update last=12
I want all this operation to be atomic. I unable to convert this in one single query. Any help/hint is appreciated.
This cannot be done with a single query. Here is how I would do it:
Have a field on the document called locked. Run a findAndModify to grab the document if the locked field is false, and set the locked field to the Date() that it is locked at. This will stop other application instances from modifying the document, providing they also check for the locked field.
Modify the document application side, then atomically update it, setting locked to false.
As long as anything modifying the document runs a findAndModify on the locked field, this entire modification should be atomic.
Make sure to have a threshold at which a lock times out, so that documents do not become locked indefinitely if a machine blows up. This means that when updating the document the second time (and releasing the lock), the application should make sure that the date in the document is what it expects, to make sure it still has the lock.