Mongodb has an option to set TTL on documents.
I would like to know if there is a similar feature that allows the update of a specific field after a specific amount of time.
Basically what I want to achieve is update the field STATUS of a document from PENDING to EXPIRED after a specific amount of time (only if it was PENDING).
I know I could use cronjobs, but first I want to check whether it's possible natively with Mongodb.
Furthermore, is it possible to set the TTL with condition? Like deleting a document after X days only if STATUS is EXPIRED?
what I want to achieve is update the field STATUS of a document from PENDING to EXPIRED after a specific amount of time
Not achievable today, you have to create your own script
Furthermore, is it possible to set the TTL with condition? Like
deleting a document after X days only if STATUS is EXPIRED?
I recommend you that the script that set the status also set readyToBeDeletedSince: Date and that you put your TTL on that field.
Regarding the TTL with condition, I found that there is an option called "partialFilterExpression" that you can add to the index.
For example the following will delete a document after 5 minutes only if status is "PENDING"
db.mymodel.createIndex({ createdAt: 1 }, { expireAfterSeconds: 300, partialFilterExpression: { status: 'PENDING' } })
Related
I'm using Algolia to index my docs. I don't want the number of records to grow unboundedly. Is there a way to set an expiry time for each record so they're automatically deleted?
The best way to achieve that would be to add an attribute to your record with the expiry dates and run a script regularly to delete the data. For instance. I would use an int so you can use numeric filters (see: https://www.algolia.com/doc/api-reference/api-parameters/filters/#numeric-filters)
{
name: "test",
expire: 20171201
}
I have users adding posts. Post looks like this (pseudo-code)
post = {
_id: random,
userId: userId,
status: 'published',
expiration: 'Date.now() + 30days'
};
I want posts to automatically change status to 'expired' after expiration time has been reached. How do I do that?
A blunt approach would be to set 1m interval check in Meteor.startup on the server.
But maybe there's a better solution? Maybe there's a Mongo-specific one (like the document updating itself or smth)?
If your need to have your collection entries status updated is tied to specific logic catchable by hooks, you could use meteor-collection-hooks.
Otherwise, you could use cron job packages such as mrt:cron or meteor-synced-cron
I have been thinking about using the build in TTL feature, but it's not easy to dynamically changing the expiration date.
Since mongodb is using a background task purging the data. Is there any downside just coding my own purging function based on "> certain_date" and run say once a day?
This way, I can dynamically changing the TTL value, and this date field won't have to be single indexed. I can reuse this field as part of the complex indexing to minimize number of indexes.
There are 2 ways to set the expiration date on a TTL collection:
at a global level, when creating the index
per document, as a field in the document
Those modes are exclusive.
Global expiry
If you want all your documents to expire 3 months after creation, use the first mode by creating the index like the following:
db.events.ensureIndex({ "createdAt": 1 }, { expireAfterSeconds: 7776000 })
If you later decide to change the expiry to "4 months", you just need to update the expireAfterSeconds value using the collMod command:
db.runCommand({"collMod" : "events" , "index" : { "keyPattern" : {"createdAt" : 1 } , "expireAfterSeconds" : 10368000 } })
Per-document expiry
If you want to have every document has its own expiration date, save the specific date in a field like "expiresAt", then index your collection with:
db.events.ensureIndex({ "expiresAt": 1 }, { expireAfterSeconds: 0 })
I have been thinking about using the build in TTL feature, but it's not easy to dynamically changing the expiration date
That's odd. Why would that be a problem? If your document has a field Expires, you can update that field at any time to dynamically prolong or shorten the life of the document.
Is there any downside just coding my own purging function based on "> certain_date" and run say once a day?
You have to code, document and maintain it
Deleting a whole lot of documents can be expensive and lead to a lot of re-ordering. It's probably helpful to run the purging more often
Minimizing the number of indexes is a good thing, but the question is whether it's really worth the effort. Only you can give an answer to this question. My advice is: start with something that's already there if any possible and come up with something better if and only if you really have to.
I am attempting to expire a collection in Mongo using Casbah's ensureIndex API.
Based on this document
http://docs.mongodb.org/manual/tutorial/expire-data/
I am using casbah's proposed ensureIndex API
collection.ensureIndex(DBObject("status" -> 1, "expireAfterSeconds" -> 120))
to expire the collection in 2 minutes ...
The collection is not being evicted or expired.
Am I missing anything else here?
Thanks
There are a couple things to check:
Were you just following the docs to a T and tried to create an index on a status field which doesn't actually exist in your documents? (had to atleast ask...)
Does the status field contain JUST dates? It can theoretically be mixed, but only documents with a date type would be considered for expiration.
Have you checked your collection indexes to make sure the index was properly created?
To check for the index from the console run: db.collection.getIndexes(). If the index was created successfully, then double check you have the corresponding status fields in your documents and that they are proper dates.
Adding the index alone, doesn't create the date field for you - you would need to add it to the documents or use an existing date field that is not part of any other index.
Also note, from the docs:
TTL indexes expire data by removing documents in a background task
that runs every 60 seconds
So, if you have a 120 second expiration, bear in mind, that its possible the documents could remain for 120 seconds up to 179 seconds, give or take, depending on when the document expired and the background task last ran.
edit: As noted in the comments - a collection itself cannot be removed based on a TTL index, the index only expires the documents in the collection.
I think, you are passing the Options in the wrong way.
It should be-
collection.ensureIndex(DBObject("status" -> 1), DBObject("expireAfterSeconds" -> 120))
Instead of-
collection.ensureIndex(DBObject("status" -> 1, "expireAfterSeconds" -> 120))
I have a collections of documents (named posts) that each contain a field named category.
Each category is part of a categories collection. There are a fixed number of them (say 15).
How do I fetch the last 10 tldrs from each category?
Another solution would be to set a "flag" in each post which is actually part of the result, like:
topTen: true
Defining a sparse index on that flag would give the fastest query - at the price, of course, of the maintenance of that flag:
set the flag at insertion time (impact: one more index to update)
if it is tolerable that for a certain period the query returns 11 posts instead of 10, then trigger a background process that deletes (unsets) the 11th flag for that category
if it is not tolerable, find and unset the 11th flag at insert time
if the category of an existing post is altered, make sure the flags get set all right (for the old and the new category)
if a post gets removed that has the flag set: find and set the flag for the new 10th post
may be you'd want to provide a periodically ran process, that makes sure the flags are all set as they should be
For more information on sparse indexes, see http://docs.mongodb.org/manual/core/indexes/#index-type-sparse
Probably it will be better to just at first get the list of all categories and then for each of them get their 10 latest posts by separate queries.