We are using express and mongoose, we are trying to remove the document every 1000 seconds in the background, but MongoDB removes at an unexpected time. how to solve it?. also would like to know the difference between expires and expireAfterSeconds.
MongoDB - v3.6.5,
mongoose - 5.4.3,
express - 4.16.4
Sample Model :
const mongoose = require('mongoose');
mongoose.set('useCreateIndex', true);
const forgotPassword = mongoose.Schema({
email: { type: String, required: [true, 'Email field is required']},
expiresAt: { type: Date, expires: '2m', default: Date.now }
}, { timestamps: true, versionKey: false, strict: false });
forgotPassword.index({ expiresAt: 1 }, { expireAfterSeconds : 1000 });
module.exports = mongoose.model('forgotpassword', forgotPassword);
Both expires and expireAfterSeconds uses TTL index:
The background task that removes expired documents runs every 60 seconds. As a result, documents may remain in a collection during the period between the expiration of the document and the running of the background task.
Your documents are expected to be removed between 2 and 3 min.
UPDATE:
Check if the collection has correct indexes. Mongoose do not update indexes if the collection already have it.
If expiration time was 0 when you first created the index the documents will be removed within a minute whatever changes you do in your js code until you drop the index, collection, or the whole database.
Use syncIndexes to update indexes on the database side, but be careful to ensure it doesn't happen often on production. It may be quite expensive on large collections.
Related
I am building a mongoose model to store survey response data. There are, however, different types of surveys with different response rates. One type of survey has frequent answers (perhaps every few seconds) and data is normally queried in chunks of time, eg from startDate to endDate of the response. However, some surveys only get responses maybe a 20 times a month, and sometimes I would want to get all the data for that survey just based on the survey_id, and, not using any time field constraints.
So my question is, do secondary indexes on time series collections work as well as they would on a non-time series collection?
My model looks like this:
const responseSchema = mongoose.Schema(
{
metaData: {
type: new mongoose.Schema({
survey_id: { type: mongoose.Schema.Types.ObjectId, ref: "survey", required: true },
}),
required: true,
},
createdAt: Date,
answers: { type: Map, of: mongoose.Mixed },
},
{
timeseries: {
timeField: "createdAt",
metaField: "metaData",
granularity: "seconds",
},
}
);
responseSchema.plugin(ts);
responseSchema.index({ "metaData.survey_id": 1, "createdAt": 1 });
I would expect normal querys using the createdAt field as filters to work well, but what if I only query by survey_id and don't use the time field. Will that still work well? or do I get performance degradation by not using the time field with a time series collection.
querys of this collection will always be based on the survey_id
I have the following Mongoose Model that I wish to auto-delete after 2mins. Unfortunately, the auto-delete is not working. Note that, I wish to keep the created_at field as a Number in milliseconds not as a date. How do I go about getting the below code to work for me.
const mongoose = require("mongoose");
const TokenSchema = new mongoose.Schema(
{
_id: mongoose.Schema.Types.ObjectId,
token: String,
deleted: Boolean,
deleted_at: Number,
created_at: { type: Number, expires: '2m', default: new Date().getTime() },//Auto-Delete after 2minutes
updated_at: Number,
}
);
TokenSchema.pre('save', function (next) {
let shadow = this;
let now = new Date().getTime();
shadow.updated_at = now;
if (!shadow.created_at) {
shadow.created_at = now;
}
next();
});
Thank you
Mongoose uses MongoDB TTL Indexes for expiring documents, which only functions on fields containing either a Date or array of Date values.
If the indexed field for a document contains any other type, it will not be automatically expired, so to get auto-expiry working, you will need to have created_at store type: Date.
MongoDB internally stores dates as the number of milliseconds since epoch, which you can extract with the valueOf() method, and the mongo query language permits querying a date field by pass a number of milliseconds.
const mongoose = require('mongoose');
const schema = new mongoose.Schema({
number: {
type: String,
required: true,
},
otp: {
type: String,
required: true,
},
});
schema.index({createdAt: 1}, {expireAfterSeconds: 30}); //The otp fails to work after 30 seconds
const model = mongoose.model('otp', schema);
module.exports = model;
This OTP document should expire but is not expiring. I tried to recreate the database a few times and look up for possibilities but couldn't find the error.
The background task that removes expired documents runs every 60 seconds.
https://docs.mongodb.com/manual/core/index-ttl/#timing-of-the-delete-operation
The background task that removes expired documents runs every 60 seconds. As a result, documents may remain in a collection during the period between the expiration of the document and the running of the background task.
Because the duration of the removal operation depends on the workload of your mongod instance, expired data may exist for some time beyond the 60 second period between runs of the background task.
The TTL index does not guarantee that expired data will be deleted immediately upon expiration. There may be a delay between the time a document expires and the time that MongoDB removes the document from the database.
Read - http://hassansin.github.io/working-with-mongodb-ttl-index#ttlmonitor-sleep-interval
you need to have a field with createdAt date type , see in mongodb documentation :
db.log_events.createIndex( { "createdAt": 1 }, { expireAfterSeconds: 3600 } );
db.log_events.insert( {
"createdAt": new Date(),
"logEvent": 2,
"logMessage": "Success!"
} )
I am storing my refresh tokens in mongodb database. For testing purposes I want them to expire in 4 minutes. For some reason, this thing is not working for me.
const mongoose = require('mongoose');
let schema = new mongoose.Schema({
token: {
type: String,
required: true
},
username: {
type: String,
required: true
},
email: {
type: String,
required: true
},
});
schema.index({expireAt: 1}, {expiresAfterSeconds: 240}); //4 mins * 60 seconds
let model = mongoose.model('refresh_token', schema);
module.exports = model;
This is the complete code of my file. I am using this to create the refresh tokens. The item is persisting for an hour as of now. Please shed some light on my mistake.
OK, I solved the issue and it was a blunder from my side.
If you are using mongoose and doing the testing, you would most likely be changing the TTL expire time. I was changing it to see if it was working and for different testing purposes but once the document is created in the atlas, requesting a different TTL time won't overwrite the previous one. I changed the time from 30 months to 5 minutes and did a lot of fluctuation for testing purposes.
So keep this in mind that once the model is created, the TTL will be locked and you need to delete the collection and re-build it otherwise you have to change the TTL settings manually in the atlas(I didn't checked this out because my problem was solved with this only and I was in testing mode of my application). Also
thanks to wak786
for proposing to see the documentation again. It clicked when I was reading how indexing works.
My final refresh token file looks like this after I deleted the collection(actually renamed it).
const mongoose = require('mongoose');
let schema = new mongoose.Schema({
token: {
type: String,
required: true
},
username: {
type: String,
required: true
},
email: {
type: String,
required: true
},
createdAt: {
type: Date,
default: new Date()
}
});
schema.index({"createdAt": 1}, {expireAfterSeconds: 2592000}); //30days * 24hours * 60 minutes * 60 seconds
let model = mongoose.model('token', schema);
module.exports = model;
You are trying create index using following command.
schema.index({expireAt: 1}, {expiresAfterSeconds: 240});
But the field expireAt does not exist in your schema. And as per the mongo docs :-
If a document does not contain the indexed field, the document will not expire.
Reference:- https://docs.mongodb.com/manual/core/index-ttl/
I have a very certain thing i want to accomplish, and I wanted to make sure it is not possible in mongoose/mongoDB before I go and code the whole thing myself.
I checked mongoose-ttl for nodejs and several forums and didn't find quite what I need.
here it is:
I have a schema with a date field createDate. Now i wish to place a TTL on that field, so far so good, i can do it like so (expiration in 5000 seconds):
createDate: {type: Date, default: Date.now, expires: 5000}
but I would like my users to be able to "up vote" documents they like so those documents will get a longer period of time to live, without changing the other documents in my collection.
So, Can i change a TTL of a SINGLE document somehow once a user tells me he likes that document using mongoose or other existing npm related modules?
thank you
It has been more than a year, but this may be useful for others, so here is my answer:
I was trying accomplish this same thing, in order to allow a grace period after an entry deletion, so the user can cancel the operation afterwards.
As stated by Mike Bennett, you can use a TTL index making documents expire at a specific clock time.
Yo have to create an index, setting the expireAfterSeconds to zero:
db.yourCollection.createIndex({ "expireAt": 1 }, { expireAfterSeconds: 0 });
This will not affect any of the documents in your collection, unless you set expireAfterSeconds on a particular document like so:
db.log_events.insert( {
"expireAt": new Date('July 22, 2013 14:00:00'),
"logEvent": 2,
"logMessage": "Success!"
} )
Example in mongoose
Model
var BeerSchema = new Schema({
name: {
type: String,
unique: true,
required: true
},
description: String,
alcohol: Number,
price: Number,
createdAt: { type: Date, default: Date.now }
expireAt: { type: Date, default: undefined } // you don't need to set this default, but I like it there for semantic clearness
});
BeerSchema.index({ "expireAt": 1 }, { expireAfterSeconds: 0 });
Deletion with grace period
Uses moment for date manipulation
exports.deleteBeer = function(id) {
var deferred = q.defer();
Beer.update(id, { expireAt: moment().add(10, 'seconds') }, function(err, data) {
if(err) {
deferred.reject(err);
} else {
deferred.resolve(data);
}
});
return deferred.promise;
};
Revert deletion
Uses moment for date manipulation
exports.undeleteBeer = function(id) {
var deferred = q.defer();
// Set expireAt to undefined
Beer.update(id, { $unset: { expireAt: 1 }}, function(err, data) {
if(err) {
deferred.reject(err);
} else {
deferred.resolve(data);
}
});
return deferred.promise;
};
You could use the expire at clock time feature in mongodb. You will have to update the expire time each time you want to extend the expiration of a document.
http://docs.mongodb.org/manual/tutorial/expire-data/#expire-documents-at-a-certain-clock-time