we are currently experimenting with Mongo's new TTL feature and would like records to expire based on a date that is in the "future" as of the creation time of the record. This is so different records can have different periods of validity.
For example, something like this:
db.createCollection("sushi")
db.sushi.ensureIndex({"best_before": 1}, {expireAfterSeconds: 1})
db.sushi.insert({name: "ngiri", best_before: new Date('2012/10/02')})
But in our initial tests documents do not get removed from the collection if the indexed date field is in the future at the time of the creation of the record.
Is there any reason why this shouldn't work?
Thanks
Looks like it was daylight saving that go me here. Lesson learnt.
Related
Is there a simple OR elegant method (or query that I can write) to retrieve the last updated timestamp (of the last updated document) in a collection. I can write a query like this to find the last inserted document
db.collection.find().limit(1).sort({$natural:-1})
but I need information about the last updated document (it could be an insert or an update).
I know that one way is to query the oplog collection for the last record from a collection. But it seems like an expensive operation given the fact that oplog could be of very large size (also not trustworthy as it is a capped collection). Is there a better way to do this?
Thanks!
You could get the last insert time same way you mentioned in the question:
db.collection.find().sort({'_id': -1}).limit(1)
But, There isn't any good way to see the last update/delete time. But, If you are using replica sets you could get that from the oplog.
Or, you could add new field in document as 'lastModified'.
You can also checkout collection-hooks. I hope this will help
One way to go about it is to have a field that holds the time of last update. You can name it updatedAt. Every time you make an update to the document, you'll just update the value to the current time. If you use the ISO format to store the time, you'll be able to sort without issues (that's what I use).
The other way is the _id field.
Method 1
db.collection.find().limit(1).sort({updatedAt: -1})
Method 2
db.collection.find().limit(1).sort({_id: -1})
You can try with ,
db.collection.findOne().sort({$natural:-1}).limit(1);
I know there is possible to remove some documents when the saved date is passed away with this command:
db.course.deleteMany({date: {"$lt": ISODate()}})
I trying to do the same thing but I'm trying to check is a timestamps is passed away
All saved timestamps are like this one
1492466400000
Is it possible to make a command with a condition to delete all documents with a too old timestamp?
EDIT
I use milliseconds timestamps
In MongoDB you 3.2 you can also use TTL Indexes where you can say to MongoDB "please remove all the Documents if the {fieldDate} is more older then 3600 seconds". This is pretty useful for Logging (remove all logs more older then 3 months).
Maybe is not your use case, but I think is pretty good to know.
TL indexes are special single-field indexes that MongoDB can use to automatically remove documents from a collection after a certain amount of time or at a specific clock time. Data expiration is useful for certain types of information like machine generated event data, logs, and session information that only need to persist in a database for a finite amount of time.
db.eventlog.createIndex( { "lastModifiedDate": 1 }, { expireAfterSeconds: 3600 } )
https://docs.mongodb.com/v3.2/core/index-ttl/
I find the way to do this
db.course.deleteMany({date: {"$lt": ISODate()*1}})
ISODate()*1 condition will convert the Date into a timestamp
and
ISODate(timestamp) convert the timestamps into a Date.
I am constructing a database where I might want to query a day of the week. Is it possible to use mongodb to query days in the week in a datetime (or utc timestamp) field?
Something like; get every object that has a datetime that was on a monday.
If it is not possible then the alternative seems to create dummy variables in the collection that show what day of the week it was. Preferably I would like to only query the datetime object for this as this would keep the database smaller.
There are three solutions that I can think of:
Your solution: create an extra "day_of_week" field, either an int or string, and then query against this field rather than the datetime field.
Query for everything in your collection, and then filter the results by day of the week on the client side.
Use $where, passing a javascript function which calls date.getDay(). For example, {$where: function () { return this.date.getDay() == 5; }} for getting every date on a Friday.
Solution #2 would call datetime.date.weekday() in pymongo on the client side. The downside of this method is that every document in the collection will end up being sent over the wire, which could add unnecessary network load. It's better than #1, however, in that it's more space efficient and you don't have duplicated information to keep in sync. Solution #3 has neither of these problems, but $where is slow because it requires the server to create a JavaScript execution context and cannot make use of indexes.
Pymongo can return Mongo BSON timestamp fields as python datetimes: http://api.mongodb.org/python/current/api/bson/timestamp.html
From there you can call datetime.date.weekday()
http://docs.python.org/2/library/datetime.html#datetime.date
I have a collection of data and I want to get it sorted by insertion time. I have not any additional fields to store the insert time. But as I found out I can get this time from Id.
I have tried this code:
return bookmarks.find({}, {sort: {_id.getTimestamp(): 1}, limit: 10});
or
return bookmarks.find({}, {sort: {ObjectId(_id).getTimestamp(): 1}, limit: 10});
but get the error message:
=> Your application has errors. Waiting for file change.
Is there any way to sort collection by insertion datetime using only id field ?
At the moment this isn't possible with Meteor, even if it is with MongoDB. The ObjectID's created with meteor don't bear a timestamp. See http://docs.meteor.com/#collection_object_id
The reason for this is client side code can insert code and it can arrive late on the server, hence there is no guarantee the timestamp portion of the ObjectID will be accurate. In addition to the latency the client side's date is used meaning if they're off it's going to get you incorrect data. I think this is the reason they use an ObjectID but it is completely random.
If you want to sort by date you have to store the time/date separately.
The part what i striked out is not accurate. Meteor use it is own id generation which is based on a random string that is while does not apply the doc what i linked before. Check sasha.sochka's comment under.
It is nearly but not 100% good if you just sort for the _id field . While as it is constructed the first 4 byte is the timestamp in secs (so sorting for the getTimestamps value is not better). Under one second resolution you cannot get the exact order, as it is mentioned in the documentation: http://docs.mongodb.org/manual/reference/object-id/#objectid
It is still true that you can try to check the exact order of the insert/update ops against your collection in the oplog, if you have an oplog, but as it is a capped collection anyway you will see the recent operations only. http://docs.mongodb.org/manual/core/replica-set-oplog/.
I am very new to mongodb
I have a database with sale_date and the value is saved as text and the format is "dd:mm:yyyy". Now I want to query based on the date. Like I want to query the last month's entry.
I also have field sale_time and also saved as text and the format is "hh:mm" and I want to query the last hour's entry.
**I want to query from java and also from the mongo console.
One row of my collection:
{
"_id":112350,
"sale_date":"21.07.2011",
"sale_time":"18:50",
"store_id":"OK3889-45",
"region_code":45,
"product_id":"QKDGLHX5061",
"product_catagorie":53,
"no_of_product":1,
"price":1211.37,
"total_price":1211.37
}
I have million of entries. Now I want to find the entries for the month of July 2011 or hour from 18:00 to 19:00 in 21.07.2013.
You can query with a regex matching your results. You said format dd:mm:yyyy but the example looks like dd.mm.yyyy so I used that in examples
For example:
db.sales.find({sale_date: /..\.07\.2011/})
This will be ineficient since it can't use an index, but it will get the job done.
It would be better, if you stick with dates as strings to reverse the order to yyyy:mm:dd then you could use a anchored regex, which will hit an index like:
db.sales.find({sale_date: /2011\.07/})
For the hour query:
db.sales.find({sale_date: "21.07.2013", sale_time: {$gte: "18:00", $lt: "19:00"}})
There is no efficient and reliable way to query the for a date range you want given the date structure you have used. A regex style query would scan through the entire collection for example, and if you have millions of documents, that's not acceptable.
You could theoretically create a MapReduce to better structure the data into a new collection. But, that will be more work to maintain (as MapReduces aren't automatically updated, and may make other queries and data fetching involve more steps than you'd like).
Although, if you're willing to do that, I'd strongly suggest you instead just fix your data as I mentioned in my comment to be a standard YYYYMMDD. Even better, you may want to consider merging the time and would be to include the time stamp in the same field:
2013-07-21T14:30
If you don't though, you can still do the single date query reasonably (although you'd want to index both the date and time as a compound index):
db.sales.ensureIndex({ sale_date: 1, sale_time: 1})
Regarding the code, it's basically going to look like this:
BasicDBObject date = new BasicDBObject("sale_date", "21.07.2013");
BasicDBObject time = new BasicDBObject("sale_time",
new BasicDBObject("$gte", "18:00").
append("$lte", "19:00"));
BasicDBObject andQuery = new BasicDBObject();
List<BasicDBObject> obj = new ArrayList<BasicDBObject>();
obj.add(date);
obj.add(time);
andQuery.put("$and", obj);
cursor = coll.find(andQuery);