I am having a MongoDb collection of 70M items (200GB) and I am trying to get those in the range [from_date, to_date] using the command:
from_date = datetime.datetime(2015, 1, 14, 9, 46, 23)
to_date = datetime.datetime(2015, 1, 14, 9, 46, 24)
db.collection.find({"datetime": {"$gte": from_date, "$lte": to_date}})
However it takes a lot of time to retrieve those items even for a single query. Is there any more efficient way to do this?
Related
This question already has answers here:
MongoDB / Pymongo Query with Datetime
(2 answers)
Closed 7 years ago.
I have documents in a MongoDB collection that have a filed 'date'. In the Mongo Shell they appear as ISO Date.
For example:
"date" : ISODate("2015-12-21T09:00:05.396Z")
I would like to find documents within a certain range date.
The documents were inserted with PyMongo if it makes any difference.
How can I do this?
Thanks!
Operate regular Python datetime objects (documentation sample):
from datetime import datetime
start = datetime(2015, 12, 20, 7, 51, 04)
end = datetime(2015, 12, 21, 7, 52, 04)
col.find_one({'date': {'$lt': end, '$gt': start}})
from the mongodb shell, I'm having an issue with a date query. This is a test database and the problem is better explained by example:
Example field in a collection
"createdOn" : ISODate("2015-11-23T00:49:01.800Z")
When I enter a date of this month for testing at the shell such as:
new Date(2015, 11, 23)
ISODate("2015-12-23T05:00:00Z")
You can see it's a month a head, meaning It's messing my queries. Where I have to drop back a month in order to get what I need for this month:
db.collection.find({'createdOn': {'$lte': new Date(2015, 10, 23)}}).count()
306
db.collection.find({'createdOn': {'$gte': new Date(2015, 10, 23)}}).count()
10
All entries above were created in Nov, but when using the correct month:
db.collection.find({'createdOn': {'$gte': new Date(2015, 11, 23)}}).count()
0
How does this even happen?
You should use Date like this: new Date("2015-11-23"). See Mongo Reference
What you are doing is almost right but you have missed on a point that:
Javascript counts month from 0 to 11 instead of 1 to 12
January is 0. December is 11. So to see for entries created in november you have to apply query like:
db.collection.find({'createdOn': {'$gte': new Date(2015, 10, 23)}})
I am trying to find all documents with date field within date range as shown below in my Meteor app, but for some reason it is always returning empty records, can someone please tell me what I might be doing wrong / missing here? Thanks
Sample document:
_id: "tMSfNq9JR85XDaQe5"customerid: "QDGvBQhS6vYgZtnRr"date: Sun Dec 07 2014 19:50:21 GMT+0800 (HKT)description: "Test"
Tried using queries from Chrome console as follows:
Custlog.find({date: {$gt: new Date(2014, 12, 1) , $lt: new Date(2014, 12, 10) }}).fetch()
Custlog.find({date: {$gt: new Date(01/12/2014) , $lt: new Date(10/12/2014) }}).fetch()
In their infinite wisdom, the implementors of the JavaScript Date Object made the month field zero indexed. So, January 1 is new Date(2014, 0, 1) and December 1 is new Date(2014, 11, 1). What you have Date(2014, 12, 1) is actually January 1, 2015.
Similarly, new Date(01/12/2014) is equivalent to dividing 1 by 12 and then dividing that by 2014 and then sending that as a number to the date object which is zero so it should give you the Unix Epoc, which in your time zone (GMT+08) is January 1, 1970.
On the mongo shell this returns a document just fine:
> db.orderbook_log.findOne({'time': { '$gte': new Date(2014, 9, 24, 17, 38, 20, 546000), '$lt': new Date(2014, 10, 24, 17, 39, 20, 546000)}})
//... returns document with this time stamp:
"time" : ISODate("2014-10-25T00:47:30.819Z")
Notice I used "9" for October because JavaScript's months are 0-11.
And I also tested with "23" as the day because it looks like JS days are also 0-indexed, and that also returned a document: "time" : ISODate("2014-10-24T17:32:13.595Z")
atime = datetime.datetime(2014, 10, 24, 17, 38, 20, 546000)
btime = datetime.datetime(2014, 10, 24, 17, 39, 20, 546000)
future_book = log.find_one({"time": {"$gte": atime, "$lt": btime}})
But when I execute find_one in pymongofuture_book is None
All I'm really trying to do is loop though the first 100 records or so and get a record that occurred a relative minute later.
Javascript days are not zero-indexed, alas. Only months are.
I see in your Javascript you're adding 546,000 ms to the first date, so that results in 2014-10-24 at 17:48:26. Javascript then converts to your local timezone, so in my case it adds 5 hours:
> new Date(2014, 10, 24, 17, 39, 20, 546000)
ISODate("2014-11-24T22:48:26Z")
This is then compared (ignoring timezones) with the "time" field in your documents.
Better to remove the final milliseconds argument, and use the MongoDB shell's ISODate function, which is designed to be more rational than Javascript Dates:
> ISODate('2014-10-24T17:38:20')
ISODate("2014-10-24T17:38:20Z")
That will then compare to your documents in the way you expect, and it should match the PyMongo behavior. I suggest you drop the milliseconds argument from your Python datetime constructors, too, to clarify things.
I save a mongodb field 'date' as datetime object
and use following queries
1.db.test_collection.find({'date': {'$gte': start_date_time}})
2. db.test_collection.find({"date": {"$gte": start_date_time, "$lte": end_date_time}
where start_date_time is (say)2014-06-03 00:00:00 and end_date_time 2014-06-03 23:59:59
i insert some values to mongodb, But I don't give any results.
please help me to solve the issue.
You should use the datetime type.
import datetime
db.test_collection.find({'date': {'$gte': datetime.datetime(2014, 6, 3, 16, 46) }})
You can see more in the documentation (http://api.mongodb.org/python/current/tutorial.html)
Hope it works fine for you!