I'm looking to do an extract from a MongoDB from a particular date.
Since I'm using a component in Talend that sends the query I'm kind of limited in the sense that I can't use multiple lines of code.
Can you do a date limitation directly in the find-method?
db.example.find({ ts: { $gt: lowdate} });
Where lowdate is substituted for something that I hope any of you can figure out.
Many thanks!
PS. The date format in the mongodb, if that matters, is "Dec 16, 2011 7:37:06 PM".
--- Update ---
From my MongoDB:
, "ty" : "auth", "ts" : "Dec 16, 2011 3:28:01 PM",
which suggests the format of the timestamp (ts) is a string. Correct?
If the date is stored as a string in that format, you will not be able to make a query. In order to fix this, I suggest you write a script in your favourite language that scans all the documents and convert this date-as-a-string into a timestamp. You can either overwrite the "ts" field, or create a new one, f.e. something called "ts_int".
For example, in PHP you would do:
<?php
$m = new Mongo();
$c = $m->mydbname->example;
foreach ( $c->find() as $item )
{
$item['ts_int'] = strtotime( $item['ts'] );
$c->update( $item );
}
?>
Related
I have around 30-40 records like the example before in my database and I'm looking to get the notifications that are less than 1 month old (from today's date). Is there a way in Mongo to get these results without having to pass in a today's date via JavaScript? Or if I do have to pass it in via JavaScript, how would I process this against my created date?
{
"_id" : ObjectId("48445b4dc72153e9ad7f3bfb"),
"notificationID" : "78723asd5-vnbb-xv31-afe0-fa9asf164e4",
"notification" : "Notification #1",
"created" : ISODate("2016-11-21T20:33:53.695Z")
}
Any help is appreciated, thanks.
MongoDB has its own Javascript interpreter so, unless your MongoDB server has a different date than your system, it knows the current date so you easily use simple Javascript to compute the value you're looking for using a regular Date object and use it in your query.
var d = new Date();
d.setMonth(d.getMonth() - 1); //1 month ago
db.data.find({created:{$gte:d}}); //change "data" for your collection's name
If you need a different date than your database's, I'm afraid you'll have to somehow pass it as a parameter.
const now = new Date()
const temp = new Date(now).setMonth(now.getMonth() - 6);
const priorSix = new Date(temp)
Table.find({"date" : {$gte: priorSix, $lt: new Date()}}, (err, tables) => {
if(err) throw new Error(err)
res.status(200).json(tables)
}).populate('foodList.item')
This code worked for me. It retrieves documents of the last 6 month :)
I am currently trying to filter a meteor collection based on two parameters they are: currentDate and coming from the collection dateEventEnds.
This is my current publish code:
Meteor.publish('recursos', function() {
let today = Date.now();
return Recursos.find({ 'recursos.fechaEnd': { $gt: today }});
});
Which is effectively not doing anything. I think this is due to:
Date.now() producing the date in milliseconds. Ex: 1463058648464
While recursos.fechaEnd has it stored like this: "2016-04-30T00:00:00.000Z"
Have also tried using: new Date(), but still nothing happens:
Meteor.publish('recursos', function() {
let today = new Date();
return Recursos.find({ 'recursos.fechaEnd': { $gt: today }});
});
new Date() results are like this: Thu May 12 2016 09:50:16 GMT-0400 (Eastern Daylight Time)
So it looks like the issue revolves around the way dates are presented. How could I change them to both use the same date format? Is it by changing them to Milliseconds?
The template where I need to do the filtering is suscribe to both the collection and an EasySearchIndex which looks like this:
RecursosIndex = new EasySearch.Index({
collection: Recursos,
fields: ['clase', 'direccion.city'],
engine: new EasySearch.MongoDB({
sort: () => ['fechaStart']
})
});
Should the filtering happen here?
Thanks
Your logic is perfectly fine, Mongo is quite capable of searching and sorting by date fields. The problem is that your dates in Mongo are stored as strings and not as actual dates. If you look at an object you should see fields like:
A real date:
"createdAt" : ISODate("2015-09-18T05:42:50.105Z")
A string:
"createdAt" : "2015-09-18T05:42:50.105Z"
To upgrade your existing dataset from strings to dates, run the following:
Recursos.find().forEach(function(r){
Recursos.update(r._id,{$set: { 'recursos.fechaEnd': new Date(r.fechaEnd) }});
});
Of course you can convert more than one date at a time, I suspect your object also has a fechaStart.
I'm saving data into the bongo as bulk insert. The data that's an array of JSON object contain date, numeric, alphanumeric data all saved as string.
Sample Data
[{
"CARDNO": "661",
"HOLDERNO": "661",
"HOLDERNAME": "S",
"IODATE": "4/1/2012",
"IOTIME": "00:03:27",
"IOGATENO": "01-3",
"IOGATENAME": "FWork",
"IOSTATUS": "Entry",
"DEPARTMENTNO": "1",
"UPDATE_STATUS": "1"
}, {
"CARDNO": "711",
"HOLDERNO": "711",
"HOLDERNAME": "P",
"IODATE": "4/1/2012",
"IOTIME": "04:35:33",
"IOGATENO": "01-7",
"IOGATENAME": "FDWork",
"IOSTATUS": "Exit",
"DEPARTMENTNO": "3",
"UPDATE_STATUS": "1"
}]
My Query
var start = new Date(2012, 4, 15);
var end = new Date(2012, 4, 1);
collection.find({
"IODATE": {
$gte: start,
$lt: end
}
}).toArray(function (err, data) {
if (err) {
console.log(err);
} else {
console.log(data.length)
}
//res.send(data.length);
res.send(JSON.stringify(data));
});
It's not returning result, I think it is because the value of "IODATE" is in string inside db.
How to work around this issue? I may need to do bulk insert since the data can be of 200 million or so.
One last try at this, because you don't have a good record of accepting good advice.
Your date formats as they stand are going to bite you. Even where trying to work around them. Here are the problems:
The format is not lexical. Which means that even with a string comparison operators like $gte, $lte are just not going to work. A lexical date would be "2012-01-04" in "yyyy-mm-dd" format. That would work with the operators.
You could look at $substr (and it's complete lack of documentation, search on SO for real usage) within aggregate but your date format is lacking the double digit form of day and month ie "04/01/2012", so that is going to blow up the positional nature of the operator. Also you would have to transform before any $match which means you blow up any chance of reducing your pipeline input, so you are stuck with not being able to filter your large resultset by date.
It's a horrible case, but there really is no other practical solution to the large data problem here than to convert your dates. Strings in the form that you have just do not cut it. Either, in order of preference convert to:
BSON date
epoch timestamp as long (whatever)
Lexical string representation (as described)
Your main case seems to be filtering, so updating the dataset is the only pratical alternative. Otherwise you are stuck with "paging" results and doing a lot of manual work, that could otherwise be done server side.
I've got a mongo object that follows this pattern. Actions is an array of objects. I've excluded the irrelevant fields.
{
"_id" : 141,
...
"Actions" : [{
...
"Modified" : new Date("Thu, 29 Nov 2012 14:41:20 GMT -08:00"),
...
}]
...
}
How do I query this so that I can get a list of objects which contain an object in the actions array who's modified property falls between a date range using the C# Mongo Driver.
I figured this out myself.
Query.ElemMatch("Actions",
Query.And(
Query.GTE("Modified", start),
Query.LTE("Modified", end)
)
)
Using a Query.ElemMatch against the Actions field passing in the Query for the Sub Elements.
Worked like a charm.
I've been trying to find a way to create an ISODate object whith pyMongo client, but without any success so far.
I use http://pypi.python.org/pypi/pymongo3 client, which is the only serious one available in Python 3 for now, but the problem doesn't seem to come from this specific pymongo version.
I'd like to know if any of you has found a solution to use this MongoDB object type from a pymongo client... thanks for your help !
You just need to store an instance of datetime.datetime.
Inserting from the python shell:
>>> c.test.test.insert({'date': datetime.datetime.utcnow()})
ObjectId('4e8b388367d5bd2de0000000')
>>> c.test.test.find_one()
{u'date': datetime.datetime(2011, 10, 4, 16, 46, 59, 786000), u'_id': ObjectId('4e8b388367d5bd2de0000000')}
Querying in the mongo shell:
> db.test.findOne()
{
"_id" : ObjectId("4e8b388367d5bd2de0000000"),
"date" : ISODate("2011-10-04T16:46:59.786Z")
}
For those who are wondering how to create ISODate from timestamp:
ts = time.time()
isodate = datetime.datetime.fromtimestamp(ts, None)
This will create datetime object with no timezone. When inserted to MongoDB it will get converted to proper ISODate().
Also, I strongly recommend looking at Python TimeTransitionsImage. Note that tuple here is named tuple (equivalent to struct in C). And also note that tuple fields are not the same as in C counterparts, even though the naming is the same (for instance, tm_wday starts with Monday and not Sunday).
Actually that does not work either. When you try to use either utcfromtimestamp or fromtimestamp, the program errors out saying that it needs a float. Just parse the string into a date time object and use that directly in the Mongodb. filter
from_dt = datetime.strptime('2018-04-01','%Y-%m-%d')
#from_dts = datetime.utcfromtimestamp(from_dt)
to_dt = datetime.strptime('2018-04-30','%Y-%m-%d')
#to_dts = datetime.utcfromtimestamp(to_dt)
filterCondition = {
"LastLogin" : { "$lte" : to_dt},
"LastLogin" : { "$gte" : from_dt}
}
And then
db[(colName)].find({ "<colName>" : filterCondition })
Would work...
result = db.objects.insert_one(
{"last_modified": datetime.datetime.utcnow()})
Here utc stands for Universal Time Coordinates.
For create a document with specific date, for example 03/10/1999, run this:
from datetime import datetime
from pymongo import MongoClient
db = MongoClient().db_name
date = datetime(1999, 03, 10)
db.collection.insert_one({'date': date})