I have a collection called members and here is a sample data:
{
"_id": 123,
"name": "Jackie",
"dob": "31/12/18"
}
Is it possible to convert the dd/mm/yy format to dd/mm/yyyy? As $dateFromString only accepts the latter.
(p.s. $toString/$toInt/$toDecimal is not supported as the MongoDB version is before version 4)
Use https://docs.mongodb.com/manual/reference/operator/aggregation/split/ then https://docs.mongodb.com/manual/reference/operator/aggregation/concat/ to get the format you want.
Related
I have a timstamp in milliseconds, like 1645825932144
I'd like to make a date range query with elastic search query string for being able to get all records whom timestamp is in the last 24h:
timestamp:[now-24h TO now]
This does not work as timestamp is in milliseconds and now produces strings like 2001-01-01 13:00:00
Is it possible to achieve this with a cast or something?
I read about range queries and date math, but did not find anything.
It's easy to compute the timestamp in milliseconds for now and now-24h so why not do it in your application logic and build the query out of those values?
For instance (in JS),
const now = new Date().getTime();
const now24h = now - 86400000;
const query = `timestamp:[${now24h} TO ${now}]`;
query would contain the following value:
timestamp:[1647785319578 TO 1647871719578]
UPDATE:
PS: I might have misunderstood the initial need, but I'm leaving the above answer as it might help others.
What you need to do in your case is to change your mapping so that your date field accepts both formats (normal date and timestamp), like this:
PUT your-index/_mapping
{
"properties": {
"timestamp": {
"type": "date",
"format": "date_optional_time||epoch_millis"
}
}
}
Then you'll be able to query like this by mix and matching timestamps and date math:
GET test/_search?q=timestamp:[now-24h TO 1645825932144]
and also like this:
GET test/_search?q=timestamp:[1645825932144 TO now]
I have this function which parses the dates in my database
def parserQuestions():
res=db.questionsActual.find({'date_created':{'$type':'string'}})
for doc in res:
db.questionsActual.update_one({'_id': doc.get('_id')}, {'$set': { 'date_created': parse(doc.get('date_created'))}})
My input format is:
2020-12-11T13:23:58.677-04:00
but when I apply the function, the output is:
2020-12-11T13:23:58.677.000Z
How can I keep the timezone? or even better, can I set my own timezone?
the desired output is:
2020-12-11T13:23:58.677-04:00
MongoDB's BSON date type always stores dates as UTC date/times, so you cannot store an offset directly in a BSON date field. You can choose to store your offset in a separate field with something like:
from pymongo import MongoClient
from dateutil import parser
db = MongoClient()['mydatabase']
dt = parser.parse('2020-12-11T13:23:58.677-04:00')
db.questionsActual.insert_one({'date': dt, 'offsetSeconds': dt.tzinfo.utcoffset(dt).total_seconds()})
print(db.questionsActual.find_one({}, {'_id': 0}))
gives:
{'date': datetime.datetime(2020, 12, 11, 17, 23, 58, 677000), 'offsetSeconds': -14400.0}
I have a aggregation pipeline like:
db.Collection.aggregate([{
$project:{
Name:1,
ModDate: new Date("$history.ModDate")
}}])
I tought that Date would convert it to local time but I get the default value of: 1970-01-01 00:00:00.000Z
Even doh the value of the ModDate is something like:
2015-10-18 06:50:19.000Z
And ModDate on the history subdocument is a date field
Is there a way to convert the isodate to the server local date?
Have you tried
ModDate: Date("$history.ModDate")
(i.e. without the new operator)
I have downloaded huge chunks of data in the format in csv. I am using mongoimport to enter the data into MongoDB for processing.
How do I get the date into date format recognized by MongoDB?
sample data with header
Date, Open Price, High Price, Low Price, Last Traded Price , Close Price, Total Traded Quantity, Turnover (in Lakhs)
04-Apr-2014,901,912,889.5,896.75,892.85,207149,1867.08
03-Apr-2014,908,918,897.65,900,900.75,156260,1419.9
02-Apr-2014,916,921.85,898,900.7,900.75,175990,1591.97
As far as I know, there is no way to do this with mongoimport.
But this is achievable by importing the data and then running the following script (note that there is no point of all this hastle with a monthes as in Neil's Lunn script, because mongo can properly convert your date by doing this new Date('04-Apr-2014')):
db.collName.find().forEach(function(el){
el.dateField = new Date(el.dateField);
db.collName.save(el)
});
PS If timezone is so important (I assume that it is not, if there are only dates without time information), you can just change timezone on your local machine and then run the query. (Thanks to Neil Lunn for clarification regarding this)
As of Mongo version 3.4, you can use --columnsHaveTypes option to specify the type of your field while using mongoimport to import your data.
here is the link for reference.
Sample mongoimport syntax below:
mongoimport --db XYZ --collection abc --type tsv --fields id.int32(),client_name.string(),app_name.auto(),date.date() --columnsHaveTypes --file "abc.tsv" --verbose
You basically have three options here as though you can import CSV directly using mongoimport, it has no idea how to convert dates from this format.
Convert your CSV input to JSON format by whatever means. For your date values you can use the extended JSON syntax form that will be recognized by the tool. The resulting JSON you produce can then be passed to mongoimport.
Write your own program to import the data by reading your CSV input and doing the correct conversions.
Import the CSV content as is, and then manipulate the data directly in your MongoDB collection using your language of choice.
One take on the third option would be to loop the results and update the dates accordingly:
var months = [
"Jan", "Feb", "Mar", "Apr", "May", "Jun",
"Jul", "Aug", "Sep", "Oct", "Nov", "Dec"
];
db.collection.find({ },{ "Date": 1 }).forEach(function(doc){
var splitDate = doc.Date.split("-");
var mval = months.indexOf( splitDate[1] );
mval = ( mval < 10 ) ? "0" + mval : mval
var newDate = new Date( splitDate[2] + "-" + mval + "-" + splitDate[0] );
db.collection.update(
{ _id: doc._id },
{ "$set": { "Date": newDate } }
);
})
And that would make sure your dates are then converted to the correct BSON date format with the same matching date values you are expected.
Beware of "local" timezone conversions, you will want to be storing as UTC time.
I have the following data.
{
deviceID: 186,
date: "2014-3-15"
}
{
deviceID: 186,
date: "2014-3-14"
}
{
deviceID: 186,
date: "2014-3-13"
}
And some lower dates, like 2014-3-9 , 8 ,7 ,6 etc.
When doing a db.coll.remove({date:{$lte:"2014-3-5"}})
Mongo removes the 15,14,13 aswell, but keeps single digit day dates. Is this maybe due to the date is a string?
I dont know how else to format the date so I can remove all dates below a certain date.
It is supposed to be a cleaning process, removing all documents with a date lower than specified.
Its because the date field you are querying on is a string filed and not a Date(). In your mongo documents instead of a custom date string, insert javascript date objects into date field.
like
{ deviceID: 186,,"date": new Date(2012, 7, 14) }
and when you execute the remove do it like
db.coll.remove({date:{$lte:new Date(2012, 7, 14)}})
If you want to remove data from MongoDB from the date less than specified, you MUST make sure of the date.
Easiest way for you to check whether you are inputting the right format is to test it before you use it in your query.
For example if you want to get current date in ISODate in Mongo shell, just type new Date and you will get the current date in Mongo.
I've tried the following in the Mongo shell:
new Date(2017, 11, 1)
and it returns
ISODate("2017-11-30T16:00:00Z")
which is not what I wanted.
What I want is to delete data before 1 November 2017.
Here's what works for me:
new Date("2017-11-01")
and it returns:
ISODate("2017-11-01T00:00:00Z")
Which is what I wanted.
This is because you are storing your data in a wrong format. You have a string an string
'15' is smaller than string '5'. Convert your strings in the beginning to date (read here how to use dates in mongo).
And only than you can use it to properly compare your dates:
db.coll.remove({
date:{
$lte : new Date(2012, 7, 14)
}
})
The reason for this is is your dates are strings.
So in a lexical sense when comparing strings "2014-3-5" is greater than "2014-3-15", as what is being compared is that "1" is less than "5".
Fix your dates as real ISO Dates, or you will forever have this problem.
Batch convert like this, assuming "year" "month" "day" in format:
db.eval(function(){
db.collection.find().forEach(function(doc) {
var d = doc.date.split("-");
var date = new Date(
"" + d[0] + "-" +
( d[1] <= 9 ) ? "0" + d[1] : d[1] + "-" +
( d[2] <= 9 ) ? "0" + d[2] : d[2]
);
db.collection.update(
{ "_id": doc._id },
{ "$set": { "date": date }
);
});
})
That makes sure you get the right dates on conversion.