I have my ISODate in mongo as ISODate and I want to just that in string format with a specific datetime format.
Here is the ISODate:
ISODate("2020-04-24T11:41:47.280Z")
Expected Result:
"2020-04-24T11:41:47.280Z"
I want this to be happened on mongodb only as many of my services are expecting in this format, and I don't want to make changes in all services as its a tedious job.
I got the expected result while i was trying the following.
ISODate("2020-04-24T11:41:47.280Z").toJSON()
This will give me back the string
"2020-04-24T11:41:47.280Z"
perhaps simply convert a date into string? this has less to do with mongo than js. moment is awesome but unusable in a mongo shell script.
db.events.find().forEach(function(doc) {
// printjson ("Document is " + doc);
var isoDate = doc.t; // t is correct key?
var isoString = isoDate.toISOString()
// update the collection with string using a new key
db.events.update(
{"_id":doc._id},
{
$set: {"iso_str":isoString}
}
);
// or overwrite using 't' key db.events.update({"_id":doc._id},{$set:{"t":isoString}});
})
I just came across this issue. There isn't anything built into ISODate turns out. So I'm converting the ISODate to JSON text, and then I do a substring to get the part I want. You can also use method on ISODate to get year, month, date separately and then combine them.
function formatDate(isoDate){
return isoDate.toJSON().substr(9, 20);
}
I assume that what you need is the dateToString function
https://docs.mongodb.com/manual/reference/operator/aggregation/dateToString/
If you have mixed data types[string & ISODate()] in an attribute & still want to normalize them to one data type e.g., 'string' then
typeof attr_val === 'string' ? attr_val : attr_val.toJSON()
References:
#manu answer
Typechecking
typeof
db.inventries.find().forEach(function (doc) {
let mydate = new Date(doc.date).toISOString();
db.inventries.update({ "_id": doc._id }, { $set: { "date": ISODate(mydate) } })
})
Related
I have this query which isn't working and I've tried looking for material to better understand the syntax behind it, but I'm not having much luck. If someone could point out some resources for me to look into, I would appreciate it. Specifically, I'm having trouble with ISODate(:#{#start}) and what roles the :# or {#parameter} play in this query.
Also, if you happen to know how to fix this up. Please share!
Edit The dates in the db should be engulfed by the date parameters supplied. I originally had 'foo': :#{#parameter} which worked for looking up values, but here I've had to wrap the string with ISODate() in order to convert it for comparison.
#Query("
{ 'user.id': :#{#id} },
{ 'date.events.start': { $gte: ISODate(:#{#start}) } },
{ 'date.events.end': { $lte: ISODate(:#{#currentDate}) } }
")
List<Case> getUsersInPeriod(#Param("id") String id,
#Param("start") String start, #Param("currentDate") String currentDate);
Edit Playing around in Mongo Compass query box with the following:
{'user.id': '5df2b19006f31c190cc13288' }, { 'date.events.start': {$gte: ISODate('2020-02-13')} }, {'date.events.end': {$lte: ISODate('2020-02-31')}}
and it does not work as expected. So I am not sure what the issue is.
Sample Mongo document values:
start: "2020-01-20T08:30:00.000-06:00" (String)
end: "2020-01-20T15:30:00.000-06:00" (String)
Sample Comparison values:
start: "2020-01-16" (String)
end: "2020-01-31" (String)
Which didn't work, and so I wrapped the comparison values with ISODate() and that still didn't work, which now has me looking at document string values.
Edit: #3
I've converted some values in the document to Date and changed the query in Mongo Compass to:
{$and: [{'user.id': '5df2b19006f31c190cc13288' }, { 'date.events.start': {$gte: ISODate('2020-01-21')} }, {'date.events.end': {$lte: ISODate('2020-01-31')}}]}
which only picks up the document values formed as Date instead of String... so, I think I narrowed down the problem to two things (My original issue still persists with the syntax).
How do I deal with Mongo documents with string dates? Is it possible to parse the document string dates as ISODate while doing the query?
#prasad_ answer to formatting: can parse String or Date, but both must be of the same type. Ideally Date is used.
Edit #4: What I know so far...
User.id checks out. There's no issue there. I know the $and usage is correct.
Sample Document entries:
start: "2020-01-20T08:30:00.000-06:00" (String)
end: "2020-01-20T15:30:00.000-06:00" (String)
The above values should be engulfed by the following parameters:
start: 2020-01-16T19:57:54.949-06:00
end: 2020-01-31T23:59:59.999-06:00
I've converted both sets of strings to String or to Date and neither has returned results from the query made in my application; however ...
MongoDB Compass Community Query filter:
{'$and':[{'user.id':'5df2b19006f31c190cc13288'},{'date.events.start':{'$gte':'2020-01-22T08:30:00.000-06:00'}},{'date.events.end':{'$lte':'2020-01-31T15:30:00.000-06:00'}}]}
Does filter strings correctly, and...
{'$and':[{'user.id':'5df2b19006f31c190cc13288'},{'date.events.start':{'$gte':'2020-01-22T08:30:00.000-06:00'}},{'date.events.end':{'$lte':'2020-01-31T15:30:00.000-06:00'}}]}
Works when the document fields are of type Date and
start: 2020-01-21T14:30:00.000+00:00 (Date)
end: 2020-01-21T21:30:00.000+00:00 (Date)
Since I can get results in Mongo Compass Community, there must be something wrong with my application's query here:
#Query("{'$and': [{ 'user.id': :#{#id} }, { 'date.events.start': { '$gte': :#{#startPeriod} } }, { 'date.events.end': { '$lte': :#{#currentDate} } } ]}")
List<Case> getUsersInPeriod(#Param("id") String id, #Param("startPeriod") String startPeriod, #Param("currentDate") String currentDate);
The document entry is structured as:
date: (Object)
events: (Array)
[0]: (Object)
start: (String)
end: (String)
[1]: (Object)
(...)
(...)
Solution
I was able to find something that put me on the right path:
Internally, MongoDB can store dates as either Strings or as 64-bit integers. If you intend to do any operations using the MongoDB query or aggregate functions, or if you want to index your data by date, you'll likely want to store your dates as integers. If you're using the built-in "Date" data type, or a date wrapped in the ISODate() function, you're also storing your date as an integer.
https://www.compose.com/articles/understanding-dates-in-compose-mongodb/
So I changed everything to Date and now it's working as expected. Not sure what I had done wrong the first time I checked Date types, but oh well.
I still don't understand the syntax I originally asked, so if someone wants to help by providing something to read, please and thank you.
I have a collection in which each document has a time field with value stored as similar to "21-Dec-2017".
I want to convert this to ISODate using projection.
My Query:
db.getCollection('orders').aggregate([{
$project:{time : {$add : new Date("$time")}}
}])
But this is returning me ISODate("1970-01-01T00:00:00.000Z") always.
you can try this,
db.getCollection('orders').aggregate([{
$project: {
time: {
$dateToString: {
format: "%d-%m-%G",
date: new Date("$time")
}
}
}
}
])
there is no any string function to get months name eg.Jan,Feb..Dec.
but you can refer https://docs.mongodb.com/manual/reference/operator/aggregation/dateToString/
to more information.
There is no problem in this ISODate("1970-01-01T00:00:00.000Z") format.
you should store date in ISO format but change format on client side according to you.
Basically you want to show date in dd/mm/yy format.
You can use http://momentjs.com/ to show date according to you.
I have a Mongo collection that stored a date field as a string. I ran the query below to convert this field into an ISODate, but it failed. However, there are quite a few records that now show as an ISODate.
How do I query to find all documents that are an ISODate format and all documents that are still a string?
db.TestCollection.find({}).forEach(function(doc) {
doc.LastUpdated = new Date(doc.LastUpdated );
db.TestCollection.save(doc);
});
The original question has been answered, but the follow up question is: Is there a way to aggregate the type of data present in the column? I would like to return a count of each type of data in the column. I have seen type string and type date, but want to verify there isn't any other data types.
Here's roughly what I have tried:
db.TestCollection.aggregate([{
$project: {
LastUpdated : { $type: "$LastUpdated " }
}
}])
I'm trying to use expression languge to generate ISO string in Nifi getMongo Query field using following query,
{
"remindmeDate": {
"$gte": "${now():format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}",
"$lte": "${now():toNumber():plus(359999):format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}"
}
}
But i'm getting invalid JSON error error as double quotes are not escaped. When we try to escape it using \ operator, nifi is not evaluating the expression language. Is there any method or workaround to get this working ?
Thanks in advance
GetMongo processor of nifi requires your query to be in extended json format of mongo.So you can use query of below format to query mongo based on datetime:
{"bday":{"$gt":{"$date":"2014-01-01T05:00:00.000Z"}, "$lt" :{"$date":"2019-01-
01T05:00:00.000Z"}}}
I used your not changed expression in UpdateAttribute processor to evaluate new flowFile attribute.
your expression:
{
"remindmeDate": {
"$gte": "${now():format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}",
"$lte": "${now():toNumber():plus(359999):format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}"
}
}
the result:
{
"remindmeDate": {
"$gte": "2017-06-16T07:38:04.811Z",
"$lte": "2017-06-16T07:44:04.810Z"
}
}
and this is a correct json object.
Finally I found that GetMongo.Query property does not support nifi expression language (nifi 1.2.0 and 1.3.0). Just hover the question mark near parameter.
It means no way to build dynamic query (
Seems need to register an issue... https://issues.apache.org/jira/browse/NIFI-4082
But it's possible to specify current and relative date in mongo query language. something like this:
{
"remindmeDate": {
"$gte": new Date(),
"$lte": new Date(ISODate().getTime() + 359999)
}
}
Nifi's getMongo Query field doesnt support EL. So i created a stored function in MongoDB for my dynamic query and called it from Nifi.
{
"_id" : "reminderDateGMT",
"value" : function (reminderDateGMT) {
var reminder = new Date(reminderDateGMT)
var fromDate = new Date();
var toDate = new Date(new Date().getTime()+(1000 * 60 * 60));
if ((reminder >= fromDate) && (reminder <=toDate )) {
return true;
} else {
return false;
}
}
}
In nifi GetMongo Query,
{
"$where": "reminderDateGMT(this.reminderDateGMT)"
}
I think you may be able to use the unescapeJson expression language function to handle this. You have to provide valid JSON (escaped quotes) for the field level (PropertyDescriptor in NiFi parlance) validation, but the expression language string expects unescaped JSON during expression parsing, so the unescapeJson function removes the escapes first and then format receives a properly quoted string.
{
"remindmeDate": {
"$gte": "${now():format(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\":unescapeJson(),'GMT')}",
"$lte": "${now():toNumber():plus(359999):format(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\":unescapeJson(),'GMT')}"
}
}
I had a similar discussion on the mailing list, and here is the solution I found that works:
Mongo console:
db.system.js.save({
"_id": "lastFiveMinutes",
"value": function() {
return new Date(ISODate().getTime() - (1000 * 60 * 5));
}
});
db.loadServerScripts();
Query field:
{
"$where": "obj.ts >= lastFiveMinutes()"
}
Note: you probably want to set this on a timer in the scheduling property.
I know this is pretty old post, but I spent lot many hours and found a solution which worked for me.
Use UpdateAttribute Processor and created two attributes to calculate the date range, I need to fetch the mongo documents :
startDate: "${now():format('yyyy-MM-dd')}"
endDate : "${now():toNumber():plus(86400000):format('yyyy-MM-dd')}"
enter image description here
After that pass these attributes to GetMongo processor:
Query : {"createdDate":{"$gte":ISODate(${startDate}), "$lt":ISODate(${endDate})}}
I have a simple Meteor collection and I am trying to insert a document that has a momentjs property into it. So I do:
docId = Col.insert({m: moment()});
However, when I try to get this document back with
doc = Col.findOne({_id: docId})
I get "Invalid date" for doc.m like so:
Object {_id: "wnHzTpHHxMSyMxmu3", m: "Invalid date"}
Anyone?!
I strongly recommend storing dates as Date objects and using moment to format them after they are fetched. For example:
Posts.insert({message: 'hello', createdAt: new Date});
Then later when you want to display the date:
var date = Posts.findOne().createdAt;
moment(date).format('MMMM DD, YYYY');
Moments are not designed to be directly serializable. They won't survive a round-trip to/from JSON. The best approach would be to serialize an ISO8601 formatted date, such as with moment().toISOString() or moment().format(). (toISOString is prefered, but will store at UTC instead of local+offset).
Then later, you can parse that string with moment(theString) and do what you want with it from there.
David's answer is also correct. Though it will rely on whatever Metor's default mechanism is for serializing Date objects, as Date also cannot exist directly in JSON. I don't know the specifics of Meteor - but chances are it's either storing an integer timestamp or just using Date.toString(). The ISO8601 format is much better suited for JSON than either of those.
UPDATE
I just took a glance at the docs for Meteor, which explain that they use an invented format called "EJSON". (You probably know this, but it's new to me.)
According to these docs, a Date is serialized as an integer timestamp:
{
"d": {"$date": 1358205756553}
}
So - David's answer is spot on (and should remain the accepted answer). But also, if you are doing something other than just getting the current date/time, then you might want to use moment for that. You can use yourMoment.toDate() and pass that so Meteor will treat it with the $date type in it's EJSON format.
If you would like to have moment objects on find and findOne, save it as a date then transform it on finding it. For example:
Posts = new Mongo.Collection('posts', {
transform: function (doc) {
Object.keys(doc).forEach(field => {
if (doc[field] instanceof Date) {
doc[field] = moment(doc[field]);
}
});
return doc;
}
});
Posts.insert({
title: 'Hello',
createdAt: new Date()
});