How to get ISO string in Nifi getMongo Query Field - mongodb

Question

I'm trying to use expression languge to generate ISO string in Nifi getMongo Query field using following query,
{
"remindmeDate": {
"$gte": "${now():format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}",
"$lte": "${now():toNumber():plus(359999):format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}"
}
}
But i'm getting invalid JSON error error as double quotes are not escaped. When we try to escape it using \ operator, nifi is not evaluating the expression language. Is there any method or workaround to get this working ?
Thanks in advance

Answer 1

GetMongo processor of nifi requires your query to be in extended json format of mongo.So you can use query of below format to query mongo based on datetime:
{"bday":{"$gt":{"$date":"2014-01-01T05:00:00.000Z"}, "$lt" :{"$date":"2019-01-
01T05:00:00.000Z"}}}

Answer 2

I used your not changed expression in UpdateAttribute processor to evaluate new flowFile attribute.
your expression:
{
"remindmeDate": {
"$gte": "${now():format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}",
"$lte": "${now():toNumber():plus(359999):format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}"
}
}
the result:
{
"remindmeDate": {
"$gte": "2017-06-16T07:38:04.811Z",
"$lte": "2017-06-16T07:44:04.810Z"
}
}
and this is a correct json object.
Finally I found that GetMongo.Query property does not support nifi expression language (nifi 1.2.0 and 1.3.0). Just hover the question mark near parameter.
It means no way to build dynamic query (
Seems need to register an issue... https://issues.apache.org/jira/browse/NIFI-4082
But it's possible to specify current and relative date in mongo query language. something like this:
{
"remindmeDate": {
"$gte": new Date(),
"$lte": new Date(ISODate().getTime() + 359999)
}
}

Answer 3

Nifi's getMongo Query field doesnt support EL. So i created a stored function in MongoDB for my dynamic query and called it from Nifi.
{
"_id" : "reminderDateGMT",
"value" : function (reminderDateGMT) {
var reminder = new Date(reminderDateGMT)
var fromDate = new Date();
var toDate = new Date(new Date().getTime()+(1000 * 60 * 60));
if ((reminder >= fromDate) && (reminder <=toDate )) {
return true;
} else {
return false;
}
}
}
In nifi GetMongo Query,
{
"$where": "reminderDateGMT(this.reminderDateGMT)"
}

Answer 4

I think you may be able to use the unescapeJson expression language function to handle this. You have to provide valid JSON (escaped quotes) for the field level (PropertyDescriptor in NiFi parlance) validation, but the expression language string expects unescaped JSON during expression parsing, so the unescapeJson function removes the escapes first and then format receives a properly quoted string.
{
"remindmeDate": {
"$gte": "${now():format(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\":unescapeJson(),'GMT')}",
"$lte": "${now():toNumber():plus(359999):format(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\":unescapeJson(),'GMT')}"
}
}

Answer 5

I had a similar discussion on the mailing list, and here is the solution I found that works:
Mongo console:
db.system.js.save({
"_id": "lastFiveMinutes",
"value": function() {
return new Date(ISODate().getTime() - (1000 * 60 * 5));
}
});
db.loadServerScripts();
Query field:
{
"$where": "obj.ts >= lastFiveMinutes()"
}
Note: you probably want to set this on a timer in the scheduling property.

Answer 6

I know this is pretty old post, but I spent lot many hours and found a solution which worked for me.
Use UpdateAttribute Processor and created two attributes to calculate the date range, I need to fetch the mongo documents :
startDate: "${now():format('yyyy-MM-dd')}"
endDate : "${now():toNumber():plus(86400000):format('yyyy-MM-dd')}"
enter image description here
After that pass these attributes to GetMongo processor:
Query : {"createdDate":{"$gte":ISODate(${startDate}), "$lt":ISODate(${endDate})}}