I have a Spring API connecting to a MongoDB database. I am trying to use Spring's #Aggregation to find entries in a "Shipment" document which have a "shipDate" later than the date specified by the user.
Here is an example of my repository:
import org.springframework.data.mongodb.repository.Aggregation;
import org.springframework.data.mongodb.repository.MongoRepository;
import org.springframework.stereotype.Repository;
#Repository
public interface ShipmentRepository extends MongoRepository<Shipment, String> {
#Aggregation("{ '$match': { shipDate: {$gt: ISODate(?0)} } }")
int findShippedAfterDate(String date);
}
The value being passed as date is "2020-06-16". I am using #Aggregation instead of #Query because there will be a $group opperation following the $match once I get this portion working.
It seems that using ?0 inside ISODate() does not work properly and results in the following error:
JSON reader expected a string but found '?0'
I have tried the following syntax variations:
shipDate: {$gt: ISODate('?0')}
shipDate: {$gt: ISODate('$?0')}
Both result in the following error, which I believe is caused by ?0 not actually being replaced by the value:
org.bson.json.JsonParseException: Invalid date format.
I am confident the issue is with ?0 not being replaced by the string value because the query works as expected when the string is used instead of the variable place holder:
#Aggregation("{ '$match': { shipDate: {$gt: ISODate('2020-06-16')} } }")
Is there a simple syntax mistake I am making? Thanks for any help on this!
I was able to get this working by using $dateToString to convert the ISODate stored in MongoDB to a string that can be matched against the date provided by the user and then adding it to each entry using $addFields. However, I believe this method may have the following issue:
Inefficient because it will add a dateString field to every returned entry.
I'm am not certain that this will be an issue because it depends on how MongoDB handles $addFields when followed by $match. If it only adds the field to entries that are returned by the $match, it will be much more efficient.
Working #Aggregation example:
#Aggregation(pipeline = {
"{ '$addFields': { 'dateString': { $dateToString: { format: '%Y-%m-%d', date: '$shipDate'} } } }",
"{ '$match': { dateString: {$gte: ?0} } }" })
Related
I have a collection where the same column has Date and String format dates for different records, with the same ISO pattern like 2021-04-22T14:10:48.751779Z.
And I need to build a unified query for search, with such =, >, < options for search.
The second problem is an input search query has a different pattern ("2021-04-22")
So the question is:
can I write some kind of transformation of existing table data before the search, for example from Date to String, or in case I have a string value this transformation would not cause any exception.
In this case, I would be able to perform a search by $eq, $lt, $gt, and $regex query options.
I`m not considering casting strings to dates even if this is possible because my input did not have "14:10:48.751779Z" this part of the date and query would not find anything because of dates would not match. For this case, $regex looks like the only solution.
Or your any other suggestions would be considered.
Current query which not satisfies case when DB column is String type:
{
"aggregate":"collection_name",
"pipeline":[
{
"$match":{
"$and":[
{
"some_column":{
"$eq":"some_value"
}
},
{
"date_column_with_string_or_date_type":{
"$gt":{
"$date":"1980-01-01T00:00:00Z"
}
}
}
]
}
},
{
"$project":{
"_id":1
}
}
]
}
Solved by using $expr and $toString
This works with Date And String types of "date_column"
$match: {
$expr: {
$gt : [
{ $toString:"$date_column" },
"2021-04-22"
]
}
}
operator:
I have this query which isn't working and I've tried looking for material to better understand the syntax behind it, but I'm not having much luck. If someone could point out some resources for me to look into, I would appreciate it. Specifically, I'm having trouble with ISODate(:#{#start}) and what roles the :# or {#parameter} play in this query.
Also, if you happen to know how to fix this up. Please share!
Edit The dates in the db should be engulfed by the date parameters supplied. I originally had 'foo': :#{#parameter} which worked for looking up values, but here I've had to wrap the string with ISODate() in order to convert it for comparison.
#Query("
{ 'user.id': :#{#id} },
{ 'date.events.start': { $gte: ISODate(:#{#start}) } },
{ 'date.events.end': { $lte: ISODate(:#{#currentDate}) } }
")
List<Case> getUsersInPeriod(#Param("id") String id,
#Param("start") String start, #Param("currentDate") String currentDate);
Edit Playing around in Mongo Compass query box with the following:
{'user.id': '5df2b19006f31c190cc13288' }, { 'date.events.start': {$gte: ISODate('2020-02-13')} }, {'date.events.end': {$lte: ISODate('2020-02-31')}}
and it does not work as expected. So I am not sure what the issue is.
Sample Mongo document values:
start: "2020-01-20T08:30:00.000-06:00" (String)
end: "2020-01-20T15:30:00.000-06:00" (String)
Sample Comparison values:
start: "2020-01-16" (String)
end: "2020-01-31" (String)
Which didn't work, and so I wrapped the comparison values with ISODate() and that still didn't work, which now has me looking at document string values.
Edit: #3
I've converted some values in the document to Date and changed the query in Mongo Compass to:
{$and: [{'user.id': '5df2b19006f31c190cc13288' }, { 'date.events.start': {$gte: ISODate('2020-01-21')} }, {'date.events.end': {$lte: ISODate('2020-01-31')}}]}
which only picks up the document values formed as Date instead of String... so, I think I narrowed down the problem to two things (My original issue still persists with the syntax).
How do I deal with Mongo documents with string dates? Is it possible to parse the document string dates as ISODate while doing the query?
#prasad_ answer to formatting: can parse String or Date, but both must be of the same type. Ideally Date is used.
Edit #4: What I know so far...
User.id checks out. There's no issue there. I know the $and usage is correct.
Sample Document entries:
start: "2020-01-20T08:30:00.000-06:00" (String)
end: "2020-01-20T15:30:00.000-06:00" (String)
The above values should be engulfed by the following parameters:
start: 2020-01-16T19:57:54.949-06:00
end: 2020-01-31T23:59:59.999-06:00
I've converted both sets of strings to String or to Date and neither has returned results from the query made in my application; however ...
MongoDB Compass Community Query filter:
{'$and':[{'user.id':'5df2b19006f31c190cc13288'},{'date.events.start':{'$gte':'2020-01-22T08:30:00.000-06:00'}},{'date.events.end':{'$lte':'2020-01-31T15:30:00.000-06:00'}}]}
Does filter strings correctly, and...
{'$and':[{'user.id':'5df2b19006f31c190cc13288'},{'date.events.start':{'$gte':'2020-01-22T08:30:00.000-06:00'}},{'date.events.end':{'$lte':'2020-01-31T15:30:00.000-06:00'}}]}
Works when the document fields are of type Date and
start: 2020-01-21T14:30:00.000+00:00 (Date)
end: 2020-01-21T21:30:00.000+00:00 (Date)
Since I can get results in Mongo Compass Community, there must be something wrong with my application's query here:
#Query("{'$and': [{ 'user.id': :#{#id} }, { 'date.events.start': { '$gte': :#{#startPeriod} } }, { 'date.events.end': { '$lte': :#{#currentDate} } } ]}")
List<Case> getUsersInPeriod(#Param("id") String id, #Param("startPeriod") String startPeriod, #Param("currentDate") String currentDate);
The document entry is structured as:
date: (Object)
events: (Array)
[0]: (Object)
start: (String)
end: (String)
[1]: (Object)
(...)
(...)
Solution
I was able to find something that put me on the right path:
Internally, MongoDB can store dates as either Strings or as 64-bit integers. If you intend to do any operations using the MongoDB query or aggregate functions, or if you want to index your data by date, you'll likely want to store your dates as integers. If you're using the built-in "Date" data type, or a date wrapped in the ISODate() function, you're also storing your date as an integer.
https://www.compose.com/articles/understanding-dates-in-compose-mongodb/
So I changed everything to Date and now it's working as expected. Not sure what I had done wrong the first time I checked Date types, but oh well.
I still don't understand the syntax I originally asked, so if someone wants to help by providing something to read, please and thank you.
I want to obtain the records that the "FECHA_FIN" field is greater than or equal to today's date.
this is an example of my data:
but with this query:
db.getCollection('susp_programadas').find( {"FECHA_FIN":{ $gte: new Date("YYYY-mm-dd") }} )
I do not get results, what am I doing wrong? Thank you
You can convert the date to an ISO date and query that way. Since you stored the date as a string mongo has no idea how to query it against an ISO date without conversion.
If you stored your date in mongo as the default ISO date then you could have easily done this:
db.getCollection('susp_programadas').find({"FECHA_FIN":{$gte: new Date()}})
So this is how you can do it now:
db.getCollection('susp_programadas').aggregate([
{
$project: {
date: { $dateFromString: { dateString: '$FECHA_FIN' }}
}
},
{ $match: { date: { $gte: new Date() }}}
])
You can use the $dateFromString in an aggregate query with a $match to get the results you want. Note that $dateFromString is only available in MongoDB version 3.6 and up.
If there is no way to convert your data to ISODate or upgrade your DB you could also consider another solution which via $where:
db.getCollection('susp_programadas').find({
$where: function(){
return new Date(this.FECHA_FIN) > Date.now()
}
})
However that solution suffers from the fact that $where can not use indexes so have that in mind.
I am trying to use make my own query for a mongo Repository:
#Repository
public interface LogEntryRepository extends MongoRepository<LogEntry,String> {
#Query("{'created_at' : {{ $gte: ISODate(?0)},{$lt: ISODate(?1)}}, " +
"$or: [{'site': {$regex: ?2}}, {'login': {$regex: ?2}}, {'ip': {$regex: ?2}} ]" +
"}")
public Page<LogEntry> findByDateTimeBetweenAndCriteria(String isoStartDate, String isoEndDate, String searchTerm, Pageable page);
}
What I'd like to achieve is searching though dated logs with a keyword. The above complains about a parse error:
Caused by: com.mongodb.util.JSONParseException:
{'created_at' : { $gte: ISODate("_param_0"), $lt: ISODate("_param_1")}, $or: [{'site': {$regex: "_param_2"}}, {'login': {$regex: "_param_2"}}, {'ip': {$regex: "_param_2"}} ]}
^
If I replace the ISODate(?0) with simply ?0 it produces Page 1 of 0 containing UNKNOWN instances
The Strings isoStartDate & isoEndDate are produced from java.util.Date and look like this 2017-06-27T00:00:00.000Z
How do I get my date in there?
ISODate is a Mongo shell construct to create a BSON date and definitely not valid JSON, which is what I believe your error is complaining about.
Try replacing the above ISODate calls with { '$date' : '?0' } and { '$date' : '?1' } as suggested in this answer. All the strings should probably need to be surrounded in single quotes.
I'm trying to use expression languge to generate ISO string in Nifi getMongo Query field using following query,
{
"remindmeDate": {
"$gte": "${now():format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}",
"$lte": "${now():toNumber():plus(359999):format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}"
}
}
But i'm getting invalid JSON error error as double quotes are not escaped. When we try to escape it using \ operator, nifi is not evaluating the expression language. Is there any method or workaround to get this working ?
Thanks in advance
GetMongo processor of nifi requires your query to be in extended json format of mongo.So you can use query of below format to query mongo based on datetime:
{"bday":{"$gt":{"$date":"2014-01-01T05:00:00.000Z"}, "$lt" :{"$date":"2019-01-
01T05:00:00.000Z"}}}
I used your not changed expression in UpdateAttribute processor to evaluate new flowFile attribute.
your expression:
{
"remindmeDate": {
"$gte": "${now():format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}",
"$lte": "${now():toNumber():plus(359999):format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}"
}
}
the result:
{
"remindmeDate": {
"$gte": "2017-06-16T07:38:04.811Z",
"$lte": "2017-06-16T07:44:04.810Z"
}
}
and this is a correct json object.
Finally I found that GetMongo.Query property does not support nifi expression language (nifi 1.2.0 and 1.3.0). Just hover the question mark near parameter.
It means no way to build dynamic query (
Seems need to register an issue... https://issues.apache.org/jira/browse/NIFI-4082
But it's possible to specify current and relative date in mongo query language. something like this:
{
"remindmeDate": {
"$gte": new Date(),
"$lte": new Date(ISODate().getTime() + 359999)
}
}
Nifi's getMongo Query field doesnt support EL. So i created a stored function in MongoDB for my dynamic query and called it from Nifi.
{
"_id" : "reminderDateGMT",
"value" : function (reminderDateGMT) {
var reminder = new Date(reminderDateGMT)
var fromDate = new Date();
var toDate = new Date(new Date().getTime()+(1000 * 60 * 60));
if ((reminder >= fromDate) && (reminder <=toDate )) {
return true;
} else {
return false;
}
}
}
In nifi GetMongo Query,
{
"$where": "reminderDateGMT(this.reminderDateGMT)"
}
I think you may be able to use the unescapeJson expression language function to handle this. You have to provide valid JSON (escaped quotes) for the field level (PropertyDescriptor in NiFi parlance) validation, but the expression language string expects unescaped JSON during expression parsing, so the unescapeJson function removes the escapes first and then format receives a properly quoted string.
{
"remindmeDate": {
"$gte": "${now():format(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\":unescapeJson(),'GMT')}",
"$lte": "${now():toNumber():plus(359999):format(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\":unescapeJson(),'GMT')}"
}
}
I had a similar discussion on the mailing list, and here is the solution I found that works:
Mongo console:
db.system.js.save({
"_id": "lastFiveMinutes",
"value": function() {
return new Date(ISODate().getTime() - (1000 * 60 * 5));
}
});
db.loadServerScripts();
Query field:
{
"$where": "obj.ts >= lastFiveMinutes()"
}
Note: you probably want to set this on a timer in the scheduling property.
I know this is pretty old post, but I spent lot many hours and found a solution which worked for me.
Use UpdateAttribute Processor and created two attributes to calculate the date range, I need to fetch the mongo documents :
startDate: "${now():format('yyyy-MM-dd')}"
endDate : "${now():toNumber():plus(86400000):format('yyyy-MM-dd')}"
enter image description here
After that pass these attributes to GetMongo processor:
Query : {"createdDate":{"$gte":ISODate(${startDate}), "$lt":ISODate(${endDate})}}