I've create a mongoplayground to make it easier to understand.
https://mongoplayground.net/p/8zhkcMyLkmy
Basically, I'm stuck between line 108 ~ 111, because in MongoDB Compass the same aggregate pipeline doesn't work.
Here's the images:
If I just change '$$date' to its date:
Even if I insert the variable inside date function:
Does anyone have any thoughts in why it doesn't work in compass and works in mongoplayground?
For best practice, when comparing date string, it is required to convert as date first (via $toDate).
$lte: [ { $toDate: "$date" }, { $toDate: "$$date" }]
Comparison/Sort Order (MongoDB Manual) - String section
By default, MongoDB uses the simple binary comparison to compare strings.
Hence, comparing dates in string format will lead to an inaccurate result.
Related
I have a field called file_date, and in the start of the query I've set a new field called start_date which is 10 days back using:
{ $set: { start_date:{ $subtract: [ "$$NOW", 10*24*60*60*1000 ] } } }.
I want to use $match to find all the results where file_date is bigger then start_date.
I've tried using $gte but I can't seem to get the right syntax.
I'm using mongo 4.2 so I cant use SubstructDate
Thanks for any help
Changing the dates to strings using $toString.
using $gte
I have this query which isn't working and I've tried looking for material to better understand the syntax behind it, but I'm not having much luck. If someone could point out some resources for me to look into, I would appreciate it. Specifically, I'm having trouble with ISODate(:#{#start}) and what roles the :# or {#parameter} play in this query.
Also, if you happen to know how to fix this up. Please share!
Edit The dates in the db should be engulfed by the date parameters supplied. I originally had 'foo': :#{#parameter} which worked for looking up values, but here I've had to wrap the string with ISODate() in order to convert it for comparison.
#Query("
{ 'user.id': :#{#id} },
{ 'date.events.start': { $gte: ISODate(:#{#start}) } },
{ 'date.events.end': { $lte: ISODate(:#{#currentDate}) } }
")
List<Case> getUsersInPeriod(#Param("id") String id,
#Param("start") String start, #Param("currentDate") String currentDate);
Edit Playing around in Mongo Compass query box with the following:
{'user.id': '5df2b19006f31c190cc13288' }, { 'date.events.start': {$gte: ISODate('2020-02-13')} }, {'date.events.end': {$lte: ISODate('2020-02-31')}}
and it does not work as expected. So I am not sure what the issue is.
Sample Mongo document values:
start: "2020-01-20T08:30:00.000-06:00" (String)
end: "2020-01-20T15:30:00.000-06:00" (String)
Sample Comparison values:
start: "2020-01-16" (String)
end: "2020-01-31" (String)
Which didn't work, and so I wrapped the comparison values with ISODate() and that still didn't work, which now has me looking at document string values.
Edit: #3
I've converted some values in the document to Date and changed the query in Mongo Compass to:
{$and: [{'user.id': '5df2b19006f31c190cc13288' }, { 'date.events.start': {$gte: ISODate('2020-01-21')} }, {'date.events.end': {$lte: ISODate('2020-01-31')}}]}
which only picks up the document values formed as Date instead of String... so, I think I narrowed down the problem to two things (My original issue still persists with the syntax).
How do I deal with Mongo documents with string dates? Is it possible to parse the document string dates as ISODate while doing the query?
#prasad_ answer to formatting: can parse String or Date, but both must be of the same type. Ideally Date is used.
Edit #4: What I know so far...
User.id checks out. There's no issue there. I know the $and usage is correct.
Sample Document entries:
start: "2020-01-20T08:30:00.000-06:00" (String)
end: "2020-01-20T15:30:00.000-06:00" (String)
The above values should be engulfed by the following parameters:
start: 2020-01-16T19:57:54.949-06:00
end: 2020-01-31T23:59:59.999-06:00
I've converted both sets of strings to String or to Date and neither has returned results from the query made in my application; however ...
MongoDB Compass Community Query filter:
{'$and':[{'user.id':'5df2b19006f31c190cc13288'},{'date.events.start':{'$gte':'2020-01-22T08:30:00.000-06:00'}},{'date.events.end':{'$lte':'2020-01-31T15:30:00.000-06:00'}}]}
Does filter strings correctly, and...
{'$and':[{'user.id':'5df2b19006f31c190cc13288'},{'date.events.start':{'$gte':'2020-01-22T08:30:00.000-06:00'}},{'date.events.end':{'$lte':'2020-01-31T15:30:00.000-06:00'}}]}
Works when the document fields are of type Date and
start: 2020-01-21T14:30:00.000+00:00 (Date)
end: 2020-01-21T21:30:00.000+00:00 (Date)
Since I can get results in Mongo Compass Community, there must be something wrong with my application's query here:
#Query("{'$and': [{ 'user.id': :#{#id} }, { 'date.events.start': { '$gte': :#{#startPeriod} } }, { 'date.events.end': { '$lte': :#{#currentDate} } } ]}")
List<Case> getUsersInPeriod(#Param("id") String id, #Param("startPeriod") String startPeriod, #Param("currentDate") String currentDate);
The document entry is structured as:
date: (Object)
events: (Array)
[0]: (Object)
start: (String)
end: (String)
[1]: (Object)
(...)
(...)
Solution
I was able to find something that put me on the right path:
Internally, MongoDB can store dates as either Strings or as 64-bit integers. If you intend to do any operations using the MongoDB query or aggregate functions, or if you want to index your data by date, you'll likely want to store your dates as integers. If you're using the built-in "Date" data type, or a date wrapped in the ISODate() function, you're also storing your date as an integer.
https://www.compose.com/articles/understanding-dates-in-compose-mongodb/
So I changed everything to Date and now it's working as expected. Not sure what I had done wrong the first time I checked Date types, but oh well.
I still don't understand the syntax I originally asked, so if someone wants to help by providing something to read, please and thank you.
I want to make a mongodb view called "orders4H" that always returns documents from our "orders" collection that have a createdAt field value that is <= NOW-4hours. Then, any tool that is using the "orders4H" view as a collection source will always get the most current order data for the last 4 hours. The view creation requires an aggregation pipeline. However, I cannot figure out how to make a pipeline that will $match on a relative date.
In mongodb 4.2 they added an aggregation system variable called NOW which seems like it would be the thing to use, if only I could get it to work in a query document. I can use it to make new fields, for example, if I put timeNow: "$$NOW" in a $addFields stage, I get a new field with the current datetime. Now I want to use this "$$NOW" in a query evaluation.
Short version of question - how can I write a $match pipeline stage that will perform a server-time-relative comparison with a datetime field from the document?
MongoDB tickets imply that this support has been added ... but I cannot figure out how to use the $$NOW in a query. Anyone out there know how to do this?
Here are the relevant tickets that I found:
https://jira.mongodb.org/browse/SERVER-37713
https://jira.mongodb.org/browse/SERVER-23656
With help from mongodb tech support, I was able to get a $match stage that works with mongodb 4.2 to make a time-relative query:
[
{ $match:
{ $expr:
{ $gt: [
"$_created_at",
{ $subtract: [ "$$NOW", 4 * 60 * 60 * 1000] } ]
}
}
}
]
I am trying to perform an aggregation query on a collection. I want to divide 2 of the fields and check if this is less than a certain value.
The way this db was made, it has all the fields as strings, and this isn't something I can really change right now. I wondered if there was a way to cast the values as numbers, maybe using $let or something?
As it is I currently get the error
exception: $divide only supports numeric types, not String and String
This is an example of the sort of query I am trying to run:
db.myCollection.aggregate([{
"$project": {
"mins": {
"$divide": ["$ALOWOVRLTOTL", "$FEESCALLCOST"]}
}
},
"$match": {
"mins": {
"$lte": 40
}
}
}])
You won't be able to perform the divide operation on non-numerical values. A possible solution to this would be to run an update and parse the String values to a numerical value.
Have a look at this question:
how to convert string to numerical values in mongodb
I'm saving data into the bongo as bulk insert. The data that's an array of JSON object contain date, numeric, alphanumeric data all saved as string.
Sample Data
[{
"CARDNO": "661",
"HOLDERNO": "661",
"HOLDERNAME": "S",
"IODATE": "4/1/2012",
"IOTIME": "00:03:27",
"IOGATENO": "01-3",
"IOGATENAME": "FWork",
"IOSTATUS": "Entry",
"DEPARTMENTNO": "1",
"UPDATE_STATUS": "1"
}, {
"CARDNO": "711",
"HOLDERNO": "711",
"HOLDERNAME": "P",
"IODATE": "4/1/2012",
"IOTIME": "04:35:33",
"IOGATENO": "01-7",
"IOGATENAME": "FDWork",
"IOSTATUS": "Exit",
"DEPARTMENTNO": "3",
"UPDATE_STATUS": "1"
}]
My Query
var start = new Date(2012, 4, 15);
var end = new Date(2012, 4, 1);
collection.find({
"IODATE": {
$gte: start,
$lt: end
}
}).toArray(function (err, data) {
if (err) {
console.log(err);
} else {
console.log(data.length)
}
//res.send(data.length);
res.send(JSON.stringify(data));
});
It's not returning result, I think it is because the value of "IODATE" is in string inside db.
How to work around this issue? I may need to do bulk insert since the data can be of 200 million or so.
One last try at this, because you don't have a good record of accepting good advice.
Your date formats as they stand are going to bite you. Even where trying to work around them. Here are the problems:
The format is not lexical. Which means that even with a string comparison operators like $gte, $lte are just not going to work. A lexical date would be "2012-01-04" in "yyyy-mm-dd" format. That would work with the operators.
You could look at $substr (and it's complete lack of documentation, search on SO for real usage) within aggregate but your date format is lacking the double digit form of day and month ie "04/01/2012", so that is going to blow up the positional nature of the operator. Also you would have to transform before any $match which means you blow up any chance of reducing your pipeline input, so you are stuck with not being able to filter your large resultset by date.
It's a horrible case, but there really is no other practical solution to the large data problem here than to convert your dates. Strings in the form that you have just do not cut it. Either, in order of preference convert to:
BSON date
epoch timestamp as long (whatever)
Lexical string representation (as described)
Your main case seems to be filtering, so updating the dataset is the only pratical alternative. Otherwise you are stuck with "paging" results and doing a lot of manual work, that could otherwise be done server side.