I am trying to query a row based on a date field in elasticsearch (not the range), using the lucene syntax but not getting the expected response,
I checked the mapping using _mapping it gives me the following result for the index, for the field ordered at:
ordered_at: {
type: "date",
format: "dateOptionalTime"
}
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-date-format.html#date-math
and
http://joda-time.sourceforge.net/api-release/org/joda/time/format/ISODateTimeFormat.html#dateOptionalTimeParser()
suggests that the following format should be acceptable yyyy-mm-dd (2015-01-03).
Following are some of the formats i have tried to fetch the result:
_search?pretty=true&q=ordered_at:"2015\-01\-03"
_search?pretty=true&q=ordered_at:2015\-01\-03*
_search?pretty=true&q=ordered_at:2015-01-03
_search?pretty=true&q=ordered_at:[2015\-01\-03 TO 15\-01\-20]
_search?pretty=true&q=ordered_at:[2015-01-03 TO 15-01-20]
_search?pretty=true&q=ordered_at:[20150103 TO 150120]
_search?pretty=true&q=ordered_at:01/03/2015
_search?pretty=true&q=ordered_at:"20150103"
_search?pretty=true&q=ordered_at:20150103
_search?pretty=true&q=ordered_at:"2015-01-03T18:53:37.000Z" (this does work but i need to just submit date and not time)
_search?pretty=true&q=ordered_at:"2015-01-03"
_search?pretty=true&q=ordered_at:"2015-01-03*"
_search?pretty=true&q=ordered_at:2015-01-03*
_search?pretty=true&q=ordered_at:2015/01/03
_search?pretty=true&q=ordered_at:2015\01\03
_search?pretty=true&q=ordered_at:2015-01-03
_search?pretty=true&q=ordered_at:2015\-01\-03
_search?pretty=true&q=ordered_at:2015-01-03*
_search?pretty=true&q=ordered_at:2015/-01/-03
_search?pretty=true&q=ordered_at:"2015-03-01"
_search?pretty=true&q=ordered_at:2015/03/01
_search?pretty=true&q=ordered_at:"2015-01-03"*
Following is a sample of how the data is displayed when we query by other fields:
ordered_at: [
"2015-01-03T18:53:37.000Z"
]
No idea what am i missing, any clue? Thanks..
Try this: q=date:[2015-01-03 TO 2015-01-20].
Related
So am fetching data with mongoose and i would like to modify the data like apply some date formats. Currently i have
const count = await UserModel.countDocuments();
const rows = await UserModel.find({ name:{$regex: search, $options: 'i'}, status:10 })
.sort([["updated_at", -1]])
.skip(page * perPage)
.limit(perPage)
.exec();
res.json({ count, rows });
The above UserModel is a mongoose model
I would like to modify some of objects like applying date formats before the data is returned while still paginating as above.
Currently i have added the following which works but i have to loop through all rows which will be a performance nighmare for large data.
res.json({ count, rows:rows.map(el=>({...el,created_at:'format date here'})) });
Is there a better option
As much as I understood your question, If you need to apply some date formats before showing data on frontend, you just need to pass the retrieved date in a date-formating library before displaying it, like in JS:
const d = new Date("2015-03-25T12:00:00Z");
However, if you want to get date in formatted form, than you must format it before storing. I hope that answer your question.
I think the warning from #Fabian Strathaus in the comments is an important consideration. I would strongly recommend that the approach you are trying to solve sets you up for success overall as opposed to introducing new pain points elsewhere with your project.
Assuming that you want to do this, an alternative approach is to ask the database to do this directly. More specifically, the $dateToString operator sounds like it could be of use here. This playground example demonstrates the basic behavior by adding a formatted date field which will be returned directly from the database. It takes the following document:
{
_id: 1,
created_at: ISODate("2022-01-15T08:15:39.736Z")
}
We then execute this sample aggregation:
db.collection.aggregate([
{
"$addFields": {
created_at_formatted: {
$dateToString: {
format: "%m/%d/%Y",
date: "$created_at"
}
}
}
}
])
The document that gets returned is:
{
"_id": 1,
"created_at": ISODate("2022-01-15T08:15:39.736Z"),
"created_at_formatted": "01/15/2022"
}
You could make use of this in a variety of ways, such as by creating and querying a view which will automatically create and return this formatted field.
I also want to comment on this statement that you made:
Currently i have added the following which works but i have to loop through all rows which will be a performance nighmare for large data.
It's good to hear that you're thinking about performance upfront. That said, your query includes a query predicate of name:{$regex: search, $options: 'i'}. Unanchored and/or case insensitive regex filters cannot use indexes efficiently. So if your status predicate is not selective, then you may need to take a look at alternative approaches for filtering on name to make sure that the query is performant.
I have a string field which consists of date, I am trying to run following command mentioned in Update string to Date object in mongodb
db.getCollection('rft').updateMany(
{},[{ "$set": { "createdate": { "$toDate": "$create_date" } }}]
);
Getting an error:
Error: Failed to execute script.
Error: the update operation document must contain atomic operators
Details:
DBCollection.prototype.updateMany#src/mongo/shell/crud_api.js:625:1
#(shell):1:1
Can someone please help in updating the records to get new field with date time.
Your query seems to work as expected , check playgroud unless something different is expected ...
I have this query which isn't working and I've tried looking for material to better understand the syntax behind it, but I'm not having much luck. If someone could point out some resources for me to look into, I would appreciate it. Specifically, I'm having trouble with ISODate(:#{#start}) and what roles the :# or {#parameter} play in this query.
Also, if you happen to know how to fix this up. Please share!
Edit The dates in the db should be engulfed by the date parameters supplied. I originally had 'foo': :#{#parameter} which worked for looking up values, but here I've had to wrap the string with ISODate() in order to convert it for comparison.
#Query("
{ 'user.id': :#{#id} },
{ 'date.events.start': { $gte: ISODate(:#{#start}) } },
{ 'date.events.end': { $lte: ISODate(:#{#currentDate}) } }
")
List<Case> getUsersInPeriod(#Param("id") String id,
#Param("start") String start, #Param("currentDate") String currentDate);
Edit Playing around in Mongo Compass query box with the following:
{'user.id': '5df2b19006f31c190cc13288' }, { 'date.events.start': {$gte: ISODate('2020-02-13')} }, {'date.events.end': {$lte: ISODate('2020-02-31')}}
and it does not work as expected. So I am not sure what the issue is.
Sample Mongo document values:
start: "2020-01-20T08:30:00.000-06:00" (String)
end: "2020-01-20T15:30:00.000-06:00" (String)
Sample Comparison values:
start: "2020-01-16" (String)
end: "2020-01-31" (String)
Which didn't work, and so I wrapped the comparison values with ISODate() and that still didn't work, which now has me looking at document string values.
Edit: #3
I've converted some values in the document to Date and changed the query in Mongo Compass to:
{$and: [{'user.id': '5df2b19006f31c190cc13288' }, { 'date.events.start': {$gte: ISODate('2020-01-21')} }, {'date.events.end': {$lte: ISODate('2020-01-31')}}]}
which only picks up the document values formed as Date instead of String... so, I think I narrowed down the problem to two things (My original issue still persists with the syntax).
How do I deal with Mongo documents with string dates? Is it possible to parse the document string dates as ISODate while doing the query?
#prasad_ answer to formatting: can parse String or Date, but both must be of the same type. Ideally Date is used.
Edit #4: What I know so far...
User.id checks out. There's no issue there. I know the $and usage is correct.
Sample Document entries:
start: "2020-01-20T08:30:00.000-06:00" (String)
end: "2020-01-20T15:30:00.000-06:00" (String)
The above values should be engulfed by the following parameters:
start: 2020-01-16T19:57:54.949-06:00
end: 2020-01-31T23:59:59.999-06:00
I've converted both sets of strings to String or to Date and neither has returned results from the query made in my application; however ...
MongoDB Compass Community Query filter:
{'$and':[{'user.id':'5df2b19006f31c190cc13288'},{'date.events.start':{'$gte':'2020-01-22T08:30:00.000-06:00'}},{'date.events.end':{'$lte':'2020-01-31T15:30:00.000-06:00'}}]}
Does filter strings correctly, and...
{'$and':[{'user.id':'5df2b19006f31c190cc13288'},{'date.events.start':{'$gte':'2020-01-22T08:30:00.000-06:00'}},{'date.events.end':{'$lte':'2020-01-31T15:30:00.000-06:00'}}]}
Works when the document fields are of type Date and
start: 2020-01-21T14:30:00.000+00:00 (Date)
end: 2020-01-21T21:30:00.000+00:00 (Date)
Since I can get results in Mongo Compass Community, there must be something wrong with my application's query here:
#Query("{'$and': [{ 'user.id': :#{#id} }, { 'date.events.start': { '$gte': :#{#startPeriod} } }, { 'date.events.end': { '$lte': :#{#currentDate} } } ]}")
List<Case> getUsersInPeriod(#Param("id") String id, #Param("startPeriod") String startPeriod, #Param("currentDate") String currentDate);
The document entry is structured as:
date: (Object)
events: (Array)
[0]: (Object)
start: (String)
end: (String)
[1]: (Object)
(...)
(...)
Solution
I was able to find something that put me on the right path:
Internally, MongoDB can store dates as either Strings or as 64-bit integers. If you intend to do any operations using the MongoDB query or aggregate functions, or if you want to index your data by date, you'll likely want to store your dates as integers. If you're using the built-in "Date" data type, or a date wrapped in the ISODate() function, you're also storing your date as an integer.
https://www.compose.com/articles/understanding-dates-in-compose-mongodb/
So I changed everything to Date and now it's working as expected. Not sure what I had done wrong the first time I checked Date types, but oh well.
I still don't understand the syntax I originally asked, so if someone wants to help by providing something to read, please and thank you.
got a problem with the elastic date format conversion when I parse the results from a query. So i have a default mapping on a date field as following:
"timestamp" : {
"type" : "date",
"format" : "dateOptionalTime"
}
and it is stored as "timestamp":"2015-05-06T08:52:56.387Z"
if I execute a max aggregation on that field I get a long value:
"timestamp_max": {
"value": 1430902071110
}
however I want the value be the same as it is stored. I read that one can specify the format in the aggregation but its not working. I tried:
"aggregations":{
"timestamp_max":{
"max":{
"field":"timestamp",
"format" : "dateOptionalTime"
}
}
}
but this gives a SearchParseException ... SearchParseException[[logstash-2015.05.07][0]: query[ConstantScore(BooleanFilter(+no_cache(timestamp:[1429357190515 TO 1431949190515])))],from[-1],size[-1]: Parse Failure [Unexpected token VALUE_STRING in [timestamp_max].]]; ...
What am I doing wrong?
Best regards,
Jan
You're almost there. You just need to specify the date format using the correct formatting pattern like this:
"aggregations":{
"timestamp_max":{
"max":{
"field":"timestamp",
"format" : "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
}
}
}
Please note that this is only working from ES 1.5.0 onwards. See the related issue on the ES github.
I have a collection ('stops') with 14000 records imported from a CSV. Included in each record is some geospatial data, which I've converted into an array, so each essentially looks like this:
{
_id: ...,
// other fields
"loc": [
153.019073,
-27.467834
]
}
When I run db.stops.ensureIndex({ loc: '2d' }) it gives the error:
location object expected, location array not in correct format
I guess that something's wrong in one of the fields, but I can't figure out which one. Any ideas?
The problem was that there was a single record which wasn't imported correctly (probably an empty line at the end of the file).
To find it (and later remove it), I used the $where operator:
db.stops.find({ $where: 'typeof this.loc[0] != "number"' })