assuming I have the following nested document structure, where my document contains nested routes with an array of date time values.
{
property_1: ...,
routes: [
{
start_id: 1,
end_id: 2,
execution_times: ['2016-08-28T11:11:47+02:00', ...]
}
]
}
Now I could filter my documents that match certain execution_times with something like this.
query: {
filtered: {
query: {
match_all: { }
},
filter: {
nested: {
path: 'routes',
filter: {
bool: {
must: [
{
terms: {
'routes.execution_times': ['2016-08-28T11:11:47+02:00', ...]
}
},
...
]
}
}
}
}
}
}
But what if I would like to filter my documents based on execution dates. What's the best way achieving this?
Should I use a range filter to map my dates to time ranges?
Or is it better to use a script query and do a conversion of the execution_times to dates there?
Or is the best way to change the document structure to contain both, the execution_date and execution_time?
Update
"The dates are not a range but individual dates like [today, day after tomorrow, 4 days from now, 10 days from now]"
Well, this is still a range as a day means 24 hours. So if you store your field as date time, you can use leverage range query : from 20-Nov-2010 00:00:00 TO 20-Nov-2010 23:59:59 with appropriate time zone for a specific day.
If you store it as a String then you will lose all the flexibility of date maths and you would be able to do only exact String matches. You will then have to do all the date manipulations at the client side to find exact matches and ranges.
I suggest play with range queries using Sense plugin and I am sure it will satisfy almost all your requirements.
-----------------------
You should make sure that you use appropriate date-time mapping for your field and use range filter over that field. You don't need to split into 2 separate fields. Date maths will allow you to query just based on date.
This will make your life much easier if you want to do aggregations over date time field.
Reference:
Date Maths:
https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#date-math
Date Mapping : https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html
Date Range Queries:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html
Related
I have a timstamp in milliseconds, like 1645825932144
I'd like to make a date range query with elastic search query string for being able to get all records whom timestamp is in the last 24h:
timestamp:[now-24h TO now]
This does not work as timestamp is in milliseconds and now produces strings like 2001-01-01 13:00:00
Is it possible to achieve this with a cast or something?
I read about range queries and date math, but did not find anything.
It's easy to compute the timestamp in milliseconds for now and now-24h so why not do it in your application logic and build the query out of those values?
For instance (in JS),
const now = new Date().getTime();
const now24h = now - 86400000;
const query = `timestamp:[${now24h} TO ${now}]`;
query would contain the following value:
timestamp:[1647785319578 TO 1647871719578]
UPDATE:
PS: I might have misunderstood the initial need, but I'm leaving the above answer as it might help others.
What you need to do in your case is to change your mapping so that your date field accepts both formats (normal date and timestamp), like this:
PUT your-index/_mapping
{
"properties": {
"timestamp": {
"type": "date",
"format": "date_optional_time||epoch_millis"
}
}
}
Then you'll be able to query like this by mix and matching timestamps and date math:
GET test/_search?q=timestamp:[now-24h TO 1645825932144]
and also like this:
GET test/_search?q=timestamp:[1645825932144 TO now]
I need to write a query in ES that only returns documents that have been created less than N minutes, or hours, ago.
There's a createdTimeStamp field in millis, and I am able to write a simple query like this:
{
"query": {
"match": {
"createdTimeStamp": "1526011575731"
}
}
}
However, this query returns the documents where the createdTimeStamp matches the value "1526011575731". Not sure if a range query would work here as the field stores millis values.
You can use range together with now for that. E.g. to get the last hour:
{
"query": {
"range" : {
"createdTimeStamp" : {
"gte" : "now-1h"
}
}
}
}
For minutes, use m instead of h. You can also round, etc., see Date Math documentation.
That your date is in milliseconds should not make a difference, as internally, all comparisons are handled like that anyway:
Internally, dates are converted to UTC (if the time-zone is specified) and stored as a long number representing milliseconds-since-the-epoch.
Queries on dates are internally converted to range queries on this long representation, and the result of aggregations and stored fields is converted back to a string depending on the date format that is associated with the field.
(from https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html)
I've a single date type field in MongoDB named invoiceDate which contains two types of data formats
ISO standard "2019-07-27T09:01:15.598Z"
Epoch "575288179200000"
I want to extract/identify all invoice Ids which contain the Epoch date format only. How can I write Mongodb Query for that?
How about check if it is a number or not first. For eg.
const date = "575288179200000";
if (!Number.isNaN(date)) {
//Epoch
} else {
//ISO standard
}
Based on the below query return result I want to filter the month and ther year.
For example I only want data for the month of March.
db.SBM_USER_DETAIL.aggregate([
{
$project: {
join_date: '$JOIN_DATE'
}
}
]).map(
function(d) {
d.join_date = moment(d.join_date).locale('es').tz("Asia/Kolkata").format();
return d
})
How to use the returned formatted value of join_date inside the MongoDB aggregation query?
MongoDB's ISODate is very similar to the javascript Date class. If you have a date range in the Kolkata timezone, and want to filter by that, instantiate a pair of Date objects to define the range, before running the find.
For this instance, to return all join_date values that fall within March 2017, converted to the Kolkata (UTC-07:00) timezone, filter for date greater than or equal to midnight March 1 and less than midnight April 1, then convert the results using moment:
var first = new Date("2017-03-01T00:00:00-07:00");
var last = new Date("2017-04-01T00:00:00-07:00");
db.SBM_USER_DETAIL.find(
{join_date:{$gte: first, $lt: last}}, //filter based on join_date
{join_date:1,_id:0} // only return join_date, omit this if you need all fields
).map(
function(d) {
d.join_date = moment(d.join_date).locale('es').tz("Asia/Kolkata").format();
return d;
}
);
if i want to find a document created on a specific Day, until now i used a range
from the first minute of the day, to the last minute of the day in seconds , sth like :
query":{"dtCreated":{"$gte":{"sec":1381356782,"usec":0},"$lt":{"sec":1389356782,"usec":0}}}
is is possible to to somehow find all documents where only the Day, Month and year equals "dtCreated" ?
in pseudocode like :
query:{"dtCreated":ISODate("2014-01-23")} <- i know that may not be a valid iso date
but what i want is to find all documents for one day without using lt and gt ?
Sry for bad english and for any hints thanks in advance!
You can do it with the aggregation framework using the date aggregation operators.
Assuming dtCreated is an ISODate field, you could try something like this:
query = [
{
'$project': {
'year': {'$year':'$dtCreated'},
'month': {'$month':'$dtCreated'},
'day':{'$dayOfMonth':'$dtCreated'}
}
},
{
'$match' : {'year':'2014', 'month':'1', day:'1'}
}
]
db.mycollection.aggregate(query)
Edit: as orid rightly remarks, though this is an answer to your question (query for date without using date range), it's not a good way to solve your problem. I would probably do it this way: a range greater than or equal to today, but less than tomorrow
db.foo.find({'dtCreated':{'$gte':ISODate("2014-01-23"), '$lt':ISODate("2014-01-24")}})