I am trying to specify a date format for an Elasticsearch field according to ISO 8601 as:
YYYY-MM-DDThh:mm:ssTZD
I put the field mapping like so:
"properties": {
"startDate": {
"type": "date",
"format": "YYYY-MM-DD'T'hh:mm:ss'TZD'"
}
}
When I try to insert a document with this field's value as: "startDate": "2018-01-10T07:07:07+01:00", I get this error:
"type": "mapper_parsing_exception",
"reason": "failed to parse [afield.startDate]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Invalid format: \"2018-01-10T07:07:07+01:00\" is malformed at \"+01:00\""
}
Is there something wrong in the date I am inserting? I'm following the example given in this W3C guide (https://www.w3.org/TR/NOTE-datetime):
Complete date plus hours, minutes and seconds:
YYYY-MM-DDThh:mm:ssTZD (eg 1997-07-16T19:20:30+01:00)
Elasticsearch datetime formats can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html
You want a datetime with no milliseconds. To do that, use the following:
PUT my_index
{
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"type1" : {
"properties": {
"startDate": {
"type": "basic_date_time_no_millis",
"format": "yyyyMMdd'T'HHmmssZ"
}
}
}
}
}
Timezone should be handled in the timestamp, and can be pulled from a query like:
GET /my_index/_search
{
"query": {
"range" : {
"timestamp" : {
"gte": "2018-01-10 07:07:07",
"lte": "now",
"time_zone": "+01:00"
}
}
}
}
For custom date formats in Elasticsearch you have to check the Joda-time documentation.
In your case you can try with yyyy-MM-dd'T'HH:mm:ssZ:
"properties": {
"startDate": {
"type": "date",
"format": "yyyy-MM-dd'T'HH:mm:ssZ"
}
}
With this you should be able to insert dates and make searches using the dates like the one you used in your example: 2018-01-10T07:07:07+01:00.
For all the people out there that have a date in the format :
2016-08-14T00:00:00+02:00
or :
yyyy-MM-ddTHH:mm:ss+HH:MM
You might consider using this format in elasticsearch :
format: "yyyy-MM-dd'T'HH:mm:ss+02:00"
Related
My MongoDB compass document looks like this
{
"_id": "123456789",
"code": "x123",
"title": "cool",
"createdDate":2022-07-06T08:04:52.156+00:00
"expiryDate":2023-12-31T00:00:00.000+00:00
}
I tried to create a mongo DB script in my pipeline to update the "expirydate" field to a specific date value "9999-12-31T00:00:00.000Z" format. My update script looks like this as I am trying to update it via my pipeline. The createdDate field took the current date correctly.
{
"command": "updateOne",
"datasource_name": "Austrailia",
"collection_name": "Sydney",
"query": {
"code": "x123"
},
"options": {
"upsert": true,
},
"update": {
"$currentDate": {
"createdDate": {
"$type": "date"
}
},
"$set": {
"title": "hot",
"expiryDate": {
"$date": "9999-12-31T00:00:00.000"
}
}
}
}
The script is failing as it is throwing errors -
{MongoError: Unknown modifier: expiryDate. Expected a valid modifier}
The dollar ($) prefixed field \'$date\' in \'expiryDate.$date\' is not valid for storage.
What would be the correct query syntax to update the date field "expiryDate" to the value specified above in the same format here?
I have created a index in elasticsearch with multiple date field and formatted the column as yyyy-mm-dd HH:mm:ss. Eventually I found the date is malformed and was populating wrong data into the fields. The index has more than 600 000 records and I don't want to leave any data. Now I need to create another field or new index with same date field and format as YYYY-MM-ddTHH:mm:ss.Z and need to populate all the records into new index or new fields.
I have used the date processor pipeline as below. but it fails. Correct me anything wrong here.
PUT _ingest/pipeline/date-malform
{
"description": "convert malformed date to timestamp",
"processors": [
{
"date": {
"field": "event_tm",
"target_field" : "event_tm",
"formats" : ["YYYY-MM-ddThh:mm:ss.Z"]
"timezone" : "UTC"
}
},
{
"date": {
"field": "vendor_start_dt",
"target_field" : "vendor_start_dt",
"formats" : ["YYYY-MM-ddThh:mm:ss.Z"]
"timezone" : "UTC"
}
},
{
"date": {
"field": "vendor_end_dt",
"target_field" : "vendor_end_dt",
"formats" : ["YYYY-MM-ddThh:mm:ss.Z"]
"timezone" : "UTC"
}
}
]
}
I have created the pipeline and used reindex as below
POST _reindex
{
"source": {
"index": "tog_gen_test"
},
"dest": {
"index": "data_mv",
"pipeline": "some_ingest_pipeline",
"version_type": "external"
}
}
I am getting the below error while running the reindex
"failures": [
{
"index": "data_mv",
"type": "_doc",
"id": "rwN64WgB936y_JOyjc57",
"cause": {
"type": "exception",
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2019-02-12 10:29:35]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "java.lang.IllegalArgumentException: unable to parse date [2019-02-12 10:29:35]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "unable to parse date [2019-02-12 10:29:35]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Illegal pattern component: T"
}
}
You can either use logstash like Shailesh Pratapwar suggested, but you also have the option to use elasticsearch reindex + ingest to do the same:
Create an ingest pipeline with the proper date processor in order to fix the date format/manipulation: https://www.elastic.co/guide/en/elasticsearch/reference/master/date-processor.html
reindex the data from the old index, to a new index, with the date manipulation. from: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
Reindex can also use the Ingest Node feature by specifying a pipeline
Use Logstash.
Read from ElasticSearch using LogStash.
Manipulate the date format.
Write to ElasticSearch using LogStash.
I am using rest API to get invoices whose create date is greater than some date (eg: After 1st September 2015).
GET
https://{{sluser}}:{{slkey}}#api.softlayer.com/rest/v3.1/SoftLayer_Account/getInvoices?objectFilter={"createDate":{"operation":"greaterThanDate","options":[{"name":"date", "value":"10/01/2015"}]}}&resultLimit=1,5
but this seems to be giving first 5 invoices. May I now what is wrong in the objectFilter? Also how can I specify the date format dd-MM-YYYY or mm-dd-yyyy or yyyy-mm-dd?
Try the following REST request:
https://[username]:[apikey] #api.softlayer.com/rest/v3/SoftLayer_Account/getInvoices?objectFilter={ "invoices": { "createDate": { "operation": "greaterThanDate", "options": [ { "name": "date", "value": [ "01/22/2016" ] } ] } } }&objectMask=mask[createDate]
Method: GET
Where: date format is mm-dd-yyyy.
This is other example using “betweenDate”:
https://[username]:[apikey]#api.softlayer.com/rest/v3/SoftLayer_Account/getInvoices?objectFilter={ "invoices": { "createDate": { "operation": "betweenDate", "options": [ { "name": "startDate", "value": ["02/01/2014"] }, { "name": "endDate", "value": ["02/13/2014"] } ] } } }&objectMask=mask[createDate]
Method: GET
References:
Object Filters
More SoftLayer REST API Examples
I am trying to use the strict_date formatter within ElasticSearch which is a a formatter for a full date as four digit year, two digit month of year, and two digit day of month: yyyy-MM-dd.
I am using the following code in Marvel:
PUT my_strictindex
{
"mappings": {
"my_type": {
"properties": {
"dob": {
"type": "strict_date"
}
}
}
}
}
I get the following error:
{
"error": "MapperParsingException[mapping [my_type]]; nested: MapperParsingException[No handler for type [strict_date] declared on field [dob]]; ",
"status": 400
}
Any help would be appreciated!
Refer to ES Docs
It should be
{
"mappings": {
"my_type": {
"properties": {
"dob": {
"type": "date",
"format": "strict_date"
}
}
}
}
}
I'm new with elasticsearch and I'm stuck with a query.
I want to get the next (now+3d) birthdays among my users. It looks simple but it's not because i have only the birthdate of my users.
How I can compare only months and day directly in the query when I only have a birthdate (Eg: 1984-04-15 or 2015-04-15 sometimes) ?
My field mapping:
"birthdate": {
"format": "dateOptionalTime",
"type": "date"
}
My actual query that doesn't work at all:
{
"query": {
"range": {
"birthdate": {
"format": "dd-MM",
"gte": "now",
"lte": "now+3d"
}
}
}
}
I saw this post Elasticsearch filtering by part of date but I'm not a big fan of the solution, and I would prefer instead of a wilcard a "now+3d"
Maybe I can do do something with a script ?
"Format" field was added in 1.5.0 version of elasticsearch. If your
version is below 1.5.0 format will not work. We had a same problem where we had to send an email on user's birthday and we were using version 1.4.4. So we created a separate dob field where we stored date in "dd-MM" format.
We changed the mapping of date field:
PUT /user
{
"mappings": {
"user": {
"properties": {
"dob": {
"type": "date",
"format": "dd-MM"
}
}
}
}
}
Then you can search:
GET /user/_search
{
"query": {
"filtered": {
"filter": {
"range": {
"date": {
"from": "01-01",
"to": "01-01",
"include_upper" : true
}
}
}
}
}
}