Date Filter on elastic search - date

I'm trying to create a Range Filter on elastic search using the following syntax:
{
"size": 100,
"filter": {
"and": {
"filters": [
{
"range": {
"listingDate": {
"gt": "15/07/2017 16:08:53"
}
}
}
]
}
}
}
The data format is:
"listingDate": "07/07/2015 09:30:00",
However regardless of the filter properties the same incorrect results are being returned by elastic search. I have tried adding the following format:
"format": "dd/MM/yyyy HH:mm:ss"
but I get the same incorrect results.
A fuller example is:
{
"size": 100,
"sort": [
{
"listingDate": {
"order": "asc"
}
}
],
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "Event"
}
},
{
"range": {
"listingDate": {
"gte": "15/07/2015 16:08:53"
}
}
},
{
"range": {
"endDate": {
"gte": "15/07/2015 16:08:53"
}
}
}
]
}
},
"filter": {
"and": {
"filters": [
{
"terms": {
"departments": [
"2393"
]
}
}
]
}
}
}

In JSON documents, dates are represented as strings. Elasticsearch uses a set of preconfigured formats to recognize and parse these strings into a long value representing milliseconds-since-the-epoch in UTC. It might be possible that your date field might not be listed in the set of preconfigured ES date formats.
Formatted dates will be parsed using the format specified on the date field by default, but it can be overridden by passing the format parameter to the range query.
{
"range" : {
"listingDate" : {
"gte": "07/07/2015 09:30:00",
"format": "dd/MM/yyyy HH:mm:ss"
}
}
}

let suppose "arr" argument have a date range e.g. ["2019-07-10","2019-07-11"]
let start_date_query;
let range=[];
if ( arr.date_from ){
if(arr.date_from[1]){
range.push({
"range":{
"start_date":{ "gte":arr.date_from[0] }
}
});
range.push({
"range":{
"end_date":{ "lte": arr.date_from[1] }
}
});
start_date_query = {
"query": {
"constant_score": {
"filter": {
"bool": {
"must":range
}
}
}
};
}
}
}

Related

Sort objects by key date with "different" main object name. with mongodb

I'm new to Mongodb.
Here I want to sort my objects by date, but my parent object is different (In my example eventA/B/C, but they will be totally different eventually) and I would like to keep them in my final result.
{
{
"eventA": {
"key1":"test1",
"key2":"test2",
"date": "2019"
}
},
{
"eventB": {
"key1":"test_a",
"key2":"test_b",
"date": "2021"
}
},
{
"eventC": {
"key1":"test_c",
"key2":"test_d",
"date": "2020"
}
}
}
As a result, the object will look like this:
{
{
"eventA": {
"key1":"test1",
"key2":"test2",
"date": "2019"
}
},
{
"eventC": {
"key1":"test_c",
"key2":"test_d",
"date": "2020"
}
},
{
"eventB": {
"key1":"test_a",
"key2":"test_b",
"date": "2021"
}
}
}
The end goal is to keep the most recent object.
Maybe it is not necessary to go through the sorting stage?

Jolt for complex JSON transform

Extracting values from first array element and second element and joining both.
Input String:
[
{
"creationDate": "2020-04-01T14:14:32.685+0000",
"factValues": [
{
"Factname": "Medicine",
"factvalue": "1234556",
"sourcePguid": "1"
},
{
"Factname": "Journal",
"factvalue": "123455",
"sourcePguid": "1"
}
]
},
{
"creationDate": "2020-04-01T14:14:32.685+0000",
"factValues": [
{
"Factname": "chemical",
"factvalue": "123455567",
"sourcePguid": "2"
},
{
"Factname": "Rubber",
"factvalue": "123455435",
"sourcePguid": "3"
}
]
}
]
Output excepted:
{
{
"sourcePguid" : "1",
"Medicine":"1234556",
"Journal":"123455"
}, {
"sourcePguid" : "2",
"chemical":"123455567",
"Rubber":"123455435"
}
}
Kindly help me to achieve the expected output.json in spec.json. The spec is not transforming as expected output. I want learn how to use attributes inside the string parser.It would be great.
Hope this might help,
[
{
"operation": "shift",
"spec": {
"*": {
"factValues": {
"*": {
"sourcePguid": "&3.sourcePguid",
"#(0,factvalue)": "&3.#Factname"
}
}
}
}
}, {
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"sourcePguid": "=firstElement(#(1,sourcePguid))"
}
}
}
]
Yet another answer using JSONata
For java: https://github.com/IBM/JSONata4Java
$.factValues{
sourcePguid: {
"sourcePguid": $distinct(sourcePguid),
Factname:factvalue
}
} ~> $each(function($v){$v})

Full-text search in date field with Elasticsearch

I have expiration_date date field in my Elasticsearch and request from user to be able to "full-text" search in this field and I have only one input for this.
So my initial mapping is:
PUT my_index
{
"mappings": {
"_doc": {
"properties": {
"expiration_date": {
"type": "date"
}
}
}
}
}
and as a value I have for example: 2021-08-27T10:48:00.293Z.
User would like be able to search this by 2021, 2021-08, 2021-08-27, 27-08-2021 and 08-2021. For all this search terms I have only one input field which is used to search in other fields as well (fields like title, description etc.).
My idea to achieve this was to introduce some multi-fields to the base field. So something like:
PUT my_index
{
"mappings": {
"_doc": {
"properties": {
"expiration_date": {
"type": "date",
"fields": {
"yyyy-mm-dd" : {
//what to do here?
},
"yyyy-mm" : {
//what to do here?
},
"yyyy" : {
//what to do here?
},
"mm-yyyy" : {
//what to do here?
},
"dd-mm-yyyy" : {
//what to do here?
}
}
}
}
}
}
}
But I'm wondering if this is doable this way? Is something similar doable in any way only at the Elasticsearch side? Or I should rather prepare something similar on my application side, send it to ES and just use it there?
Probably, the best solution would be to use the custom formats for date field in Elasticsearch:
PUT my_index
{
"mappings": {
"_doc": {
"properties": {
"expiration_date": {
"type": "date",
"format": "year||year_month||year_month_day||dd-MM-yyyy||MM-yyyy||strict_date_optional_time||epoch_millis"
}
}
}
}
}
Then you can range query the field:
{
"query": {
"bool": {
"must": [{
"range": {
"expiration_date": {
"gte": "27-01-2001"
}
}
}]
}
}
}
Where you can use any format specified in mapping.
This solution would be the most scalable: you can just add more formats (available here or you can construct one) and reindex the data to support any new formats.

Advanced elasticsearch query

I am using laravel 4.2, mongodb and elasticsearch. Below is a working code, I am trying to convert this advanced where queries to elasticsearch queries:
$products = Product::where(function ($query) {
$query->where (function($subquery1){
$subquery1->where('status', '=', 'discontinued')->where('inventory', '>', 0);
});
$query->orWhere (function($subquery2){
$subquery2->where('status', '<>', 'discontinued');
});
})->get();
All I can get so far is just returning discontinued products, the code below works but it is not what I need:
$must = [
['bool' =>
['should' =>
['term' =>
['status' => 'discontinued']
]
]
]
];
Can you show me how can I achieve the same query I first described above but in elasticsearch? I want to return discontinued products with inventory, then also return products that are not equal to discontinued.
The WHERE query you've described can be expressed in SQL like this
... WHERE (status = discontinued AND inventory > 0)
OR status <> discontinued
In Elasticsearch Query DSL, this can be expressed like this:
{
"query": {
"filtered": {
"filter": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"term": {
"status": "discontinued"
}
},
{
"range": {
"inventory": {
"gt": 0
}
}
}
]
}
},
{
"bool": {
"must_not": [
{
"term": {
"status": "discontinued"
}
}
]
}
}
]
}
}
}
}
}
Translating this query into PHP should now be straightforward. Give it a try.

Mongo create request for embedded subfields

I'm new to mongo and need help finding db entries created at requested time. In my example there are a lot of embedded fields, and I do not understand syntaxis for request:
{
"_id": "54e1a045e4b03f5930293da6",
"_version": 31867,
"_transId": "4ae4d0e6-d3df-4a24-9621-1cdb7f12362f-10489329",
"accountBalances": {
"BALANCE": {
"thresholds": {
},
"quotas": "ROLLOVER_QUOTA": {
"thresholds": {
},
"quotaCode": "ROLLOVER_QUOTA",
"credits": {
"_1HVa0dJoEeSUwbM1-xYKvg": {
"startDate": ISODate("2015-03-24T21:00:00Z"),
"creditAmount": "547194099151",
"endDate": ISODate("2020-03-24T21:00:00Z"),
"started": true,
"debits": {
"consolidated": {
"creationDate": ISODate("2015-04-17T18:00:01.469Z"),
"debitAmount": "547194090291",
"debitId": "consolidated"
}
},
"creditId": "_1HVa0dJoEeSUwbM1-xYKvg"
}
}
}
}
}
I need to search for entries which have debit creation date $gte:ISODate("2015-03-16T00:00:00.000Z"), $lte:ISODate("2015-03-16T04:00:00.000Z"
You can use the dot notation to access the fields of an embedded document:
db.collection.find(
{
"accountBalances.BALANCE.quotas.ROLLOVER_QUOTA.credits._1HVa0dJoEeSUwbM1-xYKvg.debits.creationDate": {
"$gte": ISODate("2015-03-16T00:00:00.000Z"),
"$lte": ISODate("2015-03-16T04:00:00.000Z")
}
}
);