I have expiration_date date field in my Elasticsearch and request from user to be able to "full-text" search in this field and I have only one input for this.
So my initial mapping is:
PUT my_index
{
"mappings": {
"_doc": {
"properties": {
"expiration_date": {
"type": "date"
}
}
}
}
}
and as a value I have for example: 2021-08-27T10:48:00.293Z.
User would like be able to search this by 2021, 2021-08, 2021-08-27, 27-08-2021 and 08-2021. For all this search terms I have only one input field which is used to search in other fields as well (fields like title, description etc.).
My idea to achieve this was to introduce some multi-fields to the base field. So something like:
PUT my_index
{
"mappings": {
"_doc": {
"properties": {
"expiration_date": {
"type": "date",
"fields": {
"yyyy-mm-dd" : {
//what to do here?
},
"yyyy-mm" : {
//what to do here?
},
"yyyy" : {
//what to do here?
},
"mm-yyyy" : {
//what to do here?
},
"dd-mm-yyyy" : {
//what to do here?
}
}
}
}
}
}
}
But I'm wondering if this is doable this way? Is something similar doable in any way only at the Elasticsearch side? Or I should rather prepare something similar on my application side, send it to ES and just use it there?
Probably, the best solution would be to use the custom formats for date field in Elasticsearch:
PUT my_index
{
"mappings": {
"_doc": {
"properties": {
"expiration_date": {
"type": "date",
"format": "year||year_month||year_month_day||dd-MM-yyyy||MM-yyyy||strict_date_optional_time||epoch_millis"
}
}
}
}
}
Then you can range query the field:
{
"query": {
"bool": {
"must": [{
"range": {
"expiration_date": {
"gte": "27-01-2001"
}
}
}]
}
}
}
Where you can use any format specified in mapping.
This solution would be the most scalable: you can just add more formats (available here or you can construct one) and reindex the data to support any new formats.
Related
I'm new to Mongodb.
Here I want to sort my objects by date, but my parent object is different (In my example eventA/B/C, but they will be totally different eventually) and I would like to keep them in my final result.
{
{
"eventA": {
"key1":"test1",
"key2":"test2",
"date": "2019"
}
},
{
"eventB": {
"key1":"test_a",
"key2":"test_b",
"date": "2021"
}
},
{
"eventC": {
"key1":"test_c",
"key2":"test_d",
"date": "2020"
}
}
}
As a result, the object will look like this:
{
{
"eventA": {
"key1":"test1",
"key2":"test2",
"date": "2019"
}
},
{
"eventC": {
"key1":"test_c",
"key2":"test_d",
"date": "2020"
}
},
{
"eventB": {
"key1":"test_a",
"key2":"test_b",
"date": "2021"
}
}
}
The end goal is to keep the most recent object.
Maybe it is not necessary to go through the sorting stage?
I am using MongoDB Atlas Search to perform a search in Collection, for this I created a Atlas Search Index:
{
"mappings": {
"dynamic": false,
"fields": {
"caption": {
"type": "string"
}
}
}
}
Here is my aggregation:
[
{
"$search":{
"text":{
"path":"caption",
"query":"Ingocnitáá",
"fuzzy":{
}
},
"highlight":{
"path":"caption"
}
}
}
]
I have below document in my collection:
{caption:"Ct tyu test Ingocnitáá"}
Issue: When I searching Ingocnitaa agreegation returning 0 result.
Is there anything wrong with my Search Index? I want an directive insensitive Search with highlight.
There are two things missing:
Include index name (Recommend to not use a default index, create a new index)
Always pass fuzzy:{}
Here is a working query:
[
{
"$search":{
"index": 'messageText',
"text":{
"path":"caption",
"query":"Ingocnitaa",
"fuzzy":{
}
},
"highlight":{
"path":"caption"
}
}
}
]
Where messageText is search index name.
Search Index Formattion:
{
"mappings": {
"dynamic": false,
"fields": {
"caption": {
"type": "string"
}
}
}
}
Reference: CLick here
I am trying to build a dynamic query using NEST which is as under
string product = "goldpgl";
string agencyid = "1123";
ISearchResponse <ProductModel> res = client().Search<ProductModel>(s => s
.Index("proddata")
.From(0)
.Size(100)
.Query(q =>
+q.Term(p => p.product, product) &&
+q.Term(p => p.agencyid, agencyid)));
If I pass, product value = "GoldPGL" [ N.B.~ Real value in the index ], I am not able to find the result.
However, if I pass the value in lowercase like "goldpgl", it works.
Also, it does not work for values like "Gold - PGL" or "SOME OTHER LOAN".
My POCO is as under
public class ProductModel
{
public string product { get; set; }
public string agencyid { get; set; }
}
What is wrong and how to rectify this?
As you have not provided the mapping and search query, I am assuming its happening because you are using the term query not the match query.
Term queries are not analyzed means whatever you entered in your search query would be matched against the tokens in the index. And by default, all the text fields in Elasticsearch use the standard analyzer which converts tokens to lowercase. hence GoldPGL doesn't match while goldpgl matches in your term query.
While match query as explained the official document is analyzed query and the same analyzer is applied on the search term, which is applied at index time, hence GoldPGL as well as goldpgl converted to goldpgl and both the query matches the documents and same is with Gold - PGL which also matches and verifies by me.
Analyze API comes very handy to troubleshoot these types of issue, where search query doesn't match the indexed tokens and one example of how GOLDPGL would be analyzed is shown below:
POST /_analyze
{
"text": "GOLDPGL",
"analyzer" : "standard"
}
{ "token": "goldpgl",}
{
"text": "GOLD - PGL",
"analyzer" : "standard"
}
{
"token": "gold",
"start_offset": 0,
"end_offset": 4,
"type": "<ALPHANUM>",
"position": 0
},
{
"token": "pgl",
"start_offset": 7,
"end_offset": 10,
"type": "<ALPHANUM>",
"position": 1
}
I reproduced your issue and as I am not familiar with NEST, showing your example using the REST API.
Index Def
POST /
{
"mappings": {
"properties": {
"product": {
"type": "text"
}
}
}
}
Index some documents
POST //_doc/1
{
"product": "GoldPGL"
}
Index 2nd doc
{
"product": "Gold - PGL"
}
Now search query using term query(as shown in your example), doesn't return any result (when GoldPGL is used)
{
"query": {
"term": {
"product": {
"value": "GoldPGL"
}
}
}
}
When used goldgpl, it gives result
{
"query": {
"term": {
"product": {
"value": "goldpgl"
}
}
}
}
Result
"hits": [
{
"_index": "so-term-nest",
"_type": "_doc",
"_id": "1",
"_score": 0.8025915,
"_source": {
"product": "GoldPGL"
}
}
]
Solution (use match query)
{
"query": {
"match" : {
"product" : {
"query" : "GoldPGL"
}
}
}
}
and this return results
"hits": [
{
"_index": "so-term-nest",
"_type": "_doc",
"_id": "1",
"_score": 0.8025915,
"_source": {
"product": "GoldPGL"
}
}
]
I'm implement query in Elastic4s library. But I don't know how to implement a following Json query for Elasticsearch.
{
"bool": {
"must": [
{
"match_all": {}
},
{
"keywordQuery": "hogehoge"
}
]
}
}
I don't know how to implement this part of Json query.
{
"keywordQuery": "hogehoge"
}
This is a code I implemented halfway.
boolQuery().must(Seq(matchAllQuery(), query("{keywordQuery: hogehoge}")))
and this is an output of an above code.
{
"bool": {
"must": [
{
"match_all": {}
},
{ "queryString": {
"query": "{keywordQuery": "hogehoge}"
}
}
]
}
}
I expect
{
"keywordQuery": "hogehoge"
}
but actually
{ "queryString": {
"query": "{keywordQuery": "hogehoge}"
}
}
Would you help me please?
I can't find a reference to keywordQuery in the ElasticSearch DSL documentation at https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl.html or https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl.html - maybe you need a Term query?
(on e.g. Logstash indices 'text' fields have a non-analysed subfield called '.keyword' so if I do a "keyword query" I normally do termQuery("field.keyword","value))
I don't think you need to include matchAllQuery() as it's kinda implied that you start off with the full set of results, so you could drop the bool and simplify the query to:
{
"query": {
"term": {
"field.keyword": "value"
}
}
}
In Elastic4s this would be:
client.execute {
termQuery("field.keyword", "value")
}
I'm trying to create a Range Filter on elastic search using the following syntax:
{
"size": 100,
"filter": {
"and": {
"filters": [
{
"range": {
"listingDate": {
"gt": "15/07/2017 16:08:53"
}
}
}
]
}
}
}
The data format is:
"listingDate": "07/07/2015 09:30:00",
However regardless of the filter properties the same incorrect results are being returned by elastic search. I have tried adding the following format:
"format": "dd/MM/yyyy HH:mm:ss"
but I get the same incorrect results.
A fuller example is:
{
"size": 100,
"sort": [
{
"listingDate": {
"order": "asc"
}
}
],
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "Event"
}
},
{
"range": {
"listingDate": {
"gte": "15/07/2015 16:08:53"
}
}
},
{
"range": {
"endDate": {
"gte": "15/07/2015 16:08:53"
}
}
}
]
}
},
"filter": {
"and": {
"filters": [
{
"terms": {
"departments": [
"2393"
]
}
}
]
}
}
}
In JSON documents, dates are represented as strings. Elasticsearch uses a set of preconfigured formats to recognize and parse these strings into a long value representing milliseconds-since-the-epoch in UTC. It might be possible that your date field might not be listed in the set of preconfigured ES date formats.
Formatted dates will be parsed using the format specified on the date field by default, but it can be overridden by passing the format parameter to the range query.
{
"range" : {
"listingDate" : {
"gte": "07/07/2015 09:30:00",
"format": "dd/MM/yyyy HH:mm:ss"
}
}
}
let suppose "arr" argument have a date range e.g. ["2019-07-10","2019-07-11"]
let start_date_query;
let range=[];
if ( arr.date_from ){
if(arr.date_from[1]){
range.push({
"range":{
"start_date":{ "gte":arr.date_from[0] }
}
});
range.push({
"range":{
"end_date":{ "lte": arr.date_from[1] }
}
});
start_date_query = {
"query": {
"constant_score": {
"filter": {
"bool": {
"must":range
}
}
}
};
}
}
}