Search internal substrings in hibernate search - hibernate-search

Question

I've defined my entity as following.
#Entity
#Indexed
#AnalyzerDef(name = "ngram_index", tokenizer = #TokenizerDef(factory = WhitespaceTokenizerFactory.class),
filters = {
#TokenFilterDef(factory = LowerCaseFilterFactory.class),
#TokenFilterDef(factory = ASCIIFoldingFilterFactory.class),
#TokenFilterDef(factory = NGramFilterFactory.class,
params = {
#Parameter(name = SearchConstants.MIN_GRAM_SIZE_NAME, value = SearchConstants.MIN_GRAM_SIZE_VALUE),
#Parameter(name = SearchConstants.MAX_GRAM_SIZE_NAME, value = SearchConstants.MAX_GRAM_SIZE_VALUE)
})
})
#AnalyzerDef(name = "ngram_query", tokenizer = #TokenizerDef(factory = WhitespaceTokenizerFactory.class),
filters = {
#TokenFilterDef(factory = LowerCaseFilterFactory.class),
#TokenFilterDef(factory = ASCIIFoldingFilterFactory.class),
})
#NormalizerDef(name = "lowercase",
filters = {
#TokenFilterDef(factory = ASCIIFoldingFilterFactory.class),
#TokenFilterDef(factory = LowerCaseFilterFactory.class)
}
)
#Table(name = "ORDER")
public class Order {
#Id
#DocumentId
#Column(name = "ID")
#GeneratedValue(strategy = GenerationType.AUTO)
private Long id;
#Field(analyzer = #Analyzer(definition = "ngram_index"))
#Field(name = "name_Sort", store = Store.YES, normalizer= #Normalizer(definition="lowercase"))
#SortableField(forField = "name_Sort")
#Column(name = "NAME")
private String name;
//other fields, getters and setters omitted for brevity
I then tried to overwrite the default analyzer that is being used during indexing for querying in another class that is not an entity.
public abstract class AbstractHibernateSearcher<S extends SearchableEntity> {
// other fields and methods omitted here
protected Query buildInputSearchQuery(String[] searchableFields) {
if(Strings.isNullOrEmpty(searchRequest.getQuery()) || searchableFields.length == 0) {
return null;
}
SimpleQueryStringMatchingContext simpleQueryStringMatchingContext = queryBuilder.simpleQueryString().onField(searchableFields[0]);
for(int i = 1; i < searchableFields.length; i++) {
simpleQueryStringMatchingContext = simpleQueryStringMatchingContext.andField(searchableFields[i]);
}
Query inputSearchQuery = simpleQueryStringMatchingContext
.withAndAsDefaultOperator()
.matching((searchRequest.getQuery()).toLowerCase()).createQuery();
QueryBuilder queryBuilder = getNGramQueryBuilder(searchableFields);
return queryBuilder.bool().must(inputSearchQuery).createQuery();
}
protected QueryBuilder getNGramQueryBuilder(String[] searchFields) {
if (searchFields.length == 0) {
return null;
}
EntityContext entityContext = fullTextEntityManager.getSearchFactory().buildQueryBuilder().forEntity(clazz);
for(String field : searchFields) {
entityContext = entityContext.overridesForField(field, "ngram_query");
}
return entityContext.get();
}
}
This gives me the following error when I do a query search.
{message: "HSEARCH000353: Unknown analyzer: 'ngram_query'. Make sure you defined this analyzer.",…}
exception: "RuntimeException"
message: "HSEARCH000353: Unknown analyzer: 'ngram_query'. Make sure you defined this analyzer."
I found this from the official document.
You can use #AnalyzerDef on any:
#Indexed entity regardless of where the analyzer is applied to;
parent class of an #Indexed entity;
package-info.java of a package containing an #Indexed entity.
Since I am seeing the unknown analyzer, I guess the class where I tried to overwrite with "ngram_query" analyzer has no visiblity on this analyzer?

Answer 1

Yes, you can create ngrams for each word: use a WhitespaceTokenizerFactory for your tokenizer, and add NGramFilterFactory to your token filters (note it's not the same class you mentioned: it's a token filter, not a tokenizer).
You will also need to use a different analyzer at query time, one that does not create ngrams. Otherwise a user typing "manhantan" may get a match for documents containing "man", for example.
See https://stackoverflow.com/a/56107399/6692043 for information on how to do that.
Note that ngrams can lead to very large indexes, especially if you're not careful with the value of the "minGramSize" and "maxGramSize" parameters.
Another solution would be to use your original analyzer and a wildcard query, but unfortunately it ignores analysis and can be quite slow when using leading wildcards (which is what you need here).
protected Query inputFilterBuilder() {
String[] searchableFields = getSearchableFields();
if(searchableFields.length == 0) {
return null;
}
TermMatchingContext termMatchingContext = queryBuilder.keyword().wildcard().onField(searchableFields[0]);
for(int i = 1; i < searchableFields.length; i++) {
termMatchingContext = termMatchingContext.andField(searchableFields[i]);
}
return termMatchingContext
.matching(("*" + searchRequest.getQuery() + "*").toLowerCase()).createQuery();
}
Note the code above will only work if there is a single search term. As soon as there are spaces in searchRequest.getQuery(), you won't get any result. There can be spaces in the indexed text, however, which is what you wanted, if I understood correctly.