If I turn on stemming/lemmatizer in sphinx can I push a term to it "as needed" that does not utilize stemming? I know I can use wordforms to always ignore that word from stemming e.g. Radiology > Radiology but that results in never stemming the word. I'm looking for a way to not add as a wordform exception but be able to in a query in essence say 'look exactly for "Radiology" and do not stem/lemmatize". I have tried "Radiology" instead of Radiology to no avail.
http://sphinxsearch.com/docs/current.html#conf-index-exact-words
:)
Then can do
=Radiology
(in extended match mode)
Related
I have implemented Postgres 9.6 trigram search https://www.postgresql.org/docs/9.6/static/pgtrgm.html into my application which works fine for a single search term.
I can't see how to allow my users to do AND OR NOT searches though.
Currently, if I put "perl" into the search field, it will return hundreds of results. That's great and works fine.
Now if I want to search for documents containing "perl" and also containing "javascript", no matter what search term I put in, no results come back.
I have tried for example:
"perl javascript"
"perl AND javascript"
"perl && javascript"
So I am trying to work out how I can provide to my end users a more sophisticated search than single term only. I would like my application users to be able to do full text searches with and/or/not.
Is it possible? If yes, what is the syntax?
This query finds ...
documents containing "perl" and also containing "javascript"
SELECT *
FROM tbl
WHERE document ~ 'perl'
AND document ~ 'javascript';
Note that "perlane" or "javascripting" or "Kaperl" also qualify. To search for whole words, you might be interested in text search instead. Overview:
Pattern matching with LIKE, SIMILAR TO or regular expressions in PostgreSQL
There are different search options possible with sphinx extended syntax.
Exact search: "I love to eat" //will match exact phrase
OR search: (eat|sleep|dream)
Is it possible to mix them and build queries like that:
"I love to (eat|sleep|dream)"
I know it is possible to simplify it and split OR condition to different exact phrases, like:
"I love to eat" | "I love to sleep" | "I love to dream"
But I plan to use lot of OR groups with lot of options inside, and extending this query will end up with huge one.
So is it possible to use OR syntax inside exact match syntax in Sphinx?
No, its not possible to use 'OR' within the Phrase Operator (the "s around words that enforces adjacent words) - the proper name for what you call 'exact match'.
Alas there isn't another combined 'strict order' operator and 'near' (ie there isnt a 'just before' operator). So you forced to use both, so something like
("I love to" << (eat|sleep|dream)) NEAR/3 ("I love to" NEAR/1 (eat|sleep|dream))
Which is no simpler, and would argue is more complicated and convoluted! The NEAR/3 in the middle is needed to make sure you matching on the same sentance within the document (otherwise there are edge cases with false positives).
An off the wall idea, if you have a lot of these 'OR' lists, is rather than implement them in the query, use wordforms instead. The drawback is you need to know them in advance (ie compiled into the index) and 'opting out' is more complicated.
we are an anatomy platform and use sphinx for our search. We want to make our search more fuzzier and started to use metaphone to correct spelling mistakes. It finds for example phalanges even though the search word is falanges.
That's good but we want more. We want that the user could type in falange or even falang and we still find phalanges. Any ideas how to accomplish this?
If you are interested you can checkout our sphinx config file here.
Thanks!
Well you can enable both metaphone and min_prefix_len on an index at once. It will sort of work.
falange*
might then just work. (to match phalanges)
The problem is the 'stripped' letters may change the 'sound' of the word (because change the pronunciation)
eg falange becomes FLNJ, but falang acully becomes FLNK - so they no longer 'substrings' of one another. (ie phalanges becomes FLNJS, which FLNK* wont match)
... to be honest I dont know a good solution. You could perhaps get better results, if was to apply stemming, BEFORE metaphone. (so the endings that change the pronouncation of the words are removed.
Alas Sphinx can't do this. If you enable stemming and metaphone together, only ONE of the processors will ever fire.
Two possible solutions, implement stemming outside of sphinx (or maybe with regexp_filter. Not sure if say a porter stemmer can be implemnented purely with regular expressions)
or modify sphinx, so that ALL morphology processors apply. (rather than just the first one that changes the word)
I am trying to improve my Sphinx configuration and I have a trouble with words ending with apostrophes.
For example, for Surfin' USA result, searching with "Surfin USA" returns match but "Surfing USA" doesn't return anything. How can I set Sphinx to return result for such situation?
Hmm, thats an interesting one. Not sure sphinx can automatically deal with this, because it has no way of knowing what the Apostrophe is meant to represent. I suppose there are cases where it could be multiple things.
The only way I can think would be to list them in exceptions, you can build a list of all words want to support
Surfin' > Surfing
Have to use exceptions to be able to use the apostrophe
You might want to add
Surfin > Surfing
too, so can search without the apostrophe too.
We need to have all these terms match each other and are running into difficulty
orthopaedic, orthopedic, orthopaedics, orthopedics
At the moment we are dealing with most other plurals using morphology stem_en
This is our current wordforms entry for this group (the pair is duplicated in reverse or
else it only works one way)
orthopaedic > orthopedic
orthopedic > orthopaedic
orthopedics > orthopaedics
orthopaedics > orthopedics
However "orthopedics" does not then match "orthopaedic" and we can't add another entry
"orthopaedic > orthopedics" because "orthopaedic" is already present and will throw an
error when indexing.
Any advice would be greatly appreciated
the pair is duplicated in reverse or else it only works one way
That's a bad idea! Putting it both way, will lead to issues (like you've found in fact!), you changing one to the other, so they wont match properly!
You need only one direction. Sphinx takes the left word, and actully stores the right in the index. So searching for the left and the right become interchangable. If you swap the words, then they nver get a chance to match.
The complication arrises because wordforms performs 'stemming exception' - ie a word in wordforms is NOT stemmed, so that means many word wont match. So you need to
perform stemming manually on the wordforms list, and
list all variations in your wordforms file, - to the same common word
Using your example above, would be something like
orthopaedic > orthopedic
orthopedic > orthopedic
orthopedics > orthopedic
orthopaedics > orthopedic
If the word did stem would have to do that, eg
bridge > bridg
bridges > bridg
bridging > bridg
etc
It vastly bloats your wordforms file, but it can be automated.