I've got input that is analogous to tags, where there are a couple of strings per record, and they should be thought of as keywords, not to be tokenized or broken up or analyzed in any particular way. I want it to show up in faceting "as-is", including spaces, slashes, dashes and ampersands.
I don't think I need multi_field here. There is one input value per record "keyPhrases" but the input value is a simple json array of strings.
I want elasticsearch to insert into the facets each of the values, and tag the record with all of the phrases.
Usually there are only one or two or three phrases per record, but there could be more. The set of keyPhrases is fairly small, like 30 or at most like 50. They could be thought of as "categories".
The faceting keeps breaking up the input strings and using lowercasing, even though I'm trying to specify not_analyzed, keyword tokenizer, keyword analyzer, and trying things like that.
I have other fields that keep their spacing and capitalization as I desire in the facets returned, however those fields are not_analyzed and are also store: true, but are also just exactly 1 string input per record, as opposed to many per record.
I could just take the top 1 keyPhrase per record and flatten it, but ideally all the tags would work and be available as facets.
Any ideas on how to do this?
Well, this is embarrassing.
My strict mapping wasn't actually committed to the server at the time I was trying this.
(I was dropping the index and creating the index again with each new mapping, and hadn't realized it, and this was not the final mapping, so it was getting loaded and then dropped.)
Related
I have a table in a word document that has three colums and all fields are mailmerge fields from an external IT system.
There are three columns displaying the fields:
Charge Description
Charge Value (£)
Eiligible? (yes/no)
I am trying to create a field that adds up all eligibale charges so that only charge values that show a "yes" in the eligigble field are included. Does anyone know if this is possible? I have tried creating a formula but can't get it to work. Also, I would assume at some point an if statment is required so that it only includes the eligible charge.
Has anyone done anything similar before and if so, would they mind sharing how it was achieved?
Many thanks
You can do some things with expression fields (created in Word with CTRL-F9). This will look like {} and you can insert the expression. eg {{MERGFIELD charge} + {MERGEFIELD charge2}}. Since however you want to check multiple values and then create an expression, its probably easier to use a macro. The macro would contain your logic, then set the fields in the document accordingly.
Here are two external links since I can't reproduce a useful amount the content here because it's a verbose answer to a potentially deep question:
Expression Fields
Merge fields
I hope that helps.
Is it possible, in SQL Server 2008, using the full text index syntax, to run a query such as this one?
SELECT *
FROM TABLE_TO_SEARCH S,
TABLE_WITH_STRINGS_TO_SEARCH SS
WHERE
CONTAINS(S.WHOLE_NAME,SS.FIRST_NAME)
OR CONTAINS(S.WHOLE_NAME,SS.LAST_NAME)
I need to search for the FIRST_NAME in table TABLE_TO_SEARCH, column WHOLE_NAME that has an full text index on it. It doesn't seem to be a valid query though... Is there any workaround to it by using the full text index search?
LATER EDIT:
Here is the business case: each night I am downloading from several websites information about "blacklisted" individuals and insert it into a table in this format: WholeName, LastName, FirstName, MiddleName. But the data is chaotic as WholeName does not necessarily contain either the last, first or middle name or the WholeName is null while the other 3 fields have values, or every of these 4 fields is null and so on. Also, the data may repeat itself as one blacklisted individual may come from 2+ of these websites. What I need to do is to compare this data, as chaotic as it is, against our customer data based on our customer's First and Last name and give it a matching score (rank) against the files we download from these websites.
First I tried with charindex or like operators but I couldn't create a scoring algorithm based on this and also it took 6 hours to compare just our customer's first and last name with only the WholeName column from the TABLE_TO_SEARCH table. I thought that perhaps implementing the full_text index it would get easier and faster but ... apparently I was wrong.
Has anyone dealt with a task like this? And if so, what was the best approach?
After skimming http://technet.microsoft.com/en-us/library/ms187787.aspx and http://technet.microsoft.com/en-us/library/ms142571.aspx I don't think it is possible to do your search in this way. Not only that, but it seems this type of index wouldn't work well with names anyway.
If you care about checking one name then all you have to do is set those values to variables. This method would allow you to use the full-text index.
Otherwise, I would suggest splitting the WHOLE_NAME column (if there is a space or unique character between the first and last name) and comparing each part to those other columns. If you are working with a huge data set, you may want to experiment with doing this at a temp table level and creating an index.
Good luck!
I have an SDO supplying data to a read-only browser. The SDO query joins several tables and has calculated fields as well as natural data fields.
The users now want a search facility so the browser will only show rows where the search word appears in ANY of the text fields.
For example they want to see rows where
customer.name matches "*bob*" OR
customer.address1 matches "*bob*" OR
product.description matches "*bob*" OR
calc_field_1 matches "*bob*" OR
calc_field_2 matches "*bob*" OR ...
Ideally the answer will filter the SDO output as it is created - but I am also happy to filter the data on the way to the smartbrowser or in the smartbrowser.
The business problem you're trying to solve in fraught with performance issues if you implement it as written. I'd suggest
adding another character column to the table or db,
putting all the words from the other columns in it,
applying a word-index to the new column,
doing a search on that column, and then linking back to the source tables.
It'll be much faster and easier to use.
I used a very simple solution in the end. Users can enter a string they are looking for. If the string is in a cell in the browser then the cell is highlighted in yellow.
Before this the users had to scroll up and down trying to spot the cells of interest in hundreds of rows. We did not have the time or budget for anything fancier.
The important bit of code in the smartbrowser is like this...
on row-display of br_table in frame f-main
do:
if rowObject.field1 matches "*BOB*" then
rowObject.field1:BGCOLOR in browse br_table = 14.
if rowObject.field2 matches "*BOB*" then
rowObject.field2:BGCOLOR in browse br_table = 14.
if rowObject.field3 matches "*BOB*" then
rowObject.field3:BGCOLOR in browse br_table = 14.
... etc ...
it's not hard-coded to only look for Bob - but you should get the idea.
i have a kinda weird problem with haystack/elasticsearch trying to find tv episodes stored in my database based on a string like this: 's04e07' which means season 4 episode 7 and is a kind of standard format, but the search index has its problems with that.
Trying a few different things it looks like numbers are not indexed in EdgeNgramFields.
In a CharField i can only find exact word matches like '2013' if contained in the titel, but i have no luck finding 's04e07'.
How do i get my results out of the index?
How could i possibly change the hardcoded default mapping in haystack to index my stuff correctly?
I actually wrote about haystack a few days ago, I would suggest reading that one first:
Django Haystack Distinct Value for Field
It's not directly on point, but my advice here is the same. Stop using haystack.
Haystack comes with an outofthebox edgengram and ngram analyzers, which is cool, except these analyzers don't work in nearly all use cases.
They will especially not work in yours because you are mixing numbers and chars.
But my first question is, why can't you index the data like this:
"season":1
"episode":1
And then at search time break down the users search into the above format?
If that isn't possible, you can still PUT a mapping manually without letting haystack do it for you (which I recommend highly anyway because it's mappings are not correct). It's pretty easy to do with elasticutils.
Keep in mind, I don't think edgengram is exactly what you want here in any event. because it only grams from the edge and is most useful for autocompletes, for example if someone is typing s04e and you want to display a list of possible matches.
So, this depends on how users will query the data. Will it always be the above string whole, or parts of the string, or will they sometimes search for e07 and you want to show all seasons with episode 7's?
The last possibility here is to just index it as normal (haystack will choose snowball) and use prefix queries / regex queries to get what you want.
Is it possible to form an Endeca query to retrieve a field that must start with certain letters? Say like get all users who's first letter is A? I checked with Range filters but it is supporting only numerical fields as well as Wild card search. But nothing worked well so far.
Creating a dimension is one way of approaching the problem as Paul Lemke mentioned.Wildcard is not an option since the performance overhead as well as irrelevant records.
But we solved it using couple of other alternatives.
Create a new property for the Object called "StartWith", store the first letter of the Object and make it searchable. We found it easier than creating a Dimension.
There is a problem where letters like 'A' are usually stop words in Endeca. We can do you a couple of work around to solve this.
Get the ASCII value of the first letter and store the numerical value in to that property. One more advantage with this approach is that we can use Range Filters. But you can't search for 'AB' kind of requirements.
Pre-pend some characters like ^^^My name and search for ^^^M. The advantage with this approach is you can search conditions like letters starts with AB.
Endeca at it's current version (6.1) does not have a search filter that works like a "startswith" function in other programming languages.
I do have two options that might possibly get you close:
If you are truly only looking for the first letter you can setup a Dimension value for each letter of the Alphabet (A,B,C...). You can then refine on each letter and see only the values that start with letter A, B, C, etc. The only downside to this is you can only filter based on how many dimension values you setup. So if you added "A", you couldn't filter anything that started with "AB". You could go down the line and add "AB", "BA, "CA", and so on but that would get unwieldy very fast.
If you want something closer to a "startswith" function the only other option is to use a wildcard search. Basically you would do a property search like this: N=0&Ntk=Username&Ntt=ab*
The trick with wildcard searching is it will do that across multiple words in that property. So assuming you had a data set of these values:
Smithers Smith
Larry Smith
Jenna-Smith
Doing a search of sm* would actually return all 3 results because "sm" was in their last name. Even the one with the dash would return as Endeca think's that is a seperate word. (It might be possible to turn that off though, not sure).
So basically it comes down to this: Stick a one word in a property, set that property to allow wildcard search, then do a "blah*" against that property and you should have the results you're looking for.
Have you tried the First relevance rank module which is supposed to rank based on proximity to the beginning of the field?
It sounds similar to what you are looking for and together with a wild card may produce your intended results.