Algolia transliteration Cyrillic strings - algolia

Does anyone have experience with Algolia & transliteration? For example indexes contain Cyrillic text but user is typing in latin letters.

Unfortunately, most search engines don't support transliteration natively; same for Algolia.
The best way to handle such a use-case is to enrich your objects with the transliterated attributes before sending them to the search engine.
Maybe you can try gausby/translitit-cyrillic-russian-to-latin?

Just to add one little clarification. Let's say we have one field like name and many name translations
you can form object like
{
name: 'whatever',
translations: {
ru: 'без разницы',
he: 'οτιδήποτε',
de: 'was auch immer'
}
}
Then you put in your searchable attributes name, translations.ru, translations.he, translations.de so search will return 'whatever' when user type in 'без ра' for example.

Related

A Problem with Plural Localization in Swift

I have a function to get a localized string for plural cases.
extension String {
static func localizedStringForPlurals(
formatKey key: String,
comment: String = "",
args: CVarArg... ) -> String
{
let format = NSLocalizedString(key, comment: comment)
var result = withVaList(args){
(NSString(format: format,
locale: NSLocale.current,
arguments: $0) as String)
}
return result
}
...
}
The key is NSLocalizedString, which gets a localized string for the plural case from the stringsdict file.
I have an example of how to get localized string for taps. If the int number is one, the localized string is "This tap"; if int is larger than one, i.e., others, the string is "These taps". See attachment: stringsdict file (English).
It works fine for English. For example:
var count: Int
...
let locS = String.localizedStringForPlurals(
formatKey: "theseTaps", args: count)
// "This tap" if count is 1
// "These taps" if count is > 1
However, it does not work well for Chinese. Here is a similar stringsdict file for Chinese.
The result in Swift code is always a plural string, i.e., "这些点击"(These tap), even if the count is 1 (expected: This tap or 这个点击)
I am not sure if this is a bug in NSLocalizedString or not. I know that in Chinese in general, there are no plural cases. However, as in this example, there are plural cases, this tap or these tap. In Chinese, there is no plural for "taps", one word "tap" can refer to (one) tap or (>1) taps. but "this" and "these" in Chinese are different, and they mean plural cases.
I do like Apple's localization for the plural framework. In Swift code, there is no awareness of what language is being used, but the localized string is picked up from the localization framework. However, I would like this framework to work as a developer's expectation, as in my example in two dict files. Not to be too smart enough to always take the plural result, assuming that there is no plural rule in a language (like in Chinese).
I am not sure if there is way to resolve the issue? For example, turn off this kind of smart way (plural rule for a specific language. zero, one, others for English, others for Chinese, ...), and let developer to decide and layout the plural rule of how to get a localized string.
To answer your question -- if you want something else, you need to program it yourself. In fact, I did do this at Trello because we wanted English to have a "zero" where we said something like "You don't have any cards" instead of "You have 0 cards".
We had a function like yours and in it, it said (pseudocode)
if (locale == "en" && numberForPlurals == 0) {
pluralCategory = "zero";
}
And we looked up the string ourselves based on the key and plural category.
Another option is to reword the phrase so that it works for any number -- this is obviously not ideal. An English example would be "You have 3 cat(s)" -- which is what you would do if you didn't have a pluralization capability.
You seem to know this, but for those that need more information:
Apple is just implementing the UNICODE/CLDR standard for pluralization -- this is not iOS specific -- it's a standard.
Not every language plurals based on 1, or 2+ like English does. Those categories you see are not really referring to the numbers -- they are categories that each language defines for itself.
I wrote an article about it here: https://www.atlassian.com/engineering/ios-i18n-plurals (note: this article and the implementation was written before plurals were supported on iOS directly)
Summary
There are six categories of numbers that each language could use: zero, one, two, few, many and other
You have to look up your language here: https://unicode-org.github.io/cldr-staging/charts/latest/supplemental/language_plural_rules.html
Then see what the mapping from number to category is -- For example: english uses one and other: https://unicode-org.github.io/cldr-staging/charts/latest/supplemental/language_plural_rules.html#en
Chinese uses only one category (other) and doesn't pluralize based on numbers: https://unicode-org.github.io/cldr-staging/charts/latest/supplemental/language_plural_rules.html#zh
So, you should only expect a Chinese translator to provide one string (just for other) and that's the only one you need. If you need something else, you have to write your own logic and function to provide the key to look up in the strings file.
NOTE: the names "zero", "one", "two", etc are misleading for most languages. There are six categories -- they could have just named them cat1, cat2, cat3, etc.

Ranking rules for camelCase attributes

I'm building an Algolia index to search through user-created communities on my site.
Just like for subreddits, the name of the communities can't contain spaces and are therefore often written by users in camelCase.
Here is an example of an object in my index:
{
"name": "headphoneAdvice",
"description": "This community is dedicated to enthusiasts and newcomers. We are all about making the right decision when purchasing new headphones."
}
Both name and description are set to be searchable attributes and i'm currently using these ranking rules :
["typo","geo","words","filters","proximity","attribute","exact","custom"]
However, this does not seem to work well with the camelCase name. For example, if I type "advice" in the search, the object above with "name": "headphonesAdvice" isn't found.
I'm guessing this is because words in camelCase are considered single words and thus do not match.
I've looked online for rules that allow indexing of camelCase attributes but couldn't find anything really.
Any ideas?
Cheers!
After asking around, found that someone at algolia thought of this and added https://www.algolia.com/doc/api-reference/api-parameters/camelCaseAttributes/ kudos!

MongoDB Text Search find similar words

we have a database with names of wines and beer(s). We're using MongoDB and our document structure for each wine or beer company looks like this:
{
name: String //this is the company name
products: [
{
name: String //this is the product's name
...
}
],
...
}
I have created a text index on both the company name and product name, the text search is working fine but it's not working as expected. For example we'd like our app to be able to return results when someone searches for a product named Bellissima and the text they send is Belisima OR Bellisima or any variations of that word.
Is that even possible with MongoDB's text search? How can I make it smarter? I'm not a MongoDB expert but I know it has something to do with word stemming or something so is there a way I can modify that?? Or any other workaround to give the user most appropriate results? Because searching for Belisima doesn't return any results because that's not the exact word saved in database.

Mongoid, find object by searching by part of the Id?

I want to be able to search for my objects by searching for the last 4 characters of the id. How can I do that?
Book.where(_id: params[:q])
Where the param would be something like a3f4, and in this case the actual id for the object that I want to be found would be:
bc313c1f5053b66121a8a3f4
Notice the last for characters are what we searched for. How can I search for just "part" of my objects id? instead of having my user search manually by typing in the entire id?
I found in MongoDB's help docs, that I can provide a regex:
db.x.find({someId : {$regex : "123\\[456\\]"}}) // use "\\" to escape
Is there a way for me to search using the regular mongo ruby driver and not using Mongoid?
Usually, in Mongoid you can search with a regexp like you normally would with a string in your call to where() ie:
Book.where(:title => /^Alice/) # returns all books with titles starting with 'Alice'
However this doesn't work in your case, because the _id field is not stored as a string, but as an ObjectID. However, you could add (and index) a field on your models which could provide this functionality for you, which you can populate in an after_create callback.
<shameless_plug>
Alternatively, if you're just looking for a shorter solution to the default Mongoid IDs, I could suggest something like mongoid_token which makes it pretty easy to add shorter tokens/ids to your Mongoid documents.
</shameless_plug>

RESTful URL design for search

I'm looking for a reasonable way to represent searches as a RESTful URLs.
The setup: I have two models, Cars and Garages, where Cars can be in Garages. So my urls look like:
/car/xxxx
xxx == car id
returns car with given id
/garage/yyy
yyy = garage id
returns garage with given id
A Car can exist on its own (hence the /car), or it can exist in a garage. What's the right way to represent, say, all the cars in a given garage? Something like:
/garage/yyy/cars ?
How about the union of cars in garage yyy and zzz?
What's the right way to represent a search for cars with certain attributes? Say: show me all blue sedans with 4 doors :
/car/search?color=blue&type=sedan&doors=4
or should it be /cars instead?
The use of "search" seems inappropriate there - what's a better way / term? Should it just be:
/cars/?color=blue&type=sedan&doors=4
Should the search parameters be part of the PATHINFO or QUERYSTRING?
In short, I'm looking for guidance for cross-model REST url design, and for search.
[Update] I like Justin's answer, but he doesn't cover the multi-field search case:
/cars/color:blue/type:sedan/doors:4
or something like that. How do we go from
/cars/color/blue
to the multiple field case?
For the searching, use querystrings. This is perfectly RESTful:
/cars?color=blue&type=sedan&doors=4
An advantage to regular querystrings is that they are standard and widely understood and that they can be generated from form-get.
The RESTful pretty URL design is about displaying a resource based on a structure (directory-like structure, date: articles/2005/5/13, object and it's attributes,..), the slash / indicates hierarchical structure, use the -id instead.
Hierarchical structure
I would personaly prefer:
/garage-id/cars/car-id
/cars/car-id #for cars not in garages
If a user removes the /car-id part, it brings the cars preview - intuitive. User exactly knows where in the tree he is, what is he looking at. He knows from the first look, that garages and cars are in relation. /car-id also denotes that it belongs together unlike /car/id.
Searching
The searchquery is OK as it is, there is only your preference, what should be taken into account. The funny part comes when joining searches (see below).
/cars?color=blue;type=sedan #most prefered by me
/cars;color-blue+doors-4+type-sedan #looks good when using car-id
/cars?color=blue&doors=4&type=sedan #also possible, but & blends in with text
Or basically anything what isn't a slash as explained above.
The formula: /cars[?;]color[=-:]blue[,;+&], though I wouldn't use the & sign as it is unrecognizable from the text at first glance if that's your thing.
** Did you know that passing JSON object in URI is RESTful? **
Lists of options
/cars?color=black,blue,red;doors=3,5;type=sedan #most prefered by me
/cars?color:black:blue:red;doors:3:5;type:sedan
/cars?color(black,blue,red);doors(3,5);type(sedan) #does not look bad at all
/cars?color:(black,blue,red);doors:(3,5);type:sedan #little difference
possible features?
Negate search strings (!)
To search any cars, but not black and red:
?color=!black,!red
color:(!black,!red)
Joined searches
Search red or blue or black cars with 3 doors in garages id 1..20 or 101..103 or 999 but not 5
/garage[id=1-20,101-103,999,!5]/cars[color=red,blue,black;doors=3]
You can then construct more complex search queries. (Look at CSS3 attribute matching for the idea of matching substrings. E.g. searching users containing "bar" user*=bar.)
Conclusion
Anyway, this might be the most important part for you, because you can do it however you like after all, just keep in mind that RESTful URI represents a structure which is easily understood e.g. directory-like /directory/file, /collection/node/item, dates /articles/{year}/{month}/{day}.. And when you omit any of last segments, you immediately know what you get.
So.., all these characters are allowed unencoded:
unreserved: a-zA-Z0-9_.-~
Typically allowed both encoded and not, both uses are then equivalent.
special characters: $-_.+!*'(),
reserved: ;/?:#=&
May be used unencoded for the purpose they represent, otherwise they must be encoded.
unsafe: <>"#%{}|^~[]`
Why unsafe and why should rather be encoded: RFC 1738 see 2.2
Also see RFC 1738#page-20 for more character classes.
RFC 3986 see 2.2
Despite of what I previously said, here is a common distinction of delimeters, meaning that some "are" more important than others.
generic delimeters: :/?#[]#
sub-delimeters: !$&'()*+,;=
More reading:
Hierarchy: see 2.3, see 1.2.3
url path parameter syntax
CSS3 attribute matching
IBM: RESTful Web services - The basics
Note: RFC 1738 was updated by RFC 3986
Although having the parameters in the path has some advantages, there are, IMO, some outweighing factors.
Not all characters needed for a search query are permitted in a URL. Most punctuation and Unicode characters would need to be URL encoded as a query string parameter. I'm wrestling with the same problem. I would like to use XPath in the URL, but not all XPath syntax is compatible with a URI path. So for simple paths, /cars/doors/driver/lock/combination would be appropriate to locate the 'combination' element in the driver's door XML document. But /car/doors[id='driver' and lock/combination='1234'] is not so friendly.
There is a difference between filtering a resource based on one of its attributes and specifying a resource.
For example, since
/cars/colors returns a list of all colors for all cars (the resource returned is a collection of color objects)
/cars/colors/red,blue,green would return a list of color objects that are red, blue or green, not a collection of cars.
To return cars, the path would be
/cars?color=red,blue,green or /cars/search?color=red,blue,green
Parameters in the path are more difficult to read because name/value pairs are not isolated from the rest of the path, which is not name/value pairs.
One last comment. I prefer /garages/yyy/cars (always plural) to /garage/yyy/cars (perhaps it was a typo in the original answer) because it avoids changing the path between singular and plural. For words with an added 's', the change is not so bad, but changing /person/yyy/friends to /people/yyy seems cumbersome.
To expand on Peter's answer - you could make Search a first-class resource:
POST /searches # create a new search
GET /searches # list all searches (admin)
GET /searches/{id} # show the results of a previously-run search
DELETE /searches/{id} # delete a search (admin)
The Search resource would have fields for color, make model, garaged status, etc and could be specified in XML, JSON, or any other format. Like the Car and Garage resource, you could restrict access to Searches based on authentication. Users who frequently run the same Searches can store them in their profiles so that they don't need to be re-created. The URLs will be short enough that in many cases they can be easily traded via email. These stored Searches can be the basis of custom RSS feeds, and so on.
There are many possibilities for using Searches when you think of them as resources.
The idea is explained in more detail in this Railscast.
Justin's answer is probably the way to go, although in some applications it might make sense to consider a particular search as a resource in its own right, such as if you want to support named saved searches:
/search/{searchQuery}
or
/search/{savedSearchName}
I use two approaches to implement searches.
1) Simplest case, to query associated elements, and for navigation.
/cars?q.garage.id.eq=1
This means, query cars that have garage ID equal to 1.
It is also possible to create more complex searches:
/cars?q.garage.street.eq=FirstStreet&q.color.ne=red&offset=300&max=100
Cars in all garages in FirstStreet that are not red (3rd page, 100 elements per page).
2) Complex queries are considered as regular resources that are created and can be recovered.
POST /searches => Create
GET /searches/1 => Recover search
GET /searches/1?offset=300&max=100 => pagination in search
The POST body for search creation is as follows:
{
"$class":"test.Car",
"$q":{
"$eq" : { "color" : "red" },
"garage" : {
"$ne" : { "street" : "FirstStreet" }
}
}
}
It is based in Grails (criteria DSL): http://grails.org/doc/2.4.3/ref/Domain%20Classes/createCriteria.html
This is not REST. You cannot define URIs for resources inside your API. Resource navigation must be hypertext-driven. It's fine if you want pretty URIs and heavy amounts of coupling, but just do not call it REST, because it directly violates the constraints of RESTful architecture.
See this article by the inventor of REST.
In addition i would also suggest:
/cars/search/all{?color,model,year}
/cars/search/by-parameters{?color,model,year}
/cars/search/by-vendor{?vendor}
Here, Search is considered as a child resource of Cars resource.
There are a lot of good options for your case here. Still you should considering using the POST body.
The query string is perfect for your example, but if you have something more complicated, e.g. an arbitrary long list of items or boolean conditionals, you might want to define the post as a document, that the client sends over POST.
This allows a more flexible description of the search, as well as avoids the Server URL length limit.
RESTful does not recommend using verbs in URL's /cars/search is not restful. The right way to filter/search/paginate your API's is through Query Parameters. However there might be cases when you have to break the norm. For example, if you are searching across multiple resources, then you have to use something like /search?q=query
You can go through http://saipraveenblog.wordpress.com/2014/09/29/rest-api-best-practices/ to understand the best practices for designing RESTful API's
Though I like Justin's response, I feel it more accurately represents a filter rather than a search. What if I want to know about cars with names that start with cam?
The way I see it, you could build it into the way you handle specific resources:
/cars/cam*
Or, you could simply add it into the filter:
/cars/doors/4/name/cam*/colors/red,blue,green
Personally, I prefer the latter, however I am by no means an expert on REST (having first heard of it only 2 or so weeks ago...)
My advice would be this:
/garages
Returns list of garages (think JSON array here)
/garages/yyy
Returns specific garage
/garage/yyy/cars
Returns list of cars in garage
/garages/cars
Returns list of all cars in all garages (may not be practical of course)
/cars
Returns list of all cars
/cars/xxx
Returns specific car
/cars/colors
Returns lists of all posible colors for cars
/cars/colors/red,blue,green
Returns list of cars of the specific colors (yes commas are allowed :) )
Edit:
/cars/colors/red,blue,green/doors/2
Returns list of all red,blue, and green cars with 2 doors.
/cars/type/hatchback,coupe/colors/red,blue,green/
Same idea as the above but a lil more intuitive.
/cars/colors/red,blue,green/doors/two-door,four-door
All cars that are red, blue, green and have either two or four doors.
Hopefully that gives you the idea. Essentially your Rest API should be easily discoverable and should enable you to browse through your data. Another advantage with using URLs and not query strings is that you are able to take advantage of the native caching mechanisms that exist on the web server for HTTP traffic.
Here's a link to a page describing the evils of query strings in REST: http://web.archive.org/web/20070815111413/http://rest.blueoxen.net/cgi-bin/wiki.pl?QueryStringsConsideredHarmful
I used Google's cache because the normal page wasn't working for me here's that link as well:
http://rest.blueoxen.net/cgi-bin/wiki.pl?QueryStringsConsideredHarmful