I'm looking for suggestions on the best way to implement a full-text search on some static data on the iPhone.
Basically I have an app that contains the offline version of a web site, about 50MB of text, and I'd like for users to be able to search for terms. I figure that I should somehow build an table of ("word", reference_to_file_containing_word) or something, put that into either Core Data or just sqlite, index the "word" column, then have the search facility search the table for search terms and take the intersection of the sets of results for the terms or something.
That wouldn't allow people to search for phrases but it would be pretty easy and probably not too slow.
I'd like to just use existing SDK features for this. Should I use Core Data or sqlite?
Does anyone have any other ideas on how this could be done?
You want to place every word in the document in its own row in a database? That's going to take up even more space than the document itself.
I would recommend just searching through the text; regex is actually pretty fast. Otherwise, you could implement Boyer-Moore fairly easily.
[Edit] If you insist on creating an index of words, you can't beat a trie. It would be faster than using a database, and most likely take up less space than the documents themselves (unlike the database)
The answer is FTS3 for SQLite. Google it, there are many tutorials on how to get it working on iPhone.
And the easy way to use SQLite on iPhone is using FMDB.
Related
I know this topic was discusses a couple of times, but none of them represents the ultimate solution for me.
Situation
I'm designing a relational mysql database which later should hold multilingual content. You know this from the Wikipedia or Microsoft Tech Support Pages. The contents should be the same for every language. e.g If translations are missing the site offers you the same content automatically translated or in the languages which the information is available in. If some values are not set, it should fallback to the second or default browser language or translate it e.g. through google. Development environment is Zend.
My ideas so far are for Solving the Problem:
Two Primary Keys: (ID, Language)
Advantage: Easy Database Access through database abstraction layers.
Problem: Foreign Keys, Relations ships, Fallbacks
Columns with language suffix:
Advantage: DB Performance, No relational Problems.
Problem: Database abstraction layers cannot handle this?
Has any concept proven itself or is preferable over the other? Has anyone already created something like this and can share his experience with me? Does a modified Zend DB Controller exist for this situation? How do you link this information to a form?
Thank you for your help, hints and suggestions!
Kind regards,
Manuel
The second option would be not maintainable (this should be added on the minuses side). To actually add another language you'll need to modify table and abstraction layers. Sounds like a nightmare.
The first option seems much more promising but unfortunately there is a lot to do to make it work. However, from my experience this is rather typical solution, so I would not reinvent the wheel.
What I have to add is, language fallback should be done on the Zend side, database would miss some information. You may think of some kind of index table to hold information such as unique id of the contents and available languages. If you need to serve something, you would read such record, compare it against of Accept Languages and ask database again for valid contents (using the most suitable language). The only problem is, you would need to create such an index table somehow (the best way I see would be trigger on inserting contents to your content table).
A lot of work but the problem is not too easy.
I am working on the exact same problem right now.
Somehow it does not make sense to me to add everything into the same database. Lets say I want to go to the extreme and support some 50 languages this would just bloat my DB. So, I tend to keep my main DB in my main language and then introduce some Zend_Translate concept into it. Zend_Translate should give you the fallback solution you are looking for. While the main navigation and core design is not much of a problem for my web site my biggest concern right now is how to store all the main content and how to translate because these elements contain HTML among other things. For the main content I will probably use some alternate approach and use a separate DB with tables for each language.
My plattform will be a community driven database. So I actually gonna rely on humans translating it. You have to store the information anyways, so my first concern is not the database size or performance, but easy usability. So far my idea is to implement some structure as described above, not yet sure if i'll do it in doctrine or not.
Language decision:
Start, application gets users preset language, secondary language, english mother-tong of the article. Fetching the article from the database I will check the following for every column: 1. is the primary language available? 2. Is the secondary language available? 3. If neither of them, display article in mother-tong or english and offer the user to translate it with suggestions from the google translate api. I guess it's gonna be quite a bit of coating and manipulating controllers or building a business model doing this.
#tawfekov is something like this or similar easily realizable with doctrine?
I'm creating a simple application, first time working with SQLite.
I want a regular search box and to display results as a user types (is this possible)
And how to make the search to be able to support misspellings, if i wrote "Canda" to find me "Canada","Candy" or whatever i have in the DB similar to the search
any help or literature will be helpful
Concerning (1), you should take a look at UISearchDisplayController, in short it'll allow you to easily search and display results as you type.
For (2), my first thought would be to perhaps represent the "likely" misspellings in your data model? In addition it would be interesting to also augment this list of misspellings - Apple's own SMS app is doing something like this, so that it learns from your misspellings as you go along.
UISearchDisplayController will kick of searches as the user types in text and thus narrowing the results down continously.
To my best of knowledge fuzzy matching is not directly supported in SQLite even though the feature have been requested numerous times.
It requires a quote complex algoritm to do this effectively as detailed in this Wikipedia article.
There exists a module called fuzzystrmatch for PostgreSQL, but it does have it's limitations, especially with multibyte strings (like UTF-8) which are the native string formats for SQLite. This might give you a hint of a possible implementation.
Current database have 200 thousand record. I want to make iPad application , can I run 200 thousand record with sqlite ? I don't want to use sqlite because searching is too slow over 32000. Any search engine like lucene for iPhone SDK ? if we can run lucene in iPad that will be awesome because current project is base on lucene. Can you give me a suggestion ?
Thank You
I suggest to build you own fulltext index. I don't think SQLite on the iPhone supports triggers, but you can still use a similar algorithm than the H2 fulltext search implementation. The idea is to split the text data into words, and then add those to an table that is indexed on the word. The algorithm for building the index is relatively simple. The native H2 fulltext search doesn't have all the features the Lucene fulltext search has, but it's much simpler to implement.
I recently built an iPhone app with 86,000 rows and SQLite.
At first I didn't index my serach row and searches were taking about 1 second to execut on an iPod touch 2nd generation. Once I added my indexes searching was instant.
To be honest you might be able to get away with "like" queries on the field you are looking for? It might actually be good enough. I think you will find that once you add indexes to your data, the database is really going to grow in size and could make you app pretty big and putting a full blown search engine in there could be a real pain.
Here is some code to quickly do some tests for a SQLite database.
http://www.rvaidya.com/blog/iphone/2009/02/14/wrapper-classes-for-using-sqlite-on-iphonecocoa/
SQLiteResult *result = [SQLite query:#"SELECT * from test;"];
NSArray *row = [result.rows objectAtIndex:0];
NSString *firstValue = [row objectAtIndex:0];
This blog post describes a port of Lucene to Objective C. You may try to use the code, though it seems a bit dated.
I understand Lucene is written in Java - that'd mean you can't use it on the iPad.
Have you actually created indexes in your Sqlite database? In my experience Sqlite is really quite fast. If Core Data is too slow you could also try using the native C API.
I wanna store a list of people while each person has some pieces of info associated with him. for example location and phone number and e-mail address.
i wanna store in this list around 10,000 persons.
After that i want to search this list dynamically (after typing each letter , the database is searched for new matches to the string written in the search box)
[if there is a scientific name for this search process , let me know it please :) ]
My Question is What do you think i should use in my implementation for best performance?
SQLite,
XML, plist
???
and is there any tutorial about this kind of search ?
Thanks in advance
You should definitely be using SQLite for this, and using the CoreData abstraction layer would probably be a good ideas as well unless you are already a pro at SQLite calls. The documentation for Core Data would be a good place to start on how to do this. Apple provides several examples that are similar to what you are doing.
You should use a database for this. Searching in a database is a lot quicker than searching in a file. Not quite sure how much memory it would take but it might even be possible to load it all in memory if it is just a list of names. That is how it would be done on a desktop application for sure, but might not work on an iPhone which has a lot less memory.
SQLite for the win!
Is there any "Best Practice" approach to storing persistent static data for iPhone apps?
I have an app that reads a dictionary of approximately 1000 items many of which are arrays. I started out using a single plist for this and it has become somewhat unwieldy especially since much of the values are HTML strings.
Is there a better way for me to approach this? I plan on scaling this app significantly and I obviously don't want to change my approach midstream.
I've googled for iphone data storage and variants but have come up short for anything even touching on a best practice.
It sounds like you intend to distribute some data with your application. A property list is probably the easiest to maintain, but it will be loaded into memory all at once. This could eat up a lot of the device's memory.
An sqlite database, on the other hand, will load only the data you request. I'm not sure how your data is structured, but you could quite easily create key-value pairs with a single database table. (A single table with a key column and a value column) Then, if it were me, I'd write an Objective-C class to wrap the database queries so I can write easy statements like:
NSString *welcomeText = [[MyData sharedData] dataWithKey:#"WelcomeText"];
Getting the data into the database in the first place doesn't have to be difficult. You can use the command line sqlite3 utility to bulk load your data. There's a command called .import that will import your data from a text file.
Hope this gets you moving in the right direction!
I'd go with a sqlite solution. The apps I am working on now, which are just apps to help me learn iPhone development, mostly all use sqlite. I use the sqlite plugin for firefox to help with maintaining the database, which works surprisingly well. https://addons.mozilla.org/en-US/firefox/addon/5817
As Alex suggested using a wrapper class would also be the best way to go.
Don't forget with 3.0 you can use a CoreData layer around SQLlite which may make it more appealing to you.
If you don't need to store any relational information about your data, why not just use files? There will be some wasted filesystem space, but plain files might be the most memory and CPU efficient solution, depending on the size and number of your items.
I've never developed an iPhone app but I have played around in the filesystem. I have seen sqlite databases floating around various places of the phone. I'm pretty sure it uses a sqlite database to store your calendar entries.
I would use sqlite. It is already there, easy to use, and will provide the most flexible path for expansion in the future.
I use sqlite for static data in my iPhone apps all the time.
All I did was save state when the app is shut down. I used a file for that.
sqlite sounds perfect for your app. sqlite is pretty easy. I've used it in Adobe AIR apps.