Loading 10k+ rows in iphone SQLite (FMDB) - iphone

Question

I am creating a dictionary app and I am trying to load the terms into an iphone dictionary for use. The terms are defined from this table (SQLite):
id -> INTEGER autoincrement PK
termtext -> TEXT
langid -> INT
normalized -> TEXT
Normalized is used because I am writing in GREEK and I don't have icu on the sqlite engine for searching diacritics, so I am making a termtext diacritics/case insensitive. It is also the main search field, in contrast of termtext which could be the "view" field.
I have defined an class (like a POJO) like this:
terms.h
#import <Foundation/Foundation.h>
#interface Terms : NSObject {
NSUInteger termId; // id
NSString* termText; // termtext
NSUInteger langId; // langid
NSString* normalized; // normalized
}
#property (nonatomic, copy, readwrite) NSString* termText;
#property (nonatomic, copy, readwrite) NSString* normalized;
#property (assign, readwrite) NSUInteger termId;
#property (assign, readwrite) NSUInteger langId;
#end
terms.c
#import "Terms.h"
#implementation Term
#synthesize termId;
#synthesize termText;
#synthesize langId;
#synthesize normalized;
#end
Now in my code I use FMDB as the wrapper for the SQLite database. I load the terms using the following code:
[... fmdb defined database as object, opened ]
NSMutableArray *termResults = [[[NSMutableArray alloc] init] autorelease];
FMResultSet *s = [database executeSQL:#"SELECT id, termtext, langid, normalized FROM terms ORDER BY normalized ASC"];
while ([s next]) {
Term* term = [[Terms alloc] init];
term.termId = [s intForColumn:#"id"];
[... other definitions]
[termResults addObject:term];
[term release];
}
The whole termResults is then loaded to a UITableView (on viewdidload) but the loading takes up to 5 seconds every time I start my app. Is there any way to speedup that process? I have indexed id, termText and normalized on SQLite.
*** UPDATE: added cellForRowAtIndexPath ****
[.. standard cell definition...]
// Configure the cell
Term* termObj = [self.termResults objectAtIndex:indexPath.row];
cell.textLabel.text = termObj.termText;
return cell;

Answer 1

Since you seem to load the whole DB into memory anyway (and think it is a must) the whole point of using SQLite is almost gone.
So if the data is static (does not change), I would turn the DB into objects once, then serialize the objects (implement the NSCoding protocol) and store this array (or better yet, dictionary) using writeToFile:atomically:. Once you have that file (do that during development) you can easily load it at runtime with arrayWithContentsOfFile: which should be faster in this case.

Answer 2

Loading that much data into your app is going to take time. The best approach to take would be to load the data from the db on a separate thread to the main application thread. As each item is loaded, post a message back to the main thread to add an item to the array backing the table view. This way, your app load time will be short and your UI will be responsive whilst the data is loading from the db. Here's the basic idea:
NSMutableArray *termResults = [NSMutableArray array];
// Load each item from the db on a separate thread using a block
dispatch_queue_t globalQueue = dispatch_get_global_queue();
dispatch_async(globalQueue, ^() {
// Init the database
...
// Load the items from the db
FMResultSet *s = [database executeSQL:#"SELECT id, termtext, langid, normalized FROM terms ORDER BY normalized ASC"];
while ([s next]) {
Term* term = [Term new];
term.termId = [s intForColumn:#"id"];
// Add the loaded item to termResults on the main thread
// and refresh the table view
dispatch_queue_t mainDispatchQueue = dispatch_get_main_queue();
dispatch_async(mainDispatchQueue, ^() {
// Add the loaded item to termResults
[termResults addObject:term];
// Refresh the table view. This will only reload the visible items.
[tableView reloadData];
});
[term release];
}
});
Hope this helps.

Answer 3

Not a problem to read 10K records from sqlite into UITableView. First, I couldn't see reasons to do that. Second, you couldn't avoid time gap due to a lot of memory allocations. Memory allocation is a system call. And system calls are often slow. During load ios will send memory warnings to run apps resulting in longer gap. What will you do if you will need a million records? sqlite is used to avoid such memory data structures at all. If you see problems using unicode in sqlite on iphone (icu) - read here. I use this technic in several of my apps and all of them were approved to AppStore without problems.