Importing large datasets on iPhone using CoreData - iphone

Question

I'm facing very annoying problem. My iPhone app is loading it's data from a network server. Data are sent as plist and when parsed, it neeeds to be stored to SQLite db using CoreData.
Issue is that in some cases those datasets are too big (5000+ records) and import takes way too long. More on that, when iPhone tries to suspend the screen, Watchdog kills the app because it's still processing the import and does not respond up to 5 seconds, so import is never finished.
I used all recommended techniques according to article "Efficiently Importing Data" http://developer.apple.com/mac/library/DOCUMENTATION/Cocoa/Conceptual/CoreData/Articles/cdImporting.html and other docs concerning this, but it's still awfully slow.
Solution I'm looking for is to let app suspend, but let import run in behind (better one) or to prevent attempts to suspend the app at all. Or any better idea is welcomed too.
Any tips on how to overcome these issues are highly appreciated!
Thanks

Answer 1

Instead of pushing plist files to the phone, you might want to send ready to use sqlite files. This has many advantages:
no need to import on the phone
more compact
If you always replace the whole content simply overwrite the persistent store in the device. Otherwise you may want to maintain an array as plist with all sqlites you have downloaded and then use this to add all stores to the persistentStoreCoordinator.
Bottom line: use several precompiled sqlite files and add them to the persistentStoreCoordinator.
You can use the iPhone Simulator to create those CoreData-SQLite-Stores or use a standalone Mac app. You will need to write both of those yourself.

Answer 2

First, if you can package the data with the app that would be ideal.
However, assuming you cannot do that then I would do then following:
Once the data is downloaded break it into multiple files before import.
Import on a background thread, one file at a time.
Once a file has been imported and saved, delete the import file.
On launch, look for those files waiting to be processed and pick up where you left off.
Ideally sending the data with the app would be far less work but the second solution will work and you can fine-tune the data break up during development.

Answer 3

I solved a similar problem by putting the insert processing in a background thread. But first I created a progress alert so the user couldn't manipulate the data store while it was inserting the entries.
This is basically the ViewControllers viewDidLoad
- (void)viewDidLoad
{
[super viewDidLoad];
NSError *error = nil;
if (![[self fetchedResultsController] performFetch:&error]) {
NSLog(#"Unresolved error %#, %#", error, [error userInfo]);
abort();
}
// Only insert those not imported, here I know it should be 2006 entries
if ([self tableView:nil numberOfRowsInSection:0] != 2006) {
// Put up an alert with a progress bar, need to implement
[self createProgressionAlertWithMessage:#"Initilizing database"];
// Spawn the insert thread making the app still "live" so it
// won't be killed by the OS
[NSThread detachNewThreadSelector:#selector(loadInitialDatabase:)
toTarget:self
withObject:[NSNumber numberWithInt:[self tableView:nil
numberOfRowsInSection:0]]];
}
}
The insert thread was done like this
- (void)loadInitialDatabase:(NSNumber*)number
{
NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
int done = [number intValue]+1; // How many done so far
// I load from a textfile (csv) but imagine you should be able to
// understand the process and make it work for your data
NSString *file = [NSString stringWithContentsOfFile:[[NSBundle mainBundle]
pathForResource:#"filename"
ofType:#"txt"]
encoding:NSUTF8StringEncoding
error:nil];
NSArray *lines = [file componentsSeparatedByString:#"\n"];
float num = [lines count];
float i = 0;
int perc = 0;
for (NSString *line in lines) {
i += 1.0;
if ((int)(i/(num*0.01)) != perc) {
// This part updates the alert with a progress bar
// setProgressValue: needs to be implemented
[self performSelectorOnMainThread:#selector(setProgressValue:)
withObject:[NSNumber numberWithFloat:i/num]
waitUntilDone:YES];
perc = (int)(i/(num*0.01));
}
if (done < i) // keep track of how much done previously
[self insertFromLine:line]; // Add to data storage...
}
progressView = nil;
[progressAlert dismissWithClickedButtonIndex:0 animated:YES];
[pool release];
}
It's a bit crude this way, it tries to init the data storage from where it left of if the user happend to stop it the previous times...

Answer 4

I had a similar problem importing many objects into CoreData. Initially i was doing a save on the managed object context after every object i wished to create & insert.
What you should do is create/initialize each object you want to save in CoreData, and after you have looped through all your remote data + created the objects, do a managed object context save.
I guess you could look at this as doing doing a transaction in a SQLite database: begin transaction, do lots of inserts/updates, end transaction.
if this still is too lengthy, just thread the darn task and prevent user interaction until complete

Answer 5

Is there any way you can pack the data ahead of time - say during development? And when you push the app to the store, some of the data is already there? That'll cut down on the amount of data you have to pull, thus helping to solve this issue?
If the data is time sensitive, or not ready, or for whatever reason you can't do that, could you compress the data using zlib compression before you ship it over the network?
Or is the problem that the phone dies doing 5K+ inserts?

Answer 6

I imagine you aren't showing all 5K records to the client? I'd recommend doing all of the aggregation you need on the server, and then only sending the necessary data to the phone. Even if this involves generating a few different data views, it'll still be orders of magnitude faster than sending (and then processing) all those rows in the iPhone.
Are you also processing the data in a separate (non event/ui) thread?

Answer 7

Any chance you can setup your server side to expose a RESTful web service for processing your data? I had a similar issue and was able to expose my information through a RESTful webservice. There are some libraries on the iphone that make reading from a webservice like that very easy. I chose to request JSON from the service and used the SBJSON library on the iphone to quickly take the results I got and convert them to dictionaries for easy use. I used the ASIHTTP library for making the web requests and queueing up follow up requests and making them run in the background.
The nice thing about REST is that it a built in way for you to grab batches of information so that you don't need to arbitrarily figure out how to break up your files you want to input. You just setup how many records you want to get back, and the next request you skip that many records. I don't know if that is even an option for you, so I'm not going into a lot of code examples right now, but if it is possible, it may be a smooth way to handle it.

Answer 8

Lets accept that Restful (lazy loading) is not an option... I understand you want to replicate. If the load problem is of the type 'less and less rows loading in more and more time) then in psuedo code...
[self sQLdropIndex(OffendingIndexName)]
[self breathInOverIP];
[self breathOutToSQLLite];
[self sQLAddIndex(OffendingIndexName)]
This should tell you lots.

Answer 9

I work on an app that regularly has to process 100K inserts, deletes, and updates with Core Data. If it is choking on 5K inserts, there is some optimization to be done.
Firstly, create some NSOperation subclass for processing the data. Override its -main method to do the processing. This method is, however, not guaranteed to run on the main thread. Indeed, its purpose is to avoid executing costly code on the main thread which would affect the user experience by making it freeze up grossly. So within the -main method, you need to create another managed object context which is the child of your main thread's managed object context.
- (void)main
{
NSManagedObjectContext *ctx = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
[ctx setPersistentStoreCoordinator:mainManagedObjectContext.persistentStoreCoordinator];
[ctx setUndoManager:nil];
// Do your insertions here!
NSError *error = nil;
[ctx save:&error];
}
Given your circumstances, I don't believe you need an undo manager. Having one will incur a performance penalty because Core Data is tracking your changes.
Use THIS context to perform all of your CRUD actions in the -main method, then save that managed object context. Whatever owns your main thread's managed object context must register to respond to the NSNotification named NSManagedObjectContextDidSaveNotification. Register like so:
[[NSNotificationCenter defaultCenter] addObserver:self selector:#selector(mocDidSaveNotification:) name:NSManagedObjectContextDidSaveNotification object:nil];
Then define that selector:
- (void)mocDidSaveNotification:(NSNotification *)notification
{
NSManagedObjectContext *ctx = [notification object];
if (ctx == mainManagedObjectContext) return;
[mainManagedObjectContext mergeChangesFromContextDidSaveNotification:notification];
}
When all of this comes together, it will allow you to perform long-running operations on background threads without blocking the UI thread. There are several variations of this architecture, but the central theme is this: processing on BG thread, merge on main thread, update your UI. Some other things to keep in mind: (1) keep an autorelease pool around during your processing and drain it every so often to keep your memory consumption down. In our case, we do it every 1000 objects. Adjust for your needs, but keep in mind that draining can be expensive depending on the amount of memory required per object, so you don't want to do it too often. (2) try to pare your data down to the absolute minimum that you need to have a functional app. By reducing the amount of data to parse, you reduce the amount of time required to save it. (3) by using this multithreaded approach, you can concurrently process your data. So create 3-4 instances of your NSOperation subclass, each of which processes only a portion of the data so that they all run concurrently, resulting in a smaller amount of real time consumed for parsing the data set.