I am running into intermittent, hard-to-reproduce errors on my iPhone app, so I am checking my assumptions around concurrency.
Running AFNetworking v0.10.x, I have the following network call:
[self postPath:#"/myEndPoint"
parameters:params
success:^(AFHTTPRequestOperation *request, id response)
{
AppDelegate *appDelegate = (AppDelegate*)[[UIApplication sharedApplication] delegate];
// do stuff with object context here
[appDelegate.objectContext save];
}
]
AppDelegate *appDelegate = (AppDelegate*)[[UIApplication sharedApplication] delegate];
// do other stuff with object context
[appDelegate.objectContext save];
In my AppDelegate:
-(NSManagedObjectContext*) objectContext
{
if(nil == _objectContext)
{
... set up sqlite persistent store coordinator and object model ...
_objectContext = [[NSManagedObjectContext alloc] init];
[_objectContext setPersistentStoreCoordinator:persistentStoreCoordinator];
[_objectContext setMergePolicy:NSOverwriteMergePolicy];
}
return _objectContext;
}
Is it possible, in this scenario, to end up with concurrency problems? Or, in other words, is AFNetworking's API thread-safe? I thought the NSOverwriteMergePolicy would cover me for conflicts, but crashing persists (albeit intermittently).
AFNetworking's callbacks are executed on the main thread. As a result, they are 'thread-safe', because there is only one thread that is interacting with CoreData. If you only have a single managed object things will be straightforward.
From Apple:
Tasks added to this queue are performed serially on the main thread itself. Therefore, you can use this queue as a synchronization point for work being done in other parts of your application.
There are still lots of considerations when using multi-threaded CoreData and multiple managed object contexts, and for those I refer you to rsswtmr's excellent answer, which doesn't correctly answer my question, but provides links to lots of good information.
You can't have multiple threads working on the same object context. Think through how Core Data might be part-way through changing/committing data while another change comes through on another thread. You need to create a separate context per thread and merge changes when it's safe/appropriate. The NSOverwriteMergePolicy will simply save you from having to manually handle conflicts at merge time.
Look here for a great explanation of threading Core Data access.
Related
I am creating and adding a number of managed objects to Core Data from a background queue. My understanding was that I could not access the context from the background thread so I was using performBlock to schedule adding to Core Data back onto the same queue that the context was created on. This works just fine ...
My question is during testing I noticed that by removing [moc performBlock:^{ ... }]; the application still performs as expected (maybe even fractions of a second faster) Do I need the performBlock? I would assume I do and its just working (for now :) in a none-threadsafe fashion, I just wanted to check to make sure my understanding is not flawed.
dispatch_queue_t backgroundQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0);
dispatch_async(backgroundQueue, ^{
// GET DATA
// PROCESS DATA
NSManagedObjectContext *context = [[self managedDocument] managedObjectContext];
[moc performBlock:^{
// ADD TO CORE DATA
[Core createRodInContext:context withDictionary:fuelRodDictionary];
}];
});
EDIT: Added implementation for createRodInContext ...
+ (Rod *)createRodInContext:(NSManagedObjectContext *)context withDictionary:(NSDictionary *)dictionary {
// CREATE
Rod *rod = [NSEntityDescription insertNewObjectForEntityForName:#"Rod" inManagedObjectContext:context];
// POPULATE
[neo setDataCode:[dictionary objectForKey:#"dataCode"]];
[neo setDataName:[dictionary objectForKey:#"dataName"]];
[neo setDataReference:[dictionary objectForKey:#"dataReference"]];
...
return rod;
}
In the background thread you have to use [moc performBlock:^{ ... }] to insert (and populate) a managed object in the main managed object context.
Omitting the performBlock means that you use the managed object context (which was created on the main thread) also in a different thread (which is associated with the background queue).
This might work by chance, but as soon as the main thread accesses the MOC in the same moment as your background thread, the results are unpredictable because (as you already said), a MOC is not thread-safe.
See also Concurrency Support for Managed Object Contexts
in the Core Data Release Notes for OS X v10.7 and iOS 5.0:
Confinement (NSConfinementConcurrencyType).
This is the default. You
promise that context will not be used by any thread other than the one
on which you created it.
But also for the other concurrency types (private queue, main queue), you always have to use performBlock (or performBlockAndWait) unless your code is already executing on the queue associated with the MOC.
Currently I'm using Restkit to control all my (Core-) data in my app. I'm using it to keep in sync with the server using RKManagedObjectMapping and I use [myMyNSManagedObject createEntitity] together with [[RKObjectManager §sharedManager].objectStore save] to manually edit items within Grand Central Dispatch.
Is there any recommendation to do this in this or an other way? Because sometimes the app freezes in a deadlock executing this code of Restkit
+ (NSArray*)objectsWithFetchRequest:(NSFetchRequest*)fetchRequest {
NSError* error = nil;
NSArray* objects = [[self managedObjectContext] executeFetchRequest:fetchRequest error:&error];
if (objects == nil) {
RKLogError(#"Error: %#", [error localizedDescription]);
}
return objects;
}
with that
- (NSError*)save {
NSManagedObjectContext* moc = [self managedObjectContext];
NSError *error;
#try {
if (![moc save:&error]) {
if (self.delegate != nil && [self.delegate respondsToSelector:#selector(managedObjectStore:didFailToSaveContext:error:exception:)]) {
…
in parallel. Before I switched to Restkit I put a "context performBlockAndWait" around each entity-editing code and was on the safe side with no deadlocks. I have no other NSManagedObjectContext or something created by myself, all comes from Restkit.
In my case, the problem was that I was passing NSManagedObjects across thread boundaries, and using them on threads other than the ones on which they were fetched from their respective NSManagedObjectContext. It was really subtle in my case, as I knew that I wasn't supposed to do this, but did it accidentally anyways. Instead of passing the managed objects, I started passing the NSManagedObjectIDs (and then fetching from the local thread's MOC), and haven't encountered a deadlock since. I'd recommend you do a deep scan of your code to make sure you are only using managed objects in the threads that spawned them.
We've encountered this exact problem in our app. Basically, CoreData nested contexts are very buggy in iOS5, and thus don't work as advertised. This has many manifestations, but one of them is the problem described above, deadlocking a fetch request vs. a background operation. This is well documented, instructive quote:
NSFetchedResultsController deadlocks
You never want your application to deadlock. With NSFetchedResultsController and nested contexts, it’s pretty easy to do. Using the same UIManagedDocument setup described above, executing fetch requests in the private queue context while using NSFetchedResultsController with the main queue context will likely deadlock. If you start both at about the same time it happens with almost 100% consistency. NSFetchedResultsController is probably acquiring a lock that it shouldn’t be. This has been reported as fixed for an upcoming release of iOS.
This SO answer has a possible fix that keeps nested contexts. I've seen others like it, basically liberally applying -performAndWait: calls. We haven't tried that yet (iOS5 has single digit percentages in our user base). Otherwise, the only other "fix" we know right now is abandoning nested contexts for iOS5 (cf. this SO answer).
That CoreData continues to be fundamentally broken for multithreading (in iOS 5) is inexcusable on Apple's part. You can make a good case now that trusting Apple on CoreData (multithreading, iCloud) seems to be opening a bag of pain.
iOS 5 introduced a new way to quickly fetch data on a background thread by initializing the MOC using NSPrivateQueueConcurrencyType and then doing the fetch in performBlock:
One of the rules of thumb of Core Data has been that you can not share a managed object between threads/queues. Is it still the case with performBlock:? Is the following:
[context performBlock:^{
// fetch request code
NSArray *results = [context executeFetchRequest:request error:nil];
dispatch_async(dispatch_get_main_queue(), ^(void) {
Class *firstObject = [results objectAtIndex:0];
// do something with firstObject
});
}];
still unacceptable since I'm sharing my results array/objects between the bg queue and the main queue? Do I still need to use the managed object IDs to do that?
When you use NSPrivateQueueConcurrencyType you need to do anything that touches that context or any object belonging to that context inside the -performBlock: method.
Your code above is illegal since you're passing those objects back to the main queue. The new API helps you in solving this, though: You create one context that's associated with the main queue, i.e. with NSMainQueueConcurrencyType:
// Assume we have these two context (They need to be set up. Assume they are.)
NSManagedObjectContext *mainMOC = [[[NSManagedObjectContext alloc] initWithConcurrencyType:NSMainQueueConcurrencyType] autorelease];
NSManagedObjectContext *backgroundMOC = [[[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType] autorelease];
// Now this can safely be called from ANY thread:
[backgroundMOC performBlock:^{
NSArray *results = [backgroundMOC executeFetchRequest:request error:nil];
for (NSManagedObject *mo in results) {
NSManagedObjectID *moid = [mo objectID];
[mainMOC performBlock:^{
NSManagedObject *mainMO = [mainMOC objectWithID:moid];
// Do stuff with 'mainMO'. Be careful NOT to use 'mo'.
}];
}
}];
This gets less confusing if you move the inner [mainMOC performBlock:] call into its own method. You may also want to pass an array of object IDs back to the main thread's context in stead of executing a block for each object ID. It depends on your needs.
As Daniel Eggert explains, this is definitely still the case. The exception is for NSMainQueueConcurrencyType, where you can also use the managed object context and objects safely on the main thread (as well as from other threads via the performBlock mechanism). The usefulness of this cannot be understated!
iOS 5 also introduced the concept of parent contexts, which also hugely simplify background operations and remove the need to worry about using notifications to propogate changes between threads.
The WWDC 2012 video "Session 214 - Core Data Best Practices" goes into a lot more detail on both subjects and is very comprehensive. The video is essential viewing for anyone using Core Data.
I have perused all related threads on SO but still confused about how to make changes to core data objects from multiple threads without having to save after every change.
I am working on an app that talks to the server constantly. The app uses Core Data for storage and NSFetchedResultsController is used in a few view controllers to fetch data from the persistence store. Usually when the user performs an action, a network request will be triggered. Before the network request is sent, usually some changes should be made to relevant Core Data objects, and upon server response, more changes would be made to those Core Data objects.
Initially all the Core Data operations were done on the main thread in the same NSManagedObjectContext. It was all well except that when network traffic is high, the app can become unresponsive for several seconds. Obviously that's not acceptable so I looked into moving some Core Data operations to run in the background.
The first approach I tried was to create an NSOperation object to process each network response. Inside the main method of the NSOperation object, I set up a dedicated MOC, make some changes, and commit the changes at the end.
- (void)main
{
#try {
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
// Create a dedicated MOC for this NSOperation
NSManagedObjectContext * context = [[NSManagedObjectContext alloc] init];
[context setPersistentStoreCoordinator:[APP_DELEGATE persistentStoreCoordinator]];
// Make change to Core Data objects
// ...
// Commit the changes
NSError *error = nil;
if ([context hasChanges] && ![context save:&error]) {
NSLog(#"Unresolved error %#, %#", error, [error userInfo]);
}
// Release the MOC
[context release];
// Drain the pool
[pool drain];
}
#catch (NSException *exception) {
// Important that we don't rethrow exception here
NSLog(#"Exception: %#", exception);
}
}
The MOC on the main thread is registered for the NSManagedObjectContextDidSaveNotification.
[[NSNotificationCenter defaultCenter] addObserver:self
selector:#selector(backgroundContextDidSave:)
name:NSManagedObjectContextDidSaveNotification
object:nil];
So when the background context commits changes, the main MOC will be notified and then will merge in the changes:
- (void)backgroundContextDidSave:(NSNotification *)notification
{
// Make sure we're on the main thread when updating the main context
if (![NSThread isMainThread]) {
[self performSelectorOnMainThread:#selector(backgroundContextDidSave:) withObject:notification waitUntilDone:NO];
return;
}
// Merge the changes into the main context
[[self managedObjectContext] mergeChangesFromContextDidSaveNotification:notification];
}
However, as I mentioned earlier, I also need to make changes to Core Data objects from the main MOC. Each change is usually very small (e.g. update one instance variable in an object) but there can be many of them. So I really don't want to save the main MOC after every single change. But if I don't do that, I run into problems when merging changes from background MOC to the main MOC. Merge conflicts occur since both MOCs have unsaved changes. Setting the merge policy doesn't help either, since I want to keep changes from both MOCs.
One possibility is to register the background MOC with the NSManagedObjectContextDidSaveNotification as well, but that approach smells like bad design for me. And I would still need to save the main MOC after every single change.
The second approach I tried was to do all Core Data changes from a dedicated background context running on a permanent background thread.
- (NSThread *)backgroundThread
{
if (backgroundThread_ == nil) {
backgroundThread_ = [[NSThread alloc] initWithTarget:self selector:#selector(backgroundThreadMain) object:nil];
// Actually start the thread
[backgroundThread_ start];
}
return backgroundThread_;
}
// Entry point of the background thread
- (void)backgroundThreadMain
{
NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
// We can't run the runloop unless it has an associated input source or a timer, so we'll just create a timer that will never fire.
[NSTimer scheduledTimerWithTimeInterval:DBL_MAX target:self selector:#selector(ignore) userInfo:nil repeats:YES];
[[NSRunLoop currentRunLoop] run];
// Create a dedicated NSManagedObjectContext for this thread.
backgroundContext_ = [[NSManagedObjectContext alloc] init];
[backgroundContext_ setPersistentStoreCoordinator:[self persistentStoreCoordinator]];
[pool drain];
}
So whenever I need to make Core Data changes from the main thread, I have to get the objectID from the main thread, and pass to the background thread to perform the change. When the background context saves, the changes will then be merged back to the main MOC.
- (void)addProduct:(Product *)product toCatalog:(Catalog *)catalog;
would change to:
- (void)addProduct:(NSManagedObjectID *)productObjectId toCatalog:(NSManagedObjectID *)catalogObjectId
{
NSArray * params = [NSArray dictionaryWithObjects:productObjectId, catalogObjectId, nil];
[self performSelector:(addProductToCatalogInBackground:) onThread:backgroundThread_ withObject:params waitUntilDone:NO];
}
But this seems so convoluted and ugly. Writing code like this seems to negate the usefulness of using Core Data in the first place. Also, I would still have to save the MOC after every single change since I can't get objectId for a new object without saving it to the datastore first.
I feel that I am missing something here. I really hope someone can shed some light on this. Thanks.
A NSManagedObjectContext is just a scratch pad. The act of saving it moves the changes in one scratch pad down to the NSPersistentStoreCoordinator and potentially down to disk. The only way one MOC can know about the changes from another MOC is through the NSPersistentStoreCoordinator. Therefore the save is required. However, saving is a LOT less expensive in the next release of iOS.
If you must be iOS4 or less compliant then saving is the only option. However you can load up the saves and do them less frequently depending on the design of your application. If you are importing data then save when the import is complete or in logical units inside of the import. There is no need to save after every entry, that is wasteful.
BTW, I would suggest using NSOperation instances instead of working with NSThread instances directly. They are easier to work with and will perform better.
Also, you do not need to wrap Objective-C code in try/catch blocks. Very few things will throw an exception; especially on iOS.
Lastly, I would suggest taking a look at my post on CIMGF about importing on a background thread.
The current app I'm developing for the iPad involves handling many network requests and persisting the processed results in core data.
The scenario is follows - the application needs to download images for objects I'm displaying in a grid view, which can show a total of 30 objects. Each object can consist of up to 15 png images (also in a grid). Due to the way the server is implemented (meaning I didn't implement it and can't change it easily), each image must be requested separately, so I need to make up to 15 requests per object versus just 1 request to download all 15 images.
For each object, I'm currently using an ASINetworkQueue to queue up the 15 image requests. Once the queue finishes, I create a thumbnail snapshot of the object with its images to display in the grid, then persist all the png files to core data.
I'm currently running everything on the main thread except the network requests which are handled by ASI asynchronously, but since there are so many requests, the app UI is essentially locked until all the requests are processed and results saved to core data.
One solution I came across was doing the core data operations and writes in a separate thread or using grand central dispatch. Another is to only download the images for the visible objects, and download the rest when the user scrolls down.
I'm looking for other suggestions to help keep the main ui responsive, or better ways to structure the network and core data operations. Thanks.
First of all, avoid storing large blobs in Core Data, saving the thumbnails is fine (although you should optimize your model for it), but you should store the full image once it's reconstructed in the Documents folder.
You should definitely use a queue, either NSOperationQueue or ASI network queue. I do something similar in my app which has multiple dependencies. So, for each of the 30 objects you need to have a block (or work function) get called when the 15 images have downloaded. You ideally want to do this work off the main thread. Put all these requirements together, and I'd say you need at least two queues, one for network requests and one for worker blocks, and you should use NSBlockOperations which makes the whole thing much easier. So, the code would be something like this...
// Loop through the objects
for (NSArray *objectParts in objectsToDownload) {
// Create our Object
Object *obj = [Object insertIntoManagedObjectContext:self.moc];
// This is the block which will do the post processing for the object
NSBlockOperation *processBlock = [NSBlockOperation blockOperationWithBlock:^{
// Do post processing here, be very careful with multi-threading CoreData
// it's likely you'll need some class to dispatch you MOCs which have all
// all the observers set up.
// We're gonna assume that all the sub-images have been stored in a instance
// variable:
[obj performPostProcessing];
}];
// Given the list of 15 images which form each object
for (NSURL *part in objectParts) {
// Create the ASI request for this part
ASIHTTPRequest *request = [ASIHTTPRequest requestWithURL:part];
// Configure the request
[request setDelegate:self];
[request setDidFailSelector:#selector(partRequestDidFail:)];
[request setDidFinishSelector:#selector(partRequestDidFinish:)];
// Store the object in the UserInfo dictionary
NSMutableDictionary *userInfo = [NSMutableDictionary dictionaryWithObjectsAndKeys:obj, #"Object", nil];
[request setUserInfo:userInfo];
// Add it as a dependency
[processBlock addDependency:request];
// Add it to our network queue
[networkQueue addOperation:request];
}
// Add the processBlock to our worker queue
[workerQueue addOperation:processBlock];
}
Then you'll also need to write the delegate methods, the didFinish one would look something like this...
- (void)partRequestDidFinish:(ASIHTTPRequest *)request {
// Remember this is one the main thread, so any heavy lifting should be
// put inside a block operation, and queued, which will complicate the
// dependencies somewhat, but is possible.
// Get the result data
NSData *data = [request responseData];
// Get the object that it belongs to from the user info dic
Object *obj = [[request userInfo] objectForKey:#"Object"];
// Keep track of the partial data in the object
[obj storePartialDataForPostProcessing:data];
}
And all of that would go into your class which connects to your server and creates your objects, so it's not a view controller or anything, just a regular NSObject subclass. It will need to have two queues, a managed object context (and more than likely a method which returns another MOC for you to use in threads, something like this:
// Fends a MOC suitable for use in the NSBlockOperations
- (NSManagedObjectContext *)moc {
// Get a blank managed object context
NSManagedObjectContext *aContext = [[UIApplication sharedApplication] managedObjectContext;
[aContext setMergePolicy:NSMergeByPropertyStoreTrumpMergePolicy];
NSNotificationCenter *nc = [NSNotificationCenter defaultCenter];
[nc addObserver:self selector:#selector(mergeChangesFromMOC:) name:NSManagedObjectContextDidSaveNotification object:aContext];
return aContext;
}
- (void)mergeChangesFromMOC:(NSNotification *)aNotification {
#try {
[self.managedObjectContext mergeChangesFromContextDidSaveNotification:aNotification];
NSNotificationCenter *nc = [NSNotificationCenter defaultCenter];
[nc removeObserver:self name:NSManagedObjectContextDidSaveNotification object:[aNotification object]];
}
#catch (NSException * e) {
NSLog(#"Stopping on exception: %#", [e description]);
}
#finally {}
}
You'll also need to hook in some way of monitoring progress, re-queuing failed downloads, cancelling, and saving the MOC at the end. Re-queuing failed downloads is quite tricky. Anyway, hope that helps.
So, just to clarify, in your delegate method, you'd store the downloaded image in a temporary instance variable on your Object. Then when all 15 dependencies finish, you can access that instance variable and do your work.
To kick things off, 1 Queue should be enough for all the image requests.
What you might want to do is keep references to the request so you can cancel them if the object is no longer needed.
For the locking, there's a few things to consider with images:
They are compressed, so they need to be inflated before being UIImages, that is very heavy on the CPU.
If you ever want to write to the filesystem, that process is locking. Do it in another Queue to avoid locking.
It's never a good idea to store Blob into CoreData, store the file path as a string in Core Data and fetch it from disk using a queue
Just using 3 different NSOperationQueue will make your app much more responsive:
1 for ASIHTTPRequests (Don't create a new one, use the default with startAsynchronous)
1 for image writing to disk
1 for image fetching from disk
Since you will display image to view, why don't you SDwebImage:
SDImageCache manages an asynchronous download queue, ties the downloader with the image cache store, maintains a memory cache and an optional disk cache. Disk cache write operations are performed asynchronous so it doesn't add unnecessary latency to the UI.
[imageView setImageWithURL:[NSURL URLWithString:#"http://www.domain.com/path/to/image.jpg"]
placeholderImage:[UIImage imageNamed:#"placeholder.png"]]
https://github.com/rs/SDWebImage