Out-Of-Memory while doing Core Data migration - iphone

I'm migrating a CoreData model between two versions of an application. I was storing binary data as blobs in the previous version and I want to take them out of the blobs for performance. My issue is that during the migration it seems that Core Data loads everything into memory which leads to Low Memory Warnings and then to my app being killed.
Apple documentation suggests the following :
http://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/CoreDataVersioning/Articles/vmCustomizingTheProcess.html#//apple_ref/doc/uid/TP40005510-SW9
However, it seems to rely on the fact that the large objects are applied different mapping.
In my case, all the objects are basically the same and the same mapping has to be applied to each of them. I don't see in this case how I could apply their technique.
How should I handle a migration with very large objects ?


I'm guessing that you have a bunch of changes you want to make in addition to pulling the data out of blobs. My suggestion is to do the migration in a few stages. I'm kind of thinking out loud here, so it might be possible to improve on this. This requires you to be using SQLite.
To make this work, you're going to have three versions of your model:
The original model
The model with the attribute removed (and possibly with a special unique ID added--see below)
The model with all of the changes you've made, including the addition of the new entity and relationships replacing the attribute
The reason to do this is that the transition from version 1 to 2 should be doable with an automatic lightweight migration. In that case Core Data doesn't need to load anything into memory--it just issues SQL statements to make the changes directly on the database.
So, you start by setting up your persistent store coordinator using the old model version. Once you've loaded the data, go through all of the objects you're migrating, extract the binary attribute, and write it to disk somehow. You can use a fetch request with batching and regular autorelease pool draining to make sure you don't use up too much memory for temporary objects. Store the data into the directory you get with NSCachesDirectory. You'll obviously want to store the data in a way that lets you relate it back to the object's managedObjectID.
Then, you shut everything down and ask Core Data to migrate the store from version 1 to version 2. See this link for details. Open up the store with version 2.
You might have to add a step where you assign some sort of unique ID to each object, because I'm not sure if Core Data maintains object IDs when it does a non-lightweight migration. If you need to do this, your version 2 model would add a new attribute to the object you're taking the binary data out of that would be either optional or have a default value set. Since lightweight migration shouldn't change the managedObjectIDs, you could at save the mapping of your new unique ID to the managedObjectIDs you saved along with the binary data two paragraphs ago.
Save the data and close the store.
Open the store and do a migration from version 2 to version 3, which should basically be the code you already had written before you posted the question. Once the store is open, add all of the objects you saved from the version 1 store and set up the relationships using the data you saved along the way.
Simple, right?

Related

is it possible to use more than one SQLite files in CoreData?

Hello fellow stackoverflow family members?
I know it is un-efficient to create one extra sqlite table in iPhone CoreData system. (X)
Currently, My app has one sqlite table but there are preset data to users to no need to waste parse time. But if I adding new entity in current structure of SQLite table, it wipes up the whole preset data. I haven't tried to use immigrate method but I don't think it wouldn't be just adding a new entity on the table. I'm thinking it also wipes up previous preset data.
Current Architecture of SQLite file.
entity : A
attributes : contains data
I want to keep A with attributes still contain data but also add new entity : B.
entity : A
attributes : previous contained data
(PLUS+)
entity : B
attributes : new data
Do I need to create extra persistence set to separately store entity B (create another SQLite file and use as storage) or Is there possible way to add entity B in current SQLite with no changes in entity A?
Thank you.

To directly answer your question: Yes, you can reference multiple SQLite files in Core Data. You'll want to use a unique persistent store for each one.
I don't understand the rest of your question though, so it may be likely that creating multiple persistent stores is not what you want at all.
I know it is in efficient to create one extra sqlite table in iPhone CoreData system.
This is untrue. Creating multiple entities (aka, tables, but that is abstracted away) can lead to much more efficient queries.
Currently, My app has one sqlite table but there is no need to change preset sqlite table but want to add some other attribute.
Just add the attribute to your entity. If your app has already been released or you want to maintain existing data, you'll have to set up a migration and/or turn on lightweight migration.
So commit with new attribute and update coredata is not valid. Because it flash off current data set and need to spend another hours to type manually.
This didn't make any sense to me. Can you clarify?

Can Core Data handle my "system vs. user data" migration needs?

My iPhone app will have read-only "system" data AND read/write "user" data (stored either using Core Data or a custom SQLite db). The user data may reference the system data. When a new version of the app is installed (e.g., via iTunes):
The new system data that comes with the update should overwrite/replace the old system data
The user data should be modified to reference the new system data (where possible).
Question: How is this kind of migration done with Core Data? Is it feasible?
For example, let's say my application is for managing recipes.
Each version of the app will come with a default set of recipes.
The user can not edit these "official" recipes.
However, the developer may modify (or delete) any "official" recipes in future versions of the app.
Users are allowed to add notes/comments to the "official" recipes (e.g., "bake for 45 min. instead of 30").
When the user upgrades to a new version of the app we want to keep the user comments and try to re-associate them with matching recipes from the new, "official" set of recipes. Is this possible/feasible with Core Data? Or perhaps I should just use a plain "database" solution (e.g., SQLite and traditional create/read/update/delete operations)?
Thanks!

You should have two persistent stores. A read only store that is in your bundle and a read/write store that is in the documents directory.
You can add both stores to the NSPersistentStoreCoordinator and access them both from one NSManagedObjectContext.
If both stores have the same entity then you will want to call -assignObject:toPersistentStore: to tell Core Data which store to save the entity into. If each underlying model has different entities then this is not necessary.
In addition you can "link" notes to a read-only recipe by making sure each recipe has a unique identifier (that you create) and the note stores that so that you can use a fetched property to retrieve the associated recipe and associates notes.
ObjectID
First, do not use the -objectID for linking between stores. It can and does change during migration (and other times) which will make your migration MUCH uglier than it needs to be.
Migration
Migration is very straight-forward. If you need to change the read-only data model, just change it and include the new version with your application.
If you need to change the read-write model, create a new model, use automatic migration during testing and when you are ready to ship, write a NSMappingModel from the old version to the new version.
Because the two persistent stores (and their associated models) are not linked there is very little risk with migration. The one "catch" is that the template code for Core Data will not be able to automatically resolve the source model for migration. To solve this issue you need to step in a little bit and help it out:
Stand up your destination NSPersistentStoreCoordinator and watch for an error. If you get a migration error:
Find the source model(s) and create an instance of NSManagedObjectModel with all of the appropriate source models.
Create an instance of NSMigrationManager and give it the source and destination models
Call - migrateStoreFromURL: type: options: withMappingModel: toDestinationURL: destinationType: destinationOptions: error: to kick off the migration
Profit!
It is a bit more work to handle the migration in this way. If your two models are very separated, you could do it a little different but it will require testing (as all things do):
Catch the migration error
Stand up a new NSPersistentStoreCoordinator with just the persistent store (and model) that needs to migrate.
Let that one migrate.
Tear down that NSPersistentStoreCoordinator and attempt to stand up your main NSPersistentStoreCoordinator again.

Can we use union of two sqlite databases with same tables for Core Data?

I have an iPhone Core Data app with a pre-populated sqlite "baseline" database. Can I add a second smaller sqlite database with the same tables as my pre-populated "baseline" database but with additional / complementary data such that Core Data will happily union the data from both databases and, ultimately, present to me as if it was all a single data source?
Idea that I had is:
1) the "baseline" database never changes.
2) I can download the smaller "complementary" sqlite database for additional data as and when I need to (I'm assuming downloading sqlite database is allowed, please comment if otherwise).
3) Core Data is then able to union data from 1 & 2. I can then reference this unified data by calling my defined Core Data managed object model.
Hope this makes sense.
Thanks in advance.

Core Data is designed to handle multiple data files via the – addPersistentStoreWithType:configuration:URL:options:error: method. This will allow you to combine all of the data files together and then access them via a single NSManagedObjectContext.
Your only issue, and it may not even be an issue for you, is that the store files cannot directly reference each other. Therefore you will need to reference data between files "manually" via unique identifiers. However I suspect you are already aware of that limitation.
Manual Relationships
The idea is that when both objects in a "relationship" are in one model and one file, Core Data does its magic and handles all of the referential integrity for you. However when they are in different files and/or models this doesn't happen automatically anymore.
The solution to this issue is to use a fetched property that looks up some unique identifier to retrieve the entity (or entities) that you want to be on the other side of the relationship. This will give you a "weak" relationship between files.
One thing to note though when doing this. The fetched property does not get updated automatically when something changes. This means when data changes that would cause that relationship to change, your application will not be automatically made aware of it and you will need to request that property again to get the updated relationship information.
Hopefully that makes it a bit clearer.
Co-existance of fetched properties and relationships
They can definitely co-exist but realize that they are two separate properties. If you want your controller code to see them as one, then I would suggest building a subclass for that entity and then adding a convenience method in there that hits both the relationship and the fetched property and then rolls them up into one NSArray or NSSet before returning it back to your controller code.

You can attach the downloaded database with ATTACH DATABASE statement and operate with unions of tables.

iPhone and Core Data: how to retain user-entered data between updates?

Consider an iPhone application that is a catalogue of animals. The application should allow the user to add custom information for each animal -- let's say a rating (on a scale of 1 to 5), as well as some notes they can enter in about the animal. However, the user won't be able to modify the animal data itself. Assume that when the application gets updated, it should be easy for the (static) catalogue part to change, but we'd like the (dynamic) custom user information part to be retained between updates, so the user doesn't lose any of their custom information.
We'd probably want to use Core Data to build this app. Let's also say that we have a previous process already in place to read in animal data to pre-populate the backing (SQLite) store that Core Data uses. We can embed this database file into the application bundle itself, since it doesn't get modified. When a user downloads an update to the application, the new version will include the latest (static) animal catalogue database, so we don't ever have to worry about it being out of date.
But, now the tricky part: how do we store the (dynamic) user custom data in a sound manner?
My first thought is that the (dynamic) database should be stored in the Documents directory for the app, so application updates don't clobber the existing data. Am I correct?
My second thought is that since the (dynamic) user custom data database is not in the same store as the (static) animal catalogue, we can't naively make a relationship between the Rating and the Notes entities (in one database) and the Animal entity (in the other database). In this case, I would imagine one solution would be to have an "animalName" string property in the Rating/Notes entity, and match it up at runtime. Is this the best way to do it, or is there a way to "sync" two different databases in Core Data?

Here's basically how I ended up solving this.
While Amorya's and MHarrison's answers were valid, they had one assumption: that once created, not only the tables but each row in each table would always be the same.
The problem is that my process to pre-populate the "Animals" database, using existing data (that is updated periodically), creates a new database file each time. In other words, I can't rely on creating a relationship between the (static) Animal entity and a (dynamic) Rating entity in Core Data, since that entity may not exist the next time I regenerate the application. Why not? Because I have no control how Core Data is storing that relationship behind the scenes. Since it's an SQLite backing store, it's likely that it's using a table with foreign key relations. But when you regenerate the database, you can't assume anything about what values each row gets for a key. The primary key for Lion may be different the second time around, if I've added a Lemur to the list.
The only way to avoid this problem would require pre-populating the database only once, and then manually updating rows each time there's an update. However, that kind of process isn't really possible in my case.
So, what's the solution? Well, since I can't rely on the foreign key relations that Core Data makes, I have to make up my own. What I do is introduce an intermediate step in my database generation process: instead of taking my raw data (which happens to be UTF-8 text but is actually MS Word files) and creating the SQLite database with Core Data directly, I introduce an intermediary step: I convert the .txt to .xml. Why XML? Well, not because it's a silver bullet, but simply because it's a data format I can parse very easily. So what does this XML file have different? A hash value that I generate for each Animal, using MD5, that I'll assume is unique. What is the hash value for? Well, now I can create two databases: one for the "static" Animal data (for which I have a process already), and one for the "dynamic" Ratings database, which the iPhone app creates and which lives in the application's Documents directory. For each Rating, I create a pseudo-relationship with the Animal by saving the Animal entity's hash value. So every time the user brings up an Animal detail view on the iPhone, I query the "dynamic" database to find if a Rating entity exists that matches the Animal.md5Hash value.
Since I'm saving this intermediate XML data file, the next time there's an update, I can diff it against the last XML file I used to see what's changed. Now, if the name of an animal was changed -- let's say a typo was corrected -- I revert the hash value for that Animal in situ. This means that even if an Animal name is changed, I'll still be able to find a matching Rating, if it exists, in the "dynamic" database.
This solution has another nice side effect: I don't need to handle any migration issues. The "static" Animal database that ships with the app can stay embedded as an app resource. It can change all it wants. The "dynamic" Ratings database may need migration at some point, if I modify its data model to add more entities, but in effect the two data models stay totally independent.

The way I'm doing this is: ship a database of the static stuff as part of your app bundle. On app launch, check if there is a database file in Documents. If not, copy the one from the app bundle to Documents. Then open the database from Documents: this is the only one you read from and edit.
When an upgrade has happened, the new static content will need to be merged with the user's editable database. Each static item (Animal, in your case) has a field called factoryID, which is a unique identifier. On the first launch after an update, load the database from the app bundle, and iterate through each Animal. For each one, find the appropriate record in the working database, and update any fields as necessary.
There may be a quicker solution, but since the upgrade process doesn't happen too often then the time taken shouldn't be too problematic.

Storing your SQLite database in the Documents directory (NSDocumentDirectory) is certainly the way to go.
In general, you should avoid application changes that modify or delete SQL tables as much as possible (adding is ok). However, when you absolutely have to make a change in an update, something like what Amorya said would work - open up the old DB, import whatever you need into the new DB, and delete the old one.
Since it sounds like you want a static database with an "Animal" table that can't be modified, then simply replacing this table with upgrades shouldn't be an issue - as long as the ID of the entries doesn't change. The way you should store user data about animals is to create a relation with a foreign key to an animal ID for each entry the user creates. This is what you would need to migrate when an upgrade changes it.

Can Core Data be used for objects with variable schemas?

I'm implementing a new iPhone app and am relatively new to Cocoa development overall. I am at the stage of choosing how the persistence layer of this app will work, and it looks like I'm basically choosing between Core Data and sqlite3.
The persisted models in this app are intended to have a schema that is loaded at runtime (from some kind of defn file, probably XML). By which I mean, this app is intended to have objects that are user-definable to some extent, e.g. the Customer type (which has certain built-in fields like "name" and "email") can be modified to have extra fields based on the user's specific needs (e.g. a user might want to add a "favourite fruit" field to their Customer type).
Having said that, will Core Data work for an app with a non-baked-in data model like this? I've just started playing around with the Core Data object designer thing in XCode and it seems like this thing wants to work with objects that have fixed fields that are compiled in.
I'm definitely trying to take the path of least resistance here, and I can see the benefits of using an Apple-supplied data framework, but don't want to start down that path if it's going to lock me into a data model that's defined at compile time.

The Core Data data model needs to be defined at compile time, but that does not mean you can't allow for custom fields to be added and used by end users.
It just means that you would define an entity for custom fields and create the fields as objects.
It is best to design a data model that meets your needs rather than think of how you would solve the problem in SQL.