I have a CoreData sqlite files *.sqlite *.sqlite-shm *.sqlite-wal in a zip format from my previous application version. The file is almost few hundred MBs.
In recent version, I have done some changes in the database schema i-e adding few new fields etc.
In Importing, I am simply replacing the current database by old database and making the user to restart application, which is crude. It does work however it causes 2 issues
1- It takes a lot of time on splash screen hanging the application.
2- If the database is big enough, the hanging time passes the Timeout
of Application and closes the application automatically.
What is the better way of importing database into core data saved in Documents as zip file.
First detect whether you are doing a migration. If you are, then display a ViewController with a spinner that explains what you are you doing ("please wait while we optimize the app"). When the database migration is done, the load your regular viewController.
This exact issue is discussed in this lecture: Core Data Potpourri (Paul Goracke, February 13, 2014) around 58:00. While the lecture is a bit dated (it was made before NSPersistentContainer) it is still one of the best I have ever seen and it is worthwhile to watch the entire video.
Related
Is it possible to load a database / entity directly into CoreData? The app relies of data, the initial data load being > 100,000 items. On an iPhone 4 it's taking about 15 minutes to complete the insertion, which is an unpleasantly long time.
Is there any way I could:
Load a pre-populated entity into CoreData
Load a pre-populated database (all entities) into CoreData
Something along these lines
to make the insertion of data significantly quicker
Thanks.
Create your database using CoreData and add it to your project bundle, then at first load, copy the database from the bundle to the file system so it will be writable, then open it from there. You may want to mark the database as don't share to iCloud depending on your needs.
Yes it is possible. The 'normal' approach to this is to pre-build the database, include it as a resource, check on startup whether you have a database in the documents folder and, if not, copy it over from the bundle.
I usually run my code to build the database on the simulator, and then go copy it from the sim folder and add to my project resources.
Copying a pre-made Core Data generated sqlite file via the application bundle is a good idea, even though I find it a bit cumbersome. This make most sense if you have a store for static data and one for dynamic data - in most cases it is feasible to copy the static one like this.
But nevertheless, for the record, 15 minutes is ridiculous. I can close to guarantee that you can cut this down to a few seconds.
I have been able to accomplish 500K inserts in less than 30 seconds on a first generation iPad. The most important optimisation techniques are:
disabling the undo manager,
using a background thread with #autoreleasepool,
saving periodically with exactly the right batch size (determine experimentally).
I have an app with a very large Core Data database. I have versioned it many times over the past year.
The last time I versioned the database I made one simple change to an entity: I added a new optional attribute. For some reason it would not migrate using Light-weight Migration. I found out much later that this was due to a bug in Apple's Light-weight Migration code resulting from the 'renaming identifiers' that I had needed back in another versioning.
Anyway, I digress...
Because of the bug that kept me from using Light-weight migration, I created a mapping file to help with the migration, not understanding that this would was a much heavier process and would force my users to wait while the app loaded the entire database into memory while doing the migration. It turns out that this is not really an option at all with very large databases and many of my users were unable to migrate the database at all due to memory problems, etc.
So now I want to re-release my app and clear up this problem. The trouble is, some of my users have a database that is somehow marked as being 'in the middle of migrating'. Even with my new code, which gets rid of the mapping file and supports Light-weight migration, users that are in this state, 'in the middle of a migration', don't seem to get reset.
What are my options for backing out a migration?
- I can detect that I am in this state because there is a '.myDB.sqlite.migrationdestination_41b5a6b5c6e848c462a8480cd24caef3' file in the Documents directory. Deleting this file does not clear up the migration. My guess is that the database is somehow flagged as being in this state, or is already partially migrated.
- I can detect this state and then delete the database altogether. But this forces my users to re-download their data.
Any Thoughts?
Thanks for you help.
The only thing that occurs to me would be crack open the SQL store of an affected file and look for flags or something else that might signal the db being in a transitory state. You might be able to write directly to the file and alter something.
That's really ugly problem.
I am still learning xcode and objective-c. I use to build app for iphone environment only.
However I am in need of realizing an application with an existing prefilled sql database.
For prefilling the database I wouldn't like to use code in the ditributed app, but I would rather prefer to have a separate app for doing that.
The reason is that, the app could only download the updated database, rather than a whole code update .
So, questions are:
is this a possible scenario
if yes, what kind of application
should I build in xcode for
prefilling database ?
thanks
There's no reason that you can't have one app that both uses the database and downloads updates. Keeping the database updated without downloading the whole thing is pretty simple.
If you record the creation and modification timestamps of rows in the database on the server and keep track of those same modification timestamps on the device, updating the database works like this:
The device determines latest modification timestamp it has for a given table. We'll call it latestTimestamp. It sends the latestTimestamp to the server.
The server compares the latestTimestamp to the creation and modification timestamps in the database. The server sends back data based on the comparison result:
If the modification timestamp is earlier than latestTimestamp it doesn't need to send the record, the device already has it;
If the modification timestamp is later than latestTimestamp and the creation timestamp is earlier than latestTimestamp, it sends the record back noting that it is to be updated in the device database;
If the modification timestamp is later than latestTimestamp and the creation timestamp is later than latestTimestamp, it sends the record back noting that it is to be added in the device database.
Lastly, the server database needs to keep track of deleted records and a deletion timestamp for every record recorded. If latestTimestamp is later than the deletion timestamp, it sends back that the record needs to be deleted.
Obviously it gets a bit more complicated when you have a variety of connected tables, but as long as things are sent back in the correct order, it works great.
Use asynchronous data requests (the ASIHTTPRequest library makes it a breeze) and update the data in the background while the user uses the app. If it's essential that the data be updated prior to any interaction with it you can display an activity indicator and have the user wait.
No need at all for a separate app.
I would discourage you from doing that. No matter it is a pre-filled-database-purpose app, or a normal-purpose app, Apple Review Team would treat them with the same procedure, leaving the developer waiting for weeks before that app is finally available on App Store.
Besides, as far as I know, communication between apps is still strictly limited. If the data you would like to transfer between your main app and your db app is larger than a few lines of, let's say, NSString, it might be technically un-plausible.
I'm using MonoTouch and also System.Data to create a DataSet (just xml to those not familiar) for simple data binding. Data on my app is minimal so no need to go all out with SQLLite. The dataset use makes it easy to pass via web services for cloud sync.
I serialize the DataSet to the personal folder on save and of course read this file when the app starts up to load up the user's data. I've had issues where this file is becoming corrupt and I'm not sure why. I assume file I/O may be slow on these devices and that could be the cause, I'm not sure, but it is happening.
I'm also concerned that maybe iTunes is passing this file back and forth between the PC/MAC when the user syncs their devices with iTunes, which may be the cause of the corruption?
I want to prevent this device file from syncing with iTunes and also reliably persist it. I'm using the NSFile.Save option to save it to the device. I'm thinking since it's a text file maybe I could more safely store it in the standard user settings area instead? This would prevent it from being synced by itunes, I presume?
What is the most reliable and safe way to handle this file i/o for the xml dataset storage?
Thank you.
You're using MonoTouch. Isn't it simply a matter of calling DataSet.WriteXml() with a FileStream object ready to write to a document in your Documents folder?
That Documents folder is backed up to iTunes. It's not synced, but it helps if your user is restoring their phone (because they bricked it, lost it, whatever). It doesn't explain why it's corrupt.
The only thing that I can think of why it's corrupt is because it took too long for your app to write it. There's a limited time from the point where the user exits the app until it's closed down, to prevent apps from keeping the system hostage and deteriorate user experience.
If writing the whole dataset takes too long, you want to think about minimizing that. Perhaps you can just store the data, and not the schema. Or you can devise a way to store only the deltas on exit and reconcile when the user has loaded your app again.
You can also prevent complete loss of data by writing to a second file, and when that operation completes delete the old file and rename. That way, the next time you start up if the write operation didn't complete, the old file would still be there and the user would have only lost their more recent changes.
In any case, if your data gets too big for a simple write operation to complete, you should look at different options such as sqlite.
Your best bet is probably to just save the XML as text. It's as simple as File.WriteAllText(...) - there's no reason to go to NSFile for this. That's part of the advantage of MonoTouch :)
Regarding syncing, here's the rule:
If you keep the file in the user's documents folder (Environment.SpecialFolder.MyDocuments and Environment.SpecialFolder.Personal BOTH point to the user's doc folder), then it's going to get backed up whenever the user syncs with iTunes.
There's nothing wrong with this. It persists the data between sessions and makes it recoverable if something goes wrong with the user's phone and they need to restore from a backup. Since your question is about persisting an XML file on the phone, this is what you want.
As for the iTunes question, there's no problem with speed and syncing because your app isn't going to be running while the phone is syncing. The file will either have been saved or it won't. Any corruption that takes place is happening while your app is running.
Reasons for files getting corrupted can include:
Not saving before the user quits. You get a chance to do this.
Not gracefully handling an incoming phone call. The system warns you about this as well.
iTunes definitely isn't corrupting your file. If that were the case, iOS apps would all be broken. It could be happening on your dev machine for whatever reason, but I've never seen this happen elsewhere, and I've done quite a bit of iOS development.
If you'd like a tutorial on reading and writing files, I posted an answer in another question.
It's lengthy, but the point was to answer as many questions as I could so nobody would be left hanging or confused.
A nice thing about iOS devices is that you're back (for most apps) in the one-person-at-a-time world. You're writing apps where you don't have to worry about 5,000 people trying to use your web-based app at the same time (that's not always true, but... you get the point). As a result, you can do things that you might normally consider bad for performance, but you're unlikely to see any performance problems (as long as the file you're saving is either small enough to be saved quickly or you're saving in the background on another thread - you never want to block the main (UI) thread with a heavy IO operation).
If you have any more questions, feel free to ask.
I hope this helps :)
Lots of frameworks are read-only, but I've found that GDataXMLNode from http://code.google.com/p/gdata-objectivec-client/ works very well read/write.
Having said that, on the iPhone you'd do yourself a big favour using Core Data with a SQLLite backend. :-) Apple has done all this for you and optimized it more than any of us every will
Cheers
Nik
Consider using SQLite, I'd go for something like
http://www.taimila.com/entify/ (not tried yet)
catnap orm
or Faks sqlite-net on google code (using this in a few apps)
entify - if it does what it says it can do - looks really good.
persisting XML on the iPhone as a means to store and access data is a nause you dont want to get into. I wrote about it here http://iwayneo.blogspot.com/2010/08/festival-star-history-serialization-as.html
Example: User installs app v.1.0, adds data. Then the dev submits 10 updates in 10 weeks. After 11 weeks, the user wants v.11.0 and grabs a copy from the app store.
Assuming that the app has got 11 .xcdatamodel versions inside, where ***11.xcdatamodel is the current one, what would happen now since the persistent store of the user is ages old? would the migration happen 10 times, step-by-step through every migration iteration?
Or does the actual migration of data (lets assume gigabytes of data) happen exactly once, after Core Data (or the persistent store coordinator) has figured out precisely what to do to go from v.1.0 to v.11.0?
As I answered in your other question on this subject, there would only be one migration as Core Data does not understand the concept of versions. There is no timeline of models; it will only work with a source and a destination.
In your example, the source would be v1.0 and the destination would be v11.0. One migration.