I'm having an issue with a new application on the iPhone SDK using SQLite as the DB backend.
Occasionally, my app will stop loading data to my UITableViews and after downloading the device DB via the Organizer I can access the SQLite DB via the command line. I can query certain tables fine but not others without getting an "SQL error: database disk image is malformed" error. See a sqlite session below:
SQLite version 3.6.17
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select * from user;
1|cpjolicoeur#gmail.com|cpjolicoeur||4d055e38bb1d3758|image/gif|cartoonme_avatar.gif||Craig|Jolicoeur|1|1
sqlite> select * from item;
SQL error: database disk image is malformed
sqlite>
In this example my user table works fine but my item table is malformed, which corresponds with what I am seeing in my app where the items dont load. The app doesnt crash, the data just doesnt load because of this malformed error.
Any ideas why this is happening? My only thought is that maybe the DB is being corrupted because I am writing to the SQLite DB via a background thread within the app. I download data from a webserver via an NSOperationQueue in a background thread and update the SQLite DB with the data downloaded. Would writing to the DB in a background thread (while potentially reading from the main thread) corrupt the DB, or is it something else?
You have to be very careful about background threads accessing the database while debugging! This is because when the debugger halts processing (such as at a breakpoint) all threads are paused, including threads that may be in the middle of a database call, somewhere in between a database "open" and a database "close" call.
If you are halted at a breakpoint, and click the stop sign in Xcode, your app exits immediately. This very often causes errors such as the one you saw, or the "corrupted database" error.
There really isn't any solution (because there is no way to modify the behavior of "stop tasks", but I have evolved some techniques to mitigate it:
1. Add code to detect the app entering the background and have your db operations gracefully stop.
2. Never use the stop sign to halt processing while debugging. Instead, when done with a breakpoint then "continue", hit the home button on the simulator or device (which should trigger the code you added in step 1), wait for the app to background, THEN you can stop the run.
In my case this had to do with iOS 7: On iOS 7 Core Data now uses the SQLite WAL journaling mode which writes data to a .db-wal file instead of directly to the .db file. In my app, I would copy a prepared .db file into Library/Application Support during an app update. The problem was that an old .db-wal file was still in that directory and I only replaced the .db file. That way, I ended up with a .db file that was out of sync with the old .db-wal file.
There are two solutions to this problem:
Make sure the .db, .db-wal and .db-shm files are deleted before you copy your new .db file into place.
Go back to the old pre-iOS 7 behavior like so: https://stackoverflow.com/a/18870738/171933
Depends on how SQLite is compiled, it may or may not be thread-safe. If you're using the built-in one, it may not have the compile-time options you're looking for.
For our app, we had to roll our own SQLite to add full text search. Take a look at this page.
Related
I'm an experienced programmer, but this is my first MacOS app (on 10.15.2), which needs to read an sqlite db of the user's choice (potentially anywhere on the machine)
At first it wouldn't open the DB 'name.sqlite' file itself;
but when I used NSOpenPanel to let the user select it, that worked.
But then the query failed because sqlite3 (via SQLite.swift) was trying to open 'name.sqlite-wal'
os_unix.c:43353: (0) open(/path/to/dbname.lrcat-wal) - Undefined error: 0
If I open the db using the command-line client, and pragma journal-mode=off then the app works fine, but I can't restrict the app based on that - most of the dbs have WAL journalling
I tried turning com.apple.security.app-sandbox false in app.entitlements, but that didn't help.
I tried moving the db to the Pictures folder and turning com.apple.security.assets.pictures.read-write true, but that didn't help.
The unix permissions for db and the containing folder (/tmp in my test) are both good.
Because allowing the user to select the db via the NSOpenPanel allowed me to open the db for reading, I assume there's a similar restriction on the creation of the '-wal' file for writing.
How can I get permission (ideally without bothering the user with details of what a wal file is) to create the file next to the db?
Edit:
Following TheNextMan's suggestion about sidecar files, I searched again.
The WWDC presentation he (guessing the pronoun) linked shows how to use CFBundleDocumentTypes to relate extra extensions with NSIsRelatedItemTypefor NSFilePresenter, but (as far as I know) I'm not going through that route. I've tried it but it hasn't helped
I also, armed with the "sidecar" search term, found Access sidecar files in a Mac sandboxed app, which looked good but didn't help.
Even better I found the four-year-old unanswered duplicate of my question (if I had enough rep I'd mark this as a duplicate), SQLite and Sandboxed OSX apps, which includes a quote from Apple documentation App Sandbox Design Guide, saying
Note: In the case of a SQLite journal file, beginning in 10.8.2, journal files, write-ahead logging files, and shared memory files are automatically added to the related items list if you open a SQLite database, so this step is unnecessary.
So apparently it should Just Work™. But it doesn't.
Thank you for this thorough question. I encountered today the same issue as you, fighting the macOS sandbox restrictions. Similar to you, I've already copied the user selected (NSOpenPanel) database to my apps temporary folder (FileManager.default.temporaryDirectory).
But still I was unable to query the database file: sqlite3_prepare_v2 always returned the os_unix.c:45340: (0) open(/var/folders/.../myDB.db-wal) - Undefined error: 0 message.
As the database already is located in the temporary folder, which definitely is writable by our app (as we copied the database file!), we can simply create the myDB.db-wal file ourselves:
let tmpLocation = /// URL of the SQLite *.db file in a writable directory
let walLocation = tmpLocation.deletingPathExtension().appendingPathExtension("db-wal")
try Data().write(to: walLocation)
Et voilà: opening the DB is working.
I use pure commands of the sqlite3 library,,, the first time you install the app, a method executes the sqlite3_open() method. It supposes to create the database. It actually creates it in a user folder (desktop folder in the mac os x) as showed in the log screen. After this step, it creates 2 tables and saves some data, and it completes this, with success.
the second time you run the app, it intents to open the database with the same method sqlite3_open(), but it presents the error showed in the image with code number 14.
After that, I made some research and found that the new version of sqlite uses 3 files (.sqlite, .sqlite-wal and .sqlite-shm)... After reading that, I started searching on how to create those 2 additional files at the moment of creating the first file (the .sqlite file)... But I only found that all the tutorials copy those 3 files (previously created) to the references folder on the project, but they don't create it.
Continuing my search, found that there is an option to change the configuration of the sqlite in my app, to prevent using this wal option... I had to execute the command SQLITE_FCNTL_PRAGMA (maybe this is not used like I'm doing).
Please if you need more info that may help solving this issue please just let me know.
image of Class method that opens/creates the Daatabase
Edit: screenshot with the extended errcode resulting on error 14 with no more details.
imglink
A handful of customers for my iPhone app are experiencing Core Data store corruption (I assume so, since the error is "Failed to save to data store: Operation could not be completed. (Cocoa error 259.)")
Has anyone else experienced this kind of store corruption? I am worried since I aim to soon push an update which performs a schema migration, and I am worried that this will expose even more problems.
I had assumed that the Core Data/SQLlite APIs use atomic operations and are immune to corruption except if the underlying filesystem experiences corruption.
Is there a way to reduce/prevent corruption, and a way to reproduce the corruption so I can test this (I have been unsuccessful thus far).
Edit:
Also getting this error: "The database at /var/mobile/Applications//Documents/foo.sqlite is corrupted. SQLite error code 11, database disk image is malformed."
It happens to me when I manually overwrote my Base.sqlite without deleting Base.sqlite-wal and Base.sqlite-shm. Indeed, these files are new SQLite 3.7 features, maybe added in iOS 7.
To resolve the problem, I deleted Base.sqlite-* and sqlite regenerated them from my new base version.
Also attempt to replicate the error by filling the drive on the Device -- you should receive a disk full error instead of the database corrupting, but it may be possible to get a corrupted database in this manner.
The error you're getting is defined in Foundation.h
NSFileReadCorruptFileError = 259, // Read error (file corrupt, bad format, etc)
I've never encountered it with an actual store but I have hit something similar with bad permissions (on the Mac.) I haven't seen anyone mention a similar error online either. The error prevention systems in Core Data are fairly robust.
I would guess that the easiest way to create this would be send the persistent store to look at the wrong file such as accidentally targeting it at a text file. If it expects an SQL store but finds something else it will complain that the file is corrupt. That's just a shot in the dark.
Edit
This will be hard to track down because errors like this are so rare in Core Data that there aren't any tools to assist finding the problem.
I would recommend:
Checking upstream of where the error is coming from code. Perhaps something is throwing the store off or is causing it to look in another place.
Check anywhere you might do something non-standard. For example, if you generate your own entity map in code, its easy to throw it off if your not careful.
For clarity, using Xcode 7.2.1, SQLite data store, Core Data object graph, for a prototype app.
My problem was detailed by the Xcode terminal as:
CoreData: error: (11) Fatal error. The database at
/Users/etc/Library/Developer/CoreSimulator/Devices/etc/data/Containers/Data/Application/etc/Library/Application Support/com.etc.etc/etc.sqlite is corrupted.
SQLite error code:11, 'database disk image is malformed'.
Effectively my app was able to load and read the SQLite data, but was unable to save.
This answer by SO user software evolved made sense to me. While using Simulator I was fairly certain I had interrupted a save operation on a managed object context with private queue concurrency type NSPrivateQueueConcurrencyType.
Further investigation (using SQLiteManager) revealed the SQLite table I had been saving to at the time was the cause of this problem.
I could have easily deleted the app (no public release yet) however I wanted to understand at least how to repair this problem.
Notes from this experience:
In the app delegate under the UIApplicationDelegate Protocol - (void)applicationWillResignActive:(UIApplication *)application, if your managed object context hasChanges be sure to include a database save method;
If using more than one queue, use the Home button to trigger the delegate method in item 1 before clicking the stop button in Xcode;
Repair the damaged database file - I developed a solution detailed below from the answer outlined on this webpage Fixing the SQLite error “The database disk image is malformed”.
SQLite Database File Repair Method:
Open terminal [terminal/sqlite commands];
Navigate to the appropriate file location [cd];
For backup purposes, make a copy of the 'malformed' database file (e.g.dbMalFormedBU.sqlite) (that can be deleted later if the repair is successful) [cp];
To be certain, delete the dbMalFormed.sqlite-shm and dbMalFormed.sqlite-wal files [rm];
Open your 'malformed' database file [sqlite3 dbMalFormed.sqlite];
Clone your database file [.clone dbMalFormedNew.sqlite];
Exit SQLite3 [.exit];
Delete the old 'malformed' database file [rm dbMalFormed.sqlite);
Rename the new database file to the name used previously [mv dbMalFormedNew.sqlite dbMalFormed.sqlite].
I experienced that error when trying to get the persistent store coordinator.
Multi-threading was the problem in my case. Fixed it wrapping the whole method with a #synchronized(self) {} block.
- (NSPersistentStoreCoordinator *)persistentStoreCoordinator
{
#synchronized(self) {
// Quickly return persistent store coordinator if available
if (__persistentStoreCoordinator != nil) {
return __persistentStoreCoordinator;
}
// Persistent store coordination initialization, to be performed once
// ...
}
return _persistentStoreCoordinator;
}
Are you ever interacting with the database using the sqlite API? Or have you used any non-Apple tools to create your seed database?
I experienced "Core Data store corruption" when I was preloading the database with a separated thread(not main thread) and it has been really difficult to find this bug since only a few of my customers will crash their app because of this.
I recently ran across this problem. In my case I was performing a search and iterating over the objects to transform the data to XML and KML. I would then spawn the email handler and attach the files. Then I would update a field in the objects and finally save to the backing store (SQL Lite). The worked fine in 3.x. In 4.x it broke.
It was stupid on my part to do all the email handling before altering and saving the DB. Moving all non essential code to the point after the save cleared up this problem.
This problem would totally corrupt the SQL DB. In my case the error is:
File at path does not appear to be a SQLite database
I currently have an iPhone app that reads data from an external XML file at start-up, and then writes this data to the database (it only reads/writes data that the user's app has not seen before, though)
My concern is that there is going to be a back catalogue of data of several years, and that the first time the user runs the app it will have to read this in and be atrociously slow.
Our proposed solution is to include this data "pre-built" into the applications database, so that it doesn't have to load in the archival data on first-load - that is already in the app when they purchase it.
My question is whether there is a way to automatically populate this data with data from, say, an XML file or something. The database is in SQLite. I would populate it by hand, but obviously this will take a very long time, so I was just wondering if anybody had a more...programmatic solution...
I'm going to flesh out Jason's answer, I've marked my post as a community wiki so I shouldn't get any points for this.
He's not talking about a dummy app - write the app as you normally would, but check to see if the database exists in your main bundle before you call the code that populates the plist. You run that in the simulator, pull out the generated sqllite database, and add it to your project - if you only need to read from it, you can read it from the main bundle directory. If you need to do further writes then copy it into the writable documents area, and use it from there. So basically for the main user, the code to populate the DB would never be called...
The only downside is you also end up including the plist files you are reading from, even though you only need the database. You could make a different build target that was a copy of the main one with the only difference being that it held the plist files, while the main target you built for the app store did not.
Not to take Jason's answering thunder, I can't comment yet so it has to be here.
The nice thing is that you can access the filesystem of the simulator right on your Mac. I am away from mine at the moment or I could tell you exactly where to look, but I just find it by putting the name of the db file into searchlight and just running with that.
You also do not need to wright any code to populate the db since you can use the command line tool to do the initial setup if that is more convenient.
You will need to copy it over though since resources are stored in the read only signed portion of the app bundle.
I had the same problem of you using sqlite, on massive insert it's really slow. So the best way it's provide directly a filled sqlite database.
You have another way, instead of INSERT INTO, to populate a sqlite db. You can produce a csv file for each table and load into the tables using your computer and the sqlite shell:
Just 2 simple commands:
.separator SEPARATOR
.import FILE TABLE
Example:
adslol:~ user$ sqlite3
SQLite version 3.6.12
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .separator ;
sqlite> .import myData.csv nameOfMyTable
sqlite> .quit
I hope it's what you was looking for :)
If you need a good client for sqlite3 try SQLite Manager, it's a Firefox add-ons.
I've created an sqlite3 database from the command line and inserted several records. My app retrieves all of them and shows them just fine. I then go back and insert a few more records via the sqlite3 cli, erase the db file out of the simulator's documents directory so that it will be recopied from the main bundle, and the run the app again only to find that it only displays the original records I inserted. The new records do not show. I verified that the new db file was copied to the simulators documents directory, and when I point the sqlite3 cli at it, I can do a select * and see all the records.
What could be going on here? It almost seems as if the previous version of the db file is being cached somewhere and used instead of my updated version.
//Scott
every time you rebuild and run an app in xcode, it creates a new folder under the iphone simulator's applications folder. If your sqlite db is being included from xcode the old db could be put in the new folder while the one your editing is in the old and now unused folder.
I haven't verified his answer, but Stephan Burlot said:
Sqlite uses a cache for requests. Close & reopen the database from time to release cache memory.
(I don't think it's true that every SQLite instance caches requests, but that might be the case on the iPhone.)
Obviously you aren't concerned with memory, but if it is caching requests, maybe just a close and reopen is all you need.
If that isn't the case, my next guess would be that your app is not pointing to the file you think it is pointing to -- did you have it pointing to a database with a different name at one point and forget to update the app? You could verify this by updating the db from within your app, then checking for those updates with the CLI. You might just find that they are not looking at the same db.