I'm an experienced programmer, but this is my first MacOS app (on 10.15.2), which needs to read an sqlite db of the user's choice (potentially anywhere on the machine)
At first it wouldn't open the DB 'name.sqlite' file itself;
but when I used NSOpenPanel to let the user select it, that worked.
But then the query failed because sqlite3 (via SQLite.swift) was trying to open 'name.sqlite-wal'
os_unix.c:43353: (0) open(/path/to/dbname.lrcat-wal) - Undefined error: 0
If I open the db using the command-line client, and pragma journal-mode=off then the app works fine, but I can't restrict the app based on that - most of the dbs have WAL journalling
I tried turning com.apple.security.app-sandbox false in app.entitlements, but that didn't help.
I tried moving the db to the Pictures folder and turning com.apple.security.assets.pictures.read-write true, but that didn't help.
The unix permissions for db and the containing folder (/tmp in my test) are both good.
Because allowing the user to select the db via the NSOpenPanel allowed me to open the db for reading, I assume there's a similar restriction on the creation of the '-wal' file for writing.
How can I get permission (ideally without bothering the user with details of what a wal file is) to create the file next to the db?
Edit:
Following TheNextMan's suggestion about sidecar files, I searched again.
The WWDC presentation he (guessing the pronoun) linked shows how to use CFBundleDocumentTypes to relate extra extensions with NSIsRelatedItemTypefor NSFilePresenter, but (as far as I know) I'm not going through that route. I've tried it but it hasn't helped
I also, armed with the "sidecar" search term, found Access sidecar files in a Mac sandboxed app, which looked good but didn't help.
Even better I found the four-year-old unanswered duplicate of my question (if I had enough rep I'd mark this as a duplicate), SQLite and Sandboxed OSX apps, which includes a quote from Apple documentation App Sandbox Design Guide, saying
Note: In the case of a SQLite journal file, beginning in 10.8.2, journal files, write-ahead logging files, and shared memory files are automatically added to the related items list if you open a SQLite database, so this step is unnecessary.
So apparently it should Just Work™. But it doesn't.
Thank you for this thorough question. I encountered today the same issue as you, fighting the macOS sandbox restrictions. Similar to you, I've already copied the user selected (NSOpenPanel) database to my apps temporary folder (FileManager.default.temporaryDirectory).
But still I was unable to query the database file: sqlite3_prepare_v2 always returned the os_unix.c:45340: (0) open(/var/folders/.../myDB.db-wal) - Undefined error: 0 message.
As the database already is located in the temporary folder, which definitely is writable by our app (as we copied the database file!), we can simply create the myDB.db-wal file ourselves:
let tmpLocation = /// URL of the SQLite *.db file in a writable directory
let walLocation = tmpLocation.deletingPathExtension().appendingPathExtension("db-wal")
try Data().write(to: walLocation)
Et voilà: opening the DB is working.
Related
I use pure commands of the sqlite3 library,,, the first time you install the app, a method executes the sqlite3_open() method. It supposes to create the database. It actually creates it in a user folder (desktop folder in the mac os x) as showed in the log screen. After this step, it creates 2 tables and saves some data, and it completes this, with success.
the second time you run the app, it intents to open the database with the same method sqlite3_open(), but it presents the error showed in the image with code number 14.
After that, I made some research and found that the new version of sqlite uses 3 files (.sqlite, .sqlite-wal and .sqlite-shm)... After reading that, I started searching on how to create those 2 additional files at the moment of creating the first file (the .sqlite file)... But I only found that all the tutorials copy those 3 files (previously created) to the references folder on the project, but they don't create it.
Continuing my search, found that there is an option to change the configuration of the sqlite in my app, to prevent using this wal option... I had to execute the command SQLITE_FCNTL_PRAGMA (maybe this is not used like I'm doing).
Please if you need more info that may help solving this issue please just let me know.
image of Class method that opens/creates the Daatabase
Edit: screenshot with the extended errcode resulting on error 14 with no more details.
imglink
do we have any command line from where i can query the sqlite database, which is created by coding, and stored in the application's default Documents folder?
Turn on file sharing for the app, copy the database file to your Mac, and use the command line tools (sqlite3) that are there.
(Note to the previous editor: I appreciate editing of answers for accuracy, format improvements, and fixing typos...but, if you want to provide completely different information, I suggest providing your own answer instead of changing the meaning of another user's response.)
My app is using iTunes file sharing which exposes everything in the Documents directory to the user, making it vulnerable to accidentally deletion or manipulation.
I spent hours in reading though these documents but it is a mess and I hope someone knows from experience. First, in one place they say that I should put these files in the Library directory.
In this technical Q & A Apple says that this is preserved. From my understanding this means that I can safely put important user data like sqlite3 database files in this directory. When the user updates to a new version, the content inside this directory will be preserved, it will survive and be available after the update:
applications can create their own
directories in
/Library/ and those
directories will be preserved in
backups and across updates
So /Library/ is preserved in backups and across updates.
For me with bad english this means: YES, the data will survive. It will not be lost when the user backs up. It will not be lost when the user updates. I looked up the word "preserved" in several dictionaries and I am sure it means "it will survive".
But then, there is this note in the iOS Application Programming Guide which tells something completely different! Here, they say about the Library directory:
<Application_Home>/Library/
You should not use this directory for user data files.
The contents of this directory (with the exception of the Caches
subdirectory) are backed up by iTunes.
Your application is generally
responsible for adding and removing
these files. It should also be able to
re-create these files as needed
because iTunes removes them during a
full restoration of the device.
"Should not use for user data files." (???)
But at the same time they admit it's backed up by iTunes. OK. So why shouldn't I put user data files into there, then?
/Library/Caches
Use this directory to write any application-specific support files
that you want to persist between
launches of the application or during
application updates. (...)
It should also be able to re-create
these files as needed because iTunes
removes them during a full restoration
of the device.
What?! I should put these files in Library/Caches. But this directory is not backed up by iTunes, as they said above. So this is only save for updates, but not for backup. And the data might be deleted anytime by the system.
Now this is completely confusing me. From what I understand I can choose between the evil and the devil: Data in /Library/ does not survive updates but is backed up by iTunes. Data in /Library/Caches does survive updates, but is not backed up by iTunes AND it might be deleted by the system (since it's a "Cache") anytime.
And on the other hand, the technical Q & A suggests putting this important user data in a custom subfolder in /Library/, for example /Library/PrivateDocuments.
In contrast to the iOS Application Programming Guide, the technical Q & A says: the entire /Library directory has always been preserved during updates and backups
So now, really, one of both documents MUST be wrong. But which one? What's the truth? Please, not speculation! I'm looking for answers from experience and I feel there is no way to figure this out except releasing an app and praying. Maybe someone wants to share his/her experience what really worked.
I've seen Library/Preferences (where NSUserDefaults are stored) be kept across restores, so I think most of Library is kept. The cache directories are probably excluded, though.
In general, just use the APIs to fetch paths and trust that iTunes will preserve them unless they're meant to represent temporary folders. That means you should use a subdirectory of your NSApplicationSupportDirectory named for your application:
NSArray * urls = [[NSFileManager defaultManager] URLsForDirectory:NSApplicationSupportDirectory inDomains:NSUserDomainMask];
NSAssert([urls count], #"Can't get app support directory");
NSURL * url = [urls objectAtIndex:0];
url = [url URLByAppendingPathComponent:#"MyAppName"];
In practice, this will end up being "Library/Application Support/MyAppName" in your sandbox, but you should use the API anyway to ensure this is kept future-proof.
(If you care about support for iOS 3 or 2, use the NSSearchPathForDirectoriesInDomains() function instead of the -URLsForDirectory:inDomains: method.)
The content inside Library folder (except cache) is backup by itunes.
What apple mean by "Should not use for user data files." (???) is that don't use this folder for data you want to be view in the File Sharing sytem via itunes.
So If you want the user to access the data via itunes write to /Document folder
If you want to hide the data from the user write to /Library folder
Why don't you try the keychain? If your data is not too extensive it can provide a quick way to store sensitive information
Keychain Documentation from Apple
I currently have an iPhone app that reads data from an external XML file at start-up, and then writes this data to the database (it only reads/writes data that the user's app has not seen before, though)
My concern is that there is going to be a back catalogue of data of several years, and that the first time the user runs the app it will have to read this in and be atrociously slow.
Our proposed solution is to include this data "pre-built" into the applications database, so that it doesn't have to load in the archival data on first-load - that is already in the app when they purchase it.
My question is whether there is a way to automatically populate this data with data from, say, an XML file or something. The database is in SQLite. I would populate it by hand, but obviously this will take a very long time, so I was just wondering if anybody had a more...programmatic solution...
I'm going to flesh out Jason's answer, I've marked my post as a community wiki so I shouldn't get any points for this.
He's not talking about a dummy app - write the app as you normally would, but check to see if the database exists in your main bundle before you call the code that populates the plist. You run that in the simulator, pull out the generated sqllite database, and add it to your project - if you only need to read from it, you can read it from the main bundle directory. If you need to do further writes then copy it into the writable documents area, and use it from there. So basically for the main user, the code to populate the DB would never be called...
The only downside is you also end up including the plist files you are reading from, even though you only need the database. You could make a different build target that was a copy of the main one with the only difference being that it held the plist files, while the main target you built for the app store did not.
Not to take Jason's answering thunder, I can't comment yet so it has to be here.
The nice thing is that you can access the filesystem of the simulator right on your Mac. I am away from mine at the moment or I could tell you exactly where to look, but I just find it by putting the name of the db file into searchlight and just running with that.
You also do not need to wright any code to populate the db since you can use the command line tool to do the initial setup if that is more convenient.
You will need to copy it over though since resources are stored in the read only signed portion of the app bundle.
I had the same problem of you using sqlite, on massive insert it's really slow. So the best way it's provide directly a filled sqlite database.
You have another way, instead of INSERT INTO, to populate a sqlite db. You can produce a csv file for each table and load into the tables using your computer and the sqlite shell:
Just 2 simple commands:
.separator SEPARATOR
.import FILE TABLE
Example:
adslol:~ user$ sqlite3
SQLite version 3.6.12
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .separator ;
sqlite> .import myData.csv nameOfMyTable
sqlite> .quit
I hope it's what you was looking for :)
If you need a good client for sqlite3 try SQLite Manager, it's a Firefox add-ons.
I've created an sqlite3 database from the command line and inserted several records. My app retrieves all of them and shows them just fine. I then go back and insert a few more records via the sqlite3 cli, erase the db file out of the simulator's documents directory so that it will be recopied from the main bundle, and the run the app again only to find that it only displays the original records I inserted. The new records do not show. I verified that the new db file was copied to the simulators documents directory, and when I point the sqlite3 cli at it, I can do a select * and see all the records.
What could be going on here? It almost seems as if the previous version of the db file is being cached somewhere and used instead of my updated version.
//Scott
every time you rebuild and run an app in xcode, it creates a new folder under the iphone simulator's applications folder. If your sqlite db is being included from xcode the old db could be put in the new folder while the one your editing is in the old and now unused folder.
I haven't verified his answer, but Stephan Burlot said:
Sqlite uses a cache for requests. Close & reopen the database from time to release cache memory.
(I don't think it's true that every SQLite instance caches requests, but that might be the case on the iPhone.)
Obviously you aren't concerned with memory, but if it is caching requests, maybe just a close and reopen is all you need.
If that isn't the case, my next guess would be that your app is not pointing to the file you think it is pointing to -- did you have it pointing to a database with a different name at one point and forget to update the app? You could verify this by updating the db from within your app, then checking for those updates with the CLI. You might just find that they are not looking at the same db.