My code needs to parse heaps of JSON files, and those files are hosted on GitHub and only available bundled as 1 single ZIP file. Because the ZIP file is only about 80 MB, I want to keep the entire unzipping operation in memory.
I'm able to load the ZIP file into memory as a Data? variable, but I'm not able to find a way to unzip a Data variable in memory and then assign the unzipped file/data to some other variables. I've tried using ZIP Foundation, but its Archive type's initializers take only file URLs. I didn't try Zip, but its documentation shows that it takes file URLs as well.
Here is my code:
import Cocoa
import Alamofire
class ViewController: NSViewController {
override func viewDidLoad() {
super.viewDidLoad()
let zipURL = URL(string: "https://github.com/KSP-CKAN/CKAN-meta/archive/master.zip")!
AF.request(zipURL).validate().responseData { response in
var zipData: Data? = response.data
// Here I want to unzip `zipData` after unwrapping it.
}
}
}
I also looked into passing a Data variable off as a file, but failed to find a way to do it.
UPDATE (2019-12-01 05:00)
According to this pull request thread on ZIPFoundation, the feature I'm looking for will be included in the next release. I tried to use the feature's contributor's fork, but somehow Swift Package Manager wouldn't allow it.
Before finding this, I tried using Python's zipfile library through Swift-Python interoperability provided by PythonKit, but it didn't work out, because Foundation's Data in Swift can not be cast into a PythonObject type.
Apple's Compression framework also looked promising, but it seems to have a soft limit of 1 MB on compressed files. The compressed file I need is about 80 MB, way larger than 1 MB.
So far, ZIPFoundation is my most hopeful solution.
UPDATE (2019-12-01 06:00)
On another try, I was able to install microtherion's fork through Swift Package Manager. The following code should work:
import Cocoa
import Alamofire
import ZIPFoundation
... // ignoring irrelevant parts of the code
let zipURL = URL(string: "https://github.com/KSP-CKAN/CKAN-meta/archive/master.zip")!
AF.request(zipURL).validate().responseData { response in
// a Data variable that holds the raw bytes
var zipData: Data? = response.data
// an Archive instance created with the Data variable
var zipArchive = Archive(data: zipData!, accessMode: .read)
// iterate over the entries in the Archive instance, and extract each entry into a Data variable
for entry in zipArchive! {
var unzippedData: Data
do {
_ = try zipArchive?.extract(entry) {unzippedData($0)}
} catch {
...
}
...
}
}
Related
I am trying to programmatically create an ePub file. I am following this tutorial. The tutorial says to "make the .epub container that all these files go in" by:
Create an empty .zip file with whatever name you like (See notes below for detailed instructions on how to do this.)
Copy the mimetype file into the zip file (don't use compression on this file)
Copy the rest of the files and folders mentioned above into the zip file *
Re-name the .zip extension to .epub
These are the last few steps of the process. I have all the files that need to go into the zip ready. I know the files work because I used a third party program (eCanCrusherMac.1.2.1) to do these finals steps and the product it creates is an ePub file that loads in the Books eReader (made by Apple).
I used the below code to zip the desired directory. I found this code on here Stack Overflow
func zip(itemAtURL itemURL: URL, in destinationFolderURL: URL, zipName: String) throws {
var error: NSError?
var internalError: NSError?
NSFileCoordinator().coordinate(readingItemAt: itemURL, options: [.forUploading], error: &error) { (zipUrl) in
// zipUrl points to the zip file created by the coordinator
// zipUrl is valid only until the end of this block, so we move the file to a temporary folder
let finalUrl = destinationFolderURL.appendingPathComponent(zipName)
do {
try FileManager.default.moveItem(at: zipUrl, to: finalUrl)
} catch let localError {
internalError = localError as NSError
}
}
if let error = error {
throw error
}
if let internalError = internalError {
throw internalError
}
}
I took the file this function gives me, made sure it had the epub extension and tried to open it using the Books app but it fails to load. The code produces a zip file that I can interact with normally in Finder so I know the function works.
I believe the issue is with the "mimetype" file getting compressed. I have taken a valid ePub, changed the file's extension to zip, unzipped it and then rezipped it using Finder and tried to open it again with no other changes and it doesn't work. As you can see in the instructions from the tutorial at the top of this post the "mimetype" file can't be compressed.
This is a Swift app on Mac.
I looked into the different NSFileCoordinator.ReadingOptions and NSFileCoordinator.WritingOptions and searched on Stack Overflow and elsewhere online but I can't find anything on how to create a zip file in Swift without compressing a file contained in the zip file.
I was able to get the ePub to open using ZIPFoundation:
try FileManager.default.zipItem(
at: bookURL,
to: ePubURL,
shouldKeepParent: false,
compressionMethod: .none,
progress: nil
)
How to change the .xcdatamodeld file i.e. the data model?
Since the program has already been run and the Persistent Store Coordinator (PSC) contains a url to .sqlite, .sqlite-shm and .sqlite-wal files on disk I think the process is as follows but am unsure. Any input would be appreciated.
Run code below to delete the url from PSC.
Delete sqlite files from disk.
Change .xcdatamodeld file.
CodeGen is set to manual so create new managed object subclasses.
Make appropriate changes to code.
Run program which I assume will enter a url into the PSC and create the 3 sqlite files on disk but now based on the new .xcdatamodeld file.
func deletePersistentStore() {
guard let persistentStoreURL = container.persistentStoreCoordinator.persistentStores.first?.url
else {
print("URL Missing")
return
}
do {
try container.persistentStoreCoordinator.destroyPersistentStore(
at: persistentStoreURL,
ofType: "SQLite",
options: nil)
} catch {
print("Persistent Store Not Deleted: \(error) - \(error.localizedDescription)")
}
print("\(container.persistentStoreCoordinator.persistentStores.count)")
// prints 0
print("\(String(describing: container.persistentStoreCoordinator.persistentStores.first?.url) )")
// prints nil
}
It sounds like you’re still developing this app and that it’s not released yet. If that’s true, you could do something like this. But it would be easier to delete the app from your phone (or simulator), change the model, and then install a new copy of the app.
That would let you skip steps 1 and 2.
If your app is already released, you should look into Core Data model migration. It’s a process that lets you update the data model without deleting existing data. In most cases it’s nearly automatic, but it depends on how much your model is changing.
The file needs to be read and write later. On Android I used Room & RoomAssetHelper.
I understand the basic logic that I need to include it in the project assets and then copy it to somewhere accessible for the app at first start. But I want to avoid writing these things manually and risking making an error (I am not too confident with reading files & DBs).
All of the answers that I find are from people giving quick & dirty advice on how to manually code the logic for this. I would like to do it on a clean & professional level.
Is there a library that would do most of the "risky" work for me?
(Meaning import & copy the .sqlite file, so I can start using it in my code)
I found GRDB.swift, but I cannot figure out if it supports prepopulated files.
Please stop looking for a magical library that will do all of this for you automatically.
What you need to do yourself without any SQLite library?
Add your prepopulated database.sqlite as an asset to your project.
When the app launches, check if the database.sqlite file is present at the expected location (inside your app's documents directory for example). You can check this using FileManager APIs.
If the file exists at the expected path, you are fine, no need to copy any file.
If the database.sqlite file does not exist at the expected path, you need to copy your database.sqlite file at the path using FileManager APIs.
CAUTION :
Be aware that in this step, you may encounter an error while copying the file. This should not happen for most cases. In rare caes that it does happen, you should adjust your app accordingly - indicate to user somehow that initialization failed, free some space on your phone, restart app etc.
Steps 2-4 need to be checked on every app launch - put this logic somewhere close your app startup process. If all of above instructions are followed and you succeed either via step 3 OR 4, you now have the database.sqlite file where you want it to be.
Where the SQLite library comes in?
At this point, you can use any library that suits your purpose and you feel comfortable with.
As you mentioned GRDB.swift, it allows you to specify a custom path for your database file. Copy-pasting the current version minimal setup code here for reference.
import GRDB
// 1. Open a database connection
let dbQueue = try DatabaseQueue(path: "/path/to/database.sqlite")
// 2. Define the database schema
try dbQueue.write { db in
try db.create(table: "player") { t in
t.autoIncrementedPrimaryKey("id")
t.column("name", .text).notNull()
t.column("score", .integer).notNull()
}
}
// 3. Define a record type
struct Player: Codable, FetchableRecord, PersistableRecord {
var id: Int64
var name: String
var score: Int
}
// 4. Access the database
try dbQueue.write { db in
try Player(id: 1, name: "Arthur", score: 100).insert(db)
try Player(id: 2, name: "Barbara", score: 1000).insert(db)
}
let players: [Player] = try dbQueue.read { db in
try Player.fetchAll(db)
}
In my MacOS app I am downloading an encrypted .zip file to the disk. I decrypt this file and keep the decrypted version in memory in the Data type. For security reasons the decrypted .zip will only be kept in memory.
I can successfully use ZIPFoundation's Closure based reading to extract the file contents in memory, but only by using an URL pointing to the (decrypted) .zip on disk:
guard let archive = Archive(url: url!, accessMode: .read) else { return }
Is there any way I can use the library with data only existing in memory? If not, can you point me towards a library that can handle this?
I have already tried DataCompression, but I couldn't make it work.
There's a (non-merged) Pull Request open that adds in-memory processing of ZIP archives to ZIP Foundation.
Sadly there are still some unresolved issues with in-memory writing of archives. The reading part is using fmemopen and should already work.
While the PR is not finished yet, you can have a look here: https://github.com/weichsel/ZIPFoundation/pull/78/
I have encountered a very unusual problem with lazy initialization.
My application reads metadata from an mp4 file, edits some of the data, and writes it back to the file. In previous versions, I read all the data when the file was opened. In the current version, I have switched to lazy reading of the data to improve performance and lower the memory footprint. However, I have encountered a very sporadic but reproducible bug, in which a few bytes seem to be read incorrectly from the file during lazy initialization.
Here is the initialization code:
lazy var data: Data = { [unowned self] in self.readDataFromFile() }()
func readDataFromFile() -> Data {
guard let file = fileHandle else { return Data() }
file.seek(toFileOffset: dataFilePointer)
let data = file.readData(ofLength: dataCountInFile)
print(Pointer: \(dataFilePointer)" + " Data: " + data.hexBytes(length: 32))
return data
}
The print statement was added to debug this problem. The fileHandle is a FileHandle object which I create when the file is selected. The dataFilePointer and dataCountInFile variables are set when the file is opened, by parsing the file for the pointers to the data.
When I read from the file for the first time, this is what I see in the log:
Pointer: 6808113917 Data: 14600000 1EA70000 12FB0000 19260000 12E20000
If I stop app without writing any data to the file, and then restart, not doing anything else, this is what I see in the log:
Pointer: 6808113917 Data: CFCB378C CFCBCDDB 00015F90 4B4231AA 55C40000
This is actually the correct data. Subsequent running of the app consistently shows the correct data.
I have put breakpoints at all points where the file could be written to verify that no data is written to the file.
Additional observations:
This bug only occurs with a few selected files. The vast majority of
files are read correctly.
This bug only occurs with Release build configuration. It does not occur in Debug configuration. That is why
I have had to use the print statement to observe what is happening.
This bug does not occur if I do not initialize the data var lazily,
i.e. if I immediately read the data from the file when the object is
initialized.
I am having a difficult time understanding what could be causing this strange behavior.