In looking over the new CoreML API, I don't see any way to continue training the model after generating the .mlmodel and bundling it in your app. This makes me think that I won't be able to perform machine learning on my user's content or actions because the model must be entirely trained beforehand.
Is there any way to add training data to my trained model after shipping?
EDIT: I just noticed you could initialize a generated model class from a URL, so perhaps I can post new training data to my server, re-generate the trained model and download it into the app? Seems like it would work, but this completely defeats the privacy aspect of being able to use ML without the user's data leaving the device.
The .mlmodel file is compiled by Xcode into a .mlmodelc structure (which is actually a folder inside your app bundle).
Your app might be able to download a new .mlmodel from a server but I don't think you can run the Core ML compiler from inside your app.
Maybe it is possible for your app to download the compiled .mlmodelc data from a server, copy it into the app's Documents directory, and instantiate the model from that. Try it out. ;-)
(This assumes the App Store does not do any additional processing on the .mlmodelc data before it packages up your app and ships it to the user.)
Apple has recently added a new API for on-device model compilation. Now you can download your model and compile it on device
CoreML 3 now supports on-device model personalization. You can improve your model for each user while keeping its data private.
https://developer.apple.com/machine-learning/core-ml/
In order to dynamically update the model (without updating the whole app), you need to use MPS (Metal Performance Shader) directly instead of relying on .mlmodel, which must be bundled with the app.
It means you need to manually build the neural network by writing some Swift code (instead of using coremltools to converts existing models directly), and feed various weights for each layer, which is a little bit of work, but not a rocket science.
This is a good video to watch if you want to know more about MPS.
https://developer.apple.com/videos/play/wwdc2017/608/
Core ML supports inference but not training on device.
You can update the model by replacing it with a new one from a server, but that deserves its own question.
Now with iOS11 beta4, you can compile the model, download from server.
(Details)
As alternative to bundling an mlmodel with application, you can download and then compile models within your CoreML app. For this you just need to download the model definition file onto the user’s device by using, for example, URLSession. And after this you have to compile the model definition by calling a throwing compileModel(at:) type method.
let newCompiledModel = try MLModel.compileModel(at: modelDescriptionURL)
You'll get a new compiled model file with the same name as the model description but its ending will be mlmodelc. Eventually, create a new MLModel instance by passing the compiled model URL to its initialiser.
let model = try MLModel(contentsOf: newCompiledModel)
However, remember that compiling process may be time consuming and it shouldn't be done on main thread.
Related
I've looked around for this but haven't found what I'm looking for. I need some data to basically come pre-loaded in the app. I know that I could just put it all in on the first launch but would like to stay away from a long load time on the first launch and have it already loaded.
Is it possible to insert entities into core-data so that they are hard-coded in?
Yeah, you include a a pre-filled data store in your app bundle and copy it from the bundle to the documents directory as part of the app launch process - check if the data store exists and, if not, do the copy. You do this prior to accessing the Core Data stack for the first time.
There are a few ways you could do this. The lazy programmer way is to enter your default data into the app, either on the phone or in the simulator, grab the data store file, and include it in your Xcode project. The downside is it doesn't work well if you need to go back and edit the data model later.
The other option is to create an editor app on the Mac that uses the same Core Data model as your iPhone app (they're compatible) and edit the data in your Mac app. Jeff Lamarche talks a bit about this in one of his blog postings. I've done something similar, except I wrote a command line tool to download the latest data from a web site (in my case, XML data) and parse the XML into NSManagedObjects.
This StackOverflow post talks about a bit more complex option of having two data stores - one for your system data and one for your user data - and letting Core Data use both stores at runtime.
I'm in the position of looking at over the air distribution of additional content for our Core Data centric app. We need to find a good way to encapsulate new additional content bundles for import into the app.
I'm considering the following:
Option 1> Creating .sql files with insert statements to update the underlying SQLite layer directly (we get these as a by product of creating our content anyway).
Option 2> Distribution the new content in full .sqlite files, with which to create temporary managed object contexts and copy the data into the main context via the core layer. On the outset this seems like an expensive option compared the "as lean as it can be" option 1.
Can anyone suggest any other options or recommend the best approach from experience with this?
You could ship the modules as pre-processed SQLite persistent stores. These modules could then be attached to your app's store coordinator at startup. Kind of like loading plugins.
If the content really must live in the main store, you could import the content from one store to the other in the background, or something.
Maybe that's what you're suggesting by your option #2...
Another valid approach is to define a more generic external data format, eg something in JSON, XML, CSV, even vanilla SQLite, ship your data modules in the format, and import from there into your app's persistent store.
Your option #1 is risky, as it depends knowing the internal schema of the CD SQLite store, which Apple reserves the right to change at their discretion without telling you. I wouldn't ship on this option.
I'd go with either importing JSON or shipping pre-baked SQLite persistent stores.
Option1 is very dangerous because the Core Data SQLite schema is undocumented and subject to change without warning (assuming you can accurately reverse engineer it in the first place.)
Option 2 is your best option. It might seem expensive but remember that Core Data is first and foremost an object graph management API and only secondly a persistence API. Object graphs have not only data but also relationships and behaviors. The only reliable way to add objects to an object graph is to do so while the graph is "alive" i.e. by instantiating live objects and inserting them into the live (active) graph.
Writer/programmer Marcus Zarra has written about this issue several times and explains how to use configurations and even "plugin" data models and stores to import and export graphs from Core Data.
I’m an experienced developer, but new to Mac. I just don’t want to go down one path only to find out that I made some fundamental error or incorrect assumption later on.
I want to ultimately build and sell an iPhone app using Core Data. The app will be free with content available through in-app purchase. Here is what I want to be able to do:
OPTION 1
Build a Mac OS X utility app that points to the same Core Data object model, but has its own “master” database.
Populate the master database using the Mac app.
Export a subset of the master data from the Mac app to a flat file (XML?) that is a subset of the master data.
When the user purchases that data, download from the cloud and import that data into the local iPhone data store.
Number 2 should be easy enough. I have read about the XML Parser that should help me with #4. I need help with #1 and 3.
For #1, I can’t figure out how I can maintain one object model for both apps with Xcode. That data model must accept model versioning. Do I just create two Projects, one Mac and one iPhone, and point them both to the same .xcdatamodel file and the magic happens for me?
For #3, is there any sample code that someone can share that will iterate through an array of objects to create the XML?
OPTION 2
Another option I am considering was discussed below. Instead of worrying about import/export, simply create individual sql files for each set of new or updated data.
I could maintain a separate "metadata" database that has information about the individual sql files that are available to the app.
Then, I can dynamically access the individual SQL files from the local documents directory. This is similar to an iBooks model where the sql files equate to individual books.
I'm thinking I could have only two active database connections at a time... one for the metadata and the other for the specific "book". I am not sure if this will scale to many (tens or hundreds) sql files, however.
Any help is appreciated!
Jon
UPDATE: I just saw the answer from Marcus Zarra at:
Removing and adding persistent stores to a core data application
It sounds like Option 2 is a bad idea.
For (1), you can use the same object model in both apps. Indeed, if you use the same Core Data generated store, you are required to do so. Simply, include the same model file in both apps. In Xcode, the easiest way to do this is to put the model file external to the project folders of each project and then add the model file without copying it to the project folder. This will ensure that both apps use the same model file for every build.
For (3), you need to first create an "export" persistent store using the same model as the reference store and add it to the reference context. In the model, create an "Export" configuration. Create a subentity for every entity in the model but do not change any attributes or relationships. Assign those entities to the Export configuration.
You will need to add a "Clone" method to each ManagedObject subclass for the reference entities. When triggered, the method will return a subentity populated with the reference objects attributes and relationships (the relationship objects will be cloned as well.)
Be aware that cloning an object graph is recursive and can use a lot of memory.
When you save, because you assigned them to the "Export" configuration, all the cloned export entities and their relationships will be saved to the export store. You will have cloned not only the objects but the related object graph.
Include the model and the export store in the iPhone app. Write the app to make use of the export entities only. It will never notice the absence of any reference objects.
For (4), I wouldn't mess around with using XML or exporting the data outside of core data at all. I would just use the export Core Data SQL store created in (3) and be done with it.
You can give a NSManagedObjectContext instance and instance of NSPersistentStoreCoordinator. This class has options allowing you to specify a file location for sotring data and a format (SQLite, Binary, or XML)
How do you plan to actually transfer data from Mac to iPhone? Is this something you do during development, or something people do during daily use? If the latter, you are probably better off building decoupled export/import into your app right away. So the Mac would serialize data into XML or JSON, push it somewhere in the cloud (not sure if local network/bonjour transfer is easier or useful, cloud is more universal), and iPhone fetches the data and deserializes it into the local schema/repository. You should not plan to work on the SQL layer with Core Data. Different platforms may use a different storage backend.
I'm doing an iPhone application. In this app, I just want to have a database to be used as a looked up table for values in my app. The only thing the database will do was to supply me the values I needed depending on the query of the program. It won't do any addition or deletion in the database. My question was how do I initialize a default data on the application using CoreData. So that when the program starts It already had all the values needed.
If you have other suggestions of what is better do or what are other alternatives, please tell me.
Thanks.
I would create a simple Mac application, starting from the template for a Core Data document-based application. Copy your existing .xcdatamodel over the default one in the project (or add the new data model and be sure to rename anywhere that refers to the default model). Open up the XIB file for the document window in Interface Builder and drag the Core Data Entity item into it from the Interface Builder library inspector. From the resulting dialog, choose an entity to display and select an interface to display it in.
What this will do is create a fully functional interface for adding, removing, or editing the properties of that entity type. Everything should be hooked up via Cocoa Bindings so that you don't need to write a line of code for this to work. You can add interfaces for each entity type in your model this way.
This will let you quickly enter and edit data within a Core Data document, which you can then save out to disk and add as a resource to your iPhone application. The SQLite database structures are fully compatible between the desktop and iPhone Core Data implementations, so I've found that this is a quick and easy way of testing your iPhone Core Data code.
Please refer to the Core Data Programming Guide, or see below (copy from the PG):
"
How do I initialize a store with default data?
There are two issues here: creating the data, and ensuring the data is imported only once.
There are several ways to create the data.
You can create a separate persistent store that contains the default
data and include the store as an application resource. When you want
to use it, you must either copy the whole store to a suitable
location, or copy the objects from the defaults store to an existing
store. For small datasets, you can create the managed objects
directly in code.
You can create a property list—or some other file-based
representation—of the data, and store it as an application resource.
When you want to use it, you must open the file and parse the
representation to create managed objects.
You should not use this technique on iOS, and only if absolutely necessary on Mac OS X. Parsing a file to create a store incurs unnecessary overhead. It is much better to create a Core Data store yourself offline and use it directly in your application.
There are also several ways to ensure that the defaults are imported only once:
If you are using iOS or creating a non-document-based application for
Mac OS X, you can add a check on application launch to determine
whether a file exists at the location you specify for the
application’s store. If it doesn't, you need to import the data. For
an iOS-based example, see CoreDataBooks .
If you are creating a document-based application using NSPersistentDocument, you initialize
the defaults in initWithType:error:.
If there is a possibility that the store (hence file) might be
created but the data not imported, then you can add a metadata flag
to the store. You can check the metadata (using
metadataForPersistentStoreWithURL:error:) more efficiently than
executing a fetch (and it does not require you to hard code any
default data values).
"
As mentioned above, generally we need to create a pre-populated default store with code, then use it as a
resource file, and copy it from resource bundle to document directory if the coredata file is missing. Please search the CoreDataBooks code example in your Xcode Organizer (or Apple Developer Center), and look at the - (NSPersistentStoreCoordinator *)persistentStoreCoordinator method.
I racked my brain for hours attempting to solve this. What I came up with was simply not to save the database. That way, it will be initialized each time the app is opened. If you save it, it will continue to duplicate.
I would write a tool that populates a database with the data you want in it, generate the db at build time and stuff it in your resources folder. If you are not going to write to it you can just leave it there and access it directly, if you ever did want to write to it you would need to copy it somewhere you are allowed to write (like the documents folder).
I'm developing a desktop Mac OS X App that saves its very simple data into SQLite with Core Data and a companion mobile App for iPhone that simply needs to read data from the desktop App. Although they share the same Managed Object Model when I load the SQLite database on the mobile app the data takes several seconds to migrate the data. I really would like to avoid this long wait as it appears linearly related to the size of the SQLite file.
Why is data from the same Managed Object Model even needing to migrate at all?
Does anyone know of a way to avoid needing to migrate?
I've discovered what was causing this problem. Although it is probably very obvious I'll relate it so hopefully no one else will make this mistake...
I had two distinct Managed Object Models in my project. Having more than one seems to force Core Data to try migrating to figure out which data matches with which MOM.
As a solution I was able to manually merge the models so both data sources can be represented with the same MOM.