Core ML 3 now gives us the ability to perform on-device training. After creating an updatable Core ML Model, we need to perform some function to update it on-device by using the MLUpdateTask function which requires 3 parameters: Model URL, MLBatchProvider and MLModelConfiguration.
Since Core ML 3 was just released, documentation for it is very limited; specifically on how to prepare the training data or an MLBatchProvider
Question: How do you prepare training data or create an MLBatchprovider.
If your Model is Named TestModel a TestModelTrainingInput class should be available.
let singleTrainingData = try TestModelTrainingInput(input: .init[1,2,3], output_true: .init([4,5,6]))
let trainingData = MLArrayBatchProvider(array: [singleTrainingData])
To provide data to Core ML, you create an MLFeatureProvider object. This returns one or more MLFeatureValue objects, one for each input in your model. Normally the auto-generated class does this behind the scenes.
If you want to work with a batch, you create an MLBatchProvider that has multiple MLFeatureProviders, one for each example.
Making an MLBatchProvider for predictions isn't so hard: just put your MLFeatureProviders in an array and then use MLArrayBatchProvider. Again, the auto-generated class has a helper method for this.
For training you probably want to load the data on-the-fly, do random augmentations, and so on. In that case, you'll want to make a new class that adopts the MLBatchProvider protocol. It should return an MLFeatureProvider for each example. This time the MLFeatureProvider does not just have the MLFeatureValue for the example but also an MLFeatureValue for the target / true label. (The auto-generated class has a helper class for this training feature provider, but not for the training batch provider.)
I haven't actually gotten any of the new training APIs to work yet (they crash on beta 2), so I'm not 100% sure yet how the MLBatchProvider would cycle through an entire epoch of training examples.
Related
I'm thinking of deploying ml models with seldon core on kubernetes. Seldon provides ways to do pre-processing, post-processing, predicting, combining and routing models. But, I think, these all assume that the input data is fixed. Is the input data for an entire seldon graph fixed, or is it possible to call models at runtime? In other words, is it possible use the output of one model to determine which other model should be called.
The thing I'm trying to do is to run one model that has a variable number of outputs (let's say an image instance segmentation model) and run a second model for each of the outputs (let's say an image classification model). In this case the input of the second model depends on the output of the first model.
Is this supported by seldon? Is there a way to do it with seldon?
I have been working with C# using a repository design pattern with Entity Framework (EF) that allows me to interact with the database using a generic class for each table.
This repository has all the functions that I would need such as: Adding a new entity into the database, updating an existing entity, deleting an entity, saving the context and so on...
Take, for example, I have an entity called 'Person' in the database. I would create a new class called PersonRepository which had all the functions I would need to change/add a value to the database.
As a result, to interact with the database, you create an instance of the repository class. This then allows you to call the functions of this class which in turn interacts with the database. The idea of this pattern is that all your database calls for an entity are isolated into a single class, this improves testability and separation of concerns.
I am learning how to use Core Data in my Swift programs and it appears to be similar to that of EF in C#. I have created the entities in the .xcdatamodel file and created the associated Cocoa Touch class via the Editor > Create NSManageedObject subclasses option.
I used the 'Category/Extension' CodeGen option, so, I then created a new class called 'Exercise'. From what I have read, this class can contain additional logic such as overriding the prepareForDeletion method. Does this mean I can add the addEntity, updateEntity functions onto this method and then call them from other code files?
My questions are:
Can I implement the same repository pattern that I can use in C#?
I believe this would be on the generated class of my entity (see image below)
Where do I place all my database call functions? If I can use this repository pattern I will place them on there, however, if I cannot, do I place them in a separate code file? Should I place them on the generated entity file which is created for the purposes of adding additional logic?
If I can use the repository pattern, is there a way to get the context injected into this class, or, do I have to call a getContext method each time I want to change it? In C# we can use the services to inject the context into the class each time I want to make changes to the database - I am unsure if the same is possible in Swift.
If I understand correctly, in Swift the equivalent of your 'repository' class could be one of two things. It could be context of core data, which you'd run queries and saves to OR (more likely) you would create a new class or struct that obscured away core data from the business side. Assuming the later, your "dataManager" class would perform all interactions with core data. It could then either return NSmanagedobjects to the business side, or you could map/transform/convert to some other type.
Yes, Exercise is a object that subclasses nsmanagedobject, so it has all the core data features of a managed object, but you can add any custom functions your want.
Your data manager class would perform the queries and return the correct objects. For example data manger.getExercises() -> [Exercises], dataManager.save(_ exercise: Exercise), etc....
You'll need to consider thread safety in your pattern. One option is converting your nsmanagedobjects to struct in your datamanager and not returning nsmangedobjects. Structs are thread safe and using codable you can easily convert between exerciseManagedOjbect and exerciseStruct. Plus, with this dataManager class example, you could remove core data for another persistence option in the future with reduced impact to your app overall.
I know that core data should not be considered as ORM but it still offers the functionality that is similar to ORM. Just curious, is it implementing data mapper pattern? I know "The Data Mapper is a layer of software that separates the in-memory objects from the database. Its responsibility is to transfer data between the two and also to isolate them from each other." (Martin Fowler). IMHO context manager handles all SQL stuff into one transaction, so it's very performance wise design and IMHO core data might be considered implementing data mapper pattern.
One year latter, I will contribute with my two cents
I am not an ORM expert and just recently started something using a Data Mapper, but as a long time Core Data user I can say that no. The main objective of this pattern is having a clear cut of a domain object from all database related operations.
Once I start writing unit tests, the first thing I notice is that I must load a database, even if it is just some in memory store, but I do must load one. Also there are no mappers for each class, I have no control about how each relation is stored.
Core Data loads lots of meta information about your object graph and forces some structure to them. Although you can change the persistent store and bake something of your own, you will have lots of restrictions about how to do it, with a clear "relational" feeling to it.
The idea is good, we might say it is some variation of it. Something that I do love is that the save operation is done by the context, not the object itself. So there is some type of separation.
However look at those functions like "awakeFromFetch" or "didSave", both operations are related with the data store, not a plain domain object. A proper Data Mapper pattern would allow you to define those operations for each persistent store, not unified in a single object.
UPDATE:
Funny enough one day after my answer I had to deal with an old CoreData based project and must come back to improve this answer. To make things clear, I do consider that "seems like a pattern" is not enough. For example, implementation of the facade and adapter patterns is quite similar, but you name them differently depending on how you use them.
Is Core Data implementing data mapper?
I must say that my "not quite" should have been "definitely not!"
I have just been very angry because I needed to rename some fields and later add new ones. Although I do know quite well how auto-migrations work with Core Data I forgot how annoying these are.
How many times do you need some new field, rename something, experiment until you get it right.... and every single tiny change requires a full blown database migration? With Data Mappers this never happens because domain objects are perfectly decoupled. You only touch the database to catch up with the domain objects after you finish some new feature. Core Data forces you to bind at every single moment every single detail of your domain objects.
Boy, how sweet life was until I forgot that "tiny" annoyance of Core Data being the exact opposite of what you can achieve with data mappers.
I reluctantly adopted a data mapper approach for my current project but I've been left a little confused.
Business Model
A worker has a name, a set of working hours, and a cost.
Data Model
A worker is made up of a labour type and a working pattern (three different tables). The cost of the worker is calculated based on a combination of the labour type and the working pattern.
My problem...
I seem to have two different models, one that represents business logic and one that represents data structure. I was of the understanding that my model should represent the business logic, but what happens when I want to insert a new worker? This is done using a form with two drop downs, the working pattern & the cost, the id's of which are not needed by the business model.
Confused? I am.
There is no real support for data models with the zend framework. But weierophinney does a realy good job to show how they could be implemented. Another very good description is this one.
Usually a model represents the data and includes the logic. The data model is a backend independed way to write/get data. For the model and the application it doesn't matter from where the data comes. Thus the datastorage can be exchanged without having to touch anything else.
Default data modelling in Zend Framework (Zend_Db_Table) isn't probably the best choice for object oriented data modelling.
Try using ORM like Doctrine (http://www.doctrine-project.org/) it allows you to create domain object model and almost transparently store it in database. This way you can have business model and data model combined in single classes.
I'm implementing a new iPhone app and am relatively new to Cocoa development overall. I am at the stage of choosing how the persistence layer of this app will work, and it looks like I'm basically choosing between Core Data and sqlite3.
The persisted models in this app are intended to have a schema that is loaded at runtime (from some kind of defn file, probably XML). By which I mean, this app is intended to have objects that are user-definable to some extent, e.g. the Customer type (which has certain built-in fields like "name" and "email") can be modified to have extra fields based on the user's specific needs (e.g. a user might want to add a "favourite fruit" field to their Customer type).
Having said that, will Core Data work for an app with a non-baked-in data model like this? I've just started playing around with the Core Data object designer thing in XCode and it seems like this thing wants to work with objects that have fixed fields that are compiled in.
I'm definitely trying to take the path of least resistance here, and I can see the benefits of using an Apple-supplied data framework, but don't want to start down that path if it's going to lock me into a data model that's defined at compile time.
The Core Data data model needs to be defined at compile time, but that does not mean you can't allow for custom fields to be added and used by end users.
It just means that you would define an entity for custom fields and create the fields as objects.
It is best to design a data model that meets your needs rather than think of how you would solve the problem in SQL.