Swift - switch between Core ML Model - swift

Question

I'm trying to compare predictions from different MLModels in SwiftUI. To do that I have to switch between them, but can't because every ML variable has its own class, so I get the error:
Cannot assign value of type 'ModelOne' to type 'ModelTwo'
Here's an example code:
import Foundation
import CoreML
import SwiftUI
let modelone = { //declaration model 1
do {
let config = MLModelConfiguration()
return try ModelOne(configuration: config)
} catch {
/*...*/
}
}()
let modeltwo = { //declaration model 2
do {
let config = MLModelConfiguration()
return try ModelTwo(configuration: config)
} catch {
/*...*/
}
}()
var imageused : UIImage! //image to classify
var modelstring = "" //string of model user chosen
var modelchosen = modelone
Button(action: { //button user decide to use model two
modelstring = "Model Two"
}) {/*...*/}
/*...*/
func classifyphoto() {
guard let image = imageused as UIImage?,
let imagebuffer = image.convertToBuffer() else {
return
}
if modelstring == "Model Two" { //if the user chosen model two, use ModelTwo
modelchosen = modeltwo // Error: Cannot assign value of type 'ModelOne' to type 'ModelTwo'
} else {
modelchosen = modelone}
let output = try? modelchosen.prediction(image: imagebuffer) //prediction with model chosen
if let output = output {
let results = output.classLabelProbs.sorted { $0.1 > $1.1 }
_ = results.map { /*...*/
}
}
}
Thank you!

Answer 1

The issue is that the two model classes do not have a common class or common inherited class. There are several ways to implement what you want. I think this is the best way based on your example.
class MyModel {
var model: MLModel? = nil
init(modelName: String) {
let bundle = Bundle.main
if let modelURL = bundle.url(forResource: modelName, withExtension:"mlmodelc") {
do {
self.model = try MLModel(contentsOf: modelURL)
}
catch {
print("Unable to open MLModel: \(error)")
}
}
}
}
class TestModel {
class func testModels() {
let modelOne = MyModel(modelName: "ModelOne")
let modelTwo = MyModel(modelName: "ModelTwo")
var selectedModel = modelOne
selectedModel = modelTwo
}
}

Answer 2

Swift is a statically typed language which means that in the general case you cannot assign a variable of one type to a variable of another type:
var int: Int = 42
int = "Hello, world!" // Not allowed: cannot assign String to Int
The problem is that modelchosen is of type ModelOne since it is initialized with modelone, thus, you cannot later assign modeltwo to it as you are trying to do.
To make that working, you have first to identify the common capabilities of ModelOne and ModelTwo. Take a look at their definition. For instance, do their .predict(image:) method return the same type? It looks like you are trying to do image classification, so a common capability could be the capability to return a String describing the image (or a list of potential objects, etc.).
When you'll have identified the common capability, you'll be able to define the common interface of your different types. This common interface can be expressed in many ways:
Using a base class
Using a protocol
Using an enum with payloads (union)
The following examples suppose that the common capabilities are:
The two networks can both be initialized with a MLModelConfiuration
They are used for image classification, i.e. they predict label (a String) describing a given image
Using a base class
The base class definition expresses those requirements like this:
class MLClassifier {
init(from config: MLModelConfig) {
fatalError("not implemented")
}
func classify(image: ImageBuffer) -> String {
fatalError("not implemented")
}
}
You then derive this base class for the two models (example with the first one:
final class ModelOne: MLClassifier {
init(from config: MLModelConfig) {
// the specific implementation for `ModelOne`...
}
func classify(image: ImageBuffer) -> String {
// the specific implementation for `ModelOne`..
}
}
Finally, you can make the variable modelchosen to be of type MLClassifier to erase the underlying concrete type of the model:
var modelchosen: MLClassifier = ModelOne(from: config1)
As MLClassifier is a common base class for both ModelOne and ModelTwo you can dynamically change the type of modelchosen whenever you need:
// Later...
modelchosen = ModelTwo(from: config2)
The variable modelchosen being of type MLClassifier ensures that you can call the .classify(image:) method whatever the concrete model type is:
func classifyphoto() {
guard let image = imageused as UIImage?,
let imagebuffer = image.convertToBuffer() else {
return
}
let output = modelchosen.classify(image: imageBuffer)
// Update the UI...
}
Using protocols
Protocols are the modern and preferred way of expressing common interfaces in Swift, they should be used over classes when possible:
protocol MLClassifier {
init(from config: MLModelConfig)
func classify(image: ImageBuffer) -> String
}
// Implement the protocol for your models
struct ModelOne: MLClassifier {
init(from config: MLModelConfig) { ... }
func classify(image: ImageBuffer) -> String { ... }
}
// Store an instance of any `MLClassfier` using an existential
var classifier: any MLClassifier = ModelOne(from: config1)
// Later...
classifier = ModelTwo(from: config2)
To sum up, the key is to identify the common capabilities of the different types you are trying to unify. For instance, if the two models output at some point a classLabelProbs of the same type, then you could use this as the common abstraction.
As a last resort, you could wrap everything in a big if-else statement, event though it is not recommended since it is not very readable, is not a good way to encapsulate common behavior and leads to a lot of code repetition:
func classifyphoto() {
guard let image = imageused as UIImage?,
let imagebuffer = image.convertToBuffer() else {
return
}
if modelstring == "Model Two" {
// Use modeltwo
let output = try? modeltwo.prediction(image: imagebuffer)
if let output = output {
let results = output.classLabelProbs.sorted { $0.1 > $1.1 }
_ = results.map { /*...*/ }
} else {
// Use modelone
let output = try? modelone.prediction(image: imagebuffer)
if let output = output {
let results = output.classLabelProbs.sorted { $0.1 > $1.1 }
_ = results.map { /*...*/ }
}
}