How to pass an extra parameter to Vision framework? - swift

Question

I have a Swift / CoreML code that works fine. I want to simplify my code by using the Vision framework.
In this code, there is a UIImage -> CGImage -> CVPixelBuffer conversion that I would like to get rid of. I know that using Vision one can pass directly a CGImage as input parameter. The problem I have is that my model takes 2 inputs (image + MLMultiArray) and output and image:
Inputs
my_input : Image (Color 512 x 512)
my_sigma : MultiArray (Float32 1)
Outputs
my_output : Image (Color 512 x 512)
I've tried to pass the sigma parameter as follow:
guard let cgImage = uiImage.cgImage else {
return nil
}
let options:[VNImageOption: Any] = [VNImageOption(rawValue: "my_sigma"): 0.1]
let handler = VNImageRequestHandler(cgImage: cgImage, options: options)
do {
try handler.perform(visionRequest)
} catch {
print(error)
}
By doing so, I get the following error:
[coreml] Failure verifying inputs. no results:Error Domain=com.apple.vis Code=3 "The VNCoreMLTransform request failed" UserInfo={NSLocalizedDescription=The VNCoreMLTransform request failed, NSUnderlyingError=0x280cbbab0 {Error Domain=com.apple.CoreML Code=0 "Required input feature not passed to neural network."
So, it seems that I didn't pass correctly the second parameter to the request handler. I have not been able to find the answer.
What is the correct way to pass such a parameter to Vision?
Do you know if I can use Vision to output an image directly?
Thank you for you help.

Answer 1

The code I provide bellow seems to be working fine. It might not be optimal, but at least it works which is a good first step. The main difficulty was to understand how the MLFeatureProvider works:
class SigmaProvider: MLFeatureProvider {
let sigma: Double
init(sigma: Double) {
self.sigma = sigma
}
var featureNames: Set<String> {
get {
return ["my_sigma"]
}
}
func featureValue(for featureName: String) -> MLFeatureValue? {
if (featureName == "my_sigma") {
let array = try! MLMultiArray(shape: [1], dataType: .float32)
array[0] = NSNumber(value: self.sigma)
return MLFeatureValue(multiArray: array)
}
return nil
}
}
This class defines an input named my_sigma and of type MLMultiArray. Bellow is the code that applies the CoreML model (named model) to an input UIImage:
// Input CGImage
guard let cgImage = uiImage.cgImage else {
return nil
}
// Load and setup the Vision model
guard let visionModel = try? VNCoreMLModel(for: model.model) else {
fatalError("Cannot load Vision ML model")
}
visionModel.inputImageFeatureName = "my_input"
visionModel.featureProvider = SigmaProvider(sigma: self.sigma)
// Create the request
let request = VNCoreMLRequest(model: visionModel)
request.usesCPUOnly = false
// Handler
let handler = VNImageRequestHandler(cgImage: cgImage)
// Process
do {
try handler.perform([request])
}
catch {
print("Error while performing Vision request: \(error)")
return nil
}
// Get the result
guard let results = request.results else {
print("No request results")
return nil
}
// Convert the resulting CVPixelBuffer into a UIImage
for case let resultingImage as VNPixelBufferObservation in results {
if resultingImage.featureName == "my_output" {
let ciOutput = CIImage(cvPixelBuffer: resultingImage.pixelBuffer)
UIGraphicsBeginImageContextWithOptions(uiImage.size, true, 1.0)
UIImage(ciImage: ciOutput).draw(at: CGPoint(x: 0, y: 0))
let output = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
return output
}
}
Note that the first argument of my model, my_input, is not needed in the SigmaProvider class. It is specified using the inputImageFeatureName property. Finally, I access the my_output by iterating through all results.
I hope this will help someone.

Answer 2

Have a look at the featureProvider property of VNCoreMLRequest. This is an object that provides additional inputs, i.e. anything other than the image input.