I've got an array of 40 arrays with 12 double features, so the type is [[double]]. Currently I'm sending this data to Google Cloud ML API to get a related prediction.
Since Apple recently introduced CoreML and coremltools, I converted my model from keras to .mlmodel to avoid thousand of google cloud api calls and do inference directly on my iPhone:
coreml_model = coremltools.converters.keras.convert(new_Model, input_names=['accelerations'],
output_names=['scores'])
coreml_model.save('PredictionModel.mlmodel')
After adding the model to my Xcode Project, it looks like:
I have no idea, where these others inputs and outputs are comming from.
To get a prediction, I need to convert my Array of Arrays of 12 doubles to an MLMultiArray, but I don't know how to do this. Has anyone faced a similar problem? Here is my current unfinished approach:
_predictionModel = PredictionModel()
guard let mlMultiArray = try? MLMultiArray(dataPointer: <#T##UnsafeMutableRawPointer#>, shape: <#T##[NSNumber]#>, dataType: <#T##MLMultiArrayDataType#>, strides: <#T##[NSNumber]#>, deallocator: <#T##((UnsafeMutableRawPointer) -> Void)?##((UnsafeMutableRawPointer) -> Void)?##(UnsafeMutableRawPointer) -> Void#>) else {
fatalError("Unexpected runtime error.")
}
guard let predictionOutput = try? _predictionModel.prediction(accelerations: mlMultiArray, lstm_1_h_in: nil, lstm_1_c_in: nil, lstm_2_h_in: nil, lstm_2_c_in: nil) else {
fatalError("Unexpected runtime error.")
}
The related documentation can be found here.
I achieved it by reading this blog :)
let data = _currentScaledMotionArrays.reduce([], +) //result is of type [Double] with 480 elements
guard let mlMultiArray = try? MLMultiArray(shape:[40,12], dataType:MLMultiArrayDataType.double) else {
fatalError("Unexpected runtime error. MLMultiArray")
}
for (index, element) in data.enumerated() {
mlMultiArray[index] = NSNumber(floatLiteral: element)
}
let input = PredictionModelInput(accelerations: mlMultiArray)
guard let predictionOutput = try? _predictionModel.prediction(input: input) else {
fatalError("Unexpected runtime error. model.prediction")
}
This is how I did it. Probably not the best way to handle optionals but gets the job done for testing
Create an instance of the MLMultiArray object with shape and data type
let mlArray = try? MLMultiArray(shape: [3], dataType: MLMultiArrayDataType.float32)
mlArray does not have an append function so you literally have to iterate through it and add values
for i in 0..<array.count {
mlArray?[i] = NSNumber(value: input[i])
}
Full function
func convertToMLArray(_ input: [Int]) -> MLMultiArray {
let mlArray = try? MLMultiArray(shape: [3], dataType: MLMultiArrayDataType.float32)
for i in 0..<array.count {
mlArray?[i] = NSNumber(value: input[i])
}
return arr!
}
Related
I'm currently building a new app and trying to integrate a Core ML model with Vision to my app... unfortunately, Xcode shows me this message: Cannot convert value of type 'MobileNetV2' to expected argument type 'VNCoreMLModel'
How may I solve this?
Here's my code below:
let config = MLModelConfiguration()
guard let coreMLModel = try? MobileNetV2(configuration: config),
let visionModel = try? VNCoreMLModel(for: coreMLModel.model) else {
fatalError("Couldn't load model!")
}
let classificationRequest = VNCoreMLRequest(model: coreMLModel, completionHandler: classificationCompleteHandler)
classificationRequest.imageCropAndScaleOption = VNImageCropAndScaleOption.centerCrop
visionRequests = [classificationRequest]
loopCoreMLUpdate()
}
This line shouldn't use coreMLModel but visionModel:
let classificationRequest = VNCoreMLRequest(model: coreMLModel, completionHandler: classificationCompleteHandler)
I have a Swift / CoreML code that works fine. I want to simplify my code by using the Vision framework.
In this code, there is a UIImage -> CGImage -> CVPixelBuffer conversion that I would like to get rid of. I know that using Vision one can pass directly a CGImage as input parameter. The problem I have is that my model takes 2 inputs (image + MLMultiArray) and output and image:
Inputs
my_input : Image (Color 512 x 512)
my_sigma : MultiArray (Float32 1)
Outputs
my_output : Image (Color 512 x 512)
I've tried to pass the sigma parameter as follow:
guard let cgImage = uiImage.cgImage else {
return nil
}
let options:[VNImageOption: Any] = [VNImageOption(rawValue: "my_sigma"): 0.1]
let handler = VNImageRequestHandler(cgImage: cgImage, options: options)
do {
try handler.perform(visionRequest)
} catch {
print(error)
}
By doing so, I get the following error:
[coreml] Failure verifying inputs. no results:Error Domain=com.apple.vis Code=3 "The VNCoreMLTransform request failed" UserInfo={NSLocalizedDescription=The VNCoreMLTransform request failed, NSUnderlyingError=0x280cbbab0 {Error Domain=com.apple.CoreML Code=0 "Required input feature not passed to neural network."
So, it seems that I didn't pass correctly the second parameter to the request handler. I have not been able to find the answer.
What is the correct way to pass such a parameter to Vision?
Do you know if I can use Vision to output an image directly?
Thank you for you help.
The code I provide bellow seems to be working fine. It might not be optimal, but at least it works which is a good first step. The main difficulty was to understand how the MLFeatureProvider works:
class SigmaProvider: MLFeatureProvider {
let sigma: Double
init(sigma: Double) {
self.sigma = sigma
}
var featureNames: Set<String> {
get {
return ["my_sigma"]
}
}
func featureValue(for featureName: String) -> MLFeatureValue? {
if (featureName == "my_sigma") {
let array = try! MLMultiArray(shape: [1], dataType: .float32)
array[0] = NSNumber(value: self.sigma)
return MLFeatureValue(multiArray: array)
}
return nil
}
}
This class defines an input named my_sigma and of type MLMultiArray. Bellow is the code that applies the CoreML model (named model) to an input UIImage:
// Input CGImage
guard let cgImage = uiImage.cgImage else {
return nil
}
// Load and setup the Vision model
guard let visionModel = try? VNCoreMLModel(for: model.model) else {
fatalError("Cannot load Vision ML model")
}
visionModel.inputImageFeatureName = "my_input"
visionModel.featureProvider = SigmaProvider(sigma: self.sigma)
// Create the request
let request = VNCoreMLRequest(model: visionModel)
request.usesCPUOnly = false
// Handler
let handler = VNImageRequestHandler(cgImage: cgImage)
// Process
do {
try handler.perform([request])
}
catch {
print("Error while performing Vision request: \(error)")
return nil
}
// Get the result
guard let results = request.results else {
print("No request results")
return nil
}
// Convert the resulting CVPixelBuffer into a UIImage
for case let resultingImage as VNPixelBufferObservation in results {
if resultingImage.featureName == "my_output" {
let ciOutput = CIImage(cvPixelBuffer: resultingImage.pixelBuffer)
UIGraphicsBeginImageContextWithOptions(uiImage.size, true, 1.0)
UIImage(ciImage: ciOutput).draw(at: CGPoint(x: 0, y: 0))
let output = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
return output
}
}
Note that the first argument of my model, my_input, is not needed in the SigmaProvider class. It is specified using the inputImageFeatureName property. Finally, I access the my_output by iterating through all results.
I hope this will help someone.
Have a look at the featureProvider property of VNCoreMLRequest. This is an object that provides additional inputs, i.e. anything other than the image input.
I have trouble with fetching multiples places from Google Places API. The problem is... if i fetch only one pin type like Bars, its ok, no problem. But if i trying to get Restaurants, Bars, Casino... multiples types, it gives my only first place, in our case Restaurants.
I tried make same request with Postman with link below... but for example
restaurants 1030 lines of JSON
political 83 lines
trying to get restaurants + political = 965 lines of JSON.
I use this code to get places i want:
func fetchPlacesNearCoordinate(_ coordinate: CLLocationCoordinate2D, radius: Double, types:[String], completion: #escaping PlacesCompletion) -> Void {
var urlString = "https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=\(coordinate.latitude),\(coordinate.longitude)&radius=\(50000)&rankby=prominence&sensor=true&key=\(googleApiKey)"
let typesString = types.count > 0 ? types.joined(separator: "|") : "food"
urlString += "&types=\(typesString)"
urlString = urlString.addingPercentEncoding(withAllowedCharacters: CharacterSet.urlQueryAllowed) ?? urlString
guard let url = URL(string: urlString) else { completion([]); return }
if let task = placesTask, task.taskIdentifier > 0 && task.state == .running {
task.cancel()
}
DispatchQueue.main.async {
UIApplication.shared.isNetworkActivityIndicatorVisible = true
}
placesTask = session.dataTask(with: url) { data, response, error in
if data != nil{
for el in data!{
//print(el)
}
}
var placesArray: [PlaceContent] = []
defer {
DispatchQueue.main.async {
UIApplication.shared.isNetworkActivityIndicatorVisible = false
completion(placesArray)
}
}
guard let data = data else { return }
do{
let decode = try JSONDecoder().decode(GooglePlacesAnswer.self, from: data)
placesArray = (decode.results?.map{ $0.toPlaceContent() }) ?? []
} catch let value{
print(value.localizedDescription)
}
}
placesTask?.resume()
}
You can't get places for multiple types as stated in official documentation. You have to make multiple requests and combine the results.
https://developers.google.com/maps/documentation/javascript/places
type — Restricts the results to places matching the specified type.
Only one type may be specified (if more than one type is provided, all
types following the first entry are ignored). See the list of
supported types.
I am getting error:
Value for key inputBarcodeDescriptor of type CIDataMatrixCodeDescriptor is not yet supported
let string = "tempValue&123"
let data = string.data(using: String.Encoding.ascii, allowLossyConversion: false)
guard let data = data else {
return nil
}
let descriptor = CIDataMatrixCodeDescriptor(payload: data, rowCount: 1, columnCount: 1, eccVersion: CIDataMatrixCodeDescriptor.ECCVersion(rawValue: 0))
let inputParameter = ["inputBarcodeDescriptor": descriptor]
let datafilter = CIFilter(name: "CIBarcodeGenerator", parameters: inputParameter)
let image = datafilter?.outputImage
print(image)
Well, I think you should just believe this runtime warning. You can't create a barcode using a CIDataMatrixCodeDescriptor; the class is documented but it isn't actually working. Use a different CIBarcodeDescriptor subclass instead (such as CIAztecCodeDescriptor).
wow, I've just ran into this issue... in 2022 :D
message is still
[api] Value for key inputBarcodeDescriptor of type CIDataMatrixCodeDescriptor is not yet supported
so I filed a feedback, let's hope for the best, as there's no lightweight alternative to generate DataMatrix codes that I could find
I am creating simple Json Parser that works like that: I have JsonData class that contains Anyobject as data. When I use jsonData["key"] it returns JsonData to i can chain jsonData["key"]["key2"] etc.
My question is how can I implement that class so i could cast it to lets say String:
jsonData["key"] as String without using some workarouds like
jsonData["key"].data as String
Code:
class JsonData:CustomStringConvertible{
let data:AnyObject
var description: String{
get{
return "\(data)"
}
}
init(_ data: Data) {
self.data = try! JSONSerialization.jsonObject(with: data, options: []) as! [[String:AnyObject]]
}
init(_ data: AnyObject) {
self.data = data
}
subscript(key:String) -> JsonData{
let newData = data as! [String:AnyObject]
let test = newData[key]!
return JsonData(test)
}
subscript(index:Int) ->JsonData{
let newData = data[index]!
return JsonData(newData)
}
}
In order to do this, you'd add another overload, but it won't work like you're thinking.
subscript(key: String) -> String {
let newData = data as! [String:AnyObject]
return newData[key] as! String
}
So then jsonData["key"] as String works, but jsonData["key"]["key2"] is ambiguous and you'd have to write it (jsonData["key"] as JsonData)["key2"] which probably isn't what you want.
The short answer to this is don't do this. If you need this much access to JSON, you're probably storing your data incorrectly. Parse it to structs as quickly as you can, and then work with structs. Convert the structs back to JSON when you want that. Extensive work with AnyObject is going to break your brain and the compiler over and over again. AnyObject is a necessary evil, not an every day tool. Soon you will encounter that terrible day that you have an AnyObject? and the compiler just breaks down in tears. Well, at least it isn't Any.
Putting that aside, the better solution is to use labeled-subscripts.
subscript(string key: String) -> String {
let newData = data as! [String:AnyObject]
return newData[key] as! String
}
Now you can access that as json[string: "key"] rather than json["key"] as String.