I am experiencing issues with VNDetectBarcodesRequest on iOS 16. My code works as intended on iOS 15 but on iOS 16 it doesn't find any bar codes in an image.
I've separated my code to a playground and am experiencing the same issue here. Running the code below on Xcode 13.4.1's playground, I get the result:
"Google link: Optional("https://www.google.com")"
Running the same code on Xcode 14, I get an nil result. Running this in an iOS15 simulator with Xcode 14 gives a positive result, only on iOS16 and playground it is not reading the QR code.
To add to it, no exceptions are thrown either.
Has anybody experienced the same and managed to fix this?
This is my playground code:
import UIKit
import Vision
extension UIImage {
func qrCodeLink(completion: #escaping (String?) -> Void) {
guard let ciImage = CIImage(image: self) else {
completion(nil)
return
}
let imageRequestHandler = VNImageRequestHandler(ciImage: ciImage,
orientation: .up,
options: [:])
let request = VNDetectBarcodesRequest { (request,error) in
guard error == nil else {
completion(nil)
return
}
guard let observations = request.results as? [VNDetectedObjectObservation] else {
completion(nil)
return
}
let result = (observations.first as? VNBarcodeObservation)?.payloadStringValue
completion(result)
}
try? imageRequestHandler.perform([request])
}
}
if let google = UIImage(named: "google") {
google.qrCodeLink { link in
debugPrint("Google link: \(link)")
}
} else {
debugPrint("No google image")
}
With the above code I am using this image, which is merely a link to https://www.google.com:
Elaborating on #Paul Peelen's solution, to ensure that the workaround is only used where needed (Xcode 14 + iOS 16 + Simulator), we used:
#if targetEnvironment(simulator) && compiler(>=5.7)
if #available(iOS 16, *) {
request.revision = VNDetectBarcodesRequestRevision1
}
#endif
I think I've found the issue.
When running the request on Xcode 14 and iOS 16, the request revision runs on VNDetectBarcodesRequestRevision3 (which issn't documented yet on VNDetectBarcodesRequest page). However, using VNDetectBarcodesRequestRevision1 or VNDetectBarcodesRequestRevision2 works.
Adding the following before the perform task worked for me:
request.revision = VNDetectBarcodesRequestRevision1
When compiling for either iOS 15 or 16, I found a difference in VNDetectedBarcodesRequest behavior when scanning a barcode WITH, or a barcode WITHOUT, a border. That border seems to trigger the scanning code when to start.
WITH a border: it scans both black-background image and white-background image correctly.
WITHOUT a border: it scans the barcode image on the monitor with a black background correctly, but it fails when that image is printed on white paper and scanned. No border = no barcode.
NEW ISSUE: it returned an array of 17 identical barcode observations when it scanned that single bordered barcode.
Related
I have a complication working perfectly well on my Apple Watch (simulator and hardware). However, I cannot seem to get the representation to display correctly when you choose which app to assign to a complication area. In my case, graphic corner. Its showing the display name of the app with "----" under it. In my getPlaceholderTemplate protocol method, I am using CLKComplicationTemplateGraphicCornerTextImage - which is being ignored.
Here is my protocol method.
func getPlaceholderTemplate(for complication: CLKComplication,
withHandler handler: #escaping
(CLKComplicationTemplate?) -> Void)
{
let ft = CLKSimpleTextProvider(text: "Aware")
let img = UIImage(systemName: "headphones")
let tintedImageProvider = CLKImageProvider(onePieceImage: img!)
let finalImage = CLKFullColorImageProvider(fullColorImage: img!, tintedImageProvider: tintedImageProvider)
if complication.family == .graphicCorner {
let thisTemplate = CLKComplicationTemplateGraphicCornerTextImage(textProvider: ft, imageProvider: finalImage)
handler(thisTemplate)
return
} else {
print("Complication not supported.")
}
handler(nil)
}
So this seemingly isn't being used. For the Simulator I have done the "Device > Erase all content and settings" just to make sure nothing old is cached. Any idea why it's defaulting to a UI I would prefer not have? Again, this is in the complication picker only, everywhere else it's working and looking great.
Example screenshot of how it's being represented.screenshot
I am using Swift to detect squares on an image and I can't seem to get it to detect them. It seems to detect rectangles sometimes not sure if this is the correct approach. I am new to swift and image detection so if there is something else I can be doing to detect squares I would greatly appreciate getting pointed in the right direction.
From what I have found on searches is issues around detecting squares / rectangles and perspective. Not sure if this is the issue or just my lack of knowledge of Swift and image detection.
Test image
lazy var rectangleDetectionRequest: VNDetectRectanglesRequest = {
let rectDetectRequest = VNDetectRectanglesRequest(completionHandler: self.handleDetectedRectangles)
// Customize & configure the request to detect only certain rectangles.
rectDetectRequest.maximumObservations = 8 // Vision currently supports up to 16.
rectDetectRequest.minimumConfidence = 0.6 // Be confident.
rectDetectRequest.minimumAspectRatio = 0.3 // height / width
return rectDetectRequest
}()
fileprivate func handleDetectedRectangles(request: VNRequest?, error: Error?) {
if let nsError = error as NSError? {
self.presentAlert("Rectangle Detection Error", error: nsError)
return
}
// Since handlers are executing on a background thread, explicitly send draw calls to the main thread.
DispatchQueue.main.async {
guard let drawLayer = self.pathLayer,
let results = request?.results as? [VNRectangleObservation] else {
return
}
self.draw(rectangles: results, onImageWithBounds: drawLayer.bounds)
drawLayer.setNeedsDisplay()
}
}
I have also changed minimumAspectRatio to 1.0 which from what information I have found would be a square and it still did not give the expected results.
My code is to display image.jpg in a window. That was written in Swift 3. Recently, I updated to Swift 4 with Xcode 10.2 . It doesn't work at all. I also rewrote it in Swift 5. It still doesn't work. I had to download the old Xcode 10 in order to run my code in Swift 3 to work again. I wonder what happened to the code.
Here is my main code to display the picture.
override func viewDidAppear() {
super.viewDidAppear()
imageView.image = NSImage(byReferencingFile: "\(ViewController.GlobaVariable.selecFile)" )
print("\(ViewController.GlobaVariable.selecFile)")
/// The print statement work but not the image.
}
let openPanel = NSOpenPanel()
openPanel.beginSheetModal(for:self.view.window!) { (response) in
if response.rawValue == NSFileHandlingPanelOKButton
{
let fileURL = openPanel.url!
//////////////////
let ph = fileURL.absoluteString
let path = (ph as NSString).substring(from: 7)
//print(path)
//////////////////////
GlobaVariable.selecFile = path
}
}
I am trying to combine CoreML and ARKit in my project using the given inceptionV3 model on Apple website.
I am starting from the standard template for ARKit (Xcode 9 beta 3)
Instead of intanciating a new camera session, I reuse the session that has been started by the ARSCNView.
At the end of my viewDelegate, I write:
sceneView.session.delegate = self
I then extend my viewController to conform to the ARSessionDelegate protocol (optional protocol)
// MARK: ARSessionDelegate
extension ViewController: ARSessionDelegate {
func session(_ session: ARSession, didUpdate frame: ARFrame) {
do {
let prediction = try self.model.prediction(image: frame.capturedImage)
DispatchQueue.main.async {
if let prob = prediction.classLabelProbs[prediction.classLabel] {
self.textLabel.text = "\(prediction.classLabel) \(String(describing: prob))"
}
}
}
catch let error as NSError {
print("Unexpected error ocurred: \(error.localizedDescription).")
}
}
}
At first I tried that code, but then noticed that inception requires a pixel Buffer of type Image. < RGB,<299,299>.
Although not recommenced, I thought I would just resize my frame then try to get a prediction out of it. I am resizing using this function (took it from https://github.com/yulingtianxia/Core-ML-Sample)
func resize(pixelBuffer: CVPixelBuffer) -> CVPixelBuffer? {
let imageSide = 299
var ciImage = CIImage(cvPixelBuffer: pixelBuffer, options: nil)
let transform = CGAffineTransform(scaleX: CGFloat(imageSide) / CGFloat(CVPixelBufferGetWidth(pixelBuffer)), y: CGFloat(imageSide) / CGFloat(CVPixelBufferGetHeight(pixelBuffer)))
ciImage = ciImage.transformed(by: transform).cropped(to: CGRect(x: 0, y: 0, width: imageSide, height: imageSide))
let ciContext = CIContext()
var resizeBuffer: CVPixelBuffer?
CVPixelBufferCreate(kCFAllocatorDefault, imageSide, imageSide, CVPixelBufferGetPixelFormatType(pixelBuffer), nil, &resizeBuffer)
ciContext.render(ciImage, to: resizeBuffer!)
return resizeBuffer
}
Unfortunately, this is not enough to make it work. This is the error that is catched:
Unexpected error ocurred: Input image feature image does not match model description.
2017-07-20 AR+MLPhotoDuplicatePrediction[928:298214] [core]
Error Domain=com.apple.CoreML Code=1
"Input image feature image does not match model description"
UserInfo={NSLocalizedDescription=Input image feature image does not match model description,
NSUnderlyingError=0x1c4a49fc0 {Error Domain=com.apple.CoreML Code=1
"Image is not expected type 32-BGRA or 32-ARGB, instead is Unsupported (875704422)"
UserInfo={NSLocalizedDescription=Image is not expected type 32-BGRA or 32-ARGB, instead is Unsupported (875704422)}}}
Not sure what I can do from here.
If there is any better suggestion to combine both, I'm all ears.
Edit: I also tried the resizePixelBuffer method from the YOLO-CoreML-MPSNNGraph suggested by #dfd , the error is exactly the same.
Edit2: So I changed the pixel format to be kCVPixelFormatType_32BGRA (not the same format as the pixelBuffer passed in the resizePixelBuffer).
let pixelFormat = kCVPixelFormatType_32BGRA // line 48
I do not have the error anymore. But as soon as I try to make a prediction, the AVCaptureSession stops. Seems I am running into the same issue Enric_SA is running on the apple developers forum.
Edit3: So I tried implementing rickster solution. Works well with inceptionV3. I wanted to try a a feature observation (VNClassificationObservation). At this time, it is not working using TinyYolo. The bounding are wrong. Trying to figure it out.
Don't process images yourself to feed them to Core ML. Use Vision. (No, not that one. This one.) Vision takes an ML model and any of several image types (including CVPixelBuffer) and automatically gets the image to the right size and aspect ratio and pixel format for the model to evaluate, then gives you the model's results.
Here's a rough skeleton of the code you'd need:
var request: VNRequest
func setup() {
let model = try VNCoreMLModel(for: MyCoreMLGeneratedModelClass().model)
request = VNCoreMLRequest(model: model, completionHandler: myResultsMethod)
}
func classifyARFrame() {
let handler = VNImageRequestHandler(cvPixelBuffer: session.currentFrame.capturedImage,
orientation: .up) // fix based on your UI orientation
handler.perform([request])
}
func myResultsMethod(request: VNRequest, error: Error?) {
guard let results = request.results as? [VNClassificationObservation]
else { fatalError("huh") }
for classification in results {
print(classification.identifier, // the scene label
classification.confidence)
}
}
See this answer to another question for some more pointers.
This question already has answers here:
How to get the front camera in Swift?
(8 answers)
Closed 6 years ago.
Essentially what I'm trying to accomplish is having the front camera of the AVCaptureDevice be the first and only option on a application during an AVCaptureSession.
I've looked around StackOverflow and all the methods and answers provided are deprecated as of iOS 10, Swift 3 and Xcode 8.
I know you're supposed to enumerate the devices with AVCaptureDeviceDiscoverySession and look at them to distinguish front from back, but I'm unsure of how to do so.
Could anyone help? It would amazing if so!
Here's my code:
override func viewDidAppear(_ animated: Bool) {
super.viewDidAppear(animated)
previewLayer.frame = singleViewCameraSlot.bounds
self.singleViewCameraSlot.layer.addSublayer(previewLayer)
captureSession.startRunning()
}
lazy var captureSession: AVCaptureSession = {
let capture = AVCaptureSession()
capture.sessionPreset = AVCaptureSessionPreset1920x1080
return capture
}()
lazy var previewLayer: AVCaptureVideoPreviewLayer = {
let preview = AVCaptureVideoPreviewLayer(session: self.captureSession)
preview?.videoGravity = AVLayerVideoGravityResizeAspect
preview?.connection.videoOrientation = AVCaptureVideoOrientation.portrait
preview?.bounds = CGRect(x: 0, y: 0, width: self.view.bounds.width, height: self.view.bounds.height)
preview?.position = CGPoint(x: self.view.bounds.midX, y: self.view.bounds.midY)
return preview!
}()
func setupCameraSession() {
let frontCamera = AVCaptureDevice.defaultDevice(withMediaType: AVMediaTypeVideo) as AVCaptureDevice
do {
let deviceInput = try AVCaptureDeviceInput(device: frontCamera)
captureSession.beginConfiguration()
if (captureSession.canAddInput(deviceInput) == true) {
captureSession.addInput(deviceInput)
}
let dataOutput = AVCaptureVideoDataOutput()
dataOutput.videoSettings = [(kCVPixelBufferPixelFormatTypeKey as NSString) : NSNumber(value: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange as UInt32)]
dataOutput.alwaysDiscardsLateVideoFrames = true
if (captureSession.canAddOutput(dataOutput) == true) {
captureSession.addOutput(dataOutput)
}
captureSession.commitConfiguration()
let queue = DispatchQueue(label: "io.goodnight.videoQueue")
dataOutput.setSampleBufferDelegate(self, queue: queue)
}
catch let error as NSError {
NSLog("\(error), \(error.localizedDescription)")
}
}
If you just need to find a single device based on simple characteristics (like a front-facing camera that can shoot video), just use AVCaptureDevice.default(_:for:position:). For example:
guard let device = AVCaptureDevice.default(.builtInWideAngleCamera,
for: .video,
position: .front)
else { fatalError("no front camera. but don't all iOS 10 devices have them?")
// then use the device: captureSession.addInput(device) or whatever
Really that's all there is to it for most use cases.
There's also AVCaptureDeviceDiscoverySession as a replacement for the old method of iterating through the devices array. However, most of the things you'd usually iterate through the devices array for can be found using the new default(_:for:position:) method, so you might as well use that and write less code.
The cases where AVCaptureDeviceDiscoverySession is worth using are the less common, more complicated cases: say you want to find all the devices that support a certain frame rate, or use key-value observing to see when the set of available devices changes.
By the way...
I've looked around StackOverflow and all the methods and answers provided are deprecated as of iOS 10, Swift 3 and Xcode 8.
If you read Apple's docs for those methods (at least this one, this one, and this one), you'll see along with those deprecation warnings some recommendations for what to use instead. There's also a guide to the iOS 10 / Swift 3 photo capture system and some sample code that both show current best practices for these APIs.
If you explicitly need the front camera, you can use AVCaptureDeviceDiscoverySession as specified here.
https://developer.apple.com/reference/avfoundation/avcapturedevicediscoverysession/2361539-init
This allows you to specify the types of devices you want to search for. The following (untested) should give you the front facing camera.
let deviceSessions = AVCaptureDeviceDiscoverySession(deviceTypes: [AVCaptureDeviceType.builtInWideAngleCamera], mediaType: AVMediaTypeVideo, position: AVCaptureDevicePosition.front)
This deviceSessions has a devices property which is an array of AVCaptureDevice types containing only the devices matching that search criteria.
deviceSessions?.devices
That should be either 0 or 1 depending on if the device has a front facing camera or not (some iPods won't for example).