Why does AVCaptureDevice.default return nil in swiftui? - swift

I've been trying to create my own in-built camera but it's crashing when I try to set up the device.
func setUp() {
do {
self.session.beginConfiguration()
let device = AVCaptureDevice.default(.builtInDualCamera, for: .video, position: .front)
let input = try AVCaptureDeviceInput(device: device!)
if self.session.canAddInput(input) {
self.session.addInput(input)
}
if self.session.canAddOutput(self.output) {
self.session.addOutput(self.output)
}
self.session.commitConfiguration()
} catch {
print(error.localizedDescription)
}
}
When I execute the program, it crashed with the input because I try to force unwrap a nil value which is device.
I have set the required authorization so that the app can use the camera and it still end up with a nil value.
If anyone has any clue how to solve the problem it would be very appreciate

You're asking for a builtInDualCamera, i.e. one that supports:
Automatic switching from one camera to the other when the zoom factor, light level, and focus position allow.
Higher-quality zoom for still captures by fusing images from both cameras.
Depth data delivery by measuring the disparity of matched features between the wide and telephoto cameras.
Delivery of photos from constituent devices (wide and telephoto cameras) from a single photo capture request.
And you're requiring it to be on the front of the phone. I don't know any iPhone that has such a camera on the front (particularly the last one). You likely meant to request position: .back like in the example code. But keep in mind that not all phones have a dual camera on the back either.
You might want to use default(for:) to request the default "video" camera rather than requiring a specific type of camera. Alternately, you can use a AVCaptureDevice.DiscoverSession to find a camera based on specific characteristics.

Related

How do I use my own occlusion ML model with ARKit?

ARKit has built-in people occlusion, which you can enable with something like
guard let config = arView.session.configuration as? ARWorldTrackingConfiguration else {
fatalError("Unexpectedly failed to get the configuration.")
}
guard ARWorldTrackingConfiguration.supportsFrameSemantics(.personSegmentationWithDepth) else {
fatalError("People occlusion is not supported on this device.")
}
config.frameSemantics.insert(.personSegmentationWithDepth)
arView.session.run(config)
I would like to provide my own binary segmentation model ("binary" as in person/not-person for each pixel), presumably using CoreML, instead of using whatever Apple is using to segment people because I want to occlude something else instead of people. How do I do this? Is there a straightforward way to do this or will I have to re-implement parts of the rendering pipeline? They show some code about how to use people segmentation with a custom Metal renderer in this WWDC 2019 video (starts around 9:30) but it's not clear to me how to use my own model based on that, and I would prefer to use ARKit/RealityKit instead of implementing my own rendering (I am a mere mortal).

Capture if a barcode was scanned inside application

I am developing an application that creates a barcode using CIFilter. The question I have is if there is a way to capture at the app level every time the barcode is scanned? The barcode is a way for the device holder to redeem some sort of discount at different businesses. After it is scanned I would want to hit an API that captures that the barcode was scanned by the device holder. Without having to tie into the business systems. What would be the best approach for this? If there is one.
Just a snippet of how I'm creating these barcodes
func generateBarcode(from string: String) -> UIImage? {
let data = string.data(using: String.Encoding.ascii)
if let filter = CIFilter(name: "CICode128BarcodeGenerator") {
filter.setValue(data, forKey: "inputMessage")
let transform = CGAffineTransform(scaleX: 3, y: 3)
if let output = filter.outputImage?.transformed(by: transform) {
return UIImage(ciImage: output)
}
}
return nil
}
Thanks
The short answer is: most likely not.
The screen that is displaying the barcode has (physically) no way of telling if it's being looked at or scanned. It only emits light, it doesn't receive any information.
I can only think of two ways of getting that information:
Use the sensors of the device, like the front camera, to determine if the screen is being scanned. But this is a very hard task since you have to analyze the video stream for signs of scanning (probably with machine learning of some sort). It would also require that the user gives permission to use the camera just for... some kind of feedback?
The scanner needs to somehow communicate with the device through some other means, like a local network or through the internet. This an API of some sort, however.
Maybe it's enough for your use case to just track when the user opened the barcode inside the app, assuming this will most likely only happen when they let it be scanned.

Using multiple audio devices simultaneously on osx

My aim is to write an audio app for low latency realtime audio analysis on OSX. This will involve connecting to one or more USB interfaces and taking specific channels from these devices.
I started with the learning core audio book and writing this using C. As I went down this path it came to light that a lot of the old frameworks have been deprecated. It appears that the majority of what I would like to achieve can be written using AVAudioengine and connecting AVAudioUnits, digging down into core audio level only for the lower things like configuring the hardware devices.
I am confused here as to how to access two devices simultaneously. I do not want to create an aggregate device as I would like to treat the devices individually.
Using core audio I can list the audio device ID for all devices and change the default system output device here (and can do the input device using similar methods). However this only allows me one physical device, and will always track the device in system preferences.
static func setOutputDevice(newDeviceID: AudioDeviceID) {
let propertySize = UInt32(MemoryLayout<UInt32>.size)
var deviceID = newDeviceID
var propertyAddress = AudioObjectPropertyAddress(
mSelector: AudioObjectPropertySelector(kAudioHardwarePropertyDefaultOutputDevice),
mScope: AudioObjectPropertyScope(kAudioObjectPropertyScopeGlobal),
mElement: AudioObjectPropertyElement(kAudioObjectPropertyElementMaster))
AudioObjectSetPropertyData(AudioObjectID(kAudioObjectSystemObject), &propertyAddress, 0, nil, propertySize, &deviceID)
}
I then found that the kAudioUnitSubType_HALOutput is the way to go for specifying a static device only accessible through this property. I can create a component of this type using:
var outputHAL = AudioComponentDescription(componentType: kAudioUnitType_Output, componentSubType: kAudioUnitSubType_HALOutput, componentManufacturer: kAudioUnitManufacturer_Apple, componentFlags: 0, componentFlagsMask: 0)
let component = AudioComponentFindNext(nil, &outputHAL)
guard component != nil else {
print("Can't get input unit")
exit(-1)
}
However I am confused about how you create a description of this component and then find the next device that matches the description. Is there a property where I can select the audio device ID and link the AUHAL to this?
I also cannot figure out how to assign an AUHAL to an AVAudioEngine. I can create a node for the HAL but cannot attach this to the engine. Finally is it possible to create multiple kAudioUnitSubType_HALOutput components and feed these into the mixer?
I have been trying to research this for the last week, but nowhere closer to the answer. I have read up on channel mapping and everything I need to know down the line, but at this level getting the audio at. lower level seems pretty undocumented, especially when using swift.

BLE characteristic write value occasionally jumps to a random value

Using Core Bluetooth with Swift 3, I use CBPeripheral.writeValue() to write a characteristic to the Central, and I do this when the value of a UISlider changes. I was noticing that, even when dragging the slider slowly, occasionally a jump in value would be seen on the Central. I thought perhaps some over-the-air corruption was occurring, so I changed the characteristic to write the same value three times. Now, only if all the values received by the Central match will it act on them. Here's the current code:
#IBAction func slideValChanged(_ sender: UISlider)
{
let sliderVal = UInt8(sender.value.rounded(FloatingPointRoundingRule.down))
if (sliderVal != self.sliderVal)
{
self.sliderVal = sliderVal
self.bytes.removeAll()
self.bytes = [self.sliderVal, self.sliderVal, self.sliderVal]
DispatchQueue.global(qos: .userInitiated).async
{
self.data = Data(bytes: self.bytes)
self.peripheral.writeValue(self.data, for: self.writeCharacteristic!, type: CBCharacteristicWriteType.withResponse)
print("Write values sent:", self.bytes[0], self.bytes[1], self.bytes[2])
}
}
}
Even with this, I still see the value jump, and not to anything particular. The print() statement always prints the same (and correct) number three times. Similarly, on the Central, when the value jumps, I receive three equal but incorrect values. How can this be? All I can think is that something in Core Bluetooth is changing the value before it is put on air, but I'm not sure, and I don't know where to focus my attention.
As you are using CBCharacteristicWriteType.withResponse, even slow movement of your slider will send to many packets.
When you write to the writeCharacteristic command you need to wait for a callback before calling writeCharacteristic again. The callback will indicate that your data has been buffered in the Central BLE-stack and the stack is ready to receive additional data. Without this Your data can be corrupted or even lose your connection.

UWP Camera preview: "No suitable transform was found to encode or decode"

I'm previewing a camera with a CaptureElement in UWP. But why do i get the error: "No suitable transform was found to encode or decode" when using certain resolutions? It happens when using a webcam
I don't get the error when i use this method: mediacapture.VideoDeviceController.SetMediaStreamPropertiesAsync(...). but if i use this method, the captureelement doesn't get scaled to the resolution i want (capture element has it's strech attribute set to none because i don't want quality loss)
try
{
MediaCaptureInitializationSettings mediacapturesettings = new MediaCaptureInitializationSettings { VideoDeviceId = DeviceInfo.Id };
await mediacapture.InitializeAsync(mediacapturesettings);
}
catch (Exception exception)
{
var dialog = new MessageDialog(exception.Message);
await dialog.ShowAsync();
}
captureelement.Source = mediacapture;
captureelement.FlowDirection = Windows.UI.Xaml.FlowDirection.LeftToRight;
await mediacapture.StartPreviewAsync();
// await mediacapture.VideoDeviceController.SetMediaStreamPropertiesAsync(MediaStreamType.VideoPreview, Resolution.EncodingProperties);
await mediacapture.SetEncodingPropertiesAsync(MediaStreamType.VideoPreview, Resolution.EncodingProperties,null);
VideoRotation currentrotation = Rotation;
mediacapture.SetPreviewRotation(currentrotation);
I don't get the error when i use this method: mediacapture.VideoDeviceController.SetMediaStreamPropertiesAsync(...). but if i use this method, the captureelement doesn't get scaled to the resolution i want .
It only happens the first time i callstartpreview().
I've consulted internally and got a response. This problem is probably graphic driver related. Your graphic driver may require your camera device preview to be in running state before you call MediaCapture.SetEncodingPropertiesAsync, so you got this error and only got it at the first time.
i want to know the difference between the two methods.
Difference:
MediaCapture.SetEncodingPropertiesAsync set the property on the camera sink. You can set resolutions and rotation here that the camera does not support and then an encoder will do work to convert to desired format.
MediaCapture.VideoDeviceController.SetMediaStreamPropertiesAsync set the properties on the source. You can only set the configurations that the camera supports.
So the size of preview window is controlled by the Xaml CaptureElement. If you are using VideoDeviceController.SetMediaStreamPropertiesAsync and want to resize the preview window, you can only set the Stretch="UniformToFill" to let the preview window fill the parent element.