I am trying to load both people occlusion and motion capture on the same app.
Since ARBodyTrackingConfiguration does not support personSegmentationWithDepth, I am creating 2 ARViews, giving each a different configuration (ARWorldTrackingConfiguration and ARBodyTrackingConfiguration).
The problem is that for some reason only one of the delegates callback is fired, and no depth data is available.
What am I doing wrong here?
Is it not OK to have more than one ARSession live at the same time?
In ARKit 4.0 both features can be run simultaneously. However they are both CPU intensive.
override func viewDidLoad() {
super.viewDidLoad()
guard ARBodyTrackingConfiguration.isSupported
else { fatalError("MoCap is supported on devices with A12 and higher") }
guard ARBodyTrackingConfiguration.supportsFrameSemantics(
.personSegmentationWithDepth)
else { fatalError("People occlusion is not supported on this device.") }
let config = ARBodyTrackingConfiguration()
config.frameSemantics = .personSegmentationWithDepth
config.automaticSkeletonScaleEstimationEnabled = true
arView.session.run(config, options: [])
}
Related
I'm using QLPreviewController to show AR content. With the newer iPhones with LIDAR it seems that object occlusion is enabled by default.
Is there any way to disable object occlusion in the QLVideoController without having to build a custom ARKit view controller? Since my models are quite large (life-size buildings), they seem to disappear or get cut off at the end.
ARQuickLook is a library built for quick and high-quality AR visualization. It adopts RealityKit engine, so all supported here features, like occlusion, anchors, raytraced shadows, physics, DoF, motion blur, HDR, etc, look the same way as they look in RealityKit.
However, you can't turn on/off these features in QuickLook's API. They are on by default, if supported on your iPhone. In case you want to turn on/off People Occlusion you have to use ARKit/RealityKit frameworks, not QuickLook.
class ViewController: UIViewController {
#IBOutlet var arView: ARView!
override func viewDidLoad() {
super.viewDidLoad()
let box = try! Experience.loadBox()
arView.scene.anchors.append(box)
}
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
self.switchOcclusion()
}
fileprivate func switchOcclusion() {
guard let config = arView.session.configuration as?
ARWorldTrackingConfiguration
else { return }
guard ARWorldTrackingConfiguration.supportsFrameSemantics(
.personSegmentationWithDepth)
else { return }
switch config.frameSemantics {
case [.personSegmentationWithDepth]:
config.frameSemantics.remove(.personSegmentationWithDepth)
default:
config.frameSemantics.insert(.personSegmentationWithDepth)
}
arView.session.run(config)
}
}
Pay particular attention that People Occlusion is supported on A12 and later chipsets. And it works if you're running iOS 12 and higher.
P.S.
The only QuickLook's customisable object is an object from class ARQuickLookPreviewItem.
Use ARQuickLookPreviewItem class when you want to control the background, designate which content the share sheet shares, or disable scaling in case it's not appropriate to allow the user to scale a particular model.
I have added content to the face anchor in Reality Composer, later on, after loading the Experience that i created on Reality Composer, i create a face tracking session like this:
guard ARFaceTrackingConfiguration.isSupported else { return }
let configuration = ARFaceTrackingConfiguration()
configuration.maximumNumberOfTrackedFaces = ARFaceTrackingConfiguration.supportedNumberOfTrackedFaces
configuration.isLightEstimationEnabled = true
arView.session.delegate = self
arView.session.run(configuration, options: [.resetTracking, .removeExistingAnchors])
It is not adding the content to all the faces that is detecting, and i know it is detecting more than one face because the other faces occlude the content that is stick to the other face, is this a limitation on RealityKit or i am missing something in the composer? actually is pretty hard to miss somehting since it is so basic and simple.
Thanks.
You can't succeed in multi-face tracking in RealityKit in case you use models with embedded Face Anchor, i.e. the models that came from Reality Composer' Face Tracking preset (you can use just one model with .face anchor, not three). Or you MAY USE such models but you need to delete these embedded AnchorEntity(.face) anchors. Although there's a better approach – simply load models in .usdz format.
Let's see what Apple documentation says about embedded anchors:
You can manually load and anchor Reality Composer scenes using code, like you do with other ARKit content. When you anchor a scene in code, RealityKit ignores the scene's anchoring information.
Reality Composer supports 5 anchor types: Horizontal, Vertical, Image, Face & Object. It displays a different set of guides for each anchor type to help you place your content. You can change the anchor type later if you choose the wrong option or change your mind about how to anchor your scene.
There are two options:
In new Reality Composer project, deselect the Create with default content checkbox at the bottom left of the action sheet you see at startup.
In RealityKit code, delete existing Face Anchor and assign a new one. The latter option is not great because you need to recreate objects positions from scratch:
boxAnchor.removeFromParent()
Nevertheless, I've achieved a multi-face tracking using AnchorEntity() with ARAnchor intializer inside session(:didUpdate:) instance method (just like SceneKit's renderer() instance method).
Here's my code:
import ARKit
import RealityKit
extension ViewController: ARSessionDelegate {
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
guard let faceAnchor = anchors.first as? ARFaceAnchor
else { return }
let anchor1 = AnchorEntity(anchor: faceAnchor)
let anchor2 = AnchorEntity(anchor: faceAnchor)
anchor1.addChild(model01)
anchor2.addChild(model02)
arView.scene.anchors.append(anchor1)
arView.scene.anchors.append(anchor2)
}
}
class ViewController: UIViewController {
#IBOutlet var arView: ARView!
let model01 = try! Entity.load(named: "angryFace") // USDZ file
let model02 = try! FacialExpression.loadSmilingFace() // RC scene
override func viewDidLoad() {
super.viewDidLoad()
arView.session.delegate = self
guard ARFaceTrackingConfiguration.isSupported else {
fatalError("Alas, Face Tracking isn't supported")
}
}
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
let config = ARFaceTrackingConfiguration()
config.maximumNumberOfTrackedFaces = 2
arView.session.run(config)
}
}
Vertical place detection in ARKit is not very good so I am using proximity sensor to place found vertical plane. The UX is as follows:
Ask user to place front of the device on the wall
When the proximity sensor is triggered make a vertical plane using AR Camera transform
The issue that I am currently facing is that when front sensor is triggered, everything comes to a halt. All the CoreMotion sensors and render methods for ARSCNViewDelegate calls stop. This causes the world origin to move from its origin point and make the placed item also move with it.
Is there a way to get proximity sensor data without shutting down everything? Is there a better way to place vertical items?
Asynchronous functions let you perform two or more tasks almost simultaneously, without waiting until the dispatched block is executed.
So, you have to use an approach like this:
class ViewController: UIViewController {
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
DispatchQueue.main.async {
self.proximitySensorActivation()
}
}
func proximitySensorActivation() {
let device = UIDevice.current
device.isProximityMonitoringEnabled = true
if device.isProximityMonitoringEnabled {
NotificationCenter.default.addObserver(self,
selector: #selector(proximityChanged),
name: UIDevice.proximityStateDidChangeNotification,
object: device)
}
}
#objc func proximityChanged(notification: NSNotification) {
if let device = notification.object as? UIDevice {
print(device)
}
}
}
As there is no autofocus in ARKit, I wanted to load ARKit in a view that is half the screen and second half will have AVFoundation -> AVCamera.
Is it possible to load AVCamera and ARKit simultaneously in same app?
Thanks.
Nope.
ARKit uses AVCapture internally (as explained in the WWDC talk introducing ARKit). Only one AVCaptureSession can be running at a time, so if you run your own capture session it’ll suspend ARKit’s session (and break tracking).
Update: However, in iOS 11.3 (aka "ARKit 1.5"), ARKit enables autofocus by default, and you can choose to disable it with the isAutoFocusEnabled option.
Changing the camera focus would disrupt the tracking - so this is definitely not possible (right now at least).
Update: See #rickster answer above.
I managed to use AVFoundation with ARKit by calling self.sceneView.session.run(self.sceneView.session.configuration!) right after taking the photo.
Use self.captureSession?.stopRunning() right after taking photo to make the session resume faster.
self.takePhoto()
self.sceneView.session.run(self.sceneView.session.configuration!)
func startCamera() {
captureSession = AVCaptureSession()
captureSession.sessionPreset = AVCaptureSession.Preset.photo
cameraOutput = AVCapturePhotoOutput()
if let device = AVCaptureDevice.default(for: .video),
let input = try? AVCaptureDeviceInput(device: device) {
if (captureSession.canAddInput(input)) {
captureSession.addInput(input)
if (captureSession.canAddOutput(cameraOutput)) {
captureSession.addOutput(cameraOutput)
captureSession.startRunning()
}
} else {
print("issue here : captureSesssion.canAddInput")
}
} else {
print("some problem here")
}
}
func takePhoto() {
startCamera()
let settings = AVCapturePhotoSettings()
cameraOutput.capturePhoto(with: settings, delegate: self)
self.captureSession?.stopRunning()
}
This question already has answers here:
How to get the front camera in Swift?
(8 answers)
Closed 6 years ago.
Essentially what I'm trying to accomplish is having the front camera of the AVCaptureDevice be the first and only option on a application during an AVCaptureSession.
I've looked around StackOverflow and all the methods and answers provided are deprecated as of iOS 10, Swift 3 and Xcode 8.
I know you're supposed to enumerate the devices with AVCaptureDeviceDiscoverySession and look at them to distinguish front from back, but I'm unsure of how to do so.
Could anyone help? It would amazing if so!
Here's my code:
override func viewDidAppear(_ animated: Bool) {
super.viewDidAppear(animated)
previewLayer.frame = singleViewCameraSlot.bounds
self.singleViewCameraSlot.layer.addSublayer(previewLayer)
captureSession.startRunning()
}
lazy var captureSession: AVCaptureSession = {
let capture = AVCaptureSession()
capture.sessionPreset = AVCaptureSessionPreset1920x1080
return capture
}()
lazy var previewLayer: AVCaptureVideoPreviewLayer = {
let preview = AVCaptureVideoPreviewLayer(session: self.captureSession)
preview?.videoGravity = AVLayerVideoGravityResizeAspect
preview?.connection.videoOrientation = AVCaptureVideoOrientation.portrait
preview?.bounds = CGRect(x: 0, y: 0, width: self.view.bounds.width, height: self.view.bounds.height)
preview?.position = CGPoint(x: self.view.bounds.midX, y: self.view.bounds.midY)
return preview!
}()
func setupCameraSession() {
let frontCamera = AVCaptureDevice.defaultDevice(withMediaType: AVMediaTypeVideo) as AVCaptureDevice
do {
let deviceInput = try AVCaptureDeviceInput(device: frontCamera)
captureSession.beginConfiguration()
if (captureSession.canAddInput(deviceInput) == true) {
captureSession.addInput(deviceInput)
}
let dataOutput = AVCaptureVideoDataOutput()
dataOutput.videoSettings = [(kCVPixelBufferPixelFormatTypeKey as NSString) : NSNumber(value: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange as UInt32)]
dataOutput.alwaysDiscardsLateVideoFrames = true
if (captureSession.canAddOutput(dataOutput) == true) {
captureSession.addOutput(dataOutput)
}
captureSession.commitConfiguration()
let queue = DispatchQueue(label: "io.goodnight.videoQueue")
dataOutput.setSampleBufferDelegate(self, queue: queue)
}
catch let error as NSError {
NSLog("\(error), \(error.localizedDescription)")
}
}
If you just need to find a single device based on simple characteristics (like a front-facing camera that can shoot video), just use AVCaptureDevice.default(_:for:position:). For example:
guard let device = AVCaptureDevice.default(.builtInWideAngleCamera,
for: .video,
position: .front)
else { fatalError("no front camera. but don't all iOS 10 devices have them?")
// then use the device: captureSession.addInput(device) or whatever
Really that's all there is to it for most use cases.
There's also AVCaptureDeviceDiscoverySession as a replacement for the old method of iterating through the devices array. However, most of the things you'd usually iterate through the devices array for can be found using the new default(_:for:position:) method, so you might as well use that and write less code.
The cases where AVCaptureDeviceDiscoverySession is worth using are the less common, more complicated cases: say you want to find all the devices that support a certain frame rate, or use key-value observing to see when the set of available devices changes.
By the way...
I've looked around StackOverflow and all the methods and answers provided are deprecated as of iOS 10, Swift 3 and Xcode 8.
If you read Apple's docs for those methods (at least this one, this one, and this one), you'll see along with those deprecation warnings some recommendations for what to use instead. There's also a guide to the iOS 10 / Swift 3 photo capture system and some sample code that both show current best practices for these APIs.
If you explicitly need the front camera, you can use AVCaptureDeviceDiscoverySession as specified here.
https://developer.apple.com/reference/avfoundation/avcapturedevicediscoverysession/2361539-init
This allows you to specify the types of devices you want to search for. The following (untested) should give you the front facing camera.
let deviceSessions = AVCaptureDeviceDiscoverySession(deviceTypes: [AVCaptureDeviceType.builtInWideAngleCamera], mediaType: AVMediaTypeVideo, position: AVCaptureDevicePosition.front)
This deviceSessions has a devices property which is an array of AVCaptureDevice types containing only the devices matching that search criteria.
deviceSessions?.devices
That should be either 0 or 1 depending on if the device has a front facing camera or not (some iPods won't for example).