I am using RealityKit for a human body detection.
let configuration = ARBodyTrackingConfiguration()
arView.session.run(configuration, options: [.resetTracking, .removeExistingAnchors])
A person always need stay too far in front of that back camera, more than 4 m. Can I apply a negative zoom? If I use a system camera application I see that a person can stay only 3 m front in.
Thank you in advance.
You can't zoom out sadly with reality kit. Reality kit uses the camera provided by ARKit and it's fixed with a focal length of 28mm.
Related
You will understand this question better if you open Xcode, create a new Augmented Reality Project and run that project.
After the project starts running on device, you will see the image from the rear camera, shooting your room.
After 3 or 4 seconds, a cube appears.
My questions are:
what were the app doing before the cube appearance? I mean, I suppose the app were looking for tracking points on the scene, so it could anchor the cube, right?
if this is true, what elements are the app looking for?
Suppose I am not satisfied with the point the cube appeared. Is there any function I can trigger with a tap on the screen, so the tracking can search for new points again near the location I have tapped on the screen?
I know my question is generic, so please, just give me the right direction.
ARKit and RealityKit stages
There are three stages in ARKit and RealityKit when you launch AR app:
Tracking
Scene Understanding
Rendering
Each stage may considerably increase a time required for model placement (+1...+4 seconds, depending on the device). Let's talk about each stage.
Tracking
This is initial state for your AR app. Here iPhone mixes visual data coming through RGB rear camera at 60 fps and transform data coming from IMU sensors (accelerometer, gyroscope and compass) at 1000 fps. Automatically generated Feature Points helps ARKit and RealityKit track surrounding environment and build a tracking map (whether it's a World Tracking or, for example, a Face Tracking). Feature Points are spontaneously generated on a high-contract margins of real-world objects and textures, in a well-lit environments. If you already have a previously saved World Map, it reduces a time for model placement into a scene. Also you may use a ARCoachingOverlayView for useful visual instructions that guide you during session initialization and recovery.
Scene Understanding
Second stage can include a horizontal and vertical Plane Detection, Ray-Casting (or Hit-Testing) and Light Estimation. If you have activated Plane Detection feature, it takes some time to detect a plane with a corresponding ARPlaneAnchor (or AnchorEntity(.plane)) that must tether a virtual model – cube in you case. Also there's an Advanced Scene Understanding allowing you to use a Scene Reconstruction feature. You can use scene reconstruction in gadgets with a LiDAR scanner and it gives you improved depth channel for compositing elements in a scene and People Occlusion. You can always enable an Image/Object Detection feature but you must consider it's built on machine learning algorithms that increase a model's placement time in a scene.
Rendering
The last stage is made for rendering of a virtual geometry in your scene. Scenes can contain models with shaders and textures on them, a transform or asset animations, dynamics and sound. Surrounding HDR reflections for metallic shaders are calculated by neural modules. ARKit can't render an AR scene. For 3d rendering you have to use such frameworks as RealityKit, SceneKit or Metal. These frameworks have their own rendering engines.
By default, in RealityKit there are high-quality rendering effects like Motion Blur or Ray-tracing shadows that require additional computational power. Take it into consideration.
Tip
To significantly reduce the time when placing an object in the AR scene, use a LiDAR scanner that works at nanoseconds speed. If you gadget has no LiDAR, then track only a surrounding environment where lighting conditions are good, all real-world objects are clearly distinguishable and textures on them are rich and have no repetitive patterns. Also, try not to use in your project polygonal geometry with more than 10K+ polygons and hi-res textures (jpeg or png with a size 1024x1024 considered as normal).
Also, RealityKit by default has several heavy options enabled – Depth channel Compositing, Motion Blur and Ray-traced Contact Shadows (on A11 and earlier there are Projected Shadows). If you don't need all these features, just disable them. After it your app will be much faster.
Practical Solution I
(shadows, motion blur, depth comp, etc. are disabled)
Use the following properties to disable processor intensive effects:
override func viewDidLoad() {
super.viewDidLoad()
arView.renderOptions = [.disableDepthOfField,
.disableHDR,
.disableMotionBlur,
.disableFaceOcclusions,
.disablePersonOcclusion,
.disableGroundingShadows]
let boxAnchor = try! Experience.loadBox()
arView.scene.anchors.append(boxAnchor)
}
Practical Solution II
(shadows, depth comp, etc. are enabled by default)
When you use the following code in RealityKit:
override func viewDidLoad() {
super.viewDidLoad()
let boxAnchor = try! Experience.loadBox()
arView.scene.anchors.append(boxAnchor)
}
you get a Reality Composer's preconfigured scene containing horizontal plane detection property and AnchorEntity with the following settings:
AnchorEntity(.plane(.horizontal,
classification: .any,
minimumBounds: [0.25, 0.25])
Separating Tracking and Scene Understanding from Model Loading and Rendering
The problem you're having is a time lag that occurs at the moment your app launches. At the same moment starts world tracking (first stage) and then app tries simultaneously to detect a horizontal plane (second stage) and then it renders a metallic shader of a cube (third stage). To get rid of this time lag use this very simple approach (when app's launching you need to track a room and then tap on a screen to load a model):
override func viewDidLoad() {
super.viewDidLoad()
let tap = UITapGestureRecognizer(target: self,
action: #selector(self.tapped))
arView.addGestureRecognizer(tap)
}
#objc func tapped(_ sender: UITapGestureRecognizer) {
let boxAnchor = try! Experience.loadBox()
arView.scene.anchors.append(boxAnchor)
}
This way you reduce the simultaneous load on the CPU and GPU. So your cube is loading faster.
P.S.
Also, as an alternative you can use a loadModelAsync(named:in:) type method that allows you to load a model entity from a file in a bundle asynchronously:
static func loadModelAsync(named name: String,
in bundle: Bundle?) -> LoadRequest<ModelEntity>
In the default Experience.rcproject the cube has an AnchoringComponent with a horizontal plane. So basically the cube will not display until the ARSession finds any horizontal plane in your scene (for example the floor or a table). Once it finds that the cube will appear.
If you want instead to create and anchor and set that as the target when catching a tap event, you could perform a raycast. Using the result of a raycast, you can grab the worldTransform and set the cube's AnchoringComponent to that transform:
Something like this:
boxAnchor.anchoring = AnchoringComponent(.world(transform: raycastResult.worldTransform))
I am developing an app that needs to use the front facing camera of the iPhone for Augmented Reality experience using swift. I have tried to use the ARKit, but the front facing camera made by the ARKit is only supported for iPhone X.
So, which frameworks or libraries that I can use with swift to develop apps that has AR experience especially fro front facing camera, other than ARKit?
ARKit isn't the only way possible to create "AR" experiences on iOS, nor is it the only way that Apple permits creating "AR" in the App Store.
If you define "front-facing-camera AR" as something like "uses front camera, detects faces, allows placing virtual 2D/3D content overlays that appear to stay attached to the face", there are any number of technologies one could use. Apps like Snapchat have been doing this kind of "AR" since before ARKit existed, using technology they've either developed in-house or licensed from third parties. How you do it and how well it works depends on the technology you use. ARKit guarantees a certain precision of results by requiring a front-facing depth camera.
It's entirely possible to develop an app that uses ARKit for face tracking on TrueDepth devices and a different technology for other devices. For example, looking only at what you can do "out of the box" with Apple's SDK, there's the Vision framework, which locates and tracks faces in 2D. There's probably a few third party libraries out there, too... or you could go looking through academic journals, since face detection/tracking is a pretty active area of computer vision research.
ARKit 2.0
TrueDepth front-facing camera of iPhone X/Xr/Xs gives you Depth channel at 15 fps frame rate plus Image front-facing camera gives you RGB channels at 60 fps frame rate.
Principle of work: It's like Depth Sensing System in MS Xbox Kinect but more powerful. Infrared emitter projects over 30,000 dots in a known pattern onto the user’s face. Those dots are then photographed by a dedicated infrared camera for analysis. There is a proximity sensor, presumably so that the system knows when a user is close enough to activate. An ambient light sensor helps the system set output light levels.
At the moment only iPhone X/Xr/Xs models have TrueDepth Camera. If you don't have TrueDepth Camera and Sensor System in your iPhone (just like iPhone SE, iPhone 6s, iPhone 7 and iPhone 8 don't have) you cannot use your gadget for such features as Animoji, Face ID, or Depth Occlusion Effects.
In ARKit 2.0 framework a configuration that tracks the movement and expressions of the user’s face with the TrueDepth camera uses special class ARFaceTrackingConfiguration.
So, the answer is NO, you can use the front-facing camera of iPhones with A11 and A12 chipset (or higher version), or iPhone with TrueDepth Camera and its Sensor System.
ARKit 3.0 (addition)
Now ARKit allows you simultaneously track a surrounding environment with back camera and track you face with front camera. Also, you can track up to 3 faces at a time.
Here's a two code snippets how to setup you configuration.
First scenario:
let configuration = ARWorldTrackingConfiguration()
if configuration.supportsUserFaceTracking {
configuration.userFaceTrackingEnabled = true
}
session.run(configuration)
func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
for anchor in anchors where anchor is ARFaceAnchor {
// you code here...
}
}
Second scenario:
let configuration = ARFaceTrackingConfiguration()
if configuration.supportsWorldTracking {
configuration.worldTrackingEnabled = true
}
session.run(configuration)
func session(_ session: ARSession, didUpdate frame: ARFrame) {
let transform = frame.camera.transform
// you code here...
}
when i use the code
session.currentFrame?.rawFeaturePoints?.points.count
to get the point cloud data from the depth camera on the front of iPhone X，but，it turns out to be nil
Does anyone knows what I am missing？
I don’t know how to do it. Please advice.
The ARPointCloud is a set of points a representing intermediate results of the scene analysis ARKit uses to perform world tracking. Since the front camera of the iPhone X only supports face tracking, and not world tracking, maybe that's the reason you are getting nil.
Have you tried the back camera?
Feature point detection requires a ARWorldTrackingConfiguration session.
quoted from apple documentation (https://developer.apple.com/documentation/arkit/arframe/2887449-rawfeaturepoints).
You will not be able to get feature points from front camera.
If you still want to use front camera to get the depth data use this.
https://developer.apple.com/documentation/arkit/arframe/2928208-captureddepthdata
I hope it helps.
I am developing an ARKit app with OpenGL, so working directly with ARKit and not using SceneKit.
By default, ARKit is set to landscape orientation, but I have been unable to track down any documentation or examples to rotate to portrait. SceneKit example works in portrait but the Metal example only works in landscape.
Is it possible to change the ARKit tracking orientation?
I was able to solve this in the application logic by multiplying the camera matrix by a quaternion that is rotated based on device orientation.
let cameraQuaternion = simd_quatf(arCameraTransform)
minimapcamera.quaternion2 = cameraQuaternion * simd_quatf(angle: Float.pi/2, axis: SIMD3<Float>(0, 0, 1))
my solution based on accepted answer for portrait only
sorry on objective-c.
find and rewrite this
uniforms->viewMatrix = [frame.camera viewMatrixForOrientation:UIInterfaceOrientationLandscapeRight];
uniforms->projectionMatrix = [frame.camera projectionMatrixForOrientation:UIInterfaceOrientationLandscapeRight viewportSize:_viewportSize zNear:0.001 zFar:1000];
to
uniforms->viewMatrix = [frame.camera viewMatrixForOrientation:[[UIApplication sharedApplication] statusBarOrientation]];
uniforms->projectionMatrix = [frame.camera projectionMatrixForOrientation:[[UIApplication sharedApplication] statusBarOrientation] viewportSize:_viewportSize zNear:0.001 zFar:1000];
and
CGAffineTransform displayToCameraTransform = CGAffineTransformInvert([frame displayTransformForOrientation:UIInterfaceOrientationLandscapeRight viewportSize:_viewportSize]);
to
CGAffineTransform displayToCameraTransform = CGAffineTransformInvert([frame displayTransformForOrientation:[[UIApplication sharedApplication] statusBarOrientation] viewportSize:_viewportSize]);
Currently in iOS 11 Beta, there's only horizontal surfaces tracking for plane detection.
By default, plane detection is off. If you enable horizontal plane
detection, the session adds ARPlaneAnchor objects and notifies your
ARSessionDelegate , ARSCNViewDelegate , or ARSKViewDelegate object
whenever its analysis of captured video images detects an area that
appears to be a flat surface.
If you need vertical tracking, do it yourself with hitTest(_:types:) function. It allows you to check surfaces or objects in real world.
Hit testing searches for real-world objects or surfaces detected
through the AR session's processing of the camera image. A 2D point in
the image coordinates can refer to any point along a 3D line that
starts at the device camera and extends in a direction determined by
the device orientation and camera projection. This method searches
along that line, returning all objects that intersect it in order of
distance from the camera.
I am developing an augmented reality app for Project Tango using Unity3d.
Since I want to have virtual object interact with the real world, I use the Meshing with Physics scene from the examples as my basis and placed the Tango AR Camera prefab inside of the Tango Delta Camera (at the relative position (0,0,0)).
I found out, that I have to rotate the AR Camera up by about 17deg, so the Dynamic mesh matches the room, however there is still a significant offset to the live preview from the camera.
I was wondering, if anyone who had to deal with this before could share his solution to aligning the Dynamic Mesh with the real world.
How can I align the virtual world with the camera image?
I'm having similar issues. It looks like this is related to a couple of previously-answered questions:
Point cloud rendered only partially
Point Cloud Unity example only renders points for the upper half of display
You need to take into account the color camera offset from the device origin, which requires you to get the color camera pose relative to the device. You can't do this directly, but you can get the device in the IMU frame, and also the color camera in the IMU frame, to work out the color camera in the device frame. The links above show example code.
You should be looking at something like (in unity coordinates) a (0.061, 0.004, -0.001) offset and a 13 degree rotation up around the x axis.
When I try to use the examples, I get broken rotations, so take these numbers with a pinch of salt. I'm also seeing small rotations around y and z, which don't match with what I'd expect.