ARKit 1.5 – Vertical objects detection - swift

I use ARKit 1.5 and this func to highlight vertical surfaces, but it doesn't work really well.
func createPlaneNode(planeAnchor: ARPlaneAnchor) -> SCNNode {
let scenePlaneGeometry = ARSCNPlaneGeometry(device: metalDevice!)
scenePlaneGeometry?.update(from: planeAnchor.geometry)
let planeNode = SCNNode(geometry: scenePlaneGeometry)
planeNode.name = "\(currPlaneId)"
planeNode.opacity = 0.25
if planeAnchor.alignment == .vertical {
planeNode.geometry?.firstMaterial?.diffuse.contents = UIColor.red
}
currPlaneId += 1
return planeNode
}
It always finds some FeaturePoints on vertical objects but very rare it actually highlights the surface using the planeNode that I created.
I want to be able to detect and highlight things like a pillar or even a man. How would you approach this?
Image of object with featurePoints
Image with the result in best case scenario

In ARKit 1.5 and ARKit 2.0 there's .planeDetection instance property allowing you to enable .horizontal, .vertical, or both simultaneously .horizontal and .vertical detections.
var planeDetection: ARWorldTrackingConfiguration.PlaneDetection { get set }
ViewController's code:
let configuration = ARWorldTrackingConfiguration()
configuration.planeDetection = .vertical
//configuration.planeDetection = [.vertical, .horizontal]
sceneView.session.run(configuration)
If you want to successfully detect and track vertical objects in your environment, you need good lighting conditions and rich non-repetitive texture. Look at the picture below:

Related

SceneKit. How to achieve default scene light behavior without autoenablesDefaultLighting?

I tried to set autoenablesDefaultLighting=true for my SCNView and it looks good. However i want to achieve the same behavior without autoenablesDefaultLighting with setting light and adjust it a little bit.
I tried omni light with this code:
let lightNode = SCNNode()
lightNode.light = SCNLight()
lightNode.light?.castsShadow = true
lightNode.light?.type = .omni
lightNode.light?.intensity = 10000
lightNode.position = SCNVector3(x: 0, y: 0, z: 100)
scene.rootNode.addChildNode(lightNode)
And got this:
And with autoenablesDefaultLighting=true I got this:
Custom Default Lighting
I believe, in SceneKit, the default scene lighting is a Directional Light without any shadows, attached directly to the default camera node (i.e. pointOfView node). To simulate the same lighting conditions as when the .autoenablesDefaultLighting property is true, use the following code:
Delegate's renderer method – light's position orientation will be updated 60 times per second:
import SceneKit
extension GameViewController: SCNSceneRendererDelegate {
func renderer(_ renderer: SCNSceneRenderer,
updateAtTime time: TimeInterval) {
sunNode.transform = (sceneView?.pointOfView?.worldTransform)!
let cameraAngles = (self.sceneView?.pointOfView?.eulerAngles)!
let lightAngles = self.sunNode.eulerAngles
print("Camera: " + String(format: "%.2f, %.2f, %.2f", cameraAngles.x,
cameraAngles.y,
cameraAngles.z))
print("Light: " + String(format: "%.2f, %.2f, %.2f", lightAngles.x,
lightAngles.y,
lightAngles.z))
}
}
Here's GameViewController class:
class GameViewController: NSViewController {
var sceneView: SCNView? = nil
let sunNode = SCNNode()
override func viewDidLoad() {
super.viewDidLoad()
sceneView = self.view as? SCNView
sceneView?.delegate = self
let scene = SCNScene(named: "ship.scn")!
sceneView?.scene = scene
sceneView?.scene?.lightingEnvironment.contents = .none
sceneView?.scene?.background.contents = .none
sceneView?.backgroundColor = .black
sceneView?.allowsCameraControl = true
// sceneView?.autoenablesDefaultLighting = true
sunNode.light = SCNLight()
sunNode.light?.type = .directional
sceneView?.scene?.rootNode.addChildNode(sunNode)
}
}
Explanations
I'd like to add that if there is no light in the scene at all (including the autoenablesDefaultLighting parameter), then the only uncontrollable source of light in the scene will be the non-switchable Ambient Light.
In addition to the above, the Physically Based shader always requires additional Ambient Light fixture (otherwise the physically based surface will be black). The location and orientation of this light source does not matter.
If Directional Light illuminates the surface perpendicularly, then the surface is illuminated with 100% intensity (default intensity is 1000 lumens), but if the rays of the light source are parallel to the surface, then the surface is not illuminated by this source.
As you can see, the first and last images have identical lighting environment.

SceneKit – Loading HDR or EXR lightingEnvironment has no effect

I tried loading an .hdr file to use it as a skybox and use its lighting informations. This is the code I used:
backgroundColor = UIColor.gray
// check if a default skybox is added
let environment = UIImage(named: "studio_small_09_2k.hdr")
scene?.lightingEnvironment.contents = environment
scene?.lightingEnvironment.intensity = 1.0
scene?.background.contents = environment
Unfortunately I recieve a grey screen and also no errors. Has anyone experience in using hdr files in SceneKit?
XCode Version: 13.2.1
iOS version: 15.3.1
hdr file: https://polyhaven.com/a/studio_small_09
I usually use a Cube Texture Set, where each of 6 images is square (height == width).
Also, the following cube map representations are supported:
Vertical strip as single image (height == 6 * width)
Horizontal strip as single image (6 * height == width)
Spherical projection as single image (2 * height == width)
Here's a SwiftUI code:
func makeUIView(context: Context) -> SCNView {
let sceneView = SCNView(frame: .zero)
sceneView.scene = SCNScene()
// if EXR or HDR is 2:1 spherical map, it really meets the requirements
sceneView.scene?.lightingEnvironment.contents = UIImage(named: "std.exr")
sceneView.backgroundColor = .black
sceneView.autoenablesDefaultLighting = true
sceneView.allowsCameraControl = true
let node = SCNNode()
node.geometry = SCNSphere(radius: 0.1)
node.geometry?.firstMaterial?.lightingModel = .physicallyBased
node.geometry?.firstMaterial?.metalness.contents = 1.0
sceneView.scene?.rootNode.addChildNode(node)
return sceneView
}
Pay particular attention – you need .physicallyBased lighting model to get HDR or EXR reflections.
And let's set it for BG:
sceneView.scene?.background.contents = UIImage(named: "std.exr")
Why your .exr doesn't work?
The solutions is simple: delete your .exr from project, empty the Trash and after that drag-and-drop .exr file, in Choose options for adding these files window choose Add to targets:
Now your .exr must work.

SceneKit LIDAR iOS: Show unscanned regions of camera view in the background with a different color/texture

I'm building an app similar to Polycam, 3D Scanner App, Scaniverse, etc. I visualize a mesh for scanned regions and export it into different formats. I would like to show the user what regions are scanned, and what not. To do so, I need to differentiate between them.
My idea is to build something like Polycam does..
< Polycam blue background for unscanned regions >
I tried changing the background content property of the scene, but it causes the whole camera view to be replaced by the color.
arSceneView.scene.background.contents = UIColor.black
I'm using ARSCNView and setting up plane detection as follows:
private func setupPlaneDetection() {
let configuration = ARWorldTrackingConfiguration()
configuration.planeDetection = [.horizontal, .vertical]
configuration.sceneReconstruction = .meshWithClassification
configuration.frameSemantics = .smoothedSceneDepth
arSceneView.session.run(configuration)
arSceneView.session.delegate = self
// arSceneView.scene.background.contents = UIColor.black
arSceneView.delegate = self
UIApplication.shared.isIdleTimerDisabled = true
arSceneView.showsStatistics = true
}
Thanks in advance for any help you can provide!
I’ve done this before by adding a sphere to the scene with a two-sided material (slightly transparent) and with a radius large enough that the camera and the scanned surface will always be inside of it. Here’s an example of how to do that:
let backgroundSphereNode = SCNNode()
backgroundSphereNode.geometry = SCNSphere(radius: 500)
let material = SCNMaterial()
material.isDoubleSided = true
material?.diffuse.contents = UIColor(white: 0, alpha: 0.9)
backgroundSphereNode.geometry?.materials = [material]
Note that I’m using a black color - you can obviously change this to whatever you need, but keep the alpha channel slightly transparent. And tweak the radius of the sphere so it works for your scene.

SceneKit / ARKit updating a node every frame

I'm working with ARKit / SceneKit and I'm trying to have an arrow point to an arbitrary position I set in the world, but I'm having a bit of trouble. In my sceneView I have a scene set up to load in my arrow.
override func viewDidLoad() {
super.viewDidLoad()
// Set the view's delegate
sceneView.delegate = self
guard let arrowScene = SCNScene(named: "art.scnassets/arrowScene.scn") else {
fatalError("Scene arrow.scn not found")
}
let worldAnchor = ARAnchor(name: "World Anchor", transform: simd_float4x4(1))
sceneView.session.add(anchor: worldAnchor)
sceneView.scene = (arrowScene)
}
I wish to then use SCNNode.look(at:) to point my arrow at the specified anchor, however, I am unsure how to get this to occur every frame. I know that I have access to special delegates provided by ARSCNView such as renderer(willUpdate node:), but I am unsure how to use these when the anchor is not changing positions whereas the arrow is.
Thanks for the help!
You could do so by using the updateAtTime delegate function, but I strongly recommend you to use a SCNConstraint.
let lookAtConstraint = SCNLookAtConstraint(target: myLookAtNode)
lookAtConstraint.isGimbalLockEnabled = true // useful for cameras, probably also for your arrow.
// in addition you can use a position constraint
let positionConstraint = SCNReplicatorConstraint(target: myLookAtNode)
positionConstraint.positionOffset = yourOffsetSCNVector3
positionConstraint.replicatesOrientation = false
positionConstraint.replicatesScale = false
positionConstraint.replicatesPosition = true
// then add the constraints to your arrow node
arrowNode.constraints = [lookAtConstraint, positionConstraint ]
This will automatically adjust your node for each rendered frame.

AR with iOS: putting a light in the scene makes everything black?

Ok, I am trying desperately to achieve this sort of warm lighting on my objects when added to my ARScene in Swift/Xcode - warm lighting and little glowing lights around:
To be clear, I do NOT want the objects I add to my scene to look like they belong in the surrounding room. I want them to stand out/ look warm and glow.All the tutorials on ARKit teach you how to mimic the lighting of the actual room.
Xcode has several lighting options, pulling from the surroundings gathered by the camera because with:
if let lightEstimate = session.currentFrame?.lightEstimate
I can print out the warmth, intensity, etc. And I also have these properties currently set to match the light of room:
sceneView.automaticallyUpdatesLighting = true
extension ARSCNView {
func setup() { //SCENE SETUP
antialiasingMode = .multisampling4X
autoenablesDefaultLighting = true
preferredFramesPerSecond = 60
contentScaleFactor = 1.3
if let camera = pointOfView?.camera {
camera.wantsHDR = true
camera.wantsExposureAdaptation = true
camera.exposureOffset = -1
camera.minimumExposure = -1
camera.maximumExposure = 3
}
}
}
I have tried upping the emission on my object's textures and everything but nothing achieves the effect. Adding a light just turns the objects black/no color.
What is wrong here?
To create this type of glowing red neon light result in ARKit. You can do the following.
You need to create a reactor.scnp (scenekit particle System File) and make the following changes to create the glowing red halo. This should be place in your Resources directory of the playground along with the file spark.png
These are the settings to change from the default reactor type. Leave all the other settings alone.
Change the Image animate color to red/orange/red/black
speed factor = 0.1
enable lighting checked
Emitter Shape = Sphere
Image Size = 0.5
Image Intensity = 0.1
Simulation Speed Factor = 0.1
Note: The code below is playground app I use for testing purposes. You just tap anywhere to add the Neon light into the scene. You can place as many neon lights as you like.
import ARKit
import SceneKit
import PlaygroundSupport
import SceneKit.ModelIO
class ViewController: NSObject {
var sceneView: ARSCNView
init(sceneView: ARSCNView) {
self.sceneView = sceneView
super.init()
self.setupWorldTracking()
self.sceneView.addGestureRecognizer(UITapGestureRecognizer(target: self, action: #selector(ViewController.handleTap(_:))))
}
private func setupWorldTracking() {
if ARWorldTrackingConfiguration.isSupported {
let configuration = ARWorldTrackingConfiguration()
configuration.planeDetection = .horizontal
configuration.isLightEstimationEnabled = true
self.sceneView.session.run(configuration, options: [])
}
}
#objc func handleTap(_ gesture: UITapGestureRecognizer) {
let results = self.sceneView.hitTest(gesture.location(in: gesture.view), types: ARHitTestResult.ResultType.featurePoint)
guard let result: ARHitTestResult = results.first else {
return
}
let cylinder = SCNCylinder(radius: 0.05, height: 1)
cylinder.firstMaterial?.emission.contents = UIColor.red
cylinder.firstMaterial?.emission.intensity = 1
let spotLight = SCNNode()
spotLight.light = SCNLight()
spotLight.scale = SCNVector3(1,1,1)
spotLight.light?.intensity = 1000
spotLight.castsShadow = true
spotLight.position = SCNVector3Zero
spotLight.light?.type = SCNLight.LightType.directional
spotLight.light?.color = UIColor.white
let particleSystem = SCNParticleSystem(named: "reactor", inDirectory: nil)
let systemNode = SCNNode()
systemNode.addParticleSystem(particleSystem!)
let node = SCNNode(geometry: cylinder)
let position = SCNVector3Make(result.worldTransform.columns.3.x, result.worldTransform.columns.3.y, result.worldTransform.columns.3.z)
systemNode.position = position
node.position = position
self.sceneView.scene.rootNode.addChildNode(spotLight)
self.sceneView.scene.rootNode.addChildNode(node)
self.sceneView.scene.rootNode.addChildNode(systemNode)
}
}
let sceneView = ARSCNView()
let viewController = ViewController(sceneView: sceneView)
sceneView.autoenablesDefaultLighting = false
PlaygroundPage.current.needsIndefiniteExecution = true
PlaygroundPage.current.liveView = viewController.sceneView
If your looking for a neon/glowing effect in your scene... these previous answers to a similar question asked about glowing/neon lighting should give you some guidance.
As you will see from the answers provided sceneKit does not have built-in support for volumetric lighting, all the approaches are more hacks to achieve a similar effect to a glowing light.
iOS SceneKit Neon Glow
To add a "red" directional light effect to your scene... which is an alternative to using sceneView.autoenablesDefaultLighting = true
let myLight = SCNNode()
myLight.light = SCNLight()
myLight.scale = SCNVector3(1,1,1)
myLight.intensity = 1000
myLight.position = SCNVector3Zero
myLight.light?.type = SCNLight.LightType.directional
myLight.light?.color = UIColor.red
// add the light to the scene
sceneView.scene.rootNode.addChildNode(myLight)
note: This effect makes all the objects in the scene more reddish.