ARKit - 2D Projection Result for a Point in 3D Space Changes Abruptly at Certain Device Position/Orientation - sprite-kit

I'm working on an ARKit app that uses SceneKit as the rendering engine. It does some typical ARKit things like allowing the user to place 3D virtual content in the scene, detect and draw horizontal and vertical planes, etc.
The app also includes a feature to create measurement objects like those found in Apple's Measure app. The user taps an "add endpoint" button to add the first point of the measurement, moves the device to target the location for the second, and then completes the measurement by tapping the "add endpoint" button again.
Measurement Working
The issue is that with certain device orientations and distances from the measurement objects, the measurement objects behave erratically. The first end point will suddenly change position and the line and first end point will appear to "flip" about the second endpoint.
Measurement "Flipping"
The measurement feature creates SpriteKit assets and adds them to the overlaySKScene of the sceneView. These measurements record their start and end point positions in 3D space and project these points to 2D to reposition their SpriteKit components.
This is how I'm currently updating the position of the measurement items:
func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
measurementNodes.forEach { $0.updateComponentPositions(renderer: sceneView) }
//...
}
The method updateComponentPositions is called every frame using the SCNSceneRendererDelegate method renderer(_, updateAtTime).
func updateComponentPositions(renderer: SCNSceneRenderer) {
// update start circle
let projectedStartPointScreenPosition = renderer.projectPoint(startPointPosition)
let startPointSKPosition = CGPoint(x: CGFloat(projectedStartPointScreenPosition.x),
y: scene.size.height - CGFloat(projectedStartPointScreenPosition.y) - startPointNode.frame.height/2)
startPointNode.position = startPointSKPosition
// update end circle
let projectedEndPointScreenPosition = renderer.projectPoint(endPointPosition)
let endPointSKPosition = CGPoint(x: CGFloat(projectedEndPointScreenPosition.x), y: scene.size.height - CGFloat(projectedEndPointScreenPosition.y) - endPointNode.frame.height/2)
endPointNode.position = endPointSKPosition
// update label
labelBackgroundNode.position = CGPoint(x: (startPointSKPosition.x + endPointSKPosition.x) / 2,
y: (startPointSKPosition.y + endPointSKPosition.y) / 2)
// reset line path
lineNode.path = getLinePath(from: startPointSKPosition,
to: endPointSKPosition)
}
This is the updateComponentPositions method of MeasurementNode which takes a renderer as an argument, projects the 3D points to 2D locations, and then updates the position of its geometries.
When attempting to debug this problem, I focused on the result of renderer.projectPoint(startPointPosition). The startPointPosition is a static SCNVector3 that holds the 3D position when the first endpoint was created. However, the result of this projectPoint method is what becomes erratic at this specific device orientation.
Here is a sample of the log of renderer.projectPoint(startPointPosition) when I hit that "flipping" point with the device:
SCNVector3(x: -6692.817, y: -13238.885, z: 0.49546456)
SCNVector3(x: -8706.296, y: -17154.777, z: 0.49417993)
SCNVector3(x: -12774.009, y: -25042.207, z: 0.491588)
SCNVector3(x: -24064.623, y: -46880.246, z: 0.48439798)
SCNVector3(x: -164018.86, y: -317208.25, z: 0.39538962) // magnitude becomes very large for x and y
SCNVector3(x: 17587.36, y: 33517.477, z: 0.510834) // x and y "flip"
SCNVector3(x: 9163.234, y: 17199.012, z: 0.50547314)
SCNVector3(x: 7649.4766, y: 14249.298, z: 0.5045147)
SCNVector3(x: 6688.1143, y: 12365.713, z: 0.5039065)
I tried replacing the updateComponentPositions method with one that takes the ARCamera as an argument in order to see if I would have better luck with its projectPoint(_ point: simd_float3, orientation: UIInterfaceOrientation, viewportSize: CGSize) method. I called it in that same renderer(_, updateAtTime) method, but achieved the same result.

Related

ARKit – Rendering a 3D object under an invisible plane

I have an ARKit scene with an invisible SCNPlane:
plane.geometry?.firstMaterial?.colorBufferWriteMask = []
This plane is placed on the ground and is used to render deferred shadows from other objects placed in the scene.
I want to render another SCNPlane which should be on the same level as the invisible plane (same Z-coordinate). The problem is, that every time the new object is under the invisible plane, it is not rendered at all.
Is there any way to render the object when it is under the invisible plane?
You can achieve it using the following lines of code:
shadowsPlane.geometry?.materials.first?.writesToDepthBuffer = true
shadowsPlane.geometry?.materials.first?.readsFromDepthBuffer = true
Choose one of two instance properties for .colorBufferWriteMask:
shadowsPlane.geometry?.materials.first?.colorBufferWriteMask = []
Set a rendering order for your objects like:
shadowsPlane.renderingOrder = -1 // the nearest layer
And, of course, use an appropriate .lightingModel instance property:
shadowsPlane.geometry?.materials.first?.lightingModel = .constant
Remember, there will be some tiny air gap between two planes:
shadowsPlane.position = SCNVector3(x: 0, y: 0, z: 0)
floorPlane.position = SCNVector3(x: 0, y: -0.01, z: 0)

ARKit - create world boundaries/get position at edge of visible plane?

Ok, what I am trying to do is create physics body/colliding boundaries for my character, my SCNNode, in my SceneKit game Im building with ARKit. This is so my node cannot move out of the user's vision/go so far away that it isn't visible as it is currently doing. My SCNNode is moved by user input, so I need to make "world boundaries" while ARKit still doesn't have vertical wall detection
I know you can place an object some set distance ahead of you in the real world as stated here Understand coordinate spaces in ARKit
and I have done that with this, just making a box with physics body here -
let box = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0) //change to be VERY TALL - need to make it a giant room
node.physicsBody = SCNPhysicsBody(type: SCNPhysicsBodyType.static, shape: nil)
box.firstMaterial?.diffuse.contents = UIColor.red
box.firstMaterial?.isDoubleSided = true
node = SCNNode(geometry: box)
node.position = SCNVector3(view.pointOfView.simdWorldFront + float3(0, 0, -0.5)) //random distance ahead
And this works, and I could add it as a child node of camera so it moves as user moves, but I don't think Im doing this correctly.
Essentially I need 4 walls to box the user/SCNNode in (corral the character) that are infinitely high, and at the VERY edge of the horizontal plane that the user can see. Really I don't know what this distance should be in the x plane:
+ float3(0, 0, -0.5)
How would you create AR boundaries like this?

GameplayKit: NPC element is going to very high floating number coordinates and back on update

I have plugged in my GKBehavior/GKAgent2D/GKComponent system for some behavioral routines and am having trouble fixing this bug I can't figure out.
Let me first be descriptive:
My current Entity/Component system is based on apple's ECS where:
1-entitymanager:GKEntity manages all components and contains all entities in the game and uses updateWithDelta which goes into all GKComponentSystems which themselves update every component within their system.
Currently, the update order is as follows:
behavior> node> playerNode> sound> interface> physics
2-Each system is a GKComponentSystem which handles updating all elements within.
I have two extensions of NSViewController called GameRenderer and GameControls which both are delegates of SCNSceneRendererDelegate and a custom protocol called KBAndTouchDelegate which update render on screen and control respectively (will be changing this to be incorporated in the ECS once my order updates feel right).
Now, in the behavior componentSystem, it updates a MoveComponent which contains a few functions for updating (willUpdate and didUpdate). Both have a 2DAgent position that is first taken from an SCNNode's position (converted from SCNVector3 to Float2 with just x and z coordinates being changed) then reflected back onto the node after the behavioral algorithm is evaluated.
Here are snippets for the updates:
func agentWillUpdate(agent: GKAgent) {
guard let nodeComponent = entity?.componentForClass(SCNNodeComponent.self)
else{
return
}
let nodePos = nodeComponent.node.position
position = float2(Float(nodePos.x), Float(nodePos.z))
}
func agentDidUpdate(agent: GKAgent) {
guard let nodeComponent = entity?.componentForClass(SCNNodeComponent.self)
else{
return
}
let pos = SCNVector3(x: CGFloat(position.x), y: nodeComponent.node.position.y, z: CGFloat(position.y))
print("\(nodeComponent.node.name) changed to \(pos)")
nodeComponent.node.position = pos
}
The #override updateWithDeltaTime routine checks for the party in which the moveComponent's owner is and placed in an array for getting the closest enemy distance and return the enemy.
The behavior has very low maxSpeed and maxVelocity factors.
Running the application, I printed a set of movements for two nodes which are enemies and monitor each other's position. The enemy search routine works fine as they reflect each other’s position in alternate sequence. The result is below:
Optional("zombie") changed to SCNVector3(x: -20.0, y: 0.0, z: -20.0)
enemy position is: (0.0, 0.0)
Optional("player") changed to SCNVector3(x: 0.0, y: 0.0, z: 0.0)
enemy position is: (-20.0, -20.0)
Optional("zombie") changed to SCNVector3(x: -20.0, y: 0.0, z: 41567.44140625)
enemy position is: (0.0, 0.0)
Optional("player") changed to SCNVector3(x: 0.0, y: 0.0, z: 0.0)
enemy position is: (-20.0, 41567.44140625)
HRTF loaded
Optional("zombie") changed to SCNVector3(x: 30.136764526367188, y: 0.0, z: -20.2421875)
enemy position is: (0.0, 0.0)
Optional("player") changed to SCNVector3(x: 0.0, y: 0.0, z: 0.0)
enemy position is: (30.1367645263672, -20.2421875)
Optional("zombie") changed to SCNVector3(x: -38636.87109375, y: 0.0, z: 15289.939453125)
If looking at the first positions of the "zombie" you will see it starts at -20, -20 then goes to impossibly big values for both x and z positions (again, x and z in the scene's world). The player doesn't move as I don't use any controls just to see if the zombie is able to find the player with the following GKBehavior call
init(targetSpeed: Float, seek: GKAgent, avoid: [GKAgent]){
super.init()
if targetSpeed>0{
setWeight(0.1, forGoal: GKGoal(toReachTargetSpeed: targetSpeed))
setWeight(0.5, forGoal: GKGoal(toSeekAgent: seek))
setWeight(1.0, forGoal: GKGoal(toAvoidAgents: avoid, maxPredictionTime: 1.0))
}
}
The above behavior is called in MoveComponent for the zombie's enemy (the player). Meaning the zombie will seek for the player.
Can anyone tell me what the hell these ginormous values for x and z are ? The zombie and player are about sqrt(800) or 28 meters away. But the zombie goes to stuff like 35800 meters away on x then switches back to about 50 then back and forth at each frame.
There’s a definite pattern and I’m not sure whether this GKBehavior uses A star at which point the probabilistic roadmap sets some very odd coordinate samples but nowhere in my code is there any scalar that would move the zombie so far away and flip the distances back to something reasonably normal every frame.
For anyone who might be having the same issue, as of yesterday 21-03-2016, the issue has been resolved by apple with their newest public update of x-code 7.3.

SceneKit – Rotate and animate a SCNNode

I'm trying to display a pyramid that points following the z axis and then rotates on itself around z too.
As my camera is on the z axis, I'm expecting to see the pyramid from above. I managed to rotate the pyramid to see it this way but when I add the animation it seems to rotate on multiple axis.
Here is my code:
// The following create the pyramid and place it how I want
let pyramid = SCNPyramid(width: 1.0, height: 1.0, length: 1.0)
let pyramidNode = SCNNode(geometry: pyramid)
pyramidNode.position = SCNVector3(x: 0, y: 0, z: 0)
pyramidNode.rotation = SCNVector4(x: 1, y: 0, z: 0, w: Float(M_PI / 2))
scene.rootNode.addChildNode(pyramidNode)
// But the animation seems to rotate aroun 2 axis and not just z
var spin = CABasicAnimation(keyPath: "rotation")
spin.byValue = NSValue(SCNVector4: SCNVector4(x: 0, y: 0, z: 1, w: 2*Float(M_PI)))
spin.duration = 3
spin.repeatCount = HUGE
pyramidNode.addAnimation(spin, forKey: "spin around")
Trying to both manually set and animate the same property can cause issues. Using a byValue animation makes the problem worse -- that concatenates to the current transform, so it's harder to keep track of whether the current transform is what the animation expects to start with.
Instead, separate the fixed orientation of the pyramid (its apex is in the -z direction) from the animation (it spins around the axis it points in). There's two good ways to do this:
Make pyramidNode the child of another node that gets the one-time rotation (π/2 around x-axis), and apply the spin animation directly to pyramidNode. (In this case, the apex of the pyramid will still point in the +y direction of its local space, so you'll want to spin around that axis instead of the z-axis.)
Use the pivot property to transform the local space of pyramidNode's contents, and animate pyramidNode relative to its containing space.
Here's some code to show the second approach:
let pyramid = SCNPyramid(width: 1.0, height: 1.0, length: 1.0)
let pyramidNode = SCNNode(geometry: pyramid)
pyramidNode.position = SCNVector3(x: 0, y: 0, z: 0)
// Point the pyramid in the -z direction
pyramidNode.pivot = SCNMatrix4MakeRotation(CGFloat(M_PI_2), 1, 0, 0)
scene.rootNode.addChildNode(pyramidNode)
let spin = CABasicAnimation(keyPath: "rotation")
// Use from-to to explicitly make a full rotation around z
spin.fromValue = NSValue(SCNVector4: SCNVector4(x: 0, y: 0, z: 1, w: 0))
spin.toValue = NSValue(SCNVector4: SCNVector4(x: 0, y: 0, z: 1, w: CGFloat(2 * M_PI)))
spin.duration = 3
spin.repeatCount = .infinity
pyramidNode.addAnimation(spin, forKey: "spin around")
Some unrelated changes to improve code quality:
Use CGFloat when explicit conversion is required to initialize an SCNVector component; using Float or Double specifically will break on 32 or 64 bit architecture.
Use .infinity instead of the legacy BSD math constant HUGE. This type-infers to whatever the type of spin.repeatCount is, and uses a constant value that's defined for all floating-point types.
Use M_PI_2 for π/2 to be pedantic about precision.
Use let instead of var for the animation, since we never assign a different value to spin.
More on the CGFloat error business: In Swift, numeric literals have no type until the expression they're in needs one. That's why you can do things like spin.duration = 3 -- even though duration is a floating-point value, Swift lets you pass an "integer literal". But if you do let d = 3; spin.duration = d you get an error. Why? Because variables/constants have explicit types, and Swift doesn't do implicit type conversion. The 3 is typeless, but when it gets assigned to d, type inference defaults to choosing Int because you haven't specified anything else.
If you're seeing type conversion errors, you probably have code that mixes literals, constants, and/or values returned from functions. You can probably just make the errors go away by converting everything in the expression to CGFloat (or whatever the type you're passing that expression to is). Of course, that'll make your code unreadable and ugly, so once you get it working you might start removing conversions one at a time until you find the one that does the job.
SceneKit includes animation helpers which are much simpler & shorter to use than CAAnimations. This is ObjC but gets across the point:
[pyramidNode runAction:
[SCNAction repeatActionForever:
[SCNAction rotateByX:0 y:0 z:2*M_PI duration:3]]];
I changed byValue to toValue and this worked for me. So change the line...
spin.byValue = NSValue(SCNVector4: SCNVector4(...
Change it to...
spin.toValue = NSValue(SCNVector4: SCNVector4(x: 0, y:0, z:1, w: 2*float(M_PI))

Swift - How to change the Pivot of a SCNNode object

I've been playing with the SCNNode object for a while now and I'm lost with the Pivot. How can I change the pivot of a SCNNode (SCNBox as a bar) and place the pivot on one of the edge of the bar?
A node's pivot is a transformation matrix, the inverse of which is applied to the node before its transform property takes effect. For example, take a look at this bit from the default SceneKit Game template in Xcode:
let boxNode = SCNNode()
boxNode.geometry = SCNBox(width: 1, height: 1, length: 1, chamferRadius: 0.02)
If you set the boxNode's position, that point corresponds to the center of the cube, and if you rotate it (as the template does in an animation), it spins around its center.
To change the anchor point, set the pivot to a translation transform:
boxNode.pivot = SCNMatrix4MakeTranslation(0.5, 0.5, 0.5)
Now, when you set the position that point corresponds to the top-right-front corner of the cube, and when you rotate the cube it spins around that corner.
More generally, a pivot transforms the contents of a node relative to the node's own transform. Suppose you wanted to model the precession of the Earth's axis of rotation. You could do this by creating two animations: one that animates pivot to spin the node around its own Y axis, and another that animates rotation to move that axis relative to the space containing the node.
On the pivot topic:
Just in case you do not have dimensions for your geometry/node something like this might help (especially for SCNText).
var minVec = SCNVector3Zero
var maxVec = SCNVector3Zero
if node.getBoundingBoxMin(&minVec, max: &maxVec) {
let bound = SCNVector3(x: maxVec.x + minVec.x,
y: maxVec.y + minVec.y,
z: maxVec.z + minVec.z)
node.pivot = SCNMatrix4MakeTranslation(bound.x / 2,
bound.y / 2,
bound.z / 2)
}