I need to convert a point in the 2d coordinate space of my ARSCNView to a coordinate in 3d space. Basically a ray from the point of view to the touched location (up to a set distance away).
I wanted to use arView.unprojectPoint(vec2d) for that, but the point returned always seems to be located in the center of the view
vec2d is a SCNVector3 created from a 2d coordinate like this
SCNVector3(x, y, 0) // 0 specifies camera near plane
What am I doing wrong? How do I get the desired result?
I think you have at least 2 possible solutions:
First
Use hitTest(_:types:) instance method:
This method searches for real-world objects or AR anchors in the captured camera image corresponding to a point in the SceneKit view.
let sceneView = ARSCNView()
func calculateVector(point: CGPoint) -> SCNVector3? {
let hitTestResults = sceneView.hitTest(point,
types: [.existingPlane])
if let result = hitTestResults.first {
return SCNVector3.init(SIMD3(result.worldTransform.columns.3.x,
result.worldTransform.columns.3.y,
result.worldTransform.columns.3.z))
}
return nil
}
calculateVector(point: yourPoint)
Second
Use unprojectPoint(_:ontoPlane:) instance method:
This method returns the projection of a point from 2D view onto a plane in the 3D world space detected by ARKit.
#nonobjc func unprojectPoint(_ point: CGPoint,
ontoPlane planeTransform: simd_float4x4) -> simd_float3?
or:
let point = CGPoint()
var planeTransform = simd_float4x4()
sceneView.unprojectPoint(point,
ontoPlane: planeTransform)
Add a empty node infront of camera at 'x' cm offset and making it the child of camera.
//Add a node in front of camera just after creating scene
hitNode = SCNNode()
hitNode!.position = SCNVector3Make(0, 0, -0.25) //25 cm offset
sceneView.pointOfView?.addChildNode(hitNode!)
func unprojectedPosition(touch: CGPoint) -> SCNVector3 {
guard let hitNode = self.hitNode else {
return SCNVector3Zero
}
let projectedOrigin = sceneView.projectPoint(hitNode.worldPosition)
let offset = sceneView.unprojectPoint(SCNVector3Make(Float(touch.x), Float(touch.y), projectedOrigin.z))
return offset
}
See the Justaline GitHub implementation of the code here
Related
I am working on a project where I have to place a green dot to be always in the center even when we rotate the camera in ARKit. I am using ARSCNView and I have added the node so far everything is good. Now I know I need to modify the position of the node in
func session(_ session: ARSession, didUpdate frame: ARFrame)
But I have no idea how to do that. I saw some example which was close to what I have but it does not run as it suppose to.
func session(_ session: ARSession, didUpdate frame: ARFrame) {
let location = sceneView.center
let hitTest = sceneView.hitTest(location, types: .featurePoint)
if hitTest.isEmpty {
print("No Plane Detected")
return
} else {
let columns = hitTest.first?.worldTransform.columns.3
let position = SCNVector3(x: columns!.x, y: columns!.y, z: columns!.z)
var node = sceneView.scene.rootNode.childNode(withName: "CenterShip", recursively: false) ?? nil
if node == nil {
let scene = SCNScene(named: "art.scnassets/ship.scn")!
node = scene.rootNode.childNode(withName: "ship", recursively: false)
node?.opacity = 0.7
let columns = hitTest.first?.worldTransform.columns.3
node!.name = "CenterShip"
node!.position = SCNVector3(x: columns!.x, y: columns!.y, z: columns!.z)
sceneView.scene.rootNode.addChildNode(node!)
}
let position2 = node?.position
if position == position2! {
return
} else {
//action
let action = SCNAction.move(to: position, duration: 0.1)
node?.runAction(action)
}
}
}
It doesn't matter how I rotate the camera this dot must be in the middle.
It's not clear exactly what you're trying to do, but I assume its one of the following:
A) Place the green dot centered in front of the camera at a fixed distance, eg. always exactly 1 meter in front of the camera.
B) Place the green dot centered in front of the camera at the depth of the nearest detected plane, i.e. using the results of a raycast from the mid point of the ARSCNView
I would have assumed A, but your example code is using (now deprecated) sceneView.hitTest() function which in this case would give you the depth of whatever is behind the pixel at sceneView.center
Anyway here's both:
Fixed Depth Solution
This is pretty straightforward, though there are few options. The simplest is to make the green dot a child node of the scene's camera node, and give it position with a negative z value, since z increases as a position moves toward the camera.
cameraNode.addChildNode(textNode)
textNode.position = SCNVector3(x: 0, y: 0, z: -1)
As the camera moves, so too will its child nodes. More details in this very thorough answer
Scene Depth Solution
To determine the estimated depth behind a pixel, you should use ARSession.raycast instead of SceneView.hitTest, because the latter is definitely deprecated.
Note that, if the raycast() (or still hitTest()) methods return an empty result set (not uncommon given the complexity of scene estimation going on in ARKit), you won't have a position to update the node and this it might not be directly centered in every frame. To handle this is a bit more complex, as you'd need decide exactly what you want to do in that case.
The SCNAction is unnecessary and potentially causing problems. These delegate methods run 60fps, so simply updating the position directly will produce smooth results.
Adapting and simplifying the code you posted:
func createCenterShipNode() -> SCNNode {
let scene = SCNScene(named: "art.scnassets/ship.scn")!
let node = scene.rootNode.childNode(withName: "ship", recursively: false)
node!.opacity = 0.7
node!.name = "CenterShip"
sceneView.scene.rootNode.addChildNode(node!)
return node!
}
func session(_ session: ARSession, didUpdate frame: ARFrame) {
// Check the docs for what the different raycast query parameters mean, but these
// give you the depth of anything ARKit has detected
guard let query = sceneView.raycastQuery(from: sceneView.center, allowing: .estimatedPlane, alignment: .any) else {
return
}
let results = session.raycast(query)
if let hit = results.first {
let node = sceneView.scene.rootNode.childNode(withName: "CenterShip", recursively: false) ?? createCenterShipNode()
let pos = hit.worldTransform.columns.3
node.simdPosition = simd_float3(pos.x, pos.y, pos.z)
}
}
See also: ARRaycastQuery
One last note - you generally don't want to do scene manipulation within this delegate method. It runs on a different thread than the Scenekit rendering thread, and SceneKit is very thread sensitive. This will likely work fine, but beyond adding or moving a node will certainly cause crashes from time to time. You'd ideally want to store the new position, and then update the actual scene contents from within the renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) delegate method.
The ARFaceTrackingConfiguration of ARKit places ARFaceAnchor with information about the position and orientation of the face onto the scene. Among others, this anchor has the lookAtPoint property that I'm interested in. I know that this vector is relative to the face. How can I draw a point on the screen for this position, meaning how can I translate this point's coordinates?
.lookAtPoint instance property is for direction's estimation only.
Apple documentation says: .lookAtPoint is a position in face coordinate space that is estimating only the gaze of face's direction. It's a vector of three scalar values, and it's just gettable, not settable:
var lookAtPoint: SIMD3<Float> { get }
In other words, this is the resulting vector from the product of two quantities – .rightEyeTransform and .leftEyeTransform instance properties (which also are just gettable):
var rightEyeTransform: simd_float4x4 { get }
var leftEyeTransform: simd_float4x4 { get }
Here's an imaginary situation on how you could use this instance property:
func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
if let faceAnchor = anchor as? ARFaceAnchor,
let faceGeometry = node.geometry as? ARSCNFaceGeometry {
if (faceAnchor.lookAtPoint.x >= 0) { // Looking (+X)
faceGeometry.firstMaterial?.diffuse.contents = UIImage(named: "redTexture.png")
} else { // Looking (-X)
faceGeometry.firstMaterial?.diffuse.contents = UIImage(named: "cyanTexture.png")
}
faceGeometry.update(from: faceAnchor.geometry)
facialExrpession(anchor: faceAnchor)
DispatchQueue.main.async {
self.label.text = self.textBoard
}
}
}
And here's an image showing axis directions for ARFaceTrackingConfiguration():
Answering your question:
I could say that you can't manage this point's coordinates directly because it's gettable-only property (and there is just XYZ orientation, not XYZ translation).
So if you need both – translation and rotation – use .rightEyeTransform and .lefttEyeTransform instance properties instead.
There are two projection methods:
FIRST. In SceneKit/ARKit you need to take the following instance method for projecting a point onto 2D view (for sceneView instance):
func projectPoint(_ point: SCNVector3) -> SCNVector3
or:
let sceneView = ARSCNView()
sceneView.projectPoint(myPoint)
SECOND. In ARKit you need to take the following instance method for projecting a point onto 2D view (for arCamera instance):
func projectPoint(_ point: simd_float3,
orientation: UIInterfaceOrientation,
viewportSize: CGSize) -> CGPoint
or:
let camera = ARCamera()
camera.projectPoint(myPoint, orientation: myOrientation, viewportSize: vpSize)
This method helps you project a point from the 3D world coordinate system of the scene to the 2D pixel coordinate system of the renderer.
There's also the opposite method (for unprojecting a point):
func unprojectPoint(_ point: SCNVector3) -> SCNVector3
...and ARKit's opposite method (for unprojecting a point):
#nonobjc func unprojectPoint(_ point: CGPoint,
ontoPlane planeTransform: simd_float4x4,
orientation: UIInterfaceOrientation,
viewportSize: CGSize) -> simd_float3?
The following code places the node in front of the camera but always at the center 10cm away from the camera position. I want to place the node 10cm away in z-direction but at the x and y co-ordinates of where I touch the screen. So touching on different parts of the screen should result in a node being placed 10cm away in front of the camera but at the x and y location of the touch and not always at the center.
var cameraRelativePosition = SCNVector3(0,0,-0.1)
let sphere = SCNNode()
sphere.geometry = SCNSphere(radius: 0.0025)
sphere.geometry?.firstMaterial?.diffuse.contents = UIColor.white
Service.addChildNode(sphere, toNode: self.sceneView.scene.rootNode,
inView: self.sceneView, cameraRelativePosition:
cameraRelativePosition)
Service.swift
class Service: NSObject {
static func addChildNode(_ node: SCNNode, toNode: SCNNode, inView:
ARSCNView, cameraRelativePosition: SCNVector3) {
guard let currentFrame = inView.session.currentFrame else { return }
let camera = currentFrame.camera
let transform = camera.transform
var translationMatrix = matrix_identity_float4x4
translationMatrix.columns.3.x = cameraRelativePosition.x
translationMatrix.columns.3.y = cameraRelativePosition.y
translationMatrix.columns.3.z = cameraRelativePosition.z
let modifiedMatrix = simd_mul(transform, translationMatrix)
node.simdTransform = modifiedMatrix
toNode.addChildNode(node)
}
}
The result should look exactly like this : https://justaline.withgoogle.com
We can use the unprojectPoint(_:) method of SCNSceneRenderer (SCNView and ARSCNView both conform to this protocol) to convert a point on the screen to a 3D point.
When tapping the screen we can calculate a ray this way:
func getRay(for point: CGPoint, in view: SCNSceneRenderer) -> SCNVector3 {
let farPoint = view.unprojectPoint(SCNVector3(Float(point.x), Float(point.y), 1))
let nearPoint = view.unprojectPoint(SCNVector3(Float(point.x), Float(point.y), 0))
let ray = SCNVector3Make(farPoint.x - nearPoint.x, farPoint.y - nearPoint.y, farPoint.z - nearPoint.z)
// Normalize the ray
let length = sqrt(ray.x*ray.x + ray.y*ray.y + ray.z*ray.z)
return SCNVector3Make(ray.x/length, ray.y/length, ray.z/length)
}
The ray has a length of 1, so by multiplying it by 0.1 and adding the camera location we get the point you were searching for.
I am trying to put SCNCylinder node in the scene on the touch point. I always want to show the cylinder shape diameter facing towards camera. Its working fine for horizontal scene but it have a problem in vertical scene. In vertical scene I can see the cylinder sides but I want to show the full diameter facing towards the camera no matter whats the camera orientation is. I know there is some transformation needs to be applied depending on the camera transform but don't know how. I am not using plane detection its the simple node which is directly added to the scene.
Vertical Image:
Horizontal Image:
The code to insert the node is as follows,
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
guard let touch = touches.first else {
return
}
let result = sceneView.hitTest(touch.location(in: sceneView), types: [ARHitTestResult.ResultType.featurePoint])
guard let hitResult = result.last else {
print("returning because couldn't find the touch point")
return
}
let hitTransform = SCNMatrix4(hitResult.worldTransform)
let position = SCNVector3Make(hitTransform.m41, hitTransform.m42, hitTransform.m43)
let ballShape = SCNCylinder(radius: 0.02, height: 0.01)
let ballNode = SCNNode(geometry: ballShape)
ballNode.position = position
sceneView.scene.rootNode.addChildNode(ballNode)
}
Any help would be appreciated.
I'm not certain this is the right way to handle what you need but here is something which may help you.
I think CoreMotion could be useful to help you determine if the device is at a horizontal or vertical angle.
This class has a property called attitude, which describes the rotation of our device in terms of roll, pitch, and yaw. If we are holding our phone in portrait orientation, the roll describes the angle of rotation about the axis that runs through the top and bottom of the phone. The pitch describes the angle of rotation about the axis that runs through the sides of your phone (where the volume buttons are). And finally, the yaw describes the angle of rotation about the axis that runs through the front and back of your phone. With these three values, we can determine how the user is holding their phone in reference to what would be level ground (Stephan Baker).
Begin by importing CoreMotion:
import CoreMotion
Then create the following variables:
let deviceMotionDetector = CMMotionManager()
var currentAngle: Double!
We will then create a function which will check the angle of our device like so:
/// Detects The Angle Of The Device
func detectDeviceAngle(){
if deviceMotionDetector.isDeviceMotionAvailable == true {
deviceMotionDetector.deviceMotionUpdateInterval = 0.1;
let queue = OperationQueue()
deviceMotionDetector.startDeviceMotionUpdates(to: queue, withHandler: { (motion, error) -> Void in
if let attitude = motion?.attitude {
DispatchQueue.main.async {
let pitch = attitude.pitch * 180.0/Double.pi
self.currentAngle = pitch
print(pitch)
}
}
})
}
else {
print("Device Motion Unavailable");
}
}
This only needs to be called once for example in viewDidLoad:
detectDeviceAngle()
In your touchesBegan method you can add this to the end:
//1. If We Are Holding The Device Above 60 Degress Change The Node
if currentAngle > 60 {
//2a. Get The X, Y, Z Values Of The Desired Rotation
let rotation = SCNVector3(1, 0, 0)
let vector3x = rotation.x
let vector3y = rotation.y
let vector3z = rotation.z
let degreesToRotate:Float = 90
//2b. Set The Position & Rotation Of The Object
sphereNode.rotation = SCNVector4Make(vector3x, vector3y, vector3z, degreesToRotate * 180 / .pi)
}else{
}
I am sure there are better ways to achieve what you need (and I would be very interested in hearing them too), but I hope it will get you started.
Here is the result:
In my program, I am trying to make myself collidable with other objects so that I can test collisions. Is there a way to add a physics body that moves with myself in the new AR Kit which follows you? I can make one which is at my place at the beginning of the AR but I'm sure there is a way to add it to your actual self.
Good day!
There is a solution to your problem.
First of all you need to create a collider. For example:
func addCollider(position: SCNVector3, name: String) {
let collider = SCNNode(geometry: SCNSphere(radius: 0.5))
collider.position = position
collider.geometry?.firstMaterial?.diffuse.contents = UIColor.clear
collider.physicsBody?.categoryBitMask = 0 // example
collider.physicsBody?.contactTestBitMask = 1 // example
collider.name = name
sceneView.scene.rootNode.addChildNode(collider)
}
Don't forget to add proper bit masks, to collide with other objects.
Next you need to implement ARSCNViewDelegate protocol and call method willRenderScene. Next you need to locate your point of view and transform it into the matrix. Than you need to extract values from third column of the matrix to get current orientation. To get current location you need to extract values of forth column from your matrix. And finally you need to combine your location and orientation to get your current position.
To combine to SCNVector3 values you need to write global function "+". Example:
func +(left: SCNVector3, right: SCNVector3) -> SCNVector3 {
return SCNVector3Make(left.x + right.x, left.y + right.y, left.z + right.z)
}
Be sure, that you wrote this method outside of class or struct.
Final accord! A function that will move a collider accordingly to your current position.
func renderer(_ renderer: SCNSceneRenderer, willRenderScene scene: SCNScene, atTime time: TimeInterval) {
guard let pontOfView = self.sceneView.pointOfView else { return }
let transform = pontOfView.transform
let orientation = SCNVector3(-transform.m31, -transform.m32, -transform.m33)
let location = SCNVector3(transform.m41, transform.m42, transform.m43)
let currentPosition = orientation + location
for node in sceneView.scene.rootNode.childNodes {
if node.name == "your collider name" {
node.position = currentPosition
}
}
}
P.S. Don't forget to add
self.sceneView.delegate = self
to your viewDidLoad.
Best of luck. I hope it helped!