RealityKit – Convert CGPoint to a SIMD<Float> - swift

Question

I am using RealityKit and when I tap on the screen, I just want to add a box on that place where I tapped. It can be on the middle of the screen or anywhere. I will adjust Z axis so the object is placed 1 meter away from me. But I am having a hard time converting CGPoint to relevant RealityKit coordinates.
#objc func handleTap(_ recognizer: UITapGestureRecognizer) {
guard let view = view else { return }
let location = recognizer.location(in: view)
// location to SIMD3<Float> so I can create an anchor
}
Any ideas?

Answer 1

Model on a detected plane
CGPoint is a XY-point on iPhone's screen, so there's no need to convert it to XYZ-point. If you need a 3D point on a detected plane for accomodating your model, all you have to do is to generate an object from ARRaycastResult (or ARHitTestResult in case you're using ARKit+SceneKit).
import RealityKit
import ARKit
var arView = ARView(frame: .zero)
#objc func tappingScreen(_ sender: UITapGestureRecognizer) {
let results: [ARRaycastResult] = arView.raycast(from: arView.center,
allowing: .estimatedPlane,
alignment: .horizontal)
if let result: ARRaycastResult = results.first {
let model = ModelEntity(mesh: .generateSphere(radius: 0.02))
let anchor = AnchorEntity(world: result.worldTransform)
anchor.addChild(model)
arView.scene.anchors.append(anchor)
}
}
Model at camera's position
Creating a model exactly where ARCamera is located in the current frame is super easy:
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
let model = ModelEntity(mesh: .generateSphere(radius: 0.02))
let anchor = AnchorEntity(world: arView.cameraTransform.matrix)
anchor.addChild(model)
model.position.z = -0.5 // 50 cm offset
arView.scene.anchors.append(anchor)
}
Also, you can multiply camera's transform by desired offset:
var translation = matrix_identity_float4x4
translation.columns.3.z = -0.5 // 50 cm offset
// Multiplying camera matrix by offset matrix
let transform = simd_mul(arView.cameraTransform.matrix, translation)
model.transform.matrix = transform
Additional info
This post is also helpful.