I am using the following to to place an object on a plane. It works but the position of the anchor is always coming out to be 0,0,0 even though it is based on the raycast result.
let results = arView.raycast(from: tappedLocation,
allowing: .estimatedPlane,
alignment: .horizontal)
if let result = results.first {
print(result.worldTransform) // contains values
let anchor = AnchorEntity(raycastResult: result)
print(anchor.position) // why 0,0,0
}
Any ideas?
It seems that position becomes zero when anchor is nil (transform becomes identity matrix).
If we call raycast with .existingPlaneGeometry, anchor will not be nil.
AnchorEntity has world space position inside.
This is not included in transform.
If we want to know world space position, we can calculate it by the following code. (It seems that there is no property that can be obtained...)
let anchorWorldTransform = anchorEntity.convert(transform: .identity, to: nil)
Alternative approach
If .init(raycastResult:) doesn't work, use alternative approach with .init(world:) initializer.
fileprivate func raycaster() {
guard let query = arView.makeRaycastQuery(from: arView.center,
allowing: .existingPlaneInfinite,
alignment: .horizontal)
else { return }
if let result = arView.session.raycast(query).first {
let anchor = AnchorEntity(world: result.worldTransform)
anchor.addChild(self.yourModel)
arView.scene.anchors.append(anchor)
}
}
Then:
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
self.raycaster()
}
Modifying your code
Also, you may modify your code adding these lines:
let anchor = AnchorEntity(raycastResult: result)
let transform = Transform(matrix: result.worldTransform)
anchor.transform = transform
Related
Here is my previous question about in general apply force for a certain point of an AR object which had a perfect answer.
I have managed to apply force to a given point with a little bit of tinkering to have a perfect effect for me. Let me show also some code.
I get the AR object from Experience like:
if let skateAnchor = try? Experience.loadSkateboard(),
let skateEntity = skateAnchor.skateboard {
guard let entity = skateEntity as? HasPhysicsBody else { return }
skateAnchor.generateCollisionShapes(recursive: true)
entity.collision?.filter.mask = [.sceneUnderstanding]
skateboard = entity
}
Afterwards I set up the plane and the LiDAR scanner and add some gestures to it like:
let arViewTap = UITapGestureRecognizer(target: self,
action: #selector(tapped(sender:)))
arView.addGestureRecognizer(arViewTap)
let arViewLongPress = UILongPressGestureRecognizer(target: self,
action: #selector(longPressed(sender:)))
arView.addGestureRecognizer(arViewLongPress)
So far so good, on tap gesture I apply the logic from the previously linked answer and apply force impulse like:
if let sk8 = skateboard as? HasPhysics {
sk8.applyImpulse(direction, at: position, relativeTo: nil)
}
My issue comes with my "catching" logic, where I do want to use the long press, and apply downward force to my skateboard AR object like this:
#objc func longPressed(sender: UILongPressGestureRecognizer) {
if sender.state == .began || sender.state == .changed {
let location = sender.location(in:arView)
if arView.entity(at: location) is HasPhysics {
if let ray = arView.ray(through: location) {
let results = arView.scene.raycast(origin: ray.origin,
direction: ray.direction,
length: 100.0,
query: .nearest,
mask: .all,
relativeTo: nil)
if let _ = results.first,
let position = results.first?.position,
let normal = results.first?.normal {
// test different kind of forces
let direction = SIMD3<Float>(0, -20, 0)
if let sk8 = skateboard as? HasPhysics {
sk8.addForce(direction, at: position, relativeTo: nil)
}
}
}
}
}
}
Right now I know that I am ignoring the raycast results, but this is in pure development state, my issue is that when I apply positive/negative x/z the object responds well, it either slides back and forth or left or right, the positive y is also working by draging the board in the air, the only error prone force direction is the one I am striving to achieve is the downward facing negative y. The object just sits there with no effect at all.
Let also share how my object is defined inside the Reality Composer:
Ollie trick
In real life, if you shift your entire body's weight to the nose of the skateboard's deck (like doing the Ollie Maneuver), the skateboard's center of mass shifts from the middle towards the point where the force is being applied. In RealityKit, if you need to tear the rear (front) wheels of the skateboard off the floor, move the model's center of mass towards the slope.
The repositioning of the center of mass occurs in a local coordinate system.
import SwiftUI
import RealityKit
struct ContentView : View {
var body: some View {
ARViewContainer().ignoresSafeArea()
}
}
struct ARViewContainer: UIViewRepresentable {
func makeUIView(context: Context) -> ARView {
let arView = ARView(frame: .zero)
arView.debugOptions = .showPhysics // shape visualization
let scene = try! Experience.loadScene()
let name = "skateboard_01_base_stylized_lod0"
typealias ModelPack = ModelEntity & HasPhysicsBody & HasCollision
let model = scene.findEntity(named: name) as! ModelPack
model.physicsBody = .init()
model.generateCollisionShapes(recursive: true)
model.physicsBody?.massProperties.centerOfMass.position = [0, 0,-27]
arView.scene.anchors.append(scene)
return arView
}
func updateUIView(_ uiView: ARView, context: Context) { }
}
Physics shape
The second problem that you need to solve is to replace the model's box shape of the physical body (RealityKit and Reality Composer generate this type of shape by default). Its shape cannot be in the form of a monolithic box, it's quite obvious, because the box-shaped form does not allow the force to be applied appropriately. You need a shape similar to the outline of the model.
So, you can use the following code to create a custom shape:
(four spheres for wheels and box for deck)
let shapes: [ShapeResource] = [
.generateBox(size: [ 20, 4, 78])
.offsetBy(translation: [ 0.0, 11, 0.0]),
.generateSphere(radius: 3.1)
.offsetBy(translation: [ 7.5, 3, 21.4]),
.generateSphere(radius: 3.1)
.offsetBy(translation: [ 7.5, 3,-21.4]),
.generateSphere(radius: 3.1)
.offsetBy(translation: [-7.5, 3, 21.4]),
.generateSphere(radius: 3.1)
.offsetBy(translation: [-7.5, 3,-21.4])
]
// model.physicsBody = PhysicsBodyComponent(shapes: shapes, mass: 4.5)
model.collision = CollisionComponent(shapes: shapes)
P.S.
Reality Composer model's settings (I used Xcode 14.0 RC 1).
In iOS 14, hitTest(_:types:) was deprecated. It seems that you are supposed to use raycastQuery(from:allowing:alignment:) now. From the documentation:
Raycasting is the preferred method for finding positions on surfaces in the real-world environment, but the hit-testing functions remain present for compatibility. With tracked raycasting, ARKit continues to refine the results to increase the position accuracy of virtual content you place with a raycast.
However, how can I hit test SCNNodes with raycasting? I only see options to hit test a plane.
raycastQuery method documentation
Only choices for allowing: are planes
This is my current code, which uses hit-testing to detect taps on the cube node and turn it blue.
class ViewController: UIViewController {
#IBOutlet weak var sceneView: ARSCNView!
override func viewDidLoad() {
super.viewDidLoad()
/// Run the configuration
let worldTrackingConfiguration = ARWorldTrackingConfiguration()
sceneView.session.run(worldTrackingConfiguration)
/// Make the red cube
let cube = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0)
cube.materials.first?.diffuse.contents = UIColor.red
let cubeNode = SCNNode(geometry: cube)
cubeNode.position = SCNVector3(0, 0, -0.2) /// 20 cm in front of the camera
cubeNode.name = "ColorCube"
/// Add the node to the ARKit scene
sceneView.scene.rootNode.addChildNode(cubeNode)
}
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
super.touchesBegan(touches, with: event)
guard let location = touches.first?.location(in: sceneView) else { return }
let results = sceneView.hitTest(location, options: [SCNHitTestOption.searchMode : 1])
for result in results.filter( { $0.node.name == "ColorCube" }) { /// See if the beam hit the cube
let cubeNode = result.node
cubeNode.geometry?.firstMaterial?.diffuse.contents = UIColor.blue /// change to blue
}
}
}
How can I replace let results = sceneView.hitTest(location, options: [SCNHitTestOption.searchMode : 1]) with the equivalent raycastQuery code?
About Hit-Testing
Official documentation says that only ARKit's hitTest(_:types:) instance method is deprecated in iOS 14. However, in iOS 15 you can still use it. ARKit's hit-testing method is supposed to be replaced with a raycasting methods.
Deprecated hit-testing:
let results: [ARHitTestResult] = sceneView.hitTest(sceneView.center,
types: .existingPlaneUsingGeometry)
Raycasting equivalent
let raycastQuery: ARRaycastQuery? = sceneView.raycastQuery(
from: sceneView.center,
allowing: .estimatedPlane,
alignment: .any)
let results: [ARRaycastResult] = sceneView.session.raycast(raycastQuery!)
If you prefer raycasting method for hitting a node (entity), use RealityKit module instead of SceneKit:
let arView = ARView(frame: .zero)
let query: CollisionCastQueryType = .nearest
let mask: CollisionGroup = .default
let raycasts: [CollisionCastHit] = arView.scene.raycast(from: [0, 0, 0],
to: [5, 6, 7],
query: query,
mask: mask,
relativeTo: nil)
guard let raycast: CollisionCastHit = raycasts.first else { return }
print(raycast.entity.name)
P.S.
There is no need to look for a replacement for the SceneKit's hitTest(_:options:) instance method returning [SCNHitTestResult], because it works fine and it's not a time to make it deprecated.
guys
I want to visualize the detected plane in RealityKit use the code below, but the result gives that the detected plane float as the camera move (not totally float, a bit, but obviously ), so, my question is how to solve this problem ?
can any body help ?
struct ARViewContainer: UIViewRepresentable {
func makeUIView(context: Context) -> ARView {
let arView = ARView(frame: .zero)
let config = ARWorldTrackingConfiguration()
config.planeDetection = .horizontal
arView.debugOptions = [.showFeaturePoints, .showWorldOrigin]
arView.session.run(config, options:[ ])
arView.session.delegate = arView
arView.CreatePlane()
return arView
}
func updateUIView(_ uiView: ARView, context: Context) {
}
}
var planeMesh = MeshResource.generatePlane(width: 0, depth: 0)
var planeEntity = ModelEntity(mesh:planeMesh)
extension ARView : ARSessionDelegate{
func CreatePlane(){
let planeAnchor = AnchorEntity(plane:.horizontal)
//planeEntity.transform.translation = SIMD3(0,0,0)
planeAnchor.addChild(planeEntity)
self.scene.addAnchor(planeAnchor)
}
public func session(_ session: ARSession, didUpdate anchors: [ARAnchor]){
guard let planeAnchor = anchors[0] as? ARPlaneAnchor else {
return
}
DispatchQueue.main.async {
let position = planeAnchor.transform.toTranslation()
let orientation = planeAnchor.transform.toQuaternion()
let rotatedCenter = orientation.act(planeAnchor.center)
planeEntity.model?.mesh = MeshResource.generatePlane(
width: planeAnchor.extent.x,
depth: planeAnchor.extent.z
)
planeEntity.transform.translation = position + rotatedCenter
planeEntity.transform.rotation = orientation
planeEntity.model?.materials = [SimpleMaterial(color:UIColor.white.withAlphaComponent(0.5),isMetallic: false)]
}
maybe I have not make myself clear, I used the code above to visualize the detected plane in RealityKit, Yes, it works, I can see the plane, and the plane updated when the ARAnchor update, that is to say, the plane's position ,orientation, size updated when explorer goes on. but there's a problem: the rendered plane does not fixed in the space, that is to say, after I scanned the table ,the rendered plane not always fixed on the table ,it can float left、 right、below the table when I move the Camera left、right、below the table, especially in the Y axis 。
so ,my question is how this happen ? and how to solve it ?
You can try turning off planeDetection when your ARPlane already have an Anchor and is in the position that satisfies you.
ARKit will stop updating anchors so your plane anchor won't be adjusted anymore and it should stay fixed better to the surface.
You can do it by adding button to stop updates or checking if your plane already has an anchor:
planeAnchor.anchor!.isAnchored == true
In either case just change configuration for ARWorldTrackingConfiguration without planeDetection
let config = ARWorldTrackingConfiguration()
config.planeDetection = []
arView.session.run(config, options:[ ])
I place an object on a wall, then try to recognize tap on it, but hit test returns 0 objects. When I change Z position of the object and place it a little bit closer to cam, it's recognized well, but this isn't a solution, because planes are always changing and it can cover the object in any moment. How can I made hitTest work correctly and recognize my nodes behind planes? Or, maybe, I use the wrong method?
fileprivate func addNode(atPoint point: CGPoint) {
let hits = sceneView.hitTest(point, types: .existingPlaneUsingExtent)
if hits.count > 0, let firstHit = hits.first, let originNode = originNode {
let node = originNode.clone()
sceneView.scene.rootNode.addChildNode(node)
node.position = SCNVector3Make(firstHit.worldTransform.columns.3.x, firstHit.worldTransform.columns.3.y, firstHit.worldTransform.columns.3.z)
let resize = simd_float4x4(SCNMatrix4MakeScale(0.2, 0.2, 0.2))
let rotation = simd_float4x4(SCNMatrix4MakeRotation(.pi / 2, -1, 0, 0))
let transform = simd_mul(firstHit.worldTransform, resize)
let finalTransform = simd_mul(transform, rotation)
node.simdTransform = finalTransform
addedNodes.insert(node)
}
}
func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
guard let touch = touches.first else {
print("Unable to identify touches on any plane. Ignoring interaction...")
return
}
let touchPoint = touch.location(in: sceneView)
let hits = sceneView.hitTest(touchPoint, options: [SCNHitTestOption.boundingBoxOnly: true])
let filtered = hits.filter({ addedNodes.contains($0.node) })
print("\(hits.count) vs \(filtered.count), \(hits.first?.node.name ?? "no name")")
if let node = filtered.first?.node {
node.removeFromParentNode()
addedNodes.remove(node)
return
}
addPictureToPlane(atPoint: touchPoint)
}
addedNodes - set with added objects. When I added translating transform with changing Z coordinate at least on 0.05 (close to the camera) detecting working good. At least before plane changing and moving ahead the node.
I believe what you need to do is change your SCNHitTestSearchModeparameter which allows you to set:
Possible values for the searchMode option used with hit-testing
methods.
static let searchMode: SCNHitTestOption
Whereby:
The value for this key is an NSNumber object containing the raw
integer value of an SCNHitTestSearchMode constant.
From the Apple Docs there are three possible options you can use here:
case all
The hit test should return all possible results, sorted from nearest
to farthest.
case any
The hit test should return only the first object found, regardless of
distance.
case closest
The hit test should return only the closes object found.
Based on your question therefore, you would likely need to to utilise the all case.
As such your hitTest function would probably need to look something like this (remembering that self.augmentedRealityView refers to an ARSCNView):
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
//1. Get The Current Touch Location
guard let currentTouchLocation = touches.first?.location(in: self.augmentedRealityView) else { return }
//2. Perform An SCNHitTest Setting The SearchMode To 1 (All) Which Returns A List Of Results Sorted From Nearest To Farthest
if #available(iOS 11.0, *) {
let hitTestResults = self.augmentedRealityView.hitTest(currentTouchLocation, options: [SCNHitTestOption.searchMode: 1])
//3. Loop Through The Results & Get The Nodes
for index in 0..<hitTestResults.count{
let node = hitTestResults[index]
print(node)
}
}
}
I'm actually trying to put a 3D Object on QRCode with ARKit
For that I use a AVCaptureDevice to detect a QRCode and establish the area of the QRCode that gives me a CGRect.
Then, I make a hitTest on every point of the CGRect to get the average 3D coordinates like so :
positionGiven = SCNVector3(0, 0, 0)
for column in Int(qrZone.origin.x)...2*Int(qrZone.origin.x + qrZone.width) {
for row in Int(qrZone.origin.y)...2*Int(qrZone.origin.y + qrZone.height) {
for result in sceneView.hitTest(CGPoint(x: CGFloat(column)/2,y:CGFloat(row)/2), types: [.existingPlaneUsingExtent,.featurePoint]) {
positionGiven.x+=result.worldTransform.columns.3.x
positionGiven.y+=result.worldTransform.columns.3.y
positionGiven.z+=result.worldTransform.columns.3.z
cpts += 1
}
}
}
positionGiven.x=positionGiven.x/cpts
positionGiven.y=positionGiven.y/cpts
positionGiven.z=positionGiven.z/cpts
But the hitTest doesn't detect any result and freeze the camera while when I make a hitTest with a touch on screen it works.
Do you have any idea why it's not working ?
Do you have an other idea that can help me to achieve what I want to do ?
I already thought about 3D translation with CoreMotion that can give me the tilt of the device but that seems really tedious.
I also heard about ARWorldAlignmentCamera that can locked the scene coordinate to match the orientation of the camera but I don't know how to use it !
Edit : I try to move my 3D Object every time I touch the screen and the hitTest is positive, and it's pretty accurate ! I really don't understand why hitTest on an area of pixels doesn't work...
Edit 2 : Here is the code of the hitTest who works with 2-5 touches on the screen:
#objc func touch(sender : UITapGestureRecognizer) {
for result in sceneView.hitTest(CGPoint(x: sender.location(in: view).x,y: sender.location(in: view).y), types: [.existingPlaneUsingExtent,.featurePoint]) {
//Pop up message for testing
alert("\(sender.location(in: view))", message: "\(result.worldTransform.columns.3)")
//Moving the 3D Object to the new coordinates
let objectList = sceneView.scene.rootNode.childNodes
for object : SCNNode in objectList {
object.removeFromParentNode()
}
addObject(SCNVector3(result.worldTransform.columns.3.x,result.worldTransform.columns.3.y,result.worldTransform.columns.3.z))
}
}
Edit 3 :
I manage to resolve my problem partially.
I take the transform matrix of the camera (session.currentFrame.camera.transform) so that the object is in front of the camera.
Then I apply a translation on (x,y) with the position of the CGRect.
However i can't translate the z-axis because i don't have enough informations.
And I will probably need a estimation of z coordinate like the hitTest do..
Thanks in advance ! :)
You could use Apple's Vision API to detect the QR code and place an anchor.
To start detecting QR codes, use:
var qrRequests = [VNRequest]()
var detectedDataAnchor: ARAnchor?
var processing = false
func startQrCodeDetection() {
// Create a Barcode Detection Request
let request = VNDetectBarcodesRequest(completionHandler: self.requestHandler)
// Set it to recognize QR code only
request.symbologies = [.QR]
self.qrRequests = [request]
}
In ARSession's didUpdate Frame
public func session(_ session: ARSession, didUpdate frame: ARFrame) {
DispatchQueue.global(qos: .userInitiated).async {
do {
if self.processing {
return
}
self.processing = true
// Create a request handler using the captured image from the ARFrame
let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: frame.capturedImage,
options: [:])
// Process the request
try imageRequestHandler.perform(self.qrRequests)
} catch {
}
}
}
Handle the Vision QR request and trigger the hit test
func requestHandler(request: VNRequest, error: Error?) {
// Get the first result out of the results, if there are any
if let results = request.results, let result = results.first as? VNBarcodeObservation {
guard let payload = result.payloadStringValue else {return}
// Get the bounding box for the bar code and find the center
var rect = result.boundingBox
// Flip coordinates
rect = rect.applying(CGAffineTransform(scaleX: 1, y: -1))
rect = rect.applying(CGAffineTransform(translationX: 0, y: 1))
// Get center
let center = CGPoint(x: rect.midX, y: rect.midY)
DispatchQueue.main.async {
self.hitTestQrCode(center: center)
self.processing = false
}
} else {
self.processing = false
}
}
func hitTestQrCode(center: CGPoint) {
if let hitTestResults = self.latestFrame?.hitTest(center, types: [.featurePoint] ),
let hitTestResult = hitTestResults.first {
if let detectedDataAnchor = self.detectedDataAnchor,
let node = self.sceneView.node(for: detectedDataAnchor) {
let previousQrPosition = node.position
node.transform = SCNMatrix4(hitTestResult.worldTransform)
} else {
// Create an anchor. The node will be created in delegate methods
self.detectedDataAnchor = ARAnchor(transform: hitTestResult.worldTransform)
self.sceneView.session.add(anchor: self.detectedDataAnchor!)
}
}
}
Then handle adding the node when the anchor is added.
func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
// If this is our anchor, create a node
if self.detectedDataAnchor?.identifier == anchor.identifier {
let sphere = SCNSphere(radius: 1.0)
sphere.firstMaterial?.diffuse.contents = UIColor.redColor()
let sphereNode = SCNNode(geometry: sphere)
sphereNode.transform = SCNMatrix4(anchor.transform)
return sphereNode
}
return nil
}
Source