I'm facing an issue where SCNView.hitTest does not detect hits against geometry that I'm modifying dynamically on the cpu.
Here's the overview: I have a node that uses an SCNGeometry created from a MTLBuffer of vertices:
func createGeometry(vertexBuffer: MTLBuffer, vertexCount: Int) -> SCNGeometry {
let geometry = SCNGeometry(sources: [
SCNGeometrySource.init(
buffer: vertexBuffer,
vertexFormat: .float3,
semantic: .vertex,
vertexCount: vertexCount,
dataOffset: 0,
dataStride: MemoryLayout<SIMD3<Float>>.stride),
], elements: [
SCNGeometryElement(indices: ..., primitiveType: .triangles)
])
}
let vertexBuffer: MTLBuffer = // shared buffer
let vertexCount = ...
let node = SCNNode(geometry: createGeometry(vertexBuffer: vertexBuffer, vertexCount: vertexCount))
As the app is running, I then dynamically modify the vertex buffer in the SceneKit update loop:
// In SceneKit update function
var ptr = vertexBuffer.contents().bindMemory(to: SIMD3<Float>.self, capacity: vertexCount)
for i in 0..<vertexCount {
ptr[i] = // modify vertex
}
This dynamic geometry is correctly rendered by SceneKit. However when I then try hit testing against node using SCNView.hitTest, no hits are detected against the modified geometry.
I can work around this by re-creating the node's geometry after modifying the data:
// after updating data
node.geometry = createGeometry(vertexBuffer: vertexBuffer, vertexCount: vertexCount)
However this feels like a hack.
What is the proper way to have hit testing work reliably for a node with dynamically changing SCNGeometry?
I think there's no proper way to make hit-testing work reliably in your situation. It would apparently be possible if it didn't depend on the SceneKit/Metal render loop and delegate pattern. But since it entirely depends on them, this is an unrealistically expensive operation to recreate SCNGeometry's instances, as you said earlier. So, I totally agree with #HamidYusifli.
When you perform a hit-test search, SceneKit looks for SCNGeometry objects along the ray you specify. For each intersection between the ray and a geometry, SceneKit creates a hit-test result to provide information about both the SCNNode object containing the geometry and the location of the intersection on the geometry’s surface.
The problem in your case is that when you modify the buffer’s contents (MTLBuffer) at render time, SceneKit does not know about it, and therefore cannot update SCNGeometry object which is used for performing hit-test.
So the only way I can see to solve this issue is to recreate your SCNGeometry object.
Related
EDIT: I have solved the problem and will post the solution in the next couple of days.
I'm building 3D achievements similar to Apple's Activity app.
I've already loaded my 3D model (a scene with a single node), can show it, and can tap on it to apply a rotational force:
#objc func objectTapped(_ gesture: UITapGestureRecognizer) {
let tapLocation = gesture.location(in: scnView)
let hitResults = scnView.hitTest(tapLocation, options: [:])
if let tappedNode = (hitResults.first { $0.node === badgeNode })?.node {
let pos = Float(tapLocation.x) - tappedNode.boundingBox.max.x
let tappedVector = SCNVector4(x: 0, y: pos, z: 0, w: 0.1)
tappedNode.physicsBody?.applyTorque(tappedVector,
asImpulse: true)
}
}
This works fine. Now to the tricky part:
I want the node to rotate until it either shows its front or backside (like in the Activity app), where it then should stop. It should stop naturally, which means it can overshoot a bit and then return.
To describe it with pictures - here I am holding the node in this position...
...and if I let go of the node, it will rotate to show the front side, which includes a little bit of overshooting. This is the ending position:
Since I'm quite new to SceneKit, I have troubles figuring out how to achieve this effect. It seems like I can achieve that by using SceneKit objects like gravity fields, without having to calculate a whole lot of stuff by myself, or at least that's what I'm hoping for.
I don't necessarily ask for a full solution, I basically just need a point in the right direction. Thanks in advance!
According to this article by apple Ray-Casting and Hit-Testing. I should use ray casting provided by RealityKit to detect the surfaces instead of hit testing provided by ARKit as apple says
but the hit-testing functions remain present for compatibility
. However,I can't find a way to know the extent of the surface detected by the raycast query.
So according to this code:
func startRayCasting() {
guard let raycastQuery = arView.makeRaycastQuery(from: arView.center,
allowing: .estimatedPlane,
alignment: .vertical) else {
return
}
guard let result = arView.session.raycast(raycastQuery).first else {
return
}
let transformation = Transform(matrix: result.worldTransform)
let plane = Plane(color: .green, transformation: transformation)
plane.transform = transformation
let raycastAnchor = AnchorEntity(raycastResult: result)
raycastAnchor.addChild(plane)
arView.scene.addAnchor(raycastAnchor)
}
I would expect that the plane I am creating would get the size and position of the plane detected. However this does not happen.
So my question is, Is ray casting suitable for detecting the surfaces size and location. Or its just for checking the 2d point location on a surface.
An Apple Documentation says here:
Raycast instance method performs a convex ray cast against all the geometry in the scene for a ray of a given origin, direction, and length.
and here:
Raycast instance method performs a convex ray cast against all the geometry in the scene for a ray between two end points.
In both cases raycast methods are used for detecting intersections. And in both cases these methods return an array of collision cast hit results.
That's all raycast was made for.
This may be an obscure question, but I see lots of very cool samples online of how people are using the new ARKit people occlusion technology in ARKit 3 to effectively "separate" the people from the background, and apply some sort of filtering to the "people" (see here).
In looking at Apple's provided source code and documentation, I see that I can retrieve the segmentationBuffer from an ARFrame, which I've done, like so;
func session(_ session: ARSession, didUpdate frame: ARFrame) {
let image = frame.capturedImage
if let segementationBuffer = frame.segmentationBuffer {
// Get the segmentation's width
let segmentedWidth = CVPixelBufferGetWidth(segementationBuffer)
// Create the mask from that pixel buffer.
let sementationMaskImage = CIImage(cvPixelBuffer: segementationBuffer, options: [:])
// Smooth edges to create an alpha matte, then upscale it to the RGB resolution.
let alphaUpscaleFactor = Float(CVPixelBufferGetWidth(image)) / Float(segmentedWidth)
let alphaMatte = sementationMaskImage.clampedToExtent()
.applyingFilter("CIGaussianBlur", parameters: ["inputRadius": 2.0)
.cropped(to: sementationMaskImage.extent)
.applyingFilter("CIBicubicScaleTransform", parameters: ["inputScale": alphaUpscaleFactor])
// Unknown...
}
}
In the "unknown" section, I am trying to determine how I would render my new "blurred" person on top of the original camera feed. There does not seem to be any methods to draw the new CIImage on "top" of the original camera feed, as the ARView has no way of being manually updated.
In the following code snippet we see personSegmentationWithDepth type property for depth compositing (there are RGB, Alpha and Depth channels):
// Automatically segmenting and then compositing foreground (people),
// middle-ground (3D model) and background.
let session = ARSession()
if let configuration = session.configuration as? ARWorldTrackingConfiguration {
configuration.frameSemantics.insert(.personSegmentationWithDepth)
session.run(configuration)
}
You can manually access a Depth Data of World Tracking in CVPixelBuffer (depth values for a performed segmentation):
let image = frame.estimatedDepthData
And you can manually access a Depth Data of Face Tracking in CVPixelBuffer (from TrueDepth camera):
let image = session.currentFrame?.capturedDepthData?.depthDataMap
Also, there's a generateDilatedDepth instance method in ARKit 3.0:
func generateDilatedDepth(from frame: ARFrame,
commandBuffer: MTLCommandBuffer) -> MTLTexture
In your case you have to use estimatedDepthData because Apple documentation says:
It's a buffer that represents the estimated depth values from the camera feed that you use to occlude virtual content.
var estimatedDepthData: CVPixelBuffer? { get }
If you multiply DEPTH data from this buffer (at first you have to convert Depth channel to RGB) by RGB or ALPHA using compositing techniques and you'll get awesome effects.
Look at these 6 images: the lower row represents three RGB-images corrected with Depth channel: depth grading, depth blurring, depth point position pass.
the Bringing People into AR WWDC session has some information, especially about ARMatteGenerator. The session also comes with a sample code.
I have the following node setup on my scene:
A container node with child nodes: earth, torus, and moon. I applied to the earth node's filter property the following custom HighlightFilter with CIBloom and CISourceOverCompositing filter for the glow effect:
CIFilter Class (code from awesome blog: Highlighting SCNNode with Glow)
class HighlightFilter: CIFilter {
static let filterName = "highlightFilter"
#objc dynamic var inputImage: CIImage?
#objc dynamic var inputIntensity: NSNumber?
#objc dynamic var inputRadius: NSNumber?
override var outputImage: CIImage? {
guard let inputImage = inputImage else {
return nil
}
let bloomFilter = CIFilter(name:"CIBloom")!
bloomFilter.setValue(inputImage, forKey: kCIInputImageKey)
bloomFilter.setValue(inputIntensity, forKey: "inputIntensity")
bloomFilter.setValue(inputRadius, forKey: "inputRadius")
let sourceOverCompositing = CIFilter(name:"CISourceOverCompositing")!
sourceOverCompositing.setValue(inputImage, forKey: "inputImage")
sourceOverCompositing.setValue(bloomFilter.outputImage, forKey: "inputBackgroundImage")
return sourceOverCompositing.outputImage
}}
I don't understand this invisible rectangle. I assume it is because of the CIFilter SourceOverCompositing which overlays the modified image over the original. But why is the torus hidden and the moon not?
I added the torus material to the moons'. material property, to see if something is wrong with the material. But still the same, moon is visible, torus is hidden. The moon rotates around with a helperNode as a child to the containerNode.
This may happen due to two potential problems.
SOLUTION 1.
When you're using CISourceOverCompositing you need a premultiplied RGBA image (RGB * A) for foreground, where an alpha channel has the same shape as the Earth (left picture). But you're having an alpha channel covering all the image (right picture).
If you wanna know what's the shape of Alpha channel in your Earth image – use one of these compositing applications: The Foundry Nuke, Adobe After Effect, Apple Motion, Blackmagic Fusion, etc.
Also, if you want to composite the Moon and the Earth separately, you have to have them as two different images.
In compositing classical OVER operation has the following formula:
(RGB_image1 * A_image1) + (RGB_image2 * (1 – A_image1))
The first part of this formula is a premultiplied foreground image (the Earth) – RGB1 * A1.
The second part of this formula is a background image with a hole – RGB2 * inverted_A1. You've got the inversion of the alpha channel using (1-A). The background image itself could have only three components – RGB (without A).
Then you add two images together using simple addition operation. If you have several OVER operations – the order of these ops is crucial.
SOLUTION 2.
It could be due to a Depth Buffer. Disable writesToDepthBuffer instance property. It is a Boolean value that determines whether SceneKit produces depth information when rendering the material.
yourTorusNode.geometry?.materials.first?.writesToDepthBuffer = false
Hope this helps.
I don't know exactly why this strange behaviour occured, but deleting the following line made it disappear.
self.torus.firstMaterial?.writesToDepthBuffer = false
The writesToDepthBuffer property on the material on the torus was set to false.
I am adding a 3D model containing animations to the scene that I previously download from the internet. Before adding this node I use prepare function on it because I wan't to avoid frame drop. But still I get a very short frame drop to about 47 fps. This is caused by executing this prepare function. I also tried using prepare(_:, shouldAbortBlock:) on other dispatch queue, but this still didn't help. Can someone help me resolve this or tell me why there is this happening?
arView.sceneView.prepare([mediaNode]) { [mediaNode, weak self] (success) in
guard let `self` = self else { return }
guard
let currentMediaNode = self.mediaNode as? SCNNode,
currentMediaNode === mediaNode,
!self.mainNode.childNodes.contains(mediaNode)
else { return }
self.mainNode.addChildNode(mediaNode)
}
By the way this is a list of files I'm using to load this model:
https://www.dropbox.com/s/7968fe5wfdcxbyu/Serah-iOS.dae?dl=1
https://www.dropbox.com/s/zqb6b6rxynnvc5e/0001.png?dl=1
https://www.dropbox.com/s/hy9y8qyazkcnvef/0002.tga?dl=1
https://www.dropbox.com/s/fll9jbjud7zjlsq/0004.tga?dl=1
https://www.dropbox.com/s/4niq12mezlvi5oz/0005.png?dl=1
https://www.dropbox.com/s/wikqgd46643327i/0007.png?dl=1
https://www.dropbox.com/s/fioj9bqt90vq70c/0008.tga?dl=1
https://www.dropbox.com/s/4a5jtmccyx413j7/0010.png?dl=1
DAE file is already compiled by Xcode tools so that it can be loaded after being downloaded from the internet. And this is the code I use to load it after it's downloaded:
class func loadModel(fromURL url: URL) -> SCNNode? {
let options = [SCNSceneSource.LoadingOption.animationImportPolicy : SCNSceneSource.AnimationImportPolicy.playRepeatedly]
let sceneSource = SCNSceneSource(url: url, options: options)
let node = sceneSource?.entryWithIdentifier("MDL_Obj", withClass: SCNNode.self)
return node
}
I was experiencing the same issue. My nodes were all taking advantage of physically-based rendering (PBR) and the first time I added a node to the scene, the frame rate dropped significantly, but was fine after that. I could add as many other nodes without a frame rate drop.
I figured out a work around to this issue. What I do is after I create my ARConfiguration and before I call session.run(configuration) I add a test node with PBR to the scene. In order for that node to not appear, I set the node's material's colorBufferWriteMask to an empty array (see this answer: ARKit hide objects behind walls) Then before I add my content I remove that node. Adding and removing this test node does the trick for me.
Here is an example:
var pbrTestNode: SCNNode!
func addPBRTestNode() {
let testGeometrie = SCNBox(width: 0.5, height: 0.5, length: 0.5, chamferRadius: 0)
testGeometrie.materials.first?.diffuse.contents = UIColor.blue
testGeometrie.materials.first?.colorBufferWriteMask = []
testGeometrie.materials.first?.lightingModel = .physicallyBased
pbrTestNode = SCNNode(geometry: testGeometrie)
scene.rootNode.addChildNode(pbrTestNode)
}
func removePBRTestNode() {
pbrTestNode.removeFromParentNode()
}
func startSessionWithPlaneDetection() {
// Create a session configuration
let configuration = ARWorldTrackingConfiguration()
if #available(iOS 11.3, *) {
configuration.planeDetection = [.horizontal, .vertical]
} else {
configuration.planeDetection = .horizontal
}
configuration.isLightEstimationEnabled = true
// this prevents the delay when adding any nodes with PBR later
sceneController.addPBRTestNode()
// Run the view's session
sceneView.session.run(configuration)
}
Call removePBRTestNode() when you add your content to the scene.
Firstly
Get 3D model for AR app with no more than 10K polygons and a texture of 1K x 1K. The best result can be accomplished with 5K...7K polygons per each model. Totally, SceneKit's scene may contain not more than 100K polygons. This recommendation helps you considerably improve rendering performance and, I suppose, you'll have a minimal drop frame.
Secondly
The simplest way to get rid of drop frame in ARKit/SceneKit/AVKit is to use Metal framework. Just imagine: a simple image filter can be more than a hundred times faster to perform on the GPU than an equivalent CPU-based filter. The same things I could say about realtime AV-video and 3D animation – they perform much better on GPU.
For instance, you can read this useful post about using Metal rendering for AVCaptureSession. There's awesome workflow how to use Metal.
P.S. Check your animated object/scene in 3D authoring tool (if it's OK) before writing a code.