How to combine SCNRenderer with an existing MTLCommandBuffer? - swift

Question

I successfully integrated the Vuforia SDK Image Target Tracking feature into an iOS project by combining the OpenGL context (EAGLContext) that the SDK provides, with an instance of SceneKit's SCNRenderer. That allowed me to leverage the simplicity of the SceneKit's 3D API and at the same time benefiting from Vuforia's high precision image detection. Now, I'd like to do the same by replacing OpenGL with Metal.
Some background story
I was able to draw SceneKit objects on top of the live video texture drawn by Vuforia using OpenGL without major problems.
Here's the simplified setup I used with OpenGL:
func configureRenderer(for context: EAGLContext) {
self.renderer = SCNRenderer(context: context, options: nil)
self.scene = SCNScene()
renderer.scene = scene
// other scenekit setup
}
func render() {
// manipulate scenekit nodes
renderer.render(atTime: CFAbsoluteTimeGetCurrent())
}
Apple deprecates OpenGL on iOS 12
Since Apple announced that it is deprecating OpenGL on iOS 12, I figured it would be a good idea to try to migrate this project to use the Metal instead of OpenGL.
That should be simple in theory as Vuforia supports Metal out of the box. However, when trying to integrate it, I hit the wall.
The question
The view seems to ever only render results of the SceneKit renderer, or the textures encoded by Vuforia, but never both at the same time. It depends what is encoded first. What do I have to do to blend both results togeter?
Here's the problematic setup in a nutshell:
func configureRenderer(for device: MTLDevice) {
let renderer = SCNRenderer(device: device, options: nil)
self.scene = SCNScene()
renderer.scene = scene
// other scenekit setup
}
func render(viewport: CGRect, commandBuffer: MTLCommandBuffer, drawable: CAMetalDrawable) {
// manipulate scenekit nodes
let renderPassDescriptor = MTLRenderPassDescriptor()
renderPassDescriptor.colorAttachments[0].texture = drawable.texture
renderPassDescriptor.colorAttachments[0].loadAction = .load
renderPassDescriptor.colorAttachments[0].storeAction = .store
renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColor(red: 0.0, green: 0, blue: 0, alpha: 0)
renderer!.render(withViewport: viewport, commandBuffer: commandBuffer, passDescriptor: renderPassDescriptor)
}
I tried calling render either after encoder.endEncoding or before the commandBuffer.renderCommandEncoderWithDescriptor :
metalDevice = MTLCreateSystemDefaultDevice();
metalCommandQueue = [metalDevice newCommandQueue];
id<MTLCommandBuffer>commandBuffer = [metalCommandQueue commandBuffer];
//// -----> call the `render(viewport:commandBuffer:drawable) here <------- \\\\
id<MTLRenderCommandEncoder> encoder = [commandBuffer renderCommandEncoderWithDescriptor:renderPassDescriptor];
// calls to encoder to render textures from Vuforia
[encoder endEncoding];
//// -----> or here <------- \\\\
[commandBuffer presentDrawable:drawable];
[commandBuffer commit];
In either case, I only see results of SCNRenderer OR results of the encoder, but never both in the same view.
It seems to me as if the encoding pass above, and the SCNRenderer.render, are overwriting each other's buffers.
What am I missing here?

Answer 1

I think I've found an answer.
I am rendering scnrenderer after endEncoding, but I'm creating a new descriptor.
// Pass Metal context data to Vuforia Engine (we may have changed the encoder since
// calling Vuforia::Renderer::begin)
finishRender(UnsafeMutableRawPointer(Unmanaged.passRetained(drawable!.texture).toOpaque()), UnsafeMutableRawPointer(Unmanaged.passRetained(encoder!).toOpaque()))
// ========== Finish Metal rendering ==========
encoder?.endEncoding()
// Commit the rendering commands
// Command completed handler
commandBuffer?.addCompletedHandler { _ in self.mCommandExecutingSemaphore.signal()}
let screenSize = UIScreen.main.bounds.size
let newDescriptor = MTLRenderPassDescriptor()
// Draw to the drawable's texture
newDescriptor.colorAttachments[0].texture = drawable?.texture
// Store the data in the texture when rendering is complete
newDescriptor.colorAttachments[0].storeAction = MTLStoreAction.store
// Use textureDepth for depth operations.
newDescriptor.depthAttachment.texture = mDepthTexture;
renderer?.render(atTime: 0, viewport: CGRect(x: 0, y: 0, width: screenSize.width, height: screenSize.height), commandBuffer: commandBuffer!, passDescriptor: newDescriptor)
// Present the drawable when the command buffer has been executed (Metal
// calls to CoreAnimation to tell it to put the texture on the display when
// the rendering is complete)
commandBuffer?.present(drawable!)
// Commit the command buffer for execution as soon as possible
commandBuffer?.commit()