How to achieve realistic Depth of Field effect in SceneKit? - swift

I'm trying to render a frame, with realistic depth of field effect. I've already tried the depth of field properties in the camera node, but it doesn't produce usable results.
Is there a switch to max-out rendering quality of the depth of field effect? Performance is not a factor, I just need to render a frame, and user can wait for it.

Realistic Depth of Field effect in SceneKit
In SceneKit you can easily accomplish cool-looking shallow/deep depth of field (DoF). And it's not extremely intense for processing. .focusDistance and .fStop parameters are crucial for applying DoF:
cameraNode.camera?.wantsDepthOfField = true
cameraNode.camera?.focusDistance = 5
cameraNode.camera?.fStop = 0.01
cameraNode.camera?.focalLength = 24
Use the following code for testing (it's macOS version):
import SceneKit
import Cocoa
class GameViewController: NSViewController {
override func viewDidLoad() {
super.viewDidLoad()
let scene = SCNScene()
let cameraNode = SCNNode()
cameraNode.camera = SCNCamera()
cameraNode.camera?.wantsDepthOfField = true
cameraNode.camera?.focusDistance = 5
cameraNode.camera?.fStop = 0.01
cameraNode.camera?.focalLength = 24
scene.rootNode.addChildNode(cameraNode)
cameraNode.position = SCNVector3(x: 0, y: 0, z: 15)
let lightNode = SCNNode()
lightNode.light = SCNLight()
lightNode.light!.type = .omni
lightNode.position = SCNVector3(x: 0, y: 10, z: 10)
scene.rootNode.addChildNode(lightNode)
let ambientLightNode = SCNNode()
ambientLightNode.light = SCNLight()
ambientLightNode.light!.type = .ambient
ambientLightNode.light!.color = NSColor.darkGray
scene.rootNode.addChildNode(ambientLightNode)
let cylinderNode01 = SCNNode()
cylinderNode01.geometry = SCNCylinder(radius: 2, height: 10)
cylinderNode01.position = SCNVector3(0, 0, 0)
cylinderNode01.geometry?.materials.first?.diffuse.contents = NSImage(named: NSImage.Name("checker01.png"))
scene.rootNode.addChildNode(cylinderNode01)
let cylinderNode02 = SCNNode()
cylinderNode02.geometry = SCNCylinder(radius: 2, height: 10)
cylinderNode02.position = SCNVector3(5, 0, 5)
cylinderNode02.geometry?.materials.first?.diffuse.contents = NSImage(named: NSImage.Name("checker02.jpg"))
scene.rootNode.addChildNode(cylinderNode02)
let cylinderNode03 = SCNNode()
cylinderNode03.geometry = SCNCylinder(radius: 2, height: 10)
cylinderNode03.position = SCNVector3(10, 0, 10)
cylinderNode03.geometry?.materials.first?.diffuse.contents = NSImage(named: NSImage.Name("checker01.png"))
scene.rootNode.addChildNode(cylinderNode03)
let cylinderNode04 = SCNNode()
cylinderNode04.geometry = SCNCylinder(radius: 2, height: 10)
cylinderNode04.position = SCNVector3(-5, 0, -5)
cylinderNode04.geometry?.materials.first?.diffuse.contents = NSImage(named: NSImage.Name("checker02.jpg"))
scene.rootNode.addChildNode(cylinderNode04)
let cylinderNode05 = SCNNode()
cylinderNode05.geometry = SCNCylinder(radius: 2, height: 10)
cylinderNode05.position = SCNVector3(-10, 0, -10)
cylinderNode05.geometry?.materials.first?.diffuse.contents = NSImage(named: NSImage.Name("checker01.png"))
scene.rootNode.addChildNode(cylinderNode05)
let scnView = self.view as! SCNView
scnView.scene = scene
scnView.allowsCameraControl = true
scnView.backgroundColor = NSColor.black
}
}

SceneKit isn't able to do (out of the box) heavy, high quality post processing or still image rendering computation of this type. Theoretically you could probably build a setup that uses its rendering approaches to do both. But it's not a high quality renderer. If the user can wait, and you really want to focus on quality of imagery, Unreal Engine has the capacity to do this sort of thing, built in, and far higher quality post processing, effects, lights, materials, particles and rendering.

Related

Increase displacement resolution / smooth vertices in SceneKit tessellated mesh

I'm trying to generate a mesh from a depth/displacement map using SceneKit
The source depth map I'm using looks like this:
I then generate a plane with an increased segment count and heavily tessellate it and apply the displacement material. Here's the code I'm using:
import Cocoa
import SceneKit
import PlaygroundSupport
// MARK: - View setup
let scene = SCNScene()
let cameraNode = SCNNode()
cameraNode.camera = SCNCamera()
cameraNode.camera?.zFar = 1000
scene.rootNode.addChildNode(cameraNode)
cameraNode.position = SCNVector3(x: 0, y: 0, z: 15)
let lightNode = SCNNode()
lightNode.light = SCNLight()
lightNode.light!.type = .omni
lightNode.position = SCNVector3(x: 0, y: 10, z: 10)
scene.rootNode.addChildNode(lightNode)
let ambientLightNode = SCNNode()
ambientLightNode.light = SCNLight()
ambientLightNode.light!.type = .ambient
ambientLightNode.light!.color = NSColor.darkGray
scene.rootNode.addChildNode(ambientLightNode)
let scnView = SCNView(frame: NSRect(x: 0, y: 0, width: 400, height: 400))
scnView.autoenablesDefaultLighting = true
scnView.scene = scene
scnView.allowsCameraControl = true
scnView.showsStatistics = true
scnView.backgroundColor = NSColor.black
PlaygroundPage.current.liveView = scnView
// MARK: - Plane
let plane = SCNPlane(width: 200, height: 200)
plane.widthSegmentCount = 10
plane.heightSegmentCount = 10
let planeNode = SCNNode(geometry: plane)
planeNode.name = "Plane"
planeNode.geometry?.firstMaterial?.diffuse.contents = NSColor.white
let tessellator = SCNGeometryTessellator()
tessellator.tessellationFactorScale = 25
tessellator.tessellationPartitionMode = .pow2
tessellator.insideTessellationFactor = 4
tessellator.edgeTessellationFactor = 4
tessellator.smoothingMode = .phong
planeNode.geometry?.tessellator = tessellator
planeNode.geometry?.firstMaterial?.displacement.contents = "rabbit"
planeNode.geometry?.firstMaterial?.displacement.textureComponents = .red
planeNode.geometry?.firstMaterial?.displacement.intensity = 200
planeNode.geometry?.firstMaterial?.displacement.maxAnisotropy = 1
planeNode.geometry?.firstMaterial?.displacement.magnificationFilter = .none
planeNode.geometry?.firstMaterial?.lightingModel = .phong
scene.rootNode.addChildNode(planeNode)
Which mostly gives me the 3D mesh I'm aiming for:
But the resulting "blockiness" messes with the lighting. Wireframe view shows it clearer:
Is there any way to "smoothen" the resulting mesh, average vertex positions, or something similar?
I'm very tangentially familiar with 3D and basically unfamiliar with SceneKit and google/docs haven't yielded much.
Subdivision / adaptive subdivision doesn't solve the problem, neither does increasing tessellation detail to the highest count possible. It can make the shadow/highlight patches smaller, but they're still there.
Any help or pointers are much appreciated!
I figured it out!
The ladder effect is an artifact of 8-bit banding in the depth map
Converting the image to 32-bit depth and adding a slight blur pretty much fixed it:

Adding SCNLight inside SK3DNode

I am creating an SK3DNode inside an SKScene:
let ball: SK3DNode = {
let scnScene = SCNScene()
let ballGeometry = SCNSphere(radius: 200)
let ballNode = SCNNode(geometry: ballGeometry)
ballNode.position = SCNVector3(0, 0, 0)
let material = SCNMaterial()
material.diffuse.contents = UIImage(named: "wall")
ballGeometry.materials = [material]
let light = SCNLight()
light.type = .omni
light.color = UIColor.white
let lightNode = SCNNode()
lightNode.light = light
scnScene.rootNode.addChildNode(ballNode)
scnScene.rootNode.addChildNode(lightNode)
let node = SK3DNode(viewportSize: CGSize(width: 1000, height: 1000))
node.scnScene = scnScene
node.autoenablesDefaultLighting = false
return node
}()
However, the sphere renders black. Tried it with or without the material. Is there something I am missing?
The sphere is manually placed at (0, 0, 0) and so is the light (default value). This means that the light is placed inside the sphere. This means that the surface of the sphere is facing away from the light source and thus isn't lit.

3D Positional Audio – Move SCNAudioPlayer along Y and Z Axis

Using SceneKit, I can move audioNode from left to right on x axis, but I'm having problem moving on y and z axis. I'm wearing headphone, so I can hear the binaural (3d audio) effects. Also I'm running this on MacOS.
My testing code is below. Could someone let me know what I'm missing? I'd appreciate it!
import Cocoa
import SceneKit
class ViewController: NSViewController {
#IBOutlet weak var sceneView: SCNView!
override func viewDidLoad() {
super.viewDidLoad()
let path = Bundle.main.path(forResource: "Sounds/Test.mp3",
ofType: nil)
let url = URL(fileURLWithPath: path!)
let source = SCNAudioSource(url:url)!
source.loops = true
source.shouldStream = false
source.isPositional = true
source.load()
let player = SCNAudioPlayer(source: source)
let box = SCNBox(width: 100.0,
height: 100.0,
length: 100.0,
chamferRadius: 100.0)
let boxNode = SCNNode(geometry: box)
let audioNode = SCNNode()
boxNode.addChildNode(audioNode)
let scene = SCNScene()
scene.rootNode.addChildNode(boxNode)
sceneView.scene = scene
audioNode.addAudioPlayer(player)
let avm = player.audioNode as! AVAudioMixing
avm.volume = 1.0
let up = SCNAction.moveBy(x: 0, y: 100, z: 0, duration: 5)
let down = SCNAction.moveBy(x: 0, y: -100, z: 0, duration: 5)
let sequence = SCNAction.sequence([up, down])
let loop = SCNAction.repeatForever(sequence)
boxNode.runAction(loop)
// Do any additional setup after loading the view.
}
}
Updated.
You're casting the player.audioNode to AVAudioMixing protocol:
let avm = player.audioNode as! AVAudioMixing
But instead of it, you have to cast it to a class. A code looks like this:
let avm = player.audioNode as? AVAudioEnvironmentNode
Any node that conforms to the AVAudioMixing protocol (for example, AVAudioPlayerNode) can act as a source in this environment. The environment has an implicit listener. By controlling the listener’s position and orientation, the application controls the way the user experiences the virtual world. This node also defines properties for distance attenuation and reverberation that help characterize the environment.
And take into account !
Only inputs with a mono channel connection format to the environment node are spatialized. If the input is stereo, the audio is passed through without being spatialized. Inputs with connection formats of more than two channels aren't supported.
And, of course, you need to implement AVAudio3DMixing protocol.
Here's a working code:
import SceneKit
import AVFoundation
class ViewController: NSViewController, AVAudio3DMixing {
// YOU NEED MONO AUDIO !!!
var renderingAlgorithm = AVAudio3DMixingRenderingAlgorithm.sphericalHead
var rate: Float = 0.0
var reverbBlend: Float = 0.5
var obstruction: Float = -100.0
var occlusion: Float = -100.0
var position: AVAudio3DPoint = AVAudio3DPoint(x: 0, y: 0, z: -100)
override func viewDidLoad() {
super.viewDidLoad()
let scene = SCNScene()
let cameraNode = SCNNode()
cameraNode.camera = SCNCamera()
cameraNode.camera?.zFar = 200
cameraNode.position = SCNVector3(x: 0, y: 0, z: 40)
scene.rootNode.addChildNode(cameraNode)
let sceneView = self.view as! SCNView
sceneView.scene = scene
sceneView.backgroundColor = NSColor.black
sceneView.autoenablesDefaultLighting = true
let path = Bundle.main.path(forResource: "Test_Mono", ofType: "mp3")
let url = URL(fileURLWithPath: path!)
let source = SCNAudioSource(url: url)!
source.loops = true
source.shouldStream = false // MUST BE FALSE
source.isPositional = true
source.load()
let player = SCNAudioPlayer(source: source)
let audioNode = SCNNode()
let box = SCNBox(width: 1, height: 1, length: 1, chamferRadius: 0.2)
let boxNode = SCNNode(geometry: box)
boxNode.addChildNode(audioNode)
scene.rootNode.addChildNode(boxNode)
audioNode.addAudioPlayer(player)
let avm = player.audioNode as? AVAudioEnvironmentNode
avm?.reverbBlend = reverbBlend
avm?.renderingAlgorithm = renderingAlgorithm
avm?.occlusion = occlusion
avm?.obstruction = obstruction
let up = SCNAction.moveBy(x: 0, y: 0, z: 70, duration: 5)
let down = SCNAction.moveBy(x: 0, y: 0, z: -70 , duration: 5)
let sequence = SCNAction.sequence([up, down])
let loop = SCNAction.repeatForever(sequence)
boxNode.runAction(loop)
avm?.position = AVAudio3DPoint(
x: Float(boxNode.position.x),
y: Float(boxNode.position.y),
z: Float(boxNode.position.z))
}
}
After researching and experimenting for a hwile, I finally figured it out. There were two things that I needed to fix.
I had to change the default renderingAlgorithm for SCNAudioPlayer.AVAudioNode from equalPowerPanning to either HRTF or HRTFHQ. However, AVAudioNode does not have renderingAlgorithm property. However, I was able to cast SCNAudioPlayer.AVAudioNode as AVAudioPlayerNode, and AVAudioPlayerNode does have renderingAlgorithm property. Here's the relevant code.
if let apn = player.audioNode as? AVAudioPlayerNode {
apn.renderingAlgorithm = .HRTFHQ
}
I had to assign a node with SCNCamera to pointOfView for SCNView. Also I had to change the position of the camera node further away from the audioNode. Otherwise, I heard the drastic movement in the beginning. Here's the relevant code.
let cameraNode = SCNNode(geometry: SCNBox(width:1, height:1, length:1, chamferRadius: 0.1))
cameraNode.camera = SCNCamera()
cameraNode.position = SCNVector3(x: 0, y: 0, z: -10)
sceneView.pointOfView = cameraNode
My scene.rootNode is a box geometry with 100x100x100 dimension. Inside scene.rootNode, I have a boxNode with 50x5050 dimension. Then inside the boxNode, I have audioNode generating sound with 1x1x1 dimension as well as cameraNode with 1x1x1 dimension. AudioNode's start position is 0,0,0, and the position for the cameraNode is 0,0,-20.
Finally here's the entire working code.
import Cocoa
import AVFoundation
import SceneKit
class ViewController: NSViewController {
#IBOutlet weak var sceneView: SCNView!
override func viewDidLoad() {
super.viewDidLoad()
let path = Bundle.main.path(forResource: "Sounds/Test_mono.mp3", ofType: nil)
let url = URL(fileURLWithPath: path!)
let source = SCNAudioSource(url: url)!
source.loops = true
source.shouldStream = false
source.isPositional = true
source.load()
let player = SCNAudioPlayer(source: source)
if let apn = player.audioNode as? AVAudioPlayerNode {
apn.renderingAlgorithm = .HRTFHQ
}
let audioNode = SCNNode(geometry: SCNBox(width:1, height:1, length:1, chamferRadius: 0.1))
let cameraNode = SCNNode(geometry: SCNBox(width:1, height:1, length:1, chamferRadius: 0.1))
cameraNode.camera = SCNCamera()
let boxNode = SCNNode(geometry: SCNBox(width:50, height:50, length:50, chamferRadius: 1))
boxNode.addChildNode(audioNode)
audioNode.position = SCNVector3(x: 0, y: 0, z: 0)
boxNode.addChildNode(cameraNode)
cameraNode.position = SCNVector3(x: 0, y: 0, z: -10)
let scene = SCNScene()
scene.rootNode.geometry = SCNBox(width:100, height:100, length:100, chamferRadius: 0.1)
scene.rootNode.addChildNode(boxNode)
boxNode.position = SCNVector3(x: 0, y: 0, z: 0)
sceneView.scene = scene
sceneView.pointOfView = cameraNode
sceneView.audioListener = cameraNode
audioNode.addAudioPlayer(player)
let move = SCNAction.moveBy(x:1, y:0, z:0, duration: 1)
let sequence = SCNAction.sequence([move])
let loop = SCNAction.repeatForever(sequence)
audioNode.runAction(loop)
// Do any additional setup after loading the view.
}
override var representedObject: Any? {
didSet {
// Update the view, if already loaded.
}
}
}

How does SCNIKConstraint work?

I have the following code (this can be run by replacing the standard ViewController code in the Game base project for macOS):
let scene = SCNScene()
let cameraNode = SCNNode()
cameraNode.camera = SCNCamera()
scene.rootNode.addChildNode(cameraNode)
cameraNode.position = SCNVector3(x: 0, y: 0, z: 15)
let lightNode = SCNNode()
lightNode.light = SCNLight()
lightNode.light!.type = .omni
lightNode.position = SCNVector3(x: 0, y: 10, z: 10)
scene.rootNode.addChildNode(lightNode)
let ambientLightNode = SCNNode()
ambientLightNode.light = SCNLight()
ambientLightNode.light!.type = .ambient
ambientLightNode.light!.color = NSColor.darkGray
scene.rootNode.addChildNode(ambientLightNode)
/* RELEVANT CODE BEGINS */
let boxGeo = SCNBox(width: 1, height: 1, length: 1, chamferRadius: 0)
let boxMaterial = SCNMaterial()
boxMaterial.diffuse.contents = NSColor.gray
boxGeo.firstMaterial = boxMaterial
let boxNode = SCNNode(geometry: boxGeo)
scene.rootNode.addChildNode(boxNode)
boxNode.name = "box0"
let sphereGeo = SCNSphere(radius: 0.5)
let sphereMaterial = SCNMaterial()
sphereMaterial.diffuse.contents = NSColor.green
sphereGeo.firstMaterial = sphereMaterial
let sphereNode = SCNNode(geometry: sphereGeo)
boxNode.addChildNode(sphereNode)
sphereNode.name = "sphere0"
sphereNode.constraints = [SCNConstraint]()
let distance = SCNDistanceConstraint(target: boxNode)
distance.minimumDistance = 2.0
distance.maximumDistance = 5.0
sphereNode.constraints?.append(distance)
let ik = SCNIKConstraint.inverseKinematicsConstraint(chainRootNode: boxNode)
sphereNode.constraints?.append(ik)
let anim = CABasicAnimation(keyPath: "targetPosition.y")
anim.fromValue = -2.0
anim.toValue = 2.0
anim.duration = 1
anim.autoreverses = true
anim.repeatCount = .infinity
ik.addAnimation(anim, forKey: nil)
/* RELEVANT CODE ENDS */
let scnView = self.view as! SCNView
scnView.scene = scene
scnView.allowsCameraControl = true
scnView.showsStatistics = true
scnView.backgroundColor = NSColor.black
From what I can gather from the documentation, the animation (and yes, the scene kit view animation setting is set to both play and loop in IB) should move the sphere as close as possible to the points 2.0 and -2.0 on the y-axis by rotating the cube. However, the sphere simply stays stationary. I have also tried setting the initial position of the sphere and cube by manipulating their position vectors directly instead of via the distance constraint, but again the animation did nothing.
Additionally, I have attempted to use the distance constraint in combination with the box having a lookAt constraint to make it rotate to constantly look at the sphere - these caused the rendering of the box and sphere to completely freak out.
I feel as though maybe I am missing something in the documentation here, such as another constraint or some kind of transform matrix to setup some kind of initial value. But I have encountered some other issues with constraints, animations and skeletons that is making me begin to believe that there is either a bug or some undocumented aspects of SceneKit.
You have added the sphereNode as child of the boxNode. If you move the boxNode all childs are also moved and the constraint has no effect.

Using SceneKit for hitTesting not returning a hit with SCNNode

The documentation in XCode clearly states that hitTesting a geometry in SceneKit can be done with SCNRender, SCNView or the SCNNode themselves when one plans to test a 3D line segment. I have a use for SCNScene with its nodes without a renderer or a view, therefore I am planning to use SCNNode hitTesting. I create a SCNScene, put a SCNNode in it and test a simple ray that goes through, but I always get an empty hitList and I don't understand why:
import Swift
import SceneKit
let boxGeometry = SCNBox(width: 1.0, height: 1.0, length: 1.0, chamferRadius: 0)
let boxNode = SCNNode(geometry: boxGeometry)
var scene = SCNScene()
scene.rootNode.addChildNode(boxNode)
let from = SCNVector3(x: 0, y: -2, z: 0)
let to = SCNVector3(x: 0, y: 2 , z: 0)
var hits = scene.rootNode.hitTestWithSegmentFromPoint(from, toPoint: to, options:nil) // this is always empty
if hits != nil {
if hits!.count > 0 {
var hit = (hits!.first as! SCNHitTestResult).node as SCNNode
}
}
I have tried passing various forms of options but nothing changes.
SCNHitTestFirstFoundOnlyKey: yes or no does not change anything
SCNHitTestSortResultsKey: yes or no does not change anything
SCNHitTestClipToZRangeKey: invalid for SCNNode
SCNHitTestBackFaceCullingKey: yes or no does not change anything
SCNHitTestBoundingBoxOnlyKey: yes or no does not change anything
SCNHitTestRootNodeKey: rootNOde of scene or boxNode does not change
anything
SCNHitTestIgnoreHiddenNodesKey: yes or no does not change anything
What am I doing wrong?
I have found the answer, which is either a bug or a feature: using SCNScene and its nodes SCNNode for 3D hitTesting, in particular the method: "hitTestWithSegmentFromPoint(toPoint:options:)" does not return a hit unless the scene is included in an SCNView. It appears it cannot be used offscreen. My guess is yours for why this is the case, although I can imagine it has something to do with performing some of these quite expensive calculations on the graphics card.
I have tested this using the GameView SCNScene starter project. The critical line is self.gameView!.scene = scene
override func awakeFromNib(){
let scene = SCNScene()
let boxGeometry = SCNBox(width: 1.0, height: 1.0, length: 1.0, chamferRadius: 0.0)
let boxNode = SCNNode(geometry: boxGeometry)
boxNode.position=SCNVector3(x: 0, y: 0, z: 0)
scene.rootNode.addChildNode(boxNode)
// create and add a camera to the scene
let cameraNode = SCNNode()
cameraNode.camera = SCNCamera()
scene.rootNode.addChildNode(cameraNode)
// place the camera
cameraNode.position = SCNVector3(x: 0, y: 0, z: 15)
// create and add a light to the scene
let lightNode = SCNNode()
lightNode.light = SCNLight()
lightNode.light!.type = SCNLightTypeOmni
lightNode.position = SCNVector3(x: 0, y: 10, z: 10)
scene.rootNode.addChildNode(lightNode)
// create and add an ambient light to the scene
let ambientLightNode = SCNNode()
ambientLightNode.light = SCNLight()
ambientLightNode.light!.type = SCNLightTypeAmbient
ambientLightNode.light!.color = NSColor.darkGrayColor()
scene.rootNode.addChildNode(ambientLightNode)
// set the scene to the view
// uncomment this to fail
self.gameView!.scene = scene
// allows the user to manipulate the camera
self.gameView!.allowsCameraControl = true
// show statistics such as fps and timing information
self.gameView!.showsStatistics = true
// configure the view
self.gameView!.backgroundColor = NSColor.blackColor()
let hitList = scene.rootNode.hitTestWithSegmentFromPoint(SCNVector3(x:-10,y:0,z:0), toPoint: SCNVector3(x:10,y:0,z:0), options:[SCNHitTestBackFaceCullingKey:false, SCNHitTestSortResultsKey:true, SCNHitTestIgnoreHiddenNodesKey:false])
if hitList?.count > 0 {
println("Hit found: \n\n\( hitList![0] )") // assign self.gameView!.scene = scene to reach this point.
} else {
println("No hit") // uncomment self.gameView!.scene = scene to reach this point.
}
}
I've also had trouble with hitTestWithSegmentFromPoint.
I was calling it in viewDidLoad() and it returned a 0 elements array, though I was sure there was a hit.
Calling it in viewDidAppear() (or later) solved my problem.