I need to get the points of lips and chin in ARKit but ARSCNFaceGeometry provides the data of complete mesh of face. Can you let me know how can I get points of only lips and chin from the all the vertices available in faceAnchor.geometry.vertices.
Basically need to write the points on the mesh in the ARKit so that I can find out which point is linked to which part of the face.
Any help is appreciated.
Thank You
I was able to draw the numbers using this code
if(drawSphere){
for index in 0...faceAnchor.geometry.vertices.count{
let text = SCNText(string: String(index), extrusionDepth: 4)
let material = SCNMaterial()
material.diffuse.contents = UIColor.green
material.specular.contents = UIColor.green
text.materials = [material]
let newNode = SCNNode()
newNode.position = SCNVector3Make(faceAnchor.geometry.vertices[index].x*1.0,faceAnchor.geometry.vertices[index].y*1.0,faceAnchor.geometry.vertices[index].z)
newNode.scale = SCNVector3(x:0.00025, y:0.00025, z:0.0001)
newNode.geometry = text
node.addChildNode(newNode)
}
}
If the numbers are not clear you can multiply x and y of position with some constant to get bigger view.
Related
I'm using ARKit and trying to apply a texture to a face anchor, following an Apple example. However, the texture has an extremely bright light applied to it.
How can I reduce the intensity or color of the light created by physicallyBased lightingModel?
func createFaceNode(_ renderer: SCNSceneRenderer) {
guard let sceneView = renderer as? ARSCNView,
let geometry = ARSCNFaceGeometry(device: sceneView.device!),
let material = geometry.firstMaterial
else { return }
material.diffuse.contents = #imageLiteral(resourceName: "texture")
material.normal.contents = #imageLiteral(resourceName: "normal")
material.lightingModel = .physicallyBased
material.ambientOcclusion.contents = UIColor.darkGray
}
To reduce an intensity of light's diffusion for physically based shader is as easy as this (but consider that intensity's range is normalised from 0 to 1):
node.geometry?.materials.first?.diffuse.intensity = 0.1
Or surface's reaction to light is normal:
node.geometry?.materials.first?.diffuse.intensity = 1.0
I am trying to attach an object in front of the camera, but the issue is that it is always in relation to the initial camera direction. How can I adjust/get the SCNVector3 position to place the object in front, even if the direction of the camera is up or down?
This is how I do it now:
let ballShape = SCNSphere(radius: 0.03)
let ballNode = SCNNode(geometry: ballShape)
let viewPosition = sceneView.pointOfView!.position
ballNode.position = SCNVector3Make(viewPosition.x, viewPosition.y, viewPosition.z - 0.4)
sceneView.scene.rootNode.addChildNode(ballNode)
Edited to better answer the question now that it's clarified in a comment
New Answer:
You are using only the position of the camera, so if the camera is rotated, it doesn't affect the ball.
What you can do is get the transform matrix of the ball and multiply it by the transform matrix of the camera, that way the ball position will be relative to the full transformation of the camera, including rotation.
e.g.
let ballShape = SCNSphere(radius: 0.03)
let ballNode = SCNNode(geometry: ballShape)
ballNode.position = SCNVector3Make(0.0, 0.0, -0.4)
let ballMatrix = ballNode.transform
let cameraMatrix = sceneView.pointOfView!.transform
let newBallMatrix = SCNMatrix4Mult(ballMatrix, cameraMatrix)
ballNode.transform = newBallMatrix
sceneView.scene.rootNode.addChildNode(ballNode)
Or if you only want the SCNVector3 position, to answer exactly to your question (this way the ball will not rotate):
...
let newBallMatrix = SCNMatrix4Mult(ballMatrix, cameraMatrix)
let newBallPosition = SCNVector3Make(newBallMatrix.m41, newBallMatrix.m42, newBallMatrix.m43)
ballNode.position = newBallPosition
sceneView.scene.rootNode.addChildNode(ballNode)
Old Answer:
You are using only the position of the camera, so when the camera rotates, it doesn't affect the ball.
SceneKit uses a hierarchy of nodes, so when a node is "child" of another node, it follows the position, rotation and scale of its "parent". The proper way of attaching an object to another object, in this case the camera, is to make it "child" of the camera.
Then, when you set the position, rotation or any other aspect of the transform of the "child" node, you are setting it relative to its parent. So if you set the position to SCNVector3Make(0.0, 0.0, -0.4), it's translated -0.4 units in Z on top of its "parent" translation.
So to make what you want, it should be:
let ballShape = SCNSphere(radius: 0.03)
let ballNode = SCNNode(geometry: ballShape)
ballNode.position = SCNVector3Make(0.0, 0.0, -0.4)
let cameraNode = sceneView.pointOfView
cameraNode?.addChildNode(ballNode)
This way, when the camera rotates, the ball follows exactly its rotation, but separated -0.4 units from the camera.
I have the following code that creates a SCNBox and shoots it on the screen. This works but as soon as I turn the phone in any other direction then the force impulse does not get updated and it always shoots the box in the same old position.
Here is the code:
#objc func tapped(recognizer :UIGestureRecognizer) {
guard let currentFrame = self.sceneView.session.currentFrame else {
return
}
/
let box = SCNBox(width: 0.2, height: 0.2, length: 0.2, chamferRadius: 0)
let material = SCNMaterial()
material.diffuse.contents = UIColor.red
material.lightingModel = .constant
var translation = matrix_identity_float4x4
translation.columns.3.z = -0.01
let node = SCNNode()
node.geometry = box
node.geometry?.materials = [material]
print(currentFrame.camera.transform)
node.physicsBody = SCNPhysicsBody(type: .dynamic, shape: nil)
node.simdTransform = matrix_multiply(currentFrame.camera.transform, translation)
node.physicsBody?.applyForce(SCNVector3(0,2,-10), asImpulse: true)
self.sceneView.scene.rootNode.addChildNode(node)
}
Line 26 is where I apply the force but it does not take into account the user's current phone orientation. How can I fix that?
On line 26 you're passing a constant vector to applyForce. That method takes a vector in world space, so passing a constant vector means you're always applying a force in the same direction — if you want a direction that's based on the direction the camera or something else is pointing, you'll need to calculate a vector based on that direction.
The (new) SCNNode property worldFront might prove helpful here — it gives you the direction a node is pointing, automatically converted to world space, so it's useful with physics methods. (Though you might want to scale it.)
I use ARKit 1.5 and this func to highlight vertical surfaces, but it doesn't work really well.
func createPlaneNode(planeAnchor: ARPlaneAnchor) -> SCNNode {
let scenePlaneGeometry = ARSCNPlaneGeometry(device: metalDevice!)
scenePlaneGeometry?.update(from: planeAnchor.geometry)
let planeNode = SCNNode(geometry: scenePlaneGeometry)
planeNode.name = "\(currPlaneId)"
planeNode.opacity = 0.25
if planeAnchor.alignment == .vertical {
planeNode.geometry?.firstMaterial?.diffuse.contents = UIColor.red
}
currPlaneId += 1
return planeNode
}
It always finds some FeaturePoints on vertical objects but very rare it actually highlights the surface using the planeNode that I created.
I want to be able to detect and highlight things like a pillar or even a man. How would you approach this?
Image of object with featurePoints
Image with the result in best case scenario
In ARKit 1.5 and ARKit 2.0 there's .planeDetection instance property allowing you to enable .horizontal, .vertical, or both simultaneously .horizontal and .vertical detections.
var planeDetection: ARWorldTrackingConfiguration.PlaneDetection { get set }
ViewController's code:
let configuration = ARWorldTrackingConfiguration()
configuration.planeDetection = .vertical
//configuration.planeDetection = [.vertical, .horizontal]
sceneView.session.run(configuration)
If you want to successfully detect and track vertical objects in your environment, you need good lighting conditions and rich non-repetitive texture. Look at the picture below:
I have a 32x32 .png image that I want to repeat over a SCNPlane. The code I've got (See below) results in the image being stretched to fit the size of the plane, rather than repeated.
CODE:
let planeGeo = SCNPlane(width: 15, height: 15)
let imageMaterial = SCNMaterial()
imageMaterial.diffuse.contents = UIImage(named: "art.scnassets/grid.png")
planeGeo.firstMaterial = imageMaterial
let plane = SCNNode(geometry: planeGeo)
plane.geometry?.firstMaterial?.diffuse.wrapS = SCNWrapMode.repeat
plane.geometry?.firstMaterial?.diffuse.wrapT = SCNWrapMode.repeat
I fixed it. It seems like the image was zoomed in. If I do imageMaterial.diffuse.contentsTransform = SCNMatrix4MakeScale(32, 32, 0), the image repeats.
I faced an identical issue when implementing plane visualisation in ARKit. I wanted to visualise the detected plane as a checkerboard pattern. I fixed it by creating a custom SCNNode called a "PlaneNode" with a correctly configured SCNMaterial. The material uses wrapS, wrapT = .repeat and calculates the scale correctly based on the size of the plane itself.
Looks like this:
Have a look at the code below, the inline comments contain the explanation.
class PlaneNode : SCNNode {
init(planeAnchor: ARPlaneAnchor) {
super.init()
// Create the 3D plane geometry with the dimensions reported
// by ARKit in the ARPlaneAnchor instance
let planeGeometry = SCNPlane(width:CGFloat(planeAnchor.extent.x), height:CGFloat(planeAnchor.extent.z))
// Instead of just visualizing the grid as a gray plane, we will render
// it in some Tron style colours.
let material = SCNMaterial()
material.diffuse.contents = PaintCode.imageOfViewARPlane
//the scale gives the number of times the image is repeated
//ARKit givest the width and height in meters, in this case we want to repeat
//the pattern each 2cm = 0.02m so we divide the width/height to find the number of patterns
//we then round this so that we always have a clean repeat and not a truncated one
let scaleX = (Float(planeGeometry.width) / 0.02).rounded()
let scaleY = (Float(planeGeometry.height) / 0.02).rounded()
//we then apply the scaling
material.diffuse.contentsTransform = SCNMatrix4MakeScale(scaleX, scaleY, 0)
//set repeat mode in both direction otherwise the patern is stretched!
material.diffuse.wrapS = .repeat
material.diffuse.wrapT = .repeat
//apply material
planeGeometry.materials = [material];
//make a node for it
self.geometry = planeGeometry
// Move the plane to the position reported by ARKit
position.x = planeAnchor.center.x
position.y = 0
position.z = planeAnchor.center.z
// Planes in SceneKit are vertical by default so we need to rotate
// 90 degrees to match planes in ARKit
transform = SCNMatrix4MakeRotation(-Float.pi / 2.0, 1.0, 0.0, 0.0);
}
required init?(coder aDecoder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
func update(planeAnchor: ARPlaneAnchor) {
guard let planeGeometry = geometry as? SCNPlane else {
fatalError("update(planeAnchor: ARPlaneAnchor) called on node that has no SCNPlane geometry")
}
//update the size
planeGeometry.width = CGFloat(planeAnchor.extent.x)
planeGeometry.height = CGFloat(planeAnchor.extent.z)
//and material properties
let scaleX = (Float(planeGeometry.width) / 0.02).rounded()
let scaleY = (Float(planeGeometry.height) / 0.02).rounded()
planeGeometry.firstMaterial?.diffuse.contentsTransform = SCNMatrix4MakeScale(scaleX, scaleY, 0)
// Move the plane to the position reported by ARKit
position.x = planeAnchor.center.x
position.y = 0
position.z = planeAnchor.center.z
}
}
To do this in the SceneKit editor, select your plane (add one if needed) in the scene and then select the "Material Inspector" tab on the top right. Then, under "Properties" and where it says "Diffuse", select your texture. Now, expand the diffuse section by clicking the carat to the left of "Diffuse" and go down to where it says "Scale". Here, you can increase the scaling so that the texture can look repeated rather than stretched. For this question, the OP would have to set the scaling to 32x32.
You can learn it from Scene kit viewer Suppose You have SCNplane in your scene kit
Create scene file drag a plane
Which size is 12 inches in meter it is 0.3048
and select image in diffuse
now You have image with 4 Grid as shown in image
we want each box to be show in each inches so for 12 Inches we need 12 box * 12 box as we have 12 inches box
to calculate it. First we need convert 0.3048 meter to inches
which is meters / 0.0254 answer is 12.
but we need each grid to show in each inch so we also need to divide 12 / 4 = 3
now goto show material inspector and change scale value to 3
you can see 12 boxes for 12 inch plane.
Hope it is helpful