I am using ARKits image detection to place a 3D object when a certain image is detected. Everything works fine except for the 3D model that is being created. It is shaking like crazy. I double checked and the reference image has the right measures.
I call addModel() when the image is detected. Here is how my code looks like.
Finding reference Image:
func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
let node = SCNNode()
if let imageAnchor = anchor as? ARImageAnchor{
let plane = SCNPlane(width: imageAnchor.referenceImage.physicalSize.width, height: imageAnchor.referenceImage.physicalSize.height)
let planeNode = SCNNode(geometry: plane)
addModel(addTo: planeNode)
node.addChildNode(planeNode)
}
return node
}
The addModel() functions looks like this:
func addModel(addTo: SCNNode){
let testScene = SCNScene(named: "art.scnassets/testModel.scn")
let testNode = testScene?.rootNode.childNode(withName: "test", recursively: true)
let testMaterial = SCNMaterial()
testMaterial.diffuse.contents = UIImage(named: "art.scnassets/bricks")
testNode?.geometry?.materials = [testMaterial]
testNode!.position = SCNVector3Zero
testNode!.position.x = -0.3
testNode!.position.z = 0.3
addTo.addChildNode(testNode!)
}
Did your try to delete:
testNode!.position.x = -0.3
testNode!.position.z = 0.3 ?
Because when you type testNode!.position = SCNVector3Zero it is says, that
x = 0.0, y = 0.0, z = 0.0 and after that you type another coordinates.
Related
I am trying to follow the steps to create this following this article (image below from article):
It is basically recognising a face to put something on the face (a tattoo) and placing a background image behind it.
I am using an iPhone X device to test the code, but every time I test if .personSegmentation is supported, it is false:
if ARFaceTrackingConfiguration.supportsFrameSemantics(.personSegmentation) {
configuration.frameSemantics.insert(.personSegmentation) // code never executed.
}
My whole code for adding the plane to put on top of the face plus the background is:
The ARSCNViewDelegate delegate to add the nodes:
private var virtualBackgroundNode = SCNNode()
func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
guard let device = sceneView.device else {
return nil
}
let faceGeometry = ARSCNFaceGeometry(device: device)
let faceNode = SCNNode(geometry: faceGeometry)
faceNode.geometry?.firstMaterial?.transparency = 0
let tattooPlane = SCNPlane(width: 0.13, height: 0.06)
tattooPlane.firstMaterial?.diffuse.contents = UIImage(named: "Tattoos/tattoo0")!
tattooPlane.firstMaterial?.isDoubleSided = true
let tattooNode = SCNNode()
tattooNode.position.z = faceNode.boundingBox.max.z * 3 / 4
tattooNode.position.y = 0.027
tattooNode.geometry = tattooPlane
faceNode.addChildNode(tattooNode)
configureBackgroundView()
sceneView.scene.rootNode.addChildNode(virtualBackgroundNode)
return faceNode
}
Resizing the background image, setting it and positioning the background node:
func configureBackgroundView() {
let (skScene, mediaAspectRatio) = makeImageBackgroundScene(image: UIImage(named: "Cats/cat0")!)
let size = skScene.size
virtualBackgroundNode.geometry = SCNGeometry.Plane(width: size.width, height: size.height)
let material = SCNMaterial()
material.diffuse.contents = skScene
virtualBackgroundNode.geometry?.materials = [material]
virtualBackgroundNode.scale = SCNVector3(1.7 * mediaAspectRatio, 1.7, 1)
let cameraPosition = sceneView.pointOfView?.scale
let position = SCNVector3(cameraPosition!.x, cameraPosition!.y, cameraPosition!.z - 1000)
virtualBackgroundNode.position = position
}
This method creates a SpriteKit image by resizing the original asset:
func makeImageBackgroundScene(image: UIImage) -> (scene: SKScene, mediaAspectRatio: Double) {
//Adjusted so that the aspect ratio of the image is not distorted
let width = image.size.width
let height = image.size.height
let mediaAspectRatio = Double(width / height)
let cgImage = image.cgImage!
let newImage = UIImage(cgImage: cgImage, scale: 1.0, orientation: .up)
let skScene = SKScene(size: CGSize(width: 1000 * mediaAspectRatio, height: 1000))
let texture = SKTexture(image: newImage)
let skNode = SKSpriteNode(texture:texture)
skNode.position = CGPoint(x: skScene.size.width / 2.0, y: skScene.size.height / 2.0)
skNode.size = skScene.size
skNode.yScale = -1.0
skScene.addChild(skNode)
return (skScene, mediaAspectRatio)
}
Any advice on what to try? Snapchat and TikTok have similar Face recognition + background setups and they work in my device.
Thanks for any help.
I currently have my app detecting a postcard picture. Once detected I place a 3D model of a dog on top of it. Problem is, the model is currently having the back of the dog sit sitting the card rather than it’s feet.
Is there a way to programmatically rotate the model so that the feet is standing on the postcard?
class ViewController: UIViewController, ARSCNViewDelegate {
#IBOutlet var sceneView: ARSCNView!
override func viewDidLoad() {
super.viewDidLoad()
sceneView.delegate = self
}
//MARK-: WHERE I CONFIGURE MY APP TO DETECT AN IMAGE
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
let configuration = ARImageTrackingConfiguration()
guard let trackedImages = ARReferenceImage.referenceImages(inGroupNamed: "Photos", bundle: Bundle.main) else {
print("No images available")
return
}
configuration.trackingImages = trackedImages
configuration.maximumNumberOfTrackedImages = 7
sceneView.session.run(configuration)
}
//MARK-: WHERE I PLACE A PLANE/DOG MODEL (named: shipScene) OVER THE DETECTED IMAGE
func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
let node = SCNNode()
if let imageAnchor = anchor as? ARImageAnchor {
let plane = SCNPlane(width: imageAnchor.referenceImage.physicalSize.width, height: imageAnchor.referenceImage.physicalSize.height)
plane.firstMaterial?.diffuse.contents = UIColor(white: 1, alpha: 0.8)
let planeNode = SCNNode(geometry: plane)
planeNode.eulerAngles.x = -.pi / 2
guard let url = Bundle.main.url(forResource: "shipScene", withExtension: "usdz") else { fatalError() }
let mdlAsset = MDLAsset(url: url)
let dogScene = SCNScene(mdlAsset: mdlAsset)
let dogNode = shipScene.rootNode.childNodes.first!
shipNode.position = SCNVector3Zero
shipNode.position.z = 0.15
planeNode.addChildNode(shipNode)
node.addChildNode(planeNode)
}
return node
}
}
I figure I can use a rendition of
shipNode.position.x
To move the model, but any attempts has just moved the model around (but not rotating it).
At first you have to retrieve a model node from scene. Take into account that node's name is not the same as scene's name! You can find your node in Scene graph hierarchy.
let model = scene.rootNode.childNode(withName: "myModel", recursively: true)!
Then you can rotate it using rotation property:
model.rotation.y = -CGFloat.pi / 2
or using eulerAngles property:
model.eulerAngles = SCNVector3(0, -CGFloat.pi/2, 0)
or using orientation property:
model.orientation = SCNVector4(0, 1, 0, -CGFloat.pi/2)
Pay attention where a pivot point of your node is. A rotation should be around the pivot point.
Swift version 5.7
The accepted answer is already solving the issue.
Also, for those who need some spice, here is how you can rotate+scale together with animation.
let node = scene.rootNode.childNode(withName: "Cube_001", recursively: true)!
node.runAction(SCNAction.rotateBy(x: 1, y: 1, z: 3, duration: 3.1))
node.runAction(SCNAction.scale(by: 2, duration: 3.1))
I created two nodes: a sphere and a box:
var sphere = SCNNode(geometry: SCNSphere(radius: 0.005))
//I get the box node from scn file
let boxScene = SCNScene(named: "art.scnassets/world.scn")!
var boxNode: SCNNode?
I want two nodes or physicsBody's to interact, so I created a category for categoryBitMask and contactTestBitMask:
struct CollisionCategory: OptionSet {
let rawValue: Int
static let box = CollisionCategory(rawValue: 1)
static let sphere = CollisionCategory(rawValue: 2)
}
Here I set the box node as a physics body:
self.boxScene.rootNode.enumerateChildNodes { (node, _) in
if node.name == "box" {
boxNode = node
let boxBodyShape = SCNPhysicsShape(geometry: SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0.1), options: nil)
let physicsBody = SCNPhysicsBody(type: .static, shape: boxBodyShape)
boxNode!.physicsBody = physicsBody
boxNode!.physicsBody?.categoryBitMask = CollisionCategory.box.rawValue
boxNode!.physicsBody?.contactTestBitMask = CollisionCategory.sphere.rawValue
boxNode!.physicsBody?.collisionBitMask = boxNode!.physicsBody!.contactTestBitMask
}
}
Here I set the sphere node in the render function, which you can move around the view:
func setUpSphere() {
let sphereBodySphere = SCNPhysicsShape(geometry: SCNSphere(radius: 0.005))
let physicsBody = SCNPhysicsBody(type: .kinematic, shape: sphereBodySphere)
sphere.physicsBody = physicsBody
sphere.physicsBody?.categoryBitMask = CollisionCategory.sphere.rawValue
sphere.physicsBody?.contactTestBitMask = CollisionCategory.box.rawValue
sphere.geometry?.firstMaterial?.diffuse.contents = UIColor.blue
sphere.physicsBody?.collisionBitMask = sphere.physicsBody!.contactTestBitMask
previousPoint = currentPosition
}
///It Adds a sphere and changes his position
func renderer(_ renderer: SCNSceneRenderer, willRenderScene scene: SCNScene, atTime time: TimeInterval) {
guard let pointOfView = sceneView.pointOfView else { return }
let mat = pointOfView.transform
let dir = SCNVector3(-1 * mat.m31, -1 * mat.m32, -1 * mat.m33)
let currentPosition = pointOfView.position + (dir * 0.185)
if buttonPressed {
if let previousPoint = previousPoint {
sphere.position = currentPosition
sceneView.scene.rootNode.addChildNode(sphere)
}
}
}
I added the protocol SCNPhysicsContactDelegate to the ViewController,
and I set in ViewDidLoad():
override func viewDidLoad() {
super.viewDidLoad()
sceneView.delegate = self
sceneView.scene.physicsWorld.contactDelegate = self
///I correctly see the shapes of the sphere and the box physics bodies using
sceneView.debugOptions = .showPhysicsShapes
createBox()
setUpSphere()
sceneView.scene = boxScene
sceneView.scene.physicsWorld.contactDelegate = self
}
Then I added that function:
func physicsWorld(_ world: SCNPhysicsWorld, didEnd contact: SCNPhysicsContact) {
print("Collision!")
}
This is what happens.
When the two nodes collide nothing happens, so I can't know if the two bodies are touching. Could the problem be about .kinematic, .static or about the function render()?
I followed step by step different tutorials about collisions in ARKit: Tutorial 1, Tutorial 2.
I have no idea why it doesn't work like expected.
Is there something wrong in my code?
Download file code link: https://ufile.io/20sla
willRenderScene is called uptown 60 times a second for every time the scene is going to be rendered. Since you're recreating the physics body every time it's probably messing up the physics engine determine collisions.
Try changing your code to only create the physics body once during setup.
Now I am able to show different SCNPlane, when card detected. After displaying SCNPlanes, the user touches any plane to show new SCNPlane. But right now touch is working properly but new SCNPlane is not showing.
Here is the code I've tried:
var cake_1_PlaneNode : SCNNode? = nil
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
guard let imageAnchor = anchor as? ARImageAnchor else { return }
if let imageName = imageAnchor.referenceImage.name {
print(imageName)
if imageName == "menu" {
// Check To See The Detected Size Of Our menu Card (Should By 5cm*3cm)
let menuCardWidth = imageAnchor.referenceImage.physicalSize.width
let menuCardHeight = imageAnchor.referenceImage.physicalSize.height
print(
"""
We Have Detected menu Card With Name \(imageName)
\(imageName)'s Width Is \(menuCardWidth)
\(imageName)'s Height Is \(menuCardHeight)
""")
//raspberry
//cake 1
let cake_1_Plane = SCNPlane(width: 0.045, height: 0.045)
cake_1_Plane.firstMaterial?.diffuse.contents = UIImage(named: "france")
cake_1_Plane.cornerRadius = 0.01
let cake_1_PlaneNode = SCNNode(geometry: cake_1_Plane)
self.cake_1_PlaneNode = cake_1_PlaneNode
cake_1_PlaneNode.eulerAngles.x = -.pi/2
cake_1_PlaneNode.runAction(SCNAction.moveBy(x: 0.15, y: 0, z: -0.125, duration: 0.75))
node.addChildNode(cake_1_PlaneNode)
self.sceneView.scene.rootNode.addChildNode(node)
}
}
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
let touch = touches.first as! UITouch
if(touch.view == self.sceneView){
//print("touch working")
let viewTouchLocation:CGPoint = touch.location(in: sceneView)
guard let result = sceneView.hitTest(viewTouchLocation, options: nil).first else {
return
}
if let planeNode = cake_1_PlaneNode, cake_1_PlaneNode == result.node{
print("match")
cake_1()
}
}
}
func cake_1() {
let plane = SCNPlane(width: 0.15 , height: 0.15)
plane.firstMaterial?.diffuse.contents = UIColor.black.withAlphaComponent(0.75)
let planeNodee = SCNNode(geometry: plane)
planeNodee.eulerAngles.x = -.pi / 2
planeNodee.runAction(SCNAction.moveBy(x: 0.21, y: 0, z: 0, duration: 0))
} //cake_1
Follow this link: Detect touch on SCNNode in ARKit.
Looking at your code I can see several issues (not to mention the naming conventions for your variables and methods).
Firstly, you are creating a Global Variable which you have declared like so:
var cake_1_PlaneNode : SCNNode? = nil
However you use both a Local and Global Variable for your cake_1_PlaneNode in yourDelegate Callback:
let cake_1_PlaneNode = SCNNode(geometry: cake_1_Plane)
self.cake_1_PlaneNode = cake_1_PlaneNode
Which should simply read like so:
self.cake_1_PlaneNode = SCNNode(geometry: cake_1_Plane)
Secondly, you are adding your cake_1_PlaneNode to the rootNode of your ARSCNView rather than your detected ARImageAnchor which is probably what you don't want to do, since when an ARAnchor is detected:
You can provide visual content for the anchor by attaching geometry
(or other SceneKit features) to this node or by adding child nodes.
As such, this method (unless you actually want to do it like this) is unnecessary.
The remaining issues lie within your cake_1 function itself.
Firstly you are not actually adding your planeNodee to your sceneHierachy.
Since you haven't specified whether or not the newly initialised planeNode should be added directly to your ARSCNView or as a childNode of your cake_1_planeNode your function should include one of the following:
self.sceneView.scene.rootNode.addChildNode(planeNodee)
self.cake_1_planeNode.addChildNode(planeNodee)
In addition there is probably also no need to rotate your planeNodee since by default an SCNPlane is rendered vertically.
Since you haven't stipulated where you will be placing your content, it could be that using -.pi / 2 is unnecessary, since this could make it virtually invisible to the naked eye.
One other issue which could also account for you not seeing your node, when you actually add it, is the Z position.
If you set 2 nodes at the same position you will likely experience an issue know as Z-fighting (which you can read more about here). As such you should probably move your added node forward slightly when adding it e.g. SCNVector3 (0,0,0.001)to account for this.
Based on all of these points, I have provided a fully working and commented example below:
import UIKit
import ARKit
//-------------------------
//MARK: - ARSCNViewDelegate
//-------------------------
extension ViewController: ARSCNViewDelegate {
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
//1. Check We Have An ARImageAnchor, Then Get It's Reference Image & Name
guard let imageAnchor = anchor as? ARImageAnchor else { return }
let detectedTarget = imageAnchor.referenceImage
guard let detectedTargetName = detectedTarget.name else { return }
//2. If We Have Detected Our Virtual Menu Then Add The CakeOnePlane
if detectedTargetName == "cakeMenu" {
let cakeOnePlaneGeometry = SCNPlane(width: 0.045, height: 0.045)
cakeOnePlaneGeometry.firstMaterial?.diffuse.contents = UIColor.cyan
cakeOnePlaneGeometry.cornerRadius = 0.01
let cakeOnPlaneNode = SCNNode(geometry: cakeOnePlaneGeometry)
cakeOnPlaneNode.eulerAngles.x = -.pi/2
//3. To Allow Us To Easily Keep Track Our Our Currently Added Node We Will Assign It A Unique Name
cakeOnPlaneNode.name = "Strawberry Cake"
node.addChildNode(cakeOnPlaneNode)
cakeOnPlaneNode.runAction(SCNAction.moveBy(x: 0.15, y: 0, z: 0, duration: 0.75))
}
}
}
class ViewController: UIViewController {
#IBOutlet var augmentedRealityView: ARSCNView!
let augmentedRealitySession = ARSession()
let configuration = ARWorldTrackingConfiguration()
//------------------
//MARK: - Life Cycle
//------------------
override func viewDidLoad() {
super.viewDidLoad()
setupARSession()
}
//-----------------
//MARK: - ARSession
//-----------------
/// Runs The ARSession
func setupARSession(){
//1. Load Our Detection Images
guard let detectionImages = ARReferenceImage.referenceImages(inGroupNamed: "AR Resources", bundle: nil) else { return }
//2. Configure & Run Our ARSession
augmentedRealityView.session = augmentedRealitySession
augmentedRealityView.delegate = self
configuration.detectionImages = detectionImages
augmentedRealitySession.run(configuration, options: [.resetTracking, .removeExistingAnchors])
}
//--------------------
//MARK: - Interaction
//--------------------
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
//1. Get The Current Touch Location & Perform An ARSCNHitTest To Check For Any Hit SCNNode's
guard let currentTouchLocation = touches.first?.location(in: self.augmentedRealityView),
let hitTestNode = self.augmentedRealityView.hitTest(currentTouchLocation, options: nil).first?.node else { return }
//2. If We Have Hit Our Strawberry Cake Then We Call Our makeCakeOnNode Function
if let cakeID = hitTestNode.name, cakeID == "Strawberry Cake"{
makeCakeOnNode(hitTestNode)
}
}
/// Adds An SCNPlane To A Detected Cake Target
///
/// - Parameter node: SCNNode
func makeCakeOnNode(_ node: SCNNode){
let planeGeometry = SCNPlane(width: 0.15 , height: 0.15)
planeGeometry.firstMaterial?.diffuse.contents = UIColor.black.withAlphaComponent(0.75)
let planeNode = SCNNode(geometry: planeGeometry)
planeNode.position = SCNVector3(0, 0, 0.001)
planeNode.runAction(SCNAction.moveBy(x: 0.21, y: 0, z: 0, duration: 0))
node.addChildNode(planeNode)
}
}
Which yields the following on my device:
For your information, this seems to show that your calculations for placing your content are off (unless of course this is the desired result).
As you can see, all of content rendered correctly, however the spacing of these was quite large, and as such you will likely need to pan your device somewhat to see it all when testing and developing further.
Hope it helps...
***Please use descriptive and clear names for your variables and functions, it is very hard to read and understand your code. You can read more about swift styling guidelines here: https://github.com/raywenderlich/swift-style-guide#naming
You are creating a new plane when the user touches the screen, but you are not adding that plane to the scene, therefore your "cake_1()" function only creates a new plane.
When ARKit detects an image, it automatically creates an empty node and adds it to our scene, at the center of the detected image. We must first keep a reference to the node ARKit has added for us when the image is detected.
Add this variable to the top of your class:
var detectedImageNode: SCNNode?
Then in func renderer(renderer: didAdd node:, for anchor:) add the following line:
detectedImageNode = node
Now that we have a reference to the node, we can easily add and remove other nodes.
Add the following line at the end of cake_1():
if let detectedImageNode = detectedImageNode {
cake_1_PlaneNode?.removeFromParentNode()
detectedImageNode.addChildNode(planeNodee)
}
Your final code should look like this:
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
guard let imageAnchor = anchor as? ARImageAnchor else { return }
if let imageName = imageAnchor.referenceImage.name {
print(imageName)
if imageName == "menu" {
let cake_1_Plane = SCNPlane(width: 0.045, height: 0.045)
cake_1_Plane.firstMaterial?.diffuse.contents = UIImage(named: "france")
cake_1_Plane.cornerRadius = 0.01
let cake_1_PlaneNode = SCNNode(geometry: cake_1_Plane)
self.cake_1_PlaneNode = cake_1_PlaneNode
cake_1_PlaneNode.eulerAngles.x = -.pi/2
cake_1_PlaneNode.runAction(SCNAction.moveBy(x: 0.15, y: 0, z: -0.125, duration: 0.75))
node.addChildNode(cake_1_PlaneNode)
// No need to add the following line. The node is already added to the scene
//self.sceneView.scene.rootNode.addChildNode(node)
detectedImageNode = node
}
}
}
func cake_1() {
let plane = SCNPlane(width: 0.15 , height: 0.15)
plane.firstMaterial?.diffuse.contents = UIColor.black.withAlphaComponent(0.75)
let planeNodee = SCNNode(geometry: plane)
planeNodee.eulerAngles.x = -.pi / 2
if let detectedImageNode = detectedImageNode {
cake_1_PlaneNode?.removeFromParentNode()
detectedImageNode.addChildNode(planeNodee)
}
}
Alternative solution
If you are just trying to change the image of the plane then an easier way to approach this is to just change the texture of the plane.
Replace the contents of cake_1() with:
if let planeGeometry = cake_1_PlaneNode?.geometry {
planeGeometry.firstMaterial?.diffuse.contents = UIImage(named: "newImage")
}
I'm using ARKit 1.5 (beta) for image detection. Once I detect my image I would like to then place a AR scene image using the plane detected. How can this be done?
My code so far which detects the image (which is in my assets folder):
/// - Tag: ARImageAnchor-Visualizing
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
guard let imageAnchor = anchor as? ARImageAnchor else { return }
let referenceImage = imageAnchor.referenceImage
updateQueue.async {
// Create a plane to visualize the initial position of the detected image.
let plane = SCNPlane(width: referenceImage.physicalSize.width,
height: referenceImage.physicalSize.height)
let planeNode = SCNNode(geometry: plane)
planeNode.opacity = 0.25
/*
`SCNPlane` is vertically oriented in its local coordinate space, but
`ARImageAnchor` assumes the image is horizontal in its local space, so
rotate the plane to match.
*/
planeNode.eulerAngles.x = -.pi / 2
/*
Image anchors are not tracked after initial detection, so create an
animation that limits the duration for which the plane visualization appears.
*/
planeNode.runAction(self.imageHighlightAction)
// Add the plane visualization to the scene.
node.addChildNode(planeNode)
}
DispatchQueue.main.async {
let imageName = referenceImage.name ?? ""
self.statusViewController.cancelAllScheduledMessages()
self.statusViewController.showMessage("Detected image “\(imageName)”")
}
}
var imageHighlightAction: SCNAction {
return .sequence([
.wait(duration: 0.25),
.fadeOpacity(to: 0.85, duration: 1.50),
.fadeOpacity(to: 0.15, duration: 1.50),
.fadeOpacity(to: 0.85, duration: 1.50),
.fadeOut(duration: 0.75),
.removeFromParentNode()
])
Assuming that referenceImage.name's value is the actual image filename.
if let imageName = referenceImage.name {
plane.materials = [SCNMaterial()]
plane.materials[0].diffuse.contents = UIImage(named: imageName)
}