ARKit image detection, add image - swift

I'm using ARKit 1.5 (beta) for image detection. Once I detect my image I would like to then place a AR scene image using the plane detected. How can this be done?
My code so far which detects the image (which is in my assets folder):
/// - Tag: ARImageAnchor-Visualizing
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
guard let imageAnchor = anchor as? ARImageAnchor else { return }
let referenceImage = imageAnchor.referenceImage
updateQueue.async {
// Create a plane to visualize the initial position of the detected image.
let plane = SCNPlane(width: referenceImage.physicalSize.width,
height: referenceImage.physicalSize.height)
let planeNode = SCNNode(geometry: plane)
planeNode.opacity = 0.25
/*
`SCNPlane` is vertically oriented in its local coordinate space, but
`ARImageAnchor` assumes the image is horizontal in its local space, so
rotate the plane to match.
*/
planeNode.eulerAngles.x = -.pi / 2
/*
Image anchors are not tracked after initial detection, so create an
animation that limits the duration for which the plane visualization appears.
*/
planeNode.runAction(self.imageHighlightAction)
// Add the plane visualization to the scene.
node.addChildNode(planeNode)
}
DispatchQueue.main.async {
let imageName = referenceImage.name ?? ""
self.statusViewController.cancelAllScheduledMessages()
self.statusViewController.showMessage("Detected image “\(imageName)”")
}
}
var imageHighlightAction: SCNAction {
return .sequence([
.wait(duration: 0.25),
.fadeOpacity(to: 0.85, duration: 1.50),
.fadeOpacity(to: 0.15, duration: 1.50),
.fadeOpacity(to: 0.85, duration: 1.50),
.fadeOut(duration: 0.75),
.removeFromParentNode()
])

Assuming that referenceImage.name's value is the actual image filename.
if let imageName = referenceImage.name {
plane.materials = [SCNMaterial()]
plane.materials[0].diffuse.contents = UIImage(named: imageName)
}

Related

How can I replay the paused (removed) video

I am creating a AR Application to track 2 images and play videos for that images. The problem I got is I am not able to replay the first video because it get paused (and removed).
I tried to create a storage for removed videos, and then used that stored data to replay that video. But it doesnt work as desired.
The video which is removed first start playing at every image after scanning the 1st image again.
The Code:
func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
let node = SCNNode()
if let imageAnchor = anchor as? ARImageAnchor {
let videos = ["harrypotter": "harrypotter.mp4", "deatheater": "deatheater.mp4"]
if let videoName = videos[imageAnchor.referenceImage.name!] {
if let currentVideoNode = currentVideoNode {
currentVideoNode.pause()
currentVideoNode.removeFromParent()
}
let videoNode = SKVideoNode(fileNamed: videoName)
videoNode.play()
currentVideoNode = videoNode
let videoScene = SKScene(size: CGSize(width: 480, height: 360))
videoNode.position = CGPoint(x: videoScene.size.width / 2, y: videoScene.size.height / 2)
videoNode.yScale = -1.0
videoScene.addChild(videoNode)
let plane = SCNPlane(width: imageAnchor.referenceImage.physicalSize.width, height: imageAnchor.referenceImage.physicalSize.height)
plane.firstMaterial?.diffuse.contents = videoScene
let planeNode = SCNNode(geometry: plane)
planeNode.eulerAngles.x = -.pi / 2
node.addChildNode(planeNode)
}
}
return node
}
}

How to place an AR object on plane WITHOUT ARHitTest?

I'm working on an AR project and attempting to add an object to a horizontal plane. My intention is for the user to search around for the object in their environment and once a horizontal plane is detected for the object to appear on the center point of the plane. Here's the code:
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
if anchor is ARPlaneAnchor {
guard let planeAnchor = anchor as? ARPlaneAnchor else { return }
let plane = SCNPlane(width: CGFloat(planeAnchor.extent.x), height: CGFloat(planeAnchor.extent.z))
let planeNode = SCNNode()
planeNode.position = SCNVector3(x: planeAnchor.center.x, y: 0, z: planeAnchor.center.z)
planeNode.geometry = plane
print("Plane Node: \(planeNode.position)")
planeNode.transform = SCNMatrix4MakeRotation(-Float.pi / 2, 1, 0, 0)
node.addChildNode(planeNode)
let chestScene = SCNScene(named: "art.scnassets/empty_treasure_chest copy.scn")
guard let chestNode = chestScene?.rootNode.childNode(withName: "chest", recursively: true) else { return }
sceneView.scene.rootNode.addChildNode(chestNode)
chestNode.position = planeNode.position
print("Chest Node: \(chestNode.position)")
sceneView.scene.rootNode.enumerateChildNodes { (node, _) in
if node.name == "chest_lid_top_wood" {
print("found top")
top = node
top.physicsBody = SCNPhysicsBody(type: .static, shape: SCNPhysicsShape(node: top, options: nil))
} else if node.name == "chest_bottom_wood" {
print("found bottom")
bottom = node
bottom.physicsBody = SCNPhysicsBody(type: .static, shape: SCNPhysicsShape(node: bottom, options: nil))
}
}
} else {
return
}
}
The app will detect the plane properly however the object won't be placed directly on the plane, most often it is elevated off of the plane at the world origin position. Even more confusing, I have print statements showing the position of the planeNode and chestNode, these values end up being different despite me assigning the position of the chestNode to the planeNode position. I understand most people use ARHitTest to place AR objects but doing that would hinder the UX I'm trying to achieve.
Any suggestions? Thanks in advance!
Cheers!
From my experience ARKit calls the delegate methods "to early", before the worldPosition of the node has been calculated. It is still [0, 0, 0], which is why your objects appear at the world origin.
There are two workarounds:
You could add your chestNode as a child of node instead of the scnene's rootNode.
However, this would result in the node moving every time ARKit updates the plane, because that would update the position of the node. This is usually what you want.
The alternative would be to wait for at least 1 frame before updating the position of your chestNode by wrapping the actual placement of your node inside something like
DispatchQueue.main.asyncAfter(deadline: .now() + 0.05) { [weak sceneView] in
sceneView?.scene.rootNode.addChildNode(chestNode)
chestNode.position = planeNode.position
}

Which image is being used as reference in ARSceneView?

I am currently making an AR app that uses image tracking.
I want it to have 2 images in the reference images folder (Image A: ~670x210, and Image B: ~1123x794).
How do I check which image is the camera detecting?
For example, if the camera is seeing Image A - I want to make a plane twice the size of the physical object, if it's seeing Image B - I want to make the plane the same size as the physical object.
Here's a snippet of my code, if needed:
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
// Create a session configuration
let configuration = ARImageTrackingConfiguration()
if let trackedImage = ARReferenceImage.referenceImages(inGroupNamed: "ARpaperImage", bundle: Bundle.main) {
configuration.trackingImages = trackedImage
configuration.maximumNumberOfTrackedImages = 1
}
// Run the view's session
sceneView.session.run(configuration)
}
// MARK: - ARSCNViewDelegate
func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
let node = SCNNode()
if let imageAnchor = anchor as? ARImageAnchor {
let plane = SCNPlane(width: imageAnchor.referenceImage.physicalSize.width, height: imageAnchor.referenceImage.physicalSize.height)
plane.firstMaterial?.diffuse.contents = UIColor(white: 1.0, alpha: 0,5)
let planeNode = SCNNode(geometry: plane)
planeNode.eulerAngles.x = -.pi / 2
node.addChildNode(planeNode)
}
return node
}
An ARReferenceImage has a name property which you can use to determine which ReferenceImage has been detected which is simply:
A descriptive name for the image
As such when you put your ARReferenceImages into your resource bundle you can and (probably already have) given them a name e.g:
You can the use these names within the following delegate callback to handle your requirements.
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor)
As such lets assume we have two ARReferenceImages called Brown and Choco. Using these names we can then create logic to display different content e.g:
//-------------------------
//MARK: - ARSCNViewDelegate
//-------------------------
extension ViewController: ARSCNViewDelegate{
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
//1. Check We Have Detected An ARImageAnchor
guard let validAnchor = anchor as? ARImageAnchor, let referenceImageName = validAnchor.referenceImage.name else { return }
//2. Get The Physical Size Of The Reference Image (This Is Specificed When You Create Your Target In The ARResource Bundle)
let physicalSizeOfReferenceImage = CGSize(width: validAnchor.referenceImage.physicalSize.width, height: validAnchor.referenceImage.physicalSize.height)
///3. Log The Data For Debugging Purposes
print("""
Detected ARAnchorID = \(validAnchor.identifier)
Detected Reference Image Name = \(referenceImageName)
Detected Reference Image Physical Size = (width) \(physicalSizeOfReferenceImage.width),
Detected Reference Image Physical Size = (height) \(physicalSizeOfReferenceImage.height)
""")
//4. Perform Instanciation & Other Login Based On The Reference Image Detected
if referenceImageName == "Brown"{
node.addChildNode(planeNodeOfSize(CGSize(width: physicalSizeOfReferenceImage.width * 2, height: physicalSizeOfReferenceImage.height * 2)))
}else if referenceImageName == "Choco"{
node.addChildNode(planeNodeOfSize(CGSize(width: physicalSizeOfReferenceImage.width, height: physicalSizeOfReferenceImage.height)))
}
}
/// Generates An SCNNode With An SCNPlaneGeometry Of A Specified Width & Height
///
/// - Parameter size: CGSize
/// - Returns: SCNNode
func planeNodeOfSize(_ size: CGSize) -> SCNNode{
//1. Create An SCNPlane Of Our Chosen Size
let planeNode = SCNNode()
let planeGeometry = SCNPlane(width: size.width, height: size.height )
planeGeometry.firstMaterial?.diffuse.contents = UIColor.white
planeNode.opacity = 0.25
planeNode.geometry = planeGeometry
//2. Rotate The PlaneNode To Horizontal
planeNode.eulerAngles.x = -.pi/2
//3. Return Our Node
return planeNode
}
}
Hope it helps...

3D Model is shaky with ARKit in Xcode

I am using ARKits image detection to place a 3D object when a certain image is detected. Everything works fine except for the 3D model that is being created. It is shaking like crazy. I double checked and the reference image has the right measures.
I call addModel() when the image is detected. Here is how my code looks like.
Finding reference Image:
func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
let node = SCNNode()
if let imageAnchor = anchor as? ARImageAnchor{
let plane = SCNPlane(width: imageAnchor.referenceImage.physicalSize.width, height: imageAnchor.referenceImage.physicalSize.height)
let planeNode = SCNNode(geometry: plane)
addModel(addTo: planeNode)
node.addChildNode(planeNode)
}
return node
}
The addModel() functions looks like this:
func addModel(addTo: SCNNode){
let testScene = SCNScene(named: "art.scnassets/testModel.scn")
let testNode = testScene?.rootNode.childNode(withName: "test", recursively: true)
let testMaterial = SCNMaterial()
testMaterial.diffuse.contents = UIImage(named: "art.scnassets/bricks")
testNode?.geometry?.materials = [testMaterial]
testNode!.position = SCNVector3Zero
testNode!.position.x = -0.3
testNode!.position.z = 0.3
addTo.addChildNode(testNode!)
}
Did your try to delete:
testNode!.position.x = -0.3
testNode!.position.z = 0.3 ?
Because when you type testNode!.position = SCNVector3Zero it is says, that
x = 0.0, y = 0.0, z = 0.0 and after that you type another coordinates.

How to detect touch and show new SCNPlane using ARKit?

Now I am able to show different SCNPlane, when card detected. After displaying SCNPlanes, the user touches any plane to show new SCNPlane. But right now touch is working properly but new SCNPlane is not showing.
Here is the code I've tried:
var cake_1_PlaneNode : SCNNode? = nil
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
guard let imageAnchor = anchor as? ARImageAnchor else { return }
if let imageName = imageAnchor.referenceImage.name {
print(imageName)
if imageName == "menu" {
// Check To See The Detected Size Of Our menu Card (Should By 5cm*3cm)
let menuCardWidth = imageAnchor.referenceImage.physicalSize.width
let menuCardHeight = imageAnchor.referenceImage.physicalSize.height
print(
"""
We Have Detected menu Card With Name \(imageName)
\(imageName)'s Width Is \(menuCardWidth)
\(imageName)'s Height Is \(menuCardHeight)
""")
//raspberry
//cake 1
let cake_1_Plane = SCNPlane(width: 0.045, height: 0.045)
cake_1_Plane.firstMaterial?.diffuse.contents = UIImage(named: "france")
cake_1_Plane.cornerRadius = 0.01
let cake_1_PlaneNode = SCNNode(geometry: cake_1_Plane)
self.cake_1_PlaneNode = cake_1_PlaneNode
cake_1_PlaneNode.eulerAngles.x = -.pi/2
cake_1_PlaneNode.runAction(SCNAction.moveBy(x: 0.15, y: 0, z: -0.125, duration: 0.75))
node.addChildNode(cake_1_PlaneNode)
self.sceneView.scene.rootNode.addChildNode(node)
}
}
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
let touch = touches.first as! UITouch
if(touch.view == self.sceneView){
//print("touch working")
let viewTouchLocation:CGPoint = touch.location(in: sceneView)
guard let result = sceneView.hitTest(viewTouchLocation, options: nil).first else {
return
}
if let planeNode = cake_1_PlaneNode, cake_1_PlaneNode == result.node{
print("match")
cake_1()
}
}
}
func cake_1() {
let plane = SCNPlane(width: 0.15 , height: 0.15)
plane.firstMaterial?.diffuse.contents = UIColor.black.withAlphaComponent(0.75)
let planeNodee = SCNNode(geometry: plane)
planeNodee.eulerAngles.x = -.pi / 2
planeNodee.runAction(SCNAction.moveBy(x: 0.21, y: 0, z: 0, duration: 0))
} //cake_1
Follow this link: Detect touch on SCNNode in ARKit.
Looking at your code I can see several issues (not to mention the naming conventions for your variables and methods).
Firstly, you are creating a Global Variable which you have declared like so:
var cake_1_PlaneNode : SCNNode? = nil
However you use both a Local and Global Variable for your cake_1_PlaneNode in yourDelegate Callback:
let cake_1_PlaneNode = SCNNode(geometry: cake_1_Plane)
self.cake_1_PlaneNode = cake_1_PlaneNode
Which should simply read like so:
self.cake_1_PlaneNode = SCNNode(geometry: cake_1_Plane)
Secondly, you are adding your cake_1_PlaneNode to the rootNode of your ARSCNView rather than your detected ARImageAnchor which is probably what you don't want to do, since when an ARAnchor is detected:
You can provide visual content for the anchor by attaching geometry
(or other SceneKit features) to this node or by adding child nodes.
As such, this method (unless you actually want to do it like this) is unnecessary.
The remaining issues lie within your cake_1 function itself.
Firstly you are not actually adding your planeNodee to your sceneHierachy.
Since you haven't specified whether or not the newly initialised planeNode should be added directly to your ARSCNView or as a childNode of your cake_1_planeNode your function should include one of the following:
self.sceneView.scene.rootNode.addChildNode(planeNodee)
self.cake_1_planeNode.addChildNode(planeNodee)
In addition there is probably also no need to rotate your planeNodee since by default an SCNPlane is rendered vertically.
Since you haven't stipulated where you will be placing your content, it could be that using -.pi / 2 is unnecessary, since this could make it virtually invisible to the naked eye.
One other issue which could also account for you not seeing your node, when you actually add it, is the Z position.
If you set 2 nodes at the same position you will likely experience an issue know as Z-fighting (which you can read more about here). As such you should probably move your added node forward slightly when adding it e.g. SCNVector3 (0,0,0.001)to account for this.
Based on all of these points, I have provided a fully working and commented example below:
import UIKit
import ARKit
//-------------------------
//MARK: - ARSCNViewDelegate
//-------------------------
extension ViewController: ARSCNViewDelegate {
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
//1. Check We Have An ARImageAnchor, Then Get It's Reference Image & Name
guard let imageAnchor = anchor as? ARImageAnchor else { return }
let detectedTarget = imageAnchor.referenceImage
guard let detectedTargetName = detectedTarget.name else { return }
//2. If We Have Detected Our Virtual Menu Then Add The CakeOnePlane
if detectedTargetName == "cakeMenu" {
let cakeOnePlaneGeometry = SCNPlane(width: 0.045, height: 0.045)
cakeOnePlaneGeometry.firstMaterial?.diffuse.contents = UIColor.cyan
cakeOnePlaneGeometry.cornerRadius = 0.01
let cakeOnPlaneNode = SCNNode(geometry: cakeOnePlaneGeometry)
cakeOnPlaneNode.eulerAngles.x = -.pi/2
//3. To Allow Us To Easily Keep Track Our Our Currently Added Node We Will Assign It A Unique Name
cakeOnPlaneNode.name = "Strawberry Cake"
node.addChildNode(cakeOnPlaneNode)
cakeOnPlaneNode.runAction(SCNAction.moveBy(x: 0.15, y: 0, z: 0, duration: 0.75))
}
}
}
class ViewController: UIViewController {
#IBOutlet var augmentedRealityView: ARSCNView!
let augmentedRealitySession = ARSession()
let configuration = ARWorldTrackingConfiguration()
//------------------
//MARK: - Life Cycle
//------------------
override func viewDidLoad() {
super.viewDidLoad()
setupARSession()
}
//-----------------
//MARK: - ARSession
//-----------------
/// Runs The ARSession
func setupARSession(){
//1. Load Our Detection Images
guard let detectionImages = ARReferenceImage.referenceImages(inGroupNamed: "AR Resources", bundle: nil) else { return }
//2. Configure & Run Our ARSession
augmentedRealityView.session = augmentedRealitySession
augmentedRealityView.delegate = self
configuration.detectionImages = detectionImages
augmentedRealitySession.run(configuration, options: [.resetTracking, .removeExistingAnchors])
}
//--------------------
//MARK: - Interaction
//--------------------
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
//1. Get The Current Touch Location & Perform An ARSCNHitTest To Check For Any Hit SCNNode's
guard let currentTouchLocation = touches.first?.location(in: self.augmentedRealityView),
let hitTestNode = self.augmentedRealityView.hitTest(currentTouchLocation, options: nil).first?.node else { return }
//2. If We Have Hit Our Strawberry Cake Then We Call Our makeCakeOnNode Function
if let cakeID = hitTestNode.name, cakeID == "Strawberry Cake"{
makeCakeOnNode(hitTestNode)
}
}
/// Adds An SCNPlane To A Detected Cake Target
///
/// - Parameter node: SCNNode
func makeCakeOnNode(_ node: SCNNode){
let planeGeometry = SCNPlane(width: 0.15 , height: 0.15)
planeGeometry.firstMaterial?.diffuse.contents = UIColor.black.withAlphaComponent(0.75)
let planeNode = SCNNode(geometry: planeGeometry)
planeNode.position = SCNVector3(0, 0, 0.001)
planeNode.runAction(SCNAction.moveBy(x: 0.21, y: 0, z: 0, duration: 0))
node.addChildNode(planeNode)
}
}
Which yields the following on my device:
For your information, this seems to show that your calculations for placing your content are off (unless of course this is the desired result).
As you can see, all of content rendered correctly, however the spacing of these was quite large, and as such you will likely need to pan your device somewhat to see it all when testing and developing further.
Hope it helps...
***Please use descriptive and clear names for your variables and functions, it is very hard to read and understand your code. You can read more about swift styling guidelines here: https://github.com/raywenderlich/swift-style-guide#naming
You are creating a new plane when the user touches the screen, but you are not adding that plane to the scene, therefore your "cake_1()" function only creates a new plane.
When ARKit detects an image, it automatically creates an empty node and adds it to our scene, at the center of the detected image. We must first keep a reference to the node ARKit has added for us when the image is detected.
Add this variable to the top of your class:
var detectedImageNode: SCNNode?
Then in func renderer(renderer: didAdd node:, for anchor:) add the following line:
detectedImageNode = node
Now that we have a reference to the node, we can easily add and remove other nodes.
Add the following line at the end of cake_1():
if let detectedImageNode = detectedImageNode {
cake_1_PlaneNode?.removeFromParentNode()
detectedImageNode.addChildNode(planeNodee)
}
Your final code should look like this:
func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
guard let imageAnchor = anchor as? ARImageAnchor else { return }
if let imageName = imageAnchor.referenceImage.name {
print(imageName)
if imageName == "menu" {
let cake_1_Plane = SCNPlane(width: 0.045, height: 0.045)
cake_1_Plane.firstMaterial?.diffuse.contents = UIImage(named: "france")
cake_1_Plane.cornerRadius = 0.01
let cake_1_PlaneNode = SCNNode(geometry: cake_1_Plane)
self.cake_1_PlaneNode = cake_1_PlaneNode
cake_1_PlaneNode.eulerAngles.x = -.pi/2
cake_1_PlaneNode.runAction(SCNAction.moveBy(x: 0.15, y: 0, z: -0.125, duration: 0.75))
node.addChildNode(cake_1_PlaneNode)
// No need to add the following line. The node is already added to the scene
//self.sceneView.scene.rootNode.addChildNode(node)
detectedImageNode = node
}
}
}
func cake_1() {
let plane = SCNPlane(width: 0.15 , height: 0.15)
plane.firstMaterial?.diffuse.contents = UIColor.black.withAlphaComponent(0.75)
let planeNodee = SCNNode(geometry: plane)
planeNodee.eulerAngles.x = -.pi / 2
if let detectedImageNode = detectedImageNode {
cake_1_PlaneNode?.removeFromParentNode()
detectedImageNode.addChildNode(planeNodee)
}
}
Alternative solution
If you are just trying to change the image of the plane then an easier way to approach this is to just change the texture of the plane.
Replace the contents of cake_1() with:
if let planeGeometry = cake_1_PlaneNode?.geometry {
planeGeometry.firstMaterial?.diffuse.contents = UIImage(named: "newImage")
}