How to create AR labels for ModelEntity in RealityKit? - swift

Question

I have a very simple app which puts down an .rcproject file.
import ARKit
import RealityKit
class ViewController: UIViewController {
private var marLent: Bool = false
private lazy var arView: ARView = {
let arview = ARView()
arview.translatesAutoresizingMaskIntoConstraints = false
arview.isUserInteractionEnabled = true
return arview
}()
override func viewDidLoad() {
super.viewDidLoad()
// Do any additional setup after loading the view.
let scene = try! Experience.loadScene()
arView.scene.anchors.append(scene)
configureUI()
setupARView()
}
private func configureUI() {
view.addSubview(arView)
arView.translatesAutoresizingMaskIntoConstraints = false
NSLayoutConstraint.activate([
arView.topAnchor.constraint(equalTo: view.topAnchor),
arView.leadingAnchor.constraint(equalTo: view.leadingAnchor),
arView.bottomAnchor.constraint(equalTo: view.bottomAnchor),
arView.trailingAnchor.constraint(equalTo: view.trailingAnchor),
])
}
private func setupARView() {
arView.automaticallyConfigureSession = false
let configuration = ARWorldTrackingConfiguration()
configuration.planeDetection = [.horizontal]
configuration.environmentTexturing = .automatic
arView.session.run(configuration)
}
}
How could I create a label for the placed down Entity which looks like something like these. So basically have a text that points on the entity and the text would be the entity's name for example.

Answer 1

There are 4 ways to create info dots with text plates for AR scenes. Here's an animated .gif.
the first way – using Autodesk Maya with pre-installed USD plugin (it's the most preferable way, because you can apply both animation and Python scripting techniques);
the second one – using Reality Composer (it's quite fast way, but you won't be able to exactly replicate info dots animation, like in Apple's .reality sample files);
the third one – programmatically in RealityKit;
the fourth way – programmatically using Pythonic USD Schema.
Nonetheless, for brevity, let's see how we can do it in Reality Composer app.
Reality Composer's behaviors
In Reality Composer's scene, drag and drop .png files with transparency (8-bit RGBA) to create an info dot and an info plate – each file will be turned into plane with its corresponding image. After that, you can apply Reality Composer's behaviors to any separate part of your model.
Create first custom behavior with a Scene Start trigger, then add LookAtCamera and Hide actions (when scene starts, both, a cylinder primitive and info plate must be hidden).
Create second behavior with a Tap trigger, then add LookAtCamera, Show, Wait and Hide actions (three actions must be merged together). If you tap an info dot, both hidden objects will be shown with the help of fade in/out animation.
Final step: save the scene as .reality file.
Hope, now you have an idea how it's done.