How to set a view to NOT update when a state changes - swift

I am trying to integrate an ARKit view which processes frames with machine learning and shows the results on the screen. I have gotten the ARKit view to work with UIViewRepresentable and everything works until a state changes. How do I make the AR view static and not update when a state changes. I only want to update the label that shows the result.
This is the error that I receive when the state changes: [CAMetalLayer nextDrawable] returning nil because allocation failed.
This presumably happens because the arView is being constantly reloaded as it processes the frames? Not too sure though.
This is the code for the view:
struct ARControlView: View {
#EnvironmentObject var resultHandler: ResultHandler
var body: some View {
let arView = ARViewContainer() // This is the UIViewRepresentable containing the ARKit view.
return ZStack {
arView
VStack {
Text(self.resultHandler.gesture.rawValue)
}
.onAppear {
arView.restartARSession()
}
.onDisappear {
arView.pauseArSession()
}
}
}
}
This is for the ARViewContainer:
struct ARViewContainer: UIViewRepresentable {
var arView = ARView(frame: .zero)
#EnvironmentObject var resultHandler: ResultHandler
func makeUIView(context: Context) -> ARView {
arView.session.delegate = context.coordinator
arView.session.run(AROrientationTrackingConfiguration())
return arView
}
func pauseArSession() {
arView.session.pause()
}
func restartARSession() {
arView.session.run(AROrientationTrackingConfiguration())
}
func updateUIView(_ uiView: ARView, context: Context) {}
class Coordinator: NSObject, ARSessionDelegate {
// Process frames here...
}
func makeCoordinator() -> ARViewContainer.Coordinator {
Coordinator(self)
}
}

Every time there is a state change in resultHandler, body is re-evaluated in ARControlView.
This causes a new ARViewContainer to be instantiated due to
let arView = ARViewContainer() being inside the body variable.
If you move let arView = ARViewContainer() outside of the body variable, arView won't be reinstantiated every time there is a state change.

I found out that the issue actually wasn't as it seemed. The error I was receiving was due to the frame being set to .zero which for some reason made it return nil. Also set it to not automatically configure because that also created weird issue causing the image to be stretched.
This was the line that I changed:
From:
var arView = ARView(frame: .zero)
To:
var arView = ARView(frame: .init(x: 1, y: 1, width: 1, height: 1), cameraMode: .ar, automaticallyConfigureSession: false)
Thanks for anyones else's help I appreciate it!

Related

Compass Heading information in RealityKit

I'm new to SwiftUI, and I've learned some basics of my project's RealityKit and ARKit. I just want to display an arrow that always faces towards the north, or at least get heading information displayed as text when I open the camera (AR Experience).
Waiting for someone to solve this fundamental problem.
Thanks in advance!
Use the following code to create AR experience that depends on device's geo position.
Reality Composer
Code
import SwiftUI
import RealityKit
import ARKit
struct ARViewContainer: UIViewRepresentable {
func makeUIView(context: Context) -> ARView {
let arView = ARView(frame: .zero)
arView.cameraMode = .ar
arView.automaticallyConfigureSession = false
let config = ARWorldTrackingConfiguration()
config.worldAlignment = .gravityAndHeading // case = 1
arView.session.run(config)
let arrowScene = try! Experience.loadNorth()
arView.scene.anchors.append(arrowScene)
return arView
}
func updateUIView(_ uiView: ARView, context: Context) { }
}
struct ContentView : View {
var body: some View {
ZStack {
ARViewContainer().ignoresSafeArea()
}
}
}
Settings
On device, go to Settings–Privacy–Location Services–On. After that, in Xcode, append Privacy LocationUsageDescription and LocationWhenInUseUsageDescription and then CameraUsageDescription keys in info tab.

How set Position of window on the Desktop in SwiftUI?

How to set window coordinates in SwiftUI on MacOS Desktop? For example, should the window appear always in the center or always in the upper right corner?
Here is my version, however, I shift the code and close it, when I open it, it appears first in the old place, and then jumps to a new place.
import SwiftUI
let WIDTH: CGFloat = 400
let HEIGTH: CGFloat = 200
#main
struct ForVSCode_MacOSApp: App {
#State var window : NSWindow?
var body: some Scene {
WindowGroup {
ContentView(win: $window)
}
}
}
struct WindowAccessor: NSViewRepresentable{
#Binding var window: NSWindow?
func makeNSView(context: Context) -> some NSView {
let view = NSView()
let width = (NSScreen.main?.frame.width)!
let heigth = (NSScreen.main?.frame.height)!
let resWidth: CGFloat = (width / 2) - (WIDTH / 2)
let resHeigt: CGFloat = (heigth / 2) - (HEIGTH / 2)
DispatchQueue.main.async {
self.window = view.window
self.window?.setFrameOrigin(NSPoint(x: resWidth, y: resHeigt))
self.window?.setFrameAutosaveName("mainWindow")
self.window?.isReleasedWhenClosed = false
self.window?.makeKeyAndOrderFront(nil)
}
return view
}
func updateNSView(_ nsView: NSViewType, context: Context) {
}
}
and ContentView
import SwiftUI
struct ContentView: View {
#Binding var win: NSWindow?
var body: some View {
VStack {
Text("it finally works!")
}
.font(.largeTitle)
.frame(width: WIDTH, height: HEIGTH, alignment: .center)
.background(WindowAccessor(window: $win))
}
}
struct ContentView_Previews: PreviewProvider {
#Binding var win: NSWindow?
static var previews: some View {
ContentView(win: .constant(NSWindow()))
.frame(width: 250, height: 150, alignment: .center)
}
}
I do have the same issue in one of my projects and thought I will investigate a bit deeper and I found two approaches to control the window position.
So my first approach to influence the window position is by pre-defining the windows last position on screen.
Indirect control: Frame autosave name
When the first window of an app is opened, macOS will try to restore the last window position when it was last closed. To distinguish the different windows, each window has its own frameAutosaveName.
The windows frame is persisted automatically in a text format in the apps preferences (UserDefaults.standard) with the key derived from the frameAutosaveName: "NSWindow Frame <frameAutosaveName>" (see docs for saveFrame).
If you do not specify an ID in your WindowGroup, SwiftUI will derive the autosave name from your main views class name. The first three windows will have the following autosave names:
<ModuleName>.ContentView-1-AppWindow-1
<ModuleName>.ContentView-1-AppWindow-2
<ModuleName>.ContentView-1-AppWindow-3
By setting an ID for example WindowGroup(id: "main"), the following autosave names are used (again for the first three windows):
main-AppWindow-1
main-AppWindow-2
main-AppWindow-3
When you check in your apps preferences directory (where UserDefaults.standard is stored), you will see in the plist one entry:
NSWindow Frame main-AppWindow-1 1304 545 400 228 0 0 3008 1228
There are a lot of numbers to digest. The first 4 integers describe the windows frame (origin and size), the next 4 integers describe the screens frame.
There are a few things to keep in mind when manually setting those value:
macOS coordinate system has it origin (0,0) in the bottom left corner.
the windows height includes the window title bar (28px on macOS Monterey but may be different on other versions)
the screens height excludes the title bar
I don't have documentation on this format and used trial and error to gain knowledge about it...
So to fake the initial position in the center of the screen I used the following function which I run in the apps (or the ContentView) initializer. But keep in mind: with this method only the first window will be centered. All the following windows are going to be put down and right of the previous window.
func fakeWindowPositionPreferences() {
let main = NSScreen.main!
let screenWidth = main.frame.width
let screenHeightWithoutMenuBar = main.frame.height - 25 // menu bar
let visibleFrame = main.visibleFrame
let contentWidth = WIDTH
let contentHeight = HEIGHT + 28 // window title bar
let windowX = visibleFrame.midX - contentWidth/2
let windowY = visibleFrame.midY - contentHeight/2
let newFramePreference = "\(Int(windowX)) \(Int(windowY)) \(Int(contentWidth)) \(Int(contentHeight)) 0 0 \(Int(screenWidth)) \(Int(screenHeightWithoutMenuBar))"
UserDefaults.standard.set(newFramePreference, forKey: "NSWindow Frame main-AppWindow-1")
}
My second approach is by directly manipulating the underlying NSWindow similar to your WindowAccessor.
Direct control: Manipulating NSWindow
Your implementation of WindowAccessor has a specific flaw: Your block which is reading view.window to extract the NSWindow instance is run asynchronously: some time in the future (due to DispatchQueue.main.async).
This is why the window appears on screen on the SwiftUI configured position, then disappears again to finally move to your desired location. You need more control, which involves first monitoring the NSView to get informed as soon as possible when the window property is set and then monitoring the NSWindow instance to get to know when the view is becoming visible.
I'm using the following implementation of WindowAccessor. It takes a onChange callback closure which is called whenever window is changing. First it starts monitoring the NSViews window property to get informed when the view is added to a window. When this happened, it starts listening for NSWindow.willCloseNotification notifications to detect when the window is closing. At this point it will stop any monitoring to avoid leaking memory.
import SwiftUI
import Combine
struct WindowAccessor: NSViewRepresentable {
let onChange: (NSWindow?) -> Void
func makeNSView(context: Context) -> NSView {
let view = NSView()
context.coordinator.monitorView(view)
return view
}
func updateNSView(_ view: NSView, context: Context) {
}
func makeCoordinator() -> WindowMonitor {
WindowMonitor(onChange)
}
class WindowMonitor: NSObject {
private var cancellables = Set<AnyCancellable>()
private var onChange: (NSWindow?) -> Void
init(_ onChange: #escaping (NSWindow?) -> Void) {
self.onChange = onChange
}
/// This function uses KVO to observe the `window` property of `view` and calls `onChange()`
func monitorView(_ view: NSView) {
view.publisher(for: \.window)
.removeDuplicates()
.dropFirst()
.sink { [weak self] newWindow in
guard let self = self else { return }
self.onChange(newWindow)
if let newWindow = newWindow {
self.monitorClosing(of: newWindow)
}
}
.store(in: &cancellables)
}
/// This function uses notifications to track closing of `window`
private func monitorClosing(of window: NSWindow) {
NotificationCenter.default
.publisher(for: NSWindow.willCloseNotification, object: window)
.sink { [weak self] notification in
guard let self = self else { return }
self.onChange(nil)
self.cancellables.removeAll()
}
.store(in: &cancellables)
}
}
}
This implementation can then be used to get a handle to NSWindow as soon as possible. The issue we still face: we don't have full control of the window. We are just monitoring what happens and can interact with the NSWindow instance. This means: we can set the position, but we don't know exactly at which instant this should happen. E.g. setting the windows frame directly after the view has been added to the window, will have no impact as SwiftUI is first doing layout calculations to decide afterwards where it will place the window.
After some fiddling around, I started tracking the NSWindow.isVisible property. This allows me to set the position whenever the window becomes visible. Using above WindowAccessor my ContentView implementation looks as follows:
import SwiftUI
import Combine
let WIDTH: CGFloat = 400
let HEIGHT: CGFloat = 200
struct ContentView: View {
#State var window : NSWindow?
#State private var cancellables = Set<AnyCancellable>()
var body: some View {
VStack {
Text("it finally works!")
.font(.largeTitle)
Text(window?.frameAutosaveName ?? "-")
}
.frame(width: WIDTH, height: HEIGHT, alignment: .center)
.background(WindowAccessor { newWindow in
if let newWindow = newWindow {
monitorVisibility(window: newWindow)
} else {
// window closed: release all references
self.window = nil
self.cancellables.removeAll()
}
})
}
private func monitorVisibility(window: NSWindow) {
window.publisher(for: \.isVisible)
.dropFirst() // we know: the first value is not interesting
.sink(receiveValue: { isVisible in
if isVisible {
self.window = window
placeWindow(window)
}
})
.store(in: &cancellables)
}
private func placeWindow(_ window: NSWindow) {
let main = NSScreen.main!
let visibleFrame = main.visibleFrame
let windowSize = window.frame.size
let windowX = visibleFrame.midX - windowSize.width/2
let windowY = visibleFrame.midY - windowSize.height/2
let desiredOrigin = CGPoint(x: windowX, y: windowY)
window.setFrameOrigin(desiredOrigin)
}
}
I hope this solution helps others who want to get more control to the window in SwiftUI.

Swift UI - How to access a #State variable externally?

I have ContentView.swift
struct ContentView: View {
#State var seconds: String = "60"
init(_ appDelegate: AppDelegate) {
launchTimer()
}
func launchTimer() {
let timer = DispatchTimeInterval.seconds(5)
let currentDispatchWorkItem = DispatchWorkItem {
print("in timer", "seconds", seconds) // Always `"60"`
print("in timer", "$seconds.wrappedValue", $seconds.wrappedValue) // Always `"60"`
launchTimer()
}
DispatchQueue.main.asyncAfter(deadline: .now() + timer, execute: currentDispatchWorkItem)
}
var body: some View {
TextField("60", text: $seconds)
}
func getSeconds() -> String {
return $seconds.wrappedValue;
}
}
And AppDelegate.swift
class AppDelegate: NSObject, NSApplicationDelegate {
var contentView = ContentView()
func applicationDidFinishLaunching(_ aNotification: Notification) {
launchTimer()
}
func launchTimer() {
print(contentView.getSeconds()) // Always `"60"`
let timer = DispatchTimeInterval.seconds(Int(contentView.getSeconds()) ?? 0)
DispatchQueue.main.asyncAfter(deadline: .now() + timer) {
self.launchTimer()
}
}
#objc func showPreferenceWindow(_ sender: Any?) {
if(window != nil) {
window.close()
}
window = NSWindow(
contentRect: NSRect(x: 0, y: 0, width: 480, height: 300),
styleMask: [.titled, .closable, .miniaturizable, .resizable, .fullSizeContentView],
backing: .buffered, defer: false)
window.isReleasedWhenClosed = false
window.contentView = NSHostingView(rootView: contentView)
window.center()
window.makeKeyAndOrderFront(nil)
NSApplication.shared.activate(ignoringOtherApps: true)
}
}
contentView.getSeconds() always returns 60, even if I modify the value of the TexField in my content view (by manually typing in the text field).
How does one get the wrapped value/real value of a state variable from the app delegate?
#State/#StateObject is tricky business, what happens is that SwiftUI connects the state values to a certain view instance from the UI hierarchy.
In your case, the #State var seconds: String = "60" is connected to the view below (simplified scheme):
NSWindow
-> NSHostingView
-> ContentView <----- this is where the #State usage is valid
And as others said, ContentView being a struct, it's a value type, meaning that its contents are copied in all usage sites, so instead of having a unique instance, like a class has, you end up with multiple instances.
And only one of those instances, the copy SwiftUI makes when it adds the view to the UI tree, is the one that is connected to the text field, and gets updated.
To make things even funnier, the State property wrapper is also a struct, meaning that it "suffers" from the same symptoms as the view.
That's one of the perks of using SwiftUI, in contrast with UIKit, you don't care at all about the view instances.
Now, if you want to use the value of seconds from AppDelegate, or any other place, you will have to circulate the data storage instead of the view.
For starters, change the view to receive a #Binding instead
struct ContentView: View {
#Binding var seconds: String = "60"
// the rest of the view code remains the same
Then create an ObservableObject that stores the data, and inject its $seconds binding when creating the view:
class AppData: ObservableObject {
#Published var seconds: String = "60"
}
class AppDelegate {
let appData = AppData()
// ... other code left out for clarity
#objc func showPreferenceWindow(_ sender: Any?) {
// ...
let contentView = ContentView(seconds: appData.$seconds
window.contentView = NSHostingView(rootView: contentView)
// ...
}
}
SwiftUI is more about the data than the view itself, so if you want to access the same data in multiple places, make sure you circulate the storage instead of the view the data is attached to. And also make sure that the source of truth of that data is a class, as object references are guaranteed to point to the same memory location (assuming of course the reference doesn't change).
Your ContentView (as well as any other SwiftUI view) is a struct, ie. value type, so in every place where you use it - you use a new copy of initially created value, ie.:
class AppDelegate: NSObject, NSApplicationDelegate {
var contentView = ContentView() // << 1st copy
...
func launchTimer() {
print(contentView.getSeconds()) // << 2nd copy
let timer = DispatchTimeInterval.seconds(Int(contentView.getSeconds()) ?? 0) // << 3d copy
...
}
...
#objc func showPreferenceWindow(_ sender: Any?) {
...
window.contentView = NSHostingView(rootView: contentView) // << 4th copy
}
}
You should not use any #State externally, because it is valid only inside view's body.
If you need something to share/access in different places then use class confirming to ObservableObject as view model type, and as instances of such class is a reference type then passing it here and there you can use/access same object from different places.
Update:
why the value stays to "60" even inside the ContentView
You call launchTimer in init, but in that place State is not constructed yet, so binding to it is not valid. As was already written above state is valid in body, so you have to set your timer also in body when binding will be ready, say in .onAppear, like below
var body: some View {
TextField("60", text: $seconds)
.onAppear {
launchTimer()
}
}
Prepared & tested with Xcode 13 / iOS 15
add a variable to appDelegate
var seconds : Int?
then do this in your contentView :
.onChange(of: seconds) { seconds in
guard let appDelegate = UIApplication.shared.delegate as? AppDelegate else {return}
appDelegate.seconds = seconds
}
Ended up instantiating the ContentView with the AppDelegate as a param. I added a seconds attribute to the AppDelegate to be modified by the ContentView.
AppDelegate.swift
import SwiftUI
#NSApplicationMain
class AppDelegate: NSObject, NSApplicationDelegate {
var contentView: ContentView?
var seconds = 60
func applicationDidFinishLaunching(_ aNotification: Notification) {
self.contentView = ContentView(self)
launchTimer()
}
func launchTimer() {
let timer = DispatchTimeInterval.seconds(seconds)
print(timer)
DispatchQueue.main.asyncAfter(deadline: .now() + timer) {
self.launchTimer()
}
}
}
ContentView.swift
import SwiftUI
struct ContentView: View {
#State var seconds: String = "60"
var appDelegate: AppDelegate
init(_ appDelegate: AppDelegate) {
self.appDelegate = appDelegate
}
var body: some View {
TextField("60", text: $seconds, onCommit: {
self.appDelegate.seconds = Int($seconds.wrappedValue) ?? 0
})
}
}
This code hurts my brain and surely there is a better way of doing this 😅. But I'm only so far in my Swift journey, so that will do for now lol. Please post if you have a better solution.

RealityKit – Image recognition and working with many scenes

I've created an app using the RealityKit template file. Inside RealityComposer there are multiple scenes, all the scenes use image recognition that activates some animations.
Inside Xcode I have to load all the scenes as anchors and append those anchors to arView.scene.anchors array. The issue is an obvious one, as I present the physical 2D image one after the other I get multiple anchors piled on top of each other which is not desirable. I'm aware of arView.scene.anchors.removeAll() prior to loading the new anchor but my issue is this:
How do I check when a certain image has appeared to therefore remove the existing anchor and load the correct one? I've tried to look for something like there is in ARKit as didUpdate but I can't see anything similar in RealityKit.
Many thanks
Foreword
RealityKit's AnchorEntity(.image) coming from RC, matches ARKit's ARImageTrackingConfig. When iOS device recognises a reference image, it creates Image Anchor (that conforms to ARTrackable protocol) that tethers a corresponding 3D model. And, as you understand, you must show just one reference image at a time (in your particular case AR app can't operate normally when you give it two or more images simultaneously).
Code snippet showing how if condition logic might look like:
import SwiftUI
import RealityKit
struct ContentView : View {
var body: some View {
return ARViewContainer().edgesIgnoringSafeArea(.all)
}
}
struct ARViewContainer: UIViewRepresentable {
func makeUIView(context: Context) -> ARView {
let arView = ARView(frame: .zero)
let id02Scene = try! Experience.loadID2()
print(id02Scene) // prints scene hierarchy
let anchor = id02Scene.children[0]
print(anchor.components[AnchoringComponent] as Any)
if anchor.components[AnchoringComponent] == AnchoringComponent(
.image(group: "Experience.reality",
name: "assets/MainID_4b51de84.jpeg")) {
arView.scene.anchors.removeAll()
print("LOAD SCENE")
arView.scene.anchors.append(id02Scene)
}
return arView
}
func updateUIView(_ uiView: ARView, context: Context) { }
}
ID2 scene hierarchy printed in console:
P.S.
You should implement SwiftUI Coordinator class (read about it here), and inside Coordinator use ARSessionDelegate's session(_:didUpdate:) instance method to update anchors properties at 60 fps.
Also you may use the following logic – if anchor of scene 1 is active or anchor of scene 3 is active, just delete all anchors from collection and load scene 2.
var arView = ARView(frame: .zero)
let id01Scene = try! Experience.loadID1()
let id02Scene = try! Experience.loadID2()
let id03Scene = try! Experience.loadID3()
func makeUIView(context: Context) -> ARView {
arView.session.delegate = context.coordinator
arView.scene.anchors.append(id01Scene)
arView.scene.anchors.append(id02Scene)
arView.scene.anchors.append(id03Scene)
return arView
}
...
func session(_ session: ARSession, didUpdate frame: ARFrame) {
if arView.scene.anchors[0].isActive || arView.scene.anchors[2].isActive {
arView.scene.anchors.removeAll()
arView.scene.anchors.append(id02Scene)
print("Load Scene Two")
}
}

Preview error with SwiftUI and RealityKit - Value of type 'ARView' has no member 'raycast'

I've got some trouble having a preview in Xcode 11.4. My code is working when my phone is plugged, so it's not a code problem, but when unplugged, build always failed. I'd like to be able to work on my project, on the other files not using AR, without this error. When I resume the preview on those other files, I'm blocked because of this error.
I've already put some strings in the info.plist file (privacy camera usage and required device capabilities) but still not working. Have an idea ?
import SwiftUI
import RealityKit
struct ContentView : View {
var body: some View {
return ARViewContainer().edgesIgnoringSafeArea(.all)
}
}
struct ARViewContainer: UIViewRepresentable {
func makeUIView(context: Context) -> ARView {
let arView = ARView(frame: .zero)
arView.enablePlacement()
return arView
}
func updateUIView(_ uiView: ARView, context: Context) {}
}
extension ARView {
func enablePlacement() {
let tapGestureRecognizer = UITapGestureRecognizer(target: self, action: #selector(handleTap(recognizer:)))
self.addGestureRecognizer(tapGestureRecognizer)
}
#objc func handleTap(recognizer: UITapGestureRecognizer) {
let location = recognizer.location(in: self)
let results = self.raycast(from: location, allowing: .estimatedPlane, alignment: .vertical)
if let firstResult = results.first {
let mesh = MeshResource.generateBox(width: 0.5, height: 0.02, depth: 0.2)
var material = SimpleMaterial()
material.baseColor = try! MaterialColorParameter.texture(TextureResource.load(named: "glacier"))
let modelEntity = ModelEntity(mesh: mesh,materials: [material])
let anchorEntity = AnchorEntity(world: firstResult.worldTransform)
anchorEntity.addChild(modelEntity)
self.backgroundColor = .orange
self.scene.addAnchor(anchorEntity)
}else{
print("No Surface detected - move around device")
}
}
}
#if DEBUG
struct ContentView_Previews : PreviewProvider {
static var previews: some View {
ContentView()
}
}
#endif
error value of type 'ARView' has no member 'raycast'.
Cannot infer contextual base in reference to member 'estimatedPlane'.
Cannot infer contextual base in reference to member 'vertical'.
Screen Capture
A lot of RealityKit symbols are not available in the Simulator. I think your only solution is to remove them from Simulator builds by using
#if !targetEnvironment(simulator)
/ ** /
#endif
This error occurs because you have a device selected in the simulator and the simulator cannot be used with AR apps. Change the dropdown next to the play and stop buttons at the top of Xcode to Any iOS Device (arm64)
You will need to connect a physical device to Xcode or push your app to the AppStoreConnect and use Testflight to test your code instead of the simulator.