RealityKit – Synchronization Component - swift

I am trying to build a collaborative session using RealityKit. As far as I understand I need to conform to synchronisation component to my Entity.
So my question is: is there any source of documentation about using this component? Specially that I don't understand if I should handle from the recipient side any delegate methods for handing received data.
One thing that I want to stress: I am searching for solution with RealityKit not ARKit 3. And as per apple in the WWDC they say:
If you are using RealityKit, this is the only new code you need to add
to use collaborative session. If you are not using RealityKit, then
you need to implement additional two delegate functions to transmit
the collaboration data.
Thanks.

Adopt ARSessionDelegate and MCSessionDelegate protocols, implement their delegates and set isCollaborationEnabled instance property to true:
import RealityKit
import ARKit
import MultipeerConnectivity
class ViewController: UIViewController, ARSessionDelegate, MCSessionDelegate {
#IBOutlet var arView: ARView!
var mcSession: MCSession?
override func viewDidLoad() {
super.viewDidLoad()
arView.session.delegate = self
mcSession!.delegate = self
let config = ARWorldTrackingConfiguration()
config.planeDetection = [.horizontal]
config.isCollaborationEnabled = true
arView.debugOptions = [.showFeaturePoints]
arView.session.run(config)
}
}
Then use ARSessionDelegate's and MCSessionDelegate's session() instance methods:
extension ViewController {
let name = UIDevice.current.name
let myPeerID = MCPeerID(displayName: name)
var peers = [MCPeerID]()
peers.append(myPeerID)
func session(_ session: ARSession,
didOutputCollaborationData data: ARSession.CollaborationData) {
self.mcSession = MCSession(peer: myPeerID,
securityIdentity: nil,
encryptionPreference: .required)
do {
try self.mcSession.send(data.collaborationData(),
toPeers: peers,
with: .reliable)
} catch {
print("Get Error while outputting Collaboration Data")
}
}
func session(_ session: MCSession,
didReceive data: Data,
fromPeer peerID: MCPeerID) {
self.arView.session.update(with: data.data())
}
func session(_ session: ARSession,
didRemove anchors: [ARAnchor]) {
for anchor in anchors {
if anchor.sessionIdentifier = session.identifier {
// your anchors here...
}
}
}
}
extension ARSession.CollaborationData {
func collaborationData() -> Data {
let data = Data()
// ...
return data
}
}
extension Data {
func data() -> ARSession.CollaborationData {
let data = ARSession.CollaborationData(coder: nsCoder)
// ...
return data!
}
}
You can read and watch about it here.

Related

Selector type of expression is ambiguous without more context

I'm trying to get ARSessionDelegate method using selector, but im getting this error:
Type of expression is ambiguous without more context
There is mine code:
#selector(ARSessionDelegate.session(_:didUpdate:) as ((ARSessionDelegate) -> (ARSession, ARFrame) -> Void))
That's how this method looks like:
public protocol ARSessionDelegate : ARSessionObserver {
optional func session(_ session: ARSession, didUpdate frame: ARFrame)
}
And also, I am trying to make an rx extension for the ARKit session using this answer, but im not sure it's caused the problem.
Because there are multiple methods with the same selector name, you are forced to implement the method in the delegate and forward the calls using a subject. Like this:
extension ARSession: HasDelegate { }
extension Reactive where Base: ARSession {
var delegate: ARSessionDelegateProxy {
return ARSessionDelegateProxy.proxy(for: base)
}
var didUpdate: Observable<ARFrame> {
return delegate.didUpdate.asObservable()
}
}
final class ARSessionDelegateProxy
: DelegateProxy<ARSession, ARSessionDelegate>
, DelegateProxyType
, ARSessionDelegate {
init(parentObject: ARSession) {
super.init(
parentObject: parentObject,
delegateProxy: ARSessionDelegateProxy.self
)
}
deinit {
didUpdate.onCompleted()
}
public static func registerKnownImplementations() {
self.register { ARSessionDelegateProxy(parentObject: $0) }
}
func session(_ session: ARSession, didUpdate frame: ARFrame) {
didUpdate.onNext(frame)
}
fileprivate let didUpdate = PublishSubject<ARFrame>()
}

How can I move a model using facial expression like look-right?

I have set a virtual object in rear camera view. I want to move that object using facial expression with respect to world origin and measure the displacement angles of the virtual object.
Is that possible using ARKit or RealityKit?
Use the following solution. At first setup a configuration:
import RealityKit
import ARKit
class ViewController: UIViewController, ARSessionDelegate {
#IBOutlet var arView: ARView!
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
arView.session.delegate = self
arView.automaticallyConfigureSession = false
let config = ARFaceTrackingConfiguration()
config.isWorldTrackingEnabled = true // Simultaneous tracking
arView.session.run(config)
}
}
Run your transform animation when a defined facial expression occurs:
func facialExpression(anchor: ARFaceAnchor) {
let eyeUpLeft = anchor.blendShapes[.eyeLookUpLeft]
let eyeUpRight = anchor.blendShapes[.eyeLookUpRight]
if ((eyeUpLeft?.decimalValue ?? 0.0) +
(eyeUpRight?.decimalValue ?? 0.0)) > 0.75 {
// ModelEntity's animation goes here
}
}
Delegate's method (running at 60 fps):
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
guard let faceAnchor = anchors[0] as? ARFaceAnchor else { return }
self.facialExpression(anchor: faceAnchor)
}
The answer to your second question you can see HERE.

Using UIApplicationDelegateAdaptor to get callbacks from userDidAcceptCloudKitShareWith not working

I'm trying to get notified when userDidAcceptCloudKitShareWith gets called. Traditionally this was called in the App Delegate but since I am building an iOS 14+ using App as my root object. I couldn't find any documentation out yet as far as how to add userDidAcceptCloudKitShareWith to my App class, so I am using UIApplicationDelegateAdaptor to use an App Delegate class, however it doesn't seem like userDidAcceptCloudKitShareWith is ever getting called?
import SwiftUI
import CloudKit
// Our observable object class
class ShareDataStore: ObservableObject {
static let shared = ShareDataStore()
#Published var didRecieveShare = false
#Published var shareInfo = ""
}
#main
struct SocialTestAppApp: App {
#StateObject var shareDataStore = ShareDataStore.shared
#UIApplicationDelegateAdaptor(AppDelegate.self) var appDelegate
var body: some Scene {
WindowGroup {
ContentView().environmentObject(shareDataStore)
}
}
}
class AppDelegate: NSObject, UIApplicationDelegate {
let container = CKContainer(identifier: "iCloud.com.TestApp")
func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey : Any]? = nil) -> Bool {
print("did finish launching called")
return true
}
func application(_ application: UIApplication, userDidAcceptCloudKitShareWith cloudKitShareMetadata: CKShare.Metadata) {
print("delegate callback called!! ")
acceptShare(metadata: cloudKitShareMetadata) { result in
switch result {
case .success(let recordID):
print("successful share!")
ShareDataStore.shared.didRecieveShare = true
ShareDataStore.shared.shareInfo = recordID.recordName
case .failure(let error):
print("failure in share = \(error)")
}
} }
func acceptShare(metadata: CKShare.Metadata,
completion: #escaping (Result<CKRecord.ID, Error>) -> Void) {
// Create a reference to the share's container so the operation
// executes in the correct context.
let container = CKContainer(identifier: metadata.containerIdentifier)
// Create the operation using the metadata the caller provides.
let operation = CKAcceptSharesOperation(shareMetadatas: [metadata])
var rootRecordID: CKRecord.ID!
// If CloudKit accepts the share, cache the root record's ID.
// The completion closure handles any errors.
operation.perShareCompletionBlock = { metadata, share, error in
if let _ = share, error == nil {
rootRecordID = metadata.rootRecordID
}
}
// If the operation fails, return the error to the caller.
// Otherwise, return the record ID of the share's root record.
operation.acceptSharesCompletionBlock = { error in
if let error = error {
completion(.failure(error))
} else {
completion(.success(rootRecordID))
}
}
// Set an appropriate QoS and add the operation to the
// container's queue to execute it.
operation.qualityOfService = .utility
container.add(operation)
}
}
Updated based on Asperi's Answer:
import SwiftUI
import CloudKit
class ShareDataStore: ObservableObject {
static let shared = ShareDataStore()
#Published var didRecieveShare = false
#Published var shareInfo = ""
}
#main
struct athlyticSocialTestAppApp: App {
#StateObject var shareDataStore = ShareDataStore.shared
#UIApplicationDelegateAdaptor(AppDelegate.self) var appDelegate
let sceneDelegate = MySceneDelegate()
var body: some Scene {
WindowGroup {
ContentView().environmentObject(shareDataStore)
.withHostingWindow { window in
sceneDelegate.originalDelegate = window.windowScene.delegate
window.windowScene.delegate = sceneDelegate
}
}
}
}
class MySceneDelegate: NSObject, UIWindowSceneDelegate {
let container = CKContainer(identifier: "iCloud.com...")
var originalDelegate: UIWindowSceneDelegate?
var window: UIWindow?
func sceneWillEnterForeground(_ scene: UIScene) {
print("scene is active")
}
func sceneWillResignActive(_ scene: UIScene) {
print("scene will resign active")
}
// forward all other UIWindowSceneDelegate/UISceneDelegate callbacks to original, like
func scene(_ scene: UIScene, willConnectTo session: UISceneSession, options connectionOptions: UIScene.ConnectionOptions) {
originalDelegate?.scene!(scene, willConnectTo: session, options: connectionOptions)
}
func windowScene(_ windowScene: UIWindowScene, userDidAcceptCloudKitShareWith cloudKitShareMetadata: CKShare.Metadata) {
print("delegate callback called!! ")
acceptShare(metadata: cloudKitShareMetadata) { result in
switch result {
case .success(let recordID):
print("successful share!")
ShareDataStore.shared.didRecieveShare = true
ShareDataStore.shared.shareInfo = recordID.recordName
case .failure(let error):
print("failure in share = \(error)")
}
}
}
}
extension View {
func withHostingWindow(_ callback: #escaping (UIWindow?) -> Void) -> some View {
self.background(HostingWindowFinder(callback: callback))
}
}
struct HostingWindowFinder: UIViewRepresentable {
var callback: (UIWindow?) -> ()
func makeUIView(context: Context) -> UIView {
let view = UIView()
DispatchQueue.main.async { [weak view] in
self.callback(view?.window)
}
return view
}
func updateUIView(_ uiView: UIView, context: Context) {
}
}
In Scene-based application the userDidAcceptCloudKitShareWith callback is posted to Scene delegate, but in SwiftUI 2.0 App-based application the scene delegate is used by SwiftUI itself to provide scenePhase events, but does not provide native way to handle topic callback.
The possible approach to solve this is to find a window and inject own scene delegate wrapper, which will handle userDidAcceptCloudKitShareWith and forward others to original SwiftUI delegate (to keep standard SwiftUI events working).
Here is a couple of demo snapshots based on https://stackoverflow.com/a/63276688/12299030 window access (however you can use any other preferable way to get window)
#main
struct athlyticSocialTestAppApp: App {
#StateObject var shareDataStore = ShareDataStore.shared
#UIApplicationDelegateAdaptor(AppDelegate.self) var appDelegate
let sceneDelegate = MySceneDelegate()
var body: some Scene {
WindowGroup {
ContentView().environmentObject(shareDataStore)
.withHostingWindow { window in
sceneDelegate.originalDelegate = window?.windowScene.delegate
window?.windowScene.delegate = sceneDelegate
}
}
}
}
class MySceneDelegate : NSObject, UIWindowSceneDelegate {
var originalDelegate: UISceneDelegate?
func windowScene(_ windowScene: UIWindowScene, userDidAcceptCloudKitShareWith cloudKitShareMetadata: CKShareMetadata) {
// your code here
}
// forward all other UIWindowSceneDelegate/UISceneDelegate callbacks to original, like
func scene(_ scene: UIScene, willConnectTo session: UISceneSession, options connectionOptions: UIScene.ConnectionOptions) {
originalDelegate?.scene(scene, willConnectTo: session, options: connectionOptions)
}
}
Check out this question that has a lot of useful things to check across several possible answers:
CloudKit CKShare userDidAcceptCloudKitShareWith Never Fires on Mac App
Be sure to add the CKSharingSupported key to your info.plist, and then try putting the userDidAcceptCloudKitShareWith in different places using the answers in the above link (where you put it will depend on what kind of app you're building).

Face Recognition iPhone X Swift

I done some experiments with ARFaceAnchor for recognize some emotions like blinking eyes and so on. For sure I set correctly the FaceAnchor because on the debugger I'm able to see the coordinates but it seems that it's not recognizing any emotions that I set...
Attached you will find the ViewController and in a separate Class you will find the Emotions.
Any ideas? Thank you!
// ViewController.swift
//
import UIKit
import SceneKit
import ARKit
class ViewController: UIViewController, ARSessionDelegate {
#IBOutlet var sceneView: ARSCNView!
let session = ARSession()
override func viewDidLoad() {
super.viewDidLoad()
self.sceneView.scene = SCNScene()
self.sceneView.rendersContinuously = true
// Configure our ARKit tracking session for facial recognition
let config = ARFaceTrackingConfiguration()
config.worldAlignment = .gravity
session.delegate = self
session.run(config, options: [])
}
// AR Session
var currentFaceAnchor: ARFaceAnchor?
var currentFrame: ARFrame?
func session(_ session: ARSession, didUpdate frame: ARFrame) {
self.currentFrame = frame
DispatchQueue.main.async {
}
}
func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
}
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
guard let faceAnchor = anchors.first as? ARFaceAnchor else { return }
self.currentFaceAnchor = faceAnchor
print("Face",faceAnchor)
}
func session(_ session: ARSession, didRemove anchors: [ARAnchor]) {
}
var expressionsToUse: [Expression] = [SmileExpression(), EyebrowsRaisedExpression(), EyeBlinkLeftExpression(), EyeBlinkRightExpression(), JawOpenExpression(), LookLeftExpression(), LookRightExpression()] //All the expressions
var currentExpression: Expression? = nil {
didSet {
if currentExpression != nil {
self.currentExpressionShownAt = Date()
} else {
self.currentExpressionShownAt = nil
}
}
}
var currentExpressionShownAt: Date? = nil
}
The reason no Expressions are being detected is because you aren't actually doing anything with them, apart from adding them to expressionsToUse Array.
Each Expression has three functions, which you aren't currently using:
func name() -> String {}
func isExpressing(from: ARFaceAnchor) -> Bool {}
func isDoingWrongExpression(from: ARFaceAnchor) -> Bool {}
Since you want to detect the Emotions you need to hook these functions into the following delegate method:
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) { }
As such something like this will point you in the right direction:
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
//1. Check To See We Have A Valid ARFaceAnchor
guard let faceAnchor = anchors.first as? ARFaceAnchor else { return }
self.currentFaceAnchor = faceAnchor
//2. Loop Through Each Of The Expression & Determine Which One Is Being Used
expressionsToUse.forEach { (possibleExpression) in
//a. If The The User Is Doing A Particular Expression Then Assign It To The currentExpression Variable
if possibleExpression.isExpressing(from: faceAnchor){
currentExpression = possibleExpression
print("""
Current Detected Expression = \(possibleExpression.name())
It Was Detected On \(currentExpressionShownAt!)
""")
}else if possibleExpression.isDoingWrongExpression(from: faceAnchor){
print("Incorrect Detected Expression = \(possibleExpression.name())")
}
}
}
Hope it helps...

Sending variable with watchConnectivity but can't get the picker in the watch app to update

I've been trying to send a variable to the watch from the iPhone. I've managed to send it with watchConnectivity but I can't get the picker in the watch app to update with the new variable I sent through.
Here's the code for the watch app:
import WatchKit
import Foundation
import WatchConnectivity
var bigDict = ["":""]
class InterfaceController: WKInterfaceController, WCSessionDelegate {
lazy var keys = Array(bigDict.keys)
lazy var values = Array(bigDict.values)
var pickerItems: [WKPickerItem] = []
#IBOutlet var pickerW: WKInterfacePicker!
#IBAction func pickerDidChange(_ value: Int) {
}
#IBAction func updateButton() {
for item in keys{
let pickerItem = WKPickerItem()
pickerItem.title = item
pickerItem.caption = bigDict[item]
pickerItems += [pickerItem]
}
pickerW.setItems(pickerItems)
}
//func refreshPickerItems() {
//for item in keys{
//let pickerItem = WKPickerItem()
//pickerItem.title = item
//pickerItem.caption = bigDict[item]
//pickerItems += [pickerItem]
// }
//pickerW.setItems(pickerItems)
// }
override func awake(withContext context: Any?) {
super.awake(withContext: context)
// Configure interface objects here.
//refreshPickerItems()
}
override func willActivate() {
// This method is called when watch view controller is about to be visible to user
super.willActivate()
if WCSession.isSupported() {
let session = WCSession.default
session.delegate = self
session.activate()
}
}
func session(_ session: WCSession, activationDidCompleteWith activationState: WCSessionActivationState, error: Error?) {
}
func session(_ session: WCSession, didReceiveMessage message: [String : Any]) {
print(message)
bigDict = message as! [String : String]
print(bigDict)
}
}
Shouldn't you call "updateButton()" when you receive the data?
func session(_ session: WCSession, didReceiveMessage message: [String : Any]) {
print(message)
bigDict = message as! [String : String]
updateButton()
print(bigDict)
}