So I am trying to add a multipeer element to this Sticky Note app from Apple's own Sample Code. Link to Sample Code page There are several examples of multipeer ARKit apps but the problem here is, with the app I am working from, the Sticky Note is NOT a 3D element but
For the purposes of this sample app, the sticky note entity has no geometry and thus, no appearance. Its anchor provides a 3D location only, and itʼs the sticky noteʼs screen-space annotation that has an appearance. To display it, you define the sticky noteʼs annotation. Following RealityKitʼs entity-component model, design a component that houses the annotation, which in this case is a view. See ScreenSpaceComponent.
I have been trying to use the example of multipeer apps in ARthat use the ARKit element with 3D elements stored as either assets [the "Collaborative Session" example ] or using ModelEntity geometry [the Creating a Multiuser AR Experience example ] but I haven't been successful in translating this app which uses screen space only.
I am able to get the message on the screen that it's connected to a peer, but that is as far as it goes. It will not render the notes on the second phone. I am burned out from all the attempts of making it work:(
One alternative is to forget about the notes being tethered to the screen space, and recreating this as a regular 3D space and 2D geometry thing using SpriteKit.
The system will not render the apps sticky notes on the other phone. I know there is a way around this, but I have been trying for days and haven't been able to do it.
I have been testing this using 2 phones.
I have
Added the info on the p.list
Added the Multipeer Session file
Added the code on the ViewController file related to multipeer
Added code to the arGestureSetUp() extension file which has the rendering info for the sticky notes.
What works: I can see the notes on both phones, and I get the messages saying that a peer has joined. What I can't do is view the
other user's notes like I would in a regular 3D ARkit app. It will not
render.
This is what I have added to the insertNewSticky function
func insertNewSticky(_ sender: UITapGestureRecognizer)
from one of the other examples:
let anchor = ARAnchor(name: "Anchor for object placement", transform: raycastResult.worldTransform)
arView.session.add(anchor: anchor)
Below is the full code for the Gesture Recognizer Setup
import UIKit
import ARKit
extension ViewController {
// MARK: - Gesture recognizer setup
// - Tag: AddViewTapGesture
func arViewGestureSetup() {
let tapGesture = UITapGestureRecognizer(target: self, action: #selector(tappedOnARView))
arView.addGestureRecognizer(tapGesture)
let swipeGesture = UISwipeGestureRecognizer(target: self, action: #selector(swipedDownOnARView))
swipeGesture.direction = .down
arView.addGestureRecognizer(swipeGesture)
}
func stickyNoteGestureSetup(_ note: StickyNoteEntity) {
let panGesture = UIPanGestureRecognizer(target: self, action: #selector(panOnStickyView))
note.view?.addGestureRecognizer(panGesture)
let tapOnStickyView = UITapGestureRecognizer(target: self, action: #selector(tappedOnStickyView(_:)))
note.view?.addGestureRecognizer(tapOnStickyView)
}
// MARK: - Gesture recognizer callbacks
/// Tap gesture input handler.
/// - Tag: TapHandler
#objc
func tappedOnARView(_ sender: UITapGestureRecognizer) {
// Ignore the tap if the user is editing a sticky note.
for note in stickyNotes where note.isEditing { return }
// Create a new sticky note at the tap location.
insertNewSticky(sender)
}
/**
Hit test the feature point cloud and use any hit as the position of a new StickyNote. Otherwise, display a tip.
- Tag: ScreenSpaceViewInsertionTag
*/
func insertNewSticky(_ sender: UITapGestureRecognizer) {
// Get the user's tap screen location.
let touchLocation = sender.location(in: arView)
// Cast a ray to check for its intersection with any planes.
guard let raycastResult = arView.raycast(from: touchLocation, allowing: .estimatedPlane, alignment: .any).first
else {
messageLabel.displayMessage("No surface detected, try getting closer.", duration: 2.0)
return
}
// Create a new sticky note positioned at the hit test result's world position.
let frame = CGRect(origin: touchLocation, size: CGSize(width: 200, height: 200))
let note = StickyNoteEntity(frame: frame, worldTransform: raycastResult.worldTransform)
// Center the sticky note's view on the tap's screen location.
note.setPositionCenter(touchLocation)
// Add the sticky note to the scene's entity hierarchy.
arView.scene.addAnchor(note)
// Add the sticky note's view to the view hierarchy.
guard let stickyView = note.view else { return }
arView.insertSubview(stickyView, belowSubview: trashZone)
// Enable gestures on the sticky note.
stickyNoteGestureSetup(note)
// Save a reference to the sticky note.
stickyNotes.append(note)
// Volunteer to handle text view callbacks.
stickyView.textView.delegate = self
let anchor = ARAnchor(name: "Anchor for object placement", transform: raycastResult.worldTransform)
arView.session.add(anchor: anchor)
}
/// Dismisses the keyboard.
#objc
func swipedDownOnARView(_ sender: UISwipeGestureRecognizer) {
dismissKeyboard()
}
fileprivate func dismissKeyboard() {
for note in stickyNotes {
guard let textView = note.view?.textView else { continue }
if textView.isFirstResponder {
textView.resignFirstResponder()
return
}
}
}
#objc
func tappedOnStickyView(_ sender: UITapGestureRecognizer) {
guard let stickyView = sender.view as? StickyNoteView else { return }
stickyView.textView.becomeFirstResponder()
}
//- Tag: PanOnStickyView
fileprivate func panStickyNote(_ sender: UIPanGestureRecognizer, _ stickyView: StickyNoteView, _ panLocation: CGPoint) {
messageLabel.isHidden = true
let feedbackGenerator = UIImpactFeedbackGenerator()
switch sender.state {
case .began:
// Prepare the taptic engine to reduce latency in delivering feedback.
feedbackGenerator.prepare()
// Drag if the gesture is beginning.
stickyView.stickyNote.isDragging = true
// Save offsets to implement smooth panning.
guard let frame = sender.view?.frame else { return }
stickyView.xOffset = panLocation.x - frame.origin.x
stickyView.yOffset = panLocation.y - frame.origin.y
// Fade in the widget that's used to delete sticky notes.
trashZone.fadeIn(duration: 0.4)
case .ended:
// Stop dragging if the gesture is ending.
stickyView.stickyNote.isDragging = false
// Delete the sticky note if the gesture ended on the trash widget.
if stickyView.isInTrashZone {
deleteStickyNote(stickyView.stickyNote)
// ...
} else {
attemptRepositioning(stickyView)
}
// Fades out the widget that's used to delete sticky notes when there are no sticky notes currently being dragged.
if !stickyNotes.contains(where: { $0.isDragging }) {
trashZone.fadeOut(duration: 0.2)
}
default:
// Update the sticky note's screen position based on the pan location, and initial offset.
stickyView.frame.origin.x = panLocation.x - stickyView.xOffset
stickyView.frame.origin.y = panLocation.y - stickyView.yOffset
// Give feedback whenever the pan location is near the widget used to delete sticky notes.
trashZoneThresholdFeedback(sender, feedbackGenerator)
}
}
/// Sticky note pan-gesture handler.
/// - Tag: PanHandler
#objc
func panOnStickyView(_ sender: UIPanGestureRecognizer) {
guard let stickyView = sender.view as? StickyNoteView else { return }
let panLocation = sender.location(in: arView)
// Ignore the pan if any StickyViews are being edited.
for note in stickyNotes where note.isEditing { return }
panStickyNote(sender, stickyView, panLocation)
}
func deleteStickyNote(_ note: StickyNoteEntity) {
guard let index = stickyNotes.firstIndex(of: note) else { return }
note.removeFromParent()
stickyNotes.remove(at: index)
note.view?.removeFromSuperview()
note.view?.isInTrashZone = false
}
/// - Tag: AttemptRepositioning
fileprivate func attemptRepositioning(_ stickyView: StickyNoteView) {
// Conducts a ray-cast for feature points using the panned position of the StickyNoteView
let point = CGPoint(x: stickyView.frame.midX, y: stickyView.frame.midY)
if let result = arView.raycast(from: point, allowing: .estimatedPlane, alignment: .any).first {
stickyView.stickyNote.transform.matrix = result.worldTransform
} else {
messageLabel.displayMessage("No surface detected, unable to reposition note.", duration: 2.0)
stickyView.stickyNote.shouldAnimate = true
}
}
fileprivate func trashZoneThresholdFeedback(_ sender: UIPanGestureRecognizer, _ feedbackGenerator: UIImpactFeedbackGenerator) {
guard let stickyView = sender.view as? StickyNoteView else { return }
let panLocation = sender.location(in: trashZone)
if trashZone.frame.contains(panLocation), !stickyView.isInTrashZone {
stickyView.isInTrashZone = true
feedbackGenerator.impactOccurred()
} else if !trashZone.frame.contains(panLocation), stickyView.isInTrashZone {
stickyView.isInTrashZone = false
feedbackGenerator.impactOccurred()
}
}
#objc
func tappedReset(_ sender: UIButton) {
reset()
}
}
and this is the full code for the ViewController file
/*
See LICENSE folder for this sample’s licensing information.
Abstract:
Main view controller for the AR experience.
*/
import UIKit
import RealityKit
import Combine
import ARKit
import MultipeerConnectivity
class ViewController: UIViewController, ARSessionDelegate {
// MARK: - Class variable declarations
#IBOutlet var arView: ARView!
#IBOutlet weak var messageLabel: MessageLabel!
var trashZone: GradientView!
var shadeView: UIView!
var resetButton: UIButton!
var keyboardHeight: CGFloat!
var stickyNotes = [StickyNoteEntity]()
var subscription: Cancellable!
//added Sat May 28 5:12pm
var multipeerSession: MultipeerSession?
// end of added Sat May 28 5:12pm
//added Sat May 28 5:12pm
// A dictionary to map MultiPeer IDs to ARSession ID's.
// This is useful for keeping track of which peer created which ARAnchors.
var peerSessionIDs = [MCPeerID: String]()
var sessionIDObservation: NSKeyValueObservation?
var configuration: ARWorldTrackingConfiguration?
// end of added Sat May 28 5:12pm
// MARK: - View Controller Life Cycle
override func viewDidLoad() {
super.viewDidLoad()
subscription = arView.scene.subscribe(to: SceneEvents.Update.self) { [unowned self] in
self.updateScene(on: $0)
}
arViewGestureSetup()
overlayUISetup()
arView.session.delegate = self
}
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
// Add observer to the keyboardWillShowNotification to get the height of the keyboard every time it is shown
let notificationName = UIResponder.keyboardWillShowNotification
let selector = #selector(keyboardIsPoppingUp(notification:))
NotificationCenter.default.addObserver(self, selector: selector, name: notificationName, object: nil)
}
override func viewDidAppear(_ animated: Bool) {
super.viewDidAppear(animated)
arView.session.delegate = self
// Prevent the screen from being dimmed to avoid interuppting the AR experience.
UIApplication.shared.isIdleTimerDisabled = true
// Turn off ARView's automatically-configured session
// to create and set up your own configuration.
arView.automaticallyConfigureSession = false
configuration = ARWorldTrackingConfiguration()
// Enable a collaborative session.
configuration?.isCollaborationEnabled = true
// Enable realistic reflections.
configuration?.environmentTexturing = .automatic
// Begin the session.
arView.session.run(configuration!)
// Use key-value observation to monitor your ARSession's identifier.
sessionIDObservation = observe(\.arView.session.identifier, options: [.new]) { object, change in
print("SessionID changed to: \(change.newValue!)")
// Tell all other peers about your ARSession's changed ID, so
// that they can keep track of which ARAnchors are yours.
guard let multipeerSession = self.multipeerSession else { return }
self.sendARSessionIDTo(peers: multipeerSession.connectedPeers)
}
// Start looking for other players via MultiPeerConnectivity.
multipeerSession = MultipeerSession(receivedDataHandler: receivedData, peerJoinedHandler:
peerJoined, peerLeftHandler: peerLeft, peerDiscoveredHandler: peerDiscovered)
//arView.addGestureRecognizer(UITapGestureRecognizer(target: self, action: #selector(handleTap(recognizer:))))
messageLabel.displayMessage("Tap the screen to place cubes.\nInvite others to launch this app to join you.", duration: 60.0)
}
//peerDiscovered
func peerDiscovered(_ peer: MCPeerID) -> Bool {
guard let multipeerSession = multipeerSession else { return false }
if multipeerSession.connectedPeers.count > 3 {
// Do not accept more than four users in the experience.
messageLabel.displayMessage("A fifth peer wants to join the experience.\nThis app is limited to four users.", duration: 6.0)
return false
} else {
return true
}
}
// end of added Sat May 28 5:12pm
/// - Tag: PeerJoined
// added Sat May 28 5:12pm
func peerJoined(_ peer: MCPeerID) {
messageLabel.displayMessage("""
A peer has joined the experience.
Hold the phones next to each other.
""", duration: 6.0)
// Provide your session ID to the new user so they can keep track of your anchors.
sendARSessionIDTo(peers: [peer])
}
// end of added Sat May 28 5:12pm
// added Sat May 28 5:12pm
func peerLeft(_ peer: MCPeerID) {
messageLabel.displayMessage("A peer has left the shared experience.")
// Remove all ARAnchors associated with the peer that just left the experience.
if let sessionID = peerSessionIDs[peer] {
removeAllAnchorsOriginatingFromARSessionWithID(sessionID)
peerSessionIDs.removeValue(forKey: peer)
}
}
// end of added Sat May 28 5:12pm
//added Sat May 28 5:12pm
func receivedData(_ data: Data, from peer: MCPeerID) {
if let collaborationData = try? NSKeyedUnarchiver.unarchivedObject(ofClass: ARSession.CollaborationData.self, from: data) {
arView.session.update(with: collaborationData)
return
}
// ...
let sessionIDCommandString = "SessionID:"
if let commandString = String(data: data, encoding: .utf8), commandString.starts(with: sessionIDCommandString) {
let newSessionID = String(commandString[commandString.index(commandString.startIndex,
offsetBy: sessionIDCommandString.count)...])
// If this peer was using a different session ID before, remove all its associated anchors.
// This will remove the old participant anchor and its geometry from the scene.
if let oldSessionID = peerSessionIDs[peer] {
removeAllAnchorsOriginatingFromARSessionWithID(oldSessionID)
}
peerSessionIDs[peer] = newSessionID
}
}
// end of added Sat May 28 5:12pm
func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
messageLabel.displayMessage("Established joint experience with a peer.")
// ...
}
func updateScene(on event: SceneEvents.Update) {
let notesToUpdate = stickyNotes.compactMap { !$0.isEditing && !$0.isDragging ? $0 : nil }
for note in notesToUpdate {
// Gets the 2D screen point of the 3D world point.
guard let projectedPoint = arView.project(note.position) else { return }
// Calculates whether the note can be currently visible by the camera.
let cameraForward = arView.cameraTransform.matrix.columns.2.xyz
let cameraToWorldPointDirection = normalize(note.transform.translation - arView.cameraTransform.translation)
let dotProduct = dot(cameraForward, cameraToWorldPointDirection)
let isVisible = dotProduct < 0
// Updates the screen position of the note based on its visibility
note.projection = Projection(projectedPoint: projectedPoint, isVisible: isVisible)
note.updateScreenPosition()
}
}
func reset() {
guard let configuration = arView.session.configuration else { return }
arView.session.run(configuration, options: .removeExistingAnchors)
for note in stickyNotes {
deleteStickyNote(note)
}
}
func session(_ session: ARSession, didFailWithError error: Error) {
guard error is ARError else { return }
let errorWithInfo = error as NSError
let messages = [
errorWithInfo.localizedDescription,
errorWithInfo.localizedFailureReason,
errorWithInfo.localizedRecoverySuggestion
]
let errorMessage = messages.compactMap({ $0 }).joined(separator: "\n")
DispatchQueue.main.async {
// Present an alert informing about the error that has occurred.
let alertController = UIAlertController(title: "The AR session failed.", message: errorMessage, preferredStyle: .alert)
let restartAction = UIAlertAction(title: "Restart Session", style: .default) { _ in
alertController.dismiss(animated: true, completion: nil)
self.reset()
}
alertController.addAction(restartAction)
self.present(alertController, animated: true, completion: nil)
}
}
override var prefersStatusBarHidden: Bool {
return true
}
override var prefersHomeIndicatorAutoHidden: Bool {
return true
}
private func sendARSessionIDTo(peers: [MCPeerID]) {
guard let multipeerSession = multipeerSession else { return }
let idString = arView.session.identifier.uuidString
let command = "SessionID:" + idString
if let commandData = command.data(using: .utf8) {
multipeerSession.sendToPeers(commandData, reliably: true, peers: peers)
}
}
private func removeAllAnchorsOriginatingFromARSessionWithID(_ identifier: String) {
guard let frame = arView.session.currentFrame else { return }
for anchor in frame.anchors {
guard let anchorSessionID = anchor.sessionIdentifier else { continue }
if anchorSessionID.uuidString == identifier {
arView.session.remove(anchor: anchor)
}
}
}
}
Update: I spoke to a Staff Engineer on Apple's RealityKit team who explained to me that what I was trying to accomplish is not feasible because the note had an embedded subclass that is not 'codable' as per Swift's Codable Protocol
I will have to rebuild the note differently than the example i had been working with to ensure it fits within the Codable protocol which will then ensure the data can travel across the network via Multipeer Connectivity Framework.
I'm trying (unsuccessfully) to get the node from a disclosure button clicked
I think this function is the more appropriate:
func outlineViewItemDidExpand(_ notification: Notification) {
let nodeToExpand = notification.userInfo as! Node
let nodeToExpand2 = notification.userInfo["NSObject"] as! Node
//Error #selector(_outlineControlClicked:) from sender NSButton 0x10053d710
}
NSTreeController wraps your nodes in NSTreeNode objects. Your node is representedObject of the NSTreeNode.
if let treeNode = notification.userInfo["NSObject"] as? NSTreeNode,
let node = treeNode.representedObject as? Node {
…
}
I currently have multiple balls all on the same SKScene. I handle all touches and gestures within GameScene. Below is the code I use to detect which node was touched, which works.
What I am unsure of, since there are always some touchesMoved when using this on a real device, is there anyway possible for more than one node to receive a tap at the same time? If so I obviously would need to write my code differently.
#objc func tappedView(_ sender:UITapGestureRecognizer) {
if sender.state == .ended{
let point : CGPoint = sender.location(in: self.view)
var post = sender.location(in: sender.view)
post = self.convertPoint(fromView: post)
if let touchNode = self.atPoint(post) as? MyBall{
//the declaration below is just so I have somewhere to stop in the debugger
var x = 1
}
}
}
Use nodes(at:) to get multiple nodes at a point.
#objc func tappedView(_ sender:UITapGestureRecognizer) {
if sender.state == .ended{
let point : CGPoint = sender.location(in: self.view)
var post = sender.location(in: sender.view)
post = self.convertPoint(fromView: post)
for touchNode in self.nodes(at:post){
//the declaration below is just so I have somewhere to stop in the debugger
var x = 1
}
}
}
https://developer.apple.com/documentation/spritekit/sknode/1483072-nodes
I would like to get the location point when I tap somewhere on an image. The issue is depending on which device/resolution (iphones, ipads, etc..) I don't get the same coordinate.
I'm looking at a way to get the same location regardless of the devices/resolution.
#IBAction func tapGestureAction(_ sender: Any) {
guard let gestureView = tapGesture.view else {
return
}
let tappedLocationPoint = tapGesture.location(in: gestureView)
print(tappedLocationPoint.x)
print(tappedLocationPoint.y)
}
I am new to swift and I can't figure out how to handle closures and closures concept.
I recently asked question and I find out that my variable is nil because geocodeAddressString runs asynchronously so app printing latLong well before this property was eventually set.
But here is new question that I can't understand:
import UIKit
import CoreLocation
import Firebase
var latLong: String!
override func viewDidLoad() {
super.viewDidLoad()
}
func findCordiante(adress:String){
let geocoder = CLGeocoder()
geocoder.geocodeAddressString(adress) {
placemarks, error in
if (placemarks != nil){
let placemark = placemarks?.first
let lat = placemark?.location?.coordinate.latitude
let lon = placemark?.location?.coordinate.longitude
self.latLong = String(describing: lat!) + "," + String(describing: lon!)
}else{
//handle no adress
self.latLong = ""
}
}
}
#IBAction func createSchool(_ sender: Any) {
//when user press button i want execute function and assign value to variable latLong
findCordiante(adress: "Cupertino, California, U.S.")
//so then I need to manipulate with that value here, let's say example
//example
print("Hi user, your coordinates is \(latLong)")
}
When I add print(latLong) inside closure it is printing, but I DONT WANT to do all functionality inside closure.
Simply I WANT to add result of func findCordiante() to variable latLong, so after that I can manipulate with that variable everywhere inside class
The main thing to understand is that resolving an address into coordinates (and many other geolocation operations) take time and therefore return a result with considerable delay. During the delay, the app continues to run and should be responsive to user actions.
That's the reason why closures are used, namely to split the operation into two parts:
Start the operation (your findCoordinate function)
Complete the action after the operation has finished (the closure, used as a callback)
Between these two parts, your application runs normally. It does not wait or block. If you want a waiting behaviour, you have to implement it yourself (e.g. disable buttons, ignore user gestures etc.(.
You can easily move part of the code within the closure into a separate function:
func findCordiante(adress:String){
let geocoder = CLGeocoder()
geocoder.geocodeAddressString(adress) {
placemarks, error in
if let placemarks = placemarks {
self.findCoordinateCompleted(placemarks)
} else {
self.findCoordinateFailed()
}
}
}
func findCoordinateCompleted(placemarks: [CLPlacemark]) {
let placemark = placemarks.first!
let lat = placemark.location!.coordinate.latitude
let lon = placemark.location!.coordinate.longitude
latLong = String(describing: lat) + "," + String(describing: lon)
completeCreatingSchool()
}
func findCoordinateFailed() {
latLong = ""
print("Hi user, invalid address")
// do more stuff here
}
#IBAction func createSchool(_ sender: Any) {
findCoordinate(adress: "Cupertino, California, U.S.")
}
func completeCreatingSchool() {
//example
print("Hi user, your coordinates is \(latLong)")
}
After you set latLong in the closure, it will be available to the rest of your class. The only problem you have is if createSchool gets called before the closure is done.
Solution to this is to have the button and/or menu item that points to createSchool start out disabled. You then enable it after the closure finishes.
The closest solution according to you code is to use a completion handler and return a boolean value success in the closure. Then you can use the latLong variable (wouldn't be CLLocationCoordinate2D the better type?) right after setting it.
func findCoordinate(adress:String, completion: #escaping (Bool)->()){
let geocoder = CLGeocoder()
geocoder.geocodeAddressString(adress) {
placemarks, error in
if (placemarks != nil){
let placemark = placemarks?.first
let lat = placemark?.location?.coordinate.latitude
let lon = placemark?.location?.coordinate.longitude
self.latLong = String(describing: lat!) + "," + String(describing: lon!)
completion(true)
}else{
//handle no adress
self.latLong = ""
completion(false)
}
}
}
#IBAction func createSchool(_ sender: Any) {
//when user press button i want execute function and assign value to variable latLong
findCoordinate(adress: "Cupertino, California, U.S.") { success in
//so then I need to manipulate with that value here, let's say example
//example
if success {
print("Hi user, your coordinates is \(latLong)")
}
}
}