I was following Apple Documentation and example project to load 3d Object using .SCN file with Virtual Object (subclass of SCNReferenceNode) class but suddenly i needed to change the model from .scn to usdz . Now my usdz object is loading successfully but it is not on surface (midway in the air) and i can't interact with it like (tap , pan , rotate) ... Is there any other way to get interaction with usdz object and how can I place it on the surface like I was doing it before with .scn file
For getting model URL (downloaded from server)
static let loadDownloadedModel : VirtualObject = {
let downloadedScenePath = getDocumentsDirectory().appendingPathComponent("\(Api.Params.inputModelName).usdz")
return VirtualObject(url: downloadedScenePath)!
}()
Loading it from URL
func loadVirtualObject(_ object: VirtualObject, loadedHandler: #escaping (VirtualObject) -> Void) {
isLoading = true
loadedObjects.append(object)
// Load the content asynchronously.
DispatchQueue.global(qos: .userInitiated).async {
object.reset()
object.load()
self.isLoading = false
loadedHandler(object)
}
}
Placing in the scene
func placeObjectOnFocusSquare() {
virtualObjectLoader.loadVirtualObject(VirtualObject.loadDownloadedModel) { (loadedObject) in
DispatchQueue.main.async {
self.placeVirtualObject(loadedObject)
self.setupBottomButtons(isSelected: true)
}
}
}
func placeVirtualObject(_ virtualObject: VirtualObject) {
guard let cameraTransform = session.currentFrame?.camera.transform,
let focusSquarePosition = focusSquare.lastPosition else {
statusViewController.showMessage("CANNOT PLACE OBJECT\nTry moving left or right.")
return
}
Api.Params.selectedModel = virtualObject
virtualObject.name = String(Api.Params.inputPreviewId)
virtualObject.scale = SCNVector3(Api.Params.modelXAxis, Api.Params.modelYAxis, Api.Params.modelZAxis)
virtualObject.setPosition(focusSquarePosition, relativeTo: cameraTransform, smoothMovement: false)
updateQueue.async {
self.sceneView.scene.rootNode.addChildNode(virtualObject)
}
}
.usdz object in sceneview
After so many tries , finally i found out that dynamic scaling of the model causing problem , Reference to this
iOS ARKit: Large size object always appears to move with the change in the position of the device camera
I scaled the object to 0.01 for all the axis (x,y and z)
virtualObject.scale = SCNVector3Make(0.01, 0.01, 0.01)
Related
I have quite a specific problem but hopefully someone can help me. I'm using AVFoundation to create a video camera with a live preview. I use AVCaptureVideoDataOutput to get individual frames and AVCaptureMetadataOutput to detect a face. I'm also using Dlib's facial landmarks predictor to show the landmark points on the users face and measure the interocular distance between their eyes. Finally I'm using AVAssetWriter so that a video can be recorded.
The view controller has an ellipse shape on it so the user knows where to put their face. When the interocular distance is between a certain distance I want the ellipse to turn blue so the user knows their face is in the right place.
At the minute I've achieved this by sending a notification from my SessionHandler class to the View Controller. This works, however it's causing the frames per second in the video to drop badly. I was getting 25fps (manually set by me) and now it's ranging between 8-16.
Is there another way to notify the view controller that the ellipse should be turned green?
Here's my code where the problem is occurring. I know there's a lot going on.
// MARK: AVCaptureVideoDataOutputSampleBufferDelegate, AVCaptureAudioOutputSampleBufferDelegate
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
if !currentMetadata.isEmpty {
let boundsArray = currentMetadata
.compactMap { $0 as? AVMetadataFaceObject }
.map { (faceObject) -> NSValue in
let convertedObject = output.transformedMetadataObject(for: faceObject, connection: connection)
return NSValue(cgRect: convertedObject!.bounds)
}
if user.hasDlib {
wrapper?.doWork(on: sampleBuffer, inRects: boundsArray)
// Get the interocular distance so face is the correct place in the oval
let interocularDistance = wrapper?.getEyeDistance()
//print("Interocular Distance: \(interocularDistance)")
if user.hasInterocularDistance {
if interocularDistance! < 240 || interocularDistance! > 315 {
let name = Notification.Name(rawValue: setRemoveGreenEllipse)
NotificationCenter.default.post(name: name, object: nil)
//print("face not correct distance")
if videoRecorder.isRecording {
eyeDistanceCounter += 1
//print(eyeDistanceCounter)
if eyeDistanceCounter == 30 {
cancelledByUser = false
cancelledByEyeDistance = true
videoRecorder.cancel()
eyeDistanceCounter = 0
}
}
} else {
//print("face correct distance")
eyeDistanceCounter = 0
let name = Notification.Name(rawValue: setGreenEllipse)
NotificationCenter.default.post(name: name, object: nil)
}
}
}
} else {
// Check if face is detected during recording. If it isn't, then cancel recording
if videoRecorder.isRecording {
noFaceCount += 1
if noFaceCount == 50 {
cancelledByUser = false
videoRecorder.cancel()
noFaceCount = 0
}
}
}
if layer.status == .failed {
layer.flush()
}
layer.enqueue(sampleBuffer)
let writable = videoRecorder.canWrite()
if writable {
if videoRecorder.sessionAtSourceTime == nil {
// Start Writing
videoRecorder.sessionAtSourceTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
videoRecorder.videoWriter.startSession(atSourceTime: videoRecorder.sessionAtSourceTime!)
print("video session started")
}
if videoRecorder.videoWriterInput.isReadyForMoreMediaData {
// write video buffer
videoRecorder.videoWriterInput.append(sampleBuffer)
//print("video buffering")
}
}
}
You could probably call the notification once per 30 frames, for example, instead of every frame.
You could also call the color changing function directly if it's in the same view controller. If not, you could define a delegate method and call it directly as oppose to sending notifications.
Ok, I am working with ARKit and SceneKit here and I am having trouble looking at the other questions dealing with just SceneKit trying to have a model in .dae format and load in various animations to have that model run - now that we're in iOS11 seems that some solutions don't work.
Here is how I get my model - from a base .dae scene where no animations are applied. I am importing these with Maya -
var modelScene = SCNScene(named: "art.scnassets/ryderFinal3.dae")!
if let d = modelScene.rootNode.childNodes.first {
theDude.node = d
theDude.setupNode()
}
Then in Dude class:
func setupNode() {
node.scale = SCNVector3(x: modifier, y: modifier, z: modifier)
center(node: node)
}
the scaling and centering of axes is needing because my model was just not at the origin. That worked. Then now with a different scene called "Idle.dae" I try to load in an animation to later run on the model:
func animationFromSceneNamed(path: String) -> CAAnimation? {
let scene = SCNScene(named: path)
var animation:CAAnimation?
scene?.rootNode.enumerateChildNodes({ child, stop in
if let animKey = child.animationKeys.first {
animation = child.animation(forKey: animKey)
stop.pointee = true
}
})
return animation
}
I was going to do this for all my animations scenes that I import into Xcode and store all the animations in
var animations = [CAAnimation]()
First Xcode says animation(forKey: is deprecated and This does not work it seems to (from what I can tell) de-center and de-scale my model back to the huge size it was. It screws up its position because I expect making the model move in an animation, for example, would make the instantiated model in my game snap to that same position.
and other attempts cause crashes. I am very new to scene kit and trying to get a grip on how to properly animate a .dae model that I instantiate anywhere in the scene -
How in iOS11 does one load in an array of animations to apply to their SCNNode?
How do you make it so those animations are run on the model WHEREVER THE MODEL IS (not snapping it to some other position)?
At first I should confirm that CoreAnimation framework and some of its methods like animation(forKey:) instance method are really deprecated in iOS and macOS. But some parts of CoreAnimation framework are now implemented into SceneKit and other modules. In iOS 11+ and macOS 10.13+ you can use SCNAnimation class:
let animation = CAAnimation(scnAnimation: SCNAnimation)
and here SCNAnimation class has three useful initializers:
SCNAnimation(caAnimation: CAAnimation)
SCNAnimation(contentsOf: URL)
SCNAnimation(named: String)
In addition I should add that you can use not only animations baked in .dae file format, but also in .abc, .scn and .usdz.
Also, you can use SCNSceneSource class (iOS 8+ and macOS 10.8+) to examine the contents of a SCNScene file or to selectively extract certain elements of a scene without keeping the entire scene and all the assets it contains.
Here's how a code with implemented SCNSceneSource might look like:
#IBOutlet var sceneView: ARSCNView!
var animations = [String: CAAnimation]()
var idle: Bool = true
override func viewDidLoad() {
super.viewDidLoad()
sceneView.delegate = self
let scene = SCNScene()
sceneView.scene = scene
loadMultipleAnimations()
}
func loadMultipleAnimations() {
let idleScene = SCNScene(named: "art.scnassets/model.dae")!
let node = SCNNode()
for child in idleScene.rootNode.childNodes {
node.addChildNode(child)
}
node.position = SCNVector3(0, 0, -5)
node.scale = SCNVector3(0.45, 0.45, 0.45)
sceneView.scene.rootNode.addChildNode(node)
loadAnimation(withKey: "walking",
sceneName: "art.scnassets/walk_straight",
animationIdentifier: "walk_version02")
}
...
func loadAnimation(withKey: String, sceneName: String, animationIdentifier: String) {
let sceneURL = Bundle.main.url(forResource: sceneName, withExtension: "dae")
let sceneSource = SCNSceneSource(url: sceneURL!, options: nil)
if let animationObj = sceneSource?.entryWithIdentifier(animationIdentifier,
withClass: CAAnimation.self) {
animationObj.repeatCount = 1
animationObj.fadeInDuration = CGFloat(1)
animationObj.fadeOutDuration = CGFloat(0.5)
animations[withKey] = animationObj
}
}
...
func playAnimation(key: String) {
sceneView.scene.rootNode.addAnimation(animations[key]!, forKey: key)
}
I'm actually trying to put a 3D Object on QRCode with ARKit
For that I use a AVCaptureDevice to detect a QRCode and establish the area of the QRCode that gives me a CGRect.
Then, I make a hitTest on every point of the CGRect to get the average 3D coordinates like so :
positionGiven = SCNVector3(0, 0, 0)
for column in Int(qrZone.origin.x)...2*Int(qrZone.origin.x + qrZone.width) {
for row in Int(qrZone.origin.y)...2*Int(qrZone.origin.y + qrZone.height) {
for result in sceneView.hitTest(CGPoint(x: CGFloat(column)/2,y:CGFloat(row)/2), types: [.existingPlaneUsingExtent,.featurePoint]) {
positionGiven.x+=result.worldTransform.columns.3.x
positionGiven.y+=result.worldTransform.columns.3.y
positionGiven.z+=result.worldTransform.columns.3.z
cpts += 1
}
}
}
positionGiven.x=positionGiven.x/cpts
positionGiven.y=positionGiven.y/cpts
positionGiven.z=positionGiven.z/cpts
But the hitTest doesn't detect any result and freeze the camera while when I make a hitTest with a touch on screen it works.
Do you have any idea why it's not working ?
Do you have an other idea that can help me to achieve what I want to do ?
I already thought about 3D translation with CoreMotion that can give me the tilt of the device but that seems really tedious.
I also heard about ARWorldAlignmentCamera that can locked the scene coordinate to match the orientation of the camera but I don't know how to use it !
Edit : I try to move my 3D Object every time I touch the screen and the hitTest is positive, and it's pretty accurate ! I really don't understand why hitTest on an area of pixels doesn't work...
Edit 2 : Here is the code of the hitTest who works with 2-5 touches on the screen:
#objc func touch(sender : UITapGestureRecognizer) {
for result in sceneView.hitTest(CGPoint(x: sender.location(in: view).x,y: sender.location(in: view).y), types: [.existingPlaneUsingExtent,.featurePoint]) {
//Pop up message for testing
alert("\(sender.location(in: view))", message: "\(result.worldTransform.columns.3)")
//Moving the 3D Object to the new coordinates
let objectList = sceneView.scene.rootNode.childNodes
for object : SCNNode in objectList {
object.removeFromParentNode()
}
addObject(SCNVector3(result.worldTransform.columns.3.x,result.worldTransform.columns.3.y,result.worldTransform.columns.3.z))
}
}
Edit 3 :
I manage to resolve my problem partially.
I take the transform matrix of the camera (session.currentFrame.camera.transform) so that the object is in front of the camera.
Then I apply a translation on (x,y) with the position of the CGRect.
However i can't translate the z-axis because i don't have enough informations.
And I will probably need a estimation of z coordinate like the hitTest do..
Thanks in advance ! :)
You could use Apple's Vision API to detect the QR code and place an anchor.
To start detecting QR codes, use:
var qrRequests = [VNRequest]()
var detectedDataAnchor: ARAnchor?
var processing = false
func startQrCodeDetection() {
// Create a Barcode Detection Request
let request = VNDetectBarcodesRequest(completionHandler: self.requestHandler)
// Set it to recognize QR code only
request.symbologies = [.QR]
self.qrRequests = [request]
}
In ARSession's didUpdate Frame
public func session(_ session: ARSession, didUpdate frame: ARFrame) {
DispatchQueue.global(qos: .userInitiated).async {
do {
if self.processing {
return
}
self.processing = true
// Create a request handler using the captured image from the ARFrame
let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: frame.capturedImage,
options: [:])
// Process the request
try imageRequestHandler.perform(self.qrRequests)
} catch {
}
}
}
Handle the Vision QR request and trigger the hit test
func requestHandler(request: VNRequest, error: Error?) {
// Get the first result out of the results, if there are any
if let results = request.results, let result = results.first as? VNBarcodeObservation {
guard let payload = result.payloadStringValue else {return}
// Get the bounding box for the bar code and find the center
var rect = result.boundingBox
// Flip coordinates
rect = rect.applying(CGAffineTransform(scaleX: 1, y: -1))
rect = rect.applying(CGAffineTransform(translationX: 0, y: 1))
// Get center
let center = CGPoint(x: rect.midX, y: rect.midY)
DispatchQueue.main.async {
self.hitTestQrCode(center: center)
self.processing = false
}
} else {
self.processing = false
}
}
func hitTestQrCode(center: CGPoint) {
if let hitTestResults = self.latestFrame?.hitTest(center, types: [.featurePoint] ),
let hitTestResult = hitTestResults.first {
if let detectedDataAnchor = self.detectedDataAnchor,
let node = self.sceneView.node(for: detectedDataAnchor) {
let previousQrPosition = node.position
node.transform = SCNMatrix4(hitTestResult.worldTransform)
} else {
// Create an anchor. The node will be created in delegate methods
self.detectedDataAnchor = ARAnchor(transform: hitTestResult.worldTransform)
self.sceneView.session.add(anchor: self.detectedDataAnchor!)
}
}
}
Then handle adding the node when the anchor is added.
func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
// If this is our anchor, create a node
if self.detectedDataAnchor?.identifier == anchor.identifier {
let sphere = SCNSphere(radius: 1.0)
sphere.firstMaterial?.diffuse.contents = UIColor.redColor()
let sphereNode = SCNNode(geometry: sphere)
sphereNode.transform = SCNMatrix4(anchor.transform)
return sphereNode
}
return nil
}
Source
I have the following code in my main.swift:
let strategist = GKMinmaxStrategist()
strategist.gameModel = position
strategist.maxLookAheadDepth = 1
strategist.randomSource = nil
let move = strategist.bestMoveForActivePlayer()
...where position is an instance of my GKGameModel subclass Position. After this code is run, move is nil. bestMoveForPlayer(position.activePlayer!) also results in nil (but position.activePlayer! results in a Player object).
However,
let moves = position.gameModelUpdatesForPlayer(position.activePlayer!)!
results in a non-empty array of possible moves. From Apple's documentation (about bestMoveForPlayer(_:)):
Returns nil if the player is invalid, the player is not a part of the game model, or the player has no valid moves available.
As far as I know, none of this is the case, but the function still returns nil. What could be going on here?
If it can be of any help, here's my implementation of the GKGameModel protocol:
var players: [GKGameModelPlayer]? = [Player.whitePlayer, Player.blackPlayer]
var activePlayer: GKGameModelPlayer? {
return playerToMove
}
func setGameModel(gameModel: GKGameModel) {
let position = gameModel as! Position
pieces = position.pieces
ply = position.ply
reloadLegalMoves()
}
func gameModelUpdatesForPlayer(thePlayer: GKGameModelPlayer) -> [GKGameModelUpdate]? {
let player = thePlayer as! Player
let moves = legalMoves(ofPlayer: player)
return moves.count > 0 ? moves : nil
}
func applyGameModelUpdate(gameModelUpdate: GKGameModelUpdate) {
let move = gameModelUpdate as! Move
playMove(move)
}
func unapplyGameModelUpdate(gameModelUpdate: GKGameModelUpdate) {
let move = gameModelUpdate as! Move
undoMove(move)
}
func scoreForPlayer(thePlayer: GKGameModelPlayer) -> Int {
let player = thePlayer as! Player
var score = 0
for (_, piece) in pieces {
score += piece.player == player ? 1 : -1
}
return score
}
func isLossForPlayer(thePlayer: GKGameModelPlayer) -> Bool {
let player = thePlayer as! Player
return legalMoves(ofPlayer: player).count == 0
}
func isWinForPlayer(thePlayer: GKGameModelPlayer) -> Bool {
let player = thePlayer as! Player
return isLossForPlayer(player.opponent)
}
func copyWithZone(zone: NSZone) -> AnyObject {
let copy = Position(withPieces: pieces.map({ $0.1 }), playerToMove: playerToMove)
copy.setGameModel(self)
return copy
}
If there's any other code I should show, let me know.
You need to change the activePlayer after apply or unapply a move.
In your case that would be playerToMove.
The player whose turn it is to perform an update to the game model. GKMinmaxStrategist assumes that the next call to the applyGameModelUpdate: method will perform a move on behalf of this player.
and, of course:
Function applyGameModelUpdate Applies a GKGameModelUpdate to the game model, potentially resulting in a new activePlayer. GKMinmaxStrategist will call this method on a copy of the primary game model to speculate about possible future moves and their effects. It is assumed that calling this method performs a move on behalf of the player identified by the activePlayer property.
func applyGameModelUpdate(gameModelUpdate: GKGameModelUpdate) {
let move = gameModelUpdate as! Move
playMove(move)
//Here change the current Player
let player = playerToMove as! Player
playerToMove = player.opponent
}
The same goes for your unapplyGameModelUpdate implementation.
Also, keep special attention to your setGameModelimplementation as it should copy All data in your model. This includes activePlayer
Sets the data of another game model. All data should be copied over, and should not maintain any pointers to the copied game state. This is used by the GKMinmaxStrategist to process permutations of the game without needing to apply potentially destructive updates to the primary game model.
I had the same problem. Turns out, .activePlayer has to return one of the instances returned by .players. It's not enough to return a new instance with matching .playerId.
simple checklist:
GKMinmaxStrategist's .bestMove(for:) is called
GKMinmaxStrategist's .gameModel is set
GKGameModel's isWin(for:) does not return true before move
GKGameModel's isLoss(for:) does not return true before move
GKGameModel's gameModelUpdates(for:) does not return nil all the time
GKGameModel's score(for:) is implemented
I'm attempting to add support for Voice Over accessibility in a puzzle game which has a fixed board. However, I'm having trouble getting UIAccessibilityElements to show up.
Right now I'm overriding accessibilityElementAtIndex, accessibilityElementCount and indexOfAccessibilityElement in my SKScene.
They are returning an array of accessible elements as such:
func loadAccessibleElements()
{
self.isAccessibilityElement = false
let pieces = getAllPieces()
accessibleElements.removeAll(keepCapacity: false)
for piece in pieces
{
let element = UIAccessibilityElement(accessibilityContainer: self.usableView!)
element.accessibilityFrame = piece.getAccessibilityFrame()
element.accessibilityLabel = piece.getText()
element.accessibilityTraits = UIAccessibilityTraitButton
accessibleElements.append(element)
}
}
Where piece is a subclass of SKSpriteNode and getAccessibilityFrame is defined:
func getAccessibilityFrame() -> CGRect
{
return parentView!.convertRect(frame, toView: nil)
}
Right now one (wrongly sized) accessibility element seems to appear on the screen in the wrong place.
Could someone point me in the right direction?
Many thanks
EDIT:
I've tried a hack-ish work around by placing a UIView over the SKView with UIButton elements in the same location as the SKSpriteNodes. However, accessibility still doesn't want to work. The view is loaded as such:
func loadAccessibilityView()
{
view.isAccessibilityElement = false
view.accessibilityElementsHidden = false
skView.accessibilityElementsHidden = false
let accessibleSubview = UIView(frame: view.frame)
accessibleSubview.userInteractionEnabled = true
accessibleSubview.isAccessibilityElement = false
view.addSubview(accessibleSubview)
view.bringSubviewToFront(accessibleSubview)
let pieces = (skView.scene! as! GameScene).getAllPieces()
for piece in pieces
{
let pieceButton = UIButton(frame: piece.getAccessibilityFrame())
pieceButton.isAccessibilityElement = true
pieceButton.accessibilityElementsHidden = false
pieceButton.accessibilityTraits = UIAccessibilityTraitButton
pieceButton.setTitle(piece.getText(), forState: UIControlState.Normal)
pieceButton.setBackgroundImage(UIImage(named: "blue-button"), forState: UIControlState.Normal)
pieceButton.alpha = 0.2
pieceButton.accessibilityLabel = piece.getText()
pieceButton.accessibilityFrame = pieceButton.frame
pieceButton.addTarget(self, action: Selector("didTap:"), forControlEvents: UIControlEvents.TouchUpInside)
accessibleSubview.addSubview(pieceButton)
}
UIAccessibilityPostNotification(UIAccessibilityScreenChangedNotification, nil)
}
The buttons are placed correctly, however accessibility just isn't working at all. Something seems to be preventing it from working.
I've searched in vain for a description of how to implement VoiceOver in Swift using SpriteKit, so I finally figured out how to do it. Here's some working code that converts a SKNode to an accessible pushbutton when added to a SKScene class:
// Add the following code to a scene where you want to make the SKNode variable named “leave” an accessible button
// leave must already be initialized and added as a child of the scene, or a child of other SKNodes in the scene
// screenHeight must already be defined as the height of the device screen, in points
// Accessibility
private var accessibleElements: [UIAccessibilityElement] = []
private func nodeToDevicePointsFrame(node: SKNode) -> CGRect {
// first convert from frame in SKNode to frame in SKScene's coordinates
var sceneFrame = node.frame
sceneFrame.origin = node.scene!.convertPoint(node.frame.origin, fromNode: node.parent!)
// convert frame from SKScene coordinates to device points
// sprite kit scene origin is in lower left, accessibility device screen origin is at upper left
// assumes scene is initialized using SKSceneScaleMode.Fill using dimensions same as device points
var deviceFrame = sceneFrame
deviceFrame.origin.y = CGFloat(screenHeight-1) - (sceneFrame.origin.y + sceneFrame.size.height)
return deviceFrame
}
private func initAccessibility() {
if accessibleElements.count == 0 {
let accessibleLeave = UIAccessibilityElement(accessibilityContainer: self.view!)
accessibleLeave.accessibilityFrame = nodeToDevicePointsFrame(leave)
accessibleLeave.accessibilityTraits = UIAccessibilityTraitButton
accessibleLeave.accessibilityLabel = “leave” // the accessible name of the button
accessibleElements.append(accessibleLeave)
}
}
override func didMoveToView(view: SKView) {
self.isAccessibilityElement = false
leave.isAccessibilityElement = true
}
override func willMoveFromView(view: SKView) {
accessibleElements = []
}
override func accessibilityElementCount() -> Int {
initAccessibility()
return accessibleElements.count
}
override func accessibilityElementAtIndex(index: Int) -> AnyObject? {
initAccessibility()
if (index < accessibleElements.count) {
return accessibleElements[index] as AnyObject
} else {
return nil
}
}
override func indexOfAccessibilityElement(element: AnyObject) -> Int {
initAccessibility()
return accessibleElements.indexOf(element as! UIAccessibilityElement)!
}
Accessibility frames are defined in the fixed physical screen coordinates, not UIView coordinates, and transforming between them is kind of tricky.
The device origin is the lower left of the screen, with X up, when the device is in landscape right mode.
It's a pain converting, I've no idea why Apple did it that way.