Converting UIKit to SceneKit - swift

Trouble converting program from UIKit to SceneKit. Biggest difficulty for me is understanding how delegate file, Tile, synched with array, Board, is set up with SceneKit. It is a simple project. A screenshot: http://imgur.com/9hsv7X5. It displays a 3 x 5 array. User taps an item and it becomes highlighted. Then tap another item, it becomes highlighted, previous item, unhighlighted.
Here is the UIKit project composed of 3 files:
VIEWCONTROLLER
import UIKit
struct BoardLoc {
var x: Int
var y: Int
}
class ViewController: UIViewController, TileDelegate {
var tile: Tile!
override func viewDidLoad() {
super.viewDidLoad()
let scene = Board()
tile.tileDelegate = self
tile.board = scene
}
func getTileAtLoc(tile: Tile, _ boardLoc: BoardLoc) {
tile.boardLoc = boardLoc
}
}
BOARD
import Foundation
class Board {
var board: Array<Array<String>> = Array(count:3, repeatedValue:Array(count:5, repeatedValue:"foo"))
func putTileAt(boardLoc: BoardLoc) -> String {
return board[boardLoc.x][boardLoc.y]
}
}
TILE
import UIKit
protocol TileDelegate {
func getTileAtLoc(tile: Tile, _ boardLoc: BoardLoc)
}
class Tile: UIView {
var boardLoc: BoardLoc?
var board: Board?
var tileDelegate: TileDelegate?
required init(coder aDecoder: NSCoder) {
super.init(coder: aDecoder)
addGestureRecognizer(UITapGestureRecognizer(target: self, action:"handleTap:"))
}
override func drawRect(rect: CGRect) {
for x in 0...2 {
for y in 0...4 {
let context = UIGraphicsGetCurrentContext()
let red = UIColor.redColor().CGColor
let orange = UIColor.orangeColor().CGColor
let bigCircle = CGRectMake(CGFloat(106 * x),CGFloat(106 * y), 106, 106)
let smallCircle = CGRectMake(CGFloat(106 * x) + 3, CGFloat(106 * y) + 3, 100, 100)
if (boardLoc != nil && boardLoc!.x == x && boardLoc!.y == y) {
CGContextSetFillColorWithColor(context, red)
CGContextFillEllipseInRect(context, bigCircle)
}
if board!.putTileAt(BoardLoc(x: x, y: y)) == "foo" {
CGContextSetFillColorWithColor(context, orange)
CGContextFillEllipseInRect(context, smallCircle)
}
}
}
}
func handleTap(gestureRecognizer: UIGestureRecognizer) {
let point = gestureRecognizer.locationInView(self)
let boardLoc = BoardLoc(x: Int(point.x) / 106, y: Int(point.y) / 106)
tileDelegate!.getTileAtLoc(self, boardLoc)
setNeedsDisplay()
}
}

First of all, I recommend you to read Apple SceneKit document and some tutorials.
Scene Kit is a 3D-rendering Objective-C framework that combines a high-performance rendering engine with a high-level, descriptive API. Scene Kit supports the import, manipulation, and rendering of 3D assets without requiring the exact steps to render a scene the way OpenGL does.
http://www.objc.io/issue-18/scenekit.html
https://www.weheartswift.com/introduction-scenekit-part-1/
http://www.raywenderlich.com/83748/beginning-scene-kit-tutorial
http://tacow.org/assets/attachments/SceneKit.pdf
Scene Kit allows you to render 3D scene easily, without OpenGL ES APIs. However you should understand how Scene Kit works.
Basically, Scene Kit provides a view controller that maintains an animation loop. This loop follows a design pattern common in games and simulations, with two phases: update and render. In the implementation, Scene Kit has more phases like the following figure (from http://www.objc.io/issue-18/scenekit.html), but basically, two phases, update and render.
So how to create Scene Kit project, the basics is
Prepare SCNView
Initialize 3D scene
Create touch event handler
Implement Update phase: Update game board using the touched object or the touched position, Update the animation of the objects, or some sort of stuff.
Implement Render phase: Basically, Scene Kit automatically renders registered 3D objects and models.
Thus, you should implement as the following.
Use SCNView instead of ViewController
Create a scene
Place Board and Tiles as Scene Kit 3D objects
Use hitTest for touching Tile and update Tiles in Update phase

import SceneKit
class GameViewController: UIViewController {
struct BoardLoc {
var x: Int
var y: Int
}
enum Type {
case Yellow
case Orange
}
var boardArray: Array<Array<Type>> = []
override func viewDidLoad() {
super.viewDidLoad()
for x in 0...2 {
boardArray.append(Array(count:5, repeatedValue:Type.Orange))
for y in 0...4 {
boardArray[x][y] = Type.Orange
}
}
let scene = SCNScene(named: "art.scnassets/balls8.dae")
let scnView = self.view as SCNView
scnView.scene = scene
scnView.autoenablesDefaultLighting = true
let taps = NSMutableArray()
let tap = UITapGestureRecognizer(target: self, action: "handleTap:")
taps.addObject(tap)
scnView.gestureRecognizers = taps
}
func handleTap(gestureRecognizer: UIGestureRecognizer) {
let scnView = view as SCNView
let point = gestureRecognizer.locationInView(scnView)
if let hitResults = scnView.hitTest(point, options: nil) {
if hitResults.count > 0 {
let result: AnyObject! = hitResults[0]
if !result.node!.name!.hasPrefix("Orange") {
return
}
let tapLoc = BoardLoc(x: Int(point.x) / 106, y: Int(point.y) / 106)
boardArray[tapLoc.x][tapLoc.y] = Type.Yellow
for col in 0...2 {
for row in 0...4 {
var yellowBall = scnView.scene!.rootNode.childNodeWithName("Yellow", recursively: true)
var secxtorX = Float(col) * 16.5 - 16
var sectorY = 34 - (Float(row) * 16.5)
if boardArray[col][row] == Type.Yellow {
yellowBall!.runAction(SCNAction.moveTo(SCNVector3(x: secxtorX, y: sectorY, z: 25), duration: 0.01))
boardArray[tapLoc.x][tapLoc.y] = Type.Orange
}
}
}
}
}
}
}

Related

ARKit – How to know if 3d object is in the center of a screen?

I place 3d object in the world space. After that I try to move camera randomly. Then right now I need to know after I knew object has became inside frustum by isNode method, if the object is in center, top or bottom of camera view.
For a solution that's not a hack you can use the projectPoint: API.
It's probably better to work with pixel coordinates because this method uses the actual camera's settings to determine where the object appears on screen.
let projectedPoint = sceneView.projectPoint(self.sphereNode.worldPosition)
let xOffset = projectedPoint.x - screenCenter.x;
let yOffset = projectedPoint.y - screenCenter.y;
if xOffset * xOffset + yOffset * yOffset < R_squared {
// inside a disc of radius 'R' at the center of the screen
}
Solution
To achieve this you need to use a trick. Create new SCNCamera, make it a child of pointOfView default camera and set its FoV to approximately 10 degrees.
Then inside renderer(_:updateAtTime:) instance method use isNode(:insideFrustumOf:) method.
Here's working code:
import ARKit
class ViewController: UIViewController,
ARSCNViewDelegate,
SCNSceneRendererDelegate {
#IBOutlet var sceneView: ARSCNView!
#IBOutlet var label: UILabel!
let cameraNode = SCNNode()
let sphereNode = SCNNode()
let config = ARWorldTrackingConfiguration()
public func renderer(_ renderer: SCNSceneRenderer,
updateAtTime time: TimeInterval) {
DispatchQueue.main.async {
if self.sceneView.isNode(self.sphereNode,
insideFrustumOf: self.cameraNode) {
self.label.text = "In the center..."
} else {
self.label.text = "Out OF CENTER"
}
}
}
override func viewDidLoad() {
super.viewDidLoad()
sceneView.delegate = self
sceneView.allowsCameraControl = true
let scene = SCNScene()
sceneView.scene = scene
cameraNode.camera = SCNCamera()
cameraNode.camera?.fieldOfView = 10
DispatchQueue.main.asyncAfter(deadline: .now() + 0.5) {
self.sceneView.pointOfView!.addChildNode(self.cameraNode)
}
sphereNode.geometry = SCNSphere(radius: 0.05)
sphereNode.geometry?.firstMaterial?.diffuse.contents = UIColor.red
sphereNode.position.z = -1.0
sceneView.scene.rootNode.addChildNode(sphereNode)
sceneView.session.run(config)
}
}
Also, in this solution you may turn on an orthographic projection for child camera, instead of perspective one. It helps when a model is far from the camera.
cameraNode.camera?.usesOrthographicProjection = true
Here's how your screen might look like:
Next steps
The same way you can append two additional SCNCameras, place them above and below central SCNCamera, and test your object with two extra isNode(:insideFrustumOf:) instance methods.
I solved problem with another way:
let results = self.sceneView.hitTest(screenCenter!, options: [SCNHitTestOption.rootNode: parentnode])
where parentnode is the parent of target node, because I have multiple nodes.
func nodeInCenter() -> SCNNode? {
let x = (Int(sceneView.projectPoint(sceneView.pointOfView!.worldPosition).x - sceneView.projectPoint(sphereNode.worldPosition).x) ^^ 2) < 9
let y = (Int(sceneView.projectPoint(sceneView.pointOfView!.worldPosition).y - sceneView.projectPoint(sphereNode.worldPosition).y) ^^ 2) < 9
if x && y {
return node
}
return nil
}

Losing reference to GameViewController from inside SpriteKit when moving between scenes - want to move between SKScene and UITableView

I have a simple (multi scene) spritekit game and would like to implement and display a settings screen using a UITableViewContoller. I am close but keep losing my reference to the games view controller when i change scene, which means i can only segue to the UITableViewController on the first scene when my game starts. I'm hoping someone can help. I setup the UITableView in the storyboard and setup a segue to the tableview from my GameViewController in there.
I have the following set up in my GameViewController file:
override func viewDidLoad() {
super.viewDidLoad()
if let view = self.view as! SKView? {
// Load the SKScene
if let scene = SKScene(fileNamed: "MenuScene") {
// Set the scale mode to scale to fit the window
scene.scaleMode = .aspectFill
//I'VE ADDED THIS
if let gameScene = scene as? MenuScene {
gameScene.viewController = self
}
// Present the scene
view.presentScene(scene)
}
view.ignoresSiblingOrder = true
}
}
func showSettings() {
performSegue(withIdentifier: "openSettings", sender: self)
}
which gives me a reference to the GameViewController from my SKScene. I then load the UITableViewController from my SKScene in spritekit when a user touches an SKLabelNode. A simpler version of my MenuScene code looks something like this:
class MenuScene: SKScene {
var viewController: GameViewController?
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?)
{
let touch:UITouch = touches.first!
let positionInScene = touch.location(in: self)
let touchedNode = self.atPoint(positionInScene)
if let name = touchedNode.name
{
let transition:SKTransition = SKTransition.fade(withDuration: 0.75)
var scene:SKScene = SKScene()
if (name == "settingsBTN"){
viewController?.showSettings()
}
else if (name == "randomBTN"){
scene = StarScene(size: self.size)
scene.backgroundColor = SKColor.black
self.view?.presentScene(scene, transition: transition)
}
}
}
}//end of MenuScene
The table view is presented fine when i tap the settingsBTN when first starting the game. however, if i visit another scene in my game (e.g. by pressing randomBTN) and return back to the menu scene i can no longer segue to the Settings view as i have lost the reference to the viewcontroller (it shows as nil) so i can't call the showSettings method in the GameViewController that performs the segue (presumably because i set up the reference to the view controller in the view controller itself).
I can't figure out how to maintain the reference to my GameViewController so i can present the settings screen at any point. Please could someone help.
There are two simple methods to call a GameViewController function from other classes: through a delegate or by making the class instance static.
// DELEGATE
//
// GameViewController.swift
protocol GameView_Delegate
{
func show_ui()
}
class GameViewController: UIViewController, GameView_Delegate
{
override func viewDidLoad()
{
// ...
if let view = self.view as! SKView?
{
if let scene = SKScene(fileNamed: "MenuScene")
{
scene.scaleMode = .aspectFill
// pass GameViewController reference
scene.gv_delegate = self
}
view.presentScene(scene)
}
// ...
}
func show_ui()
{
// show ui
// ...
}
}
// MenuScene.swift
class MenuScene: SKScene
{
var gv_delegate : GameView_Delegate!
func settings()
{
// call function show_ui from GameViewController
gv_delegate.show_ui()
}
func game_scene()
{
let transition = SKTransition.fade(with: UIColor.white, duration: 1.0)
let next_scene = GameScene()
next_scene.anchorPoint = CGPoint(x: 0.5, y: 0.5)
next_scene.scaleMode = self.scaleMode
next_scene.size = self.size
// before moving to next scene, we must pass the delegate, so we
// don't lose the connection with the GameViewController
next_scene.gv_delegate = gv_delegate
self.view?.presentScene(next_scene, transition: transition)
}
}
// GameScene.swift
class GameScene: SKScene
{
var gv_delegate : GameView_Delegate!
func settings()
{
// call function show_ui from GameViewController
gv_delegate.show_ui()
}
func main_scene()
{
let transition = SKTransition.fade(with: UIColor.white, duration: 1.0)
let next_scene = MainScene()
next_scene.anchorPoint = CGPoint(x: 0.5, y: 0.5)
next_scene.scaleMode = self.scaleMode
next_scene.size = self.size
// before moving to next scene, we must pass the delegate, so we
// don't lose the connection with the GameViewController
next_scene.gv_delegate = gv_delegate
self.view?.presentScene(next_scene, transition: transition)
}
}
When you enter a new scene, for example GameScene, the old scene MainScene in our case is deallocated (all variables emptied, all references destroyed) and a new GameScene instance is created and displayed on screen. If afterwards you want to exit that scene and enter MainScene, a new instance of that class will be created with the GameViewController reference empty. We should reference it again or pass the reference before exiting the scene (as in my example).
// STATIC
//
// GameViewController.swift
class GameViewController: UIViewController
{
static var shared : GameViewController!
override func viewDidLoad()
{
super.viewDidLoad()
GameViewController.shared = self
// ...
}
func show_ui()
{
// show ui
// ...
}
}
// MenuScene.swift
class MenuScene: SKScene
{
func settings()
{
// call function show_ui from GameViewController
GameViewController.shared.show_ui()
}
}
Read pros/cons of each method and chose the one that suits your app the best. Also, avoid using UIKit inside SpriteKit.

How to Mirror(Flip) Video playing using AVPlayer?

I want to Mirror (Flip) video playing using AVPlayer.
Like : MirrorTube :- https://chrome.google.com/webstore/detail/mirrortube/olomckflnlligkboahmaihmeaffjdbfm/related?hl=en
i want to achieve same functionality.
I have tried to change CGAffineTransform but it does not work same.
Thanks in advance!
Here's an example how to flip the player vertically and horizontally by using CGAffineTransform:
PlayerView:
import AVKit
class PlayerView: UIView {
var player: AVPlayer? {
get {
return playerLayer.player
}
set {
playerLayer.player = newValue
}
}
var playerLayer: AVPlayerLayer {
return layer as! AVPlayerLayer
}
// Override UIView property
override static var layerClass: AnyClass {
return AVPlayerLayer.self
}
}
ViewController using the playerView defined in xib/storyboard:
#IBOutlet var playerView: PlayerView!
#IBAction func flipVerticallyBarButtonItemTouched(_ sender: UIBarButtonItem) {
UIView.animate(withDuration: 0.2) { [unowned self] in
self.playerView.transform = self.playerView.transform.concatenating(CGAffineTransform(scaleX: 1, y: -1))
}
}
#IBAction func flipHorizontallyBarButtonItemTouched(_ sender: UIBarButtonItem) {
UIView.animate(withDuration: 0.2) { [unowned self] in
self.playerView.transform = self.playerView.transform.concatenating(CGAffineTransform(scaleX: -1, y: 1))
}
}
Note: For an iOS app in Mac Catalyst (with a AVPlayerController subclass) the root view is AVPlayerView.
But oddly AVPlayerView isn't public on iOS > Mac Catalyst so self.playerView wont work so you cant cast it. You get class not found for 'AVPlayerView'
But when you run the app in Mac Catalyst and inspect it the self.view is an AVPlayerView
workaround just flip the root view itself without casting - self.view
`class MyAVPlayerController: AVPlayerController`
to flip horizontal etc
self.view.transform = self.view.transform.concatenating(CGAffineTransform(scaleX: -1, y: 1))
to flip vertical etc
self.view.transform = self.view.transform.concatenating(CGAffineTransform(scaleX: 1, y: -1))
KNOWN ISSUE this flips the whole AVPlayerView including controls. in iOS 16 the actual player is a view where Swift.type(of:view) is "__AVPlayerLayerView" so you can walk the hierarchy and find the UIView return it then apply the transform to only that subview
func flipHorizontal(){
print("TradAVPlayerViewController.swift flipHorizontal")
if self.player != nil{
//------------------------------------------------------------------
//v1 - flips VC.view which is AVPlayerView
//but also flips TOAST and CONTROLS
//https://developer.apple.com/documentation/uikit/uiview/1622459-transform
//------------------------------------------------------------------
//OK ROTATES VIEW
//self.view.transform = CGAffineTransform(rotationAngle: CGFloat((90 * Double.pi)/180))
//OK FLIP VERTICAL
//self.view.transform = self.view.transform.concatenating(CGAffineTransform(scaleX: 1, y: -1))
//OK FLIPS HORIZONTAL
// self.view.transform = self.view.transform.concatenating(CGAffineTransform(scaleX: -1, y: 1))
//note the flip remains in place as long as self.player isnt recreated so PREV/NEXT will stay flipped
//------------------------------------------------------------------
//ISSUE flipping the whole view also flips the TOAST and SCRUBBER
//------------------------------------------------------------------
//in iOS 15 the player is a class with name "__AVPlayerLayerView"
//find it in the hierarchy and only apply transform to that on
if let rootView = self.view{
let className = "__AVPlayerLayerView"
if let subViewFound
= findSubViewByName(name: className,
view: rootView)
{
print("FOUND className:'\(className)' flipping it")
subViewFound.transform = subViewFound.transform.concatenating(CGAffineTransform(scaleX: -1, y: 1))
}else{
print("NOT FOUND className:'\(className)' CANT FLIP IT - debug hierarchy in 3d and get class name for the AVPlayerLayerView")
}
}else{
logger.error("self.view is nil")
}
print()
}else{
logger.error("self.player is nil")
}
}
func findSubViewByName(name: String, view: UIView) -> UIView?{
var viewFound: UIView? = nil
for subView in view.subviews{
//MUST WRAP IN STRING else types wont match
let typeName = "\(Swift.type(of: subView))"
//print("typeName:\(typeName)")
if typeName == name {
print("SUB VIEW - typeName:\(typeName) - MATCH!!")
viewFound = subView
break
}else{
print("SUB VIEW - typeName:\(typeName) - NO MATCH")
}
//recurse depth first
if let subViewFound = findSubViewByName(name: name, view: subView){
viewFound = subViewFound
break
}else{
//no match in subviewsf
}
}
return viewFound
}

ARKit – How to put 3D Object on QRCode?

I'm actually trying to put a 3D Object on QRCode with ARKit
For that I use a AVCaptureDevice to detect a QRCode and establish the area of the QRCode that gives me a CGRect.
Then, I make a hitTest on every point of the CGRect to get the average 3D coordinates like so :
positionGiven = SCNVector3(0, 0, 0)
for column in Int(qrZone.origin.x)...2*Int(qrZone.origin.x + qrZone.width) {
for row in Int(qrZone.origin.y)...2*Int(qrZone.origin.y + qrZone.height) {
for result in sceneView.hitTest(CGPoint(x: CGFloat(column)/2,y:CGFloat(row)/2), types: [.existingPlaneUsingExtent,.featurePoint]) {
positionGiven.x+=result.worldTransform.columns.3.x
positionGiven.y+=result.worldTransform.columns.3.y
positionGiven.z+=result.worldTransform.columns.3.z
cpts += 1
}
}
}
positionGiven.x=positionGiven.x/cpts
positionGiven.y=positionGiven.y/cpts
positionGiven.z=positionGiven.z/cpts
But the hitTest doesn't detect any result and freeze the camera while when I make a hitTest with a touch on screen it works.
Do you have any idea why it's not working ?
Do you have an other idea that can help me to achieve what I want to do ?
I already thought about 3D translation with CoreMotion that can give me the tilt of the device but that seems really tedious.
I also heard about ARWorldAlignmentCamera that can locked the scene coordinate to match the orientation of the camera but I don't know how to use it !
Edit : I try to move my 3D Object every time I touch the screen and the hitTest is positive, and it's pretty accurate ! I really don't understand why hitTest on an area of pixels doesn't work...
Edit 2 : Here is the code of the hitTest who works with 2-5 touches on the screen:
#objc func touch(sender : UITapGestureRecognizer) {
for result in sceneView.hitTest(CGPoint(x: sender.location(in: view).x,y: sender.location(in: view).y), types: [.existingPlaneUsingExtent,.featurePoint]) {
//Pop up message for testing
alert("\(sender.location(in: view))", message: "\(result.worldTransform.columns.3)")
//Moving the 3D Object to the new coordinates
let objectList = sceneView.scene.rootNode.childNodes
for object : SCNNode in objectList {
object.removeFromParentNode()
}
addObject(SCNVector3(result.worldTransform.columns.3.x,result.worldTransform.columns.3.y,result.worldTransform.columns.3.z))
}
}
Edit 3 :
I manage to resolve my problem partially.
I take the transform matrix of the camera (session.currentFrame.camera.transform) so that the object is in front of the camera.
Then I apply a translation on (x,y) with the position of the CGRect.
However i can't translate the z-axis because i don't have enough informations.
And I will probably need a estimation of z coordinate like the hitTest do..
Thanks in advance ! :)
You could use Apple's Vision API to detect the QR code and place an anchor.
To start detecting QR codes, use:
var qrRequests = [VNRequest]()
var detectedDataAnchor: ARAnchor?
var processing = false
func startQrCodeDetection() {
// Create a Barcode Detection Request
let request = VNDetectBarcodesRequest(completionHandler: self.requestHandler)
// Set it to recognize QR code only
request.symbologies = [.QR]
self.qrRequests = [request]
}
In ARSession's didUpdate Frame
public func session(_ session: ARSession, didUpdate frame: ARFrame) {
DispatchQueue.global(qos: .userInitiated).async {
do {
if self.processing {
return
}
self.processing = true
// Create a request handler using the captured image from the ARFrame
let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: frame.capturedImage,
options: [:])
// Process the request
try imageRequestHandler.perform(self.qrRequests)
} catch {
}
}
}
Handle the Vision QR request and trigger the hit test
func requestHandler(request: VNRequest, error: Error?) {
// Get the first result out of the results, if there are any
if let results = request.results, let result = results.first as? VNBarcodeObservation {
guard let payload = result.payloadStringValue else {return}
// Get the bounding box for the bar code and find the center
var rect = result.boundingBox
// Flip coordinates
rect = rect.applying(CGAffineTransform(scaleX: 1, y: -1))
rect = rect.applying(CGAffineTransform(translationX: 0, y: 1))
// Get center
let center = CGPoint(x: rect.midX, y: rect.midY)
DispatchQueue.main.async {
self.hitTestQrCode(center: center)
self.processing = false
}
} else {
self.processing = false
}
}
func hitTestQrCode(center: CGPoint) {
if let hitTestResults = self.latestFrame?.hitTest(center, types: [.featurePoint] ),
let hitTestResult = hitTestResults.first {
if let detectedDataAnchor = self.detectedDataAnchor,
let node = self.sceneView.node(for: detectedDataAnchor) {
let previousQrPosition = node.position
node.transform = SCNMatrix4(hitTestResult.worldTransform)
} else {
// Create an anchor. The node will be created in delegate methods
self.detectedDataAnchor = ARAnchor(transform: hitTestResult.worldTransform)
self.sceneView.session.add(anchor: self.detectedDataAnchor!)
}
}
}
Then handle adding the node when the anchor is added.
func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
// If this is our anchor, create a node
if self.detectedDataAnchor?.identifier == anchor.identifier {
let sphere = SCNSphere(radius: 1.0)
sphere.firstMaterial?.diffuse.contents = UIColor.redColor()
let sphereNode = SCNNode(geometry: sphere)
sphereNode.transform = SCNMatrix4(anchor.transform)
return sphereNode
}
return nil
}
Source

Accessibility (Voice Over) with Sprite Kit

I'm attempting to add support for Voice Over accessibility in a puzzle game which has a fixed board. However, I'm having trouble getting UIAccessibilityElements to show up.
Right now I'm overriding accessibilityElementAtIndex, accessibilityElementCount and indexOfAccessibilityElement in my SKScene.
They are returning an array of accessible elements as such:
func loadAccessibleElements()
{
self.isAccessibilityElement = false
let pieces = getAllPieces()
accessibleElements.removeAll(keepCapacity: false)
for piece in pieces
{
let element = UIAccessibilityElement(accessibilityContainer: self.usableView!)
element.accessibilityFrame = piece.getAccessibilityFrame()
element.accessibilityLabel = piece.getText()
element.accessibilityTraits = UIAccessibilityTraitButton
accessibleElements.append(element)
}
}
Where piece is a subclass of SKSpriteNode and getAccessibilityFrame is defined:
func getAccessibilityFrame() -> CGRect
{
return parentView!.convertRect(frame, toView: nil)
}
Right now one (wrongly sized) accessibility element seems to appear on the screen in the wrong place.
Could someone point me in the right direction?
Many thanks
EDIT:
I've tried a hack-ish work around by placing a UIView over the SKView with UIButton elements in the same location as the SKSpriteNodes. However, accessibility still doesn't want to work. The view is loaded as such:
func loadAccessibilityView()
{
view.isAccessibilityElement = false
view.accessibilityElementsHidden = false
skView.accessibilityElementsHidden = false
let accessibleSubview = UIView(frame: view.frame)
accessibleSubview.userInteractionEnabled = true
accessibleSubview.isAccessibilityElement = false
view.addSubview(accessibleSubview)
view.bringSubviewToFront(accessibleSubview)
let pieces = (skView.scene! as! GameScene).getAllPieces()
for piece in pieces
{
let pieceButton = UIButton(frame: piece.getAccessibilityFrame())
pieceButton.isAccessibilityElement = true
pieceButton.accessibilityElementsHidden = false
pieceButton.accessibilityTraits = UIAccessibilityTraitButton
pieceButton.setTitle(piece.getText(), forState: UIControlState.Normal)
pieceButton.setBackgroundImage(UIImage(named: "blue-button"), forState: UIControlState.Normal)
pieceButton.alpha = 0.2
pieceButton.accessibilityLabel = piece.getText()
pieceButton.accessibilityFrame = pieceButton.frame
pieceButton.addTarget(self, action: Selector("didTap:"), forControlEvents: UIControlEvents.TouchUpInside)
accessibleSubview.addSubview(pieceButton)
}
UIAccessibilityPostNotification(UIAccessibilityScreenChangedNotification, nil)
}
The buttons are placed correctly, however accessibility just isn't working at all. Something seems to be preventing it from working.
I've searched in vain for a description of how to implement VoiceOver in Swift using SpriteKit, so I finally figured out how to do it. Here's some working code that converts a SKNode to an accessible pushbutton when added to a SKScene class:
// Add the following code to a scene where you want to make the SKNode variable named “leave” an accessible button
// leave must already be initialized and added as a child of the scene, or a child of other SKNodes in the scene
// screenHeight must already be defined as the height of the device screen, in points
// Accessibility
private var accessibleElements: [UIAccessibilityElement] = []
private func nodeToDevicePointsFrame(node: SKNode) -> CGRect {
// first convert from frame in SKNode to frame in SKScene's coordinates
var sceneFrame = node.frame
sceneFrame.origin = node.scene!.convertPoint(node.frame.origin, fromNode: node.parent!)
// convert frame from SKScene coordinates to device points
// sprite kit scene origin is in lower left, accessibility device screen origin is at upper left
// assumes scene is initialized using SKSceneScaleMode.Fill using dimensions same as device points
var deviceFrame = sceneFrame
deviceFrame.origin.y = CGFloat(screenHeight-1) - (sceneFrame.origin.y + sceneFrame.size.height)
return deviceFrame
}
private func initAccessibility() {
if accessibleElements.count == 0 {
let accessibleLeave = UIAccessibilityElement(accessibilityContainer: self.view!)
accessibleLeave.accessibilityFrame = nodeToDevicePointsFrame(leave)
accessibleLeave.accessibilityTraits = UIAccessibilityTraitButton
accessibleLeave.accessibilityLabel = “leave” // the accessible name of the button
accessibleElements.append(accessibleLeave)
}
}
override func didMoveToView(view: SKView) {
self.isAccessibilityElement = false
leave.isAccessibilityElement = true
}
override func willMoveFromView(view: SKView) {
accessibleElements = []
}
override func accessibilityElementCount() -> Int {
initAccessibility()
return accessibleElements.count
}
override func accessibilityElementAtIndex(index: Int) -> AnyObject? {
initAccessibility()
if (index < accessibleElements.count) {
return accessibleElements[index] as AnyObject
} else {
return nil
}
}
override func indexOfAccessibilityElement(element: AnyObject) -> Int {
initAccessibility()
return accessibleElements.indexOf(element as! UIAccessibilityElement)!
}
Accessibility frames are defined in the fixed physical screen coordinates, not UIView coordinates, and transforming between them is kind of tricky.
The device origin is the lower left of the screen, with X up, when the device is in landscape right mode.
It's a pain converting, I've no idea why Apple did it that way.