I've been having some fun using GameplayKit in a Scenekit (and ARKit, although that isn't important to this question) application.
Specifically, I have been using Entities, Components, and Agents for Behavior, and it has all been working great, except for one thing.
I have managed to use an GKAgent inside a component to move around a scene, and to avoid other objects. That all seems to be working. However, I can only get the GKAgent position working, not the "rotation" property.
import Foundation
import SceneKit
import GameplayKit
class MoveComponent: GKScnComponent, GKAgentDelegate {
//this class is abbreviated for perfunctory sakes
func agentWillUpdate(_ agent: GKAgent) {
guard let visualComponent = entity?.component(ofType: self.visualType) as? VisualComponent else {
return
}
//works with just "position", but turns possessed with both rotation and position
agentSeeking.rotation = convertToFloat3x3(float4x4: visualComponent.parentMostNode.simdTransform)
agentSeeking.position = visualComponent.parentMostNode.simdPosition
}
func agentDidUpdate(_ agent: GKAgent) {
guard let visualComponent = entity?.component(ofType: self.visualType) as? VisualComponent else {
return
}
//works with just "position", but turns possessed with both rotation and position
visualComponent.parentMostNode.simdPosition = agentSeeking.position
visualComponent.parentMostNode.simdTransform = convertToFloat4x4(float3x3: agentSeeking.rotation)
}
}
I have written some conversion code between 3x3 matrices and 4x4 matrices, thanks to this question:
Converting matrix_float3x3 rotation to SceneKit
import Foundation
import GameplayKit
func convertToFloat3x3(float4x4: simd_float4x4) -> simd_float3x3 {
let column0 = convertToFloat3 ( float4: float4x4.columns.0 )
let column1 = convertToFloat3 ( float4: float4x4.columns.1 )
let column2 = convertToFloat3 ( float4: float4x4.columns.2 )
return simd_float3x3.init(column0, column1, column2)
}
func convertToFloat3(float4: simd_float4) -> simd_float3 {
return simd_float3.init(float4.x, float4.y, float4.z)
}
func convertToFloat4x4(float3x3: simd_float3x3) -> simd_float4x4 {
let column0 = convertToFloat4 ( float3: float3x3.columns.0 )
let column1 = convertToFloat4 ( float3: float3x3.columns.1 )
let column2 = convertToFloat4 ( float3: float3x3.columns.2 )
let identity3 = simd_float4.init(x: 0, y: 0, z: 0, w: 1)
return simd_float4x4.init(column0, column1, column2, identity3)
}
func convertToFloat4(float3: simd_float3) -> simd_float4 {
return simd_float4.init(float3.x, float3.y, float3.z, 0)
}
I'm a little new to all this, and I'm not a linear algebra guru, so I'm not 100% certain if my matrix conversion functions are doing exactly what they are supposed to do.
When I just use the "position" property of the agent, everything is fine, but when I add in rotation/transform from the agent to the node, everything starts acting possessed.
Any thoughts/pointers/help with what I'm doing wrong?
I figured it out. Matrix multiplication is the key to getting this working. Anyone with experience with 3d game development probably could have told me that without any mental exertion :)
func agentWillUpdate(_ agent: GKAgent) {
guard let visualComponent = entity?.component(ofType: self.visualType) as? VisualComponent else {
return
}
agentSeeking.position = visualComponent.parentMostNode.simdPosition
let rotation = visualComponent.parentMostNode.rotation
let rotationMatrix = SCNMatrix4MakeRotation(rotation.w, rotation.x, rotation.y, rotation.z)
let float4x4 = SCNMatrix4ToMat4(rotationMatrix)
agentSeeking.rotation = convertToFloat3x3(float4x4: float4x4)
}
func agentDidUpdate(_ agent: GKAgent) {
guard let visualComponent = entity?.component(ofType: self.visualType) as? VisualComponent else {
return
}
// visualComponent.parentMostNode.simdPosition = agentSeeking.position
let rotation3x3 = agentSeeking.rotation
let rotation4x4 = convertToFloat4x4(float3x3: rotation3x3)
let rotationMatrix = SCNMatrix4FromMat4(rotation4x4)
let position = agentSeeking.position
let translationMatrix = SCNMatrix4MakeTranslation(position.x, position.y, position.z)
let transformMatrix = SCNMatrix4Mult(rotationMatrix, translationMatrix)
visualComponent.parentMostNode.transform = transformMatrix
}
I hope somebody finds this useful.
For converting from an agent's rotation 3x3 matrix to a scene node rotation, one convenient way is to use simd_quatf, so you can do something like:
node.simdWorldOrientation = simd_quatf(agent.rotation)
Similarly, using simd_float3x3 lets one convert from a quaternion (e.g. simdWorldOrientation) to a 3x3 rotation matrix. For example:
agent.rotation = simd_float3x3(node.simdWorldOrientation)
These let you avoid the extra conversion functions.
Related
I am trying to create some AR experience.
I load the Model with animations as an Entity. Lets call it a Toy.
I create an AnchorEntity.
I attach the Toy to the AnchorEntity. Up to this point everything works great.
I want the Toy to walk in random directions. And it does for the first time. Then it gets interesting, allow me to share my code:
First method uses a newly created Transform for the Toy with the modified translation x, y, to make the Toy move and that is it.
func walk(completion: #escaping () -> Void) {
guard let robot = robot else {
return
}
let currentTransform = robot.transform
guard let path = randomPath(from: currentTransform) else {
return
}
let (newTranslation , travelTime) = path
let newTransform = Transform(scale: currentTransform.scale,
rotation: currentTransform.rotation,
translation: newTranslation)
robot.move(to: newTransform, relativeTo: nil, duration: travelTime)
DispatchQueue.main.asyncAfter(deadline: .now() + travelTime + 1) {
completion()
}
}
We get that new Transform from the method below.
func randomPath(from currentTransform: Transform) -> (SIMD3<Float>, TimeInterval)? {
// Get the robot's current transform and translation
let robotTranslation = currentTransform.translation
// Generate random distances for a model to cross, relative to origin
let randomXTranslation = Float.random(in: 0.1...0.4) * [-1.0,1.0].randomElement()!
let randomZTranslation = Float.random(in: 0.1...0.4) * [-1.0,1.0].randomElement()!
// Create a translation relative to the current transform
let relativeXTranslation = robotTranslation.x + randomXTranslation
let relativeZTranslation = robotTranslation.z + randomZTranslation
// Find a path
var path = (randomXTranslation * randomXTranslation + randomZTranslation * randomZTranslation).squareRoot()
// Path only positive
if path < 0 { path = -path }
// Calculate the time of walking based on the distance and default speed
let timeOfWalking: Float = path / settings.robotSpeed
// Based on old trasnlation calculate the new one
let newTranslation: SIMD3<Float> = [relativeXTranslation,
Float(0),
relativeZTranslation]
return (newTranslation, TimeInterval(timeOfWalking))
}
The problem is that the value of Entity.transform.translation.y grows from 0 to some random value < 1. Always after the second time the walk() method is being called.
As you can see, every time the method is called, newTranslation sets the Y value to be 0. And yet the Toy's translation:
I am out of ideas any help is appreciated. I can share the whole code if needed.
I have managed to fix the issue by specifying parameter relativeTo as Toy's AnchorEntity:
toy.move(to: newTransform, relativeTo: anchorEntity, duration: travelTime)
I'm working on a project which converts user's face to emoji. I use Apple's ARKit in this purpose.
I need to get the most probable option. I wrote this code:
func renderer(for anchor: ARAnchor) {
guard let faceAnchor = anchor as? ARFaceAnchor else {
return
}
let shapes = faceAnchor.blendShapes
let browDownLeft = shapes[.browDownLeft]!.doubleValue
let browInnerUp = shapes[.browInnerUp]!.doubleValue
let browOuterUpLeft = shapes[.browOuterUpLeft]!.doubleValue
let leftBrowMax = max(browDownLeft, browInnerUp, browOuterUpLeft)
switch leftBrowMax {
case browDownLeft:
userFace.leftBrow = .browDown
case browInnerUp:
userFace.leftBrow = .browInnerUp
case browOuterUpLeft:
userFace.leftBrow = .browOuterUp
default:
userFace.leftBrow = .any
}
}
I need to duplicate function's body six time (for brows, eyes and mouth sides), so I want to write it in a more convenient way. Is there any options in Swift like numpy's argmax function? Also I need to specify arguments range, because arguments for mouth should not be compared with arguments for brows.
You can use something like this:
func maxBlendShape(for blendShapes: [ARFaceAnchor.BlendShapeLocation], in shape: [ARFaceAnchor.BlendShapeLocation: NSNumber]) -> Double? {
blendShapes
.compactMap { shape[$0] }
.map(\.doubleValue)
.max()
}
Usage would then be something like this:
maxBlendShape(for: [.browDownLeft, .browInnerUp, .browOuterUpLeft], in: faceAnchor.blendShapes)
Note: Nothing here is specific to ARKit, you just filter some keys from the dictionary and find their max value. A generic solution could look like this:
extension Dictionary where Value == NSNumber {
func maxDouble(for keys: [Key]) -> Double? {
keys
.compactMap({self[$0]})
.map(\.doubleValue)
.max()
}
}
faceAnchor.blendShapes.maxDouble(for: [.browInnerUp, .browDownLeft, .browOuterUpLeft])
Recently, I make it about lidar scan project.
It is very difficult. And I need to manipulate vertex data.
So I tried by this code
guard let meshAnchors = arView.session.currentFrame?.anchors.compactMap { $0 as? ARMeshAnchor }
else { return }
meshAnchors.first?.geometry.vertices // I want to get vertex position
There is no position of vertex, only buffer data
How can I do that? is it change from buffer data to array?
Plz help me.
Just went through this myself so I figured I'd drop ya my solution.
First grab this extension from Apple's Documentation to get a vertex at a specific index:
extension ARMeshGeometry {
func vertex(at index: UInt32) -> SIMD3<Float> {
assert(vertices.format == MTLVertexFormat.float3, "Expected three floats (twelve bytes) per vertex.")
let vertexPointer = vertices.buffer.contents().advanced(by: vertices.offset + (vertices.stride * Int(index)))
let vertex = vertexPointer.assumingMemoryBound(to: SIMD3<Float>.self).pointee
return vertex
}
}
Then, to get the positions in ARKit world space, you can do something like this:
func getVertexWorldPositions(frame: ARFrame) {
let anchors = frame.anchors.filter { $0 is ARMeshAnchor } as! [ARMeshAnchor]
// Each mesh geometry lives in its own anchor
for anchor in anchors {
// Anchor's transform in world space
let aTrans = SCNMatrix4(anchor.transform)
let meshGeometry = anchor.geometry
let vertices: ARGeometrySource = meshGeometry.vertices
for vIndex in 0..<vertices.count {
// This will give you a vertex in local (anchor) space
let vertex = meshGeometry.vertex(at: UInt32(vIndex))
// Create a new matrix with the vertex coordinates
let vTrans = SCNMatrix4MakeTranslation(vertex[0], vertex[1], vertex[2])
// Multiply it by the anchors's transform to get it into world space
let wTrans = SCNMatrix4Mult(vTrans, aTrans)
// Use the coordinates for something!
let vPos = SCNVector3(wTrans.m41, wTrans.m42, wTrans.m43)
print(vPos)
}
}
}
I am pretty Newbie to programming. And I am trying to pile up the random blocks dynamically till it hits the upper frame. But it seems that Swift doesn't let me to do so. Did I miss anything please? Any input are appreciated.
let blocks =[block1,block2,block3,block4,block5,block6,block7,block8,block9,block10,block11,block12]
var block:SKSpriteNode!
let blockX:Double = 0.0
var blockY:Double = -(self.size.height/2)
repeat{
block = blocks.randomBlock()
block.zPosition = 2
block.position = CGPoint(x:blockX, y:blockY)
block.size.height = 50
block.size.width = 50
self.addChild(block)
blockY += 50
} while( block.position.y < self.size.height)
extension Array {
func randomBlock()-> Element {
let randint = Int(arc4random_uniform(UInt32(self.count)))
return self[randint]
}
}
you need to have someway of tracking which blocks have been selected and ensure that they don't get selected again. The method below uses an array to store the indexes of selected blocks and then uses recursion to find a cycle through until an unused match is found.
private var usedBlocks = [Int]()
func randomBlock() -> Int {
guard usedBlocks.count != blocks.count else { return -1 }
let random = Int(arc4random_uniform(UInt32(blocks.count)))
if usedBlocks.contains(random) {
return randomBlock()
}
usedBlocks.append(random)
return random
}
in your loop change your initializer to
let index = randomBlock()
if index > -1 {
block = blocks[index]
block.zPosition = 2
block.position = CGPoint(x:blockX, y:blockY)
}
remember that if you restart the game or start a new level, etc. you must clear all of the objects from usedBlocks
usedBlocks.removeAll()
I'm new to programming and have been taking online courses in swift and spritekit trying to create my first working game from scratch.
Currently I'm having an issue trying to create a sequence of functions that run independently, wait so that only one is running at a time, and loops indefinitely.
The first function:
func shootTwentyArrows() {
var oneArrow = SKAction.runBlock{
self.shootArrow()
}
var waitBetweenArrows = SKAction.waitForDuration(arrowSpeed)
var fireAnArrow = SKAction.sequence([oneArrow, waitBetweenArrows])
self.runAction(SKAction.repeatAction(fireAnArrow, count: 20))
}
And the second function:
func shootSpiral1() {
var leftArrow = SKAction.runBlock{
self.arrowFromLeft()
}
var rightArrow = SKAction.runBlock{
self.arrowFromRight()
}
var waitBetweenArrows = SKAction.waitForDuration(arrowSpeed)
var sequence = SKAction.sequence([leftArrow, waitBetweenArrows, rightArrow, waitBetweenArrows])
var spiral1 = SKAction.repeatAction(sequence, count: 5)
self.runAction(spiral1)
to clarify, I'm trying to run something like:
shootTwentyArrows()
when that's done, shootSpiral1(), when that's done repeat.
Thanks in advance for any responses.
I guess the most correct way to do that would be to refactor code a little bit:
func shootTwentyArrows() -> SKAction {
let oneArrow = SKAction.runBlock{
self.shootArrow()
}
let waitBetweenArrows = SKAction.waitForDuration(arrowSpeed)
let fireAnArrow = SKAction.sequence([oneArrow, waitBetweenArrows])
return SKAction.repeatAction(fireAnArrow, count: 20)
}
func shootSpiral1() -> SKAction {
let leftArrow = SKAction.runBlock{
self.arrowFromLeft()
}
let rightArrow = SKAction.runBlock{
self.arrowFromRight()
}
let waitBetweenArrows = SKAction.waitForDuration(arrowSpeed)
let sequence = SKAction.sequence([leftArrow, waitBetweenArrows, rightArrow, waitBetweenArrows])
let spiral1 = SKAction.repeatAction(sequence, count: 5)
return spiral1
}
Then somewhere in the code you can just do something like that :
let spiralAction = shootSpiral1()
let oneArrowAction = shootTwentyArrows()
let sequence = SKAction.sequence([spiralAction, oneArrowAction])
let infiniteSequence = SKAction.repeatActionForever(sequence)
self.runAction(infiniteSequence)
I left function names the same on purpose, so you get the idea.
P.S. It is a common practice to declare a variable as let, declare it as var only when you have to modify it later.
The best way to do this is have an SKAction sequence that calls both of your functions. You would call this SKAction from wherever you want to initialize the arrow shooting actions like viewDidLoad for example. This would be the code to call the actions...
var actionShootingArrows = SKAction.sequence([shootSpiral1(), shootTwentyArrows()])
self.runAction(SKAction.repeatActionForever(actionShootingArrows))
Hope this helps!