I have quite a specific problem but hopefully someone can help me. I'm using AVFoundation to create a video camera with a live preview. I use AVCaptureVideoDataOutput to get individual frames and AVCaptureMetadataOutput to detect a face. I'm also using Dlib's facial landmarks predictor to show the landmark points on the users face and measure the interocular distance between their eyes. Finally I'm using AVAssetWriter so that a video can be recorded.
The view controller has an ellipse shape on it so the user knows where to put their face. When the interocular distance is between a certain distance I want the ellipse to turn blue so the user knows their face is in the right place.
At the minute I've achieved this by sending a notification from my SessionHandler class to the View Controller. This works, however it's causing the frames per second in the video to drop badly. I was getting 25fps (manually set by me) and now it's ranging between 8-16.
Is there another way to notify the view controller that the ellipse should be turned green?
Here's my code where the problem is occurring. I know there's a lot going on.
// MARK: AVCaptureVideoDataOutputSampleBufferDelegate, AVCaptureAudioOutputSampleBufferDelegate
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
if !currentMetadata.isEmpty {
let boundsArray = currentMetadata
.compactMap { $0 as? AVMetadataFaceObject }
.map { (faceObject) -> NSValue in
let convertedObject = output.transformedMetadataObject(for: faceObject, connection: connection)
return NSValue(cgRect: convertedObject!.bounds)
}
if user.hasDlib {
wrapper?.doWork(on: sampleBuffer, inRects: boundsArray)
// Get the interocular distance so face is the correct place in the oval
let interocularDistance = wrapper?.getEyeDistance()
//print("Interocular Distance: \(interocularDistance)")
if user.hasInterocularDistance {
if interocularDistance! < 240 || interocularDistance! > 315 {
let name = Notification.Name(rawValue: setRemoveGreenEllipse)
NotificationCenter.default.post(name: name, object: nil)
//print("face not correct distance")
if videoRecorder.isRecording {
eyeDistanceCounter += 1
//print(eyeDistanceCounter)
if eyeDistanceCounter == 30 {
cancelledByUser = false
cancelledByEyeDistance = true
videoRecorder.cancel()
eyeDistanceCounter = 0
}
}
} else {
//print("face correct distance")
eyeDistanceCounter = 0
let name = Notification.Name(rawValue: setGreenEllipse)
NotificationCenter.default.post(name: name, object: nil)
}
}
}
} else {
// Check if face is detected during recording. If it isn't, then cancel recording
if videoRecorder.isRecording {
noFaceCount += 1
if noFaceCount == 50 {
cancelledByUser = false
videoRecorder.cancel()
noFaceCount = 0
}
}
}
if layer.status == .failed {
layer.flush()
}
layer.enqueue(sampleBuffer)
let writable = videoRecorder.canWrite()
if writable {
if videoRecorder.sessionAtSourceTime == nil {
// Start Writing
videoRecorder.sessionAtSourceTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
videoRecorder.videoWriter.startSession(atSourceTime: videoRecorder.sessionAtSourceTime!)
print("video session started")
}
if videoRecorder.videoWriterInput.isReadyForMoreMediaData {
// write video buffer
videoRecorder.videoWriterInput.append(sampleBuffer)
//print("video buffering")
}
}
}
You could probably call the notification once per 30 frames, for example, instead of every frame.
You could also call the color changing function directly if it's in the same view controller. If not, you could define a delegate method and call it directly as oppose to sending notifications.
Related
I was following Apple Documentation and example project to load 3d Object using .SCN file with Virtual Object (subclass of SCNReferenceNode) class but suddenly i needed to change the model from .scn to usdz . Now my usdz object is loading successfully but it is not on surface (midway in the air) and i can't interact with it like (tap , pan , rotate) ... Is there any other way to get interaction with usdz object and how can I place it on the surface like I was doing it before with .scn file
For getting model URL (downloaded from server)
static let loadDownloadedModel : VirtualObject = {
let downloadedScenePath = getDocumentsDirectory().appendingPathComponent("\(Api.Params.inputModelName).usdz")
return VirtualObject(url: downloadedScenePath)!
}()
Loading it from URL
func loadVirtualObject(_ object: VirtualObject, loadedHandler: #escaping (VirtualObject) -> Void) {
isLoading = true
loadedObjects.append(object)
// Load the content asynchronously.
DispatchQueue.global(qos: .userInitiated).async {
object.reset()
object.load()
self.isLoading = false
loadedHandler(object)
}
}
Placing in the scene
func placeObjectOnFocusSquare() {
virtualObjectLoader.loadVirtualObject(VirtualObject.loadDownloadedModel) { (loadedObject) in
DispatchQueue.main.async {
self.placeVirtualObject(loadedObject)
self.setupBottomButtons(isSelected: true)
}
}
}
func placeVirtualObject(_ virtualObject: VirtualObject) {
guard let cameraTransform = session.currentFrame?.camera.transform,
let focusSquarePosition = focusSquare.lastPosition else {
statusViewController.showMessage("CANNOT PLACE OBJECT\nTry moving left or right.")
return
}
Api.Params.selectedModel = virtualObject
virtualObject.name = String(Api.Params.inputPreviewId)
virtualObject.scale = SCNVector3(Api.Params.modelXAxis, Api.Params.modelYAxis, Api.Params.modelZAxis)
virtualObject.setPosition(focusSquarePosition, relativeTo: cameraTransform, smoothMovement: false)
updateQueue.async {
self.sceneView.scene.rootNode.addChildNode(virtualObject)
}
}
.usdz object in sceneview
After so many tries , finally i found out that dynamic scaling of the model causing problem , Reference to this
iOS ARKit: Large size object always appears to move with the change in the position of the device camera
I scaled the object to 0.01 for all the axis (x,y and z)
virtualObject.scale = SCNVector3Make(0.01, 0.01, 0.01)
I using Apple's example https://developer.apple.com/library/ios/samplecode/AVCustomEdit/Introduction/Intro.html and have some issues with video transformation.
If source assets have preferredTransform other than identity, output video will have incorrectly rotated frames. This problem can be fixed if AVMutableVideoComposition doesn't have value in property customVideoCompositorClass and when AVMutableVideoCompositionLayerInstruction's transform is setted up with asset.preferredTransform. But in reason of using custom video compositor, which adopting an AVVideoCompositing protocol I can't use standard video compositing instructions.
How can I pre-transform input asset tracks before it's CVPixelBuffer's putted into Metal shaders? Or there are any other way to fix it?
Fragment of original code:
func buildCompositionObjectsForPlayback(_ forPlayback: Bool, overwriteExistingObjects: Bool) {
// Proceed only if the composition objects have not already been created.
if self.composition != nil && !overwriteExistingObjects { return }
if self.videoComposition != nil && !overwriteExistingObjects { return }
guard !clips.isEmpty else { return }
// Use the naturalSize of the first video track.
let videoTracks = clips[0].tracks(withMediaType: AVMediaType.video)
let videoSize = videoTracks[0].naturalSize
let composition = AVMutableComposition()
composition.naturalSize = videoSize
/*
With transitions:
Place clips into alternating video & audio tracks in composition, overlapped by transitionDuration.
Set up the video composition to cycle between "pass through A", "transition from A to B", "pass through B".
*/
let videoComposition = AVMutableVideoComposition()
if self.transitionType == TransitionType.diagonalWipe.rawValue {
videoComposition.customVideoCompositorClass = APLDiagonalWipeCompositor.self
} else {
videoComposition.customVideoCompositorClass = APLCrossDissolveCompositor.self
}
// Every videoComposition needs these properties to be set:
videoComposition.frameDuration = CMTimeMake(1, 30) // 30 fps.
videoComposition.renderSize = videoSize
buildTransitionComposition(composition, andVideoComposition: videoComposition)
self.composition = composition
self.videoComposition = videoComposition
}
UPDATE:
I did workaround for transforming like this:
private func makeTransformedPixelBuffer(fromBuffer buffer: CVPixelBuffer, withTransform transform: CGAffineTransform) -> CVPixelBuffer? {
guard let newBuffer = renderContext?.newPixelBuffer() else {
return nil
}
// Correct transformation example I took from https://stackoverflow.com/questions/29967700/coreimage-coordinate-system
var preferredTransform = transform
preferredTransform.b *= -1
preferredTransform.c *= -1
var transformedImage = CIImage(cvPixelBuffer: buffer).transformed(by: preferredTransform)
preferredTransform = CGAffineTransform(translationX: -transformedImage.extent.origin.x, y: -transformedImage.extent.origin.y)
transformedImage = transformedImage.transformed(by: preferredTransform)
let filterContext = CIContext(mtlDevice: MTLCreateSystemDefaultDevice()!)
filterContext.render(transformedImage, to: newBuffer)
return newBuffer
}
But wondering if there are more memory-effective way without creation of new pixel buffers
How can I pre-transform input asset tracks before it's CVPixelBuffer's
putted into Metal shaders?
The best way to achieve maximum performance is to transform your video frame directly in shader. You just need to add rotation matrix in your Vertex shader.
I'm trying to animate something that spins / left to right, but whenever I call
spinLeft() or spinRight() then the animation always starts from frame 0.
In other words, I want to be able to say spin something 4 out of 10 frames, stop, then
spin in the opposite direction, FROM frame 4. Right now, it resets to frame 0.
var textures = [SKTexture]() // Loaded with 10 images later on.
var sprite = SKSpriteNode()
func spinLeft() {
let action = SKAction.repeatForever(.animate(with: textures, timePerFrame: 0.1))
sprite.run(action)
}
func spinRight() {
let action = SKAction.repeatForever(.animate(with: textures, timePerFrame: 0.1)).reversed()
sprite.run(action)
}
You could do this (Syntax may be a little off, but you get the point):
The key here is the .index(of: ... ) which will get you the index.
func spinUp() {
let index = textures.index(of: sprite.texture)
if index == textures.count - 1 {
sprite.texture = textures[0]
}
else {
sprite.texture = textures[index + 1]
}
}
func spinDown() {
let index = textures.index(of: sprite.texture)
if index == 0 {
sprite.texture = textures[textures.count - 1]
}
else {
sprite.texture = textures[index - 1]
}
}
func changeImage(_ isUp: Bool, _ amountOfTime: CGFloat) {
let wait = SKAction.wait(duration: amountOfTime)
if isUp {
run(wait) {
self.imageUp()
}
}
else {
run(wait) {
self.imageDown()
}
}
}
If you use something like a swipe gesture recognizer, you can use it's direction to set the isUp Bool value and the velocity of that swipe for the amountOfTime for the changeImage function.
The changeImage will only change the image once, so you will need to handle this somewhere else, or create another function if you want it to continuously spin or die off eventually.
Hope this helps!
I'm doing a small game in Swift 3 and SpriteKit. I want to do a collision with my character and a special object that increases my score in 1, but for some reason, when I detect the collision, the score increases in 2 or 3.
I'm removing from parent my SpriteKitNode but it seems that it doesn't work.
Here's my code:
func checkCollisionsObject(){
enumerateChildNodes(withName: "objeto") {node, _ in
let objeto = node as! SKSpriteNode
if objeto.frame.intersects(self.personaje.frame){
objeto.removeFromParent()
self.actualizarPoints()
//self.labelNivel.text = "Level: \(self.nivel)"
}
}
}
func actualizarPoints() {
self.pointsCounter += 1
points.text = "Points: \(pointsCounter)"
}
The problem is that the collision detection is happening at 60fps (pretty fast). So in that time multiple collision detections are occurring. You are just handling the first one.
I usually like to have a property on the object that I can trigger so that I know whether or not the object has collided, and set it so that it doesn't detect anymore collisions.
In your case the object is just a SKSpriteNode so you would have to set the property in userData or make the object a custom object and have the property in the custom object class
func checkCollisionsObject(){
enumerateChildNodes(withName: "objeto") {node, _ in
let objeto = node as! CustomObject
if objeto.frame.intersects(self.personaje.frame) && objeto.hasCollided == false {
objeto.hasCollided = true
objeto.removeFromParent()
self.actualizarPoints()
}
}
}
func actualizarPoints() {
self.pointsCounter += 1
points.text = "Points: \(pointsCounter)"
}
I have the following code in my main.swift:
let strategist = GKMinmaxStrategist()
strategist.gameModel = position
strategist.maxLookAheadDepth = 1
strategist.randomSource = nil
let move = strategist.bestMoveForActivePlayer()
...where position is an instance of my GKGameModel subclass Position. After this code is run, move is nil. bestMoveForPlayer(position.activePlayer!) also results in nil (but position.activePlayer! results in a Player object).
However,
let moves = position.gameModelUpdatesForPlayer(position.activePlayer!)!
results in a non-empty array of possible moves. From Apple's documentation (about bestMoveForPlayer(_:)):
Returns nil if the player is invalid, the player is not a part of the game model, or the player has no valid moves available.
As far as I know, none of this is the case, but the function still returns nil. What could be going on here?
If it can be of any help, here's my implementation of the GKGameModel protocol:
var players: [GKGameModelPlayer]? = [Player.whitePlayer, Player.blackPlayer]
var activePlayer: GKGameModelPlayer? {
return playerToMove
}
func setGameModel(gameModel: GKGameModel) {
let position = gameModel as! Position
pieces = position.pieces
ply = position.ply
reloadLegalMoves()
}
func gameModelUpdatesForPlayer(thePlayer: GKGameModelPlayer) -> [GKGameModelUpdate]? {
let player = thePlayer as! Player
let moves = legalMoves(ofPlayer: player)
return moves.count > 0 ? moves : nil
}
func applyGameModelUpdate(gameModelUpdate: GKGameModelUpdate) {
let move = gameModelUpdate as! Move
playMove(move)
}
func unapplyGameModelUpdate(gameModelUpdate: GKGameModelUpdate) {
let move = gameModelUpdate as! Move
undoMove(move)
}
func scoreForPlayer(thePlayer: GKGameModelPlayer) -> Int {
let player = thePlayer as! Player
var score = 0
for (_, piece) in pieces {
score += piece.player == player ? 1 : -1
}
return score
}
func isLossForPlayer(thePlayer: GKGameModelPlayer) -> Bool {
let player = thePlayer as! Player
return legalMoves(ofPlayer: player).count == 0
}
func isWinForPlayer(thePlayer: GKGameModelPlayer) -> Bool {
let player = thePlayer as! Player
return isLossForPlayer(player.opponent)
}
func copyWithZone(zone: NSZone) -> AnyObject {
let copy = Position(withPieces: pieces.map({ $0.1 }), playerToMove: playerToMove)
copy.setGameModel(self)
return copy
}
If there's any other code I should show, let me know.
You need to change the activePlayer after apply or unapply a move.
In your case that would be playerToMove.
The player whose turn it is to perform an update to the game model. GKMinmaxStrategist assumes that the next call to the applyGameModelUpdate: method will perform a move on behalf of this player.
and, of course:
Function applyGameModelUpdate Applies a GKGameModelUpdate to the game model, potentially resulting in a new activePlayer. GKMinmaxStrategist will call this method on a copy of the primary game model to speculate about possible future moves and their effects. It is assumed that calling this method performs a move on behalf of the player identified by the activePlayer property.
func applyGameModelUpdate(gameModelUpdate: GKGameModelUpdate) {
let move = gameModelUpdate as! Move
playMove(move)
//Here change the current Player
let player = playerToMove as! Player
playerToMove = player.opponent
}
The same goes for your unapplyGameModelUpdate implementation.
Also, keep special attention to your setGameModelimplementation as it should copy All data in your model. This includes activePlayer
Sets the data of another game model. All data should be copied over, and should not maintain any pointers to the copied game state. This is used by the GKMinmaxStrategist to process permutations of the game without needing to apply potentially destructive updates to the primary game model.
I had the same problem. Turns out, .activePlayer has to return one of the instances returned by .players. It's not enough to return a new instance with matching .playerId.
simple checklist:
GKMinmaxStrategist's .bestMove(for:) is called
GKMinmaxStrategist's .gameModel is set
GKGameModel's isWin(for:) does not return true before move
GKGameModel's isLoss(for:) does not return true before move
GKGameModel's gameModelUpdates(for:) does not return nil all the time
GKGameModel's score(for:) is implemented