Observer is causing lag in AVFoundation captureOutput method - swift

Question

I have quite a specific problem but hopefully someone can help me. I'm using AVFoundation to create a video camera with a live preview. I use AVCaptureVideoDataOutput to get individual frames and AVCaptureMetadataOutput to detect a face. I'm also using Dlib's facial landmarks predictor to show the landmark points on the users face and measure the interocular distance between their eyes. Finally I'm using AVAssetWriter so that a video can be recorded.
The view controller has an ellipse shape on it so the user knows where to put their face. When the interocular distance is between a certain distance I want the ellipse to turn blue so the user knows their face is in the right place.
At the minute I've achieved this by sending a notification from my SessionHandler class to the View Controller. This works, however it's causing the frames per second in the video to drop badly. I was getting 25fps (manually set by me) and now it's ranging between 8-16.
Is there another way to notify the view controller that the ellipse should be turned green?
Here's my code where the problem is occurring. I know there's a lot going on.
// MARK: AVCaptureVideoDataOutputSampleBufferDelegate, AVCaptureAudioOutputSampleBufferDelegate
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
if !currentMetadata.isEmpty {
let boundsArray = currentMetadata
.compactMap { $0 as? AVMetadataFaceObject }
.map { (faceObject) -> NSValue in
let convertedObject = output.transformedMetadataObject(for: faceObject, connection: connection)
return NSValue(cgRect: convertedObject!.bounds)
}
if user.hasDlib {
wrapper?.doWork(on: sampleBuffer, inRects: boundsArray)
// Get the interocular distance so face is the correct place in the oval
let interocularDistance = wrapper?.getEyeDistance()
//print("Interocular Distance: \(interocularDistance)")
if user.hasInterocularDistance {
if interocularDistance! < 240 || interocularDistance! > 315 {
let name = Notification.Name(rawValue: setRemoveGreenEllipse)
NotificationCenter.default.post(name: name, object: nil)
//print("face not correct distance")
if videoRecorder.isRecording {
eyeDistanceCounter += 1
//print(eyeDistanceCounter)
if eyeDistanceCounter == 30 {
cancelledByUser = false
cancelledByEyeDistance = true
videoRecorder.cancel()
eyeDistanceCounter = 0
}
}
} else {
//print("face correct distance")
eyeDistanceCounter = 0
let name = Notification.Name(rawValue: setGreenEllipse)
NotificationCenter.default.post(name: name, object: nil)
}
}
}
} else {
// Check if face is detected during recording. If it isn't, then cancel recording
if videoRecorder.isRecording {
noFaceCount += 1
if noFaceCount == 50 {
cancelledByUser = false
videoRecorder.cancel()
noFaceCount = 0
}
}
}
if layer.status == .failed {
layer.flush()
}
layer.enqueue(sampleBuffer)
let writable = videoRecorder.canWrite()
if writable {
if videoRecorder.sessionAtSourceTime == nil {
// Start Writing
videoRecorder.sessionAtSourceTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
videoRecorder.videoWriter.startSession(atSourceTime: videoRecorder.sessionAtSourceTime!)
print("video session started")
}
if videoRecorder.videoWriterInput.isReadyForMoreMediaData {
// write video buffer
videoRecorder.videoWriterInput.append(sampleBuffer)
//print("video buffering")
}
}
}

Answer 1

You could probably call the notification once per 30 frames, for example, instead of every frame.
You could also call the color changing function directly if it's in the same view controller. If not, you could define a delegate method and call it directly as oppose to sending notifications.