How do I maintain an 'image size' in the previewLayer similar to that of the camera app on the iPhone?
At the moment my AVCaptureVideoPreviewLayer is presenting the frames from the capture session fitted to the rectangle size of the of the preview layer. My preview layer dimension are quite small so I’d like to maintain the same sort of scale as the iPhone’s camera app and view whatever portion of the image can be seen within the bounds of the preview layer - effectively leaving same image size as the built in camera app but with a smaller field of view.
I’ve tried setting the preview layer’s contents to .resizeAspectFill but it still just fits the video output to the frame. I’ve also tried to change the videoScaleAndCropFactor and the AVCaptureDevice’s videoZoomFactor with no success.
// how I've set up my device:
let availableDevices = AVCaptureDevice.DiscoverySession(
deviceTypes: [.builtInWideAngleCamera],
mediaType: AVMediaType.video,
position: .back).devices
// the preview layer
let previewLayer = AVCaptureVideoPreviewLayer(session: self.captureSession)
self.previewLayer = previewLayer
self.previewLayer.frame = CGRect(x: 0, y: 0,
width: self.spineView.spine.frame.width,
height: self.spineView.spine.frame.height)
self.previewLayer.cornerRadius = 5.0
self.previewLayer.videoGravity = .resizeAspectFill
self.spineView.spine.layer.addSublayer(self.previewLayer)
Related
I have a swift based iPhone application (this tutorial https://www.kodeco.com/1163620-face-detection-tutorial-using-the-vision-framework-for-ios). It takes the camera feed and processes each frame using the vision API to find the landmarks on a face, and then draws and overlay on the video of the landmarks. All I was trying to do was take the position of a landmark and crop a rectangle around that position from the original image (after that I was going to run it through an ML model to determine some things). However, I have an issue translating the vision API landmark position back to the original image location to do the cropping. Below is hopefully the relevant portions of the code that show how I attempted to do this and failed (I pulled code from a number of functions/classes just to focus it on the problem).
Capture video frame
let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let ciimage = CIImage(cvImageBuffer: imageBuffer)
Find the Face Landmarks
let detectFaceRequest = VNDetectFaceLandmarksRequest(completionHandler: detectedFace)
sequenceHandler.perform([detectFaceRequest],on: imageBuffer,orientation: .leftMirrored)
Get the left Eye Pupil Location
let point = result.landmarks?.leftPupil?.pointsInImage(imageSize: ciimage.extent.size)
Draw the cropped image around the leftPupil
let cropped = ciimage?.cropped(to: CGRect(x: point.x-100, y: point.y-100, width: 200, height: 200))
let uicropped = UIImage(ciImage: cropped!)
uicropped.draw(at: CGPoint(x:100,y:100))
The issue is the cropped image is not positioned over the left pupil.
When I enable LivePhotoCapture on my AVCapturePhotoOutput and switch to builtInUltraWideCamera on my iPhone 12, I get a distorted image on the preview layer. The issue goes away if LivePhotoCapture is disabled.
This issue isn't reproducible on iPhone 13 Pro.
Tried to play with videoGravity settings, but no luck. Any tips are appreciated!
On my AVCapturePhotoOutput:
if self.photoOutput.isLivePhotoCaptureSupported {
self.photoOutput.isLivePhotoCaptureEnabled = true
}
Preview layer:
videoPreviewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
videoPreviewLayer.videoGravity = .resizeAspect
videoPreviewLayer.connection?.videoOrientation = .portrait
previewView.layer.addSublayer((videoPreviewLayer)!)
self.captureSession.startRunning()
self.videoPreviewLayer.frame = self.previewView.bounds
Result (the picture is mirrored, but it's not a problem, the problem is on the right and bottom edges of the picture):
My goal is to render an SCNScene off screen with a transparent background, as a PNG. Full reproducing project here.
It works, but when I enable jittering, the resulting render is semitransparent. In this example I pasted in the resulting PNG on top of an image with black squares, and you will notice that the black squares are in fact visible:
As you can see, the black boxes are visible through the 3D objects.
But, when I disable jittering, I get this:
As you can see, the black boxes are not visible.
I'm on Monterrey 12.1 (21C52). I'm testing the images in Preview and in Figma.
I'm using standard SDK features only. Here's what I do:
scene.background.contents = NSColor.clear
let snapshotRenderer = SCNRenderer(device: MTLCreateSystemDefaultDevice())
snapshotRenderer.pointOfView = sceneView.pointOfView
snapshotRenderer.scene = scene
snapshotRenderer.scene!.background.contents = NSColor.clear
snapshotRenderer.autoenablesDefaultLighting = true
// setting this to false does not make the image semi-transparent
snapshotRenderer.isJitteringEnabled = true
let size = CGSize(width: 1000, height: 1000)
let image = snapshotRenderer.snapshot(atTime: .zero, with: size, antialiasingMode: .multisampling16X)
let imageRep = NSBitmapImageRep(data: image.tiffRepresentation!)
let pngData = imageRep?.representation(using: .png, properties: [:])
try! pngData!.write(to: destination)
The docs for jittering says
Jittering is a process that SceneKit uses to improve the visual quality of a rendered scene. While the scene’s content is still, SceneKit moves the pointOfView location very slightly (by less than a pixel in projected screen space). It then composites images rendered after several such moves to create the final rendered scene, creating an antialiasing effect that smooths the edges of rendered geometry.
To me, that doesn't sound like something that is expected to produce semi-transparency?
I'm capturing the output of a playing video using AVPlayerItemVideoOutput.copyPixelBuffer
I'm able to convert the pixel buffer into a CIImage, then render it back into a pixel buffer again, and then an AVAssetWriter writes the buffer stream out to a new movie clip successfully.
The reason I'm converting to CIImage is I want to do some manipulation of each frame. (So far I don't understand how to manipulate pixel buffers directly).
In this case I want to overlay a "scribble" style drawing that the user does with their finger. While the video plays, they can draw over it. I'm capturing this drawing successfully into a CAShapeLayer.
The code below outputs just the overlay CAShapeLayer successfully. When I try to reincorporate the original frame by uncommenting the lines shown, the entire process bogs down drastically and drops from 60fps to an unstable 10fps or so on an iPhone 12. I get stable 60fps in all cases except when I uncomment that code.
What's the best way to incorporate the shape layer into this stream of pixel buffers in 60fps "real time"?
Note: some of this code is not finalized -- setting bounds correctly, etc. However this is not related to my question and I'm aware that has to be done. The rotation/translation are there to orient the shape layer -- this all works for now.
func addShapesToBuffer(buffer: CVPixelBuffer, shapeLayer: CAShapeLayer) -> CVPixelBuffer? {
let coreImage = CoreImage.CIImage.init(cvImageBuffer: buffer)
let newBuffer = getBuffer(from: coreImage)
CVPixelBufferLockBaseAddress(newBuffer!, [])
let rect = CGRect(origin: CGPoint.zero, size: CGSize(width: 800, height: 390))
shapeLayer.shouldRasterize = true
shapeLayer.rasterizationScale = UIScreen.main.scale
shapeLayer.backgroundColor = UIColor.clear.cgColor
let renderer = UIGraphicsImageRenderer(size: rect.size)
let uiImageDrawing = renderer.image {
context in
// let videoImage = UIImage(ciImage: coreImage)
// videoImage.draw(in: rect)
let cgContext = context.cgContext
cgContext.rotate(by: deg2rad(-90))
cgContext.translateBy(x: -390, y: 0)
return shapeLayer.render(in: cgContext)
}
let ciContext = CIContext()
let newImage = CIImage(cgImage: uiImageDrawing.cgImage!)
ciContext.render(_: newImage, to: newBuffer!)
CVPixelBufferUnlockBaseAddress(newBuffer!, [])
return newBuffer
}
I found similar threads like this two: Save what user see, Add layer on top But they didn't help.
The following section shows a PreviewLayer of the captureSession. Then I put a blue CGLayer on top of it. Everything is displayed correctly.
func showCameraOutput(in videoView: UIView) {
previewLayer = AVCaptureVideoPreviewLayer(session: self.captureSession)
previewLayer?.videoGravity = .resizeAspectFill
previewLayer?.frame = CGRect(x: 0, y: 0, width: videoView.frame.width, height: videoView.frame.height)
previewLayer?.connection?.videoOrientation = .landscapeRight
let blueLayer = CALayer()
blueLayer.backgroundColor = UIColor.blue.withAlphaComponent(0.5).cgColor
blueLayer.frame = CGRect(x: 0, y: 0, width: videoView.frame.width/2, height: videoView.frame.height)
previewLayer?.addSublayer(blueLayer)
videoView.layer.sublayers?.removeAll(where: { $0 is AVCaptureVideoPreviewLayer })
if let previewLayer = previewLayer {
videoView.layer.insertSublayer(previewLayer, at: 0)
}
}
When I save the video, the blue layer is no longer there.
func fileOutput(_ output: AVCaptureFileOutput, didFinishRecordingTo outputFileURL: URL, from connections: [AVCaptureConnection], error: Error?) {
UISaveVideoAtPathToSavedPhotosAlbum(outputFileURL.path, nil, nil, nil)
}
Is it not possible to save layers to a live video? Or is something else saved? How can I access this resource then?
I do not want to not edit the video afterwards. The goal is to put a layer on top of a live video and save it with it. Maybe I just need a hint for google.
In your setup, the video frames take two different routes through your app:
Camera (AVCaptureDeviceInput) → AVCaptureVideoPreviewLayer → Core Animation (CALayer composition and blending) → Screen
Camera (AVCaptureDeviceInput) → AVCaptureFileOutput (writing video frames to a file)
So path 2 doesn't know anything about what's happening on the Core Animation layer. It just writes the unmodified camera frames to a file.
In order to have a composition/blending phase that is visible in preview and output, you need a much more elaborate setup, I'm afraid. It would look something like this:
AVCaptureDeviceInput → AVCaptureVideoDataOutput → AVCaptureVideoDataOutputSampleBufferDelegate → CVPixelBuffer → Composition/Blending (probably with Core Image) → new, edited CVPixelBuffer
The new CVPixelBuffer will then go to two targets:
display with MTKView
AVAssetWriterInputPixelBufferAdaptor → AVAssetWriterInput → AVAssetWriter → video file
I hope this helps with googling. 🙂