I found similar threads like this two: Save what user see, Add layer on top But they didn't help.
The following section shows a PreviewLayer of the captureSession. Then I put a blue CGLayer on top of it. Everything is displayed correctly.
func showCameraOutput(in videoView: UIView) {
previewLayer = AVCaptureVideoPreviewLayer(session: self.captureSession)
previewLayer?.videoGravity = .resizeAspectFill
previewLayer?.frame = CGRect(x: 0, y: 0, width: videoView.frame.width, height: videoView.frame.height)
previewLayer?.connection?.videoOrientation = .landscapeRight
let blueLayer = CALayer()
blueLayer.backgroundColor = UIColor.blue.withAlphaComponent(0.5).cgColor
blueLayer.frame = CGRect(x: 0, y: 0, width: videoView.frame.width/2, height: videoView.frame.height)
previewLayer?.addSublayer(blueLayer)
videoView.layer.sublayers?.removeAll(where: { $0 is AVCaptureVideoPreviewLayer })
if let previewLayer = previewLayer {
videoView.layer.insertSublayer(previewLayer, at: 0)
}
}
When I save the video, the blue layer is no longer there.
func fileOutput(_ output: AVCaptureFileOutput, didFinishRecordingTo outputFileURL: URL, from connections: [AVCaptureConnection], error: Error?) {
UISaveVideoAtPathToSavedPhotosAlbum(outputFileURL.path, nil, nil, nil)
}
Is it not possible to save layers to a live video? Or is something else saved? How can I access this resource then?
I do not want to not edit the video afterwards. The goal is to put a layer on top of a live video and save it with it. Maybe I just need a hint for google.
In your setup, the video frames take two different routes through your app:
Camera (AVCaptureDeviceInput) → AVCaptureVideoPreviewLayer → Core Animation (CALayer composition and blending) → Screen
Camera (AVCaptureDeviceInput) → AVCaptureFileOutput (writing video frames to a file)
So path 2 doesn't know anything about what's happening on the Core Animation layer. It just writes the unmodified camera frames to a file.
In order to have a composition/blending phase that is visible in preview and output, you need a much more elaborate setup, I'm afraid. It would look something like this:
AVCaptureDeviceInput → AVCaptureVideoDataOutput → AVCaptureVideoDataOutputSampleBufferDelegate → CVPixelBuffer → Composition/Blending (probably with Core Image) → new, edited CVPixelBuffer
The new CVPixelBuffer will then go to two targets:
display with MTKView
AVAssetWriterInputPixelBufferAdaptor → AVAssetWriterInput → AVAssetWriter → video file
I hope this helps with googling. 🙂
Related
I am trying to record, and then save, a video in Swift using AVFoundation. This works. I am also trying to add an overlay, such as a text label containing the date, to the video.
For example: the video saved is not only what the camera sees, but the timestamp as well.
Here is how I am saving the video:
func fileOutput(_ output: AVCaptureFileOutput, didFinishRecordingTo outputFileURL: URL, from connections: [AVCaptureConnection], error: Error?) {
saveVideo(toURL: movieURL!)
}
private func saveVideo(toURL url: URL) {
PHPhotoLibrary.shared().performChanges({
PHAssetChangeRequest.creationRequestForAssetFromVideo(atFileURL: url)
}) { (success, error) in
if(success) {
print("Video saved to Camera Roll.")
} else {
print("Video failed to save.")
}
}
}
I have a movieOuput that is an AVCaptureMovieFileOutput. My preview layer does not contain any sublayers. I tried adding the timestamp label's layer to the previewLayer, but this did not succeed.
I have tried Ray Wenderlich's example as well as this stack overflow question. Lastly, I also tried this tutorial, all of which to no avail.
How can I add an overlay to my video that is in the saved video in the camera roll?
Without more information it sounds like what you are asking for is a WATERMARK.
Not an overlay.
A watermark is a markup on the video that will be saved with the video.
An overlay is generally showed as subviews on the preview layer and will not be saved with the video.
Check this out here: https://stackoverflow.com/a/47742108/8272698
func addWatermark(inputURL: URL, outputURL: URL, handler:#escaping (_ exportSession: AVAssetExportSession?)-> Void) {
let mixComposition = AVMutableComposition()
let asset = AVAsset(url: inputURL)
let videoTrack = asset.tracks(withMediaType: AVMediaType.video)[0]
let timerange = CMTimeRangeMake(kCMTimeZero, asset.duration)
let compositionVideoTrack:AVMutableCompositionTrack = mixComposition.addMutableTrack(withMediaType: AVMediaType.video, preferredTrackID: CMPersistentTrackID(kCMPersistentTrackID_Invalid))!
do {
try compositionVideoTrack.insertTimeRange(timerange, of: videoTrack, at: kCMTimeZero)
compositionVideoTrack.preferredTransform = videoTrack.preferredTransform
} catch {
print(error)
}
let watermarkFilter = CIFilter(name: "CISourceOverCompositing")!
let watermarkImage = CIImage(image: UIImage(named: "waterMark")!)
let videoComposition = AVVideoComposition(asset: asset) { (filteringRequest) in
let source = filteringRequest.sourceImage.clampedToExtent()
watermarkFilter.setValue(source, forKey: "inputBackgroundImage")
let transform = CGAffineTransform(translationX: filteringRequest.sourceImage.extent.width - (watermarkImage?.extent.width)! - 2, y: 0)
watermarkFilter.setValue(watermarkImage?.transformed(by: transform), forKey: "inputImage")
filteringRequest.finish(with: watermarkFilter.outputImage!, context: nil)
}
guard let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPreset640x480) else {
handler(nil)
return
}
exportSession.outputURL = outputURL
exportSession.outputFileType = AVFileType.mp4
exportSession.shouldOptimizeForNetworkUse = true
exportSession.videoComposition = videoComposition
exportSession.exportAsynchronously { () -> Void in
handler(exportSession)
}
}
And heres how to call the function.
let outputURL = NSURL.fileURL(withPath: "TempPath")
let inputURL = NSURL.fileURL(withPath: "VideoWithWatermarkPath")
addWatermark(inputURL: inputURL, outputURL: outputURL, handler: { (exportSession) in
guard let session = exportSession else {
// Error
return
}
switch session.status {
case .completed:
guard NSData(contentsOf: outputURL) != nil else {
// Error
return
}
// Now you can find the video with the watermark in the location outputURL
default:
// Error
}
})
Let me know if this code works for you.
It is in swift 3 so some changes will be needed.
I currently am using this code on an app of mine. Have not updated it to swift 5 yet
I do not have an actual development environment for Swift that can utilize AVFoundation. Thus, I can't provide you with any example code.
For adding meta data(date, location, timestamp, watermark, frame rate, etc...) as an overlay to the video while recording, you would have to process the video feed, frame by frame, live, while recording. Most likely you would have to store the frames in a buffer and process them before actually record them.
Now when it come to the meta data, there are two type, static and dynamic. For static type such as a watermark, it should be easy enough, as all the frames will get the same thing.
However, for dynamic meta data type such as timestamp or GPS location, there are a few things that needed to be taken into consideration. It takes computational power and time to process the video frames. Thus, depends on the type of dynamic data and how you got them, sometime the processed value may not be a correct value. For example, if you got a frame at 1:00:01, you process it and add a timestamp to it. Just pretend that it took 2 seconds to process the timestamp. The next frame you got is at 1:00:02, but you couldn't process it until 1:00:03 because processing the previous frame took 2 seconds. Thus, depend on how you got that new timestamp for the new frame, that timestamp value may not be the value that you wanted.
For processing dynamic meta data, you should also take into consideration of hardware lag. For example, the software is supposed to add live GPS location data to each frame and there weren't any lags in development or in testing. However, in real life, a user used the software in an area with a bad connection, and his phone lag while obtaining his GPS location. Some of his lags lasted as long as 5 seconds. What do you do in that situation? Do you set a time out for the GPS location and used the last good position? Do you report the error? Do you defer that frame to be process later when the GPS data become available(This may ruin live recording) and using an expensive algorithm to try to predict the user's location for that frame?
Besides those to take into consideration, I have some references here that I think may help you. I thought the one from medium.com looked pretty good.
https://medium.com/ios-os-x-development/ios-camera-frames-extraction-d2c0f80ed05a
Adding watermark to currently recording video and save with watermark
Render dynamic text onto CVPixelBufferRef while recording video
Adding on to #Kevin Ng, you can do an overlay on video frames with an UIViewController and an UIView.
UIViewController will have:
property to work with video stream
private var videoSession: AVCaptureSession?
property to work with overlay(the UIView class)
private var myOverlay: MyUIView{view as! MyUIView}
property to work with video output queue
private let videoOutputQueue = DispatchQueue(label:
"outputQueue", qos: .userInteractive)
method to create video session
method to process and display overlay
UIView will have task-specific helper methods needed to to act as overlay. For example, if you are doing hand detection, this overlay class can have helper methods to draw points on coordinates(ViewController class will detect coordinates of hand features, do necessary coordinate conversions, then pass the coordinates to the UIView class to display coordinates as an overlay)
I successfully integrated the Vuforia SDK Image Target Tracking feature into an iOS project by combining the OpenGL context (EAGLContext) that the SDK provides, with an instance of SceneKit's SCNRenderer. That allowed me to leverage the simplicity of the SceneKit's 3D API and at the same time benefiting from Vuforia's high precision image detection. Now, I'd like to do the same by replacing OpenGL with Metal.
Some background story
I was able to draw SceneKit objects on top of the live video texture drawn by Vuforia using OpenGL without major problems.
Here's the simplified setup I used with OpenGL:
func configureRenderer(for context: EAGLContext) {
self.renderer = SCNRenderer(context: context, options: nil)
self.scene = SCNScene()
renderer.scene = scene
// other scenekit setup
}
func render() {
// manipulate scenekit nodes
renderer.render(atTime: CFAbsoluteTimeGetCurrent())
}
Apple deprecates OpenGL on iOS 12
Since Apple announced that it is deprecating OpenGL on iOS 12, I figured it would be a good idea to try to migrate this project to use the Metal instead of OpenGL.
That should be simple in theory as Vuforia supports Metal out of the box. However, when trying to integrate it, I hit the wall.
The question
The view seems to ever only render results of the SceneKit renderer, or the textures encoded by Vuforia, but never both at the same time. It depends what is encoded first. What do I have to do to blend both results togeter?
Here's the problematic setup in a nutshell:
func configureRenderer(for device: MTLDevice) {
let renderer = SCNRenderer(device: device, options: nil)
self.scene = SCNScene()
renderer.scene = scene
// other scenekit setup
}
func render(viewport: CGRect, commandBuffer: MTLCommandBuffer, drawable: CAMetalDrawable) {
// manipulate scenekit nodes
let renderPassDescriptor = MTLRenderPassDescriptor()
renderPassDescriptor.colorAttachments[0].texture = drawable.texture
renderPassDescriptor.colorAttachments[0].loadAction = .load
renderPassDescriptor.colorAttachments[0].storeAction = .store
renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColor(red: 0.0, green: 0, blue: 0, alpha: 0)
renderer!.render(withViewport: viewport, commandBuffer: commandBuffer, passDescriptor: renderPassDescriptor)
}
I tried calling render either after encoder.endEncoding or before the commandBuffer.renderCommandEncoderWithDescriptor :
metalDevice = MTLCreateSystemDefaultDevice();
metalCommandQueue = [metalDevice newCommandQueue];
id<MTLCommandBuffer>commandBuffer = [metalCommandQueue commandBuffer];
//// -----> call the `render(viewport:commandBuffer:drawable) here <------- \\\\
id<MTLRenderCommandEncoder> encoder = [commandBuffer renderCommandEncoderWithDescriptor:renderPassDescriptor];
// calls to encoder to render textures from Vuforia
[encoder endEncoding];
//// -----> or here <------- \\\\
[commandBuffer presentDrawable:drawable];
[commandBuffer commit];
In either case, I only see results of SCNRenderer OR results of the encoder, but never both in the same view.
It seems to me as if the encoding pass above, and the SCNRenderer.render, are overwriting each other's buffers.
What am I missing here?
I think I've found an answer.
I am rendering scnrenderer after endEncoding, but I'm creating a new descriptor.
// Pass Metal context data to Vuforia Engine (we may have changed the encoder since
// calling Vuforia::Renderer::begin)
finishRender(UnsafeMutableRawPointer(Unmanaged.passRetained(drawable!.texture).toOpaque()), UnsafeMutableRawPointer(Unmanaged.passRetained(encoder!).toOpaque()))
// ========== Finish Metal rendering ==========
encoder?.endEncoding()
// Commit the rendering commands
// Command completed handler
commandBuffer?.addCompletedHandler { _ in self.mCommandExecutingSemaphore.signal()}
let screenSize = UIScreen.main.bounds.size
let newDescriptor = MTLRenderPassDescriptor()
// Draw to the drawable's texture
newDescriptor.colorAttachments[0].texture = drawable?.texture
// Store the data in the texture when rendering is complete
newDescriptor.colorAttachments[0].storeAction = MTLStoreAction.store
// Use textureDepth for depth operations.
newDescriptor.depthAttachment.texture = mDepthTexture;
renderer?.render(atTime: 0, viewport: CGRect(x: 0, y: 0, width: screenSize.width, height: screenSize.height), commandBuffer: commandBuffer!, passDescriptor: newDescriptor)
// Present the drawable when the command buffer has been executed (Metal
// calls to CoreAnimation to tell it to put the texture on the display when
// the rendering is complete)
commandBuffer?.present(drawable!)
// Commit the command buffer for execution as soon as possible
commandBuffer?.commit()
The project I am working on takes an h264-encoded bitstream and displays the video frames.
The decoder––which takes raw bytes and outputs a CMSampleBuffer containing a video frame––is working well. It's formatDescription contains the expected values, it correctly interprets NALU types, all that. I may be wrong though, as this is my first time working with all of this, so if anyone suspects that the problem may be in the decoder I would be happy to provide code.
My problem arises when I pass a CMSampleBuffer to an AVSampleBufferDisplayLayer. The video frames aren't appearing on screen.
This is the function that is supposed to display the video frames:
func videoFrameReceived(_ frame: CMSampleBuffer) {
videoLayer.enqueue(frame)
DispatchQueue.main.async { [weak self] in
self?.videoLayer.setNeedsDisplay()
}
}
This is how I initialized my AVSampleBufferDisplayLayer:
if let layer = videoLayer {
layer.frame = CGRect(x: 0, y: 150, width: 640, height: 480)
layer.videoGravity = .resizeAspect
let cmTimebasePointer = UnsafeMutablePointer<CMTimebase?>.allocate(capacity: 1)
let status = CMTimebaseCreateWithMasterClock(kCFAllocatorDefault, CMClockGetHostTimeClock(), cmTimebasePointer)
layer.controlTimebase = cmTimebasePointer.pointee
if let controlTimeBase = layer.controlTimebase, status == noErr {
CMTimebaseSetTime(controlTimeBase, kCMTimeZero)
CMTimebaseSetRate(controlTimeBase, 1.0)
}
self.layer.addSublayer(layer)
}
Thanks!
User figured out how to do it and posted a link to the solution as a comment to the question. The link was broken,
here is the updated link to solution
How do I maintain an 'image size' in the previewLayer similar to that of the camera app on the iPhone?
At the moment my AVCaptureVideoPreviewLayer is presenting the frames from the capture session fitted to the rectangle size of the of the preview layer. My preview layer dimension are quite small so I’d like to maintain the same sort of scale as the iPhone’s camera app and view whatever portion of the image can be seen within the bounds of the preview layer - effectively leaving same image size as the built in camera app but with a smaller field of view.
I’ve tried setting the preview layer’s contents to .resizeAspectFill but it still just fits the video output to the frame. I’ve also tried to change the videoScaleAndCropFactor and the AVCaptureDevice’s videoZoomFactor with no success.
// how I've set up my device:
let availableDevices = AVCaptureDevice.DiscoverySession(
deviceTypes: [.builtInWideAngleCamera],
mediaType: AVMediaType.video,
position: .back).devices
// the preview layer
let previewLayer = AVCaptureVideoPreviewLayer(session: self.captureSession)
self.previewLayer = previewLayer
self.previewLayer.frame = CGRect(x: 0, y: 0,
width: self.spineView.spine.frame.width,
height: self.spineView.spine.frame.height)
self.previewLayer.cornerRadius = 5.0
self.previewLayer.videoGravity = .resizeAspectFill
self.spineView.spine.layer.addSublayer(self.previewLayer)
I am new to swift and I am trying to make a Mac OS app that loops a video from the app's resources using AVPlayer as the background of the window once the app has been launched. When the user selects a menu item/clicks a button the background video will instantly change to a different video from the app's resources and start looping that video as the window's background.
I was able to play the first video once the app launches following this tutorial: (https://youtu.be/QgeQc587w70) and I also successfully made the video loop itself seamlessly following this post: (Looping AVPlayer seamlessly).
The problem I am now facing is changing the video to the other one once a menu item was selected/a button was clicked. The approach I was going for is to change the url and create a new AVPlayer using the new URL and affect it to the playerView.player following this post: (Swift How to update url of video in AVPlayer when clicking on a button?). However every time the menu item is selected the app crashes with the error "thread 1 exc_bad_instruction (code=exc_i386_invop subcode=0x0)". This is apparently caused by the value of playerView being nil. I don't really understand the reason for this as playerView is an AVPlayerView object that I created using the xib file and linked to the swift file by control-dragging and I couldn't seem to find another appropriate method of doing the thing I wanted to do. If you know the reason for this and the way of fixing it please provide me some help or if you know a better method of doing what I've mention above please tell me as well. Any help would be much appreciated!
My code is as follow, the line that crashes the app is at the bottom:
import Cocoa
import AppKit
import AVKit
import AVFoundation
struct videoVariables {
static var videoName = "Test_Video" //declaring the video name as a global variable
}
var videoIsPlaying = true
var theURL = Bundle.main.url(forResource:videoVariables.videoName, withExtension: "mp4") //creating the video url
var player = AVPlayer.init(url: theURL!)
class BackgroundWindow: NSWindowController {
#IBOutlet weak var playerView: AVPlayerView! // AVPlayerView Linked using control-drag from xib file
#IBOutlet var mainWindow: NSWindow!
#IBOutlet weak var TempBG: NSImageView!
override var windowNibName : String! {
return "BackgroundWindow"
}
//function used for resizing the temporary background image and the playerView to the window’s size
func resizeBG() {
var scrn: NSScreen = NSScreen.main()!
var rect: NSRect = scrn.frame
var height = rect.size.height
var width = rect.size.width
TempBG.setFrameSize(NSSize(width: Int(width), height: Int(height)))
TempBG.frame.origin = CGPoint(x: 0, y: 0)
playerView!.setFrameSize(NSSize(width: Int(width), height: Int(height)))
playerView!.frame.origin = CGPoint(x: 0, y: 0)
}
override func windowDidLoad() {
super.windowDidLoad()
self.window?.titleVisibility = NSWindowTitleVisibility.hidden //hide window’s title
self.window?.styleMask = NSBorderlessWindowMask //hide window’s border
self.window?.hasShadow = false //hide window’s shadow
self.window?.level = Int(CGWindowLevelForKey(CGWindowLevelKey.desktopWindow)) //set window’s layer as desktopWindow layer
self.window?.center()
self.window?.makeKeyAndOrderFront(nil)
NSApp.activate(ignoringOtherApps: true)
if let screen = NSScreen.main() {
self.window?.setFrame(screen.visibleFrame, display: true, animate: false) //resizing the window to cover the whole screen
}
resizeBG() //resizing the temporary background image and the playerView to the window’s size
startVideo() //start playing and loop the first video as the window’s background
}
//function used for starting the video again once it has been played fully
func playerItemDidReachEnd(notification: NSNotification) {
playerView.player?.seek(to: kCMTimeZero)
playerView.player?.play()
}
//function used for starting and looping the video
func startVideo() {
//set the seeking time to be 2ms ahead to prevent a black screen every time the video loops
let playAhead = CMTimeMake(2, 100);
//loops the video
NotificationCenter.default.addObserver(forName: .AVPlayerItemDidPlayToEndTime, object:
playerView.player?.currentItem, queue: nil, using: { (_) in
DispatchQueue.main.async {
self.playerView.player?.seek(to: playAhead)
self.playerView.player?.play()
}
})
var playerLayer: AVPlayerLayer?
playerLayer = AVPlayerLayer(player: player)
playerView?.player = player
print(playerView?.player)
playerLayer?.videoGravity = AVLayerVideoGravityResizeAspectFill
player.play()
}
//changing the url to the new url and create a new AVPlayer then affect it to the playerView.player once the menu item is being selected
#IBAction func renderBG(_ sender: NSMenuItem) {
videoVariables.videoName = "Test_Video_2"
var theNewURL = Bundle.main.url(forResource:videoVariables.videoName, withExtension: "mp4")
player = AVPlayer.init(url: theNewURL!)
//!!this line crashes the app with the error "thread 1 exc_bad_instruction (code=exc_i386_invop subcode=0x0)" every time the menu item is being selected!!
playerView.player = player
}
}
Additionally, the background video is not supposed to be interactive(E.g. User cannot pause/ fast-forward the video), so any issues that might be caused by user interactivity can be ignored. The purpose of the app is to play a video on the user's desktop creating the exact same effect of running the command:
"/System/Library/Frameworks/ScreenSaver.framework/Resources/
ScreenSaverEngine.app/Contents/MacOS/ScreenSaverEngine -background" in terminal.
Any help would be much appreciated!
You don't need to create AVPlayer from url. There is AVPlayerItem class to manipulate player playback queue.
let firstAsset = AVURLAsset(url: firstVideoUrl)
let firstPlayerItem = AVPlayerItem(asset: firstAsset)
let player = AVPlayer(playerItem: firstPlayerItem)
let secondAsset = AVURLAsset(url: secondVideoUrl)
let secondPlayerItem = AVPlayerItem(asset: secondAsset)
player.replaceCurrentItem(with: secondPlayerItem)
Docs about AVPlayerItem