I was experimenting with Open AI's ChatGPT and when asked to give me a code for playing a video in a RealityKit's AR scene when reference image is tracked it used ARVideoNode instead of my expected AVPlayer and VideoMaterial solution. It even gave me an answer why ARVideoNode is better than AVPlayer on a VideoMaterial when I asked, but I never heard of ARVideoNode in RealityKit.
Am I missing something or is it just a flaw in the AI?
import RealityKit
// Set up image tracking
let configuration = ARImageTrackingConfiguration()
configuration.detectionImages = ["reference-image-1", "reference-image-2"]
arView.session.run(configuration)
// Create a dictionary to map reference image names to video file names
let videoFileNames = ["reference-image-1": "video-1.mp4",
"reference-image-2": "video-2.mp4"]
// Track the reference images and display the corresponding
// videos on top of them
var videoNodes = [String: ARVideoNode]()
arView.scene.subscribe(to: ARImageAnchor.self) { (anchor: ARImageAnchor) in
// Get the video file name for the tracked image
guard let videoFileName = videoFileNames[anchor.name] else { return }
// Load the video file
let videoURL = URL(fileURLWithPath: "path/to/\(videoFileName)")
let videoAsset = VideoAsset(url: videoURL)
// Create an ARVideoNode and add it to the scene
let videoNode = ARVideoNode(asset: videoAsset)
arView.scene.anchors.append(videoNode)
videoNodes[anchor.name] = videoNode
// Position the video node on top of the image
videoNode.transform = anchor.transform
// Play the video
videoNode.play()
}
// Monitor the tracking status of the reference images
// and pause/resume the videos as needed
arView.scene.subscribe(to: ARImageAnchor.self) { (anchor: ARImageAnchor) in
// Get the video node for the tracked image
guard let videoNode = videoNodes[anchor.name] else { return }
if anchor.isTracked {
// Resume playing the video if the image is being tracked
videoNode.play()
} else {
// Pause the video if the image is not being tracked
videoNode.pause()
}
}
It's not a flaw, it's rather ChatGPT's wrong answer. As far as I know, Kudan's ARVideoNode is a subclass of an ARNode parent class that is used to render video content. It has nothing to do with RealityKit even in terms of programming language - KudanAR SDK natively uses Objective-C for iOS and Java for Android.
Here's SO temporary policy regarding ChatGPT:
Overall, because the average rate of getting correct answers from ChatGPT is too low, the posting of answers created by ChatGPT is substantially harmful to the site and to users who are asking and looking for correct answers.
I have an AV player which streams video from url, I want to show thumbnail of video while dragging slider in forward or backward direction(like feature in youtube). can you explain how to add this feature using swift.
LIKE THIS
https://drive.google.com/file/d/1g9ngVJbhTDvWABBt49T-QrSfA354bERa/view
If your playlist files already support the EXT-X-IMAGE-STREAM-INF tag and you are using the standard player controller (AVPlayerViewController) then you will have that feature for free. But if you are using customized UI for your player or your playlist doesn't support that tag, you have to build that feature by your self. It's possible because all you have to do is just display correct frames on top of the AVPlayer layer in proper positions.
if fileUrl.containsVideo {
let generateThumbnail = (try? AVAssetImageGenerator(asset: asset).copyCGImage(at: .init(seconds: 30, preferredTimescale: 60), actualTime: nil)) ?? UIImage(systemName: "music.note")?.cgImage!
img = UIImage(cgImage: generateThumbnail!)
itemtitle = fileUrl.lastPathComponent
}
I am trying to record, and then save, a video in Swift using AVFoundation. This works. I am also trying to add an overlay, such as a text label containing the date, to the video.
For example: the video saved is not only what the camera sees, but the timestamp as well.
Here is how I am saving the video:
func fileOutput(_ output: AVCaptureFileOutput, didFinishRecordingTo outputFileURL: URL, from connections: [AVCaptureConnection], error: Error?) {
saveVideo(toURL: movieURL!)
}
private func saveVideo(toURL url: URL) {
PHPhotoLibrary.shared().performChanges({
PHAssetChangeRequest.creationRequestForAssetFromVideo(atFileURL: url)
}) { (success, error) in
if(success) {
print("Video saved to Camera Roll.")
} else {
print("Video failed to save.")
}
}
}
I have a movieOuput that is an AVCaptureMovieFileOutput. My preview layer does not contain any sublayers. I tried adding the timestamp label's layer to the previewLayer, but this did not succeed.
I have tried Ray Wenderlich's example as well as this stack overflow question. Lastly, I also tried this tutorial, all of which to no avail.
How can I add an overlay to my video that is in the saved video in the camera roll?
Without more information it sounds like what you are asking for is a WATERMARK.
Not an overlay.
A watermark is a markup on the video that will be saved with the video.
An overlay is generally showed as subviews on the preview layer and will not be saved with the video.
Check this out here: https://stackoverflow.com/a/47742108/8272698
func addWatermark(inputURL: URL, outputURL: URL, handler:#escaping (_ exportSession: AVAssetExportSession?)-> Void) {
let mixComposition = AVMutableComposition()
let asset = AVAsset(url: inputURL)
let videoTrack = asset.tracks(withMediaType: AVMediaType.video)[0]
let timerange = CMTimeRangeMake(kCMTimeZero, asset.duration)
let compositionVideoTrack:AVMutableCompositionTrack = mixComposition.addMutableTrack(withMediaType: AVMediaType.video, preferredTrackID: CMPersistentTrackID(kCMPersistentTrackID_Invalid))!
do {
try compositionVideoTrack.insertTimeRange(timerange, of: videoTrack, at: kCMTimeZero)
compositionVideoTrack.preferredTransform = videoTrack.preferredTransform
} catch {
print(error)
}
let watermarkFilter = CIFilter(name: "CISourceOverCompositing")!
let watermarkImage = CIImage(image: UIImage(named: "waterMark")!)
let videoComposition = AVVideoComposition(asset: asset) { (filteringRequest) in
let source = filteringRequest.sourceImage.clampedToExtent()
watermarkFilter.setValue(source, forKey: "inputBackgroundImage")
let transform = CGAffineTransform(translationX: filteringRequest.sourceImage.extent.width - (watermarkImage?.extent.width)! - 2, y: 0)
watermarkFilter.setValue(watermarkImage?.transformed(by: transform), forKey: "inputImage")
filteringRequest.finish(with: watermarkFilter.outputImage!, context: nil)
}
guard let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPreset640x480) else {
handler(nil)
return
}
exportSession.outputURL = outputURL
exportSession.outputFileType = AVFileType.mp4
exportSession.shouldOptimizeForNetworkUse = true
exportSession.videoComposition = videoComposition
exportSession.exportAsynchronously { () -> Void in
handler(exportSession)
}
}
And heres how to call the function.
let outputURL = NSURL.fileURL(withPath: "TempPath")
let inputURL = NSURL.fileURL(withPath: "VideoWithWatermarkPath")
addWatermark(inputURL: inputURL, outputURL: outputURL, handler: { (exportSession) in
guard let session = exportSession else {
// Error
return
}
switch session.status {
case .completed:
guard NSData(contentsOf: outputURL) != nil else {
// Error
return
}
// Now you can find the video with the watermark in the location outputURL
default:
// Error
}
})
Let me know if this code works for you.
It is in swift 3 so some changes will be needed.
I currently am using this code on an app of mine. Have not updated it to swift 5 yet
I do not have an actual development environment for Swift that can utilize AVFoundation. Thus, I can't provide you with any example code.
For adding meta data(date, location, timestamp, watermark, frame rate, etc...) as an overlay to the video while recording, you would have to process the video feed, frame by frame, live, while recording. Most likely you would have to store the frames in a buffer and process them before actually record them.
Now when it come to the meta data, there are two type, static and dynamic. For static type such as a watermark, it should be easy enough, as all the frames will get the same thing.
However, for dynamic meta data type such as timestamp or GPS location, there are a few things that needed to be taken into consideration. It takes computational power and time to process the video frames. Thus, depends on the type of dynamic data and how you got them, sometime the processed value may not be a correct value. For example, if you got a frame at 1:00:01, you process it and add a timestamp to it. Just pretend that it took 2 seconds to process the timestamp. The next frame you got is at 1:00:02, but you couldn't process it until 1:00:03 because processing the previous frame took 2 seconds. Thus, depend on how you got that new timestamp for the new frame, that timestamp value may not be the value that you wanted.
For processing dynamic meta data, you should also take into consideration of hardware lag. For example, the software is supposed to add live GPS location data to each frame and there weren't any lags in development or in testing. However, in real life, a user used the software in an area with a bad connection, and his phone lag while obtaining his GPS location. Some of his lags lasted as long as 5 seconds. What do you do in that situation? Do you set a time out for the GPS location and used the last good position? Do you report the error? Do you defer that frame to be process later when the GPS data become available(This may ruin live recording) and using an expensive algorithm to try to predict the user's location for that frame?
Besides those to take into consideration, I have some references here that I think may help you. I thought the one from medium.com looked pretty good.
https://medium.com/ios-os-x-development/ios-camera-frames-extraction-d2c0f80ed05a
Adding watermark to currently recording video and save with watermark
Render dynamic text onto CVPixelBufferRef while recording video
Adding on to #Kevin Ng, you can do an overlay on video frames with an UIViewController and an UIView.
UIViewController will have:
property to work with video stream
private var videoSession: AVCaptureSession?
property to work with overlay(the UIView class)
private var myOverlay: MyUIView{view as! MyUIView}
property to work with video output queue
private let videoOutputQueue = DispatchQueue(label:
"outputQueue", qos: .userInteractive)
method to create video session
method to process and display overlay
UIView will have task-specific helper methods needed to to act as overlay. For example, if you are doing hand detection, this overlay class can have helper methods to draw points on coordinates(ViewController class will detect coordinates of hand features, do necessary coordinate conversions, then pass the coordinates to the UIView class to display coordinates as an overlay)
Is it possible to make peview image while listing video timeline, like in Netflix App with AVPlayer?
exmple here
I found that mp4 video has that preview window, but m3u8 format has not it
func playVideo(cell: MovieCell) {
guard let urlVideo = URL(string: cell.movieSrc) else { return }
// Create an AVPlayer, passing it the HTTP Live Streaming URL.
let player = AVPlayer(url: urlVideo)
// Create a new AVPlayerViewController and pass it a reference to the player.
let controller = AVPlayerViewController()
controller.player = player
// Modally present the player and call the player's play() method when complete.
self.present(controller, animated: true) {
player.play()
}
}
This is called "Trick Play" and must be included in the HLS manifest for AVPlayer to handle this for you.
See the Trick Play section in HLS Authoring Specification for Apple Devices:
6.1. I-frame playlists (EXT-X-I-FRAME-STREAM-INF) MUST be provided to support scrubbing and scanning UI.
I am creating iMessage app and trying to send audio or video file to other user.
Video file works and looks fine but its not working as expected with audio file.
My current code is:
let destinationFilename = mp3FileNames[i]
let destinationURL = docDirectoryURL.appendingPathComponent(destinationFilename)
if let conversation = activeConversation {
let layout = MSMessageTemplateLayout()
layout.image = UIImage.init(named: "audio-x-generic-icon")
layout.mediaFileURL = destinationURL
layout.caption = selectedSongObj.name
let message = MSMessage()
message.layout = layout
message.url = URL(string: "emptyURL")
conversation.insert(message, completionHandler: nil)
return
}
Looks like layout.mediaFileURL = destinationURL is not adding any file into message.
And when I try to send file with above code.It looks like shown below:
It looks fine but there is no audio to play but if I try this way:
let destinationFilename = mp3FileNames[i]
let destinationURL = docDirectoryURL.appendingPathComponent(destinationFilename)
if let conversation = activeConversation {
conversation.insertAttachment(destinationURL!, withAlternateFilename: nil, completionHandler: nil)
return
}
And result with above code is:
I can play audio for that message because it's there. But problem with that message is I can not attach any image or caption with it.
How can I attach image and audio file into same message.
And if possible instead of image can I add GIF?
Any help would be much appreciated, Thank you.
Not necessary to use GIF, iMessage extensions supports also PNGand JPEG image formats. Recommended image size is 300x300 points at #3x scale.
If the MSMessageTemplateLayout's image property has a non-nil value then
mediaFileURL property is ignored. So you can't send an image and an audio file at the same time. Docs