How Can I delay the stream to UIImageview using AVCaptureVideoPreviewLayer from camera?
See below how I bind them, but I just can't figure how to delay it (I don't want it in real time)
AVCaptureVideoPreviewLayer* captureVideoPreviewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:session];
captureVideoPreviewLayer.frame = self.imageView.bounds;
[self.imageView.layer addSublayer:captureVideoPreviewLayer];
First you're going to want to remove the preview layer frame you have right now, as there is no method to delay those preview frames out of the box.
You're going to want to create a buffer. If we're talking about a few frames, you could have a NSMutableArray that you're filling up on one end with UIImages while you're feeding your image view from the other end.
Your UIImage would come from the didOutputSampleBuffer method, use something like this UIImage created from CMSampleBufferRef not displayed in UIImageView?
Now, few challenges you will have to deal with:
you're talking about having a multiple seconds delay, 5 seconds would be about 150 frames. Storing 150 UIImage in memory isn't gonna happen, unless they're very tiny and on the latest devices
You would solve that by saving the images to disk and have your array only store the path of those images instead of the images themselves. Now you're probably going to run into performance issues, as you're going to do read/write operations in real time, your framerate is going to suffer from that
Because of that bad frame rate, you're going to have to make sure you're not losing synchronization between recorded feed and live feed, otherwise you'll start with a 5 sec delay and end up with a much longer delay
Good luck with that, it can be done with some trade-off (slower frame rate...) but it can be done. (I have done something very similar myself multiple times, can't share the code for IP reasons).
Related
I am Creating application for coaching. I struck with the marking on video. So I choose ffmpeg for converting video to image frame. That make me time delay as well as memory issues. I need to provide the user play the video slowly as frame by frame. Is there any other way to do that with out image conversion. V1 Golf did that process very quick manner. Please help me.
I would try converting video frame in separate thread and I would extract a few frames ahead as images in the background when user gets into 'slow motion mode'.
Here is example for one frame, so you should be quick with others: Video frame capture by NSOperation.
This should reduce delays and frames could be converted while user is eye-consuming subsequent frames.
Good day, hope someone can help me up with this situation:
I working on an iPhone app that takes series of images, assisted by gyroscope.
So both AVCamCaptureManager and CMMotionManager sessions are running at the same time.
after taking a still image, i am:
- processing the image in a background thread (which works fine without affecting anything)
- then saving processed image data to disk
[imageData writeToFile:imagePath atomically:YES];
The issue: both AVCamCaptureManager and CMMotionManager sessions freeze for less then 1/2 second, right after initiating writeToFile function.
Does anyone have any experience with such scenario?
Thanks for your time! :)
It appears that saving to disk does not affect sessions.
I am also setting UIImageView.image to a large image in the end of my routine, and this is what was freezing everything for 1/2 second.
I would like to render multiple H264 mp4 videos on multiple views at the same time. Target is to read about 8 short videos, each at a size of 100x100 pixels and let them display their content on multiple positions on the screen, simultaneously.
Imagine 24 squares on the screen, each showing one video out of pool of 8 videos.
MoviePlayer doesn't work, for it's only showing one fullscreen video. An AVPlayer with multiple AVPlayerLayers is limited, because only the most-recently-created Layer will show it's content on screen (according to the documentation and my testing).
So, i wrote a short video class and created an instance for every .mp4 file in my package, using AVAssetReader to read it's content. On update, every videoframe is retreived converted to an UIImage and displayed, according to the video's framerate. Furthermore, these images are cached for a fast access on looping.
- (id) initWithAsset:(AVURLAsset*)asset withTrack:(AVAssetTrack*)track
{
self = [super init];
if (self)
{
NSDictionary* settings = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA], (NSString*)kCVPixelBufferPixelFormatTypeKey, nil];
mOutput = [[AVAssetReaderTrackOutput alloc] initWithTrack:track outputSettings:settings];
mReader = [[AVAssetReader alloc] initWithAsset:asset error:nil];
[mReader addOutput:mOutput];
BOOL status = [mReader startReading];
}
return self;
}
- (void) update:(double)elapsed
{
CMSampleBufferRef buffer = [mOutput copyNextSampleBuffer];
if (buffer)
{
UIImage* image = [self imageFromSampleBuffer:sampleBuffer];
}
[...]
}
Actually this works pretty well, but only for 4 videos. The fifth one never shows up. First I thought of memory issues, but I tested it on the following devices:
iPhone 3GS
iPhone 4
iPad
iPad 2
I had the same behaviour on each device: 4 videos playing in a smooth loop, no differences.
If it would have been a memory issue, I would have expect at least either the iPad 2 to show 5 or 6 videos (due to it's better hardware) or the 3GS to show only 1 or a crash somewhere.
The simulator shows all videos, though.
Debugging on the device shows, that
BOOL status = [mReader startReading];
returns false for video 5,6,7 and 8.
So, is there some kind of hardware setting (or restriction) that doesn't allow more than 4 simultaneous AVAssetReaders? Because, I can't really explain this behaviour. I don't think that all devices have the exact same amount of video memory.
Yes, iOS has an upper limit on the number of videos that can be decoded at one time. While your approach is good, I don't know of any way to work around this upper limit as far as having that many h.264 decoders active at once. If you are interested, please have a look at my solution to this problem, this is an xcode project called Fireworks. Basically, this demo shows decoding a bunch of alpha channel videos to disk, then each one is played by mapping a portion of the video files into memory. This approach makes it possible to decode more than 4 movies at the same time without using up all the system memory and without running into the hard limit of the number of h.264 decoder objects.
Have you tried creating separate AVPlayerItems based on the same AVAsset for each AVPlayerLayer?
Here's my latest iteration of a perfectly smooth-scrolling collection view with real-time video previews (up to 16 at a time):
https://youtu.be/7QlaO7WxjGg
It even uses a cover flow custom layout and "reflection" view that mirrors the video preview perfectly. The source code is here:
http://www.mediafire.com/download/ivecygnlhqxwynr/VideoWallCollectionView.zip
My goal is to write a custom camera view controller that:
Can take photos in all four interface orientations with both the back and, when available, front camera.
Properly rotates and scales the preview "video" as well as the full resolution photo.
Allows a (simple) effect to be applied to BOTH the preview "video" and full resolution photo.
Implementation (on iOS 4.2 / Xcode 3.2.5):
Due to requirement (3), I needed to drop down to AVFoundation.
I started with Technical Q&A QA1702 and made these changes:
Changed the sessionPreset to AVCaptureSessionPresetPhoto.
Added an AVCaptureStillImageOutput as an additional output before starting the session.
The issue that I am having is with the performance of processing the preview image (a frame of the preview "video").
First, I get the UIImage result of imageFromSampleBuffer: on the sample buffer from captureOutput:didOutputSampleBuffer:fromConnection:. Then, I scale and rotate it for the screen using a CGGraphicsContext.
At this point, the frame rate is already under the 15 FPS that is specified in the video output of the session and when I add in the effect, it drops to under or around 10. Quickly the app crashes due to low memory.
I have had some success with dropping the frame rate to 9 FPS on the iPhone 4 and 8 FPS on the iPod Touch (4th gen).
I have also added in some code to "flush" the dispatch queue, but I am not sure how much it is actually helping. Basically, every 8-10 frames, a flag is set that signals captureOutput:didOutputSampleBuffer:fromConnection: to return right away rather than process the frame. The flag is reset after a sync operation on the output dispatch queue finishes.
At this point I don't even mind the low frame rates, but obviously we can't ship with the low memory crashes. Anyone have any idea how to take action to prevent the low memory conditions in this case (and/or a better way to "flush" the dispatch queue)?
To prevent the memory issues, simply create an autorelease pool in captureOutput:didOutputSampleBuffer:fromConnection:.
This makes sense since imageFromSampleBuffer: returns an autoreleased UIImage object. Plus it frees up any autoreleased objects created by image processing code right away.
// Delegate routine that is called when a sample buffer was written
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
// Create a UIImage from the sample buffer data
UIImage *image = [self imageFromSampleBuffer:sampleBuffer];
< Add your code here that uses the image >
[pool release];
}
My testing has shown that this will run without memory warnings on an iPhone 4 or iPod Touch (4th gen) even if requested FPS is very high (e.g. 60) and image processing is very slow (e.g. 0.5+ secs).
OLD SOLUTION:
As Brad pointed out, Apple recommends image processing be on a background thread so as to not interfere with the UI responsiveness. I didn't notice much lag in this case, but best practices are best practices, so use the above solution with autorelease pool instead of running this on the main dispatch queue / main thread.
To prevent the memory issues, simply use the main dispatch queue instead of creating a new one.
This also means that you don't have to switch to the main thread in captureOutput:didOutputSampleBuffer:fromConnection: when you want to update the UI.
In setupCaptureSession, change FROM:
// Configure your output.
dispatch_queue_t queue = dispatch_queue_create("myQueue", NULL);
[output setSampleBufferDelegate:self queue:queue];
dispatch_release(queue);
TO:
// we want our dispatch to be on the main thread
[output setSampleBufferDelegate:self queue:dispatch_get_main_queue()];
A fundamentally better approach would be to use OpenGL to handle as much of the image-related heavy lifting for you (as I see you're trying in your latest attempt). However, even then you might have issues with building up frames to be processed.
While it seems strange that you'd be running into memory accumulation when processing frames (in my experience, you just stop getting them if you can't process them fast enough), Grand Central Dispatch queues can get jammed up if they are waiting on I/O.
Perhaps a dispatch semaphore would let you throttle the addition of new items to the processing queues. For more on this, I highly recommend Mike Ash's "GCD Practicum" article, where he looks at optimizing an I/O bound thumbnail processing operation using dispatch semaphores.
Now that Apple is officially allowing UIGetScreenImage() to be used in iPhone apps, I've seen a number of blogs saying that this "opens the floodgates" for video capture on iPhones, including older models. But I've also seen blogs that say the fastest frame rate they can get with UIGetScreenImage() is like 6 FPS.
Can anyone share specific frame-rate results you've gotten with UIGetScreenImage() (or other approved APIs)? Does restricting the area of the screen captured improve frame rate significantly?
Also, for the wishful thinking segment of today's program, does anyone have pointers to code/library that uses UIGetScreenImage() to capture video? For instance, I'd like an API something like Capture( int fps, Rect bounds, int durationMs ) that would turn on the camera and for the given duration record a sequence of .png files at the given frame rate, copying from the given screen rect.
There is no specific frame rate. UIGetScreenImage() is not a movie recorder. It just try to return as soon as it could, unfortunately still very slow.
Restricting the area of the screen captured is useless. UIGetScreenImage doesn't take any input parameters. Cropping the output image could make the frame rate even worse due to the excess work.
UIGetScreenImage() returns an image of current screen display. It's said to be slow but whether it's fast enough depends on the use case. The video recording app iCamcorder is using this function.
According to their blog,
iCamcorder records at an remarkable average minimum of 10 frames per second and a maximum of 15 frames per second.
The UIGetScreenImage method Apple recently allowed developers to use captures the current screen contents. Unfortunately it is really slow, about 15% of the processing time of the App just goes into calling this method. http://www.drahtwerk.biz/EN/Blog.aspx/iCamcorder-v19-and-Giveaway/?newsID=27
So the raw performance of UIGetScreenImage() should be at least much higher than 15 fps.
To crop the returned image, you can try
extern CGImageRef UIGetScreenImage(void);
...
CGImageRef cgoriginal = UIGetScreenImage();
CGImageRef cgimg = CGImageCreateWithImageInRect(cgoriginal, rect);
UIImage *viewImage = [UIImage imageWithCGImage:cgimg];
CGImageRelease(cgoriginal);
CGImageRelease(cgimg);