Good day, hope someone can help me up with this situation:
I working on an iPhone app that takes series of images, assisted by gyroscope.
So both AVCamCaptureManager and CMMotionManager sessions are running at the same time.
after taking a still image, i am:
- processing the image in a background thread (which works fine without affecting anything)
- then saving processed image data to disk
[imageData writeToFile:imagePath atomically:YES];
The issue: both AVCamCaptureManager and CMMotionManager sessions freeze for less then 1/2 second, right after initiating writeToFile function.
Does anyone have any experience with such scenario?
Thanks for your time! :)
It appears that saving to disk does not affect sessions.
I am also setting UIImageView.image to a large image in the end of my routine, and this is what was freezing everything for 1/2 second.
Related
How Can I delay the stream to UIImageview using AVCaptureVideoPreviewLayer from camera?
See below how I bind them, but I just can't figure how to delay it (I don't want it in real time)
AVCaptureVideoPreviewLayer* captureVideoPreviewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:session];
captureVideoPreviewLayer.frame = self.imageView.bounds;
[self.imageView.layer addSublayer:captureVideoPreviewLayer];
First you're going to want to remove the preview layer frame you have right now, as there is no method to delay those preview frames out of the box.
You're going to want to create a buffer. If we're talking about a few frames, you could have a NSMutableArray that you're filling up on one end with UIImages while you're feeding your image view from the other end.
Your UIImage would come from the didOutputSampleBuffer method, use something like this UIImage created from CMSampleBufferRef not displayed in UIImageView?
Now, few challenges you will have to deal with:
you're talking about having a multiple seconds delay, 5 seconds would be about 150 frames. Storing 150 UIImage in memory isn't gonna happen, unless they're very tiny and on the latest devices
You would solve that by saving the images to disk and have your array only store the path of those images instead of the images themselves. Now you're probably going to run into performance issues, as you're going to do read/write operations in real time, your framerate is going to suffer from that
Because of that bad frame rate, you're going to have to make sure you're not losing synchronization between recorded feed and live feed, otherwise you'll start with a 5 sec delay and end up with a much longer delay
Good luck with that, it can be done with some trade-off (slower frame rate...) but it can be done. (I have done something very similar myself multiple times, can't share the code for IP reasons).
My goal is to write a custom camera view controller that:
Can take photos in all four interface orientations with both the back and, when available, front camera.
Properly rotates and scales the preview "video" as well as the full resolution photo.
Allows a (simple) effect to be applied to BOTH the preview "video" and full resolution photo.
Implementation (on iOS 4.2 / Xcode 3.2.5):
Due to requirement (3), I needed to drop down to AVFoundation.
I started with Technical Q&A QA1702 and made these changes:
Changed the sessionPreset to AVCaptureSessionPresetPhoto.
Added an AVCaptureStillImageOutput as an additional output before starting the session.
The issue that I am having is with the performance of processing the preview image (a frame of the preview "video").
First, I get the UIImage result of imageFromSampleBuffer: on the sample buffer from captureOutput:didOutputSampleBuffer:fromConnection:. Then, I scale and rotate it for the screen using a CGGraphicsContext.
At this point, the frame rate is already under the 15 FPS that is specified in the video output of the session and when I add in the effect, it drops to under or around 10. Quickly the app crashes due to low memory.
I have had some success with dropping the frame rate to 9 FPS on the iPhone 4 and 8 FPS on the iPod Touch (4th gen).
I have also added in some code to "flush" the dispatch queue, but I am not sure how much it is actually helping. Basically, every 8-10 frames, a flag is set that signals captureOutput:didOutputSampleBuffer:fromConnection: to return right away rather than process the frame. The flag is reset after a sync operation on the output dispatch queue finishes.
At this point I don't even mind the low frame rates, but obviously we can't ship with the low memory crashes. Anyone have any idea how to take action to prevent the low memory conditions in this case (and/or a better way to "flush" the dispatch queue)?
To prevent the memory issues, simply create an autorelease pool in captureOutput:didOutputSampleBuffer:fromConnection:.
This makes sense since imageFromSampleBuffer: returns an autoreleased UIImage object. Plus it frees up any autoreleased objects created by image processing code right away.
// Delegate routine that is called when a sample buffer was written
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
// Create a UIImage from the sample buffer data
UIImage *image = [self imageFromSampleBuffer:sampleBuffer];
< Add your code here that uses the image >
[pool release];
}
My testing has shown that this will run without memory warnings on an iPhone 4 or iPod Touch (4th gen) even if requested FPS is very high (e.g. 60) and image processing is very slow (e.g. 0.5+ secs).
OLD SOLUTION:
As Brad pointed out, Apple recommends image processing be on a background thread so as to not interfere with the UI responsiveness. I didn't notice much lag in this case, but best practices are best practices, so use the above solution with autorelease pool instead of running this on the main dispatch queue / main thread.
To prevent the memory issues, simply use the main dispatch queue instead of creating a new one.
This also means that you don't have to switch to the main thread in captureOutput:didOutputSampleBuffer:fromConnection: when you want to update the UI.
In setupCaptureSession, change FROM:
// Configure your output.
dispatch_queue_t queue = dispatch_queue_create("myQueue", NULL);
[output setSampleBufferDelegate:self queue:queue];
dispatch_release(queue);
TO:
// we want our dispatch to be on the main thread
[output setSampleBufferDelegate:self queue:dispatch_get_main_queue()];
A fundamentally better approach would be to use OpenGL to handle as much of the image-related heavy lifting for you (as I see you're trying in your latest attempt). However, even then you might have issues with building up frames to be processed.
While it seems strange that you'd be running into memory accumulation when processing frames (in my experience, you just stop getting them if you can't process them fast enough), Grand Central Dispatch queues can get jammed up if they are waiting on I/O.
Perhaps a dispatch semaphore would let you throttle the addition of new items to the processing queues. For more on this, I highly recommend Mike Ash's "GCD Practicum" article, where he looks at optimizing an I/O bound thumbnail processing operation using dispatch semaphores.
I'm building an app which includes a number of image sequences (5 sequences with about 80 images each). It runs nicely in the iPhone simulator, but causes my iPhone to reboot when I test it. By the way, each png image is about 8k in size.
Has anyone successfully built a similar app?
Am I using too many resources for the iPhone to handle?
Anyone?
UPDATE:
Thanks to all for you answers! I've modified my code to use [UIImage imageWithContentsOfFile:] instead of [UIImage imageNamed:]
However I'm still unable to prevent the app from crashing my iPhone.
(please note that my pngs are not that big about 400x400px / 8k)
Does anyone have any suggestions?
Here's my code:
// code snippet:
myFrames = [[NSMutableArray alloc] initWithCapacity:maxFrames];
NSMutableString *curFrame;
num = 0;
// loop (maxframes = 80)
for(int f = 1; f < maxFrames+1; f++)
{
curFrame = [NSMutableString stringWithString:tName];
if(f < 10) [curFrame appendString:[NSString stringWithFormat:#"00%i",f]];
else if(f>9 && f<100) [curFrame appendString:[NSString stringWithFormat:#"0%i",f]];
else [curFrame appendString:[NSString stringWithFormat:#"%i",f]];
UIImage *img = [UIImage imageWithContentsOfFile: [[NSBundle mainBundle] pathForResource:curFrame ofType:#"png"]];
if(img) [myFrames addObject:img];
[img release];
}
// animate the images!
self.animationImages = myFrames;
self.animationDuration = (maxFrames * .05); // Seconds
[self startAnimating];
The best way to find out is to run the application under Instruments using Leaks or Object Alloc. If you see an upward trend that keeps rising, you might have a leak.
If you're using [UIImage imageNamed:], you should be aware that it pre-caches an optimized version which takes up more memory when compared with [UIImage imageWithContentsOfFile:]. Additionally, until iPhone 3.0, the cache created by [UIImage imageNamed:] doesn't get released when there's a memory warning.
The current-gen iPhone only has 128MB of ram, some of which is used by the OS itself. A 320x480 image fully uncompressed with an alpha channel can take 614k. If you have 400 unique images that are full screen, that's well over 128MB of ram, assuming it is loaded up and cached uncompressed.
The number one reason why an app would not crash on the simulator but on the phone would be memory
On the iphone simulator AFAIK the memory is not limited to 128Mb while on the iphone once it reaches 128Mb it restarts. So check your memory usage on the simulator. You have to change the way you are loading the images and or check for leaks. Also check if your getting low memory warnings by implementing the methods (I forgot what they are called :()
I've seen apps run in the simulator and not on the phone because of improper PNG formatting (even a single improperly formatted image can cause this crash). Check to make sure that the format of your images matches those of PNG files provided by apple in their example apps.
That being said 400 full screen images would easily cause it to run out of memory as in memory they will occupy far more than the 8kb. Not sure how big those images are, but if they're all in memory they will need to be very, very small on the iPhone.
The first answer to your question states that while your PNGs may take up only 8K on disk, that is the compressed on-disk form. When it is loaded into memory, it is decompressed and is much larger than 8K. At 32-bits per pixel, a 400x400 image will be 640K.
Even without the alpha channel, you're looking at 480K. 480K x 80 frames, that is 38.4MB, which is definitely creeping into using more memory than the iphone has available to give your app at once. Here is an article about some of the troubles with obtaining a substantial about of memory from the iPhone OS.
So far I've managed to create an app for iPhone that takes multiple images with about a 3 second interval between each. I`m processing each image in a separate thread asynchronously and everything is great till it gets to the moment for saving the image on the iPhone disk. Then it takes about 12 seconds to save the image to the disk using JPEG representation.
How does Apple do it, how do they manage to save a single image so fast to the disk is there a trick they are using? I saw that the animations distract the user for a while, but still the time needed is below 12 seconds!
Thanks in advance.
Actually apple uses its kernal driver AppleJPEGDriver, It is a hardware jpeg encoding api and is much faster than software encoding (JPEGRepresnetaion) and some of the people using it in their jailbreak apps(cycorder video recording application).
Apple should give the same functionality to its users but they are apple :)
I haven't tried this but I wouldn't be so sure that Apple isn't using the same methods. A big part of the Apple design philosophy relies on hiding operational interruptions from the user. The Apple code may take as much time as yours but simply be adroit at hiding the entire save time from the perception of the user.
If someone can't tell you how Apple actually does save faster I would suggest looking at ways to disguise the save time.
If you google around a bit... there is a whole bunch of people with the same problem.
I didn't find an answer. The general conclusion seems to be that apple either uses some internal api and bypass public api overhead or some hardware encoder.
Guess you are out of luck for fast image saving
I was having this problem in my app, on saving it would hang so I used Grand central dispatch.
Below is the setImage method out of my image cache class, if UIImage has a image it saves it otherwise it deletes it. You can adapt this to suit your needs hopefully, will only work on iOS 4+. The code is ARC enabled.
-(void)setImage:(UIImage *)image{
if (image == nil){
NSLog(#"Deleting Image");
// Since we have no image let's remove the cached image if it exists
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
NSString *cachePath = [NSSearchPathForDirectoriesInDomains(NSCachesDirectory,
NSUserDomainMask, YES) objectAtIndex:0];
[[NSFileManager defaultManager] removeItemAtPath:[cachePath
stringByAppendingPathComponent:#"capturedimage.jpg"] error:nil];
});
}
else {
NSLog(#"Saving Image");
// We've got an image, let's save it to flash memory.
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
NSString *cachePath =
[NSSearchPathForDirectoriesInDomains(NSCachesDirectory,
NSUserDomainMask, YES) objectAtIndex:0];
NSData *dataObj = UIImagePNGRepresentation(image);
[dataObj writeToFile:[cachePath
stringByAppendingPathComponent:#"capturedimage.jpg"] atomically:NO];
});
}
imageCache = image;
}
I'm trying to build a video recorder without jailbreaking my iPhone (i've a Developer license).
I began using PhotoLibrary private framework, but i can only reach 2ftp (too slow).
Cycoder app have a fps of 15, i think it uses a different approach.
I tried to create a bitmap from the previewView of the CameraController, but it always returns e black bitmap.
I wonder if there's a way to directly access the video buffer, maybe with IOKit framework.
Thanks
Marco
Here is the code:
image = [window _createCGImageRefRepresentationInFrame:rectToCapture];
Marco
That is the big problem. So far i've solved using some temp fixed size buffers and detach a thread for every buffer when is full. The thread will save the buffer content in the Flash memory. Launching some heavy threads, heavy beacause each thread access the flash, will slow the device down and the refresh of the camera view.
Buffers cannot be big, because you will get memory warning, and cannot be small because you will freeze the device, because of too many threads and accesses to the flash memory at a time.
The solution resides in balancing buffer size and number of threads.
I haven't already tried to use sqlite3 db to store images binary data, but i don't if will be a better solution.
PS: to speed up class methods call, avoid the common solution [object method] because of how method call works, but try to get and save the method address as below.
From Apple ObjC doc:
"The example below shows how the procedure that implements the setFilled: method might be
called:
void (*setter)(id, SEL, BOOL);
int i;
setter = (void (*)(id, SEL, BOOL))[target methodForSelector:#selector(setFilled:)];
for ( i = 0; i < 1000, i++ )
setter(targetList[i], #selector(setFilled:), YES); "
Marco
If you're intending to ever release your app on the App Store, using a private framework will ensure that it will be rejected. Video, using the SDK, simply isn't supported.
To capture the video you can see when the Camera is active requires fairly sophisticate techniques, not exposed by any framework/lib out of the box.
I used a non documented UIWindow method to get the current displayed frame as CGImageRef.
Now it works successfully!!
If you would, and if i'm allowded, i can post the code that do the trick.
Marco