I'm trying to build a video recorder without jailbreaking my iPhone (i've a Developer license).
I began using PhotoLibrary private framework, but i can only reach 2ftp (too slow).
Cycoder app have a fps of 15, i think it uses a different approach.
I tried to create a bitmap from the previewView of the CameraController, but it always returns e black bitmap.
I wonder if there's a way to directly access the video buffer, maybe with IOKit framework.
Thanks
Marco
Here is the code:
image = [window _createCGImageRefRepresentationInFrame:rectToCapture];
Marco
That is the big problem. So far i've solved using some temp fixed size buffers and detach a thread for every buffer when is full. The thread will save the buffer content in the Flash memory. Launching some heavy threads, heavy beacause each thread access the flash, will slow the device down and the refresh of the camera view.
Buffers cannot be big, because you will get memory warning, and cannot be small because you will freeze the device, because of too many threads and accesses to the flash memory at a time.
The solution resides in balancing buffer size and number of threads.
I haven't already tried to use sqlite3 db to store images binary data, but i don't if will be a better solution.
PS: to speed up class methods call, avoid the common solution [object method] because of how method call works, but try to get and save the method address as below.
From Apple ObjC doc:
"The example below shows how the procedure that implements the setFilled: method might be
called:
void (*setter)(id, SEL, BOOL);
int i;
setter = (void (*)(id, SEL, BOOL))[target methodForSelector:#selector(setFilled:)];
for ( i = 0; i < 1000, i++ )
setter(targetList[i], #selector(setFilled:), YES); "
Marco
If you're intending to ever release your app on the App Store, using a private framework will ensure that it will be rejected. Video, using the SDK, simply isn't supported.
To capture the video you can see when the Camera is active requires fairly sophisticate techniques, not exposed by any framework/lib out of the box.
I used a non documented UIWindow method to get the current displayed frame as CGImageRef.
Now it works successfully!!
If you would, and if i'm allowded, i can post the code that do the trick.
Marco
Related
On Apple's iOS 6.0 feature page, it used to say
Take advantage of the built-in camera’s advanced features. New APIs let you control focus, exposure, and region of interest. You can also access and display faces with face detection APIs, and leverage hardware-enabled video stabilization.
This text has since been removed, and I can't find new methods in the API for controlling exposure. In class AVCaptureDevice under "Exposure Settings" there is no new property/method for iOS 6.0. Do you know where i can find new features for exposure in API?
It's true that there is an -exposureMode property on AVCaptureDevice, but that's only for setting the mode (off/auto/continuous) and not the actual f-stop, SS, or ISO. Camera apps that provide "exposure" control all seem to do it through post-processing.
However, it seems there are undocumented APIs in the framework to do this. Check out the full headers for AVCaptureDevice.h (via a class-dump) and note the following methods:
- (void)setManualExposureSupportEnabled:(BOOL)arg1;
- (BOOL)isManualExposureSupportEnabled;
- (void)setExposureGain:(float)arg1;
- (float)exposureGain;
- (void)setExposureDuration:(struct { long long x1; int x2; unsigned int x3; long long x4; })arg1;
- (struct { long long x1; int x2; unsigned int x3; long long x4; })exposureDuration;
- (void)setExposureMode:(int)arg1;
- (int)exposureMode;
- (BOOL)isExposureModeSupported:(int)arg1;
My guess is gain is equivalent f-stop (fixed aperture), and duration is shutter speed. I wonder if these are used for the iPhone 5's low-light boost mode.
You can also use otool to poke around and try to piece together the symbols. There's likely a new constant in exposureMode for enabling manual control, and exposureDuration seems like it has flags too. When calling these, make sure to use the new -isExposureModeSupported: and also call -respondsToSelector: to check compatibility.
As always, using private APIs is frowned upon by Apple and is cause for rejection from the App Store. There might be ways around this, such as hiding the calls using -performSelector: or obc_msgsend with rot13 strings or something, as I'm pretty sure they only do static analysis on the app binary.
I've managed to 'trick' the camera into running a shorter exposure time, but I suspect it will only be of use to those doing similar (macro) image acquires. I first set up AVCaptureDevice to use AVCaptureExposureModeContinuousAutoExposure and set the flash to TorchMode. I then UnlockForConfiguration and set up a key-value observer to watch for adjustingExposure to finish. I then re-lock the device, flip to AVCaptureExposureModeLocked, and turn off the Torch. This has the effect of brute-force setting a shorter shutter speed than what the camera would select on the un-illuminated scene. By playing with the Torch level I can set any relative shutter speed value I want (it would be best of course to leave the torch on, but in my application it produces glare on the subject). Again this only really works when your object distance is very close (less than say 6 inches), but it's allowed me to eliminate hand shake blurring in my close-up images. The down side is that the images are darker since I don't have a way of spoofing the camera gain, but not a problem in my particular application.
It looks like they've updated that linked text—there's no mention of new APIs for exposure:
Use powerful new features of the built-in camera. New APIs support real-time video stabilization, an improved LED flash, and face detection and display. You can get reports of dropped frames during capture and leverage new utilities to map UI touches to focus and exposure commands. And apps that support iPhone 5 can take advantage of low light boost mode.
There is an opt-in low-light boost mode for iPhone 5, detailed here by Jim Rhoades (and in this developer forum post, log-in required).
As a follow-up to Michael Grinich's excellent information, I found that there is an order dependency on some of the calls in the private API. To use "manual" exposure controls, you have to enable them before you set the mode, like so:
#define AVCaptureExposureModeManual 3
NSError* error = nil;
if ([captureDevice lockForConfiguration:&error]) {
captureDevice.manualExposureSupportEnabled = YES;
if ([captureDevice isExposureModeSupported:AVCaptureExposureModeManual]) {
captureDevice.exposureMode = AVCaptureExposureModeManual;
captureDevice.exposureGain = ...;
captureDevice.exposureDuration = {...};
}
[captureDevice unlockForConfiguration];
}
All of this is demonstrated in iOS-ManualCamera.
Starting with iOS 8.0, this is now finally possible.
See setExposureModeCustomWithDuration etc. in the Apple documentation.
Here is an article discussing how to use the APIs.
I'm trying to modify the GLCameraRipple sample application from Apple to process video frames on a background thread. In this example, it handles each frame on the main thread using the following code:
// Set dispatch to be on the main thread so OpenGL can do things with the data
[dataOutput setSampleBufferDelegate:self queue:dispatch_get_main_queue()];
If I change this code to process in a background thread:
dispatch_queue_t videoQueue = dispatch_queue_create("com.test.queue", NULL);
[dataOutput setSampleBufferDelegate:self queue:videoQueue];
then program crashes.
When I try to create a second EAGLContext with sharing, as specified in Apple's documentation, then I only see a green or black screen.
How can I modify this sample application to run on a background thread?
This was actually fairly interesting, after I tinkered with the sample. The problem here is with the CVOpenGLESTextureCacheCreateTextureFromImage() function. If you look at the console when you get the green texture, you'll see something like the following being logged:
Error at CVOpenGLESTextureCacheCreateTextureFromImage -6661
-6661, according to the headers (the only place I could find documentation on these new functions currently), is a kCVReturnInvalidArgument error. Something's obviously wrong with one of the arguments to this function.
It turns out that it is the CVImageBufferRef that is the problem here. It looks like this is being deallocated or otherwise changed while the block that handles this texture cache update is happening.
I tried a few ways of solving this, and ended up using a dispatch queue and dispatch semaphore like I describe in this answer, having the delegate still call back on the main thread, and within the delegate do something like the following:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
if (dispatch_semaphore_wait(frameRenderingSemaphore, DISPATCH_TIME_NOW) != 0)
{
return;
}
CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
CFRetain(pixelBuffer);
dispatch_async(openGLESContextQueue, ^{
[EAGLContext setCurrentContext:_context];
// Rest of your processing
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
CFRelease(pixelBuffer);
dispatch_semaphore_signal(frameRenderingSemaphore);
});
}
By creating the CVImageBufferRef on the main thread, locking the bytes it points to, and retaining it, then handing it off to the asynchronous block, that seems to fix this error. A full project that shows this modification can be downloaded from here.
I should say one thing here: this doesn't appear to gain you anything. If you look at the way that the GLCameraRipple sample is set up, the heaviest operation in the application, the calculation of the ripple effect, is already dispatched to a background queue. This is also using the new fast upload path for providing camera data to OpenGL ES, so that's not a bottleneck here when run on the main thread.
In my Instruments profiling on a dual-core iPhone 4S, I see no significant difference in rendering speed or CPU usage between the stock version of this sample application and my modified one that runs the frame upload on a background queue. Still, it was an interesting problem to diagnose.
I am currently using openAl in my game . I am getting the memory leak whose responsible frame is `
OALSource:AddPlaybackMessage
` What I am doing is that after my game finishes i delete all the buffers allocated to the Source and free other pcmdata pointers etc .. before the starting the game i re-initialize the source and their buffers with the audio data. At this stage am getting this leak. Any idea why its happening ..... Some posts say its a bug in apple's OpenAl library but i dont think so .. Apple must have done something about it ....
Thanks
ok Now am not getting any leaks from openAL .. What i have done is :-
While initializing sources again delete them first with their respective buffers and free any pcmData pointer.
regenerate sources and buffers.
Dont ever Delete openALContext and openALDevice . get context and device only once when your app starts . Delete them on Dealloc().
Hope this solves your openAl leak issues.
Cheers
I'm developing an interactive storybook type application for the iPhone and I've recently encountered a frustrating bug concerning audio mixing on the device.
Firstly, I setup an audio session. I set the category to AVAudioSessionCategoryAmbient and then init and play my AVAudioPlayer instance. Now, in the background whilst the audio is playing I'm pre-loading a video to play using an MPMoviePlayerController followed by a call to prepareToPlay. The reason I pre-load the video this way is because I need it to play instantly later on cue with fairly strict timing.
In this configuration, the audio/movie works fine and they mix and do not interrupt each other. However, this particular audio session category does not permit audio to continue playing while the device is locked, a feature I really need. As a result I'm forced to consider a different category: AVAudioSessionCategoryPlayback.
By default this category does not permit mixing with other audio, according to the Apple docs. To enable mixing with other audio I am overriding the relevant category:
OSStatus propertySetError = 0;
UInt32 setProperty = 1;
propertySetError = AudioSessionSetProperty(kAudioSessionProperty_OverrideCategoryMixWithOthers, sizeof(setProperty), &setProperty);
assert(propertySetError == 0);
Unfortunately, this solves my playing whilst locked issue but introduces another issue: the AVAudioPlayer audio is interrupted briefly as the video loads with a minor stutter. The stutter is small, perhaps less than a second but is enough to disrupt the user experience. I've read this related post which enabled me to pre-load the video with the AVAudioSessionCategoryAmbient, but unfortunately the same approach doesn't seem to work with the new category.
The audio session category is applied successfully, according to the return code. Does anyone know why enabling audio mixing with this category is not the same as the mixing facility provided by ambient category?
The best way I've found working a similar problem is to use the newer AVPlayer (+1 #adam) and set your app to enable background audio and receive remote control notifications. I was tipped off to this by #MarquelV following How can you play music from the iPod app while still receiving remote control events in your app?
If you can get backgrounding working properly, that should enable you to continue playing while the device is locked. Oh, and don't forget to add keys to info.plist, its easy to do and then have no idea why it isn't working.
My goal is to write a custom camera view controller that:
Can take photos in all four interface orientations with both the back and, when available, front camera.
Properly rotates and scales the preview "video" as well as the full resolution photo.
Allows a (simple) effect to be applied to BOTH the preview "video" and full resolution photo.
Implementation (on iOS 4.2 / Xcode 3.2.5):
Due to requirement (3), I needed to drop down to AVFoundation.
I started with Technical Q&A QA1702 and made these changes:
Changed the sessionPreset to AVCaptureSessionPresetPhoto.
Added an AVCaptureStillImageOutput as an additional output before starting the session.
The issue that I am having is with the performance of processing the preview image (a frame of the preview "video").
First, I get the UIImage result of imageFromSampleBuffer: on the sample buffer from captureOutput:didOutputSampleBuffer:fromConnection:. Then, I scale and rotate it for the screen using a CGGraphicsContext.
At this point, the frame rate is already under the 15 FPS that is specified in the video output of the session and when I add in the effect, it drops to under or around 10. Quickly the app crashes due to low memory.
I have had some success with dropping the frame rate to 9 FPS on the iPhone 4 and 8 FPS on the iPod Touch (4th gen).
I have also added in some code to "flush" the dispatch queue, but I am not sure how much it is actually helping. Basically, every 8-10 frames, a flag is set that signals captureOutput:didOutputSampleBuffer:fromConnection: to return right away rather than process the frame. The flag is reset after a sync operation on the output dispatch queue finishes.
At this point I don't even mind the low frame rates, but obviously we can't ship with the low memory crashes. Anyone have any idea how to take action to prevent the low memory conditions in this case (and/or a better way to "flush" the dispatch queue)?
To prevent the memory issues, simply create an autorelease pool in captureOutput:didOutputSampleBuffer:fromConnection:.
This makes sense since imageFromSampleBuffer: returns an autoreleased UIImage object. Plus it frees up any autoreleased objects created by image processing code right away.
// Delegate routine that is called when a sample buffer was written
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
// Create a UIImage from the sample buffer data
UIImage *image = [self imageFromSampleBuffer:sampleBuffer];
< Add your code here that uses the image >
[pool release];
}
My testing has shown that this will run without memory warnings on an iPhone 4 or iPod Touch (4th gen) even if requested FPS is very high (e.g. 60) and image processing is very slow (e.g. 0.5+ secs).
OLD SOLUTION:
As Brad pointed out, Apple recommends image processing be on a background thread so as to not interfere with the UI responsiveness. I didn't notice much lag in this case, but best practices are best practices, so use the above solution with autorelease pool instead of running this on the main dispatch queue / main thread.
To prevent the memory issues, simply use the main dispatch queue instead of creating a new one.
This also means that you don't have to switch to the main thread in captureOutput:didOutputSampleBuffer:fromConnection: when you want to update the UI.
In setupCaptureSession, change FROM:
// Configure your output.
dispatch_queue_t queue = dispatch_queue_create("myQueue", NULL);
[output setSampleBufferDelegate:self queue:queue];
dispatch_release(queue);
TO:
// we want our dispatch to be on the main thread
[output setSampleBufferDelegate:self queue:dispatch_get_main_queue()];
A fundamentally better approach would be to use OpenGL to handle as much of the image-related heavy lifting for you (as I see you're trying in your latest attempt). However, even then you might have issues with building up frames to be processed.
While it seems strange that you'd be running into memory accumulation when processing frames (in my experience, you just stop getting them if you can't process them fast enough), Grand Central Dispatch queues can get jammed up if they are waiting on I/O.
Perhaps a dispatch semaphore would let you throttle the addition of new items to the processing queues. For more on this, I highly recommend Mike Ash's "GCD Practicum" article, where he looks at optimizing an I/O bound thumbnail processing operation using dispatch semaphores.