I'm developing an iPhone 4 (iOS 4) application that show an level meter.
This app measures human voice. But it has a problem. When there is a lot of noise, it doesn't work. It measures also background noise.
To measure sound, I use this:
- (void) initWithPattern:(Pattern *)pattern
{
mode = figureMode;
[self showFigureMeter];
patternView.pattern = pattern;
NSURL *url = [NSURL fileURLWithPath:#"/dev/null"];
NSDictionary *settings = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithFloat: 44100.0], AVSampleRateKey,
[NSNumber numberWithInt: kAudioFormatAppleLossless], AVFormatIDKey,
[NSNumber numberWithInt: 2], AVNumberOfChannelsKey,
[NSNumber numberWithInt: AVAudioQualityMax], AVEncoderAudioQualityKey,
nil];
NSError *error;
if (recorder == nil)
recorder = [[AVAudioRecorder alloc] initWithURL:url settings:settings error:&error];
if (recorder) {
[recorder prepareToRecord];
recorder.meteringEnabled = YES;
[recorder record];
levelTimer = [NSTimer scheduledTimerWithTimeInterval: 0.03
target: self
selector: #selector(levelTimerCallback:)
userInfo: nil
repeats: YES];
}
}
- (void)levelTimerCallback:(NSTimer *)timer
{
[recorder updateMeters];
float peakPower = [recorder peakPowerForChannel:0];
if (mode == figureMode)
{
if (peakPower < -40) {
;
} else if ((peakPower > -40) && (peakPower < -30)) {
;
} else if ((peakPower > -30) && (peakPower < -20)) {
;
} else if ((peakPower > -20) && (peakPower < -10)) {
;
} else if (peakPower > -10) {
;
}
}
}
Is there any way to remove background noise?
Noise reduction usually involves sampling the sound (as raw PCM samples), and doing some non-trivial digital signal processing (DSP). One needs a well defined characterization of the noise and how it is different from the desired signal (frequency bands, time, external gating function, etc.) for this to be tractable at all.
You can't just use AVAudioRecorder metering.
You can measure the noise level when no one's speaking (either ask for silence or simply select the lowest measured level) then subtract from the instantaneous level.
Or you can use an FFT to attempt to filter out the background noise by selecting only "voice" frequencies (no success guaranteed).
Related
i was wondering how would i be able to create mini videos every certain amount of time from my recording without stopping my recording? i tried to look for an equivalent of AvAssetImageGenerator for videos an example would be nice.
The easiest way is to use two AVAssetWriters and set up the next writer while the current one is recording, then stop after x time and swap the writers. You should be able to swap the writers without dropping any frames.
Edit:
How to do AVAssetWriter "juggling"
Step 1: Create instance objects for the writers and pixelbuffer adaptors (and you'll want file names for these files as well that you know)
AVAssetWriter* mWriter[2];
AVAssetWriterInputPixelBufferAdaptor* mPBAdaptor[2];
NSString* mOutFile[2];
int mCurrentWriter, mFrameCount, mTargetFrameCount;
Step 2: Create a method for setting up a writer (since you'll be doing this over and over again)
-(int) setupWriter: (int) writer
{
NSAutoreleasePool* p = [[NSAutoreleasePool alloc] init];
NSDictionary* writerSettings = [NSDictionary dictionaryWithObjectsAndKeys: AVVideoCodecH264, AVVideoCodecKey, [NSNumber numberWithInt: mVideoWidth], AVVideoWidthKey, [NSNumber numberWithInt: mVideoHeight], AVVideoHeightKey, nil];
NSDictionary* pbSettings = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithInt:mVivdeoWidth],kCVPixelBufferWidthKey,
[NSNumber numberWithInt:mVideoHeight], kCVPixelBufferHeightKey,
[NSNumber numberWithInt:0],kCVPixelBufferExtendedPixelsLeftKey,
[NSNumber numberWithInt:0],kCVPixelBufferExtendedPixelsRightKey,
[NSNumber numberWithInt:0],kCVPixelBufferExtendedPixelsTopKey,
[NSNumber numberWithInt:0],kCVPixelBufferExtendedPixelsBottomKey,
[NSNumber numberWithInt:mVideoWidth],kCVPixelBufferBytesPerRowAlignmentKey,
[NSNumber numberWithUnsignedInt:kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange], kCVPixelBufferPixelFormatTypeKey, nil];
AVAssetWriterInput* writerInput = [AVAssetWriterInput assetWriterWithMediaType: AVMediaTypeVideo outputSettings: writerSettings];
// Create an audio input here if you want...
mWriter[writer] = [[AVAssetWriter alloc] initWithURL: [NSURL fileURLWithPath:mOutfile[writer]] fileType: AVFileTypeMPEG4 error:nil];
mPBAdaptor[writer] = [[AVAssetWriterInputPixelBufferAdaptor alloc] initWithAssetWriterInput: writerInput sourcePixelBufferAttributes: pbSettings];
[mWriter[writer] addInput: writerInput];
// Add your audio input here if you want it
[p release];
}
Step 3: Gotta tear these things down!
- (void) tearDownWriter: (int) writer
{
if(mWriter[writer]) {
if(mWriter[writer].status == 1) [mWriter[writer] finishWriting]; // This will complete the movie.
[mWriter[writer] release]; mWriter[writer] = nil;
[mPBAdaptor[writer] release]; mPBAdaptor[writer] = nil;
}
}
Step 4: Swap! Tear down the current writer and recreate it asynchronously while the other writer is writing.
- (void) swapWriters
{
NSAutoreleasePool * p = [[NSAutoreleasePool alloc] init];
if(++mFrameCount > mCurrentTargetFrameCount)
{
mFrameCount = 0;
int c, n;
c = mCurrentWriter^1;
n = mCurrentWriter; // swap.
[self tearDownWriter:n];
__block VideoCaptureClass* bSelf = self;
dispatch_async(dispatch_get_global_queue(0,0), ^{
[bSelf setupWriter:n];
CMTime time;
time.value = 0;
time.timescale = 15; // or whatever the correct timescale for your movie is
time.epoch = 0;
time.flags = kCMTimeFlags_Valid;
[bSelf->mWriter[n] startWriting];
[bSelf->mWriter[n] startSessionAtSourceTime:time];
});
mCurrentWriter = c;
}
[p release];
}
Note: When starting up you will have to create and start both writers.
Step 5: Capturing output
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
// This method will only work with video; you'll have to check for audio if you're using that.
CMTime time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer); // Note: you may have to create your own PTS.
CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
[mPBAdaptor[mCurrentWriter] appendPixelBuffer:pixelBuffer withPresentationTime: time];
[self swapBuffers];
}
You can probably skip the pixel buffer adaptor if you don't need it. This should give you an approximate idea of how to do what you want to do. mTargetFrameCount represents how many frames you want the current video to be in length. Audio will probably take additional consideration, you may want to base your length off your audio stream instead of the video stream if you are using audio.
I'm having lag issues when I'm recording audio+video by using AVCaptureVideoDataOutput and AVCaptureAudioDataOutput. Sometimes the video blocks for a few milliseconds, sometimes the audio is not in sync with the video.
I inserted some logs and observed that first I get a lot of video buffers in captureOutput callback, and after some time I get the audio buffers(sometimes I don't receive the audio buffers at all, and the resulting video is without sound). If I comment the code that handles the video buffers, I get the audio buffers without problems.
This is the code I'm using:
-(void)initMovieOutput:(AVCaptureSession *)captureSessionLocal
{
AVCaptureVideoDataOutput *dataOutput = [[AVCaptureVideoDataOutput alloc] init];
self._videoOutput = dataOutput;
[dataOutput release];
self._videoOutput.alwaysDiscardsLateVideoFrames = NO;
self._videoOutput.videoSettings = [NSDictionary dictionaryWithObject: [NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange]
forKey:(id)kCVPixelBufferPixelFormatTypeKey
];
AVCaptureAudioDataOutput *audioOutput = [[AVCaptureAudioDataOutput alloc] init];
self._audioOutput = audioOutput;
[audioOutput release];
[captureSessionLocal addOutput:self._videoOutput];
[captureSessionLocal addOutput:self._audioOutput];
// Setup the queue
dispatch_queue_t queue = dispatch_queue_create("MyQueue", NULL);
[self._videoOutput setSampleBufferDelegate:self queue:queue];
[self._audioOutput setSampleBufferDelegate:self queue:queue];
dispatch_release(queue);
}
Here I set up the writer:
-(BOOL) setupWriter:(NSURL *)videoURL session:(AVCaptureSession *)captureSessionLocal
{
NSError *error = nil;
self._videoWriter = [[AVAssetWriter alloc] initWithURL:videoURL fileType:AVFileTypeQuickTimeMovie
error:&error];
NSParameterAssert(self._videoWriter);
// Add video input
NSDictionary *videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:
AVVideoCodecH264, AVVideoCodecKey,
[NSNumber numberWithInt:640], AVVideoWidthKey,
[NSNumber numberWithInt:480], AVVideoHeightKey,
nil];
self._videoWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo
outputSettings:videoSettings];
NSParameterAssert(self._videoWriterInput);
self._videoWriterInput.expectsMediaDataInRealTime = YES;
self._videoWriterInput.transform = [self returnOrientation];
// Add the audio input
AudioChannelLayout acl;
bzero( &acl, sizeof(acl));
acl.mChannelLayoutTag = kAudioChannelLayoutTag_Mono;
NSDictionary* audioOutputSettings = nil;
// Both type of audio inputs causes output video file to be corrupted.
// should work on any device requires more space
audioOutputSettings = [ NSDictionary dictionaryWithObjectsAndKeys:
[ NSNumber numberWithInt: kAudioFormatAppleLossless ], AVFormatIDKey,
[ NSNumber numberWithInt: 16 ], AVEncoderBitDepthHintKey,
[ NSNumber numberWithFloat: 44100.0 ], AVSampleRateKey,
[ NSNumber numberWithInt: 1 ], AVNumberOfChannelsKey,
[ NSData dataWithBytes: &acl length: sizeof( acl ) ], AVChannelLayoutKey,
nil ];
self._audioWriterInput = [AVAssetWriterInput
assetWriterInputWithMediaType: AVMediaTypeAudio
outputSettings: audioOutputSettings ];
self._audioWriterInput.expectsMediaDataInRealTime = YES;
// add input
[self._videoWriter addInput:_videoWriterInput];
[self._videoWriter addInput:_audioWriterInput];
return YES;
}
And here is the callback:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
if( !CMSampleBufferDataIsReady(sampleBuffer) )
{
NSLog( #"sample buffer is not ready. Skipping sample" );
return;
}
if( _videoWriter.status != AVAssetWriterStatusCompleted )
{
if( _videoWriter.status != AVAssetWriterStatusWriting )
{
CMTime lastSampleTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
[_videoWriter startWriting];
[_videoWriter startSessionAtSourceTime:lastSampleTime];
}
if( captureOutput == _videoOutput )
{
if( [self._videoWriterInput isReadyForMoreMediaData] )
{
[self newVideoSample:sampleBuffer];
}
}
else if( captureOutput == _audioOutput )
{
if( [self._audioWriterInput isReadyForMoreMediaData] )
{
[self newAudioSample:sampleBuffer];
}
}
}
}
-(void) newAudioSample:(CMSampleBufferRef)sampleBuffer
{
if( _videoWriter.status > AVAssetWriterStatusWriting )
{
[self NSLogPrint:[NSString stringWithFormat:#"Audio:Warning: writer status is %d", _videoWriter.status]];
if( _videoWriter.status == AVAssetWriterStatusFailed )
[self NSLogPrint:[NSString stringWithFormat:#"Audio:Error: %#", _videoWriter.error]];
return;
}
if( ![_audioWriterInput appendSampleBuffer:sampleBuffer] )
[self NSLogPrint:[NSString stringWithFormat:#"Unable to write to audio input"]];
}
-(void) newVideoSample:(CMSampleBufferRef)sampleBuffer
{
if( _videoWriter.status > AVAssetWriterStatusWriting )
{
[self NSLogPrint:[NSString stringWithFormat:#"Video:Warning: writer status is %d", _videoWriter.status]];
if( _videoWriter.status == AVAssetWriterStatusFailed )
[self NSLogPrint:[NSString stringWithFormat:#"Video:Error: %#", _videoWriter.error]];
return;
}
if( ![_videoWriterInput appendSampleBuffer:sampleBuffer] )
[self NSLogPrint:[NSString stringWithFormat:#"Unable to write to video input"]];
}
Is there something wrong in my code, why does the video lag?
(I'm testing it on a Iphone 4 ios 4.2.1)
It looks like you are using serial queues. The audio Output queue is right after the video output queue. Consider using concurrent queues.
I've got a method that takes in the annotations (custom PostLocationAnnotation class) to be displayed on a map view and clusters close ones together, outputting an array of MKAnnotation of PostLocationAnnotations and LocationGroupAnnotations (the clusters, which each contain some PostLocationAnnotations). Here's how I call the function (from within an 'updateAnnotations' method, called when the viewport of the map changes):
[annotationsToAdd addObjectsFromArray:[ffMapView annotations]];
[ffMapView addAnnotations:[self clusterAnnotations:annotationsToAdd WithEpsilon:20.0f andMinPts:4]];
annotationsToAdd is initially populated by the annotations that have been retrieved from the server that have not already been added to the map. Therefore I am passing the full list of the annotations that should be put on the map into the clusterAnnotations method. Here is the body of the method:
- (NSArray *)clusterAnnotations:(NSArray *)annotations WithEpsilon:(float)eps andMinPts:(int)minPts
{
NSMutableSet *D = [[NSMutableSet alloc] initWithCapacity:[annotations count]];
NSMutableArray *C = [[NSMutableArray alloc] init];
for (id <MKAnnotation> annotation in annotations)
{
if ([annotation isKindOfClass:[PostLocationAnnotation class]])
{
NSMutableDictionary *dictEntry = [NSMutableDictionary dictionaryWithObjectsAndKeys:
annotation, #"point",
[NSNumber numberWithBool:NO], #"visited",
[NSNumber numberWithBool:NO], #"noise",
[NSNumber numberWithBool:NO], #"clustered", nil];
[D addObject:dictEntry];
[dictEntry release];
} else if ([annotation isKindOfClass:[LocationGroupAnnotation class]])
{
for (PostLocationAnnotation *location in [(LocationGroupAnnotation *)annotation locations])
{
NSMutableDictionary *dictEntry = [NSMutableDictionary dictionaryWithObjectsAndKeys:
location, #"point",
[NSNumber numberWithBool:NO], #"visited",
[NSNumber numberWithBool:NO], #"noise",
[NSNumber numberWithBool:NO], #"clustered", nil];
[D addObject:dictEntry];
[dictEntry release];
}
}
}
for (NSMutableDictionary *P in D)
{
if ([P objectForKey:#"visited"] == [NSNumber numberWithBool:NO])
{
[P setValue:[NSNumber numberWithBool:YES] forKey:#"visited"];
NSMutableSet *N = [[NSMutableSet alloc] initWithSet:[self regionQueryForPoint:P andEpsilon:eps fromList:D]];
if ([N count] < minPts)
{
[P setValue:[NSNumber numberWithBool:YES] forKey:#"noise"];
} else {
LocationGroupAnnotation *newCluster = [[LocationGroupAnnotation alloc] initWithLocations:nil];
[C addObject:newCluster];
[self expandDbscanClusterWithPoint:P andRegion:N andCluster:newCluster andEpsilon:eps andMinPts:minPts fromList:D];
[newCluster release];
}
[N release];
}
}
NSMutableArray *annotationsToAdd = [[[NSMutableArray alloc] initWithCapacity:[annotations count]] autorelease];
for (NSMutableDictionary *P in D)
{
if ([P objectForKey:#"clustered"] == [NSNumber numberWithBool:NO])
{
[annotationsToAdd addObject:[P objectForKey:#"point"]];
}
}
for (LocationGroupAnnotation *cluster in C)
{
[cluster updateCenterCoordinate];
}
[annotationsToAdd addObjectsFromArray:(NSArray *)C];
[D release];
[C release];
return (NSArray *)annotationsToAdd;
}
When I run this I get a zombie message, and I have found that removing [D release] fixes the zombie but causes a leak. Looking at Instruments I can see that the memory address is first Malloc'd in clusterAnnotations, then retained and released a couple of times, then retained a large number of times by regionQueryForPoint (reaching a peak of 47 references), then released twice by clusterAnnotations, then released by [NSAutoreleasePool drain] until the refcount reaches -1 and I get the zombie message error. Here is the code for regionQueryForPoint:
- (NSSet *)regionQueryForPoint:(NSMutableDictionary *)P andEpsilon:(float)eps fromList:(NSMutableSet *)D
{
NSMutableSet *N = [[[NSMutableSet alloc] init] autorelease];
for (NSMutableDictionary *dictEntry in D)
{
if ((dictEntry != P) &&
([[dictEntry objectForKey:#"point"] isKindOfClass:[PostLocationAnnotation class]]))
{
CGPoint p1 = [ffMapView convertCoordinate:[[P objectForKey:#"point"] coordinate] toPointToView:self.view];
CGPoint p2 = [ffMapView convertCoordinate:[[dictEntry objectForKey:#"point"] coordinate] toPointToView:self.view];
float dX = p1.x - p2.x;
float dY = p1.y - p2.y;
if (sqrt(pow(dX,2)+pow(dY,2)) < eps)
{
[N addObject:dictEntry];
}
}
}
return (NSSet *)N;
}
The large number of retains appear to happen when regionQueryForPoint is called from the expandDbScanClusterWithPoint method, so I've included that here for completeness:
- (void)expandDbscanClusterWithPoint:(NSMutableDictionary *)P andRegion:(NSMutableSet *)N
andCluster:(LocationGroupAnnotation *)cluster
andEpsilon:(float)eps
andMinPts:(int)minPts
fromList:(NSMutableSet *)D
{
[cluster addAnnotation:(PostLocationAnnotation *)[P objectForKey:#"point"]];
[P setValue:[NSNumber numberWithBool:YES] forKey:#"clustered"];
BOOL finished = NO;
while (!finished)
{
finished = YES;
for (NSMutableDictionary *nextP in N)
{
if ([nextP objectForKey:#"visited"] == [NSNumber numberWithBool:NO])
{
[nextP setValue:[NSNumber numberWithBool:YES] forKey:#"visited"];
NSSet *nextN = [self regionQueryForPoint:nextP andEpsilon:eps fromList:D];
if ([nextN count] >= minPts)
{
[N unionSet:nextN];
finished = NO;
break;
}
}
if ([nextP objectForKey:#"clustered"] == [NSNumber numberWithBool:NO])
{
[cluster addAnnotation:[nextP objectForKey:#"point"]];
[nextP setValue:[NSNumber numberWithBool:YES] forKey:#"clustered"];
}
}
}
}
I've been dissecting this for ages, counting references, watching pointers and everything but I just can't work out how to safely release this D set. Can anyone see anything I'm not seeing?
You seem to be over-releasing dictEntry with [dictEntry release];. When using dictionaryWithObjectsAndKeys, you're getting an autoreleased object back. So releasing it again will decrease the retain count.
EDIT: If you're unsure how it works and when you're actually retaining objects, you might want to have a look at the memory management docs:
You create an object using a method whose name begins with “alloc”,
“new”, “copy”, or “mutableCopy” (for example, alloc, newObject, or
mutableCopy).
This question already has answers here:
OpenGL ES 2.0 to Video on iPad/iPhone
(7 answers)
Closed 2 years ago.
I am trying to create a AVAssetWriter to screen capture an openGL project. I have never written a AVAssetWriter or an AVAssetWriterInputPixelBufferAdaptor so I am not sure if I did anything correctly.
- (id) initWithOutputFileURL:(NSURL *)anOutputFileURL {
if ((self = [super init])) {
NSError *error;
movieWriter = [[AVAssetWriter alloc] initWithURL:anOutputFileURL fileType:AVFileTypeMPEG4 error:&error];
NSDictionary *videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:
AVVideoCodecH264, AVVideoCodecKey,
[NSNumber numberWithInt:640], AVVideoWidthKey,
[NSNumber numberWithInt:480], AVVideoHeightKey,
nil];
writerInput = [[AVAssetWriterInput
assetWriterInputWithMediaType:AVMediaTypeVideo
outputSettings:videoSettings] retain];
writer = [[AVAssetWriterInputPixelBufferAdaptor alloc] initWithAssetWriterInput:writerInput sourcePixelBufferAttributes:[NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA], kCVPixelBufferPixelFormatTypeKey,nil]];
[movieWriter addInput:writerInput];
writerInput.expectsMediaDataInRealTime = YES;
}
return self;
}
Other parts of the class:
- (void)getFrame:(CVPixelBufferRef)SampleBuffer:(int64_t)frame{
frameNumber = frame;
[writer appendPixelBuffer:SampleBuffer withPresentationTime:CMTimeMake(frame, 24)];
}
- (void)startRecording {
[movieWriter startWriting];
[movieWriter startSessionAtSourceTime:kCMTimeZero];
}
- (void)stopRecording {
[writerInput markAsFinished];
[movieWriter endSessionAtSourceTime:CMTimeMake(frameNumber, 24)];
[movieWriter finishWriting];
}
The assetwriter is initiated by:
NSURL *outputFileURL = [NSURL fileURLWithPath:[NSString stringWithFormat:#"%#%#", NSTemporaryDirectory(), #"output.mov"]];
recorder = [[GLRecorder alloc] initWithOutputFileURL:outputFileURL];
The view is recorded this way:
glReadPixels(0, 0, 480, 320, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
for(int y = 0; y <320; y++) {
for(int x = 0; x <480 * 4; x++) {
int b2 = ((320 - 1 - y) * 480 * 4 + x);
int b1 = (y * 4 * 480 + x);
buffer2[b2] = buffer[b1];
}
}
pixelBuffer = NULL;
CVPixelBufferCreateWithBytes (NULL,480,320,kCVPixelFormatType_32BGRA,buffer2,1920,NULL,0,NULL,&pixelBuffer);
[recorder getFrame:pixelBuffer :framenumber];
framenumber++;
Note:
pixelBuffer is a CVPixelBufferRef.
framenumber is an int64_t.
buffer and buffer2 are GLubyte.
I get no errors but when I finish recording there is no file. Any help or links to help would greatly be appreciated. The opengl has from live feed from the camera. I've been able to save the screen as a UIImage but want to get a movie of what I created.
If you're writing RGBA frames, I think you may need to use a AVAssetWriterInputPixelBufferAdaptor to write them out. This class is supposed to manage a pool of pixel buffers, but I get the impression that it actually massages your data into YUV.
If that works, then I think you'll find that your colours are all swapped at which point you'll probably have to write pixel shader to convert them to BGRA. Or (shudder) do it on the CPU. Up to you.
First time asking a question here. I'm hoping the post is clear and sample code is formatted correctly.
I'm experimenting with AVFoundation and time lapse photography.
My intent is to grab every Nth frame from the video camera of an iOS device (my iPod touch, version 4) and write each of those frames out to a file to create a timelapse. I'm using AVCaptureVideoDataOutput, AVAssetWriter and AVAssetWriterInput.
The problem is, if I use the CMSampleBufferRef passed to captureOutput:idOutputSampleBuffer:fromConnection:, the playback of each frame is the length of time between original input frames. A frame rate of say 1fps. I'm looking to get 30fps.
I've tried using CMSampleBufferCreateCopyWithNewTiming(), but then after 13 frames are written to the file, the captureOutput:idOutputSampleBuffer:fromConnection: stops being called. The interface is active and I can tap a button to stop the capture and save it to the photo library for playback. It appears to play back as I want it, 30fps, but it only has those 13 frames.
How can I accomplish my goal of 30fps playback?
How can I tell where the app is getting lost and why?
I've placed a flag called useNativeTime so I can test both cases. When set to YES, I get all frames I'm interested in as the callback doesn't 'get lost'. When I set that flag to NO, I only ever get 13 frames processed and am never returned to that method again. As mentioned above, in both cases I can playback the video.
Thanks for any help.
Here is where I'm trying to do the retiming.
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
BOOL useNativeTime = NO;
BOOL appendSuccessFlag = NO;
//NSLog(#"in captureOutpput sample buffer method");
if( !CMSampleBufferDataIsReady(sampleBuffer) )
{
NSLog( #"sample buffer is not ready. Skipping sample" );
//CMSampleBufferInvalidate(sampleBuffer);
return;
}
if (! [inputWriterBuffer isReadyForMoreMediaData])
{
NSLog(#"Not ready for data.");
}
else {
// Write every first frame of n frames (30 native from camera).
intervalFrames++;
if (intervalFrames > 30) {
intervalFrames = 1;
}
else if (intervalFrames != 1) {
//CMSampleBufferInvalidate(sampleBuffer);
return;
}
// Need to initialize start session time.
if (writtenFrames < 1) {
if (useNativeTime) imageSourceTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
else imageSourceTime = CMTimeMake( 0 * 20 ,600); //CMTimeMake(1,30);
[outputWriter startSessionAtSourceTime: imageSourceTime];
NSLog(#"Starting CMtime");
CMTimeShow(imageSourceTime);
}
if (useNativeTime) {
imageSourceTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
CMTimeShow(imageSourceTime);
// CMTime myTiming = CMTimeMake(writtenFrames * 20,600);
// CMSampleBufferSetOutputPresentationTimeStamp(sampleBuffer, myTiming); // Tried but has no affect.
appendSuccessFlag = [inputWriterBuffer appendSampleBuffer:sampleBuffer];
}
else {
CMSampleBufferRef newSampleBuffer;
CMSampleTimingInfo sampleTimingInfo;
sampleTimingInfo.duration = CMTimeMake(20,600);
sampleTimingInfo.presentationTimeStamp = CMTimeMake( (writtenFrames + 0) * 20,600);
sampleTimingInfo.decodeTimeStamp = kCMTimeInvalid;
OSStatus myStatus;
//NSLog(#"numSamples of sampleBuffer: %i", CMSampleBufferGetNumSamples(sampleBuffer) );
myStatus = CMSampleBufferCreateCopyWithNewTiming(kCFAllocatorDefault,
sampleBuffer,
1,
&sampleTimingInfo, // maybe a little confused on this param.
&newSampleBuffer);
// These confirm the good heath of our newSampleBuffer.
if (myStatus != 0) NSLog(#"CMSampleBufferCreateCopyWithNewTiming() myStatus: %i",myStatus);
if (! CMSampleBufferIsValid(newSampleBuffer)) NSLog(#"CMSampleBufferIsValid NOT!");
// No affect.
//myStatus = CMSampleBufferMakeDataReady(newSampleBuffer); // How is this different; CMSampleBufferSetDataReady ?
//if (myStatus != 0) NSLog(#"CMSampleBufferMakeDataReady() myStatus: %i",myStatus);
imageSourceTime = CMSampleBufferGetPresentationTimeStamp(newSampleBuffer);
CMTimeShow(imageSourceTime);
appendSuccessFlag = [inputWriterBuffer appendSampleBuffer:newSampleBuffer];
//CMSampleBufferInvalidate(sampleBuffer); // Docs don't describe action. WTF does it do? Doesn't seem to affect my problem. Used with CMSampleBufferSetInvalidateCallback maybe?
//CFRelease(sampleBuffer); // - Not surprisingly - “EXC_BAD_ACCESS”
}
if (!appendSuccessFlag)
{
NSLog(#"Failed to append pixel buffer");
}
else {
writtenFrames++;
NSLog(#"writtenFrames: %i", writtenFrames);
}
}
//[self displayOuptutWritterStatus]; // Expect and see AVAssetWriterStatusWriting.
}
My setup routine.
- (IBAction) recordingStartStop: (id) sender
{
NSError * error;
if (self.isRecording) {
NSLog(#"~~~~~~~~~ STOPPING RECORDING ~~~~~~~~~");
self.isRecording = NO;
[recordingStarStop setTitle: #"Record" forState: UIControlStateNormal];
//[self.captureSession stopRunning];
[inputWriterBuffer markAsFinished];
[outputWriter endSessionAtSourceTime:imageSourceTime];
[outputWriter finishWriting]; // Blocks until file is completely written, or an error occurs.
NSLog(#"finished CMtime");
CMTimeShow(imageSourceTime);
// Really, I should loop through the outputs and close all of them or target specific ones.
// Since I'm only recording video right now, I feel safe doing this.
[self.captureSession removeOutput: [[self.captureSession outputs] objectAtIndex: 0]];
[videoOutput release];
[inputWriterBuffer release];
[outputWriter release];
videoOutput = nil;
inputWriterBuffer = nil;
outputWriter = nil;
NSLog(#"~~~~~~~~~ STOPPED RECORDING ~~~~~~~~~");
NSLog(#"Calling UIVideoAtPathIsCompatibleWithSavedPhotosAlbum.");
NSLog(#"filePath: %#", [projectPaths movieFilePath]);
if (UIVideoAtPathIsCompatibleWithSavedPhotosAlbum([projectPaths movieFilePath])) {
NSLog(#"Calling UISaveVideoAtPathToSavedPhotosAlbum.");
UISaveVideoAtPathToSavedPhotosAlbum ([projectPaths movieFilePath], self, #selector(video:didFinishSavingWithError: contextInfo:), nil);
}
NSLog(#"~~~~~~~~~ WROTE RECORDING to PhotosAlbum ~~~~~~~~~");
}
else {
NSLog(#"~~~~~~~~~ STARTING RECORDING ~~~~~~~~~");
projectPaths = [[ProjectPaths alloc] initWithProjectFolder: #"TestProject"];
intervalFrames = 30;
videoOutput = [[AVCaptureVideoDataOutput alloc] init];
NSMutableDictionary * cameraVideoSettings = [[[NSMutableDictionary alloc] init] autorelease];
NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;
NSNumber* value = [NSNumber numberWithUnsignedInt: kCVPixelFormatType_32BGRA]; //kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange];
[cameraVideoSettings setValue: value forKey: key];
[videoOutput setVideoSettings: cameraVideoSettings];
[videoOutput setMinFrameDuration: CMTimeMake(20, 600)]; //CMTimeMake(1, 30)]; // 30fps
[videoOutput setAlwaysDiscardsLateVideoFrames: YES];
queue = dispatch_queue_create("cameraQueue", NULL);
[videoOutput setSampleBufferDelegate: self queue: queue];
dispatch_release(queue);
NSMutableDictionary *outputSettings = [[[NSMutableDictionary alloc] init] autorelease];
[outputSettings setValue: AVVideoCodecH264 forKey: AVVideoCodecKey];
[outputSettings setValue: [NSNumber numberWithInt: 1280] forKey: AVVideoWidthKey]; // currently assuming
[outputSettings setValue: [NSNumber numberWithInt: 720] forKey: AVVideoHeightKey];
NSMutableDictionary *compressionSettings = [[[NSMutableDictionary alloc] init] autorelease];
[compressionSettings setValue: AVVideoProfileLevelH264Main30 forKey: AVVideoProfileLevelKey];
//[compressionSettings setValue: [NSNumber numberWithDouble:1024.0*1024.0] forKey: AVVideoAverageBitRateKey];
[outputSettings setValue: compressionSettings forKey: AVVideoCompressionPropertiesKey];
inputWriterBuffer = [AVAssetWriterInput assetWriterInputWithMediaType: AVMediaTypeVideo outputSettings: outputSettings];
[inputWriterBuffer retain];
inputWriterBuffer.expectsMediaDataInRealTime = YES;
outputWriter = [AVAssetWriter assetWriterWithURL: [projectPaths movieURLPath] fileType: AVFileTypeQuickTimeMovie error: &error];
[outputWriter retain];
if (error) NSLog(#"error for outputWriter = [AVAssetWriter assetWriterWithURL:fileType:error:");
if ([outputWriter canAddInput: inputWriterBuffer]) [outputWriter addInput: inputWriterBuffer];
else NSLog(#"can not add input");
if (![outputWriter canApplyOutputSettings: outputSettings forMediaType:AVMediaTypeVideo]) NSLog(#"ouptutSettings are NOT supported");
if ([captureSession canAddOutput: videoOutput]) [self.captureSession addOutput: videoOutput];
else NSLog(#"could not addOutput: videoOutput to captureSession");
//[self.captureSession startRunning];
self.isRecording = YES;
[recordingStarStop setTitle: #"Stop" forState: UIControlStateNormal];
writtenFrames = 0;
imageSourceTime = kCMTimeZero;
[outputWriter startWriting];
//[outputWriter startSessionAtSourceTime: imageSourceTime];
NSLog(#"~~~~~~~~~ STARTED RECORDING ~~~~~~~~~");
NSLog (#"recording to fileURL: %#", [projectPaths movieURLPath]);
}
NSLog(#"isRecording: %#", self.isRecording ? #"YES" : #"NO");
[self displayOuptutWritterStatus];
}
OK, I found the bug in my first post.
When using
myStatus = CMSampleBufferCreateCopyWithNewTiming(kCFAllocatorDefault,
sampleBuffer,
1,
&sampleTimingInfo,
&newSampleBuffer);
you need to balance that with a CFRelease(newSampleBuffer);
The same idea holds true when using a CVPixelBufferRef with a piexBufferPool of an AVAssetWriterInputPixelBufferAdaptor instance. You would use CVPixelBufferRelease(yourCVPixelBufferRef); after calling the appendPixelBuffer: withPresentationTime: method.
Hope this is helpful to someone else.
With a little more searching and reading I have a working solution. Don't know that it is best method, but so far, so good.
In my setup area I've setup an AVAssetWriterInputPixelBufferAdaptor. The code addition looks like this.
InputWriterBufferAdaptor = [AVAssetWriterInputPixelBufferAdaptor
assetWriterInputPixelBufferAdaptorWithAssetWriterInput: inputWriterBuffer
sourcePixelBufferAttributes: nil];
[inputWriterBufferAdaptor retain];
For completeness to understand the code below, I also have these three lines in the setup method.
fpsOutput = 30; //Some possible values: 30, 10, 15 24, 25, 30/1.001 or 29.97;
cmTimeSecondsDenominatorTimescale = 600 * 100000; //To more precisely handle 29.97.
cmTimeNumeratorValue = cmTimeSecondsDenominatorTimescale / fpsOutput;
Instead of applying a retiming to a copy of the sample buffer. I now have the following three lines of code that effectively does the same thing. Notice the withPresentationTime parameter for the adapter. By passing my custom value to that, I gain the correct timing I'm seeking.
CVPixelBufferRef myImage = CMSampleBufferGetImageBuffer( sampleBuffer );
imageSourceTime = CMTimeMake( writtenFrames * cmTimeNumeratorValue, cmTimeSecondsDenominatorTimescale);
appendSuccessFlag = [inputWriterBufferAdaptor appendPixelBuffer: myImage withPresentationTime: imageSourceTime];
Use of the AVAssetWriterInputPixelBufferAdaptor.pixelBufferPool property may have some gains, but I haven't figured that out.