What is the best/fastest way to convert CMSampleBufferRef to OpenCV IplImage? - iphone

Question

I am writing an iPhone app that does some sort of real-time image detection with OpenCV. What is the best way to convert a CMSampleBufferRef image from the camera (I'm using AVCaptureVideoDataOutputSampleBufferDelegate of AVFoundation) into an IplImage that OpenCV understands? The conversion needs to be fast enough so it can run realtime.
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
// Convert CMSampleBufferRef into IplImage
IplImage *openCVImage = ???(sampleBuffer);
// Do OpenCV computations realtime
// ...
[pool release];
}
Thanks in advance.

Answer 1

This sample code is based on Apple's sample to manage CMSampleBuffer's pointer:
- (IplImage *)createIplImageFromSampleBuffer:(CMSampleBufferRef)sampleBuffer {
IplImage *iplimage = 0;
if (sampleBuffer) {
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer, 0);
// get information of the image in the buffer
uint8_t *bufferBaseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
size_t bufferWidth = CVPixelBufferGetWidth(imageBuffer);
size_t bufferHeight = CVPixelBufferGetHeight(imageBuffer);
// create IplImage
if (bufferBaseAddress) {
iplimage = cvCreateImage(cvSize(bufferWidth, bufferHeight), IPL_DEPTH_8U, 4);
iplimage->imageData = (char*)bufferBaseAddress;
}
// release memory
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
}
else
DLog(#"No sampleBuffer!!");
return iplimage;
}
You need to create a 4-channel IplImage because the Phone's camera buffer is in BGRA.
To my experience, this conversion is fast enough to be done in a real-time application, but of course, anything you will add to it will cost time, especially with OpenCV.

Answer 2

"iplimage->imageData = (char*)bufferBaseAddress;" will lead to memory leak.
It should be "memcpy(iplimage->imageData, (char*)bufferBaseAddress, iplimage->imageSize);"
so the complete coded is:
-(IplImage *)createIplImageFromSampleBuffer:(CMSampleBufferRef)sampleBuffer {
IplImage *iplimage = 0;
if (sampleBuffer) {
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer, 0);
// get information of the image in the buffer
uint8_t *bufferBaseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
size_t bufferWidth = CVPixelBufferGetWidth(imageBuffer);
size_t bufferHeight = CVPixelBufferGetHeight(imageBuffer);
// create IplImage
if (bufferBaseAddress) {
iplimage = cvCreateImage(cvSize(bufferWidth, bufferHeight), IPL_DEPTH_8U, 4);
//iplimage->imageData = (char*)bufferBaseAddress;
memcpy(iplimage->imageData, (char*)bufferBaseAddress, iplimage->imageSize);
}
// release memory
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
}
else
DLog(#"No sampleBuffer!!");
return iplimage;
}