How to detect and show eyes and mouth in camera in iPhone? - iphone

Question

For detecting face in camera I am reffering to SquareCam(iOS developer library). But I am unable to show left eye, right eye and mouth position. I am using the following code
NSArray *sublayers = [NSArray arrayWithArray:[previewLayer sublayers]];
NSInteger sublayersCount = [sublayers count], currentSublayer = 0;
NSInteger featuresCount = [features count], currentFeature = 0;
[CATransaction begin];
[CATransaction setValue:(id)kCFBooleanTrue forKey:kCATransactionDisableActions];
// hide all the face layers
for ( CALayer *layer in sublayers )
{
if ( [[layer name] isEqualToString:#"FaceView"] )
[layer setHidden:YES];
}
if ( featuresCount == 0 || !detectFaces ) {
[CATransaction commit];
return; // early bail.
}
CGSize parentFrameSize = [previewView frame].size;
NSString *gravity = [previewLayer videoGravity];
BOOL isMirrored = [previewLayer isMirrored];
CGRect previewBox = [SquareCamViewController videoPreviewBoxForGravity:gravity
frameSize:parentFrameSize
apertureSize:clap.size];
for ( CIFaceFeature *ff in features ) {
// find the correct position for the square layer within the previewLayer
// the feature box originates in the bottom left of the video frame.
// (Bottom right if mirroring is turned on)
CGRect faceRect = [ff bounds];
CGRect leftEyeFrameRect;
CGFloat temp = faceRect.size.width;
faceRect.size.width = faceRect.size.height;
faceRect.size.height = temp;
temp = faceRect.origin.x;
faceRect.origin.x = faceRect.origin.y;
faceRect.origin.y = temp;
// scale coordinates so they fit in the preview box, which may be scaled
CGFloat widthScaleBy = previewBox.size.width / clap.size.height;
CGFloat heightScaleBy = previewBox.size.height / clap.size.width;
faceRect.size.width *= widthScaleBy;
faceRect.size.height *= heightScaleBy;
faceRect.origin.x *= widthScaleBy;
faceRect.origin.y *= heightScaleBy;
if ( isMirrored )
{
faceRect = CGRectOffset(faceRect, previewBox.origin.x + previewBox.size.width - faceRect.size.width - (faceRect.origin.x * 2), previewBox.origin.y);
}
else
{
faceRect = CGRectOffset(faceRect, previewBox.origin.x, previewBox.origin.y);
leftEyeFrameRect=CGRectOffset(faceRect,ff.leftEyePosition.x, ff.leftEyePosition.y);
}
CALayer *featureLayer = nil;
CALayer *eyeLayer = nil;
// re-use an existing layer if possible
while ( !featureLayer && (currentSublayer < sublayersCount) )
{
CALayer *currentLayer = [sublayers objectAtIndex:currentSublayer++];
if ( [[currentLayer name] isEqualToString:#"FaceLayer"] ) {
featureLayer = currentLayer;
[currentLayer setHidden:NO];
}
}
// create a new one if necessary
if ( !featureLayer ) {
featureLayer = [CALayer new];
[featureLayer setContents:(id)[square CGImage]];
[featureLayer setName:#"FaceLayer"];
[previewLayer addSublayer:featureLayer];
[featureLayer release];
}
[featureLayer setFrame:faceRect];
if (faceView !=nil) {
[faceView removeFromSuperview];
[faceView release];
}
if (leftEyeView != nil) {
[leftEyeView removeFromSuperview];
[leftEyeView release];
}
faceView = [[UIView alloc] initWithFrame:CGRectMake(faceRect.origin.x, faceRect.origin.y ,faceRect.size.width, faceRect.size.height)];
faceView.layer.borderWidth = 1;
faceView.layer.borderColor = [[UIColor redColor] CGColor];
[self.view addSubview:faceView];
leftEyeView = [[UIView alloc] initWithFrame:CGRectMake(faceView.frame.origin.x+(faceView.frame.size.height/2), faceView.frame.origin.y+(faceView.frame.size.height*0.10) ,faceView.frame.size.width*0.40, faceView.frame.size.height*0.40)];
UIImageView *leftEyeImageView=[[UIImageView alloc] initWithImage:[UIImage imageNamed:#"eye.png"]];
leftEyeImageView.frame = CGRectMake(0, 0, faceView.frame.size.width*0.40, faceView.frame.size.height*0.40);
[leftEyeView addSubview:leftEyeImageView];
[self.view addSubview:leftEyeView];
if (ff.hasLeftEyePosition) {
CGPoint leftEyeCenter= ff.leftEyePosition;
UIView *vv= [[UIView alloc] initWithFrame:CGRectMake(leftEyeCenter.x, leftEyeCenter.y, 50, 50)];
vv.center = leftEyeCenter;
vv.layer.borderWidth= 4.0;
vv.layer.borderColor= [[UIColor blackColor]CGColor];
[self.view addSubview:vv];
}
It is detecting eye but not showing in right position. Can anyone help to resolve this.
Thanks in advance.

Answer 1

I faced the same problem as you in the preview when using the front camera, because the preview is mirrored and I could not find any good information to scale.
The following code is the closest I got. Please note that I have an image defined as property that is called heartImage and that I assume you are using the Apple Sample called SquareCam.
In the method - (void)drawFaceBoxesForFeatures:(NSArray *)features forVideoBox:(CGRect)clap orientation:(UIDeviceOrientation)orientation
if(ff.hasLeftEyePosition)
{
//swap coordinates
CGFloat leftEyeRectOriginX = ff.leftEyePosition.y ;
CGFloat leftEyeRectOriginY = ff.leftEyePosition.x ;
CGFloat leftEyeRectWidth = faceRect.size.width*0.3;
CGFloat leftEyeRectHeight = faceRect.size.width*0.3;
//adjust scale
leftEyeRectOriginX *= widthScaleBy;
leftEyeRectOriginY *= heightScaleBy;
NSLog(#"LeftEyePosition: %#", NSStringFromCGPoint(ff.leftEyePosition));
CGRect r = CGRectMake(leftEyeRectOriginX - (leftEyeRectWidth/2) , leftEyeRectOriginY - (leftEyeRectHeight/2), leftEyeRectWidth, leftEyeRectHeight);
if ( isMirrored ){
r = CGRectOffset(r, previewBox.origin.x + previewBox.size.width - (rightEyeRectOriginX*2) - rightEyeRectWidth+ faceRect.origin.x, previewBox.origin.y);
NSLog(#"LeftEyeRect mirrored: %#", NSStringFromCGRect(r));
}
else{
r = CGRectOffset(r, previewBox.origin.x, previewBox.origin.y);
}
while ( !leftEyeEyeLayer && (currentSublayer < sublayersCount) ) {
CALayer *currentLayer = [sublayers objectAtIndex:currentSublayer++];
if ( [[currentLayer name] isEqualToString:#"LeftEyeLayer"] ) {
leftEyeEyeLayer = currentLayer;
[currentLayer setHidden:NO];
}
}
// create a new one if necessary
if ( !leftEyeEyeLayer ) {
leftEyeEyeLayer = [CALayer new];
[leftEyeEyeLayer setContents:(id)[heartImage CGImage]];
[leftEyeEyeLayer setName:#"LeftEyeLayer"];
[previewLayer addSublayer:leftEyeEyeLayer];
[leftEyeEyeLayer release];
}
[leftEyeEyeLayer setFrame:r];
}
The same applies for the Right Eye, with the exception that I use this in case it's mirrored: r = CGRectOffset(r, previewBox.origin.x + previewBox.size.width - (rightEyeRectOriginX*2) - rightEyeRectWidth+ faceRect.origin.x, previewBox.origin.y);
.
The only difference with the sample code is that you first want to remove all the featureLayers, so some lines above my code looks like this:
// hide all the face layers
for ( CALayer *layer in sublayers ) {
if ( [[layer name] isEqualToString:#"FaceLayer"] || [[layer name] isEqualToString:#"LeftEyeLayer"] || [[layer name] isEqualToString:#"RightEyeLayer"] )
[layer setHidden:YES];
}
To be precise, I have troubles only in the live camera preview. When using the method to save picture in library (- (CGImageRef)newSquareOverlayedImageForFeatures:(NSArray *)features
inCGImage:(CGImageRef)backgroundImage
withOrientation:(UIDeviceOrientation)orientation
frontFacing:(BOOL)isFrontFacing
) it works correctly by using:
if(ff.hasLeftEyePosition)
{
CGRect r = CGRectMake(ff.leftEyePosition.x-faceWidth*0.15, ff.leftEyePosition.y-faceWidth*0.15, faceWidth*0.3, faceWidth*0.3);
CGContextDrawImage(bitmapContext, r, [rotatedHeartImage CGImage]);
}
Please let me know if and how I should improve my answer.

Answer 2

This is probably caused by incorrect orientations between your input, detector and output. If the faces are detected, then probably only the output coordinates need to be translated from landscape to portrait or vice versa. Otherwise, take a look here.

Answer 3

You could investigate in Haar cascade training files and OPENGL but that's a total different approuch. It does support iOS versions below 6.0 that's a pro. But it's way harder then the Squarecam sample of Apple (con).
This OpenGL was able to detect ears, eyes etc. There could already be some training files on the web.