I am a long time reader, first time poster on StackOverflow, and must say it has been a great source of knowledge for me.
I am trying to get to know the AVFoundation framework.
What I want to do is save what the camera sees and then detect when something changes.
Here is the part where I save the image to a UIImage :
if (shouldSetBackgroundImage) {
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
// Create a bitmap graphics context with the sample buffer data
CGContextRef context = CGBitmapContextCreate(rowBase, bufferWidth,
bufferHeight, 8, bytesPerRow,
colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
// Create a Quartz image from the pixel data in the bitmap graphics context
CGImageRef quartzImage = CGBitmapContextCreateImage(context);
// Free up the context and color space
CGContextRelease(context);
CGColorSpaceRelease(colorSpace);
// Create an image object from the Quartz image
UIImage * image = [UIImage imageWithCGImage:quartzImage];
[self setBackgroundImage:image];
NSLog(#"reference image actually set");
// Release the Quartz image
CGImageRelease(quartzImage);
//Signal that the image has been saved
shouldSetBackgroundImage = NO;
}
and here is the part where I check if there is any change in the image seen by the camera :
else {
CGImageRef cgImage = [backgroundImage CGImage];
CGDataProviderRef provider = CGImageGetDataProvider(cgImage);
CFDataRef bitmapData = CGDataProviderCopyData(provider);
char* data = CFDataGetBytePtr(bitmapData);
if (data != NULL)
{
int64_t numDiffer = 0, pixelCount = 0;
NSMutableArray * pointsMutable = [NSMutableArray array];
for( int row = 0; row < bufferHeight; row += 8 ) {
for( int column = 0; column < bufferWidth; column += 8 ) {
//we get one pixel from each source (buffer and saved image)
unsigned char *pixel = rowBase + (row * bytesPerRow) + (column * BYTES_PER_PIXEL);
unsigned char *referencePixel = data + (row * bytesPerRow) + (column * BYTES_PER_PIXEL);
pixelCount++;
if ( !match(pixel, referencePixel, matchThreshold) ) {
numDiffer++;
[pointsMutable addObject:[NSValue valueWithCGPoint:CGPointMake(SCREEN_WIDTH - (column/ (float) bufferHeight)* SCREEN_WIDTH - 4.0, (row/ (float) bufferWidth)* SCREEN_HEIGHT- 4.0)]];
}
}
}
numberOfPixelsThatDiffer = numDiffer;
points = [pointsMutable copy];
}
For some reason, this doesn't work, meaning that the iPhone detects almost everything as being different from the saved image, even though I set a very low threshold for detection in the match function...
Do you have any idea of what I am doing wrong?
There are three possibilities I can think of for why you might be seeing nearly every pixel be different: colorspace conversions, incorrect mapping of pixel locations, or your thresholding being too sensitive for the actual movement of the iPhone camera. The first two aren't very likely, so I think it might be the third, but they're worth checking.
There might be some color correction going on when you place your pixels within a UIImage, then extract them later. You could try simply storing them in their native state from the buffer, then using that original buffer as the point of comparison, not the UIImage's backing data.
Also, check to make sure that your row / column arithmetic works out for the actual pixel locations in both images. Perhaps generate a difference image the absolute difference of subtracting the two images, then use a simple black / white divided area as a test image for the camera.
The most likely case is that the overall image is shifting by more than one pixel simply through the act of a human hand holding it. These whole-frame image shifts could cause almost every pixel to be different in a simple comparison. You may need to adjust your thresholding or do more intelligent motion estimation, like is used in video compression routines.
Finally, when it comes to the comparison operation, I'd recommend taking a look at OpenGL ES 2.0 shaders for performing this. You should see a huge speedup (14-28X in my benchmarks) over doing this pixel-by-pixel comparison on the CPU. I show how to do color-based thresholding using the GPU in this article, which has this iPhone sample application that tracks colored objects in real time using GLSL shaders.
Human eyes are way much different than a camera (even a very expensive one) in the way that we don't perceive minimal light changes or small motion changes. Cameras DO, they are very sensitive but not smart at all!
With your current approach (it seems you are comparing each pixel):
What would happen if the frame is shifted only 1 pixel to the right?! You can image right the result of your algorithm, right?. Humans will perceive nothing or almost nothing.
There is also the camera shutter problem: That means that every frame might not have the same amount of light. Hence, a pixel-by-pixel comparison method is too prone to fail.
You want to at least pre-process your image and extract some basic features. Maybe edges, corners, etc. OpenCV is easy for that but I am not sure that doing such a processing will be fast in the iPhone. (It depends on your image size)
Alternatively you can try the naive template matching algorithm with a template size that will be a little short than your hole view size.
Image Processing is computationally expensive so don't expect it to be fast from the first time, specially in a mobile device and even more if you don't have experience in Image Processing/Computer Vision stuff.
Hope it helps ;)
Related
I have a set of mask images that I need to use everytime I recognise a previously-known scene on my camera. All the mask images are in IplImage format. There will be instances where, for example, the camera has panned to a slightly different but nearby location. this means that if I do a template matching somewhere in the middle of the current scene, I will be able to recognise the scene with some amount of shift of the template in this scene. All I need to do is use those shifts to adjust the mask image ROIs so that they can be overlayed appropriately based on the template-matching. I know that there are functions such as:
cvSetImageROI(Iplimage* img, CvRect roi)
cvResetImageROI(IplImage* img);
Which I can use to set crop/uncrop my image. However, it didn't work for me quit the way I expected. I would really appreciate if someone could suggest an alternative or what I am doing wrong, or even what I haven't thought of!
**I must also point out that I need to keep the image size same at all times. The only thing that will be different is the actual area of interest in the image. I can probably use the zero/one padding to cover the unused areas.
I believe a solution without making too many copies of the original image would be:
// Make a new IplImage
IplImage* img_src_cpy = cvCreateImage(cvGetSize(img_src), img_src->depth, img_src->nChannels);
// Crop Original Image without changing the ROI
for(int rows = roi.y; rows < roi.height; rows++) {
for(int cols = roi.x; rows < roi.width; cols++) {
img_src_cpy->imageData[(rows-roi.y)*img_src_cpy->widthStep + (cols-roi.x)] = img_src[rows*img_src + cols];
}
{
//Now copy everything to the original image OR simply return the new image if calling from a function
cvCopy(img_src_cpy, img_src); // OR return img_src_cpy;
I tried the code out on itself and it is also fast enough for me (executes in about 1 ms for 332 x 332 Greyscale image)
Original image:
Filtered image:
I am trying to crop UIImages (photos in a phone's camera roll) into squares. Here is part of the code I am using, where 'image' is the image that is being cropped:
if( image.size.height > image.size.width )
{
dimension = image.size.width;
imageRef = CGImageCreateWithImageInRect([image CGImage], CGRectMake((image.size.height-dimension)/2, 0, dimension, dimension));
If I am using the original image, it looks like this at this point:
Which is fine and what I expect - I have a rotation algorithm not shown here that sorts this out.
If I'm using the filtered image, it looks like this:
...not square cropped, but weirdly zoomed in instead. So this seems to be where the problem lies, and I don't know why these filtered images are behaving differently.
}
else
{
dimension = image.size.height;
imageRef = CGImageCreateWithImageInRect([image CGImage], CGRectMake((image.size.width-dimension)/2, 0, dimension, dimension));
}
CGBitmapInfo bitmapInfo = CGImageGetBitmapInfo(imageRef);
CGColorSpaceRef colorSpaceInfo = CGImageGetColorSpace(imageRef);
CGContextRef bitmap;
bitmap = CGBitmapContextCreate(NULL, dimension, dimension, CGImageGetBitsPerComponent(imageRef), CGImageGetBytesPerRow(imageRef), colorSpaceInfo, bitmapInfo);
My problem is that at that last line, CGBitmapContextCreate, I sometimes get the following error:
<Error>: CGBitmapContextCreate: invalid data bytes/row: should be at least 7744 for 8 integer bits/component, 3 components, kCGImageAlphaNoneSkipLast.
What's odd is that usually, this doesn't happen - so far, I've only encountered this error when the original image has height greater than width, and it has been filtered by another app called Camera+...the exact same photo before filtering causes no problems, and a filtered landscape photo also seems to be fine.
Can anyone guide me here or help me explain what's actually going on? I understand enough from the error message to know that if I replace CGImageGetBytesPerRow(imageRef) with some arbitrary number higher than 7744, the error no longer happens, but I don't understand enough about this CGImage stuff to know what effect that's actually having on anything, and it doesn't seem to me like much of an actual solution. This code is based on other cropping examples I've seen on the web, and so my understanding of these bitmap functions is limited.
Any thoughts would be hugely appreciated!
EDIT
I found this question on SO: In this CGBitmapContextCreate, why is bytesPerRow 0? and it prompted me to try setting the bytesPerRow parameter to 0. Turns out this eliminates the error, but my cropping routine doesn't work properly in the same situations as when this error was occurring before. This might take a special person to answer, but does anyone know enough about image filtering to take a guess at why portrait-oriented, camera+ filtered photos are somehow being treated differently by this code? I've updated the title since the question has changed slightly.
EDIT2
I've added example images into the code above, and in the end, after any necessary rotating, the final cropped images look like this:
with original image: - perfect!
with filtered image: - terrible!
The code that's used to create these final, supposedly cropped images is this:
CGContextDrawImage(bitmap, CGRectMake(0, 0, dimension, dimension), imageRef);
CGImageRef ref = CGBitmapContextCreateImage(bitmap);
image = [UIImage imageWithCGImage:ref];
UIImageWriteToSavedPhotosAlbum(image, self, #selector(image:didFinishSavingWithError:contextInfo:), nil);
In short - camera+ filtered images, for some reason, are coming in with a different imageOrientation value than the original images. Somehow, this fact causes a line like this:
imageRef = CGImageCreateWithImageInRect([image CGImage], CGRectMake((image.size.height-dimension)/2, 0, dimension, dimension));
to behave differently depending on the imageOrientation of image. So, while the original image, whose image orientation was either right or left, was being rotated onto its side by this line (which is why I was cropping with an x offset in both portrait and landscape sizes), the filtered image's orientation is up. Because of this, I wasn't getting the rotation I expected, and so the filtered images were getting stretched. To solve this, I am checking the orientation of the image before calling CGImageCreateWithImageInRect, and if it is portrait-sized but has an up orientation, I crop with a y offset instead of x (like the line of code David H mentions below).
My guess is that calling [image CGImage] rotates the image to a relative up position...so if the orientation is right, the image gets rotated 90 degrees counter-clockwise, but if the orientation is up, it doesn't get rotated at all. I still don't understand why filtered images end up with a different orientation than their originals, but I assume it is just some sort of side effect in camera+'s filtering code. All this orientation stuff could stand to be a lot simpler, but this seems to be the solution for now.
A few comments:
1) you want to round the hex value of the value when you divide by 2 so as to not get on a fractional pixel boundary (user roundf())
2) you do not handle the case of both dimensions the same
3) in the first create, you are setting the x offset not the y - use this modfied line:
imageRef = CGImageCreateWithImageInRect([image CGImage], CGRectMake(0, (image.size.height-dimension)/2, dimension, dimension));
I need to change the white skin face to dark skin face...
For example American white face to African face(i.e color tone)...
I pick the color value of the pixel by digital color meter it gives the RGB value[red=101,green=63 and blue=43] for dark skin and for white skin it gives the RGB value as [red=253,green=210 and blue=176]...
Then i am setting that value in my code it gives the false result...
Here is my code...
-(UIImage*)customBlackFilterOriginal
{
CGImageRef imgSource=self.duplicateImage.image.CGImage;
CFDataRef m_DataRef1 = CGDataProviderCopyData(CGImageGetDataProvider(imgSource));
UInt8 *dataOriginal=(UInt8 *)CFDataGetBytePtr(m_DataRef1);
double lengthSource=CFDataGetLength(m_DataRef1);
NSLog(#"length::%f",lengthSource);
int redPixel;
int greenPixel;
int bluePixel;
for(int index=0;index<lengthSource;index+=4)
{
dataOriginal[index]=dataOriginal[index];
dataOriginal[index+1]= 101;
dataOriginal[index+2]= 63;
dataOriginal[index+3]=43;
}
NSUInteger width =CGImageGetWidth(imgSource);
size_t height=CGImageGetHeight(imgSource);
size_t bitsPerComponent=CGImageGetBitsPerComponent(imgSource);
size_t bitsPerPixel=CGImageGetBitsPerPixel(imgSource);
size_t bytesPerRow=CGImageGetBytesPerRow(imgSource);
NSLog(#"the w:%u H:%lu",width,height);
CGColorSpaceRef colorspace=CGImageGetColorSpace(imgSource);
CGBitmapInfo bitmapInfo=CGImageGetBitmapInfo(imgSource);
CFDataRef newData=CFDataCreate(NULL,dataOriginal,lengthSource);
CGDataProviderRef provider=CGDataProviderCreateWithCFData(newData);
CGImageRef newImg=CGImageCreate(width,height,bitsPerComponent,bitsPerPixel,bytesPerRow,colorspace,bitmapInfo,provider,NULL,true,kCGRenderingIntentDefault);
return [UIImage imageWithCGImage:newImg];
}
please share any idea about the above color changing....
what mistake i did in the code?..
I am not an IPhone programmer so I can't test anything but some things are odd in your code:
Pixel size
When reading your data, you seem to assume you have a 32bits ARGB picture, did you validate it's the case?
CFDataGetBytePtr
According to the docs, it Returns a read-only pointer to the bytes of a CFData object., are you sure you're not looking for CFDataGetBytes which Copies the byte contents of a CFData object to an external buffer. In which case you'll have to allocate your buffer to contain width * height * bpp. Once you have this copy, you can manipulate it anyway you want to create the new picture.
Pixel Selection
According to your question, I seem to understand that you want to change skin color from White to Black. Your current code iterates over every pixel to change its color. You should evaluate the "distance" between the pixel color and what you're looking for, and if it's below a certain threshold process it. It might be easier to perform the operation in HSV than by dealing with RGB colors.
I would like to render image in OpenGL ES, pixel by pixel. I want to do it this way because I plan to move those pixels over time to create various effect.
For performance and design reasons I decided to use only every other pixel in both directions (thus reducing their number to one quarter)
I have only very basic understanding of opengl, so I am probably missing some key knowledge to achieve this.
What is the best way to achieve this? Do I have to really render it pixel by pixel? Or can I somehow create texture out of array of pixels?
I would like to make this work on as much devices as possible (so OpenGL ES 1.1 solution is preffered, but if it is not possible or it would be really inconvenient or slow, 2.0 can be used)
I tried to do this using VBO with mixed results. I am not sure I have done it properly, because there are some problems (and it is very slow). Here is my code:
Initialization:
glGenBuffers(1, &pointsVBO);
glBindBuffer(GL_ARRAY_BUFFER, pointsVBO);
glBufferData(GL_ARRAY_BUFFER, 160*240*sizeof(Vertex), 0, GL_DYNAMIC_DRAW);
glEnableClientState(GL_VERTEX_ARRAY);
glEnableClientState(GL_COLOR_ARRAY);
Rendering:
- (void)renderPoints:(ImagePixel**)imagePixels {
int count = 160 * 240;
for(int i = 0; i < count; ++i) {
vertices[i].v[0] = imagePixels[i]->positionX;
vertices[i].v[1] = imagePixels[i]->positionY;
vertices[i].color[0] = imagePixels[i]->red;
vertices[i].color[1] = imagePixels[i]->green;
vertices[i].color[2] = imagePixels[i]->blue;
vertices[i].color[3] = 1;
}
glVertexPointer(2, GL_FLOAT, sizeof(Vertex), vertices[0].v);
glColorPointer(4, GL_FLOAT, sizeof(Vertex), vertices[0].color);
// update vbo
GLvoid *vbo_buffer = glMapBufferOES(GL_ARRAY_BUFFER, GL_WRITE_ONLY_OES);
memcpy(vbo_buffer, vertices, count * sizeof(Vertex));
glUnmapBufferOES(GL_ARRAY_BUFFER);
// draw contents of vbo
glDrawArrays(GL_POINTS, 0, count);
}
Vertex struct:
typedef struct Vertex
{
float v[2];
float color[4];
} Vertex;
imagePixels array is filled with data from image.
When I do this, I get most of my image but I am missing few last rows and I can see some random pixels around the screen. Is it possible that I hit some limit in drawArrays that uses only portion of data?
Second problem is, that points in second half of columns aren't aligned properly. I guess this is caused by rounding errors in float math when computing position during rendering itself (supplied coordinates are all multiples of 2). Is there any way how to prevent this? I need all points to be aligned in proper grid.
I will provide you with screenshot as soon as I get my iphone back
If you really are wanting to manipulate every pixel, you should probably just use a single full-screen quad in OpenGL and update its texture each frame.
You can create a texture out of a bitmap array of pixels using glTexImage2D.
How could I implement image's pixel data as
Array[i][j]= 255; without calculating byteIndex of raw data with bytesPerPixel and bytesPerRow like that:
int byteIndex = (bytesPerRow * j) + i * bytesPerPixel;
i and j are x,y coordinates of image.
Well, you can do this by allocating N rows of K elements, where N is the height of the image - and K - it's width.
Like this:
char ** image;
image = new char*[height];
for (size_t i=0;i<height;i++)
{
image[i] = new char[width];
}
But! Don't do this. It's the words idea to operate pixels in this way. First of all, when the image data are in planar form, it operates more efficiently since there are less cache-misses. Another reason - almost all image processing libraries uses planar memory model for storing the image data in memory.
I suggest to use ready image containers from either OpenCV (cv::Mat type) or Boost::Gil library. I prefefer Gil much more, since in provide explicilty typed image containers, like:
boost::gil::bgra8_image_t
boost::gil::gray32f_image_t
and so on...
And provide efficient and flexible way for doing color conversion (CMYK <-> HSV <-> RGBA <-> GRAY) and image transformations (90 degreee rotation, flipping, cropping, etc)