I am taking an image, loading it via the screen context, and changing it pixel by pixel. I have a number of different filters that I am applying to the images, but the last thing I need to do is shift the color balance (similar to Photoshop) to make the red more cyan.
The code below shows how I am taking the image, getting the data, and going through the r/g/b values pixel by pixel:
CGImageRef sourceImage = theImage.image.CGImage;
CFDataRef theData;
theData = CGDataProviderCopyData(CGImageGetDataProvider(sourceImage));
UInt8 *pixelData = (UInt8 *) CFDataGetBytePtr(theData);
int dataLength = CFDataGetLength(theData);
int red = 0;
int green = 1;
int blue = 2;
for (int index = 0; index < dataLength; index += 4) {
int r = pixelData[index + red];
int g = pixelData[index + green];
int b = pixelData[index + blue];
// the color balancing would go here...
if (r < 0) r = 0;
if (g < 0) g = 0;
if (b < 0) b = 0;
if (r > 255) r = 255;
if (g > 255) g = 255;
if (b > 255) b = 255;
pixelData[index + red] = r;
pixelData[index + green] = g;
pixelData[index + blue] = b;
}
CGContextRef context;
context = CGBitmapContextCreate(pixelData,
CGImageGetWidth(sourceImage),
CGImageGetHeight(sourceImage),
8,
CGImageGetBytesPerRow(sourceImage),
CGImageGetColorSpace(sourceImage),
kCGImageAlphaPremultipliedLast);
CGImageRef newCGImage = CGBitmapContextCreateImage(context);
UIImage *newImage = [UIImage imageWithCGImage:newCGImage];
CGContextRelease(context);
CFRelease(theData);
CGImageRelease(newCGImage);
theImage.image = newImage;
I am doing a number of other things to the pixel data (setting the levels, desaturating) but I need to shift the red value toward cyan.
In Photoshop, I am able to do it via:
Image: Adjustment: Color Balance
and setting it to -30 0 0
I have not been able to find an algorithm or an example explaining how this color shift is performed. I have tried to subtract 30 from every red value, or setting a hard maximum at 255 - 30 (225), but those seem to clip the colors and not shift the values... Right now, I'm just hacking around on it, but pointers to a reference would help out a lot.
(NB: I am not able to use an OpenGL solution because I have to take the same algorithm and convert it to PHP/gd for a web server version of this application)
What you need to do is subtract the red and then correct the "lightness" to match the old color, for whatever measure of lightness you choose. Something like this should work:
// First, calculate the current lightness.
float oldL = r * 0.30 + g * 0.59 + b * 0.11;
// Adjust the color components. This changes lightness.
r = r - 30;
if (r < 0) r = 0;
// Now correct the color back to the old lightness.
float newL = r * 0.30 + g * 0.59 + b * 0.11;
if (newL > 0) {
r = r * oldL / newL;
g = g * oldL / newL;
b = b * oldL / newL;
}
Note this behaves somewhat oddly when given pure red (it will leave it unchanged). OTOH, GIMP (version 2.6.11) does the same thing in its color balance tool so I'm not too worried about it.
Related
const shift = (0xFF << 24);
Future<Image> convertYUV420toImageColor(CameraImage image) async {
try {
final int width = image.width;
final int height = image.height;
final int uvRowStride = image.planes[1].bytesPerRow;
final int uvPixelStride = image.planes[1].bytesPerPixel;
print("uvRowStride: " + uvRowStride.toString());
print("uvPixelStride: " + uvPixelStride.toString());
// imgLib -> Image package from https://pub.dartlang.org/packages/image
var img = imglib.Image(width, height); // Create Image buffer
// Fill image buffer with plane[0] from YUV420_888
for(int x=0; x < width; x++) {
for(int y=0; y < height; y++) {
final int uvIndex = uvPixelStride * (x/2).floor() + uvRowStride*(y/2).floor();
final int index = y * width + x;
final yp = image.planes[0].bytes[index];
final up = image.planes[1].bytes[uvIndex];
final vp = image.planes[2].bytes[uvIndex];
// Calculate pixel color
int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
int g = (yp - up * 46549 / 131072 + 44 -vp * 93604 / 131072 + 91).round().clamp(0, 255);
int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);
// color: 0x FF FF FF FF
// A B G R
img.data[index] = shift | (b << 16) | (g << 8) | r;
}
}
imglib.PngEncoder pngEncoder = new imglib.PngEncoder(level: 0, filter: 0);
List<int> png = pngEncoder.encodeImage(img);
muteYUVProcessing = false;
return Image.memory(png);
} catch (e) {
print(">>>>>>>>>>>> ERROR:" + e.toString());
}
return null;
}
I have been following this code snippet from How to convert Camera Image to Image in Flutter? to convert YUV to RGB to send the images via WebSockets for ML prediction.
Although it works to convert, the resulting image is rotated 90 degrees and the performance is a little bit slow. How I can rotate it?
replace img.data[index] = shift | (b << 16) | (g << 8) | r;
with
if (img.boundsSafe(height-y, x)){
img.setPixelRgba(height-y, x, r , g ,b ,shift);
}
and replace var img = imglib.Image(width, height);
with
var img = imglib.Image(height, width);
For IOS version, the CameraImage is returned as biplanar which has only two planes.
Quote from image_format_group.dart:
/// Multi-plane YUV 420 format.
/// This format is a generic YCbCr format, capable of describing any 4:2:0
/// chroma-subsampled planar or semiplanar buffer (but not fully interleaved),
/// with 8 bits per color sample.
/// On Android, this is `android.graphics.ImageFormat.YUV_420_888`. See
/// https://developer.android.com/reference/android/graphics/ImageFormat.html#YUV_420_888
/// On iOS, this is `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange`. See
/// https://developer.apple.com/documentation/corevideo/1563591-pixel_format_identifiers/kcvpixelformattype_420ypcbcr8biplanarvideorange?language=objc
yuv420
For my approach, I use dart:ffi and link to c++ for Android CameraImage following the tutorial from here. The conversion between YUV420p and YUV420sp can be found here. There is no complete code for the conversion yet, but neither any solution for IOS around the forum.
Currently I have a Uint8List, formatted like [R,G,B,R,G,B,...] for all the pixels of the image. And of course I have its width and height.
I found decodeImageFromPixels while searching but it only takes RGBA/BGRA format. I converted my pixmap from RGB to RGBA and this function works fine.
However, my code now looks like this:
Uint8List rawPixel = raw.value.asTypedList(w * h * channel);
List<int> rgba = [];
for (int i = 0; i < rawPixel.length; i++) {
rgba.add(rawPixel[i]);
if ((i + 1) % 3 == 0) {
rgba.add(0);
}
}
Uint8List rgbaList = Uint8List.fromList(rgba);
Completer<Image> c = Completer<Image>();
decodeImageFromPixels(rgbaList, w, h, PixelFormat.rgba8888, (Image img) {
c.complete(img);
});
I have to make a new list(waste in space) and iterate through the entire list(waste in time).
This is too inefficient in my opinion, is there any way to make this more elegant? Like add a new PixelFormat.rgb888?
Thanks in advance.
You may find that this loop is faster as it doesn't keep appending to the list and then copy it at the end.
final rawPixel = raw.value.asTypedList(w * h * channel);
final rgbaList = Uint8List(w * h * 4); // create the Uint8List directly as we know the width and height
for (var i = 0; i < w * h; i++) {
final rgbOffset = i * 3;
final rgbaOffset = i * 4;
rgbaList[rgbaOffset] = rawPixel[rgbOffset]; // red
rgbaList[rgbaOffset + 1] = rawPixel[rgbOffset + 1]; // green
rgbaList[rgbaOffset + 2] = rawPixel[rgbOffset + 2]; // blue
rgbaList[rgbaOffset + 3] = 255; // a
}
An alternative is to prepend the array with a BMP header by adapting this answer (though it would simpler as there would be no palette) and passing that bitmap to instantiateImageCodec as that code is presumably highly optimized for parsing bitmaps.
I'm trying to create image filter that will shift color of the image. In order to do this I need to convert rgb color to hsl and after shift, convert hsl back to rgb. I make some researches and found formulas that can help me with this task.
I implement them in my playground using Swift just to test if they are reliable and they are. I won't post Swift code here just to keep things clean, but I'll show my test results:
input: rgb (61, 117,237) or (0.24,0.46,0.93)
result:
rgb2hsl [0.613527 0.831325 0.585] or (221, 83, 58.5) //hsl
hsl2rgb [0.24 0.46 0.93] //back to rgb
Great! So far so good.
Now we need to convert our Swift code to Core Image Kernel Language (CIKL).
And here it is:
float hue2rgb(float f1, float f2, float hue) {
if (hue < 0) {
hue += 1.0;
}
else if (hue > 1) {
hue -= 1.0;
}
float res;
if (6*hue<1) {
res = f1 + (f2 - f1) * 6 * hue;
}
else if (2*hue<1) {
res = f2;
}
else if (3*hue<2) {
res = f1 + (f2 - f1) * (2.0/3.0 - hue) * 6;
}
else {
res = f1;
}
return res;
}
vec3 hsl2rgb(vec3 hsl) {
vec3 rgb;
if (hsl.y == 0) {
rgb = vec3(hsl.z,hsl.z,hsl.z);
}
else {
float f2;
if (hsl.z < 0.5) {
f2 = hsl.z * (1.0 + hsl.y);
}
else {
f2 = hsl.z + hsl.y - hsl.y * hsl.z;
}
float f1 = 2 * hsl.z - f2;
float r = hue2rgb(f1, f2, hsl.x + 1.0/3.0);
float g = hue2rgb(f1, f2, hsl.x);
float b = hue2rgb(f1, f2, hsl.x - 1.0/3.0);
rgb = vec3(r,g,b);
}
return rgb;
}
vec3 rgb2hsl(vec3 rgb) {
float maxC = max(rgb.x, max(rgb.y,rgb.z));
float minC = min(rgb.x, min(rgb.y,rgb.z));
float l = (maxC + maxC)/2.0;
float h = 0;
float s = 0;
if (maxC != minC) {
float d = maxC - minC;
s = l > 0.5 ? d / (2.0 - maxC - minC) : d / (maxC + minC);
if (maxC == rgb.x) {
h = (rgb.y - rgb.z) / d + (rgb.y < rgb.z ? 6.0 : 0);
} else if (maxC == rgb.y) {
h = (rgb.z - rgb.x) / d + 2.0;
}
else {
h = (rgb.x - rgb.y) / d + 4.0;
}
h /= 6.0;
}
return vec3(h,s,l);
}
And here comes the problem. I'm not able to get right values using this functions in my filter. To check everything I made a Quartz Composer Patch.
Since I didn't find any print/log option in CIKL, I made this to check if my conversions work right:
The logic of this patch: my filter takes color as an input, convert it to hsl and back to rgb and returns it; image input ignored for now.
Kernel func of my filter:
kernel vec4 kernelFunc(__sample pixel, __color color) {
vec3 vec = color.rgb;
vec3 hsl = rgb2hsl(vec);
return vec4(hsl2rgb(hsl), 1);
}
Filter includes functions listed above.
The result I see in the viewer is:
Image on the right is cropped constant color image from the input color.
The left image is the output from our filter.
Digital color picker returns rgb (237, 239.7, 252) for left image.
I have no more ideas how to debug this thing and find a problem. Any help will be highly appreciated. Thanks.
I found the problem. It was me, converting code from Swift to CIKL I made a stupid mistake that was very hard to find, because you have no print / log tools in CIKL or I don't know about it.
Anyway, the problem was in the rgb2hsl function:
float l = (maxC + maxC)/2.0; // WRONG
it should be:
float l = (maxC + minC)/2.0;
Hope it'll help someone in the future.
How can I save a frame via TangoService_connectOnFrameAvailable() and display it correctly on my computer? As this reference page mentions, the pixels are stored in the HAL_PIXEL_FORMAT_YV12 format. In my callback function for TangoService_connectOnFrameAvailable, I save the frame like this:
static void onColorFrameAvailable(void* context, TangoCameraId id, const TangoImageBuffer* buffer)
{
...
std::ofstream fp;
fp.open(imagefile, std::ios::out | std::ios::binary );
int offset = 0;
for(int i = 0; i < buffer->height*2 + 1; i++) {
fp.write((char*)(buffer->data + offset), buffer->width);
offset += buffer->stride;
}
fp.close();
}
Then to get rid of the meta data in the first row and to display the image I run:
$ dd if="input.raw" of="new.raw" bs=1 skip=1280
$ vooya new.raw
I was careful to make sure in vooya that the channel order is yvu. The resulting output is:
What am I doing wrong in saving the image and displaying it?
UPDATE per Mark Mullin's response:
int offset = buffer->stride; // header offset
// copy Y channel
for(int i = 0; i < buffer->height; i++) {
fp.write((char*)(buffer->data + offset), buffer->width);
offset += buffer->stride;
}
// copy V channel
for(int i = 0; i < buffer->height / 2; i++) {
fp.write((char*)(buffer->data + offset), buffer->width / 2);
offset += buffer->stride / 2;
}
// copy U channel
for(int i = 0; i < buffer->height / 2; i++) {
fp.write((char*)(buffer->data + offset), buffer->width / 2);
offset += buffer->stride / 2;
}
This now shows the picture below, but there are still some artifacts; I wonder if that's from the Tango tablet camera or my processing of the raw data... any thoughts?
Can't say exactly what you're doing wrong AND tango images often have artifacts in them - yours are new, but I often see baby blue as a color where glare seems to be annoying deeper systems, and as it begins to loose sync with the depth system under load, you'll often see what looks like a shiny grid (its the IR pattern, I think) - At the end, any rational attempt to handle the image with openCV etc failed, so I hand wrote the decoder with some help from SO thread here
That said, given imagebuffer contains a pointer to the raw data from Tango, and various other variables like height and stride are filled in from the data received in the callback, then this logic will create an RGBA map - yeah, I optimized the math in it, so it's a little ugly - it's slower but functionally equivalent twin is listed second. My own experience says its a horrible idea to try and do this decode right in the callback (I believe Tango is capable of loosing sync with the flash for depth for purely spiteful reasons), so mine runs at the render stage.
Fast
uchar* pData = TangoData::cameraImageBuffer;
uchar* iData = TangoData::cameraImageBufferRGBA;
int size = (int)(TangoData::imageBufferStride * TangoData::imageBufferHeight);
float invByte = 0.0039215686274509803921568627451; // ( 1 / 255)
int halfi, uvOffset, halfj, uvOffsetHalfj;
float y_scaled, v_scaled, u_scaled;
int uOffset = size / 4 + size;
int halfstride = TangoData::imageBufferStride / 2;
for (int i = 0; i < TangoData::imageBufferHeight; ++i)
{
halfi = i / 2;
uvOffset = halfi * halfstride;
for (int j = 0; j < TangoData::imageBufferWidth; ++j)
{
halfj = j / 2;
uvOffsetHalfj = uvOffset + halfj;
y_scaled = pData[i * TangoData::imageBufferStride + j] * invByte;
v_scaled = 2 * (pData[uvOffsetHalfj + size] * invByte - 0.5f) * Vmax;
u_scaled = 2 * (pData[uvOffsetHalfj + uOffset] * invByte - 0.5f) * Umax;
*iData++ = (uchar)((y_scaled + 1.13983f * v_scaled) * 255.0);;
*iData++ = (uchar)((y_scaled - 0.39465f * u_scaled - 0.58060f * v_scaled) * 255.0);
*iData++ = (uchar)((y_scaled + 2.03211f * u_scaled) * 255.0);
*iData++ = 255;
}
}
Understandable
for (int i = 0; i < TangoData::imageBufferHeight; ++i)
{
for (int j = 0; j < TangoData::imageBufferWidth; ++j)
{
uchar y = pData[i * image->stride + j];
uchar v = pData[(i / 2) * (TangoData::imageBufferStride / 2) + (j / 2) + size];
uchar u = pData[(i / 2) * (TangoData::imageBufferStride / 2) + (j / 2) + size + (size / 4)];
YUV2RGB(y, u, v);
*iData++ = y;
*iData++ = u;
*iData++ = v;
*iData++ = 255;
}
}
I think that there is a better way to do if you can to do it offline.
The best way to save the image should be something like this (don't forgot to create the folder Pictures or you won't save anything)
void onFrameAvailableRouter(void* context, TangoCameraId id, const TangoImageBuffer* buffer) {
//To write the image in a txt file.
std::stringstream name_stream;
name_stream.setf(std::ios_base::fixed, std::ios_base::floatfield);
name_stream.precision(3);
name_stream << "/storage/emulated/0/Pictures/"
<<cur_frame_timstamp_
<<".txt";
std::fstream f(name_stream.str().c_str(), std::ios::out | std::ios::binary);
// size = 1280*720*1.5 to save YUV or 1280*720 to save grayscale
int size = stride_ * height_ * 1.5;
f.write((const char *) buffer->data,size * sizeof(uint8_t));
f.close();
}
Then to convert the .txt file to png you can do this
inputFolder = "input"
outputFolderRGB = "output/rgb"
outputFolderGray = "output/gray"
input_filename = "timestamp.txt"
output_filename = "rgb.png"
allFile = listdir(inputFolder)
numberOfFile = len(allFile)
if "input" in glob.glob("*"):
if "output/rgb" in glob.glob("output/*"):
print ""
else:
makedirs("output/rgb")
if "output/gray" in glob.glob("output/*"):
print ""
else:
makedirs("output/gray")
#The output reportories are ready
for file in allFile:
count+=1
print "current file : ",count,"/",numberOfFile
input_filename = file
output_filename = input_filename[0:(len(input_filename)-3)]+"png"
# load file into buffer
data = np.fromfile(inputFolder+"/"+input_filename, dtype=np.uint8)
#To get RGB image
# create yuv image
yuv = np.ndarray((height + height / 2, width), dtype=np.uint8, buffer=data)
# create a height x width x channels matrix with the datatype uint8 for rgb image
img = np.zeros((height, width, channels), dtype=np.uint8);
# convert yuv image to rgb image
cv2.cvtColor(yuv, cv2.COLOR_YUV2BGRA_NV21, img, channels)
cv2.imwrite(outputFolderRGB+"/"+output_filename, img)
#If u saved the image in graysacale use this part instead
#yuvReal = np.ndarray((height, width), dtype=np.uint8, buffer=data)
#cv2.imwrite(outputFolderGray+"/"+output_filename, yuvReal)
else:
print "not any input"
You just have to put your .txt in a folder input
It's a python script but if you prefer a c++ version it's very close.
I've googled for ages and can't find a way to do this. Anybody got an idea? There is an NSColor way of doing it for mac, but nothing for iPhone that I can see. The idea in my app is that the user types in a HEX code (which i have managed to get into RGB) and it's changed into HSB.
Ideas?
It just takes a little bit of math. The following code is just the important parts of the custom class I created for conversion. The "HSBColor" class stores just the hue, saturation, and brightness and I provide functions to get the components of it or a UIColor if I need to actually use it for something in the system.
Note: This code will not work as is unless you define an HSBColor class with hue, brightness, and saturation properties.
+(void)max:(int*)max andMin:(int*)min ofArray:(float[])array
{
*min=0;
*max=0;
for(int i=1; i<3; i++)
{
if(array[i] > array[*max])
*max=i;
if(array[i] < array[*min])
*min=i;
}
}
+(HSBColor*)colorWithRed:(float)red Green:(float)green Blue:(float)blue
{
HSBColor* toReturn = [[[HSBColor alloc] init] autorelease];
float colorArray[3];
colorArray[0] = red;
colorArray[1] = green;
colorArray[2] = blue;
//NSLog(#"RGB: %f %f %f",colorArray[0],colorArray[1],colorArray[2]);
int max;
int min;
[self max:&max andMin:&min ofArray:colorArray];
if(max==min)
{
toReturn.hue=0;
toReturn.saturation=0;
toReturn.brightness=colorArray[0];
}
else
{
toReturn.brightness=colorArray[max];
toReturn.saturation=(colorArray[max]-colorArray[min])/(colorArray[max]);
if(max==0) // Red
toReturn.hue = (colorArray[1]-colorArray[2])/(colorArray[max]-colorArray[min])*60/360;
else if(max==1) // Green
toReturn.hue = (2.0 + (colorArray[2]-colorArray[0])/(colorArray[max]-colorArray[min]))*60/360;
else // Blue
toReturn.hue = (4.0 + (colorArray[0]-colorArray[1])/(colorArray[max]-colorArray[min]))*60/360;
}
return toReturn;
}
+(HSBColor*)colorWithSystemColor:(UIColor*)color
{
const CGFloat* components = CGColorGetComponents(color.CGColor);
return [self colorWithRed:components[0] Green:components[1] Blue:components[2]];
}
In case you need it:
Convert HSV to RGB
(From GLPaint sample application AppController.m)
//FUNCTIONS:
/*
HSL2RGB Converts hue, saturation, luminance values to the equivalent red, green and blue values.
For details on this conversion, see Fundamentals of Interactive Computer Graphics by Foley and van Dam (1982, Addison and Wesley)
You can also find HSL to RGB conversion algorithms by searching the Internet.
See also http://en.wikipedia.org/wiki/HSV_color_space for a theoretical explanation
*/
static void HSL2RGB(float h, float s, float l, float* outR, float* outG, float* outB)
{
float temp1,
temp2;
float temp[3];
int i;
// Check for saturation. If there isn't any just return the luminance value for each, which results in gray.
if(s == 0.0) {
if(outR)
*outR = l;
if(outG)
*outG = l;
if(outB)
*outB = l;
return;
}
// Test for luminance and compute temporary values based on luminance and saturation
if(l < 0.5)
temp2 = l * (1.0 + s);
else
temp2 = l + s - l * s;
temp1 = 2.0 * l - temp2;
// Compute intermediate values based on hue
temp[0] = h + 1.0 / 3.0;
temp[1] = h;
temp[2] = h - 1.0 / 3.0;
for(i = 0; i < 3; ++i) {
// Adjust the range
if(temp[i] < 0.0)
temp[i] += 1.0;
if(temp[i] > 1.0)
temp[i] -= 1.0;
if(6.0 * temp[i] < 1.0)
temp[i] = temp1 + (temp2 - temp1) * 6.0 * temp[i];
else {
if(2.0 * temp[i] < 1.0)
temp[i] = temp2;
else {
if(3.0 * temp[i] < 2.0)
temp[i] = temp1 + (temp2 - temp1) * ((2.0 / 3.0) - temp[i]) * 6.0;
else
temp[i] = temp1;
}
}
}
// Assign temporary values to R, G, B
if(outR)
*outR = temp[0];
if(outG)
*outG = temp[1];
if(outB)
*outB = temp[2];
}