pass matlab image to open3d three::Image in a mex script - matlab

I am trying to load an image in a mex script and cast it to the corresponding format that the Open3D library uses, i.e. three::Image. I am using the following code:
uint8_t* rgb_image = (uint8_t*) mxGetPr(prhs[3]);
int* dims = (int*) mxGetDimensions(prhs[3]);
int height = dims[0];
int width = dims[2];
int channels = dims[4];
int imsize = height * width;
Image image;
image.PrepareImage(height, width, 3, sizeof(uint8_t)); // parameters: height, width, num_of_channels, bytes_per_channel
memcpy(image.data_.data(), rgb_image, image.data_.size());
The above works well when I give a grayscale image and specify num_of_channels to 1 but not for 3 channel images as you can notice below:
Then I tried to create a function where I am manually looping through the raw data and assigning them to the output image
auto image_ptr = std::make_shared<Image>();
image_ptr->PrepareImage(height, width, channels, sizeof(uint8_t));
for (int i = 0; i < height * width; i++) {
uint8_t *p = (uint8_t *)(image_ptr->data_.data() + i * channels * sizeof(uint8_t));
*p++ = *rgb_image++;
}
But now it seems that the color channels are wrongly assigned:
Any idea how to address this issue. The point is that it seems to be something easy but since my knowledge with C++ and pointers is quite limited I cannot figure it out straight forward.
I found this solution here (Reading image in matlab in a format acceptable to mex) as well but I am not sure how exactly I can use it. To be honest I am quite of confused.

ok the solution was quite straight forward as I was though in first place. It was just playing correctly with the pointers:
std::shared_ptr<Image> CreateRGBImageFromMat(uint8_t *mat_image, int width, int height, int channels)
{
auto open3d_image = std::make_shared<Image>();
open3d_image->PrepareImage(height, width, channels, sizeof(uint8_t));
for (int i = 0; i < height * width; i++) {
uint8_t *p = (uint8_t *)(open3d_image->data_.data() + i * channels * sizeof(uint8_t));
*p++ = *(mat_image + i);
*p++ = *(mat_image + i + height*width);
*p++ = *(mat_image + i + height*width*2);
}
return open3d_image;
}
since the three::Image expects the data in contiguous order row x col x channel while from matlab the image comes in blocks rows x cols x channel_1 (after you transpose the image since matlab is column major). My question though now is whether I can do the same with memcpy() or std::copy() where I can copy the bloc data to contiguous form so that I bypass the for loop.

Related

libyuv::NV21ToI420 messes up colors

I am using libyuv to convert NV21 image format to I420:
void convert(uint8* input, int width, int height) {
int size = width * height * 3/2;
uint8* output = new uint8[size];
uint8* src_y = input;
int src_stride_y = width;
uint8* src_vu = input + (width * height);
int src_stride_vu = width / 2;
uint8* dst_y = output;
int dst_stride_y = width;
uint8* dst_u = dst_y + (width * height);
int dst_stride_u = width / 2;
uint8* dst_v = dst_u + (width * height) / 4;
int dst_stride_v = width / 2;
libyuv::NV21ToI420(src_y, src_stride_y,
src_vu, src_stride_vu,
dst_y, dst_stride_y,
dst_u, dst_stride_u,
dst_v, dst_stride_v,
width, height);
dumpToFile(dst_y, size);
...
}
The size of my input is 640x480.
I display the dumped file using ImageMagick's display:
$ display -size 640x480 -depth 8 -sampling-factor 4:2:0 -colorspace srgb MyI420_1.yuv
However, the colors are messed up in the displayed image. The other aspects of the image look okay.
I am wondering if I am making a mistake in my code. Perhaps my stride calculations are not correct.
Note that if I use my custom function to rearrange V1U1V2U2... as U1U2...V1V2... and dump the output, it displays fine. However, I prefer to use libyuv as it has some optimization for neon, SSE2, etc. Regards.
Your src_stride_vu need to be same as width since it's combined stride to UV pixels interlaced.

How does the reversebits function of HLSL SM5 work?

I am trying to implement an inverse FFT in a HLSL compute shader and don't understand how the new inversebits function works. The shader is run under Unity3D, but that shouldn't make a difference.
The problem is, that the resulting texture remains black with the exception of the leftmost one or two pixels in every row. It seems to me, as if the reversebits function wouldn't return the correct indexes.
My very simple code is as following:
#pragma kernel BitReverseHorizontal
Texture2D<float4> inTex;
RWTexture2D<float4> outTex;
uint2 getTextureThreadPosition(uint3 groupID, uint3 threadID) {
uint2 pos;
pos.x = (groupID.x * 16) + threadID.x;
pos.y = (groupID.y * 16) + threadID.y;
return pos;
}
[numthreads(16,16,1)]
void BitReverseHorizontal (uint3 threadID : SV_GroupThreadID, uint3 groupID : SV_GroupID)
{
uint2 pos = getTextureThreadPosition(groupID, threadID);
uint xPos = reversebits(pos.x);
uint2 revPos = uint2(xPos, pos.y);
float4 values;
values.x = inTex[pos].x;
values.y = inTex[pos].y;
values.z = inTex[revPos].z;
values.w = 0.0f;
outTex[revPos] = values;
}
I played around with this for quite a while and found out, that if I replace the reversebits line with this one here:
uint xPos = reversebits(pos.x << 23);
it works. Although I have no idea why. Could be just coincidence. Could someone please explain to me, how I have to use the reversebits function correctly?
Are you sure you want to reverse the bits?
x = 0: reversed: x = 0
x = 1: reversed: x = 2,147,483,648
x = 2: reversed: x = 1,073,741,824
etc....
If you fetch texels from a texture using coordinates exceeding the width of the texture then you're going to get black. Unless the texture is > 1 billion texels wide (it isn't) then you're fetching well outside the border.
I am doing the same and came to the same problem and these answers actually answered it for me but i'll give you the explanation and a whole solution.
So the solution with variable length buffers in HLSL is:
uint reversedIndx;
uint bits = 32 - log2(xLen); // sizeof(uint) - log2(numberOfIndices);
for (uint j = 0; j < xLen; j ++)
reversedIndx = reversebits(j << bits);
And what you found/noticed essentially pushes out all the leading 0 of your index so you are just reversing the least significant or rightmost bits up until the max bits we want.
for example:
int length = 8;
int bits = 32 - 3; // because 1 << 3 is 0b1000 and we want the inverse like a mask
int j = 6;
and since the size of an int is generally 32bits in binary j would be
j = 0b00000000000000000000000000000110;
and reversed it would be (AKA reversebits(j);)
j = 0b01100000000000000000000000000000;
Which was our error, so j bit shifted by bits would be
j = 0b11000000000000000000000000000000;
and then reversed and what we want would be
j = 0b00000000000000000000000000000011;

iPhone SDK - Optimize a for loop

I'm developing an image processing application and I'm looking for an advise to tune my code.
My need is to split the image into blocs (80x80), and for each blocs, calculate the average color.
My first method contains the main loops where the second method is called :
- (NSArray*)getRGBAsFromImage:(UIImage *)image {
int width = image.size.width;
int height = image.size.height;
int blocPerRow = 80;
int blocPerCol = 80;
int pixelPerRowBloc = width / blocPerRow;
int pixelPerColBloc = height / blocPerCol;
int xx,yy;
// Row loop
for (int i=0; i<blocPerRow; i++) {
xx = (i * pixelPerRowBloc) + 1;
// Colon loop
for (int j=0; j<blocPerCol; j++) {
yy = (j * pixelPerColBloc) +1;
[self getRGBAsFromImageBloc:image
atX:xx
andY:yy
withPixelPerRow:pixelPerRowBloc
AndPixelPerCol:pixelPerColBloc];
}
}
// return my NSArray not done yet !
}
My second method browses the pixel bloc and returns a ColorStruct :
- (ColorStruct*)getRGBAsFromImageBloc:(UIImage*)image
atX:(int)xx
andY:(int)yy
withPixelPerRow:(int)pixelPerRow
AndPixelPerCol:(int)pixelPerCol {
// First get the image into your data buffer
CGImageRef imageRef = [image CGImage];
NSUInteger width = CGImageGetWidth(imageRef);
NSUInteger height = CGImageGetHeight(imageRef);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
unsigned char *rawData = malloc(height * width * 4);
NSUInteger bytesPerPixel = 4;
NSUInteger bytesPerRow = bytesPerPixel * width;
NSUInteger bitsPerComponent = 8;
CGContextRef context = CGBitmapContextCreate(rawData, width, height,
bitsPerComponent, bytesPerRow, colorSpace,
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGColorSpaceRelease(colorSpace);
CGContextDrawImage(context, CGRectMake(0, 0, width, height), imageRef);
CGContextRelease(context);
// Now your rawData contains the image data in the RGBA8888 pixel format.
int byteIndex = (bytesPerRow * yy) + xx * bytesPerPixel;
int red = 0;
int green = 0;
int blue = 0;
int alpha = 0;
int currentAlpha;
// bloc loop
for (int i = 0 ; i < (pixelPerRow*pixelPerCol) ; ++i) {
currentAlpha = rawData[byteIndex + 3];
red += (rawData[byteIndex] ) * currentAlpha;
green += (rawData[byteIndex + 1]) * currentAlpha;
blue += (rawData[byteIndex + 2]) * currentAlpha;
alpha += currentAlpha;
byteIndex += 4;
if ( i == pixelPerRow ) {
byteIndex += (width-pixelPerRow) * 4;
}
}
red /= alpha;
green /= alpha;
blue /= alpha;
ColorStruct *bColorStruct = newColorStruct(red, blue, green);
free(rawData);
return bColorStruct;
}
ColorStruct :
typedef struct {
int red;
int blue;
int green;
} ColorStruct;
with constructor :
ColorStruct *newColorStruct(int red, int blue, int green) {
ColorStruct *ret = malloc(sizeof(ColorStruct));
ret->red = red;
ret->blue = blue;
ret->green = green;
return ret;
}
As you can see, I have three level of loop : the row loop, the colon loop, and the bloc loop.
I have tested my code and it takes about 5 to 6 seconds for an 320x480 pictures.
Any help is welcomed.
Thanks,
Bahaaldine
Seem like a perfect problem to give it the Grand Central Dispatch ?
I think the main problem in this code is there are too many image reads. The entire image is loaded to memory for every(!) block (malloc is expensive too). You should preload image data once (cache it) and then use that memory in getRGBAsFromImageBloc(). Now for 320x480 picture you have 4 x 6 = 24 blocks. So you can speed up you app manyfold by only using caching.
At the end of the day taking an image and performing three multiplies and five additions on each pixel sequentially is always going to be relatively slow.
Luckily, what you're doing can be thought of as a special case of interpolating an image from one size to another - i.e. the average pixel of an image is the same as that image resized to a size of 1x1 (assuming the resizing is using some form of linear interpolation, but that's usually the standard way to do it) and there's a few highly optimized (or at least more optimized than you're likely to get without enormous effort) options for doing that that are part of the iPhone's graphics libraries. At first I'd try using the Quartz methods to resize an image:
CGImageRef sourceImage = yourImage;
int numBytesPerPixel = 4;
u_char* scaledImageData = (u_char*)malloc(numBytesPerPixel);
CGColorSpaceRef colorspace = CGImageGetColorSpace(sourceImage);
CGContextRef context = CGBitmapContextCreate (scaledImageData, 1, 1, 8, numBytesPerPixel, colorspace, kCGImageAlphaNoneSkipFirst);
CGColorSpaceRelease(colorspace);
CGContextDrawImage(context, CGRectMake(0,0,1,1), sourceImage);
int a = scaledImageData[0];
int r = scaledImageData[1];
int g = scaledImageData[2];
int b = scaledImageData[3];
(this just scales the original image down to 1 pixel and doesn't show the cropping of the sub regions but unfortunately I don't have time for that code right now - if you try to implement it and get stuck add a comment and I can show you how you would do that).
If that doesn't work you could always try using OpenGL ES to do this (create a texture out of the part of your image you need to scale, render it to a 1x1 buffer, and test the result from the buffer). This is a lot more complicated but might have some advantages in that it gives you access to the GPU, which might be a lot faster for large images.
Hope that makes sense and helps...
P.S. - Definitely follow y0prst's suggestion and only read the image in once - that is an easy fix that is going to buy you a ton of performance.
P.P.S - I haven't tested the code so usual caveats apply.
You're inspecting every single pixel - something that, it would seem, is going to take roughly the same amount of time no matter how you loop through it (provided you only inspect each pixel once).
I would suggest using a random sampling within the bloc - every "n'th" pixel, which would reduce the loop time (and the accuracy), or allow for an adjustable granularity.
Now, if there is an existing algorithm for computing the average of a group of pixels - that would be something to consider as an alternative.
You can speed things up by not calling a method in the middle of your loop. Just include the code inline.
ADDED: Also, you might try doing the draw image only once, not repeated in a loop, if you have enough memory.
After you do that, you can try hoisting some of the multiplies out of the inner loop as well for a little additional performance (although the Compiler may optimize some of this for you).

Looking for some help working with premultiplied alpha

I am trying to update a source image with the contents of multiple destination images. From what I can tell using premultiplied alpha is the way to go with this, but I think I am doing something wrong (function below). the image I am starting with is initialized with all ARGB values set to 0. When I run the function once the resulting image looks great, but when I start compositing on any others all the pixels that have alpha information get really messed up. Does anyone know if I am doing something glaringly wrong or if there is something extra I need to do to modify the color values?
void CompositeImage(unsigned char *src, unsigned char *dest, int srcW, int srcH){
int w = srcW;
int h = srcH;
int px0;
int px1;
int px2;
int px3;
int inverseAlpha;
int r;
int g;
int b;
int a;
int y;
int x;
for (y = 0; y < h; y++) {
for (x= 0; x< w*4; x+=4) {
// pixel number
px0 = (y*w*4) + x;
px1 = (y*w*4) + (x+1);
px2 = (y*w*4) + (x+2);
px3 = (y*w*4) + (x+3);
inverseAlpha = 1 - src[px3];
// create new values
r = src[px0] + inverseAlpha * dest[px0];
g = src[px1] + inverseAlpha * dest[px1];
b = src[px2] + inverseAlpha * dest[px2];
a = src[px3] + inverseAlpha * dest[px3];
// update destination image
dest[px0] = r;
dest[px1] = g;
dest[px2] = b;
dest[px3] = a;
}
}
}
I'm not clear on what data you are working with. Do your source images already have the alpha values pre-multiplied as they are stored? If not, then pre-multiplied alpha does not apply here and you would need to do normal alpha blending.
Anyway, the big problem in your code is that you're not keeping track of the value ranges that you're dealing with.
inverseAlpha = 1 - src[px3];
This needs to be changed to:
inverseAlpha = 255 - src[px3];
You have all integral value types here, so the normal incoming 0..255 value range will result in an inverseAlpha range of -254..1, which will give you some truly wacky results.
After changing the 1 to 255, you also need to divide your results for each channel by 255 to scale them back down to the appropriate range. The alternative is to do the intermediate calculations using floats instead of integers and divide the initial channel values by 255.0 (instead of these other changes) to get values in the 0..1 range.
If your source data really does already have pre-multiplied alpha, then your result lines should look like this.
r = src[px0] + inverseAlpha * dest[px0] / 255;
If your source data does not have pre-multiplied alpha, then it should be:
r = src[px0] * src[px3] / 255 + inverseAlpha * dest[px0] / 255;
There's nothing special about blending the alpha channel. Use the same calculation as for r, g, and b.

How to save an image as Tiff or PNG with an alpha channel or alpha mask in iPhone SDK?

I have an image with something inside in a white backround. I want to save that image in a format that allows alpha channel or using an alpha mask in a way that the white pixels became transparents. Any light out there?
I don't know of any libraries where this is super easy. But, there's a lot of relevant sample code in the GLImageProcessing example here. (I haven't run the following)
UIImage *some_image = [UIImage imageNamed:#"somethin'.tiff"];
CGImageRef cg_image = some_image.CGImage;
CFDataRef data = CGDataProviderCopyData(CGImageGetDataProvider(cg_image));
size_t bpp = CGImageGetBitsPerPixel(CGImage);
uint32_t *stuff = (uint32_t *)CFDataGetBytePtr(data);
int w = CGImageGetWidth(CGImage);
int h = CGImageGetHeight(CGImage);
int N = w * h;
for (int i = 0; i < N; i++ ) {
// do your stuff, test for white, set the alpha mask
stuff[i] = stuff[i] & ((uint32_t)0xFFFFFFFF | alpha_mask);
}
You could instead use this function
UIKIT_EXTERN NSData *UIImagePNGRepresentation(UIImage *image);
and write the data to disk. I hope this helps. Post the solution if you find it...