How to optimize flutter CameraImage to TensorImage? - flutter

That function is too slow. So Flutter CameraImage efficiency convert to TensorImage in dart?
var img = imglib.Image(image.width, image.height); // Create Image buffer
Plane plane = image.planes[0];
const int shift = (0xFF << 24);
// Fill image buffer with plane[0] from YUV420_888
for (int x = 0; x < image.width; x++) {
for (int planeOffset = 0;
planeOffset < image.height * image.width;
planeOffset += image.width) {
final pixelColor = plane.bytes[planeOffset + x];
// color: 0x FF FF FF FF
// A B G R
// Calculate pixel color
var newVal =
shift | (pixelColor << 16) | (pixelColor << 8) | pixelColor;
img.data[planeOffset + x] = newVal;
}
}
return img;
}```

Seems your for loop is inefficient. The data for whole row (with same placeOffset, different x) will be cached at once, so would be faster to switch ordering of the two loops.
for (int y = 0; y < image.height; y++) {
for (int x = 0; x < image.width; x++) {
final pixelColor = plane.bytes[y * image.width + x];
// ...
}
}
However, your code does not seems to be reading from the actual camera stream. please refer this thread for converting CameraImage to Image.
How to convert Camera Image to Image in Flutter?

Related

How to convert CameraImage first to jpeg and after to base64 Flutter

Is it possible to convert CameraImage (yuv420 or bgra) to jpeg and after that encode it in base64?
List<String> convertFaceImages(Map<String, CameraImage> images) {
final sortedImages = sortImages(images);
List<String> base64Images = [];
List<img.Image> listImage = [];
for (CameraImage image in sortedImages) {
var yuv420Image =
img.Image(image.width, image.height); // Create Image buffer
Plane plane = image.planes[0];
const int shift = (0xFF << 24);
// Fill image buffer with plane[0] from YUV420_888
for (int x = 0; x < image.width; x++) {
for (int planeOffset = 0;
planeOffset < image.height * image.width;
planeOffset += image.width) {
final pixelColor = plane.bytes[planeOffset + x];
// color: 0x FF FF FF FF
// A B G R
// Calculate pixel color
var newVal =
shift | (pixelColor << 16) | (pixelColor << 8) | pixelColor;
yuv420Image.data[planeOffset + x] = newVal;
}
}
listImage.add(yuv420Image);
}
for (img.Image image in listImage) {
final encodedImage = base64Encode(img.encodeJpg(image));
logger.w(encodedImage);
base64Images.add(encodedImage);
}
return base64Images;
}
I tried do it with this code, but when I try to decode this image on backend (or https://base64.guru/converter/decode/image) I got just a black image. More of than, when I was triyng to display this images in my app, images was black-white an rotated on 90 degrees.
Can someone help me please? I'm actually trying to get this right for a very long time.

Convert YUV420 to RGB flutter

const shift = (0xFF << 24);
Future<Image> convertYUV420toImageColor(CameraImage image) async {
try {
final int width = image.width;
final int height = image.height;
final int uvRowStride = image.planes[1].bytesPerRow;
final int uvPixelStride = image.planes[1].bytesPerPixel;
print("uvRowStride: " + uvRowStride.toString());
print("uvPixelStride: " + uvPixelStride.toString());
// imgLib -> Image package from https://pub.dartlang.org/packages/image
var img = imglib.Image(width, height); // Create Image buffer
// Fill image buffer with plane[0] from YUV420_888
for(int x=0; x < width; x++) {
for(int y=0; y < height; y++) {
final int uvIndex = uvPixelStride * (x/2).floor() + uvRowStride*(y/2).floor();
final int index = y * width + x;
final yp = image.planes[0].bytes[index];
final up = image.planes[1].bytes[uvIndex];
final vp = image.planes[2].bytes[uvIndex];
// Calculate pixel color
int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
int g = (yp - up * 46549 / 131072 + 44 -vp * 93604 / 131072 + 91).round().clamp(0, 255);
int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);
// color: 0x FF FF FF FF
// A B G R
img.data[index] = shift | (b << 16) | (g << 8) | r;
}
}
imglib.PngEncoder pngEncoder = new imglib.PngEncoder(level: 0, filter: 0);
List<int> png = pngEncoder.encodeImage(img);
muteYUVProcessing = false;
return Image.memory(png);
} catch (e) {
print(">>>>>>>>>>>> ERROR:" + e.toString());
}
return null;
}
I have been following this code snippet from How to convert Camera Image to Image in Flutter? to convert YUV to RGB to send the images via WebSockets for ML prediction.
Although it works to convert, the resulting image is rotated 90 degrees and the performance is a little bit slow. How I can rotate it?
replace img.data[index] = shift | (b << 16) | (g << 8) | r;
with
if (img.boundsSafe(height-y, x)){
img.setPixelRgba(height-y, x, r , g ,b ,shift);
}
and replace var img = imglib.Image(width, height);
with
var img = imglib.Image(height, width);
For IOS version, the CameraImage is returned as biplanar which has only two planes.
Quote from image_format_group.dart:
/// Multi-plane YUV 420 format.
/// This format is a generic YCbCr format, capable of describing any 4:2:0
/// chroma-subsampled planar or semiplanar buffer (but not fully interleaved),
/// with 8 bits per color sample.
/// On Android, this is `android.graphics.ImageFormat.YUV_420_888`. See
/// https://developer.android.com/reference/android/graphics/ImageFormat.html#YUV_420_888
/// On iOS, this is `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange`. See
/// https://developer.apple.com/documentation/corevideo/1563591-pixel_format_identifiers/kcvpixelformattype_420ypcbcr8biplanarvideorange?language=objc
yuv420
For my approach, I use dart:ffi and link to c++ for Android CameraImage following the tutorial from here. The conversion between YUV420p and YUV420sp can be found here. There is no complete code for the conversion yet, but neither any solution for IOS around the forum.

Convert Dicom image pixel value to Unity color

I use SimpleITK to read a Dicom image in Unity. I create a list of Unity Texture2D to store Dicom slices. My question is how can we convert the pixel value from Dicom image to Unity Color?
List<Texture2D> _imageListTexture = new List<Texture2D>();
for (int k = 0; k < depth; k++)
{
Texture2D _myTex = new Texture2D(width, height, TextureFormat.ARGB32, true);
for (int j = 0; j < height; j++)
{
for (int i = 0; i < width; i++)
{
int idx = i + width * (j + height * k);
float pixel = liverVoxels[idx]; //We have the pixel value
Color32 _col = new Color32(); //Unity has rgba but we have only one value
//What to do here?
_myTex.SetPixel(i, j, _col);
}
}
_myTex.Apply();
_imageListTexture.Add(_myTex);
}
It's very simple like this:
float pixel = liverVoxels[idx]; //We have the pixel value
Color32 _col = new Color32(pixel, pixel, pixel,255);
_myTex.SetPixel(i, j, _col);

Filled rect disappear after mouse releasing

I’m new to processing. I am looking to write a program in which I select an area of an image ( like “rect selection” in Photoshop for example…), this area has a red stroke and is slightly opaque, once the area is selected, the rectangle fills with the average color of the pixels of the same area and the red stroke turns off. The idea is to be able to repeat this action several times on the same image. When I released the mouse, the fill is erased because of background(buff); in void draw();. I would like the rectangle filled with the new color to be saved. I think I need to use an array, class, but I don’t understand how these work. If someone is able to help me, it would be a great help. Thank you.
PImage buff1;
int x1,y1,x2,y2,h1,h2;
void setup()
{
size(1000, 721);
buff1 = loadImage("buff1.jpg2);
background(buff1);
}
color extractColorFromImage(final PImage buff1) {
buff1.loadPixels();
color r = 1, g = 1, b = 1;
for (final color c : buff1.pixels) {
r += c >> 020 & 255;
g += c >> 010 & 255;
b += c & 255;
}
r /= buff1.pixels.length;
g /= buff1.pixels.length;
b /= buff1.pixels.length;
return color(r, g, b);
}
void draw()
{
background(buff1);
rectMode(CORNERS);
stroke(255,0,0);
strokeWeight(2);
strokeJoin(ROUND);
rect(x1,y1,x2,y2,2);
fill(255,0,0,50);
noStroke();
cursor(ARROW);
}
void mousePressed()
{
x1 = mouseX;
y1 = mouseY;
}
void mouseDragged()
{
x2 = mouseX;
y2 = mouseY;
}
void mouseReleased()
{
int H1 = abs(1+x2-x1);
int H2 = abs(1+y2-y1);
for (int i=0; i<width; i+=H1)
{
for (int j=0; j<height; j+=H2)
{
PImage newImg = buff1.get(x1, y1, H1, H2);
fill(extractColorFromImage(newImg), 40);
noStroke();
cursor(ARROW);
}
}
}
Once the pixel data for an image have been loaded by loadPixels(), the loaded pixels can be accessed and changed by pixels[].
updatePixels() updates the image with the data in its pixels[] array:
void mouseReleased()
{
int x_1 = min(x1,x2);
int y_1 = min(y1,y2);
int x_2 = max(x1,x2);
int y_2 = max(y1,y2);
PImage newImg = buff1.get(x_1, y_1, x_2-x1+1, y_2-y1+1);
color new_color = extractColorFromImage(newImg);
buff1.loadPixels();
for (int i = x_1; i <= x_2; i++)
{
for (int j = y_1; j <= y_2; j++)
{
buff1.pixels[j*buff1.width+i] = new_color;
}
}
buff1.updatePixels();
}
When the mouse button is pressed, then I recommend to set (x2, y2), too:
void mousePressed()
{
x1 = mouseX;
y1 = mouseY;
x2 = x1;
y2 = y1;
}
Optionally the original color can be mixed with the new color by lerpColor():
PImage newImg = buff1.get(x_1, y_1, x_2-x1+1, y_2-y1+1);
color new_color = extractColorFromImage(newImg);
buff1.loadPixels();
for (int i = x_1; i <= x_2; i++)
{
for (int j = y_1; j <= y_2; j++)
{
color orig_color = buff1.pixels[j*buff1.width+i];
buff1.pixels[j*buff1.width+i] = lerpColor(orig_color, new_color, 0.5);
}
}
buff1.updatePixels();

How to convert camera image to image?

I want to convert camera image from function startImageStream of camera plugin in Flutter to Image to crop that image but I only find the way to convert to FirebaseVisionImage.
Edit For Color Image
if I understand you clear. You are trying to covnert YUV420 format. The following is code snippet from: https://github.com/flutter/flutter/issues/26348
const shift = (0xFF << 24);
Future<Image> convertYUV420toImageColor(CameraImage image) async {
try {
final int width = image.width;
final int height = image.height;
final int uvRowStride = image.planes[1].bytesPerRow;
final int uvPixelStride = image.planes[1].bytesPerPixel;
print("uvRowStride: " + uvRowStride.toString());
print("uvPixelStride: " + uvPixelStride.toString());
// imgLib -> Image package from https://pub.dartlang.org/packages/image
var img = imglib.Image(width, height); // Create Image buffer
// Fill image buffer with plane[0] from YUV420_888
for (int x = 0; x < width; x++) {
for (int y = 0; y < height; y++) {
final int uvIndex = uvPixelStride * (x / 2).floor() + uvRowStride * (y / 2).floor();
final int index = y * width + x;
final yp = image.planes[0].bytes[index];
final up = image.planes[1].bytes[uvIndex];
final vp = image.planes[2].bytes[uvIndex];
// Calculate pixel color
int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
int g = (yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91).round().clamp(0, 255);
int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);
// color: 0x FF FF FF FF
// A B G R
img.data[index] = shift | (b << 16) | (g << 8) | r;
}
}
imglib.PngEncoder pngEncoder = new imglib.PngEncoder(level: 0, filter: 0);
List<int> png = pngEncoder.encodeImage(img);
muteYUVProcessing = false;
return Image.memory(png);
} catch (e) {
print(">>>>>>>>>>>> ERROR:" + e.toString());
}
return null;
}
I found, sometimes planes[0] bytes per row is not same as width. In that case, you should do something like the code below.
static image_lib.Image convertYUV420ToImage(CameraImage cameraImage) {
final width = cameraImage.width;
final height = cameraImage.height;
final yRowStride = cameraImage.planes[0].bytesPerRow;
final uvRowStride = cameraImage.planes[1].bytesPerRow;
final uvPixelStride = cameraImage.planes[1].bytesPerPixel!;
final image = image_lib.Image(width, height);
for (var w = 0; w < width; w++) {
for (var h = 0; h < height; h++) {
final uvIndex =
uvPixelStride * (w / 2).floor() + uvRowStride * (h / 2).floor();
final index = h * width + w;
final yIndex = h * yRowStride + w;
final y = cameraImage.planes[0].bytes[yIndex];
final u = cameraImage.planes[1].bytes[uvIndex];
final v = cameraImage.planes[2].bytes[uvIndex];
image.data[index] = yuv2rgb(y, u, v);
}
}
return image;
}
static int yuv2rgb(int y, int u, int v) {
// Convert yuv pixel to rgb
var r = (y + v * 1436 / 1024 - 179).round();
var g = (y - u * 46549 / 131072 + 44 - v * 93604 / 131072 + 91).round();
var b = (y + u * 1814 / 1024 - 227).round();
// Clipping RGB values to be inside boundaries [ 0 , 255 ]
r = r.clamp(0, 255);
g = g.clamp(0, 255);
b = b.clamp(0, 255);
return 0xff000000 |
((b << 16) & 0xff0000) |
((g << 8) & 0xff00) |
(r & 0xff);
}