Convert YUV420 to RGB flutter - flutter

Question

const shift = (0xFF << 24);
Future<Image> convertYUV420toImageColor(CameraImage image) async {
try {
final int width = image.width;
final int height = image.height;
final int uvRowStride = image.planes[1].bytesPerRow;
final int uvPixelStride = image.planes[1].bytesPerPixel;
print("uvRowStride: " + uvRowStride.toString());
print("uvPixelStride: " + uvPixelStride.toString());
// imgLib -> Image package from https://pub.dartlang.org/packages/image
var img = imglib.Image(width, height); // Create Image buffer
// Fill image buffer with plane[0] from YUV420_888
for(int x=0; x < width; x++) {
for(int y=0; y < height; y++) {
final int uvIndex = uvPixelStride * (x/2).floor() + uvRowStride*(y/2).floor();
final int index = y * width + x;
final yp = image.planes[0].bytes[index];
final up = image.planes[1].bytes[uvIndex];
final vp = image.planes[2].bytes[uvIndex];
// Calculate pixel color
int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
int g = (yp - up * 46549 / 131072 + 44 -vp * 93604 / 131072 + 91).round().clamp(0, 255);
int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);
// color: 0x FF FF FF FF
// A B G R
img.data[index] = shift | (b << 16) | (g << 8) | r;
}
}
imglib.PngEncoder pngEncoder = new imglib.PngEncoder(level: 0, filter: 0);
List<int> png = pngEncoder.encodeImage(img);
muteYUVProcessing = false;
return Image.memory(png);
} catch (e) {
print(">>>>>>>>>>>> ERROR:" + e.toString());
}
return null;
}
I have been following this code snippet from How to convert Camera Image to Image in Flutter? to convert YUV to RGB to send the images via WebSockets for ML prediction.
Although it works to convert, the resulting image is rotated 90 degrees and the performance is a little bit slow. How I can rotate it?

Answer 1

replace img.data[index] = shift | (b << 16) | (g << 8) | r;
with
if (img.boundsSafe(height-y, x)){
img.setPixelRgba(height-y, x, r , g ,b ,shift);
}
and replace var img = imglib.Image(width, height);
with
var img = imglib.Image(height, width);

Answer 2

For IOS version, the CameraImage is returned as biplanar which has only two planes.
Quote from image_format_group.dart:
/// Multi-plane YUV 420 format.
/// This format is a generic YCbCr format, capable of describing any 4:2:0
/// chroma-subsampled planar or semiplanar buffer (but not fully interleaved),
/// with 8 bits per color sample.
/// On Android, this is `android.graphics.ImageFormat.YUV_420_888`. See
/// https://developer.android.com/reference/android/graphics/ImageFormat.html#YUV_420_888
/// On iOS, this is `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange`. See
/// https://developer.apple.com/documentation/corevideo/1563591-pixel_format_identifiers/kcvpixelformattype_420ypcbcr8biplanarvideorange?language=objc
yuv420
For my approach, I use dart:ffi and link to c++ for Android CameraImage following the tutorial from here. The conversion between YUV420p and YUV420sp can be found here. There is no complete code for the conversion yet, but neither any solution for IOS around the forum.