I am trying to make an OBS-like app using Flutter.
I was trying to use Flutter engine to draw widgets onto the video
frames, along with screen, with separated layers.
I came up with a bad way, which:
use RenderRepaintBoundary to get images of specific area.
use ffmpeg to compose these .png series into video with h.264 encoding.
(then maybe use .mp4 files to publish as video stream??)
, which is baaad in real-time performance and efficiency apparently.
(relevant code)
// some_page.dart
int index = 0;
Future<void> onTestPressed() async {
int i = 0;
while (i++ < 600) {
try {
var render = repaintKey.currentContext!.findRenderObject()
as RenderRepaintBoundary;
double dpr = window.devicePixelRatio;
var byteData = await render
.toImage(pixelRatio: dpr)
.then((image) => image.toByteData(format: ImageByteFormat.png));
var tempDir = await getTemporaryDirectory();
var fileName = '${tempDir.path}/frame_${index++}';
var bytes = byteData!.buffer.asUint8List();
var file = File(fileName);
if (!file.existsSync()) {
file.createSync();
}
await file.writeAsBytes(bytes);
// OpenFile.open(fileName);
} catch (e) {
if (kDebugMode) {
print(e);
}
}
}
}
🌟 I know that Flutter uses Skia as its graphic engine, and could I use Skia ability (by drawing widgets) somehow so as to produce video frames more directly?
Thank you.
Related
I am working with Flutter and developing module like offline download media.
I got some solution for Video DRM tools like https://www.vdocipher.com/ but I want to implement it for audio.
The simplest way that I know about to store and hide the media as follow but it will be visible if I turn on show hidden files.
static Future<File> downloadAudioFile(String audioUrl) async {
var hiddenFolder = '.nomedia';
var directoryExternal = await getExternalStorageDirectory();
final dir = Directory('${directoryExternal!.path}/$hiddenFolder');
if ((await dir.exists())) {
} else {
dir.create();
}
final http.Response responseData = await http.get(Uri.parse(audioUrl));
var uint8list = responseData.bodyBytes;
var buffer = uint8list.buffer;
ByteData byteData = ByteData.view(buffer);
String fileName = audioUrl.split('/').last;
File file = await File('${dir.path}/.${fileName}').writeAsBytes(
buffer.asUint8List(byteData.offsetInBytes, byteData.lengthInBytes));
return file;
}
I want to do it for audio & video files, Please let me know if anyone have idea about it.
I want to resize an image and when run my code but resize process is very slow.
My code as below :
void _selectImage() async {
try {
final checkDataImage =
await _imagePicker.pickImage(source: ImageSource.camera);
if (checkDataImage != null) {
print(checkDataImage.name);
print(checkDataImage.path);
setState(() {
pickedImage = checkDataImage;
});
}
} catch (err) {
print(err);
pickedImage = null;
}
final tempDir = await getTemporaryDirectory();
final path = tempDir.path;
Img.Image image = Img.decodeImage(await pickedImage.readAsBytes());
Img.Image smallerImg = Img.copyResize(image, width: 500);
int rand = new Math.Random().nextInt(999999999);
var compressImg = new File('$path/image_$rand.jpg')
..writeAsBytesSync(Img.encodeJpg(smallerImg, quality: 90));
setState(() {
if (!mounted) return;
_imageUpload = compressImg;
});}
What wrong my code, please help to faster it process ?
Thank you for your help.
As others have said, package:image is written in pure Dart and will never be as fast as native solutions that can be multithreaded and hardware accelerated.
While using another isolate will stop the image tasks blocking the UI, if you need to do this as fast as possible, I suggest you use something like the ImageMagick library, or if you only need Android and iOS support, package:flutter_image_compress.
I have this code which loads a ui.Image using an async function:
Future<ui.Image> _loadImage() async {
final imageData = await rootBundle.load('assets/some_image.jpg');
var imageBytes = imageData.buffer.asUint8List();
return await decodeImageFromList(imageBytes);
}
But I would like to base it on AssetImage instead as it loads the correct asset version for the device pixel density. Or make it resolution aware by some other means.
I am currently making a wallpaper app in Flutter. I am using the palette_generator package to extract colors from the image. The problem is while the app is extracting colors from the image. The UI freezes for a solid 3 seconds, after which the colors are shown on the screen. After the colors have been rendered, the whole screen suffers from jank and lag. All animations on that screen perform very badly. Popping off that screen removes the jank and lag. Here is the code I am using:
Future<PaletteGenerator?> updatePalette() async {
var cache = DefaultCacheManager();
File file = await cache.getSingleFile(widget.wall.url);
ui.Image image = await load(file);
palette = await PaletteGenerator.fromImage(
image,
maximumColorCount: 7,
);
return palette;
}
Future<ui.Image> load(File file) async {
var data = await file.readAsBytes();
ui.Codec codec = await ui.instantiateImageCodec(data);
ui.FrameInfo fi = await codec.getNextFrame();
return fi.image;
}
FutureBuilder<PaletteGenerator?>(
future: updatePalette(),
builder: (context, snapshot) {
if (snapshot.hasData)
return MyBottomSheet(
wall: widget.wall, palette: snapshot.data);
return LinearProgressIndicator();
}),
Is there anything I can do to fix this issue?
I recommend using the flutter_isolate package to implement Isolates. The function you are using is a heavy function. Using an Isolated function avoids putting too much pressure on the main thread.
Example code:
void updatePalette() async {
var cache = DefaultCacheManager();
File file = await cache.getSingleFile(url);
var port = ReceivePort();
var isolate = await FlutterIsolate.spawn(
computePalette,
[file.readAsBytesSync(), port.sendPort],
);
port.listen((msg) {
List<int> data = msg;
data.forEach((d) => colorList.add(Color(d)));
isolate.kill();
port.close();
});
}
void computePalette(List<Object> args) async {
var image = args[0] as Uint8List;
var port = args[1] as SendPort;
var img = Image.memory(image);
var palette = await PaletteGenerator.fromImageProvider(
img.image,
maximumColorCount: 7,
);
List<int> colors = [];
palette.colors.forEach((element) => colors.add(element.value));
port.send(colors);
}
I'm using multi_image_picker package to pick images and upload to server, but before uploading I want to resize images. I'm trying to accomplish it using dart.ui but having a problem:
//assets is List<Asset> from MultiImagePicker.pickImages
assets.forEach((asset) {
Future<ByteData> byteData = asset.getByteData();
byteData.then((d) async {
List<int> imageData = d.buffer.asUint8List();
String b64 =base64Encode(imageData);
print(b64); // prints [/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQE...
//if i send b64 to server then decode it and save as img it's working well
//Resize
ui.instantiateImageCodec(imageData,targetHeight: 800, targetWidth: 600)
.then((codec) {
codec.getNextFrame().then((frameInfo) async {
ui.Image i = frameInfo.image;
ByteData bytes = await i.toByteData();
List<int> resizedImageData = bytes.buffer.asUint8List();
String rb64 = base64Encode(resizedImageData);
print(rb64); // prints too many backslashes:[k5KO/5qWk/+ZlZL/mpaT/5uXlP+alpP/mJSR/5iUkf+YlJH/mZSR/5uWk/+blpP/n5qX/6GcmP+gm5f/oZyY/6GcmP+fmpb/nZi..
//If i send rb64 to server then server cannot decode and save it.
});
});
});
});
This is the function I normally use to resize:
import 'dart:ui' as skia;
Future<skia.Image> resizeImage(String path, {int width, int height}) async {
Uint8List data = await File(path).readAsBytes();
final codec = await skia.instantiateImageCodec(data, targetWidth: width, targetHeight: height);
final frame = await codec.getNextFrame();
return frame.image;
}
As I mentioned in the comment, this is currently not working in Flutter Web but it's due to a bug that will be fixed soon, hopefully.