In a work for desktop flutter application (linux), I need to do some intensive computing task into an isolate. But, while this isolate is performing this long operation, it is not reading incoming messages (which seems logic).
I would like to still be able to communicate with it while it is performing.
ReceivePort rPort;
SendPort sPort;
Isolate isolate;
void main() {
rPort = ReceivePort();
isolate = await Isolate.spawn(entryPoint, receivePort.sendPort);
sPort = await rPort.first;
sendMessage("perform"); // this is being processed
sendMessage("controlMessage"); // this is being processed after the perform has ended
}
void sendMessage(String msg) {
ReceivePort localReceivePort = ReceivePort();
sendPort.send([msg, localReceivePort.sendPort]);
}
void entryPoint(SendPort sendPort)
{
ReceivePort receivePort = ReceivePort();
sendPort.send(receivePort.sendPort);
receivePort.listen((message) {
String data = msg[0];
SendPort reply = msg[1];
print(data);
if(data == "perform") performBlockingOperation();
else if(data == "controlMessage") controlPerformance();
repy.send("something");
});
}
I tried many options : making the performBlockingOperation async, but this doesn't seem to work.
Even trying to create a native thread inside the isolate returns
../../third_party/dart/runtime/vm/runtime_entry.cc: 3331: error:
Cannot invoke native callback outside an isolate.
Is there a way to achieve what I am trying ?
Making the computation asynchronous is the right first step.
The second step is to actually yield control at some points during the computation.
Then insert some await Future.delayed(Duration.zero); statements in your computation code that is blocking everything else. Not too deep in the computation, because it does introduce a delay, but often enough that your isolate gets a chance to check for new events occasionally.
Related
I have created an audio worklet that performs pitch detection , all works fine but I want to free the microphone once I am done
I get the stream and wire everything up like this
const AudioContextConstructor =
window.AudioContext || window.webkitAudioContext;
this.audioContext = new AudioContextConstructor();
await this.audioContext.audioWorklet.addModule('js/worklet_pitcher.js');
this.stream = await navigator.mediaDevices.getUserMedia({ audio: true });
var mediaStreamSource = this.audioContext.createMediaStreamSource(this.stream);
this.pitchWorklet = new AudioWorkletNode(this.audioContext, 'pitch-processor');
mediaStreamSource.connect(this.pitchWorklet);
When I am done I simply do this
stop = (): void => {
if (this.running) {
this.audioContext.close();
this.running = false;
}
}
this stops the worklet pipeline but the red dot still shows in the browser tab meaning that I still own the mic.
I looked for a stream.close so I could explicitly close the MediaStream returned by getUserMediabut there isnt one
You also need to call stop() on each MediaStreamTrack of the MediaStream obtained from the mic.
this.stream.getTracks().forEach((track) => track.stop());
https://developer.mozilla.org/en-US/docs/Web/API/MediaStreamTrack/stop
https://developer.mozilla.org/en-US/docs/Web/API/MediaStream/getTracks
I read about dart concurrency. CMIIW, we should use isolate to not block UI render in flutter. But, I found that isolate.spawn is significantly slower.
So, why we should isolate? isn't it better to just use the main than to wait a whole lot longer?
import 'dart:convert';
import 'dart:isolate';
void main() async {
final jsonStr = '{'
'"name": "alif",'
'"age": 26,'
'"gender": "male",'
'"office": "work from home"'
'}';
Future asyncRun<Q>(Function(Q) function, Q message) async {
function(message);
}
final stopwatchA = Stopwatch()..start();
for (var i = 0; i < 100; i++) {
jsonDecode(jsonStr);
}
print('100 sync run : ${stopwatchA.elapsed}');
final stopwatchB = Stopwatch()..start();
await Future.wait(
[for (var i = 0; i < 100; i++) asyncRun(jsonDecode, jsonStr)],
);
print('100 async run : ${stopwatchA.elapsed}');
final stopwatchC = Stopwatch()..start();
await Future.wait(
[for (var i = 0; i < 100; i++) Isolate.spawn(jsonDecode, jsonStr)],
);
print('100 isolate.spawn: ${stopwatchC.elapsed}');
}
after i read comment from jamesdlin about spawn time.
I update the code. I expand the json and use only a single isolate. I found the isolate run much faster. But I don't understand, why is that happened? I expect slightly worse or similar time at best.
import 'dart:convert';
import 'dart:isolate';
void main() async {
final N = 1000;
String content = '';
for (var i = 0; i < N; i++) {
content += '"content$i": $i';
if (i != N - 1) content += ',';
}
final jsonStr = '{$content}';
Future asyncRun<Q>(Function(Q) function, Q message) async {
function(message);
}
final stopwatchA = Stopwatch()..start();
jsonDecode(jsonStr);
print('$N content sync run : ${stopwatchA.elapsed}');
final stopwatchB = Stopwatch()..start();
await asyncRun(jsonDecode, jsonStr);
print('$N content async run : ${stopwatchA.elapsed}');
final stopwatchC = Stopwatch()..start();
await Isolate.spawn(jsonDecode, jsonStr);
print('$N content isolate run : ${stopwatchC.elapsed}');
}
When do somthing which use a lot of cpu time,
if it run on main isolate, the UI will blocked.
isolate.spawn is the best way.
Using isolates, your Dart code can perform multiple independent tasks at once, using additional processor cores if they're available. Isolates are like threads or processes, but each isolate has its own memory and a single thread running an event loop.
The main isolate
You often don’t need to think about isolates at all. A typical Dart app executes all its code in the app’s main isolate,
Even single-isolate programs can execute smoothly by using async-await to wait for asynchronous operations to complete before continuing to the next line of code. A well-behaved app starts quickly, getting to the event loop as soon as possible. The app then responds to each queued event promptly, using asynchronous operations as necessary.
The isolate life cycle
As the following figure shows, every isolate starts by running some Dart code, such as the main() function. This Dart code might register some event listeners—to respond to user input or file I/O, for example. When the isolate’s initial function returns, the isolate stays around if it needs to handle events. After handling the events, the isolate exits.
Event handling
In a client app, the main isolate’s event queue might contain repaint requests and notifications of tap and other UI events. For example, the following figure shows a repaint event, followed by a tap event, followed by two repaint events. The event loop takes events from the queue in first in, first out order.
Official Documentation
With my program, I need to be able to speak into 1 device and my voice play out another device. I was going to originally use TCP to do this, but then it was recommended to me that I should use UDP to reduce latency and make the audio sound better when it comes through. But as the title says, how can I take a dart Stream and send it through UDP? I know there is a function on a TCP Socket object from the dart:io library called addStream() that makes this easy, not 100% sure how it works, but I was wondering if there was something similar to this for UDP? I'm using this library for recording audio, by the way. I'm not really sure what I'm doing and was hoping that someone could explain how to do this in dart, and hopefully this could help a wandering and desperate Googler. This is what I had for my program that used TCP:
void handleClient(Socket client) {
print("Connected a client");
client.addStream(_audioStream);
}
_audioStream = _recorder.audioStream.listen((data) {
if (_isRecording) {
ServerSocket.bind(InternetAddress.anyIPv4, 4444)
.then((ServerSocket server) {
print("Server socket init");
server.listen(handleClient);
});
}
});
Again, this was the version that uses TCP, and of course this didn't work. I'm not really sure what I'm doing, but hopefully that should give you a good idea of what I'm trying to do. How do I send an audio stream over UDP with dart?
what you are asking here is essentially voice communication over the internet
check out
WebRTC and its flutter implementation Here
Ended up figuring it out almost 2+ years later. The latency isn't great and I'm still working on that. But to any lost and wandering Googlers, here you go:
You'll need the mic_stream and flutter_sound packages off pub.dev.
Sender side:
Stream<Uint8List>? stream = await MicStream.microphone(
sampleRate: 32000,
audioFormat: AudioFormat.ENCODING_PCM_8BIT,
);
int? bufferSize = await MicStream.bufferSize;
RawDatagramSocket.bind(InternetAddress.anyIPv4, port).then((socket) {
StreamSubscription<List<int>> listener = stream!.listen((sample) async {
socket.send(sample, InternetAddress(sendToAddress), port);
});
});
and the receiving side:
FlutterSoundPlayer player = FlutterSoundPlayer();
await player.openPlayer(enableVoiceProcessing: false);
await player.startPlayerFromStream(
codec: Codec.pcm16, numChannels: 1, sampleRate: 32000);
var receiver = UDP.bind(
Endpoint.unicast(InternetAddress(thisDeviceIPAddress), port: const Port(37069)));
final udpSocket = await RawDatagramSocket.bind(a5atHome, port);
udpSocket.listen((RawSocketEvent event) {
Uint8List? data = udpSocket.receive()?.data;
if (data != null) {
player.foodSink!.add(FoodData(data)); // this plays the audio
} else {}
});
I find it very difficult to find a simple reliable library to use, in Unity, for a simple websocket client. WebSocketSharp (ie, https://github.com/sta/websocket-sharp ) seems fine but many items are unexplained.
When incoming messages arrive,
wssharp_connection.OnMessage += (sender, e) =>
{
if (!e.IsText) { return; }
Debug.Log(">> OnMessage, " + e.Data);
Handle(e.Data);
// but wait, not the main Unity thread??
};
void Handle(string msg)
{
.. your conventional Unity call, in a monobehavior
}
Testing shows it seems to arrive on (always? sometimes?) not the main Unity thread.
Does anyone know
In fact is it always not the main thread, or can it vary?
If always another thread, in fact is it always the same one thread, or does it vary/many?
Since every single Unity app that uses a websocket client has to do this, I'm surprised that there are not 100s of discussion about how to handle incoming messages, including on the WebSocketSharp page,
but here's what I would do, just in the usual way you handle "threading" in a frame-based engine like Unity:
wssharp_connection.OnMessage += (sender, e) =>
{
if (!e.IsText) { return; }
incoming_messages.Enqueue(e.Data);
};
// and ...
ConcurrentQueue<string> incoming_messages = new ConcurrentQueue<string>();
void Update()
{
if (incoming_messages.TryDequeue(out var message))
{
Handle(message);
}
}
So then
Is that indeed the usual way to handle the threading problem in WebSocketSharp, when using inside Unity?
Given how many billion Unity apps there are with websockets clients, this must be a pretty well-trodden path.
I know Future will run in event queue.But event queue are also running on main isolate, if i do some heavy task (for example, calculate sum from 1 to 1000000) in future, it will block my ui code.
But Future in network operation will not block ui (such as await httpClient.getUrl(uri)).
Why does a network request using future take several seconds without blocking the UI, while computational operations block the UI？
#override
void initState() {
super.initState();
Future((){
var result;
for (var i = 0; i < 1000000; ++i) {
result = 'result is $i';
}
print(result);
});
}
if i do some heavy task using Future in initState(), the ui will be blocked.
Isolates in Dart are single-threaded. An isolate can do only one thing at a time.
Asynchronous functions are basically a form of cooperative multitasking. A function must yield (usually via await) to allow other operations to execute in the isolate.
Your computation doesn't yield, so it must run in its entirety before the UI can resume processing events, and the UI will be unresponsive. If you altered it:
Future(() async {
var result;
for (var i = 0; i < 1000000; ++i) {
result = 'result is $i';
await Future.delayed(Duration.zero);
}
print(result);
});
then you should find that the UI can process events regularly and should have the appearance of remaining responsive. (Note that your computation will take much longer to complete because of the additional extra overhead.)
Let me answer briefly, the network request (HttpClient in dart:io) actually ended up in another isolate.
find _NativeSocket section inside socket_patch.dart file, keep searching down and you will see this statement (the link is likely to point to the wrong line as the SDK is constantly updated in the future):
_EventHandler._sendData(this, eventPort!.sendPort, fullData);
Does it look familiar?