How to send big text data from tcp socket to Flutter app? - sockets

I have a 3000 rows and 20 column data most cell has 50 or 80 character long text. I test using aqueduct and Dart Server to see which one is better intermediate option in between server data and Flutter communication.
I use 2 option for the test. All I can test is that for (var data in _socket) runs about 10 or 12 times to receive whole data. On the other hand _socket.liste lost much of the data.
In both I even use await Future.delayed(Duration(seconds: 7))...
And result was same. In short data they're both very fast. but for big data I am having problem to get correct result.
How to send big text data from tcp socket to Flutter app?
// OPTION 1
_socket.listen((List<int> data) {
String _reply = new String.fromCharCodes(data).trim();
_reply = "$_reply$_tmpValue";
});
// OPTION 2
await for (var data in _socket) {
String _reply = new String.fromCharCodes(data).trim();
_reply = "$_reply$_tmpValue";
}
UPDATE:
final Socket client = await Socket.connect('192.168.22.120', 3000);
client.add(utf8.encode('$_requestedData\r\n'));
client.listen(
(var data) {
var ints = new Uint8List.fromList(data).buffer.asInt64List();
ints.forEach((i) => print('Got $i'));
},
onDone: () { print('Done'); client.close(); },
onError: (e) { print('Got error $e'); client.close(); });
print('main done');
RESULT
main done
_reply:
Got 4206144564389427761
Got 6861850004482897203
Got 2368148215793798714
UPDATE (Working Solution):
_socket = await Socket.connect('127.0.0.1’, 3000);
_socket.write(_myQuery);
await for (var data in _socket) {
_reply = new String.fromCharCodes(data);
_rawValue = _rawValue + _reply;
int dataLen;
dataLen = _reply.length;
int carriageReturnPosition;
carriageReturnPosition = dataLen - 2;
int newLinePosition;
newLinePosition = dataLen - 1;
// In my scenario it loops 32 times and it takes about 145 seconds to get big data
if (_reply.substring(carriageReturnPosition, newLinePosition) == '\r' &&
_reply.substring(newLinePosition, dataLen) == '\n') {
await _socket.close();
_returnData = _reply;
}
}

Related

How can I get multiple messages from dart isolate?

How can I get multiple messages from dart isolate?
I'm trying to create an excel file and want to do some operation on that file in an isolate. Before doing an operation on that file, I want to return an message to main isolate, that excel file is created.
Here is function goes in isolate :
foo(String filePath){
// create excel file
var bytes = File(filePath).readAsBytesSync();
var excel = Excel.decodeBytes(bytes);
//HERE I WANT TO SEND THE MESSAGE THAT CREATING EXCEL FILE IS DONE
// some operatoin on excel file
var result = doSomeOperation(excel);
return result;
}
Main isolate code :
var result = await compute(foo, filePath);
What should I do to get creating file message before the actual result comes?
For excel, I'm using excel: ^2.0.0-null-safety-3 package.
Compute only returns one result. If you want to pass multiple 'events' back to the main isolate then you need to use the full Isolate logic (with sendPort and receivePort).
For example, the following code runs in an isolate, and downloads a file while emitting float values to represent progress, potentially a String to indicate log messages and then a bool to indicate success or failure upon completion.
Future<void> isolateDownload(
DownloadRequest request) async {
final sendPort = request.sendPort;
if (sendPort != null) {
var success = false;
var errorMessage = '';
var url = Uri.parse('a_url_based_on_request');
IOSink? out;
try {
http.StreamedResponse response =
await http.Client().send(http.Request('GET', url));
if (response.statusCode == 200) {
var filePath =
join(request.destinationDirPath, '${request.fileName}.ZIP');
var contentLength = response.contentLength;
var bytesLoadedUpdateInterval = (contentLength ?? 0) / 50;
var bytesLoaded = 0;
var bytesLoadedAtLastUpdate = 0;
out = File(filePath).openWrite();
await response.stream.forEach((chunk) {
out?.add(chunk);
bytesLoaded += chunk.length;
// update if enough bytes have passed since last update
if (contentLength != null &&
bytesLoaded - bytesLoadedAtLastUpdate >
bytesLoadedUpdateInterval) {
sendPort.send(bytesLoaded / contentLength);
bytesLoadedAtLastUpdate = bytesLoaded;
}
});
success = true;
if (contentLength != null) {
sendPort.send(1.0); // send 100% downloaded message
}
} else {
errorMessage =
'Download of ${request.fileName} '
'received response ${response.statusCode} - ${response.reasonPhrase}';
}
} catch (e) {
errorMessage = 'Download of ${request.chartType}:${request.chartName} '
'received error $e';
} finally {
await out?.flush();
await out?.close();
if (errorMessage.isNotEmpty) {
sendPort.send(errorMessage);
}
sendPort.send(success);
}
}
}
The code that spawns the isolate then simply checks for the type of the message passed to it to determine the action.
Future<bool> _downloadInBackground(
DownloadRequest request) async {
var receivePort = ReceivePort();
request.sendPort = receivePort.sendPort;
var isDone = Completer();
var success = false;
receivePort.listen((message) {
if (message is double) {
showUpdate(message);
}
if (message is String) {
log.fine(message); // log error messages
}
if (message is bool) {
success = message; // end with success or failure
receivePort.close();
}
}, onDone: () => isDone.complete()); // wraps up
await Isolate.spawn(isolateDownload, request);
await isDone.future;
return success;
}

Downloading progress in dart:http

I'm trying to make a progress bar indicator for a downloading file, but if I add a listener to the StreamedResponse, the piping works, but does to not finish its future.
final client = new http.Client();
http.StreamedResponse response = await client.send(http.Request("GET", Uri.parse('someurl')));
var received = 0;
var length = response.contentLength;
//if I remove this listener, the await below gets completed
var listen = response.stream.listen((List<int> bytes) {
received += bytes.length;
print("${(received / length) * 100} %");
});
var sink = downloadFile.openWrite();
await response.stream.pipe(sink);
listen.cancel();
sink.close();
On github they already advised someone that it was supposed to work, but on StreamedResponse docs stays that This should always be a single-subscription stream.. So, adding a listener to calculate the percentage seems to bugs StreamedResponse pipe in someway. Any idea on how to get this to work?
#pskink comment let me to this solution that works for every type of sink you are using.
var length = response.contentLength;
var received = 0;
var sink = downloadFile.openWrite();
await response.stream.map((s) {
received += s.length;
print("${(received / length) * 100} %");
return s;
}).pipe(sink);
another way to accomplish this, if you are writing into a file, is to watch file length
var length = response.contentLength;
var sink = downloadFile.openWrite();
Future.doWhile(() async {
var received = await downloadFile.length();
print("${(received / length) * 100} %");
return received != length;
});
await response.stream.pipe(sink);

Weird issue while processing events coming from kinesis

I setup amazon connect on aws and if I make a test call, it will put that call in a aws kinesis stream. I am trying to write a lambda that process this records and save them to database.
If I make a simple call (call the number - asnwer - hangup) it works just fine. However if I make a multipart call (call a number - answer - trasnfer to another number - hangup) this comes to kinesis as two separate records (CTR).
My lambda process the CTR (Contact Trace Records) one by one. First it saves the CTR to a table called call_segments and then it query this table to see if the other part of this call is already there. If it is, it merges the data and save to a table called completed_calls, otherwise skips it.
If a call has more than on segment (if it was transfered to another number) it brings it to you as two events.
My problem is that even though I am processing the events one after the other it seems that when the second event is processed (technically the call segment from first event is already in database), it can not see the first segment of the call.
here is my code:
const callRecordService = require("./call-records-service");
exports.handler = async (event) => {
await Promise.all(
event.Records.map(async (record) => {
return processRecord(record);
})
);
};
const processRecord = async function(record) {
try{
const payloadStr = new Buffer(record.kinesis.data, "base64").toString("ascii");
let payload = JSON.parse(payloadStr);
await callRecordService.processCTR(payload);
}
catch(err){
// console.error(err);
}
};
and here is the service file:
async function processCTR(ctr) {
let userId = "12"
let result = await saveCtrToContactDetails(ctr, userId);
let paramsForCallSegments = [ctr.InstanceARN.split("instance/").pop(), ctr.ContactId]
let currentCallSegements = await dbHelper.getAll(dbQueries.getAllCallSegmentsQuery, paramsForCallSegments)
let completedCall = checkIfCallIsComplete(currentCallSegements);
if (completedCall) {
console.log('call is complete')
let results = await saveCallToCompletedCalls(completedCall);
}
}
//------------- Private functions --------------------
const saveCtrToContactDetails = async (ctr, userId) => {
let params = [ctr.ContactId,userId,ctr.callDuration];
let results = await dbHelper.executeQuery(dbQueries.getInsertCallDetailsRecordsQuery, params);
return results;
}
const checkIfCallIsComplete = (currentCallSegements) => {
//This function checks if all callSegments are already in call_segments table.
}
const saveCallToCompletedCalls = async (completedCall) => {
let contact_id = completedCall[0].contact_id;
let user_id = completedCall[0].user_id;
let call_duration = completedCall[0] + completedCall[1]
completedCall.forEach(callSegment => {
call_duration += callSegment.call_duration;
});
let params = [contact_id, user_id, call_duration];
let results = await dbHelper.executeQuery(dbQueries.getInsertToCompletedCallQuery, params);
};

How to use Dart socket onError as inline function?

I have dart socket that gets very long data. Lucky the third party add '\r\n' end of the data, so I can close when I find last data has '\r\n'.
In weeks I am trying to get long data due server problem. Before I used to waiting endlessly to server closes the connection (sometime took me 10 to 12 min)
Now seems everything works but a small problem. I used to get onError and onDone using void function. But I am using aqueduct so I need to return long data after I receive from server as a response.
In my below full code its keep printing "Server_Error". If I comment below data seems everything works. But my problem is that I need to return error data as well.
onError: () {
print("Server_Error");
},
onDone: () {
_socket.destroy();
},
cancelOnError: true);
If I comment above part I can print the long data.
My question is that, based on my scenario how to use Dart socket onError as inline function?
import 'dart:io';
import 'dart:convert';
import 'dart:typed_data';
Socket _socket;
String _reply;
String _testValue = "";
main() async {
String _queryA = “QueryLongData”;
await Socket.connect("192.168.22.120”, 3000).then((Socket sock) {
_socket = sock;
_socket.write('$_queryA\r\n');
_socket.listen((data) {
final List<int> byteArray = data;
_reply = String.fromCharCodes(byteArray);
int dataLen = _reply.length;
int carriageReturnPosition = dataLen - 2;
int newLinePosition = dataLen - 1;
_testValue = _testValue + _reply;
if (_reply.substring(carriageReturnPosition, newLinePosition) == '\r' &&
_reply.substring(newLinePosition, dataLen) == '\n') {
_socket.close();
print("Data: $_testValue"); // means return data
}
}, onError: () {
print("Server_Error");
},
onDone: () {
_socket.destroy();
},
cancelOnError: true);
}).catchError((e) {
print("Server_Error");
});
if(_socket.done == true) {
print("Exiting...");
exit(0);
}
}
<!-- language: dart -->
String _queryA = "QueryLongData";
Socket.connect("192.168.22.120", 3000).then((Socket sock) {
_socket = sock;
_socket.write("$_queryA\r\n");
_socket.listen((data) {
final List<int> byteArray = data;
_reply = String.fromCharCodes(byteArray);
int dataLen = _reply.length;
int carriageReturnPosition = dataLen - 2;
int newLinePosition = dataLen - 1;
_testValue = _testValue + _reply;
if (_reply.substring(carriageReturnPosition, newLinePosition) == "\r" &&
_reply.substring(newLinePosition, dataLen) == "\n") {
_socket.close();
print("Data: $_testValue"); // means return data
}
}, onError: () => print("Server_Error"),
onDone: () => _socket.destroy(),
cancelOnError: true);
}).catchError((e) => print("Server_Error"));
if( ( await _socket.done enter code here) == true) {
print("Exiting...");
exit(0);
}

Writing to sockets

I have trouble understanding the reason for the error I get when the user tries to write anything to the server:
TypeError: Object #<identifyClient> has no method 'write'
at writeToAll (/root/node/mud/server.js:13:15)
Why does identifyClient() complaints about the write(), while it happens in the writeToAll() (line 15 has comment next to it)? identifyClient() really only sets the name for the client, and should not be concerned what happens in the writeToAll().
var net = require("net");
var clients = [];
function identifyClient(client) {
this.name = null;
this.client = client;
}
function writeToAll(data, client) {
for (var i = 0; i < clients.length; i++) {
if (clients[i] != client) {
clients[i].write(data); // This is line 15
}
}
}
var server = net.createServer(function(client) {
var clientID = new identifyClient(client);
clients.push(clientID);
client.on("data", function(data) {
writeToAll(data, client);
});
});
server.listen(4444);
Replace clients[i] with clients[i].client
also you have to remove the client from the clients array once it disconnects.