Octave Sockets Package, how to receive large data streams - sockets

I still have problems with Octave. I need to receive large (>10M) images via a web socket connection, process the data and send it back. I work with Octave 4.2.1 in Windows 7, the sockets package is version 1.2.0 and can be found here: https://octave.sourceforge.io/sockets/
Here is the minimal code example:
pkg load sockets;
rcv_sck=socket(AF_INET, SOCK_STREAM, 0);
bind(rcv_sck,12345);
a=listen(rcv_sck,0);
b=accept(rcv_sck);
data = [];
bufflen = 4600;
total = 0;
count = 0;
while (total < 10000)
[buff,count]=recv(b, bufflen);
data = horzcat(data, buff);
total += count;
endwhile
disconnect (b);
disconnect (rcv_sck);
fileID = fopen('data.jpg','w');
fwrite('data.jpg', data);
fclose(fileID);
imshow ('data.jpg')
it can be tested with netcat
ncat.exe 127.0.0.1 12345 < test.jpg
My problem is, that I am not able to receive data bigger than 4608 bytes. If I send more data, the receive array is cut to this size. This is why i choose 4600 bytes as a buffer length. Now I try to put the chunks of small data blocks together to get the correct data.
But this has two serious issues:
1) I have to know the size of the data I am sending, a problem which can be solved by sending the size first as a parameter. In my example i have set the size to 10000 bytes.
2) More important: it is terribly slow. For a jpeg of 170kB it takes 7 seconds to send the data.
Any hints or tipps are greatly appreciated, thank you.
jan

If you want to read data until disconnect I would suggest
data = recv (b, Inf, MSG_WAITALL);
without any loop or horcat. And on your client side (netcat) use -q0:
netcat.exe -q0 localhost 12345 < test.jpg
Btw, if you want to store it on the server side, why don't you use
system ("netcat -l -p 12345 > data.jpg");
in the first place?

Related

Serial async write: How to add data to the output buffer & weird output buffer behaviour

I am using asynchronous serial write operation to send data. Using fwrite() I cannot send data if a write operation is already in progress. It returns an error:
Unsuccessful write: An asynchronous write is already in progress.
How can I simply add the extra data I need to write to the existing output buffer, instead of having to wait for it to finish?
Also, I tried to see the number of bytes left to output, but it would either show the full number (511, default size) or zero. So I increased the size of the output buffer to 100,000 sent that many items and plotted the BytesToOutput, this is what I got:
It drops from ~25,000 directly to 0!? Can anyone explain this please?
Here is my code:
instrreset; %closes, deletes and clears all available serial objects
s=serial('/dev/ttyACM0');
s.BaudRate= 1000000;
s.Timeout= 0.1;
s.OutputBufferSize= 100000;
fopen(s);
data=[1:99999];
bytesLeft=zeros(1,100000);
disp('sending data now... wait for it to complete')
tic()
fwrite(s, data, 'uint8', 'async');
f=1;
while ( strcmp(s.TransferStatus, 'write') )
bytesLeft(f)=s.BytesToOutput;
f=f+1;
%disp(s.BytesToOutput)
end
toc()
plot(bytesLeft);
fclose(s);
delete(s);
clear s;

Matlab TCP-IP Server Lossless Transfer

I have tried and failed to implement a TCP-Listen server in Matlab that is "lossless". By lossless, I mean using the linux socat utility to send a file:
socat -u file.bin TCP4:127.0.0.1:50000
And receive a byte-exact match to that file within Matlab:
function t = test
fid = fopen('x','W');
t =tcpip('0.0.0.0',50000,'NetWorkRole','server','InputBufferSize',50*1024^2);
t.BytesAvailableFcnMode = 'byte';
t.BytesAvailableFcnCount = 1024^2;
t.BytesAvailableFcn = {#FCN, fid };
fopen(t);
function FCN( obj, event, fid )
x=fread( igetfield(obj, 'jobject'), obj.BytesAvailable, 0, 0);
fwrite( fid, x(1),'int8' );
I've tested this a good bit, and had decent success in terms of transfer rate (without the fwrite and use /dev/zero for the file, it saturates a gigabit link), and low cpu load. The trick is bypassing Matlab's tcpip() default wrapper*, and accessing a lower-level method via:
igetfield(obj,'jobject')
For the 2139160576 Byte file I test with, it usually receives ~2139153336 bytes. I've tried various other implementations that receive the fread() output into structs, cells, and array concatenation. They also are missing a few KiB's. I've tried a repeating pattern of 512 random bytes; one test had byte mismatch at the beginning.
Socat->Socat transfer works (obviously).
Socat->my-matlab-code holds at fopen() until socat connects. No data is transferred until fread() is called. I've tried throttling the transfer with the linux "pv" utility:
pv -L 10m file.bin | socat -u - TCP4:127.0.0.1:50000
To no effect.
My question is: where are the bytes going, or what should I test next?
(Edited to include further test results):
Outputting to a file is unnecessary, i.e. the fwrite() call. It is easier and faster to execute t=test, wait for the transfer to complete, i.e. the socat client to return, then query the total bytes transferred from within Matlab:
t.BytesAvailable + t.ValuesReceived
On my Windows machine, this value is always less than the file size of 2139160576 bytes. On my Ubuntu machine, occasionally the values equate. Furthermore, when they do not equate, "netstat -s" segments retransmited and packets discarded do not change. Wireshark monitoring of the loopback interface shows a final Matlab/server ACK sequence number of 2139160578. Presumably, 2 more than the file size due to both the server and client incrementing by one.
*As an aside, Matlab's Instrument Control Toolbox implementation of fread is a terrible wrapper around lower-level code I can't see. matlab\toolbox\shared\instrument\#icinterface\fread.m, function localFormatData, LINE 296. All the data types are explicitly cast to double with numeric type conversion. This results in massive cpu load, not to mention lossy conversion between data types.

Reading raw serial data in Matlab

I'm trying to read some raw telemetry data via serial. Each message terminates with \r\n and there are 4 kinds of messages.
I setup the port like this:
if exist('s')
fclose(s);
delete(s);
clear s;
end
s = serial('/dev/ttyS99');
s.BaudRate = 57600;
s.BytesAvailableFcnMode = 'terminator';
s.Terminator = 'CR/LF';
s.DataBits = 8;
s.Parity = 'none';
s.StopBits = 1;
s.BytesAvailableFcn = #serial_callback;
s.OutputEmptyFcn = #serial_callback;
fopen(s);
For the calback, i wrote a simple function
function serial_callback(obj, event)
if (obj.BytesAvailable > 0)
[serial_out, count] = fscanf(obj);
disp(count)
end
end
end
Using fscanf, I get random message lengths. For instance, I am searching for a message with length 42, and it only retrieves messages with length near that value (40, 43, 46...).
I i use fread with unspecified size, I allways get a full 512 bytes buffer. If I specify the size in fread with get(obj, 'BytesAvailable), it degenerates in the sizes of fscanf, i.e., totally random.
So, am I doing something wrong, is matlab bad for parsing serial data...?
P.S. I am getting something like 40 messages of 20~40 bytes per second.
Call the callback function everytime a \r\n has been received, using BytesAvailableFcnMode to configure. fscanf should only be called when a line is completely received by the buffer.
Use fgets or fgetl to read one line from file. This will keep all things after the first \r\n in the buffer. I don't have a serial port nor a Tek so I can't verify this to be working for serial ports.
Probably you need a while loop to read all lines in the buffer. Again I'm not sure how serial does callback but it's unlikely to continuously triggering callback function when there's nothing newly arrived after callback has been called.

Binary files sent over socket are corrupted

I'm doing a code that will transfer files between two computers. I'm using tcp socket for the connection. The thing is I need to attach sort of headers to the file bytes that I'm sending so the receiveer know that what I'm sending is part of a file. Let's say my header is data. The string I'll send will be: data <file bytes>.
I'm able to send them and the receiver is able to receive them but the file seems corrupted. Though for unformatted text files it works well but for other files it doesn't seem to parse the file efficiently.
while(1){
fp = (char*) malloc (56);
rc = recv(connfd,fp,55,0);
if(strcmp(fp,"stop") == 0){
break;
}
fp = fp + 5; //I do this to skip the 'data<space>" header
wr = write(fd,pf2,rc-5);
tot = tot + wr;
printf("Received a total of %d bytes rc = %d \n",tot, rc);
}
But I've tried sending the file without the header and I get the file uncorrupted but I need to use those 'data' headers for this particular code. What am I doing wrong?
fp = fp + 5; //I do this to skip the 'data<space>" header
But you don't receive the data<space> header in every receive() call. You have to keep a buffer to which you add all data you receive, until you encounter another "data<space>".
Please note though that separators are generally a bad idea. What if you send a file that has the string "data<space>" in it? Your client will assume that after that, a new file will be sent, while in fact you're still receiving the original file.
Try to send some kind of message-length-header, for example an uint32, which occupies four bytes before each file you send. You can then read the first four bytes and then you know how many more bytes you can expect for that file.

unix sockets: how to send really big data with one "send" call?

I'm using unix scoket for data transferring (SOCK_STREAM mode)
I need to send a string of more than 100k chars. Firstly, I send length of a string - it's sizeof(int) bytes.
length = strlen(s)
send(sd, length, sizeof(int))
Then I send the whole string
bytesSend = send(sd, s, length)
but for my surprise "bytesSend" is less than "length".
Note, that this works fine when I send not so big strings.
May be there exist some limitations for system call "send" that I've been missing ...
The send system call is supposed to be fast, because the program may have other things useful things to do. Certainly you do not want to wait for the data to be sent out and the other computer to send a reply - that would lead to terrible throughput.
So, all send really does is queues some data for sending and returns control to the program. The kernel could copy the entire message into kernel memory, but this would consume a lot of kernel memory (not good).
Instead, the kernel only queues as much of the message as is reasonable. It is the program's responsibility to re-attempt sending of the remaining data.
In your case, use a loop to send the data that did not get sent the first time.
while(length > 0) {
bytesSent = send(sd, s, length);
if (bytesSent == 0)
break; //socket probably closed
else if (bytesSent < 0)
break; //handle errors appropriately
s += bytesSent;
length -= bytesSent;
}
At the receiving end you will likely need to do the same thing.
Your initial send() call is wrong. You need to pass send() the address of the data, i.e.:
bytesSend = send(sd, &length, sizeof(int))
Also, this runs into some classical risks, with endianness, size of int on various platforms, et cetera.