I have tried and failed to implement a TCP-Listen server in Matlab that is "lossless". By lossless, I mean using the linux socat utility to send a file:
socat -u file.bin TCP4:127.0.0.1:50000
And receive a byte-exact match to that file within Matlab:
function t = test
fid = fopen('x','W');
t =tcpip('0.0.0.0',50000,'NetWorkRole','server','InputBufferSize',50*1024^2);
t.BytesAvailableFcnMode = 'byte';
t.BytesAvailableFcnCount = 1024^2;
t.BytesAvailableFcn = {#FCN, fid };
fopen(t);
function FCN( obj, event, fid )
x=fread( igetfield(obj, 'jobject'), obj.BytesAvailable, 0, 0);
fwrite( fid, x(1),'int8' );
I've tested this a good bit, and had decent success in terms of transfer rate (without the fwrite and use /dev/zero for the file, it saturates a gigabit link), and low cpu load. The trick is bypassing Matlab's tcpip() default wrapper*, and accessing a lower-level method via:
igetfield(obj,'jobject')
For the 2139160576 Byte file I test with, it usually receives ~2139153336 bytes. I've tried various other implementations that receive the fread() output into structs, cells, and array concatenation. They also are missing a few KiB's. I've tried a repeating pattern of 512 random bytes; one test had byte mismatch at the beginning.
Socat->Socat transfer works (obviously).
Socat->my-matlab-code holds at fopen() until socat connects. No data is transferred until fread() is called. I've tried throttling the transfer with the linux "pv" utility:
pv -L 10m file.bin | socat -u - TCP4:127.0.0.1:50000
To no effect.
My question is: where are the bytes going, or what should I test next?
(Edited to include further test results):
Outputting to a file is unnecessary, i.e. the fwrite() call. It is easier and faster to execute t=test, wait for the transfer to complete, i.e. the socat client to return, then query the total bytes transferred from within Matlab:
t.BytesAvailable + t.ValuesReceived
On my Windows machine, this value is always less than the file size of 2139160576 bytes. On my Ubuntu machine, occasionally the values equate. Furthermore, when they do not equate, "netstat -s" segments retransmited and packets discarded do not change. Wireshark monitoring of the loopback interface shows a final Matlab/server ACK sequence number of 2139160578. Presumably, 2 more than the file size due to both the server and client incrementing by one.
*As an aside, Matlab's Instrument Control Toolbox implementation of fread is a terrible wrapper around lower-level code I can't see. matlab\toolbox\shared\instrument\#icinterface\fread.m, function localFormatData, LINE 296. All the data types are explicitly cast to double with numeric type conversion. This results in massive cpu load, not to mention lossy conversion between data types.
Related
I am trying to write a .dbc file for a can-open data log (example of one of lines I am trying to use below)
Time 884.163000, ID:2a1, Data Bytes (0)7b (1)00 (2)95 (3)68 (4)e5 (5)8e (6)49 (7)54
I have written .dbc files using both motorola and intel endianness using candb++ covering 16 bit data over 2 bytes, but this has always been with sequential bytes, ie- (2),(3) or (5),(6).
The bytes I need to use for the particular data in the above example are (3) and (7) in intel format (54,68 in this case). I have written a .dbc for just byte 3 shown in the snip below.
BO_ 673 Rig_Pressure: 4 Vector__XXX
SG_ Pressure_Multiplex M : 15|8#0+ (1,0) [0|0] "" Vector__XXX
SG_ Pressure m0 : 31|8#0- (0.45,58) [0.399999999999999|115.15] "Bar" Vector__XXX
I am asking if there is a way to modify the text file (or use cabdb++) to specify each bit or pick 2 non sequential bytes in the .dbc, something like modifying the bit start something like
31|8#0- to 31|8#0- 63|8#0-
I am far from a computer programmer, I much prefer GUI based programs and am only starting out in learning python, so please be gentle!!!
Thank you!
if mmap() was used to read a file, how can I find the number of data mapped by mmap().
float *map = (float *)mmap(NULL, FILESIZE, PROT_READ, MAP_SHARED, fd, 0);
The mmap system call does not read data. It just maps the data in your virtual address space (by indirectly configuring your MMU), and that virtual address space is changed by a successful mmap. Later, your program will read that data (or not). In your example, your program might later read map[356] if mmap has succeeded (and you should test against its failure).
Read carefully the documentation of mmap(2). The second argument (in your code, FILESIZE) defines the size of the mapping (in bytes). You might check that it is a multiple of sizeof(float) and divide it by sizeof(float) to get the number of elements in map that are meaningful and obtained from the file. The size of the mapping is rounded up to a multiple of pages. The man page of mmap(2) says:
A file is mapped in multiples of the page size. For a file that is
not a multiple of the page size, the remaining memory is zeroed when
mapped, and writes to that region are not written out to the file.
Data is mapped in pages. A page is usually 4096 bytes. Read more about paging.
The page size is returned by getpagesize(2) or by sysconf(3) with _SC_PAGESIZE (which usually gives 4096).
Consider reading some book like Operating Systems: Three Easy Pieces (freely downloadable) to understand how virtual memory works and what is a memory mapped file.
On Linux, the /proc/ filesystem (see proc(5)) is very useful to understand the virtual address space of some process: try cat /proc/$$/maps in your terminal, and read more to understand its output. For a process of pid 1234, try also cat /proc/1234/maps
From inside your process, you could even read sequentially the /proc/self/maps pseudo-file to understand its virtual address space, like here.
I would like to read a block of data from my Arduino Mega (and also from an Arduino Micro in another project) with my Raspberry Pi via I2C. The code has to be in Perl because it's sort of a plug-in for my Home-Automation-Server.
I'm using the Device::SMBus interface and the connection works, I'm able to write and read single Bytes. I can even use writeBlockData with register address 0x00. I randomly discovererd that this address works.
But when I want to readBlockData, no register-address seems to work.
Does anyone know the correct register-address, or is that not even the problem that causes errors?
Thanks in advance
First off, which register(s) are you wanting to read? Here's an example using my RPi::I2C software (it should be exceptionally similar with the distribution you're using), along with a sketch that has a bunch of pseudo-registers configured for reading/writing.
First, the Perl code. It reads two bytes (the output of an analogRead() of pin A0 which is set up as register 80), then bit-shifts the two bytes into a 16-bit integer to get the full 0-1023 value of the pin:
use warnings;
use strict;
use RPi::I2C;
my $arduino_addr = 0x04;
my $arduino = RPi::I2C->new($arduino_addr);
my #bytes = $arduino->read_block(2, 80);
my $a0_value = ($bytes[0] << 8) | $bytes[1];
print "$a0_value\n";
Here's a full-blown Arduino sketch you can review that sets up a half dozen or so pseudo-registers, and when each register is specified, the Arduino writes or reads the appropriate data. If no register is specified, it operates on 0x00 register.
The I2C on the Arduino always does an onReceive() call before it does the onRequest() (when using Wire), so I set up a global variable reg to hold the register value, which I populate in the onReceive() interrupt, which is then used in the onRequest() call to send you the data at the pseudo-register you've specified.
The sketch itself doesn't really do anything useful, I just presented it as an example. It's actually part of my automated unit test platform for my RPi::WiringPi distribution.
I still have problems with Octave. I need to receive large (>10M) images via a web socket connection, process the data and send it back. I work with Octave 4.2.1 in Windows 7, the sockets package is version 1.2.0 and can be found here: https://octave.sourceforge.io/sockets/
Here is the minimal code example:
pkg load sockets;
rcv_sck=socket(AF_INET, SOCK_STREAM, 0);
bind(rcv_sck,12345);
a=listen(rcv_sck,0);
b=accept(rcv_sck);
data = [];
bufflen = 4600;
total = 0;
count = 0;
while (total < 10000)
[buff,count]=recv(b, bufflen);
data = horzcat(data, buff);
total += count;
endwhile
disconnect (b);
disconnect (rcv_sck);
fileID = fopen('data.jpg','w');
fwrite('data.jpg', data);
fclose(fileID);
imshow ('data.jpg')
it can be tested with netcat
ncat.exe 127.0.0.1 12345 < test.jpg
My problem is, that I am not able to receive data bigger than 4608 bytes. If I send more data, the receive array is cut to this size. This is why i choose 4600 bytes as a buffer length. Now I try to put the chunks of small data blocks together to get the correct data.
But this has two serious issues:
1) I have to know the size of the data I am sending, a problem which can be solved by sending the size first as a parameter. In my example i have set the size to 10000 bytes.
2) More important: it is terribly slow. For a jpeg of 170kB it takes 7 seconds to send the data.
Any hints or tipps are greatly appreciated, thank you.
jan
If you want to read data until disconnect I would suggest
data = recv (b, Inf, MSG_WAITALL);
without any loop or horcat. And on your client side (netcat) use -q0:
netcat.exe -q0 localhost 12345 < test.jpg
Btw, if you want to store it on the server side, why don't you use
system ("netcat -l -p 12345 > data.jpg");
in the first place?
Quoted from MSDN entry for TransmitFile:
The maximum number of bytes that can be transmitted using a single call to the TransmitFile function is 2,147,483,646, the maximum value for a 32-bit integer minus 1. The maximum number of bytes to send in a single call includes any data sent before or after the file data pointed to by the lpTransmitBuffers parameter plus the value specified in the nNumberOfBytesToWrite parameter for the length of file data to send. If an application needs to transmit a file larger than 2,147,483,646 bytes, then multiple calls to the TransmitFile function can be used with each call transferring no more than 2,147,483,646 bytes. Setting the nNumberOfBytesToWrite parameter to zero for a file larger than 2,147,483,646 bytes will also fail since in this case the TransmitFile function will use the size of the file as the value for the number of bytes to transmit.
Alright. Sending a file of size 2*2,147,483,646 bytes (~ 4 GiB) with TransmitFile would then have to be divided into two parts at minimum (e.g. 2 GiB + 2 GiB in two calls to TransmitFile). But how exactly would one go about doing that, while preferably also keeping the underlying TCP connection alive in between?
When the file is indeed <=2,147,483,646 bytes in size, one could just write:
HANDLE fh = CreateFile(filename, GENERIC_READ, FILE_SHARE_READ, NULL,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL | FILE_FLAG_SEQUENTIAL_SCAN, NULL);
TransmitFile(SOCK_STREAM_socket, fh, 0, 0, NULL, NULL, TF_DISCONNECT);
to let Windows handle all the lower-level stuff (caching, chunking the data up into pieces for efficient transmission etc. However, unlike the comparable Linux sendfile() syscall, there is no immediately obvious offset argument in the call (although the fifth argument, LPOVERLAPPED lpOverlapped probably is exactly what I'm looking for). I suppose I could hack something together, but I'm also looking for a graceful, good practice Win32 solution from someone who actually knows about this stuff.
You can use the lpOverlapped parameter to specify a 64-bit offset within the file at which to start the file data transfer by setting the Offset and OffsetHigh member of the OVERLAPPED structure. If lpOverlapped is a NULL pointer, the transmission of data always starts at the current byte offset in the file.
So, for lack of a minimal example readily available on the net, which calls are necessary to accomplish such a task?
Managed to figure it out based on the comments.
So, if LPOVERLAPPED lpOverlapped is a null pointer, the call starts transmission at the current file offset of the file (much like the Linux sendfile() syscall and its off_t *offset parameter). This offset (pointer) can be manipulated with SetFilePointerEx, so one could write:
#define TRANSMITFILE_MAX ((2<<30) - 1)
LARGE_INTEGER total_bytes;
memset(&total_bytes, 0, sizeof(total_bytes));
while (total_bytes < filesize) {
DWORD bytes = MIN(filesize-total_bytes, TRANSMITFILE_MAX);
if (!TransmitFile(SOCK_STREAM_socket, fh, bytes, 0, NULL, NULL, 0))
{ /* error handling */ }
total_bytes.HighPart += bytes;
SetFilePointerEx(fh, total_bytes, NULL, FILE_BEGIN);
}
closesocket(SOCK_STREAM_socket);
to accomplish the task.
Not very elegant imo, but it works.