I have a fixed message protocol to work with for a COM device. How do I
specifically declare that I do not have a termination character when I write to the serial port?
If I declare that I do not have any termination character from the serial port, is it necessary to specify that in the Serial.readline() as well?
import serial
ser = serial.Serial(
port='COM4',
baudrate=115200,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS,
timeout=None,
xonxoff=False,
rtscts=False,
write_timeout=None,
dsrdtr=False,
inter_byte_timeout=None,
exclusive=None)
If you don't have a line termination character and your message is of fixed length it makes more sense to read a fixed number of bytes from the port.
If, for instance, your message is 100 bytes you can do:
serial.read(100) # Reads UP TO 100 bytes
Note that if you have a None timeout this will read up to 100 bytes. So if you have less than that it will return as many as it found (that might be 99, 5, or none).
With this in mind, it is recommended that you check you received the complete message by comparing the number of bytes received with the expected message length.
You can also define a timeout with timeout=1 and do something like this:
timeout=time.time()+3.0
while ser.inWaiting() or time.time()-timeout<0.0:
if ser.inWaiting()>0:
data+=ser.read(ser.inWaiting())
timeout=time.time()+3.0
print(data)
This will make sure you read the whole message and nothing else has been arrived to the buffer by waiting for 3 seconds after you finish reading.
Related
I'm currently attempting to read an incoming message from a client socket, that, prior to the below procedure has already been connected to the server socket. The below procedure outputs the message, one character at a time, as it retrieves it from the stream.
The problem is that, when the stream is out of information, the call to Ada.Streams.Read is blocking, and stops the application flow completely. According to some examples, it would appear as though Offset should be set to 0 automatically at the end of the stream, but that never happens. Instead the application stops at the call to Read.
procedure Read_From (Channel : Sockets.Stream_Access) is
use Ada.Text_IO;
use Ada.Streams;
Data : Stream_Element_Array (1 .. 1);
Offset : Stream_Element_Offset;
begin
loop
Read (Channel.All, Data, Offset);
exit when Offset = 0;
Put (Character'Val (Data (1)));
end loop;
-- The application never reaches this point.
New_Line;
Put_Line ("Finished reading from client!");
end Read_From;
-- #param Channel `GNAT.Sockets.Stream (Client_Socket)`
I've also attempted the same process with GNAT.Sockets.Receive_Socket, but the same issue remains: the application flow is stopped completely, assumably awaiting further information from the stream, even though there is nothing more to retrieve.
Any pointers in the right direction would be highly appreciated!
Normally, you’d read a (binary) message from a stream knowing how much data needed to be read, so you could read until you’d got that much.
But, if you’re reading a text message from an externally-defined source, as it might be an HTTP request, there needs to be some terminator sequence so you can read character-by-character until you’ve read the terminator. In the case of an HTTP request, that’s a CR/LF/CR/LF sequence. Or it could be a null-terminated C string, in which case you’d be looking for the ASCII.NUL.
The Ada way to transfer variable-length text is to use String’Output/String’Input (see ARM 13.13.2(18)ff). What happens for a String (an array of Character) is that first the bounds are sent, then the content; on reception, the bounds are read, a String with those bounds is created, and the required number of bytes are read into the new String, which is then returned.
Basically that's how Ada streams work. The end of the stream only comes once you reach the final end of the stream, not just the current end of a buffer.
If you want to be able to interrupt reading, you have to use another representation of the connection than GNAT.Sockets.Stream_Access.
Quoted from MSDN entry for TransmitFile:
The maximum number of bytes that can be transmitted using a single call to the TransmitFile function is 2,147,483,646, the maximum value for a 32-bit integer minus 1. The maximum number of bytes to send in a single call includes any data sent before or after the file data pointed to by the lpTransmitBuffers parameter plus the value specified in the nNumberOfBytesToWrite parameter for the length of file data to send. If an application needs to transmit a file larger than 2,147,483,646 bytes, then multiple calls to the TransmitFile function can be used with each call transferring no more than 2,147,483,646 bytes. Setting the nNumberOfBytesToWrite parameter to zero for a file larger than 2,147,483,646 bytes will also fail since in this case the TransmitFile function will use the size of the file as the value for the number of bytes to transmit.
Alright. Sending a file of size 2*2,147,483,646 bytes (~ 4 GiB) with TransmitFile would then have to be divided into two parts at minimum (e.g. 2 GiB + 2 GiB in two calls to TransmitFile). But how exactly would one go about doing that, while preferably also keeping the underlying TCP connection alive in between?
When the file is indeed <=2,147,483,646 bytes in size, one could just write:
HANDLE fh = CreateFile(filename, GENERIC_READ, FILE_SHARE_READ, NULL,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL | FILE_FLAG_SEQUENTIAL_SCAN, NULL);
TransmitFile(SOCK_STREAM_socket, fh, 0, 0, NULL, NULL, TF_DISCONNECT);
to let Windows handle all the lower-level stuff (caching, chunking the data up into pieces for efficient transmission etc. However, unlike the comparable Linux sendfile() syscall, there is no immediately obvious offset argument in the call (although the fifth argument, LPOVERLAPPED lpOverlapped probably is exactly what I'm looking for). I suppose I could hack something together, but I'm also looking for a graceful, good practice Win32 solution from someone who actually knows about this stuff.
You can use the lpOverlapped parameter to specify a 64-bit offset within the file at which to start the file data transfer by setting the Offset and OffsetHigh member of the OVERLAPPED structure. If lpOverlapped is a NULL pointer, the transmission of data always starts at the current byte offset in the file.
So, for lack of a minimal example readily available on the net, which calls are necessary to accomplish such a task?
Managed to figure it out based on the comments.
So, if LPOVERLAPPED lpOverlapped is a null pointer, the call starts transmission at the current file offset of the file (much like the Linux sendfile() syscall and its off_t *offset parameter). This offset (pointer) can be manipulated with SetFilePointerEx, so one could write:
#define TRANSMITFILE_MAX ((2<<30) - 1)
LARGE_INTEGER total_bytes;
memset(&total_bytes, 0, sizeof(total_bytes));
while (total_bytes < filesize) {
DWORD bytes = MIN(filesize-total_bytes, TRANSMITFILE_MAX);
if (!TransmitFile(SOCK_STREAM_socket, fh, bytes, 0, NULL, NULL, 0))
{ /* error handling */ }
total_bytes.HighPart += bytes;
SetFilePointerEx(fh, total_bytes, NULL, FILE_BEGIN);
}
closesocket(SOCK_STREAM_socket);
to accomplish the task.
Not very elegant imo, but it works.
I'm writing a little VOIP app like Skype, which works quite good right now, but I've run into a very strange problem.
In one thread, I'm calling within a while(true) loop the winsock recv() function twice per run to get data from a socket.
The first call gets 2 bytes which will be casted into a (short) while the second call gets the rest of the message which looks like:
Complete Message: [2 Byte Header | Message, length determined by the 2Byte Header]
These packets are round about 49/sec which will be round about 3000bytes/sec.
The content of these packets is audio-data that gets converted into wave.
With ioctlsocket() I determine wether there is some data on the socket or not at each "message" I receive (2byte+data). If there's something on the socket right after I received a message within the while(true) loop of the thread, the message will be received, but thrown away to work against upstacking latency.
This concept works very well, but here's the problem:
While my VOIP program is running and when I parallely download (e.g. via browser) a file, there always gets too much data stacked on the socket, because while downloading, the recv() loop seems actually to slow down. This happens in every download/upload situation besides the actual voip up/download.
I don't know where this behaviour comes from, but when I actually cancel every up/download besides the voip traffic of my application, my apps works again perfectly.
If the program runs perfectly, the ioctlsocket() function writes 0 into the bytesLeft var, defined within the class where the receive function comes from.
Does somebody know where this comes from? I'll attach my receive function down below:
std::string D_SOCKETS::receive_message(){
recv(ClientSocket,(char*)&val,sizeof(val),MSG_WAITALL);
receivedBytes = recv(ClientSocket,buffer,val,MSG_WAITALL);
if (receivedBytes != val){
printf("SHORT: %d PAKET: %d ERROR: %d",val,receivedBytes,WSAGetLastError());
exit(128);
}
ioctlsocket(ClientSocket,FIONREAD,&bytesLeft);
cout<<"Bytes left on the Socket:"<<bytesLeft<<endl;
if(bytesLeft>20)
{
// message gets received, but ignored/thrown away to throw away
return std::string();
}
else
return std::string(buffer,receivedBytes);}
There is no need to use ioctlsocket() to discard data. That would indicate a bug in your protocol design. Assuming you are using TCP (you did not say), there should not be any left over data if your 2byte header is always accurate. After reading the 2byte header and then reading the specified number of bytes, the next bytes you receive after that constitute your next message and should not be discarded simply because it exists.
The fact that ioctlsocket() reports more bytes available means that you are receiving messages faster than you are reading them from the socket. Make your reading code run faster, don't throw away good data due to your slowness.
Your reading model is not efficient. Instead of reading 2 bytes, then X bytes, then 2 bytes, and so on, you should instead use a larger buffer to read more raw data from the socket at one time (use ioctlsocket() to know how many bytes are available, and then read at least that many bytes at one time and append them to the end of your buffer), and then parse as many complete messages are in the buffer before then reading more raw data from the socket again. The more data you can read at a time, the faster you can receive data.
To help speed up the code even more, don't process the messages inside the loop directly, either. Do the processing in another thread instead. Have the reading loop put complete messages in a queue and go back to reading, and then have a processing thread pull from the queue whenever messages are available for processing.
While reading data in socket its important either keep a message terminator symbol or add the Packet size information at the begening of the message.
If a terminator symbol is used and a binary message is sent there is no guarantee that the terminator symbol would not appear in the middle of the message (unless some special encoding is used).
On the other hand if size information is attached. size information is unsigned and if one byte is used for it it cannot be used to transfer messages longer than 256 bytes. if 4 byte integer is used. its not even guaranteed that 4 bytes will come a s whole. just 2 bytes of the size information may come can assuming the size information has arrived it may use that 2 bytes and rest of the integer data will be discarded. waiting for 4 bytes to be available on read buffer may cause infinite awaiting if only 3 bytes are available on the buffer (e.g. if total buffer is 7 bytes or 4077 bytes long).
here comes two possible ways
sizeInfo separator chunk
read until the separator is found once found read until sizeInfo bytes passed
keep an unreadyBytes initialized at 4 upon receiving the sizeInfo change it accordingly
which one of these two is safer to use ? Please Criticize
Edit
My central question is how to make sure that the size bytes has arrived properly. assuming messages are of variable size.
its not even guaranteed that 4 bytes will come a s whole. just 2 bytes of the size information may come can assuming the size information has arrived it may use that 2 bytes and rest of the integer data will be discarded. waiting for 4 bytes to be available on read buffer may cause infinite awaiting if only 3 bytes are available on the buffer (e.g. if total buffer is 7 bytes or 4077 bytes long).
If you have a 4 bytes length descriptor you should always read at least 4 bytes, because the sender should have written this bytes in every message your server is receiving. If you can't get them, maybe there has been a problem in transmission. I really can't understand your problem.
Anyway I'll suggest to you not to use any separator chunk.
Put an header at data blocks you are transmitting and use a buffer to reconstruct the packet flow.
You must at least read the header of a packet to determine its length.
You can define a basic structure for a packet:
struct packet{
uint32 id;
char payload[MAX_PAYLOAD_SIZE];
};
The you read data from socket storing them into a buffer:
struct packet buffer;
Then you can read the data from the socket:
int n;
n = read(newsocket, &buffer, sizeof(uint32) + MAX_PAYLOAD_SIZE);
read returns the number of bytes read. If you read exactly a packet from the sender, then n = id. Otherwise maybe you read more data (es. the sender sent to you more packets). If you are receiving a stream of data split into unit (represented by packet structures), then you may use an array of packet to store the complete packet received and a temporarily buffer to manage incoming fragments.
Something like:
struct packet buffer[MAX_PACKET_STORED];
char temp_buffer[MAX_PAYLOAD_SIZE + 4];
int n;
n = read(newsocket, &buffer, sizeof(uint32) + MAX_PAYLOAD_SIZE);
//here suppose have received a packet of 100 Byte payload + 32 bit of length + 100 Byte
//fragments of the next packet.
//then:
int first_pack_len, second_pack_len;
first_pack_len = *((uint32 *)&temp_buffer[0]); //retrieve packet length
memcpy(&packet_buffer[0], temp_buffer, first_pack_len + sizeof(uint32)) //store the first packet into the array
second_pack_data_available_in_buffer = n - (first_pack_len + sizeof(uint32)); //total bytes read minus length of the first packet read
second_pack_len = *((int *)&temp_buffer[first_pack_len + sizeof(uint32)]);
I hope to have been clear enough. But maybe I'm misunderstanding your question.
Pay attention also that if the 2 end-systems communicating could have different endiannes, so it's a better idea use htonl/ntohl function on length when sending/receving length value. But this is another issue)
How do I interpret dwParam1 from the midiInProc delegate into midi status message like note-off, or note-on, control change?
Because as long i try dwParam1 is 254, and is not equal to note-off or anything else.
You won't necessarily receive note-offs from every input device. IIRC it is legal for a device to send a note-on with volume=0 as a substitute for note-off. Also a drum stream (from a drum machine and/or on MIDI channel 10) I believe commonly contains only note-ons, no note-offs.
Given that your question mentions dwParam1 and midiInProc, I'm assuming this is for Windows. When you receive MIM_DATA in your midiInProc, you can parse dwParam1 as follows:
For the status byte (command and channel), use LOBYTE(dwParam1).
For the first data byte, use HIBYTE(dwParam1).
If applicable, for the second data byte, use LOBYTE(HIWORD(dwParam1)).
I'm not entirely sure what you are asking, but I think you are trying to figure out how to interpret MIDI data.
I suggest this resource:
http://www.midi.org/techspecs/midimessages.php
MIDI messages related to notes are differentiated by the first 4 bits, not by the whole byte. The last four bits of the first byte specify the channel.
The answer by #Conrad Albrecht is mostly right, but I wanted to chip in with an answer (instead of a comment), as I think that the original poster is probably being confused by MIDI running status.
If you are seeing bytes which don't resemble normal MIDI status bytes, you can assume that they are of the same type as the previous byte which you received. Therefore it is not only legal, but very common, to use MIDI note on events with velocity of 0 as a substitute for MIDI note offs.
You should just interpret these bytes as the normal second two bytes of a MIDI note on event.