I'm sending data over an UDP socket and receive it in a loop with read().
The input data looks like this:
String 1
String 2
String 3
....
I write data out with send(), each string after each other (in a loop).
How do I make sure that I can reconstruct the data on the receive end in the correct fashion (as I put the strings in)?
The received data can be split anywhere in the middle of the lines like so:
Packet 0: Stri
Packet 1: ng 1
Packet 2: String 2 St
Packet 3: ring 3
...
Do i have to introduce a custom END OF MESSAGE byte sequence to tell? Because EOF won't help here.
I need to be able to tell if a package is corrupted, and where the data blocks that belong together begin and end, as I sent them away beginning with S and ending with the Number! I can't use TCP because i need broadcast/multicast support.
If you want all messages to arrive, and in the same order they were sent, and to have an "end of message" indication, maybe TCP is better :-)
(TCP does this all out of the box.)
Related
My client side cannot recv the two messages if the sender sends too quickly.
sender.py
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', int(port)))
sock.listen(1)
conn, addr = sock.accept()
#conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
# sends message 1 and message 2
conn.send(pickle.dumps(message1))
#time.sleep(1)
conn.send(pickle.dumps(message2))
Where both message 1 and message 2 are pickled objects.
client.py
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((ip,int(port)))
message1 = pickle.loads(sock.recv(1024))
print(message1)
message2 = pickle.loads(sock.recv(1024))
When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.
Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages
Your code assumes that each send on the server side will match a recv on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv will already contain data from the second send which might be simply discarded by pickle.loads as junk after the pickled data. The second recv will only receive the remaining data (or just block since all data where already received) so pickle.loads will fail.
The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)) when sending and for reading first read the fixed-length size value and then read the message with the given size.
I am learning to sockets and found the word Data OR Record Boundaries in SOCK_SEQPACKET communication protocol? Can anyone explain in simple words what is Data boundary and how the SOCK_SEQPACKET is different from SOCK_STREAM & SOCK_DGRAM ?
This answer https://stackoverflow.com/a/9563694/1076479 has a good succinct explanation of message boundaries (a different name for "record boundaries").
Extending that answer to SOCK_SEQPACKET:
SOCK_STREAM provides reliable, sequenced communication of streams of data between two peers. It does not maintain message (record) boundaries, which means the application must manage its own boundaries on top of the stream provided.
SOCK_DGRAM provides unreliable transmission of datagrams. Datagrams are self-contained capsules and their boundaries are maintained. That means if you send a 20 byte buffer on peer A, peer B will receive a 20 byte message. However, they can be dropped, or received out of order, and it's up to the application to figure that out and handle it.
SOCK_SEQPACKET is a newer technology that is not yet widely used, but tries to marry the benefits of both of the above. That is, it provides reliable, sequenced communication that also transmits entire "datagrams" as a unit (and hence maintains message boundaries).
It's easiest to demonstrate the concept of message boundaries by showing what happens when they're neglected. Beginners often post client code like this here on SO (using python for convenience):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('192.168.4.122', 9000))
s.send(b'FOO') # Send string 1
s.send(b'BAR') # Send string 2
reply = s.recv(128) # Receive reply
And server code similar to this:
lsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
lsock.bind(('', 9000))
lsock.listen(5)
csock, caddr = lsock.accept()
string1 = csock.recv(128) # Receive first string
string2 = csock.recv(128) # Receive second string <== XXXXXXX
csock.send(b'Got your messages') # Send reply
They don't understand then why the server hangs on the second recv call, while the client is hung on its own recv call. That happens because both strings the client sent (may) get bundled together and received as a single unit in the first recv on the server side. That is, the message boundary between the two logical messages was not preserved, and so string1 will often contain both chunks run together: 'FOOBAR'
(Often there are other timing-related aspects to the code that influence when/whether that actually happens or not.)
I've been working in a socket tcp connection to a game server. The big problem here is that the game server send the data without any separators - since it sends the packet lenght inside the data -, making impossible to use socket:receive("*a") or "*l". The data received from the server does not have a static size and are sent in HEX format. I'm using this solution:
while true do
local rect, r, st = socket.select({_S.sockets.main, _S.sockets.bulle}, nil, 0.2)
for i, con in ipairs(rect) do
resp, err, part = con:receive(1)
if resp ~= nil then
dataRecv = dataRecv..resp
end
end
end
As you can see, I can only get all the data from the socket by reading one byte and appending it to a string, not a good way since I have two sockets to read. Is there a better way to receive data from this socket?
I don't think there is any other option; usually in a situation like this the client reads a packet of specific length to figure out how much it needs to read from the rest of the stream. Some protocols combine new line and the length; for example HTTP uses line separators for headers, with one of the headers specifying the length of the content that follows the headers.
Still, you don't need to read the stream one-by-one character as you can switch to non-blocking read and request any number of characters. If there is not enough to read, you'll get partially read content plus "timeout" signaled, which you can handle in your logic; from the documentation:
In case of error, the method returns nil followed by an error message
which can be the string 'closed' in case the connection was closed
before the transmission was completed or the string 'timeout' in case
there was a timeout during the operation. Also, after the error
message, the function returns the partial result of the transmission.
I have this code:
class SrvHandler(asyncore.dispatcher_with_send):
def handle_read(self):
data = self.recv(1024)
if data:
data = str(data, 'utf-8')
print("Received: " + data)
Now, I have a VBasic application. This application has 2 thread. This two thread works at the same time but uses same connection (static connection declared). Sometimes, this 2 threads sends data at the same time or at lease few delay between miliseconds. But the problem is, Python3 receive this 2 different data at the same time like one single line of data.
For example:
VBasic Thread 1: socket.write("Hello")
VBasic Thread 2: socket.write("12345")
Python3 Data:
Expected Result:
First recv: Hello
Second recv: 12345
Actual Result:
First recv: Hello12345
Why Python3 acts like this? What can i do to prevent this mix? What is the reason of this behavior? Is it by design? How can i prevent this? On python3 or VBasic (Client or Server)
Hope this example and code is informative enough.
This is expected behaviour for a streaming service like TCP.
TCP cannot transfer any message longer than one byte.
If you want to transfer any structure longer than one byte, you need a protocol on top of TCP - maybe put a null or something at the end of your strings and send that as well. At the receiving end, concatenate all received bytes until you find a null.
There's a thing I'm wondering about when it comes to socket programming in Perl. I'm trying to send two variabels through my socket. It works, I can send both but I want to receive them one by one. Let me show you my code and the output I get:
SERVER
my $var1 = 200;
chomp($var1);
$socket->send($var1);
my $var2 = 300;
chomp($var2);
$socket->send($var2);
CLIENT
$socket->recv(my $var1, 4000);
chomp($var1);
$socket->recv(my $var2, 4000);
chomp($var2);
print "From server: My height is: $var1 cm, weight is: $var2 kg\n";
Well, my expected output should be: From server: My height is: 400 cm, weight is: 300 cm.
Instead, my output looks like this: From server: My height is: 400300 cm, weight is:
Well, I can't see why my code is wrong. Shouldnt I be able to receive data one by one like this? How would I eventually fix this to receive the data correctly?
Short answer: Use datagram sockets or implement a communication protocol that delimits distinct messages.
You ask:
Shouldnt I be able to receive data one by one like this?
You can do that easily on datagram sockets like UDP: IO::Socket::INET->new(Type => SOCK_DGRAM, ...). There the writer/reader protocol is transmitting and receiving discrete packets of data, and send/recv will do what you want.
Stream sockets like TCP, on the other hand, have a continuous bytestream model. Here you need some sort of record separator or initial transmission of message size or similar. I suspect tis is what you are using.
Stream sockets are not unlike plain files, which appear as "bytestream" IO handles. If a file contained the string "300400", without any newlines, how would a reader of that file know that this string was actually two records instead of one (or six or five or an incomplete record, etc.)?