My client side cannot recv the two messages if the sender sends too quickly.
sender.py
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', int(port)))
sock.listen(1)
conn, addr = sock.accept()
#conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
# sends message 1 and message 2
conn.send(pickle.dumps(message1))
#time.sleep(1)
conn.send(pickle.dumps(message2))
Where both message 1 and message 2 are pickled objects.
client.py
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((ip,int(port)))
message1 = pickle.loads(sock.recv(1024))
print(message1)
message2 = pickle.loads(sock.recv(1024))
When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.
Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages
Your code assumes that each send on the server side will match a recv on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv will already contain data from the second send which might be simply discarded by pickle.loads as junk after the pickled data. The second recv will only receive the remaining data (or just block since all data where already received) so pickle.loads will fail.
The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)) when sending and for reading first read the fixed-length size value and then read the message with the given size.
Related
I'm writing simple http server.
I want to shutdown socket after server send all data.
I considered that compare return byte of write() to socket with actuall content length, but I had read that the return value just means that data moved to send-buffer of the socket. (Im not sure and I don't know how can I check it)
If so, can I shutdown the socket just after check the bytes are same? What if the datas sended need to be retransmitted at TCP level after server send FIN flag?
The OS does not discard data you have written when you call shutdown(SHUT_WR). If the other end already shut down its end (you can tell because you received 0 bytes) then you should be able to close the socket, and the OS will keep it open until it has finished sending everything.
The FIN is treated like part of the data. It has to be retransmitted if the other end doesn't receive it, and it doesn't get processed until everything before it has been received. This is called "graceful shutdown" or "graceful close". This is unlike RST, which signals that the connection should be aborted immediately.
What will happen if I will establish a connection between a client and a server, and configure a different buffer size for each of them.
This is my client's code:
import socket,sys
TCP_IP = sys.argv[1]
TCP_PORT = int(sys.argv[2])
BUFFER_SIZE = 1024
MESSAGE = "World! Hello, World!"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send(MESSAGE)
data = s.recv(BUFFER_SIZE)
s.close()
print "received data:", data
Server's code:
import socket,sys
TCP_IP = '0.0.0.0'
TCP_PORT = int(sys.argv[1])
BUFFER_SIZE = 5
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
while True:
conn, addr = s.accept()
print 'New connection from:', addr
while True:
data = conn.recv(BUFFER_SIZE)
if not data: break
print "received:", data
conn.send(data.upper())
conn.close()
That means I will be limited to only 5 bytes? Which means I won't be able to receive the full packet and will lose 1024-5 packets?
I or does it mean I am able to get only packets of 5 bytes, which means that instead of receiving one packets of 1024 bytes as the client sent it, I'll have to divide 1024 by 5 and get 204.8 packets (?) which sounds not possible.
What in general is happing in that code?
Thanks.
Your arguments are based on the assumption that a single send should match a single recv. But this is not the case. TCP is a byte stream and not a message based protocol. This means all what matters are the transferred bytes. And for this is does not matter if it does not matter if one or 10 recv are needed to read 50 bytes.
Apart from that send is not guaranteed to send the full buffer either. It might only send parts of the buffer, i.e. the sender need actually check the return code to find out how much of the given buffer was actually send now and how much need to be retried for sending later.
And note that the underlying "packet" is again a different thing. If there is a send for 2000 bytes it will usually need multiple packets to be send (depending on the maximum transfer unit of the underlying data link layer). But this does not mean that one also need multiple recv. If all the 2000 bytes are already transferred to the OS level receive buffer at the recipient then they can be also be read at once, even if they traveled in multiple packets.
Your socket won't lose the remaining 1024 - 5 (1019) bytes.it just stored on the socket and ready to read again! so , all you need to do is to read from the socket again. the size of buffer you want to read to is decided by yourself. and you are not limited to 5 bytes, you are just limiting the read buffer for each single read to 5 bytes. so for 1024 bytes to read you have to read for 204 times plus another time read which would be the last one. but remember that the last time read fills your last buffer index with null. and that means there is no more bytes available for now.
For c send function(blocking way) it's specified what function returns with size of sent bytes when it's received on destinations. I'm not sure that I understand all nuances, also after writing "demo" app with WSAIoctl and WSARecv on server side.
When send returns with less bytes number than asked in buffer-length parameter?
What is considered as "received on destinations"? My first guess it's when it sit on server's OS buffer and server application is notified. My second one it's when server application recv call have read it fully?
Unless you are using a (somewhat exotic) library, a send on a socket will return the number of bytes passed to the TCP buffer successfully, not the number of bytes received by the peer (see Microsoft´s docs for example).
When you are streaming data via a socket, you need to check the bytes effectively accepted into the TCP send buffer. That´s why usually a send command is inside a loop that will issue several sends if needed.
Errors in send are local: for example if the socket is closed by the peer during a sending operation (making your socket invalid) or if the operation times out (TCP buffer not emptying, i. e. peer not receiving data fast enough or some other trouble).
After all send is completed you have no easy way of knowing if the peer received all the bytes you sent. You´ll usually just issue closesocket and make sure that your socket has a proper linger option set (i. e. only close after timeout or sucessfully finishing the send). Alternatively you wait for a confirmation by the peer (for example via a recv that returns zero bytes, indicating that the connection was gracefully closed).
Edit: typo
Look at this small basic python programs:
import socket
tcpsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcpsock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
tcpsock.bind(("", 10000))
tcpsock.listen(10)
(sock, (ip, port)) = tcpsock.accept()
s = sock.recv(1024)
print(s)
Second program:
import socket
import time
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('localhost', 10000))
time.sleep(1)
sock.sendall(b'hello world')
The first program is a socket server. It recv a message through the socket and display it on the console. The second program is a client which connects to the server and sends it a message.
As you can see, the server reads a 1024 bytes max length message. My client send a few bytes.
My question is: How does the server knows the message ends after the 'd' char ?
I am working with sockets since years and i have always implemented a delimiter mechanism in order to know when the message stops.
But it seems to work automaticly. My question is: How ?
I know TCP car fragment messages. So what's happen if the paquet is trucated in the middle of my message ? Is it managed by OS ?
Thanks
How does the server knows the message ends after the 'd' char ?
It does not. There is not even a concept of a message in TCP. recv simply returns what is there: it blocks if no data are available and returns what can be read up to the given size if data are available. "Data available" means that there are data in the sockets receive buffer, which are put by the OS kernel there. In other words: recv will not block until the requested number of bytes can be returned but it will already return when at least a single byte is in the sockets receive buffer.
For example if the client would do two send or sendall shortly after each other a single recv might return both "messages" together. This can be easily triggered by deferring the recv (add some sleep before it) so that both "messages" are guaranteed to be arrived at the client.
I have to call the function repeatedly to get all data, given that the len argument is set to 10240. But this results in blocking at last. How can I get all the data and safely return in a platform independent way?
BTW, I use netcat at the sender side:
cat ocr_pi.png | nc -u server 5555
Is this issue relative to nc's behavior? I didn't find any parameter to set UDP packet size(-O is for TCP).
Thanks.
UDP sends and receives data as messages. In the len argument, you tell recvfrom() the max message size you can receive, and then recvfrom() waits until a full message arrives, regardless of its size. UDP messages are self-contained. Unlike TCP, a UDP message cannot be partially sent/received. It is an all-or-nothing thing. If the size of the received message is greater than the len value you specify, the message is discarded and you get an error.
The only way recvfrom() blocks is if there is no message available to read. If you don't want to block, use select() (or pselect() or epoll or other platform equivalent) to specify a timeout to wait for a message to arrive, and then call recvfrom() only if there is actually something to read.