What is the lowest possible TCP receive window size can be announced by Linux kernel TCP/IP stack impl and how can i configure to make it announce such one? My goal is to achieve low latency and sacrificing throughput?
Right now my test client application reads about 128 bytes each second from buffer and TCP stack waits till there are free SO_RCVBUF/2 bytes space before notifying server about new window size. I'd like to make my client announce window size lower than SO_RCVBUF/2 if its possible.
I do not think that changing the receive window size will have any effect on latency.
However, since your messages are small (128 bytes), you may like to disable Nagle algorithm in the sender that makes the sender wait when sending TCP payload smaller than MSS:
Applications that expect real-time responses and low latency can react poorly with Nagle's algorithm. Applications such as networked multiplayer video games or the movement of the mouse in a remotely controlled operating system, expect that actions are sent immediately, while the algorithm purposefully delays transmission, increasing bandwidth efficiency at the expense of latency. For this reason applications with low-bandwidth time-sensitive transmissions typically use TCP_NODELAY to bypass the Nagle delay.
On the receiver you may like to disable TCP delayed acknowledgements:
The additional wait time introduced by the delayed ACK can cause further delays when interacting with certain applications and configurations. If Nagle's algorithm is being used by the sending party, data will be queued by the sender until an ACK is received. If the sender does not send enough data to fill the maximum segment size (for example, if it performs two small writes followed by a blocking read) then the transfer will pause up to the ACK delay timeout. Linux 2.4.4+ supports a TCP_QUICKACK socket option that disables delayed ACK.
C++ code:
void disableTcpNagle(int sock) {
int value = 1;
if(::setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, &value, sizeof value))
throw std::system_error(errno, std::system_category(), "setsockopt(TCP_NODELAY)");
}
void enableTcpQuickAck(int sock) {
int value = 1;
if(::setsockopt(sock, IPPROTO_TCP, TCP_QUICKACK, &value, sizeof value))
throw std::system_error(errno, std::system_category(), "setsockopt(TCP_QUICKACK)");
}
Related
I have a single server and multiple UDP clients in my setup and each sending a boolean string (true/false 5 byte max) at regular intervals of time (few milliseconds). Based on which i am determining if the device is alive or not.The number of clients are unknown before starting the program and using c++ as my programming language.
Is it possible to have a multi threading in UDP socket connection ? I came across and example for TCP where they create a new socket descriptor and spawn a thread. If its possible to write a multi threading for UDP serves please provide example/reference code.
During
while (true)
{
if ((numbytes = recvfrom(sockfd, buf, MAXBUFLEN-1 , 0,
(struct sockaddr *)&cient_addr, &client_addr_len)) == -1)
{
perror("recvfrom");
exit(1);
}
// Process buffer...
// Do something with buffer
}
In multiple client scenario if i receive message from one client its copied to local buffer of maximum buffer size. And while process this message if we receive one more message from another client where will this message store intermittently before reading to local buffer ? Does the socket file descriptor has its own buffer ?
What happens if both clients send message at the same time ? Only one is read to local buffer what will happen to another message ? will it wait for the next recvfrom ?
I read that if Maximum buffer is less that the message / packet size received then the recvfrom will only read the max buffer length and might through some error...Although all of my client only send 5 bytes maximum..If I assign Maximum bytes to 1024 bytes which is way far than expected what prices am I gonna pay ?
Thank you.
I'm currently implementing IOCP server for a game and I'm trying zero bytes recv technic.
I have 3 questions.
If you know disconnection from the client by checking if bytestransferred is 0, then how do you distinguish between receive completion and disconnection?
I'm not performing non-block mode recv() when I process actual receive process because clients send the bytes of actual data first, so I know how many bytes I'm receiving. Do I need to use non-block mode recv() still?
I'm doing it like so.
InputMemoryBitStream incomming;
std::string data;
uint32_t strLen = 0;
recv(socket, reinterpret_cast<char*>(&strLen), sizeof(uint32_t), 0);
incomming.resize(strLen);
recv(socket, reinterpret_cast<char*>(incomming.getBufferPtr()), strLen, 0);
incomming.read(data, strLen);
(InputMemoryBitStream is for reading compressed data.)
I'm dynamically allocating per io data every time I do WSARecv() and WSASend() and I free them as soon as I finish processing completed I/Os. Is it inefficient to do that? or is it acceptable? Should I reuse per io data maybe?
Thank you in advance.
After studying the "window size" concept, what I understood is that it keeps packet before sending over wire and till acknowledgement come for earliest packet . Once this gets filled up, subsequent packet will be dropped. Somewhere I also have read that TCP is a streaming protocol, and packet is what related to IP protocol at Network layer .
What I assumed till was that I have declared a Buffer (inside code) which I fill with some data and send this Buffer using socket. I declared a buffer of 10000 bytes and send it repeatedly using socket over 10 Gbps link .
I have following assumptions and questions. Please verify and help
If I want to send a packet of 64,256,512 etc. bytes, declared buffer inside code of that much space and send over socket. Each execution of send() command will send one packet of that much size .
So if I want to study the packet size variation effect on throughput, what do I have to do? Do I need to vary buffer size in code?
What are the socket buffer which we set using SO_SNDBUF and SO_RECVBUF? Google says it's buffer space for socket. Is it same as TCP window size or something different? Which parameter is more suitable to vary or to increase throughput?
Also there are three parameter in socket buffer: Min, Default and Max. Which one should I vary to my experiment and to get more relevance?
If I want to send a packet of 64,256,512 etc. bytes , Declared buffer inside code of that much space and send over socket .Each execution of send() command will send one packet of that much size.
Only if you disable the Nagle algorithm and the size is less than the path MTU. You mustn't rely on this.
So if I want to Study the Packet size variation effect on throughput, What I have to do , vary buffer space in Code?
No. Vary SO_RCVBUF at the receiver. This is the single biggest determinant of throughput, as it determines the maximum receive window.
what are the socket buffer which we set using SO_SNDBUF and SO_RCVBUF
Send buffer size at the sender, and receive buffer size at the receiver. In the kernel.
It's Same as TCP Window size
See above.
or else different ? Which parameter is more suitable to vary to increase throughput ?
See above.
Also there are three parameter in Socket Buffer min Default and Max . Which one should I vary for My experiment to get more relevance
None of them. These are the system-wide parameters. Just play with SO_SNDBUF and SO_RCVBUF for the specific sockets in your application.
TCP does not directly expose a way to control the way packets are sent since it is a stream protocol. But you can make the TCP stack send packets by disabling the Nagle algorithm. That way all data that you send will be sent out immediately instead of being buffered. Data will be split into packets of MTU size which is like ~1400 bytes. Depends on the link.
To answer (2): Disable nagling and invoke send with buffers of < 1400 bytes. Use Wireshark to make sure you got what you wanted.
The buffer settings have nothing to do with any of this. I know of no valid reason to touch them.
In general this question is probably moot since you seem to want to send a lot of data. Just leave Nagling enabled and send big buffers (such as 64KB).
I do some experience on Windows 10:
code from https://docs.python.org/3/library/socketserver.html#asynchronous-mixins,
RawCap for loopback capture,
WireShark for watching result.
The primary client code is:
def client(ip, port, message):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF, 100000)
sock.connect((ip, port))
sock.sendall(bytes(message, 'ascii'))
response = str(sock.recv(1024), 'ascii')
print("Received: {}".format(response))
Here is the result(the server port is 11111):
you can see, the tcp recive window size is the same as SO_RCVBUF, may it is platform indepent, you can verify it on other platform.
on https://msdn.microsoft.com/en-us/library/windows/hardware/ff570832(v=vs.85).aspx
The SO_RCVBUF socket option determines the size of a socket's receive buffer that is used by the underlying transport.
verified this.
Also, when I set SO_SNDBUF = 100000, it have no affects on the tcp transmission between client and server, as server just can discard data if client send much data one time.
So, if you want to change SO_RCVBUF to max Throughput, you can refer http://packetbomb.com/understanding-throughput-and-tcp-windows/, the os may offer func to detect ideal send backlog (ISB).
I have read that the combination of three things causes something like a 200ms delay with TCP: Nagle's algorithm, delayed acknowledgement, and the "write-write-read" combination. However, I cannot reproduce this delay with Java sockets and I am therefore not sure if I have understood correctly.
I am running a test on Windows 7 with Java 7 with two threads using sockets over the loopback address. I have not touched the tcpNoDelay option on any socket (false by default) nor played around with any TCP settings on the OS. The main piece of the code in the client is as below. The server is responding with a byte after each two bytes it receives from the client.
for (int i = 0; i < 100; i++) {
client.getOutputStream().write(1);
client.getOutputStream().write(2);
System.out.println(client.getInputStream().read());
}
I do not see any delay. Why not?
I believe you see delay acknowledgment.
You write 4 and 4 bytes to the socket. The server's TCP stack receives a segment (that probably contains at least 4 bytes from an int number) and wakes up the server application thread. This thread writes a byte back to the stream and this byte is sent to the client within ACK segment. I.e. TCP stack gives a chance to an application to send a reply immediately. So you see no delay.
You can write a dump of traffic and also make an experiment between two computers to see what really happens.
I starts learning TCP protocol from internet and having some experiments. After I read an article from http://www.diffen.com/difference/TCP_vs_UDP
"TCP is more reliable since it manages message acknowledgment and retransmissions in case of lost parts. Thus there is absolutely no missing data."
Then I do my experiment, I write a block of code with TCP socket:
while( ! EOF (file))
{
data = read_from(file, 5KB); //read 5KB from file
write(data, socket); //write data to socket to send
}
I think it's good because "TCP is reliable" and it "retransmissions lost parts"... But it's not good at all. A small file is OK but when it comes to about 2MB, sometimes it's OK but not always...
Now, I try another one:
while( ! EOF (file))
{
wait_for_ACK();//or sleep 5 seconds
data = read_from(file, 5KB); //read 5KB from file
write(data, socket); //write data to socket to send
}
It's good now...
All I can think of is that the 1st one fails because of:
1. buffer overflow on sender because the sending rate is slower than the writing rate of the program (the sending rate is controlled by TCP)
2. Maybe the sending rate is greater than writing rate but some packets are lost (after some retransmission, still fails and then TCP gives up...)
Any ideas?
Thanks.
TCP will ensure that you don't lose data but you should check how many bytes actually got accepted for transmission... the typical loop is
while (size > 0)
{
int sz = send(socket, bufptr, size, 0);
if (sz == -1) ... whoops, error ...
size -= sz; bufptr += sz;
}
when the send call accepts some data from your program then it's a job of the OS to get that to destination (including retransmission), but the buffer for sending may be smaller than the size you need to send, and that's why the resulting sz (number of bytes accepted for transmission) may be less than size.
It's also important to consider that sending is asynchronous, i.e. after the send function returns the data is not already at the destination, it's has been only assigned to the TCP transport system to be delivered. If you want to know when it will be received then you'll have to use other systems (e.g. a reply message from your counterpart).
You have to check write(socket) to make sure it writes what you ask.
Loop until you've sent everything or you've calculated a time out.
Do not use indefinite timeouts on socket read/write. You're asking for trouble if you do, especially on Windows.