I have UDP implementation with facility to get back acknowledge from server. The client re-sends packets for which acknowledgement is not received from server with in a specified time. Clients send around 10 packets while waiting for acknowledgement from server for 1st packet. It then repeats sending packets for which acknowledgement is not received. This works fine in normal scenario with minor delay in network.
The real issue is being experienced on a low bandwidth connection where round trip delay is a bit significant. Clients keeps on adding packets in send queue based on acknowledgement timeouts. This results into many duplicate packets getting added to queue.
Tried to find any elegant solution to avoid duplicate packets in send queue with no luck. Any help will be appreciated.
If I can get a way to mark/set a property of a packet such that if packet is not send within NN ms then it will be removed from queue then I can build algorithm around it.
UDP has no builtin duplicate detection as is the case with TCP. This means any kind of such detection has to be done by the application itself. Since the only way an application can interact with the send queue is to send datagrams any kind of duplicate detection on the sender side has to be done before the packet gets put into the send queue.
How you figure out at this stage if this is really a duplicate packet to a previous one which should not be sent or if this a duplicate packet which should be sent because the original one got lost is fully up to the application. And any "...not send within NN ms..." has to be implemented in the application too with timers or similar. You might additionally try to get more control of the queue by reducing the size of the send queue with SO_SNDBUF.
Related
I came across this term "Outstanding frames" while studying about packet transmission between client and server, but I have doubts around this term.
There is an older question on the same topic but it doesn't seem to have a definitive answer in it: what is an 'outstanding' frame?
I got 2 possible definitions of what it can be:
Packets which are queued up for transmission but not yet sent and
packets sent to client but Acknowledgement has not been received yet for the packets.
So which is the correct meaning for the term?
The second definition is more accurate: outstanding frames refer to packets that have been sent to the client, but for which an acknowledgement has not yet been received by the server. This means that the server is still waiting for confirmation that the packets were successfully received by the client. In contrast, packets that are queued up for transmission but not yet sent are referred to as "pending frames."
I'm having issues understanding what socket types are negatively impacted in the event that TCP must try retransmmitting messages.
We have a distributed system that uses a combination of inprocess and TCP connections for internal processes and external devices and applications. My concern is that in the event there is a significant traffic that causes latency and dropped packets, that a TCP retransmit will cause delay in the system.
What I'd like to avoid is an application that has messages compile in a queue waiting to be sent (via a single ZeroMQ TCP socket) because TCP is forcing the socket to repeatedly retransmit messages that never sent an acknowledge.
Is this an issue that can happen using ZeroMQ? Currently I am using PUSH/PULL on a Linux OS.
Or is this not a concern, and if not, why?
It is crucial that messages from the external devices/applications do not feed stale data.
First, the only transport where retransmits are possible is TCP over an actual physical network. And then likely not on a LAN, as it's unlikely that Ethernet packets will go missing on a LAN.
TCP internal to a computer, and especially IPC, INPROC, etc will all have guaranteed delivery of data first time, every time. There is no retransmit mechanism.
If one of the transports being used by socket does experience delays due to transmission errors, that will slow things up. ZMQ cannot consider a message "sent" until it's been propagated via all the transports used by the socket. The external visibility of "sent" is that the outbound message queue has moved away from the high water mark by 1.
It's possible that any one single message will arrive sooner over IPC than TCP, and possible that message 2 will arrive over IPC before message 1 has arrived via TCP. But if you're relying on message timing / relative order, you shouldn't be using ZMQ in the first place; it's Actor model, not CSP.
EDIT For Frank
The difference between Actor and CSP is that the former is asynchronous, the latter is synchronous. Thus for Actor model, the sender has zero information as to when the receiver actually gets a message. For CSP, the sending / receiving is an execution rendevous - the send completes only when the receive is complete.
This can be remarkably useful. If in your system it makes no sense for A to instruct C to do something before (in time, not just in A's code flow) instructing B, then you can do that with CSP (but not Actor model). That's because when A sends to B, B receives the message before A's send completes, freeing A to then send to C.
Unsurprisingly it's real time systems that benefit from CSP.
So consider ZMQ's Actor model with a mix of TCP, IPC and INPROC transports in ZMQ. There's a good chance that messages send via TCP will arrive a good deal later than messages sent through INPROC, even if they were sent first.
I'm writing a game server for a turn-based game. One criteria is that the game needs to be as fair for all players as possible.
So far it works like this:
Each client has a TCP connection. (If relevant, the connection is opened via WebSockets)
While running, continually check for incoming socket messages via epoll.
Iterate through clients with sockets ready to read:
Read all messages from the client.
Update the internal game state for each message.
Queue outgoing messages to affected clients.
At the end of each "window" (turn):
Iterate through clients and write all queued outgoing messages to their sockets
My concern for fairness raises the following questions:
Does it matter in which order I send messages to the clients?
Calling write() on all the sockets takes only a fraction of a second for my program, but somewhere in the underlying OS or networking would it make a difference if I sorted the client list?
Perhaps I should be sending to the highest-latency clients first?
Does it matter how I write the outgoing messages to the sockets?
Currently I'm writing them as one large chunk. The size can exceed a single packet.
Would it be faster for the client to begin its processing if I sent messages in smaller chunks than 1 packet?
Would it be better to write 1 packet worth to each client at a time, and iterate over the clients multiple times?
Are there any linux/networking configurations that would bear impact here?
Thanks in advance for your feedback and tips.
Does it matter in which order I send messages to the clients?
Yes, by fractions of milliseconds. If the network interface is available for sending the OS will immediately start sending. Why would it wait?
Perhaps I should be sending to the highest-latency clients first?
I think you should be sending in random order. Shuffle the list prior to sending. This makes it fair. I think your question is valid and this should be addressed.
Currently I'm writing them as one large chunk. [...]
First, realize that TCP is stream-based and that there are no packets/messages at the protocol level. On a physical level data is indeed packetized.
It is not necessary to manually split off packets because clients will read data as it arrives anyway. If a client issues a read, that read will complete immediately once the first packet has arrived. There is no artificial waiting in the OS.
Are there any linux/networking configurations that would bear impact here?
I don't know. Be sure to disable nagling.
We are using recv(2) in order to listen to incoming ICMP packets in a network management application in which there is a status poll feature to ping all managed hosts.
In a productive environment, the number of managed hosts can become pretty large (1,000+), and at the moment, when said status poll is performed, we send out all the ping requests sequentially with no delay inbetween.
Obviously, this will lead to many ping replies being sent back almost simultaneously, and it appears that our dispatching routine cannot catch up. This seems to cause that packets are dropped and the ping replies are never actually received by our application. We believe so because many hosts are falsely detected as being "unavailable".
The dispatcher doesn't do anything more than adding incoming replies to an "inbox" of sorts, which is processed later by another thread, ie it does not take much time and can probably not be optimized any further.
Is it possible that the internal packet buffer used by recv (in the kernel? in the network hardware?) fills up and starts dropping packets? If so, is there a good way to determine a reasonable maximum amount of simultaneous pings that can be performed safely (ie by getting that buffer size from the OS)?
While sending packets across a network, how can one determine where one packet ends and where another starts?
Is sending/receiving acknowledgment one of the ways of doing so?
TCP is a stream-based protocol. That is, it provides a stream vs. packet or message-based interface to the application. If using TCP, an application must implement its own method of determining packets or messages. For example, (a) all message are a fixed size, or (b) each message is prefixed with its subsequent size, or (c) there is a special "end-of-record" sequence in the data stream to indicate a message boundary. Search google for lots of information on how one can implement message boundaries in TCP.
I assume here that you mean application-level 'packets'.
If you use UDP, you don't need to since it's a message protocol. TCP is a byte streaming protocol, so it cannot send packets, just bytes. If you need to send anything more complex than a byte-stream across TCP, you have to add another protocol on top - HTTP is one such protocol. Text is fairly easy since lines have terminating characters, usually CR/LF/CRLF. Sending non-text messages will require a different protocol.
One approach that is often used with TCP is to connect, stream a protocol-unit, disconnect. This works OK, but slowly because of the huge latency of continually opening and closing TCP connections. HTTP usually works like this in order to serve up web pages to large numbers of users who, if left permanently connected while they viewed pages, would needlessly use up all the server sockets.
Waiting for an application-level ACK from the peer is sometimes necessary if it absolutely essential that peer receipt is known before the next message is sent, but again, this is slow because of the connection latency. TCP was not designed with this approach in mind.
If the commonly available IP protocols cannot directly provide what you need, you will have to resort to implementing your own.
What sort of 'packet' are you sending?
Rgds,
Martin
With TCP sockets, you just see the datastream where you can receive and send bytes. You have no way of knowing where a packet ends and another begins.
This is a feature (and a problem) of TCP. Most people just read data into a buffer until a linefeed (\n) is seen. Then process the data and wait for the next line. If transferring chunks of binary data, one can first inform the receiver of how many bytes of data are coming.
If packet boundaries are important, you could use UDP but then the packet order might change or some packets might be lost on the way without you knowing.
The newer SCTP protocol behaves much like TCP (lost packets are resend, packet ordering is retained) but with SCTP sockets you can send packets so that receiver gets exactly the same packet.