I am working on a project wherein I have multiple push sockets( created dynamically ) connected to a single pull socket. Here, each push socket sends a unique kind of data to the pull socket and the data size for each messages is less than 1000Bytes and each push socket sends upto 20,000 to 30,0000 such messages. During the peak load the number of push sockets can scale upto 100. So here, there are 100 push sockets connecting to one PULL receiver and each push socket sending 20-30K of messages with size almost 1000Bytes.
Now, On the PULL side receiver I process these each unique packets and put it into the respective queue, For Ex: If I have 50 PUSH sockets sending in data than on my receiver side there will be 50 queues processing the data. This is required as each queue holds unique set of data which needs to be processed uniquely. Here, I cannot have multiple PULL receivers as the data would then be routed in a round-robin fashion and one set of unique data might go into another one which I don't need.
Here, the problem I am facing is sometimes I see I don't receive all the data sent by the PUSH sockets. I believe there is some data loss that I happening. In order to control this I have setup HWM on both the sides. I tried the below combinations:
1) SNDHWM set to 100 and RCVHWM set to 100
2) SNDHWM set to 10 and RCVHWM set to 10
3) SNDHWM set to 10 and RCVHWM set to 100
Apart from the above configs I have also set the context threads on sender side to be 3 and on the receiver side to be 1. Here, on the sender side there is only one context created from which multiple push sockets are created dynamically as in when the data receives.
But, in all these combinations I find still some packets are missing.
Can someone guide me as in which parameters I need to configure in ZMQ so that I don't end up in packet loss and all the packets I receive on my receiver side. Also, I somehow believe my receiver side processing might be slow due to which this could have happened.
Please pen your thoughts here and guide me if possible.
Related
I saw the socket option TCP_NODELAY, which is used to turn on or off the Nagle alorithm.
I checked what the Nagle algorithm is, and it seems similar to 'stop and wait'.
Can someone give me a clear difference between these two concepts?
In a stop and wait protocol, one
sends a message to the peer
waits for an ack for that message
sends the next message
(i.e. one cannot send a new message until the previous one has been acknowledged)
Nagle's algorithem as used in TCP is orthoginal to this concept. When the TCP application sends some data, the protocol buffers the data and waits a little while to see if there's more data to be sent instead of sending data to the peer immediately.
If the application has more data to send in this small timeframe, the protocol stack merges that data into the current buffer and can send it as one large message.
This concept could very well be applied to a stop and go protocol as well. (Note that TCP is not a stop and wait protocol)
The Nagle Algorithm is used to control whether the socket provider sends outgoing data immediately as-is at the cost of less efficient network transmissions (off), or if it buffers outgoing data so it can make more efficient network transmissions at the cost of speed (on).
Stop and Wait is a mechanism used to ensure the integrity of transmitted data, by making the sender send a frame of data and then wait for an acknowledgement from the receiver before sending another frame, thus ensuring frames are received in the same order in which they are sent.
These two features operate independently of each other.
I have a ZMQ_PUB socket sending messages out at ~50Hz. One destination needs to react to each message, so it has a standard ZMQ_SUB socket with a while(true) loop checking for new messages. A second destination should only react once a second to the "most recent" message. That is, my second destination needs to subsample.
For the second destination, I believe I'd want to have a time-based loop that is called at my desired rate (1Hz) and recv() the latest message, dropping the rest. I believe this is done via a ZMQ_HWM on the subscriber. Is there another option that needs to be set somewhere?
Do I need to worry about the different subscribers having different HWMs? Will the publisher become angry? It's a shame ZMQ_RATE only applies to multicast sockets.
Is there a best way to accomplish what I'm attempting?
zmq v3.2.4
The HighWaterMark will not be a fantastic solution for your problem. Setting it on the subscriber to, let's say, 10 and reading 1 message per second, will just give you the old messages first, slowly, and throw away all the new, because it's limit are reached.
You could either use a topic on you publisher that makes you able to filter out every 50th message like making the topic messageCount % 50 and subscribe to 0.
Otherwise maybe you shouldn't use zmq's pub/sub, but instead do you own look alike with router/dealer that allows you to subscribe to sampled messages.
Lastly you could also just send them all. 50 m/s is hardly anything in zmq (if they aren't heavy on data, like megs) and then only use every 50th message.
Just assume we are sending a packet with TCP and we found that packet has been dropped by network. After expiry of the time we try to resend the packet. In the meanwhile we got a new segment from application layer, and we are now trying to send both segments, the older one and new one with the sequence number which has not been acknowledged. Now the packet size is greater than older size. Just assume the older packet was delivered successfully but its acknowledgement was lost.
I explain this by some steps:-
1. (from sender) packet[SEQ=100,SEG_LEN=3,SEG="ABC"]----(To receiver)--->Receiver got it
2. (from receiver) packet[ACK=121]-----(To sender)---->Packet lost(Sender couldn't receive it)
3. We got new segment from application SEG="XYZ" and time expired for the previous packet
4. (from sender) packet[SEQ=100,SEQ_LEN=6,SEG="ABCXYZ"]----(To receiver)--->Receiver got it
So, now I want to know that what will happen at the receiver side,
Will it drop the packet by just assuming duplicate? or
It will accept the extra("XYZ") segment or total("ABCXYZ") segment.(?)
After expiry of the time we try to resend the packet.
Not necessarily, see below.
In the meanwhile we got a new segment from application layer, and we are now trying to send both segments, the older one and new one with the sequence number which has not been acknowledged.
Not necessarily. It is entirely possible that both segments are now coalesced and that you're only trying to send one segment at this point.
Now the packet size is greater than older size.
The segment size may be older than the old size, if what I said above is true. If it isn't, the sizes of the two packets you originally talked about haven't changed at all.
Just assume the older packet was delivered successfully but its acknowledgement was lost. So, now I want to know that what will happen at the receiver side
Will it drop the packet by just assuming duplicate?
It will drop the original packet if it was resent according to your postulate. If it was coalesced according to my postulate it will accept the new part of the packet, from the currently accepted point onwards.
It will accept the extra segment or total segment.
This question is too confused to answer. You need to make up your mind whether you're talking about two packets or a single coalesced packet. You're talking about both at the same time.
So, now I want to know that what will happen at the receiver side,
Will it drop the packet by just assuming duplicate?
No.
It will accept the extra("XYZ") segment
Yes.
or total("ABCXYZ") segment.(?)
No.
You need to think about this. TCP/IP works. Seriously. For 25 years at least. If any of the alternative scenarios you've posted were true it wouldn't be usable.
I am currently writing code to transfer data to a remote vendor. The transfer will take place over a TCP socket. The problem I have is the data is variable length and there are no framing or size markers. Sending the data is no problem, but I am unsure of the best way to handle the returned data.
The data is comprised of distinct "messages" but they do not have a fixed size. Each message has an 8 or 16 byte bitmap that indicates what components are included in this message. Some components are fixed length and some are variable. Each variable length component has a size prefix for that portion of the overall message.
When I first open the socket I will send over messages and each one should receive a response. When I begin reading data I should be at the start of a message. I will need to interpret the bitmap to know what message fields are included. As the data arrives I will have to validate that each field indicated by the bitmap is present and of the correct size.
Once I have read all of the first message, the next one starts. My concern is if the transmission gets cut partway through a message, how can I recover and correctly find the next message start?
I will have to simulate a connection failure and my code needs to automatically retry a set number of times before canceling that message.
I have no control over the code on the remote end and cannot get framing bytes or size prefixes added to the messages.
Best practices, design patterns, or ideas on the best way to handle this are all welcomed.
From a user's point of view, TCP is a stream of data, just like you might receive over a serial port. There are no packets and no markers.
A non-blocking read/recv call will return you what has currently arrived at which point you can parse that. If, while parsing, you run out of data before reaching the end of the message, read/recv more data and continue parsing. Rinse. Repeat. Note that you could get more bytes than needed for a specific message if another has followed on its heels.
A TCP stream will not lose or re-order bytes. A message will not get truncated unless the connection gets broken or the sender has a bug (e.g. was only able to write/send part and then never tried to write/send the rest). You cannot continue a TCP stream that is broken. You can only open a new one and start fresh.
A TCP stream cannot be "cut" mid-message and then resumed.
If there is a short enough break in transmission then the O/S at each end will cope, and packets retransmitted as necessary, but that is invisible to the end user application - as far as it's concerned the stream is contiguous.
If the TCP connection does drop completely, both ends will have to re-open the connection. At that point, the transmitting system ought to start over at a new message boundary.
For something like this you would probably have a lot easier of a time using a networking framework (like netty), or a different IO mechansim entirely, like Iteratee IO with Play 2.0.
I know that it is possible that multiple packets would be stacked to the buffer to be read from and that a long packet might require a loop of multiple send attempts to be fully sent. But I have a question about packaging in these cases:
If I call recv (or any alternative (low-level) function) when there are multiple packets awaiting to be read, would it return them all stacked into my buffer or only one of them (or part of the first one if my buffer is insufficient)?
If I send a long packet which requires multiple iterations to be sent fully, does it count as a single packet or multiple packets? It's basically a question whether it marks that the package sent is not full?
These questions came to my mind when I thought about web sockets packaging. Special characters are used to mark the beginning and end of a packet which sorta leads to a conclusion that it's not possible to separate multiple packages.
P.S. All the questions are about TCP/IP but you are welcomed to share information (answers) about UDP as well.
TCP sockets are stream based. The order is guaranteed but the number of bytes you receive with each recv/read could be any chunk of the pending bytes from the sender. You can layer a message based transport on top of TCP by adding framing information to indicate the way that the payload should be chunked into messages. This is what WebSockets does. Each WebSocket message/frame starts with at least 2 bytes of header information which contains the length of the payload to follow. This allows the receiver to wait for and re-assemble complete messages.
For example, libraries/interfaces that implement the standard Websocket API or a similar API (such as a browser), the onmessage event will fire once for each message received and the data attribute of the event will contain the entire message.
Note that in the older Hixie version of WebSockets, each frame was started with '\x00' and terminated with '\xff'. The current standardized IETF 6455 (HyBi) version of the protocol uses the header information that contains the length which allows much easier processing of the frames (but note that both the old and new are still message based and have basically the same API).
TCP connection provides for stream of bytes, so treat it as such. No application message boundaries are preserved - one send can correspond to multiple receives and the other way around. You need loops on both sides.
UDP, on the other hand, is datagram (i.e. message) based. Here one read will always dequeue single datagram (unless you mess with low-level flags on the socket). Event if your application buffer is smaller then the pending datagram and you read only a part of it, the rest of it is lost. The way around it is to limit the size of datagrams you send to something bellow the normal MTU of 1500 (less IP and UDP headers, so actually 1472).