I have two applications communicating via TCP sockets. First one receives and the second sends.
First app:
start=clock();
recv();
end=clock();
when i run the application, (end-start) is 150-200 msecs.(always)
Second app:
while (!stop) {
start=clock();
prepare_message();
send();
end=clock();
}
when i run the application, (end-start) is 0.00 msecs. (always)
Message payload is nearly 200-300 bytes and the ping duration is <1ms. So, why does the receiver wait for 200ms while the sender does not wait?
So how can i describe the 200msecs?
Thanks
Sender is sending the message whenever it is ready. The receiver has to wait for the message, and this is where this extra time may come from. How do you ensure that recv() is called after the message has been send? If you don't, then recv() is most likely waiting for input, while the sender hasn't yet reach this part of the code.
Another thing is that depending on the methods you are using, the sender only saves the message in a buffer, as TCP can wait for more data to combine it in a single package. You should use TCP_NODELAY option to avoid that.
Related
For c send function(blocking way) it's specified what function returns with size of sent bytes when it's received on destinations. I'm not sure that I understand all nuances, also after writing "demo" app with WSAIoctl and WSARecv on server side.
When send returns with less bytes number than asked in buffer-length parameter?
What is considered as "received on destinations"? My first guess it's when it sit on server's OS buffer and server application is notified. My second one it's when server application recv call have read it fully?
Unless you are using a (somewhat exotic) library, a send on a socket will return the number of bytes passed to the TCP buffer successfully, not the number of bytes received by the peer (see Microsoft´s docs for example).
When you are streaming data via a socket, you need to check the bytes effectively accepted into the TCP send buffer. That´s why usually a send command is inside a loop that will issue several sends if needed.
Errors in send are local: for example if the socket is closed by the peer during a sending operation (making your socket invalid) or if the operation times out (TCP buffer not emptying, i. e. peer not receiving data fast enough or some other trouble).
After all send is completed you have no easy way of knowing if the peer received all the bytes you sent. You´ll usually just issue closesocket and make sure that your socket has a proper linger option set (i. e. only close after timeout or sucessfully finishing the send). Alternatively you wait for a confirmation by the peer (for example via a recv that returns zero bytes, indicating that the connection was gracefully closed).
Edit: typo
My program is like the following
sender side:
while(1){
sendto(); // send some data packets
usleep(); // delay to avoid overflow
}
receiver side:
while(1){
recvfrom(); // receive
}
I could like to have a tcp socket appeared in receiver while loop, which send a control message (to change to time of usleep) from receiver side to sender side for every 2 seconds( or for every 100 received packets). In the mean while, the sender should also be able to listen this control message in while loop. Can it be achieved without a multi-thread program?
I am make a application base on lwip,the applcation just send data to the server;
When my app works for some times (about 5 hours),I found that the send thread hung in send() function,and after about 30min send() return 0,and my thread run agin;
In the server side ,have make a keepalive,its time is 5min,when my app hungs,5min later the server close the sockect,but my app have not get this,still hungs in send() until 30min get 0 return; why this happen?
1: the up speed is not enough to send data,it will hungs in send?
2: maybe the server have not read data on time,and it make send buff full and hungs?
how can i avoid these peoblems in my code ? I have try to set TCP_NODELAY,SO_SNDTIMEO and select before send,but also have this problem.
send() blocks when the receiver is too far behind the sender. recv() returns zero when the peer has closed the connection, which means you must close the socket and stop reading.
I have to call the function repeatedly to get all data, given that the len argument is set to 10240. But this results in blocking at last. How can I get all the data and safely return in a platform independent way?
BTW, I use netcat at the sender side:
cat ocr_pi.png | nc -u server 5555
Is this issue relative to nc's behavior? I didn't find any parameter to set UDP packet size(-O is for TCP).
Thanks.
UDP sends and receives data as messages. In the len argument, you tell recvfrom() the max message size you can receive, and then recvfrom() waits until a full message arrives, regardless of its size. UDP messages are self-contained. Unlike TCP, a UDP message cannot be partially sent/received. It is an all-or-nothing thing. If the size of the received message is greater than the len value you specify, the message is discarded and you get an error.
The only way recvfrom() blocks is if there is no message available to read. If you don't want to block, use select() (or pselect() or epoll or other platform equivalent) to specify a timeout to wait for a message to arrive, and then call recvfrom() only if there is actually something to read.
I am confused as to what poller actually does in zmq. The zguide goes into it minimally, and only describes it as a way to read from multiple sockets. This is not a satisfying answer for me because it does not explain how to have timeout sockets. I know zeromq: how to prevent infinite wait? explains for push/pull, but not req/rep patterns, which is what I want to know how to use.
What I am attempting to ask is: How does poller work, and how does its function apply to keeping track of sockets and their requests?
When you need to listen on different sockets in the same thread, use a poller:
ZMQ.Socket subscriber = ctx.socket(ZMQ.SUB)
ZMQ.Socket puller = ctx.socket(ZMQ.PULL)
Register sockets with poller (POLLIN listens for incoming messages)
ZMQ.Poller poller = ZMQ.Poller(2)
poller.register(subscriber, ZMQ.Poller.POLLIN)
poller.register(puller, ZMQ.Poller.POLLIN)
When polling, use a loop:
while( notInterrupted()){
poller.poll()
//subscriber registered at index '0'
if( poller.pollin(0))
subscriber.recv(ZMQ.DONTWAIT)
//puller registered at index '1'
if( poller.pollin(1))
puller.recv( ZMQ.DONTWAIT)
}
Choose how you want to poll...
poller.poll() blocks until there's data on either socket.
poller.poll(1000) blocks for 1s, then times out.
The poller notifies when there's data (messages) available on the sockets; it's your job to read it.
When reading, do it without blocking: socket.recv( ZMQ.DONTWAIT). Even though poller.pollin(0) checks if there's data to be read, you want to avoid any blocking calls inside the polling loop, otherwise, you could end up blocking the poller due to 'stuck' socket.
So, if two separate messages are sent to subscriber, you have to invoke subscriber.recv() twice in order to clear the poller, otherwise, if you call subscriber.recv() once, the poller will keep telling you there's another message to be read. So, in essence, the poller tracks the availability and number of messages, not the actual messages.
You should run through the polling examples and play with the code, it's the best way to learn.
Does that answer your question?
In this answer i listed
Details from the documentation http://api.zeromq.org/4-1:zmq-poll
Also i added some important explanation and things that clear confusion for new commers! If you are in a hurry! You may like to start by What the poller do and What about Recieving and A note about recieving and what about one socket only sections at the end! Starting from important notes section! Where i clear things in depth! I still suggest reading well the details in the doc ref! And first section!
Doc ref and notes
Listen to multiple sockets and events
The zmq_poll() function provides a mechanism for applications to multiplex input/output events in a level-triggered fashion over a set of sockets. Each member of the array pointed to by the items argument is a zmq_pollitem_t structure. The nitems argument specifies the number of items in the items array. The zmq_pollitem_t structure is defined as follows:
typedef struct
{
void //*socket//;
int //fd//;
short //events//;
short //revents//;
} zmq_pollitem_t;
zmq Socket or standard socket through fd
For each zmq_pollitem_t item, zmq_poll() shall examine either the ØMQ socket referenced by socket or the standard socket specified by the file descriptor fd, for the event(s) specified in events. If both socket and fd are set in a single zmq_pollitem_t, the ØMQ socket referenced by socket shall take precedence and the value of fd shall be ignored.
Big note (same context):
All ØMQ sockets passed to the zmq_poll() function must share the same ØMQ context and must belong to the thread calling zmq_poll().
Revents member
For each zmq_pollitem_t item, zmq_poll() shall first clear the revents member, and then indicate any requested events that have occurred by setting the bit corresponding to the event condition in the revents member.
Upon successful completion, the zmq_poll() function shall return the number of zmq_pollitem_t structures with events signaled in revents or 0 if no events have been signaled.
Awaiting for events and blocking
If none of the requested events have occurred on any zmq_pollitem_t item, zmq_poll() shall wait timeout microseconds for an event to occur on any of the requested items. If the value of timeout is 0, zmq_poll() shall return immediately. If the value of timeout is -1, zmq_poll() shall block indefinitely until a requested event has occurred on at least one zmq_pollitem_t. The resolution of timeout is 1 millisecond.
0 => doesn't wait
-1 => block
+val => block and wait for the timeout amount
Events
The events and revents members of zmq_pollitem_t are bit masks constructed by OR'ing a combination of the following event flags:
ZMQ_POLLIN
For ØMQ sockets, at least one message may be received from the socket without blocking. For standard sockets this is equivalent to the POLLIN flag of the poll() system call and generally means that at least one byte of data may be read from fd without blocking.
ZMQ_POLLOUT
For ØMQ sockets, at least one message may be sent to the socket without blocking. For standard sockets this is equivalent to the POLLOUT flag of the poll() system call and generally means that at least one byte of data may be written to fd without blocking.
ZMQ_POLLERR
For standard sockets, this flag is passed through zmq_poll() to the underlying poll() system call and generally means that some sort of error condition is present on the socket specified by fd. For ØMQ sockets this flag has no effect if set in events, and shall never be returned in revents by zmq_poll().
Note:
The zmq_poll() function may be implemented or emulated using operating system interfaces other than poll(), and as such may be subject to the limits of those interfaces in ways not defined in this documentation.
Return value
Upon successful completion, the zmq_poll() function shall return the number of zmq_pollitem_t structures with events signaled in revents or 0 if no events have been signaled. Upon failure, zmq_poll() shall return -1 and set errno to one of the values defined below.
Example
Polling indefinitely for input events on both a 0mq socket and a standard socket.
zmq_pollitem_t items [2];
/* First item refers to ØMQ socket 'socket' */
items[0].socket = socket;
items[0].events = ZMQ_POLLIN;
/* Second item refers to standard socket 'fd' */
items[1].socket = NULL;
items[1].fd = fd;
items[1].events = ZMQ_POLLIN;
/* Poll for events indefinitely */
int rc = zmq_poll (items, 2, -1);
assert (rc >= 0); /* Returned events will be stored in items[].revents */
Important notes
What the poller do and What about Recieving
The poller only check and await for when events occure!
POLLIN is for receiving! Data is there for recieving!
Then we should read through recv()! We are responsible to read or do anything! The poller is just there to listen to the events and await for them! And through zmq_pollitem_t we can listen to multiple events! If any event happen! Then the poller unblock! We can check then the event in the recv! and zmq_pollitem_t! Note that the poller queue the events as they trigger! And next call will pick from the queue! The order because of that is also kept! And successive calls will return the next event and so on! As they came in!
A note about recieving and what about one socket only
For a Router! A one router can receive multiple requests even from a one client! And also from multiple clients at once! In a setup where multiple clients are of same nature! And are the ones connecting to the router! A question that can cross the mind of a new commer is! Do i need a poller for this async nature! The anwser is no! No need for a poller and listening for different sockets!
The big note is: Receiving calls (zmq_recv(), socket.recv() some lang binding)! Block! And are the way to read! When messages come! They are queued! The poller have nothing to do with this! The poller only listen to events from different sockets! And unblock if any of them happen! if the timeout is reached then no event happen! And does no more then that!
The nature of receiving is straight forward! The recieve call blocks! Till a message in the message queue comes! When multiple come they will get queued! Then on each next call for recv()! We will pull the next message! Or frame! (Depending on what recieving method we are using! And the api level! And abstraction from the binding library to the low level one!)
Because we can access too the messages by frame! a frame at each call!
But then here it becomes clear!
Recieve calls are the things to recieve! They block till a message enter the queue! Multiple parallel messages! Will enter the queue as they come! Then for each call! Either the queue is filled or not! consume it! or wait!
That's a very important thing to know! And which can confuse new commers!
The poller is only needed when there is multiple sockets! And they are always sockets that we declare on the process code in question (bind them, or to connect to something)! Because if not! How will you recieve the messages! You can't do it well! because you have to prioritize one or another! in a loop having one recv() go first ! Which will block! Which even if the other socket get a message in it's queue! The loop is blocked and can't proceed to the next recv()! hince the poller give us the beauty to be able to tackle this! And work well with multiple sockets!
while(true) {
socket1.recv() // this will block
socket2.recv() // this will have to wait till the first recieve! Even if messages come in in it's queue
With the poller:
While(true) {
zmq_poll() // block till one of the socket events happen! If the event was POLLIN!
// If any socket get a message to it's queue
// This will unblock
// then we check which type and which socket was
if (condition socket 1) {
// treat socket 1 request
}
if (condition socket 2) {
// treat socket 2 request
}
// ...
}
You can see real code from the doc at this section (scroll enough to see the code blocks, you can see in all different langs too)
Know that the poller just notify that there is messages in! If it's POLLIN!
In each iteration! The poller if many events triggered already! Let's give the example of 10 messages recieved 5 in each socket! Here the poller already have the events queued! And in each next call for the 9 times! Will resolve immediately! The message in question can be mapped to what socket (by using the poller object and mean! So binding libraries make it too simple and agreable)! Then the right socket block will make the recieve call! And when it does it will consume it's next message from it's queue!
And so you keep looping and each time consuming the next message! As they came in! And the poller tracked there order of comming in! And that through the subscription and the events that were choosen to listen to! In case of receiving, it should be POLLIN!
Then each socket have it's message queue! And each recieve call! Pull from it! The poller tracked them! And so when the poller resolve! It's assured that there is message for the sockets recieve calls!
Last example: The server client pattern
Let's take the example of one server (router) and many clients (Dealers) that connect to! As by the image bellow!
Question: many connections to the same router! Comming asynchronously at once! And bla bla bla! In the server side (Router)! Do i need a poller !? A lot of new commers, may think yes or question if it's needed! Yup you guess right!
BIG NO!
Why ? Because in the Server (router) code! We have only one socket we are dealing with! That we bind! The clients then are connecting to it! In that end! There is only one socket! And all recv() calls are on that one socket! And that socket have it's message queue! The recv() consume the message one after another! Doesn't matter asynchronous and how they comes! Again the poller is only when there is multiple sockets! And so having the mixing nature of treating messages comming from multiple sockets! If not! Then one recv() of one socket need to go first then the other! And will block the other! Not a good thing (it's bad)!
Note
This answer bring a nice clearing! Plus it mades a reference to the doc with good highlighting! Also show code by the low level lib (c lang)! The answer of #rafflan show a great code with a binding lib (seems c#)! And a great explanation! If you didn't check it you must!