Is recv(bufsize) guaranteed to receive all the data if sended data is smaller then bufsize? - sockets

For example:
Client Side
...
socket.connect(server_address)
data = some_message_less_than_100_bytes
socket.sendall(data)
...
Server Side
...
socket.accept()
socket.recv(1024)
...
Is the server side guaranteed to receive the data in one recv()?
If not, how does the standard solution using header for specifying message length even works?
The header itself could have been split and we have to check if header has been correctly received.
Or the header is fixed length? So that the receiver can always interpret the first few bytes in the same way no matter in how many pieces that data is sent?
Actually I'm trying to do something like this
Client
while():
send()
recv()
Server
recv()
while():
send() # Acknowledge to client
recv()
which is suggested by ravi in Linux socket: How to make send() wait for recv()
but I figured out the problem described above.
Is the ravi's answer assuming that both client and server will receive what the other sent in a single recv()?
Update
I would very like to post the image but I can't because of low reputation...
Following link is the HTTP Frame Format
https://datatracker.ietf.org/doc/html/rfc7540#section-4
It indeed used a fixed length solution, so that no matter in how many pieces the header is split it can work with the same way.
So I guess, some sort of 'fixed' length is the only solution? Even if the header size itself is variable, it then probably have some promised bits to indicate how long the header would be. Am I right?

Is the server side guaranteed to receive the data in one recv()?
For UDP, yes. recv() will return either 1 whole datagram, or an error. Though, if the buffer size is smaller than the datagram then the data will be truncated and you can't recover it.
For TCP, no. The only guarantee you have is that if no error occurs then recv() will return at least 1 byte but no more than the specified buffer size, it can return any number of bytes in between.
If not, how does the standard solution using header for specifying message length even works? The header itself could have been split and we have to check if header has been correctly received. Or the header is fixed length?
It can go either way, depending on the particular format of the header. Many protocols use fixed-length headers, and many protocols use variable-length headers.
Either way, you may have to call send() multiple times to ensure you send all the relevant bytes, and call recv() multiple times to ensure you receive all them. There is no 1:1 relationship between sends and reads in TCP.
Is the ravi's answer assuming that both client and server will receive what the other sent in a single recv()?
Ravi's answer makes no assumptions whatsoever about the number of bytes sent by send() and received by recv(). His answer is presented in a more higher-level perspective. But, it is very trivial to force the required behavior, eg:
int sendAll(int sckt, void *data, int len)
{
char *pdata = (char*) data;
while (len > 0) {
int res = send(sckt, pdata, len, 0);
if (res > 0) {
pdata += res;
len -= res;
}
else if (errno != EINTR) {
if ((errno != EWOULDBLOCK) && (errno != EAGAIN)) {
return -1;
}
/*
optional: use select() or (e)poll to
wait for the socket to be writable ...
*/
}
}
return 0;
}
int recvAll(int sckt, void *data, int len)
{
char *pdata = (char*) data;
while (len > 0) {
int res = recv(sckt, pdata, len, 0);
if (res > 0) {
pdata += res;
len -= res;
}
else if (res == 0) {
return 0;
}
else if (errno != EINTR) {
if ((errno != EWOULDBLOCK) && (errno != EAGAIN)) {
return -1;
}
/*
optional: use select() or (e)poll to
wait for the socket to be readable ...
*/
}
}
return 1;
}
This way, you can use sendAll() to send the message header followed by the message data, and recvAll() to receive the message header followed by the message data.

Is the server side guaranteed to receive the data in one recv()?
No.
TCP is a byte stream, not a message protocol. While it will likely work with small messages and an empty send buffer in most cases, it will start to fail if the data send get larger than the MTU of the underlying data link. TCP does not guarantee any atomar send-recv pair though for anything but a single octet. So don't count on it even for small data.

Related

C socket server: What's the right way to read all of an unknown length XMLHttpRequest?

I have a simple XMLHttpRequest handler written in C. It reads and processes requests coming from a JavaScript XMLHttpRequest send() running in a browser.
The parent process accepts incoming connections and forks a child process for each incoming connection to read and process the data.
It works perfectly for most requests, but fails in some cases (apparently related to the network infrastructure between the client and the server) if the request is over about 2 kB in length. I'm assuming that the request is being broken into multiple packets somewhere between the browser and my socket server.
I can't change the request format, but I can see the request being sent and verify the content. The data is a 'GET' with an encoded URI that contains a 'type' field. If the type is 'file', the request could be as long as 3 kB, otherwise it's a couple of hundred bytes at most. 'File' requests are rare - the user is providing configuration data to be written to a file on the server. All other requests work fine, and any 'file' requests shorter than about 2 kB work fine.
What's the preferred technique for ensuring that I have all of the data in this situation?
Here's the portion of the parent that accepts the connection and forks the child (non-blocking version):
for (hit = 1;; hit++) {
length = sizeof(cli_addr);
if ((socketfd = accept4(listensd, (struct sockaddr *) &cli_addr, &length, SOCK_NONBLOCK)) < 0){
//if ((socketfd = accept(listensd, (struct sockaddr *) &cli_addr, &length)) < 0){
exit(3);
}
if ((pid = fork()) < 0) {
exit(3);
} else {
if (pid == 0) { /* child */
//(void) close(listensd);
childProcess(socketfd, hit); /* never returns. Close listensd when done*/
} else { /* parent */
(void) close(socketfd);
}
}
}
Here's the portion of the child process that performs the initial recv(). In the case of long 'file' requests, the child's first socket recv() gets about 1700 bytes of payload followed by the browser-supplied connection data.
ret = recv(socketfd, recv_data, BUFSIZE, 0); // read request
if (ret == 0 || ret == -1) { // read failure stop now
sprintf(sbuff, "failed to read request: %d", ret);
logger(&shm, FATAL, sbuff, socketfd);
}
recv_data[ret] = 0;
len = ret;
If the type is 'file', there could be more data. The child process never gets the rest of the data. If the socket is blocking, a second read attempt simply hangs. If the socket is non-blocking (as in the snippet below) all subsequent reads return -1 with error 'Resource temporarily unavailable' until it times out:
// It's a file. Could be broken into multiple blocks. Try second read
sleep(1);
ret = recv(socketfd, &recv_data[len], BUFSIZE, 0); // read request
while (ret != 0){
if (ret > 0){
recv_data[len+ret] = 0;
len += ret;
} else {
sleep(1);
}
ret = recv(socketfd, &recv_data[len], BUFSIZE, 0); // read request
}
I expected that read() would return 0 when the client closes the connection, but that doesn't happen.
A GET request only has a head and no body (well, almost always), so you have everything the client has sent as soon as you have the request head, and you know when you have read the whole request head when you read a blank line i.e. two returns (and no sooner or later).
If the client sends just a part, without the blank line, you are supposed to wait for the rest. I would put a time-out on that and reject the whole request if it takes too long.
BTW there are still browsers out there, and maybe some proxies as well, with a URL length limit of about 2000 characters.

Socket read often return -1 while the buffer is not empty

I am trying to test WiFi data transfer between cell phone and Esp32 (Arduino), when ESP32 reads file data via WiFi, even there is still data in, client.read() often return -1, I have to add other conditions to check reading finished or not.
My question is why there are so many failed reads, any ideas are highly appreciated.
void setup()
{
i=0;
Serial.begin(115200);
Serial.println("begin...");
// You can remove the password parameter if you want the AP to be open.
WiFi.softAP(ssid, password);
IPAddress myIP = WiFi.softAPIP();
Serial.print("AP IP address: ");
Serial.println(myIP);
server.begin();
Serial.println("Server started");
}
// the loop function runs over and over again until power down or reset
void loop()
{
WiFiClient client = server.available(); // listen for incoming clients
if(client) // if you get a client,
{
Serial.println("New Client."); // print a message out the serial port
Serial.println(client.remoteIP().toString());
while(client.connected()) // loop while the client's connected
{
while(client.available()>0) // if there's bytes to read from the client,
{
char c = client.read(); // read a byte, then
if(DOWNLOADFILE ==c){
pretime=millis();
uint8_t filename[32]={0};
uint8_t bFilesize[8];
long filesize;
int segment=0;
int remainder=0;
uint8_t data[512];
int len=0;
int totallen=0;
delay(50);
len=client.read(filename,32);
delay(50);
len=client.read(bFilesize,8);
filesize=BytesToLong(bFilesize);
segment=(int)filesize/512;
delay(50);
i=0; //succeed times
j=0; //fail times
////////////////////////////////////////////////////////////////////
//problem occures here, to many "-1" return value
// total read 24941639 bytes, succeed 49725 times, failed 278348 times
// if there were no read problems, it should only read 48,715 times and finish.
//But it total read 328,073 times, including 278,348 falied times, wasted too much time
while(((len=client.read(data,512))!=-1) || (totallen<filesize))
{
if(len>-1) {
totallen+=len;
i++;
}
else{
j++;
}
}
///loop read end , too many times read fail//////////////////////////////////////////////////////////////////
sprintf(toClient, "\nfile name %s,size %d, total read %d, segment %d, succeed %d times, failed %d times\n",filename,filesize,totallen,segment,i,j);
Serial.write(toClient);
curtime=millis();
sprintf(toClient, "time splashed %d ms, speed %d Bps\n", curtime-pretime, filesize*1000/(curtime-pretime));
Serial.write(toClient);
client.write(RETSUCCESS);
}
else
{
Serial.write("Unknow command\n");
}
}
}
// close the connection:
client.stop();
Serial.println("Client Disconnected.");
}
When you call available() and check for > 0, you are checking to see if there is one or more characters available to read. It will be true if just one character has arrived. You read one character, which is fine, but then you start reading more without stopping to see if there are more available.
TCP doesn't guarantee that if you write 100 characters to a socket that they all arrive at once. They can arrive in arbitrary "chunks" with arbitrary delays. All that's guaranteed is that they will eventually arrive in order (or if that's not possible because of networking issues, the connection will fail.)
In the absence of a blocking read function (I don't know if those exist) you have to do something like what you are doing. You have to read one character at a time and append it to a buffer, gracefully handing the possibility of getting a -1 (the next character isn't here yet, or the connection broke). In general you never want to try to read multiple characters in a single read(buf, len) unless you've just used available() to make sure len characters are actually available. And even that can fail if your buffers are really large. Stick to one-character-at-a-time.
It's a reasonable idea to call delay(1) when available() returns 0. In the places where you try to guess at something like delay(20) before reading a buffer you are rolling the dice - there's no promise that any amount of delay will guarantee bytes get delivered. Example: Maybe a drop of water fell on the chip's antenna and it won't work until the drop evaporates. Data could be delayed for minutes.
I don't know how available() behaves if the connection fails. You might have to do a read() and get back a -1 to diagnose a failed connection. The Arduino documentation is absolutely horrible, so you'll have to experiment.
TCP is much simpler to handle on platforms that have threads, blocking read, select() and other tools to manage data. Having only non-blocking read makes things harder, but there it is.
In some situations UDP is actually a lot simpler - there are more guarantees about getting messages of certain sizes in a single chunk. But of course whole messages can go missing or show up out of order. It's a trade-off.

Cancelling retransmissions on a L2CAP socket

I was wondering if anyone can assist me with a problem that I have with C Bluetooth programming (Linux Bluez).
I am using Ubuntu 10.04, BlueZ 4.60.
My goal is to have a L2CAP socket in which there will be minimal delay for sending data between 2 computers.
So far I managed to open an L2CAP socket, but this socket has endless retransmissions and I'm trying to change it. I want to have no retransmissions at all because I need the data to be transfer fast with minimal delay and the reliability of the data is not important.
I found an example online that deals with changing the flush timout for the socket and by that causing that if a packet is not acked after a certain period of time it is dropped and the next data in the buffer is sent.
The problem is that this example doesn't work :-(
Here is my code, this method is called after the bind command:
int set_flush_timeout(bdaddr_t *ba, int timeout)
{
int err = 0, dd, dev_id;
struct hci_conn_info_req *cr = 0;
struct hci_request rq = { 0 };
struct {
uint16_t handle;
uint16_t flush_timeout;
} cmd_param;
struct {
uint8_t status;
uint16_t handle;
} cmd_response;
// find the connection handle to the specified bluetooth device
cr = (struct hci_conn_info_req*) malloc(
sizeof(struct hci_conn_info_req) +
sizeof(struct hci_conn_info));
bacpy( &cr->bdaddr, ba );
cr->type = ACL_LINK;
dev_id = hci_get_route( NULL);
dd = hci_open_dev( dev_id );
if( dd < 0 ) {
err = dd;
goto cleanup;
}
err = ioctl(dd, HCIGETCONNINFO, (unsigned long) cr );
if( err ) goto cleanup;
// build a command packet to send to the bluetooth microcontroller
cmd_param.handle = cr->conn_info->handle;
cmd_param.flush_timeout = htobs(timeout);
rq.ogf = OGF_HOST_CTL;
rq.ocf = 0x28;
rq.cparam = &cmd_param;
rq.clen = sizeof(cmd_param);
rq.rparam = &cmd_response;
rq.rlen = sizeof(cmd_response);
rq.event = EVT_CMD_COMPLETE;
// send the command and wait for the response
err = hci_send_req( dd, &rq, 1 );
if( err ) goto cleanup;
if( cmd_response.status ) {
err = -1;
errno = bt_error(cmd_response.status);
}
cleanup:
free(cr);
if( dd >= 0) close(dd);
return err;
}
What is my mistake?
Does anyone know another option that will solve my problem.
Code examples will also be great!!
Thanks!!
This code to set the automatic flush time out seems to be correct.
You can make sure by ensuring that you are getting "Success" in response to this command's command complete event.
I suspect that the issue might be in your packet sending code, note that for the automatic flush timeout to take effect the individual packets should be marked as automatically flushable, The HCI data packet has the Packet_Boundary_Flag which you can sent to indicate if individual packets are flushable.
Also note that the Flush timeout has to be large enough to allow for enough time so that the packets gets a transmission attempt, the way the flush timeout are defined can cause the packet to be flushed even without the packet being transmitted even once, so you need to tune it. By definition Flush timeout start when the packet is Queued for transmission.

SIP Registration Issue

I am currently working on SIP sample application.
I am trying to do the registration using C based Socket programming on Unix. I have been successfully being able to get register with PJSIP, but when the same parameters I am sending with normal socket programming, then I am not being able to receive any response from server.
Here is the source code:
char *server = (char *)serverAddress; // First arg: server address/name
char *echoString = "Request msg REGISTER/cseq=46476 (tdta0x8857200)\r\nREGISTER sip:DOMAIN_NAME SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.1.120:51648;rport;branch=z9hG4bKPjwEt4VvIVdjIJKRmEJbkidYDAu-zQbIqv\r\nMax-Forwards: 70\r\nFrom: <sip:USER_NAME#DOMAIN_NAME>;tag=epCBN7JXsQE1nnI5d5SOZe9a5ujRyI67\r\nTo: <sip:USER_NAME#DOMAIN_NAME>\r\nCall-ID: .5yYCqh2jEYdy5T4kxhzxwDYEkCO1XlD\r\nCSeq: 46476 REGISTER\r\nContact: <sip:USER_NAME#192.168.1.120:51648;ob>\r\nExpires: 300\r\nAllow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS\r\nAuthorization: Digest username=\"USER_NAME\", realm=\"asterisk\", nonce=\"3b63254c\", uri=\"sip:DOMAIN_NAME\", response=\"9e8fc78829d143a58fba5a79f6ad44fd\", algorithm=MD5\r\nContent-Length: 0";
size_t echoStringLen = strlen(echoString);
// Third arg (optional): server port/service
char *servPort = (char *)service;
// Tell the system what kind(s) of address info we want
struct addrinfo addrCriteria; // Criteria for address match
memset(&addrCriteria, 0, sizeof(addrCriteria)); // Zero out structure
addrCriteria.ai_family = AF_UNSPEC; // For the following fields, a zero value means
// Any address family "don't care"
addrCriteria.ai_socktype = SOCK_DGRAM; // Only datagram sockets
addrCriteria.ai_protocol = IPPROTO_UDP; // Only UDP protocol
// Get address(es)
struct addrinfo *servAddr; // List of server addresses
int rtnVal = getaddrinfo(server, servPort, &addrCriteria, &servAddr);
if (rtnVal != 0)
DieWithUserMessage("getaddrinfo() failed", gai_strerror(rtnVal));
// Create a datagram/UDP socket
int sock = socket(servAddr->ai_family, servAddr->ai_socktype,
servAddr->ai_protocol); // Socket descriptor for client
if (sock < 0)
DieWithSystemMessage("socket() failed");
// Send the string to the server
ssize_t numBytes = sendto(sock, echoString, echoStringLen, 0,
servAddr->ai_addr, servAddr->ai_addrlen);
if (numBytes < 0)
DieWithSystemMessage("sendto() failed");
else if (numBytes != echoStringLen)
DieWithUserMessage("sendto() error", "sent unexpected number of bytes");
// Receive a response
struct sockaddr_storage fromAddr; // Source address of server
// Set length of from address structure (in-out parameter)
socklen_t fromAddrLen = sizeof(fromAddr);
char buffer[100 + 1]; // I/O buffer
numBytes = recvfrom(sock, buffer, 100, 0,
(struct sockaddr *) &fromAddr, &fromAddrLen);
if (numBytes < 0)
DieWithSystemMessage("recvfrom() failed");
else if (numBytes != 100)
DieWithUserMessage("recvfrom() error", "received unexpected number of bytes");
// Verify reception from expected source
int value = SockAddrsEqual(servAddr->ai_addr, (struct sockaddr *) &fromAddr);
if (value == 0)
DieWithUserMessage("recvfrom()", "received a packet from unknown source");
freeaddrinfo(servAddr);
buffer[echoStringLen] = '\0'; // Null-terminate received data
printf("Received: %s\n", buffer); // Print the echoed string
close(sock);
exit(0);
When I am trying to debug the code, then the breakpoint gets disappeared from recvfrom method call as follows.
numBytes = recvfrom(sock, buffer, 100, 0, (struct sockaddr *) &fromAddr, &fromAddrLen);
Thanks in advance for your co-operation.
If you are not getting a response from the Registrar server it's most likely because it couldn't understand the REGISTER request you sent it or rejected it as a duplicate. More that likely you will need to get a bit more sophisticated about how you are constructing your REGISTER request rather than simply sending a hard coded string.
Apart from that one obvious thing you could try is to add the required \r\n\r\n onto the end of your echoString. All SIP requests are required to end the header portion with a double line break and that is missing from your string.
recvfrom() blocks until a response is received, unless you set the socket to non-blocking. A better approach would be to use select() with a timeout (like a second or two). Better still would be to then put the entire thing into a loop which retries some number of times (like three or four).
Remember that UDP makes no guarantees that your datagram will be delivered: it is entirely up to you to decide when you've waited long enough and need to try again.
Also, check that the values being set in servAddr by getaddrinfo() make sense. If the address or port number is wrong, your packet may silently disappear.
If you confirm that this is socket issue then good to trace.
But if you are not sure then for this kind of issue use network analyze application (wireshark) & see the packet you send & receive. Then look into socket issue.

How to flush the socket using boost

I am implementing a server that sends xml to clients using boost. The problem I am facing is that the buffer doesn't get sent immediately and accumulates to a point then sends the whole thing. This cause a problem on my client side, when it parses the xml, it may have incomplete xml tag (incomplete message). Is there a way in boost to flush out the socket whenever it needs to send out a message? Below is server's write code.
void
ClientConnection::handle_write(const boost::system::error_code& error)
{
if (!error)
{
m_push_message_queue.pop_front ();
if (!m_push_message_queue.empty () && !m_disconnected)
{
boost::asio::async_write(m_socket,
boost::asio::buffer(m_push_message_queue.front().data(),
m_push_message_queue.front().length()),
boost::bind(&ClientConnection::handle_write, this,
boost::asio::placeholders::error));
}
}
else
{
std::err << "Error writting out message...\n";
m_disconnected = true;
m_server->DisconnectedClient (this);
}
}
Typically when creating applications using TCP byte streams the sender sends a fixed length header so the receiver knows how many bytes to expect. Then the receiver reads that many bytes and parses the resulting buffer into an XML object.
I assume you are using TCP connection. TCP is stream type, so you can't assume your packet will come in one big packet. You need to fix your communication design, by sending size length first like San Miller answer, or sending flag or delimiter after all xml data has been sent.
Assuming you are definitely going to have some data on the socket you want to clear, you could do something like this:
void fulsh_socket()
{
boost::asio::streambuf b;
boost::asio::streambuf::mutable_buffers_type bufs = b.prepare(BUFFER_SIZE);
std::size_t bytes = socket_.receive(bufs); // !!! This will block until some data becomes available
b.commit(bytes);
boost::asio::socket_base::bytes_readable command(true);
socket_.io_control(command);
while(command.get())
{
bufs = b.prepare(BUFFER_SIZE);
bytes = socket_.receive(bufs);
b.commit(bytes);
socket_.io_control(command); // reset for bytes pending
}
return;
}
where socket_ is a member variable.
HTH