I am implementing a server that sends xml to clients using boost. The problem I am facing is that the buffer doesn't get sent immediately and accumulates to a point then sends the whole thing. This cause a problem on my client side, when it parses the xml, it may have incomplete xml tag (incomplete message). Is there a way in boost to flush out the socket whenever it needs to send out a message? Below is server's write code.
void
ClientConnection::handle_write(const boost::system::error_code& error)
{
if (!error)
{
m_push_message_queue.pop_front ();
if (!m_push_message_queue.empty () && !m_disconnected)
{
boost::asio::async_write(m_socket,
boost::asio::buffer(m_push_message_queue.front().data(),
m_push_message_queue.front().length()),
boost::bind(&ClientConnection::handle_write, this,
boost::asio::placeholders::error));
}
}
else
{
std::err << "Error writting out message...\n";
m_disconnected = true;
m_server->DisconnectedClient (this);
}
}
Typically when creating applications using TCP byte streams the sender sends a fixed length header so the receiver knows how many bytes to expect. Then the receiver reads that many bytes and parses the resulting buffer into an XML object.
I assume you are using TCP connection. TCP is stream type, so you can't assume your packet will come in one big packet. You need to fix your communication design, by sending size length first like San Miller answer, or sending flag or delimiter after all xml data has been sent.
Assuming you are definitely going to have some data on the socket you want to clear, you could do something like this:
void fulsh_socket()
{
boost::asio::streambuf b;
boost::asio::streambuf::mutable_buffers_type bufs = b.prepare(BUFFER_SIZE);
std::size_t bytes = socket_.receive(bufs); // !!! This will block until some data becomes available
b.commit(bytes);
boost::asio::socket_base::bytes_readable command(true);
socket_.io_control(command);
while(command.get())
{
bufs = b.prepare(BUFFER_SIZE);
bytes = socket_.receive(bufs);
b.commit(bytes);
socket_.io_control(command); // reset for bytes pending
}
return;
}
where socket_ is a member variable.
HTH
Related
For example:
Client Side
...
socket.connect(server_address)
data = some_message_less_than_100_bytes
socket.sendall(data)
...
Server Side
...
socket.accept()
socket.recv(1024)
...
Is the server side guaranteed to receive the data in one recv()?
If not, how does the standard solution using header for specifying message length even works?
The header itself could have been split and we have to check if header has been correctly received.
Or the header is fixed length? So that the receiver can always interpret the first few bytes in the same way no matter in how many pieces that data is sent?
Actually I'm trying to do something like this
Client
while():
send()
recv()
Server
recv()
while():
send() # Acknowledge to client
recv()
which is suggested by ravi in Linux socket: How to make send() wait for recv()
but I figured out the problem described above.
Is the ravi's answer assuming that both client and server will receive what the other sent in a single recv()?
Update
I would very like to post the image but I can't because of low reputation...
Following link is the HTTP Frame Format
https://datatracker.ietf.org/doc/html/rfc7540#section-4
It indeed used a fixed length solution, so that no matter in how many pieces the header is split it can work with the same way.
So I guess, some sort of 'fixed' length is the only solution? Even if the header size itself is variable, it then probably have some promised bits to indicate how long the header would be. Am I right?
Is the server side guaranteed to receive the data in one recv()?
For UDP, yes. recv() will return either 1 whole datagram, or an error. Though, if the buffer size is smaller than the datagram then the data will be truncated and you can't recover it.
For TCP, no. The only guarantee you have is that if no error occurs then recv() will return at least 1 byte but no more than the specified buffer size, it can return any number of bytes in between.
If not, how does the standard solution using header for specifying message length even works? The header itself could have been split and we have to check if header has been correctly received. Or the header is fixed length?
It can go either way, depending on the particular format of the header. Many protocols use fixed-length headers, and many protocols use variable-length headers.
Either way, you may have to call send() multiple times to ensure you send all the relevant bytes, and call recv() multiple times to ensure you receive all them. There is no 1:1 relationship between sends and reads in TCP.
Is the ravi's answer assuming that both client and server will receive what the other sent in a single recv()?
Ravi's answer makes no assumptions whatsoever about the number of bytes sent by send() and received by recv(). His answer is presented in a more higher-level perspective. But, it is very trivial to force the required behavior, eg:
int sendAll(int sckt, void *data, int len)
{
char *pdata = (char*) data;
while (len > 0) {
int res = send(sckt, pdata, len, 0);
if (res > 0) {
pdata += res;
len -= res;
}
else if (errno != EINTR) {
if ((errno != EWOULDBLOCK) && (errno != EAGAIN)) {
return -1;
}
/*
optional: use select() or (e)poll to
wait for the socket to be writable ...
*/
}
}
return 0;
}
int recvAll(int sckt, void *data, int len)
{
char *pdata = (char*) data;
while (len > 0) {
int res = recv(sckt, pdata, len, 0);
if (res > 0) {
pdata += res;
len -= res;
}
else if (res == 0) {
return 0;
}
else if (errno != EINTR) {
if ((errno != EWOULDBLOCK) && (errno != EAGAIN)) {
return -1;
}
/*
optional: use select() or (e)poll to
wait for the socket to be readable ...
*/
}
}
return 1;
}
This way, you can use sendAll() to send the message header followed by the message data, and recvAll() to receive the message header followed by the message data.
Is the server side guaranteed to receive the data in one recv()?
No.
TCP is a byte stream, not a message protocol. While it will likely work with small messages and an empty send buffer in most cases, it will start to fail if the data send get larger than the MTU of the underlying data link. TCP does not guarantee any atomar send-recv pair though for anything but a single octet. So don't count on it even for small data.
Can anyone please help me with this code , what is the use of "boost::asio::async_write" function here
Does it sends acknowledgment back to the client ?
void handle_read(const boost::system::error_code& error,
size_t bytes_transferred)
{
if (!error)
{
boost::asio::async_write(socket_,
boost::asio::buffer(data_, bytes_transferred),
boost::bind(&session::handle_write, this,
boost::asio::placeholders::error));
}
else
{
delete this;
}
}
It looks like this is from an "echo server" example. async_write writes the contents of boost::asio::buffer(data_, bytes_transferred) to the socket.
Since we're inside handle_read we can guess that this function itself is the completion handler for a likely async_read call that filled that data_ buffer. Since we use the exact number of bytes reported back by async_read (bytes_transferred) and there's no visible manipulation on data_, we can assume that this simply sends the exact message (or data in general) received to socket_. If socket_ was also the endpoint in the async_read this is the definition of an echo server.
I am trying to test WiFi data transfer between cell phone and Esp32 (Arduino), when ESP32 reads file data via WiFi, even there is still data in, client.read() often return -1, I have to add other conditions to check reading finished or not.
My question is why there are so many failed reads, any ideas are highly appreciated.
void setup()
{
i=0;
Serial.begin(115200);
Serial.println("begin...");
// You can remove the password parameter if you want the AP to be open.
WiFi.softAP(ssid, password);
IPAddress myIP = WiFi.softAPIP();
Serial.print("AP IP address: ");
Serial.println(myIP);
server.begin();
Serial.println("Server started");
}
// the loop function runs over and over again until power down or reset
void loop()
{
WiFiClient client = server.available(); // listen for incoming clients
if(client) // if you get a client,
{
Serial.println("New Client."); // print a message out the serial port
Serial.println(client.remoteIP().toString());
while(client.connected()) // loop while the client's connected
{
while(client.available()>0) // if there's bytes to read from the client,
{
char c = client.read(); // read a byte, then
if(DOWNLOADFILE ==c){
pretime=millis();
uint8_t filename[32]={0};
uint8_t bFilesize[8];
long filesize;
int segment=0;
int remainder=0;
uint8_t data[512];
int len=0;
int totallen=0;
delay(50);
len=client.read(filename,32);
delay(50);
len=client.read(bFilesize,8);
filesize=BytesToLong(bFilesize);
segment=(int)filesize/512;
delay(50);
i=0; //succeed times
j=0; //fail times
////////////////////////////////////////////////////////////////////
//problem occures here, to many "-1" return value
// total read 24941639 bytes, succeed 49725 times, failed 278348 times
// if there were no read problems, it should only read 48,715 times and finish.
//But it total read 328,073 times, including 278,348 falied times, wasted too much time
while(((len=client.read(data,512))!=-1) || (totallen<filesize))
{
if(len>-1) {
totallen+=len;
i++;
}
else{
j++;
}
}
///loop read end , too many times read fail//////////////////////////////////////////////////////////////////
sprintf(toClient, "\nfile name %s,size %d, total read %d, segment %d, succeed %d times, failed %d times\n",filename,filesize,totallen,segment,i,j);
Serial.write(toClient);
curtime=millis();
sprintf(toClient, "time splashed %d ms, speed %d Bps\n", curtime-pretime, filesize*1000/(curtime-pretime));
Serial.write(toClient);
client.write(RETSUCCESS);
}
else
{
Serial.write("Unknow command\n");
}
}
}
// close the connection:
client.stop();
Serial.println("Client Disconnected.");
}
When you call available() and check for > 0, you are checking to see if there is one or more characters available to read. It will be true if just one character has arrived. You read one character, which is fine, but then you start reading more without stopping to see if there are more available.
TCP doesn't guarantee that if you write 100 characters to a socket that they all arrive at once. They can arrive in arbitrary "chunks" with arbitrary delays. All that's guaranteed is that they will eventually arrive in order (or if that's not possible because of networking issues, the connection will fail.)
In the absence of a blocking read function (I don't know if those exist) you have to do something like what you are doing. You have to read one character at a time and append it to a buffer, gracefully handing the possibility of getting a -1 (the next character isn't here yet, or the connection broke). In general you never want to try to read multiple characters in a single read(buf, len) unless you've just used available() to make sure len characters are actually available. And even that can fail if your buffers are really large. Stick to one-character-at-a-time.
It's a reasonable idea to call delay(1) when available() returns 0. In the places where you try to guess at something like delay(20) before reading a buffer you are rolling the dice - there's no promise that any amount of delay will guarantee bytes get delivered. Example: Maybe a drop of water fell on the chip's antenna and it won't work until the drop evaporates. Data could be delayed for minutes.
I don't know how available() behaves if the connection fails. You might have to do a read() and get back a -1 to diagnose a failed connection. The Arduino documentation is absolutely horrible, so you'll have to experiment.
TCP is much simpler to handle on platforms that have threads, blocking read, select() and other tools to manage data. Having only non-blocking read makes things harder, but there it is.
In some situations UDP is actually a lot simpler - there are more guarantees about getting messages of certain sizes in a single chunk. But of course whole messages can go missing or show up out of order. It's a trade-off.
Okay this is my first question here on Stack Overflow, so bare over with it if I'm not asking properly.
Basically I'm trying to code some asynchronous sockets using std.socket, but I'm not sure if I've understood the concept correct. I've only ever worked with asynchronous sockets in C# and in D it seem to be on a much lower level. I've researched a lot and looked up a lot of code, documentation etc. both for D and C/C++ to get an understanding, however I'm not sure if I understand the concept correctly and if any of you have some examples. I tried looking at splat, but it's very outdated and vibe seems to be too complex just for a simple asynchronous socket wrapper.
If I understood correctly there is no poll() function in std.socket so you'd have to use SocketSet with a single socket on select() to poll the status of the socket right?
So basically how I'd go about handling the sockets is polling to get the read status of the socket and if it has a success (value > 0) then I can call receive() which will return 0 for disconnection else the received value, but I'd have to keep doing this until the expected bytes are received.
Of course the socket is set to nonblocked!
Is that correct?
Here is the code I've made up so far.
void HANDLE_READ()
{
while (true)
{
synchronized
{
auto events = cast(AsyncObject[int])ASYNC_EVENTS_READ;
foreach (asyncObject; events)
{
int poll = pollRecv(asyncObject.socket.m_socket);
switch (poll)
{
case 0:
{
throw new SocketException("The socket had a time out!");
continue;
}
default:
{
if (poll <= -1)
{
throw new SocketException("The socket was interrupted!");
continue;
}
int recvGetSize = (asyncObject.socket.m_readBuffer.length - asyncObject.socket.readSize);
ubyte[] recvBuffer = new ubyte[recvGetSize];
int recv = asyncObject.socket.m_socket.receive(recvBuffer);
if (recv == 0)
{
removeAsyncObject(asyncObject.event_id, true);
asyncObject.socket.disconnect();
continue;
}
asyncObject.socket.m_readBuffer ~= recvBuffer;
asyncObject.socket.readSize += recv;
if (asyncObject.socket.readSize == asyncObject.socket.expectedReadSize)
{
removeAsyncObject(asyncObject.event_id, true);
asyncObject.event(asyncObject.socket);
}
break;
}
}
}
}
}
}
So basically how I'd go about handling the sockets is polling to get the read status of the socket
Not quite right. Usually, the idea is to build an event loop around select, so that your application is idle as long as there are no network or timer events that need to be handled. With polling, you'd have to check for new events continuously or on a timer, which leads to wasted CPU cycles, and events getting handled a bit later than they occur.
In the event loop, you populate the SocketSets with sockets whose events you are interested in. If you want to be notified of new received data on a socket, it goes to the "readable" set. If you have data to send, the socket should be in the "writable" set. And all sockets should be on the "error" set.
select will then block (sleep) until an event comes in, and fill the SocketSets with the sockets which have actionable events. Your application can then respond to them appropriately: receive data for readable sockets, send queued data for writable sockets, and perform cleanup for errored sockets.
Here's my D implementation of non-fiber event-based networking: ae.net.asockets.
I am learning android but I can't get past the InputStream.read().
This is just a socket test - the server sends back two bytes when it receives a connection and I know that this working fine. All I want to do is read these values. The b = data.read reads both values in turn but then hangs, it never returns the -1 value which is what expect it to. Also it does not throw an exception.
Any ideas?
Thanks.
protected void startLongRunningOperation() {
// Fire off a thread to do some work that we shouldn't do directly in the UI thread
Thread t = new Thread() {
public void run() {
try {
Log.d("Socket", "try connect ");
Socket sock = new Socket("192.168.0.12", 5001);
Log.d("socket", "connected");
InputStream data = sock.getInputStream();
int b = 0;
while (b != -1) {
b = data.read();
}
data.close();
} catch (Exception e) {
Log.d("Socket", e.toString());
}
}
};
t.start();
}
Reaching the end of the stream is a special state. It doesn't happen just because there is nothing left to read. If the stream is still open, but there's nothing to be read, it will "hang" (or block) as you've noticed until a byte comes across.
To do what you want, the server either needs to close/end the stream, or you need to use:
while (data.available() > 0) {
..
When the number of available bytes is zero, there's nothing sitting in the stream buffer to be read.
On the other hand, if you know that there should only ever be two bytes to read, and that's the end of your data, then just read the two bytes and move on (i.e. don't use a while loop). The reason to use a while loop here would only be if you weren't sure how many total bytes to expect.