How to know if it is safe to keep reading from a socket inputStream (avoid read blocks) - sockets

I'll give you a little bit of context first:
I have a class that is supposed to write in a socket some querys and then read the answers.
Here it is the code that is supposed to read from the inputStream:
private String getDataFromInputStream() throws IOException
{
StringBuilder sb = new StringBuilder();
InputStream stream = this.socket.getInputStream();
byte[] buff = new byte[1024];
int bytesRead = 0;
while(bytesRead >= 0)
{
bytesRead = stream.read(buff);
if (bytesRead > 0) sb.append(new String(buff, 0, bytesRead));
}
return sb.toString();
}
If the returned result is OK, this method works great, but if, for some reason, the InputStream.read keeps waiting for input it, of course, blocks.
So I changed that code to this one:
private String getDataFromInputStream() throws IOException
{
StringBuilder sb = new StringBuilder();
InputStream stream = this.socket.getInputStream();
byte[] buff = new byte[1024];
int bytesRead = 0;
while(stream.available() > 0)
{
bytesRead = stream.read(buff);
if (bytesRead > 0) sb.append(new String(buff, 0, bytesRead));
}
return sb.toString();
}
But here's the new problem: The second piece of code doesn't read the whole response. It just returns a fragment of the full response.
I know that the available() method returns 0 because the stream doesn't have more data to be read inside the buffer but there is more data to be read from the socket.
So how can I make a method flexible enought to read the whole thing even if it takes some time to the Stream to buffer the response and, also doesn't block when expecting input?
I need to get the whole response if the query succeeded and to close the input so it won't block, if the command failed and the socket is expecting any input.

Since you're using a socket, you'll need the source to tell you how much data it wants to send. That may be present in the first few bytes that is sent (in which case you wait for that specific number of bytes first) otherwise you'll have to modify the source to provide that information. This is what TCP and UDP packets do as well, where the header to the packets include the length of the data so that the client knows how many bytes to wait for.

Related

Is recv(bufsize) guaranteed to receive all the data if sended data is smaller then bufsize?

For example:
Client Side
...
socket.connect(server_address)
data = some_message_less_than_100_bytes
socket.sendall(data)
...
Server Side
...
socket.accept()
socket.recv(1024)
...
Is the server side guaranteed to receive the data in one recv()?
If not, how does the standard solution using header for specifying message length even works?
The header itself could have been split and we have to check if header has been correctly received.
Or the header is fixed length? So that the receiver can always interpret the first few bytes in the same way no matter in how many pieces that data is sent?
Actually I'm trying to do something like this
Client
while():
send()
recv()
Server
recv()
while():
send() # Acknowledge to client
recv()
which is suggested by ravi in Linux socket: How to make send() wait for recv()
but I figured out the problem described above.
Is the ravi's answer assuming that both client and server will receive what the other sent in a single recv()?
Update
I would very like to post the image but I can't because of low reputation...
Following link is the HTTP Frame Format
https://datatracker.ietf.org/doc/html/rfc7540#section-4
It indeed used a fixed length solution, so that no matter in how many pieces the header is split it can work with the same way.
So I guess, some sort of 'fixed' length is the only solution? Even if the header size itself is variable, it then probably have some promised bits to indicate how long the header would be. Am I right?
Is the server side guaranteed to receive the data in one recv()?
For UDP, yes. recv() will return either 1 whole datagram, or an error. Though, if the buffer size is smaller than the datagram then the data will be truncated and you can't recover it.
For TCP, no. The only guarantee you have is that if no error occurs then recv() will return at least 1 byte but no more than the specified buffer size, it can return any number of bytes in between.
If not, how does the standard solution using header for specifying message length even works? The header itself could have been split and we have to check if header has been correctly received. Or the header is fixed length?
It can go either way, depending on the particular format of the header. Many protocols use fixed-length headers, and many protocols use variable-length headers.
Either way, you may have to call send() multiple times to ensure you send all the relevant bytes, and call recv() multiple times to ensure you receive all them. There is no 1:1 relationship between sends and reads in TCP.
Is the ravi's answer assuming that both client and server will receive what the other sent in a single recv()?
Ravi's answer makes no assumptions whatsoever about the number of bytes sent by send() and received by recv(). His answer is presented in a more higher-level perspective. But, it is very trivial to force the required behavior, eg:
int sendAll(int sckt, void *data, int len)
{
char *pdata = (char*) data;
while (len > 0) {
int res = send(sckt, pdata, len, 0);
if (res > 0) {
pdata += res;
len -= res;
}
else if (errno != EINTR) {
if ((errno != EWOULDBLOCK) && (errno != EAGAIN)) {
return -1;
}
/*
optional: use select() or (e)poll to
wait for the socket to be writable ...
*/
}
}
return 0;
}
int recvAll(int sckt, void *data, int len)
{
char *pdata = (char*) data;
while (len > 0) {
int res = recv(sckt, pdata, len, 0);
if (res > 0) {
pdata += res;
len -= res;
}
else if (res == 0) {
return 0;
}
else if (errno != EINTR) {
if ((errno != EWOULDBLOCK) && (errno != EAGAIN)) {
return -1;
}
/*
optional: use select() or (e)poll to
wait for the socket to be readable ...
*/
}
}
return 1;
}
This way, you can use sendAll() to send the message header followed by the message data, and recvAll() to receive the message header followed by the message data.
Is the server side guaranteed to receive the data in one recv()?
No.
TCP is a byte stream, not a message protocol. While it will likely work with small messages and an empty send buffer in most cases, it will start to fail if the data send get larger than the MTU of the underlying data link. TCP does not guarantee any atomar send-recv pair though for anything but a single octet. So don't count on it even for small data.

Read all available bytes from TCP Socket (unknown byte count)

I am having Problems useing the Indy TIdTCPClient.
I want to call a function, everytime if there is Data available on the socket. For this I have a Thread calling IdTCPClient->Socket->Readable(100).
The function itself looks like this:
TMemoryStream *mStream = new TMemoryStream;
int len = 0;
try
{
if(!Form1->IdTCPClient2->Connected())
Form1->IdTCPClient2->Connect();
mStream->Position = 0;
do
{
Form1->IdTCPClient2->Socket->ReadStream(mStream, 1);
}
while(Form1->IdTCPClient2->Socket->Readable(100));
len = mStream->Position;
mStream->Position = 0;
mStream->Read(Buffer, len);
}catch(Exception &Ex) {
Form1->DisplaySSH->Lines->Add(Ex.Message);
Form1->DisplaySSH->GoToTextEnd();
}
delete mStream;
It will not be called directly within the thread, but the thread triggers an event, which is calling this function. Which means I am using Readable(100) twice, without reading data in betwee.
So since I dont know how many bytes I have to read I thought I can read one byte, check if there is more available and then read another byte.
The Problem here is that the do while loop doesnt loop, it just runs once.
I am guessing that Readable does not quite wokt the way I need it to.
Is there any other way to receive all the bytes available in the Socket?
You should not be using Readable() directly in this situation. That call reports whether the underlying socket has pending unread data in its internal kernel buffer. That does not take into account that the TIdIOHandler may already have unread data in its InputBuffer that is left over from a previous read operation.
Use the TIdIOHandler::CheckForDataOnSource() method instead of TIdIOHandler::Readable():
TMemoryStream *mStream = new TMemoryStream;
try
{
if (!Form1->IdTCPClient2->Connected())
Form1->IdTCPClient2->Connect();
mStream->Position = 0;
do
{
if (Form1->IdTCPClient2->IOHander->InputBufferIsEmpty())
{
if (!Form1->IdTCPClient2->IOHander->CheckForDataOnSource(100))
break;
}
Form1->IdTCPClient2->IOHandler->ReadStream(mStream, Form1->IdTCPClient2->IOHandler->InputBuffer->Size, false);
/* alternatively:
Form1->IdTCPClient2->IOHandler->InputBuffer->ExtractToStream(mStream);
*/
}
while (true);
// use mStream as needed...
}
catch (const Exception &Ex) {
Form1->DisplaySSH->Lines->Add(Ex.Message);
Form1->DisplaySSH->GoToTextEnd();
}
delete mStream;
Or, you can alternatively use TIdIOHandler::ReadBytes() instead of TIdIOHandler::ReadStream(). If you set its AByteCount parameter to -1, it will return only the bytes that are currently available (if the InputBuffer is empty, ReadBytes() will wait up to the ReadTimeout interval for the socket to receive any new bytes) 1:
try
{
if (!Form1->IdTCPClient2->Connected())
Form1->IdTCPClient2->Connect();
TIdBytes data;
do
{
if (Form1->IdTCPClient2->IOHander->InputBufferIsEmpty())
{
if (!Form1->IdTCPClient2->IOHander->CheckForDataOnSource(100))
break;
}
Form1->IdTCPClient2->IOHandler->ReadBytes(data, -1, true);
/* alternatively:
Form1->IdTCPClient2->IOHandler->InputBuffer->ExtractToBytes(data, -1, true);
*/
}
while (true);
// use data as needed...
}
catch (const Exception &Ex) {
Form1->DisplaySSH->Lines->Add(Ex.Message);
Form1->DisplaySSH->GoToTextEnd();
}
1: make sure you are using an up-to-date snapshot of Indy 10. Prior to Oct 6 2016, there was a logic bug in ReadBytes() when AByteCount=-1 that didn't take the InputBuffer into account before checking the socket for new bytes.

Android InputStream

I am learning android but I can't get past the InputStream.read().
This is just a socket test - the server sends back two bytes when it receives a connection and I know that this working fine. All I want to do is read these values. The b = data.read reads both values in turn but then hangs, it never returns the -1 value which is what expect it to. Also it does not throw an exception.
Any ideas?
Thanks.
protected void startLongRunningOperation() {
// Fire off a thread to do some work that we shouldn't do directly in the UI thread
Thread t = new Thread() {
public void run() {
try {
Log.d("Socket", "try connect ");
Socket sock = new Socket("192.168.0.12", 5001);
Log.d("socket", "connected");
InputStream data = sock.getInputStream();
int b = 0;
while (b != -1) {
b = data.read();
}
data.close();
} catch (Exception e) {
Log.d("Socket", e.toString());
}
}
};
t.start();
}
Reaching the end of the stream is a special state. It doesn't happen just because there is nothing left to read. If the stream is still open, but there's nothing to be read, it will "hang" (or block) as you've noticed until a byte comes across.
To do what you want, the server either needs to close/end the stream, or you need to use:
while (data.available() > 0) {
..
When the number of available bytes is zero, there's nothing sitting in the stream buffer to be read.
On the other hand, if you know that there should only ever be two bytes to read, and that's the end of your data, then just read the two bytes and move on (i.e. don't use a while loop). The reason to use a while loop here would only be if you weren't sure how many total bytes to expect.

read specific number of bytes from NetworkStream

I am trying to read a message of known length from the network stream.
I was kinda expecting that NetworkStream.Read() would wait to return until buffer array I gave to it is full. If not, then what is the point of the ReadTimeout property?
Sample code I am using to test my theory
public static void Main(string[] args)
{
TcpListener listener = new TcpListener(IPAddress.Any, 10001);
listener.Start();
Console.WriteLine("Waiting for connection...");
ThreadPool.QueueUserWorkItem(WriterThread);
using (TcpClient client = listener.AcceptTcpClient())
using (NetworkStream stream = client.GetStream())
{
Console.WriteLine("Connected. Waiting for data...");
client.ReceiveTimeout = (int)new TimeSpan(0, 1, 0).TotalMilliseconds;
stream.ReadTimeout = (int)new TimeSpan(0, 1, 0).TotalMilliseconds;
byte[] buffer = new byte[1024];
int bytesRead = stream.Read(buffer, 0, buffer.Length);
Console.WriteLine("Got {0} bytes.", bytesRead);
}
listener.Stop();
Console.WriteLine("Press any key to exit...");
Console.ReadKey(true);
}
private static void WriterThread(object state)
{
using (TcpClient client = new TcpClient())
{
client.Connect(new IPEndPoint(IPAddress.Parse("127.0.0.1"), 10001));
using (NetworkStream stream = client.GetStream())
{
byte[] bytes = Encoding.UTF8.GetBytes("obviously less than 1024 bytes");
Console.WriteLine("Sending {0} bytes...", bytes.Length);
stream.Write(bytes, 0, bytes.Length);
Thread.Sleep(new TimeSpan(0, 2, 0));
}
}
}
Result of that is:
Waiting for connection...
Sending 30 bytes...
Connected. Waiting for data...
Got 30 bytes.
Press any key to exit...
Is there a standard way of making a sync read that returns only when specified number of bytes was read? I am sure it is not too complicated to write one myself, but presence of the timeout properties on both TcpClient and NetworkStream kinda suggests it should be already working that way.
All you are guaranteed is (one of):
0 bytes (end of stream)
at least 1 byte (some data available; does not mean there isn't more coming or already available)
an error (timeout, etc)
To read a specified number of bytes... loop:
int read = 0, offset = 0, toRead = ...
while(toRead > 0 && (read = stream.Read(buffer, offset, toRead)) > 0) {
toRead -= read;
offset += read;
}
if(toRead > 0) throw new EndOfStreamException();
TCP is a byte-stream protocol that does not preserve application message boundaries. It is simply not able to "glue" bytes together in that way. The purpose of the read timeout is to specify how long you would like the read to block. But as long as at least one byte of data can be returned, the read operation will not block.
If you need to call read in a loop until you read a complete message, do that. The TCP layer doesn't care what you consider to be a full message, that's not its job.

Detecting client TCP disconnection while using NetworkStream class

A friend of mine came to me with a problem: when using the NetworkStream class on the server end of the connection, if the client disconnects, NetworkStream fails to detect it.
Stripped down, his C# code looked like this:
List<TcpClient> connections = new List<TcpClient>();
TcpListener listener = new TcpListener(7777);
listener.Start();
while(true)
{
if (listener.Pending())
{
connections.Add(listener.AcceptTcpClient());
}
TcpClient deadClient = null;
foreach (TcpClient client in connections)
{
if (!client.Connected)
{
deadClient = client;
break;
}
NetworkStream ns = client.GetStream();
if (ns.DataAvailable)
{
BinaryFormatter bf = new BinaryFormatter();
object o = bf.Deserialize(ns);
ReceiveMyObject(o);
}
}
if (deadClient != null)
{
deadClient.Close();
connections.Remove(deadClient);
}
Thread.Sleep(0);
}
The code works, in that clients can successfully connect and the server can read data sent to it. However, if the remote client calls tcpClient.Close(), the server does not detect the disconnection - client.Connected remains true, and ns.DataAvailable is false.
A search of Stack Overflow provided an answer - since Socket.Receive is not being called, the socket is not detecting the disconnection. Fair enough. We can work around that:
foreach (TcpClient client in connections)
{
client.ReceiveTimeout = 0;
if (client.Client.Poll(0, SelectMode.SelectRead))
{
int bytesPeeked = 0;
byte[] buffer = new byte[1];
bytesPeeked = client.Client.Receive(buffer, SocketFlags.Peek);
if (bytesPeeked == 0)
{
deadClient = client;
break;
}
else
{
NetworkStream ns = client.GetStream();
if (ns.DataAvailable)
{
BinaryFormatter bf = new BinaryFormatter();
object o = bf.Deserialize(ns);
ReceiveMyObject(o);
}
}
}
}
(I have left out exception handling code for brevity.)
This code works, however, I would not call this solution "elegant". The other elegant solution to the problem I am aware of is to spawn a thread per TcpClient, and allow the BinaryFormatter.Deserialize (née NetworkStream.Read) call to block, which would detect the disconnection correctly. Though, this does have the overhead of creating and maintaining a thread per client.
I get the feeling that I'm missing some secret, awesome answer that would retain the clarity of the original code, but avoid the use of additional threads to perform asynchronous reads. Though, perhaps, the NetworkStream class was never designed for this sort of usage. Can anyone shed some light?
Update: Just want to clarify that I'm interested to see if the .NET framework has a solution that covers this use of NetworkStream (i.e. polling and avoiding blocking) - obviously it can be done; the NetworkStream could easily be wrapped in a supporting class that provides the functionality. It just seemed strange that the framework essentially requires you to use threads to avoid blocking on NetworkStream.Read, or, to peek on the socket itself to check for disconnections - almost like it's a bug. Or a potential lack of a feature. ;)
Is the server expecting to be sent multiple objects over the same connection? IF so I dont see how this code will work, as there is no delimiter being sent that signifies where the first object starts and the next object ends.
If only one object is being sent and the connection closed after, then the original code would work.
There has to be a network operation initiated in order to find out if the connection is still active or not. What I would do, is that instead of deserializing directly from the network stream, I would instead buffer into a MemoryStream. That would allow me to detect when the connection was lost. I would also use message framing to delimit multiple responses on the stream.
MemoryStream ms = new MemoryStream();
NetworkStream ns = client.GetStream();
BinaryReader br = new BinaryReader(ns);
// message framing. First, read the #bytes to expect.
int objectSize = br.ReadInt32();
if (objectSize == 0)
break; // client disconnected
byte [] buffer = new byte[objectSize];
int index = 0;
int read = ns.Read(buffer, index, Math.Min(objectSize, 1024);
while (read > 0)
{
objectSize -= read;
index += read;
read = ns.Read(buffer, index, Math.Min(objectSize, 1024);
}
if (objectSize > 0)
{
// client aborted connection in the middle of stream;
break;
}
else
{
BinaryFormatter bf = new BinaryFormatter();
using(MemoryStream ms = new MemoryStream(buffer))
{
object o = bf.Deserialize(ns);
ReceiveMyObject(o);
}
}
Yeah but what if you lose a connection before getting the size? i.e. right before the following line:
// message framing. First, read the #bytes to expect.
int objectSize = br.ReadInt32();
ReadInt32() will block the thread indefinitely.