I have developed a client and a server using swift nio, I have no problems sending package of all size between 12 and 1000bytes since server sends a pack of 528bytes and when client got it, it is 512bytes. I'm trying to figure out why it happens. Does anyone knows if there is any chance to set a minimum ByteBuffer capacity? or if I'm missing something.
Thanks to all.
Assuming you're using TCP (that is, using ClientBootstrap), you cannot expect that the boundaries of messages sent by the server will be reflected in your reads. TCP is "stream-oriented": this means that the messages don't have boundaries at all, they behave just like a stream of data. In the NIO case, that means you would expect to see another read shortly after that contains more data.
The initial ByteBuffer capacity used for reads is controlled by the RecvByteBufferAllocator used by the Channel. This can be overridden:
ClientBootstrap(group: group)
.channelOption(ChannelOptions.recvAllocator,
AdaptiveRecvByteBufferAllocator(minimum: 1024, initial: 1024, maximum: 65536))
The standard defaults for the AdaptiveRecvByteBufferAllocator in NIO 2.23.0 are a minimum size of 64 bytes, an initial size of 1024 bytes, and a maximum size of 65536 bytes. In general we don't recommend overriding these defaults unless you need to: for TCP NIO will ensure the buffer is appropriately sized for the reads we're seeing.
Related
I have a socket application that reads and writes data. I'm using OpenSSL to do the encryption/decryption. My question is whether the "BIO_write" method can buffer data internally or if I have to append to a growing buffer as I read more from the socket. Here's what I'm doing.
I read from the socket and I use the class method below to write all the bytes that were read into the BIO:
int CSslConnectionContext::Store(BYTE* pbData, DWORD dwDataLength)
{
int bytes = 0;
if (dwDataLength > 0)
{
bytes = BIO_write(bio[BIO_RECV], pbData, dwDataLength);
}
return bytes;
}
I then immediately call the SSL_read method to get decrypted data:
int CSslConnectionContext::Read(BYTE* pbBuffer, DWORD dwBufferSize)
{
int bytes = SSL_read(ssl, pbBuffer, dwBufferSize);
return bytes;
}
If SSL_read returns a positive number then I've got decrypted data available for my application.
What I'm not sure of is what happens when my socket read doesn't capture all of the data required for decryption in a single read.
So if I need 100 bytes to be able to decrypted the data and the first read only gets 80, can I call BIO_write() with those 80, do another socket read to get the next 20, and then call BIO_write() with just those 20 bytes?
Or do I need to write my code so when I read 80 I do something like this:
call BIO_write() with the 80 bytes.
if that returns a failure indicator - hold onto that 80 bytes.
read the next 20 bytes from the socket and append it to the buffered 80 bytes.
call BIO_write() with 100 bytes
OpenSSL holds internal buffer - let's call it SSL stack - on top of TCP stack. OpenSSL library handles SSL stack. BIO_xxx() functions can operate on different end-points: i.e. memory, sockets.
It behaves differently depending on the actual item it operates on. For instance if BIO_write() uses memory (BIO_s_mem), BIO_write never fails except insufficient memory. But if it uses socket, and socket is non-blocking it can return error on failure, or it can write some number of bytes instead of all of the requested bytes where socket buffer is full.
So, how to use/handle buffer depends many factors, but most noticable ones are:
Blocking or Nonblocking IO
BIO object that operates on (memory, socket, etc.)
For instance if you're using BIO_s_mem and non-blocking socket operations, following technique can be applied:
Write buffer using BIO_write, and check if it failed. If it did not fail, you can be sure that you've written all buffer to SSL stack.
Call Read_SSL and check for errors, if error is WANT_READ, or WANT_WRITE then you need to write more data to SSL stack to be able to read a valid record.
For the question and example:
You can write partially (As many as you can, even 1 byte). For instance if you read 80 bytes from socket, then write those using BIO_write. Then call to SSL_read may fail (WANT_READ, WANT_WRITE, or other). Then you receive 20 bytes from socket, then write these bytes using BIO_write. Then call SSL_read again. Whenever SSL_read returns without error this means SSL stack decoded a valid record.
But it is quite important to understand waiting on non-blocking sockets using select() to handle SSL reads/writes are quite cumbersome. One SSL_write can result multiple writes to socket while you already waiting for READ event for the socket.
please use bio_pending.. to know all the bytes available with openssl.. Loop using the return value of bio_pending. This should be called before bio_read.
After studying the "window size" concept, what I understood is that it keeps packet before sending over wire and till acknowledgement come for earliest packet . Once this gets filled up, subsequent packet will be dropped. Somewhere I also have read that TCP is a streaming protocol, and packet is what related to IP protocol at Network layer .
What I assumed till was that I have declared a Buffer (inside code) which I fill with some data and send this Buffer using socket. I declared a buffer of 10000 bytes and send it repeatedly using socket over 10 Gbps link .
I have following assumptions and questions. Please verify and help
If I want to send a packet of 64,256,512 etc. bytes, declared buffer inside code of that much space and send over socket. Each execution of send() command will send one packet of that much size .
So if I want to study the packet size variation effect on throughput, what do I have to do? Do I need to vary buffer size in code?
What are the socket buffer which we set using SO_SNDBUF and SO_RECVBUF? Google says it's buffer space for socket. Is it same as TCP window size or something different? Which parameter is more suitable to vary or to increase throughput?
Also there are three parameter in socket buffer: Min, Default and Max. Which one should I vary to my experiment and to get more relevance?
If I want to send a packet of 64,256,512 etc. bytes , Declared buffer inside code of that much space and send over socket .Each execution of send() command will send one packet of that much size.
Only if you disable the Nagle algorithm and the size is less than the path MTU. You mustn't rely on this.
So if I want to Study the Packet size variation effect on throughput, What I have to do , vary buffer space in Code?
No. Vary SO_RCVBUF at the receiver. This is the single biggest determinant of throughput, as it determines the maximum receive window.
what are the socket buffer which we set using SO_SNDBUF and SO_RCVBUF
Send buffer size at the sender, and receive buffer size at the receiver. In the kernel.
It's Same as TCP Window size
See above.
or else different ? Which parameter is more suitable to vary to increase throughput ?
See above.
Also there are three parameter in Socket Buffer min Default and Max . Which one should I vary for My experiment to get more relevance
None of them. These are the system-wide parameters. Just play with SO_SNDBUF and SO_RCVBUF for the specific sockets in your application.
TCP does not directly expose a way to control the way packets are sent since it is a stream protocol. But you can make the TCP stack send packets by disabling the Nagle algorithm. That way all data that you send will be sent out immediately instead of being buffered. Data will be split into packets of MTU size which is like ~1400 bytes. Depends on the link.
To answer (2): Disable nagling and invoke send with buffers of < 1400 bytes. Use Wireshark to make sure you got what you wanted.
The buffer settings have nothing to do with any of this. I know of no valid reason to touch them.
In general this question is probably moot since you seem to want to send a lot of data. Just leave Nagling enabled and send big buffers (such as 64KB).
I do some experience on Windows 10:
code from https://docs.python.org/3/library/socketserver.html#asynchronous-mixins,
RawCap for loopback capture,
WireShark for watching result.
The primary client code is:
def client(ip, port, message):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF, 100000)
sock.connect((ip, port))
sock.sendall(bytes(message, 'ascii'))
response = str(sock.recv(1024), 'ascii')
print("Received: {}".format(response))
Here is the result(the server port is 11111):
you can see, the tcp recive window size is the same as SO_RCVBUF, may it is platform indepent, you can verify it on other platform.
on https://msdn.microsoft.com/en-us/library/windows/hardware/ff570832(v=vs.85).aspx
The SO_RCVBUF socket option determines the size of a socket's receive buffer that is used by the underlying transport.
verified this.
Also, when I set SO_SNDBUF = 100000, it have no affects on the tcp transmission between client and server, as server just can discard data if client send much data one time.
So, if you want to change SO_RCVBUF to max Throughput, you can refer http://packetbomb.com/understanding-throughput-and-tcp-windows/, the os may offer func to detect ideal send backlog (ISB).
I am writing the callout driver for Hyper-V 2012 where I need to filter the packets sent from virtual machines.
I added filter at FWPM_LAYER_EGRESS_VSWITCH_TRANSPORT_V4 layer in WFP. Callout function receive packet buffer which I am typecasting it to NET_BUFFER_LIST. I am doing following to get the data pointer
pNetBuffer = NET_BUFFER_LIST_FIRST_NB((NET_BUFFER_LIST*)pClassifyData->pPacket);
pContiguousData = NdisGetDataBuffer(pNetBuffer, NET_BUFFER_DATA_LENGTH(pNetBuffer), 0, 1, 0);
I have simple client-server application to test the packet data. Client is on VM and server is another machine. As I observed, data sent from client to server is truncated and some garbage value is added at the end. There is no issue for sending message from server to client. If I dont add this layer filter client-server works without any issue.
Callback function receives the metadata which incldues ipHeaderSize and transportHeaderSize. Both these values are zero. Are these correct values or should those be non-zero??
Can somebody help me to extract the data from packet in callout function and forward it safely to further layers?
Thank You.
These are the TCP packets. I looked into size and offset information. It seems the problem is consistent across packets.
I checked below values in (NET_BUFFER_LIST*)pClassifyData->pPacket.
NET_BUFFER_LIST->NetBUfferListHeader->NetBUfferListData->FirstNetBuffer->NetBuffe rHeader->NetBufferData->CurrentMdl->MappedSystemVa
First 24 bytes are only sent correctly and remaining are garbage.
For example total size of the packet is 0x36 + 0x18 = 0x4E I don't know what is there in first 0x36 bytes which is constant for all the packets. Is it a TCP/IP header? Second part 0x18 is the actual data which i sent.
I even tried with API NdisQueryMdl() to retrieve from MDL list.
So on the receiver side I get only 24 bytes correct and remaining is the garbage. How to read the full buffer from NET_BUFFER_LIST?
I have read that the combination of three things causes something like a 200ms delay with TCP: Nagle's algorithm, delayed acknowledgement, and the "write-write-read" combination. However, I cannot reproduce this delay with Java sockets and I am therefore not sure if I have understood correctly.
I am running a test on Windows 7 with Java 7 with two threads using sockets over the loopback address. I have not touched the tcpNoDelay option on any socket (false by default) nor played around with any TCP settings on the OS. The main piece of the code in the client is as below. The server is responding with a byte after each two bytes it receives from the client.
for (int i = 0; i < 100; i++) {
client.getOutputStream().write(1);
client.getOutputStream().write(2);
System.out.println(client.getInputStream().read());
}
I do not see any delay. Why not?
I believe you see delay acknowledgment.
You write 4 and 4 bytes to the socket. The server's TCP stack receives a segment (that probably contains at least 4 bytes from an int number) and wakes up the server application thread. This thread writes a byte back to the stream and this byte is sent to the client within ACK segment. I.e. TCP stack gives a chance to an application to send a reply immediately. So you see no delay.
You can write a dump of traffic and also make an experiment between two computers to see what really happens.
I established a connection with a client this way:
gen_tcp:listen(1234,[binary,{packet,0},{reuseaddr,true},{active,false},{recbuf,2048}]).
This code performs message processing:
loop(Socket)->
inet:setops(Socket,[{active,once}],
receive
{tcp,Socket,Data}->
handle(Data),
loop(Socket);
{Pid,Cmd}->
gen_tcp:send(Socket,Cmd),
loop(Socket);
{tcp_close,Socket}->
% ...
end.
My OS is Windows. When the size of the message is 1024 bytes, I lose bytes in Data. The server sends ACK + FIN to the client.
I believe that the Erlang is limited to 1024 bytes, therefore I defined recbuf.
Where the problem is: Erlang, Windows, hardware?
Thanks.
You may be setting the receive buffer far too small. Erlang certainly isn't limited to a 1024 byte buffer. You can check for yourself by doing the following in the shell:
{ok, S} = gen_tcp:connect("www.google.com", 80, [{active,false}]),
O = inet:getopts(S, [recbuf]),
gen_tcp:close(S),
O.
On Mac OS X I get a default receive buffer size of about 512Kb.
With {packet, 0} parsing, you'll receive tcp data in whatever chunks the network stack chooses to send it in, so you have to do message boundary parsing and buffering yourself. Do you have a reliable way to check message boundaries in the wire protocol? If so, receive the tcp data and append it to a buffer variable until you have a complete message. Then call handle on the complete message and remove the complete message from the buffer before continuing.
We could probably help you more if you gave us some information on the client and the protocol in use.