My goal is to send a TCP packet with empty data field, in order to test the socket with the remote machine.
I am using the OutputStream class's method of write(byte[] b).
my attempt:
outClient = ClientSocket.getOutputStream();
outClient.write(("").getBytes());
Doing so, the packet never show up on the wire. It works fine if "" is replaced by " " or any non-zero string.
I tried jpcap, which worked with me, but didn't serve my goal.
I thought of extending the OutputStream class, and implementing my own OutputStream.write method. But this is beyond my knowledge. Please give me advice if someone have already done such a thing.
If you just want to quickly detect silently dropped connections, you can use Socket.setKeepAlive( true ). This method ask TCP/IP to handle heartbeat probing without any data packets or application programming. If you want more control on the frequency of the heartbeat, however, you should implement heartbeat with application level packets.
See here for more details: http://mindprod.com/jgloss/socket.html#DISCONNECT
There is no such thing as an empty TCP packet, because there is no such thing as a TCP packet. TCP is a byte-stream protocol. If you send zero bytes, nothing happens.
To detect broken connections, short of keepalive which only helps you after two hours, you must rely on read timeouts and write exceptions.
Related
Problem
I want to run a load test with a high number of requests per second. I have written a socket sender and a receiver in Go. The sender sends a lot of packets to port 7357, each one containing the current time in nanoseconds. The receiver listens in port 7357 and parses each message, computing the latency.
The problem is that when reading I get multiple packets in one conn.Read(). I understand that this means that I am in fact sending multiple messages per packet: each conn.Write() does not send a socket packet, but it waits for some time and then gets coalesced with the next (or the next few) before sending.
Question
How can I make sure that each conn.Write() is sent individually through the socket as a separate packet? Note: I don't want to reinvent TCP, I just want to simulate the load from a number of external entities that send a message each.
Steps Taken
I have searched the documentation but there seems to be no conn.Flush() or similar. I have tried using a buffered writer:
writer := bufio.NewWriter(conn)
...
bytes, err := writer.Write(message)
err = writer.Flush()
No errors, but still I get mixed packets at the receiving end. I have also tried doing a fake conn.Read() of 0 bytes after every conn.Write(), but it didn't work either. Sending a message terminator such as \r\n does not seem to make any difference. Finally, Nagle algorithm is disabled by default, but I have called tcp.SetNoDelay(true) for good measure.
In Node.js I managed to do the trick with a setImmediate() after each socket.write(): setImmediate() waits for all I/O to finish before continuing. How can I do the same in Go so I get separate packets?
Code Snippets
Send:
func main() {
conn, _ := net.Dial("tcp", ":7357")
defer conn.Close()
for {
timestamp := strconv.FormatInt(time.Now().UnixNano(), 10)
conn.Write([]byte(timestamp))
conn.Read(buff)
}
}
Receive:
func main() {
listen, _ := net.Listen("tcp4", ":7357")
defer listen.Close()
for {
conn, _ := listen.Accept()
go handler(conn)
}
}
func handler(conn net.Conn) {
defer conn.Close()
var buf = make([]byte, 1024)
for {
conn.Read(buf)
data := string(buf[:n])
timestamp, _ := strconv.ParseInt(data, 10, 64)
elapsed := timestamp - time.Now().UnixNano()
log.Printf("Elapsed %v", elapsed)
}
}
Error handling has been removed for legibility, but it is thoroughly checked in the actual code. It crashes when running the strconv.ParseInt() the first time, with a value out of range error since it receives a lot of timestamps coalesced.
There used to a be a rule that before anyone was permitted to write any code that uses TCP, they were required to repeat the following sentence from memory and explain what it means: "TCP is not a message protocol, it is a reliable byte-stream protocol that does not preserve application message boundaries."
Aside from the fact that your suggested solution is simply not possible reliably with TCP, it is not the solution to reducing latency. If the network is overwhelmed, using more packets to send the same data will just make the latency worse.
TCP is a byte stream protocol. The service it provides is a stream of bytes. Period.
It seems that you want a low-latency message protocol that works over TCP. Great. Design one and implement it.
The main trick to getting low latency is to use application-level acknowledgements. The TCP ACK flag will piggy-back onto the acknowledgements, providing low latency.
Do not disable Nagling. That's a hack that's only needed when you can't design a proper protocol that's intended to work with TCP in the first place. It will make latency worse under non-ideal conditions for same reason the solution you suggested, even if it were possible, would be a poor idea.
But you MUST design and implement a message protocol or use an existing one. Your code is expecting TCP, which is not a message protocol, to somehow deliver messages to it. That is just not going to happen, period.
How can I make sure that each conn.Write() is sent individually through the socket as a separate packet? Note: I don't want to reinvent TCP, I just want to simulate the load from a number of external entities that send a message each.
Even if you could, that wouldn't do what you want anyway. Even if they were sent in separate packets, that would not guarantee that read on the other side wouldn't coalesce. If you want to send and receive messages, you need a message protocol which TCP is not.
In Node.js I managed to do the trick with a setImmediate() after each socket.write(): setImmediate() waits for all I/O to finish before continuing. How can I do the same in Go so I get separate packets?
You may have changed it from "happens not to work" to "happened to work when I tried it". But for the reasons I've explained, you can never make this work reliably and you are on a fool's errand.
If you want to send and receive messages, you need to precisely defined what a "message" will be and write code to send and receive them. There are no shortcuts that are reliable. TCP is a byte stream protocol, period.
If you care about latency and throughput, design an optimized message protocol to layer over TCP that optimizes these. Do not disable Nagle as Nagle is required to prevent pathological behavior. It should only be disabled when you cannot change the protocol and are stuck with a protocol that was not designed to layer on top of TCP. Disabling Nagle ties one hand behind your back and causes dramatically worse latency and throughput under poor network conditions by increasing the number of packets required to send the data even when that doesn't make any sense.
You probably want/need application-level acknowledgements. This works nicely with TCP because TCP ACKs will piggyback on the application-level acknowledgements.
You can read predefined number of bytes from the socket on each iteration, that might help, but you need to create your own protocol, that will be handled by your application. Without proto impossible to guarantee that everything will work stable, because on the receiver you cannot understand where is the begin and where is the end of message.
start listening client with netcat -l
go program opens a conn with net.DialTCP to said client.
kill the netcat
in go program, do conn.Write() with a []byte -> it runs fine without error!
it takes another conn.Write to get the error: broken pipe
The first write is the one where data loss happens, and I want to avoid. if i only get an error I know i can just keep the data and try again later.
I've seen https://stackoverflow.com/a/15071574/2757887 which is a very similar case and the explanation seems to apply here, but it still doesn't explain how to deal with the issue, if the tcp protocol I need to implement only does one-way communication.
I've sniffed the traffic with wireshark, and when i kill the netcat, I can see that it sends FIN to the go program, to which the go program replies with ACK. For some reason the go program doesn't immediately reply with it's own FIN - and i'm curious why that is, it might help with my problem - but there's probably a good reason for it.
Either way, from the "connection termination" section # http://en.wikipedia.org/wiki/Transmission_Control_Protocol, I conclude that the socket is in the CLOSE_WAIT state at this point, which I also confirmed with "netstat -np", which shows the socket going from ESTABLISHED to CLOSE_WAIT after killing netstat.
Looking at wireshark, the first conn.write results in a packet with push and ack fields set, and of course my payload. this is the write that succeeds fine in go.
then the old socket that used to belong to netstat sends RST,
which makes sure that as soon as i try to write in go (2nd write) it fails.
So my question is:
A) why can't I get an error on the first write? if the socket received the FIN and is in CLOSE_WAIT why does Go let me write to the socket and tell me all is fine?
B) is there any way I can check in Go whether the socket is in CLOSE_WAIT? and if so, I could for this purpose consider it closed and not do the write.
thanks,
Dieter
Fundamentally, a successful write only tells you that data has been queued to be sent to the other end. If you need to make sure the other end gets that data, even if the connection closes or errors, you must store a copy of the data until the other end provides you with an application-level acknowledgment.
The man page for recvfrom summarizes its behavior as "receive a message from a socket". If the socket is of type SOCK_STREAM or SOCK_DGRAM, is "message" synonymous with "packet"? If not, how does it differ?
My first thought was recvfrom works on stream sockets just because there's no reason to ban it. As in the famous quote:
"Unix was not designed to stop its users from doing stupid things, as that would also stop them from doing clever things." – Doug Gwyn
If it did what I expected it to do, you could use it like a combination read() and getpeername() since it returns the sender's address. That might be considered clever.
But then I tried it on Linux, and it didn't work that way. The source address buffer was unchanged and the length indicator was set to 0.
So now I'm not sure what to say except don't use it on stream sockets. It's not meant for them.
ADDENDUM: And no, even in my wildest dreams I wouldn't have expected it to give you access to packet boundaries in a TCP stream. Data that has been put through the tcp receiving mechanism simply isn't made of packets anymore.
Is it wise/safe to close() a socket directly after the last send()?
I know that TCP is supposed to try to deliver all remaining data in the send buffer even after closing the socket, but can I really count on that?
I'm making sure that there is no remaining data in my receive buffer so that no RST will be sent following my close.
In my case, the close is actually the very last statement of code before calling exit().
Will the TCP stack really continue to try and transmit the data even after the process sending it has terminated? Is that as reliable as waiting for an arbitrary timeout myself before calling close() by setting SO_LINGER?
That is, do the same TCP timeouts apply, or are they shorter? With a big send buffer and a slow connection, the time to actually transfer all the buffered data could be substantial, after all.
I'm not interested at all in being notified of the last byte sent; I just want them to eventually arrive at the remote host as reliably as possible.
Application layer acknowledgements are not an option (the protocol is HTTP, and I'm writing a small server).
I've been reading the The ultimate SO_LINGER page, or: why is my tcp not reliable blog post a lot. I recommend you read it too. It discusses edge cases of large data transfers with regards to TCP sockets.
I'm not the expert at SO_LINGER, but on my server code (still in active development) I do the following:
After the last byte is sent via send(), I call shutdown(sock, SHUT_WR) to trigger a FIN to be sent.
Then wait for a subsequent recv() call on that socket to return 0 (or recv returns -1 and errno is anything other that EAGAIN/EWOULDBLOCK).
Then the server does a close() on the socket.
The assumption is that the client will close his socket first after it has received all the bytes of the response.
But I do have a timeout enforced between the final send() and when recv() indicates EOF. If the client never closes his end of the connection, the server will give up waiting and close the connection anyway. I'm at 45-90 seconds for this timeout.
All of my sockets are non-blocking and I use poll/epoll to be notified of connection events as a hint to see if it's time to try calling recv() or send() again.
Application layer acknowledgements are not an option (the protocol is HTTP, and I'm writing a small server).
HTTP protocol doesn't suffer from this problem. A HTTP server is not supposed to close the connection in any normal operation. The client closes it after recv(), and it knows exactly how many bytes it expects.
And just to be clear, the answer is "no".
Yes, it is safe that send() then close() immediately.
the kernel will sent out all data in buffer and wait ack, then fin the socket gracefully.
is it possible in any common platform - say, in Windows - to write a servient process that creates a socket, binds it to exactly one local "address:port" (fixed), and then:
use it to listen for incoming connections (on the specified port)
while at the same time
use it as a client socket to connect to some other servient (having source port identical to the one it exposes to others) ?
that is (sorry for the syntax abuse):
mySocket=socket(); mySocket.bind(myaddress, 3000);
mySocket.connectTo(neighbour, whateverport); // and present to others as port 3000
mySocket.listen(); // and it listens on 3000
mySocket.accept();
?
iirc, it's not even possible/advisable to try, even in the case an API wouldn't complain, but maybe it's me that is playing too much by the book... so I thought of asking you
thanks a lot!
No, a socket cannot be used for both listening and connecting at the same time. connect() will return a WSAEINVAL error if listen() was already called, and listen() will return a WSAEISCONN error if connect() was already called. You need to use separate sockets.
And if you could, there's be all kinds of troubles that crop up. For example, if select() returns that the socket is readable, do you do a recv() or an accept()? You want 2 sockets to play those two roles.
What advantage is there in one socket? For example, if you were hoping to do a blocking read until something interesting happens (incoming connection, incoming data), there are alternatives. In that example, you'd use select() to block on two sockets at once. The result from select() tells you which socket is ready. That tells you if you want to accept() a new connection from the one socket or recv() new data from the other socket.