I have a system with server/client applications. The client will send in socket connection request and the server will accept the socket connection when it's working correctly. However, in some situations (most likely due to ungraceful socket disconnection like system shutdown on client side or crash), the client will not be able to reconnect to the server application. The Wireshark capture shows the client will continue to try to connect; but after going through SYN->SYN,ACK->ACK, the server application will send RST. At this point, sometimes the netstat -an will show the connection is in CLOSE_WAIT state and other times would not show this connection. The capture shows 'Acknowledgment Number: Broken TCP. The ackowledge field is nonzero while the ACK flag is not set.
My questions is why the server application would send this RST?
Related
I am working on a product which uses ZeroMQ (version 4.0.1).
The server and client communicate based on ZeroMQ ROUTER-socket.
To read socket events, server and client also create socket-monitor sockets (PAIR). There are three ports on which server binds and listens. Out of these three ports, one port is in a non-secured mode. Other two ports are using md5-authentication.
The issue I am facing is that, both the server and the client spontaneously receive socket disconnect for one of the secure port sockets (please see a log below). I have checked multiple times that server and client both have L3 reachability to each other.
What else I should check for?
What really triggers this error scenario?
zmq_print_callback:ZmQ: int zmq::stream_engine_t::read(void*, size_t):923
Stream engine recv():
TCP socket (187) to unknown:0 was disconnected
with error 107 [Transport endpoint is not connected]
Below sequence of events can trigger this error on server
Server receives ACCEPTED event for clientY and gets FD1.
Link-flap/network issue happens and clientY disconnects but server does not receive this disconnect.
Network recovers and clientY connects back to server.
Server receives ACCEPTED event for clientY and gets FD2. However, packets sent to this sockets does not go out of the server.
After 1 min or so, clientY receives "Transport endpoint is not connected error" for FD1.
Application can use this to treat as client disconnect.
I have a Java Socket Server running on a Windows Server 2008.
When using a multi-threads client to send several TCP connections at the same time, the client always get the "Errno 111 connection refused" error after the establishment of the first connection.
Here's the capture trace of Wireshark (10.1.3.136 is the server, 10.34.10.132 is the client): Trace and the specific red trace goes here:Trace2
So, what's the issue?
If I delay-launch the thread by more than 5ms, or use a centos as the server, the errors disapperar. No exceptions are found in the server trace file.
The issue is that you have filled the backlog queue, whereupon Windows starts issuing resets to further incoming connection requests.
This could be because you specified a small backlog value, but the more likely cause is that your server is simply not accepting connections fast enough: your accept loop is fiddling around doing other things, such as DNS calls or even I/O with the client, all of which should be done in the client's thread. All the accept loop should do is accept sockets and start threads.
I am facing an issue with tcp connection..
I have a number of clients connected to the a remote server over tcp .
Now,If due to any issue i am not able to reach my server , after the successful establishment of the tcp connection , i do not receive any error on the client side .
On client end if i do netstat , it shows me that clients are connected the remote server , even though i am not able to ping the server.
So,now i am in the case where the server shows it is not connected to any client and on another end the client shows it is connected the server.
I have tested this for websocket also with node.js , but the same behavior persists over there also .
I have tried to google it around , but no luck .
Is there any standard solution for that ?
This is by design.
If two endpoints have a successful socket (TCP) connection between each other, but aren't sending any data, then the TCP state machines on both endpoints remains in the CONNECTED state.
Imagine if you had a shell connection open in a terminal window on your PC at work to a remote Unix machine across the Internet. You leave work that evening with the terminal window still logged in and at the shell prompt on the remote server.
Overnight, some router in between your PC and the remote computer goes out. Hours later, the router is fixed. You come into work the next day and start typing at the shell prompt. It's like the loss of connectivity never happened. How is this possible? Because neither socket on either endpoint had anything to send during the outage. Given that, there was no way that the TCP state machine was going to detect a connectivity failure - because no traffic was actually occurring. Now if you had tried to type something at the prompt during the outage, then the socket connection would eventually time out within a minute or two, and the terminal session would end.
One workaround is to to enable the SO_KEEPALIVE option on your socket. YMMV with this socket option - as this mode of TCP does not always send keep-alive messages at a rate in which you control.
A more common approach is to just have your socket send data periodically. Some protocols on top of TCP that I've worked with have their own notion of a "ping" message for this very purpose. That is, the client sends a "ping" message over the TCP socket every minute and the server responds back with "pong" or some equivalent. If neither side gets the expected ping/pong message within N minutes, then the connection, regardless of socket error state, is assumed to be dead. This approach of sending periodic messages also helps with NATs that tend to drop TCP connections for very quiet protocols when it doesn't observe traffic over a period of time.
I'm writing an application that attempts to do the following:
create a TCP server listening on an available port
create a TCP socket that connects to the server
have the server socket write data to the client
have the server socket close its end of the connection
have the client write a message to the server
Here's where the problem lies. When I attempt to run the application, the TCP exchange goes like this:
The first three packets establish the three-way handshake, and the fourth and fifth packets are the transmission of the data written by the server and its acknowledgement.
As expected, the server socket sends a packet with the FIN flag set to indicate that it is closing its end of the connection. The client acknowledges this and then attempts to write its data to the socket. The server immediately sends an RST packet, terminating the connection prematurely.
Why does this happen?
Note: the above capture was done on Windows 8.1.
The sender cannot send data after a [FIN]. Such an action will result in the receiver issuing an [RST].
The FIN probably indicates that the server has fully closed the connection in both directions. In this case if it receives any further data on the connection it will issue an RST. This suggests an application protocol error on your part. If the server sends a reply and then closes the socket, the client can't send anything else via that connection.
Possibly you need your server to call shutdown() with SHUT_WR and then read something else from the client before closing the socket. Or possibly you're just doing it wrong.
I have a client-server (Java) application using persistent TCP connections, but sometimes the Server receives java.io.IOException: Connection reset by peer exception when trying to write on the socket, however I don't see any error in the Client log.
This RST is probably caused by an intermediate proxy/router, but if that's the case, should this be seen on the client as well?
If the RST is sent by the client, it can be seen on it using a packet sniffer such as wireshark. However, it won't show up in any user-level sockets since it's sent by the OS as a response to various erroneous inputs (such as connection attempts to a closed port).
If the RST is sent by the network, then it's pretending to be the client to sever the connection. It can do so in one direction, or in both of them. In that case, the client might not see anything, except for a RST sent by the actual server when the client continues to send data to a connection it perceives as open, while the server sees it as closed.
Try capturing the traffic on both the server and the client, see where the resets are coming from.