I have a client-server (Java) application using persistent TCP connections, but sometimes the Server receives java.io.IOException: Connection reset by peer exception when trying to write on the socket, however I don't see any error in the Client log.
This RST is probably caused by an intermediate proxy/router, but if that's the case, should this be seen on the client as well?
If the RST is sent by the client, it can be seen on it using a packet sniffer such as wireshark. However, it won't show up in any user-level sockets since it's sent by the OS as a response to various erroneous inputs (such as connection attempts to a closed port).
If the RST is sent by the network, then it's pretending to be the client to sever the connection. It can do so in one direction, or in both of them. In that case, the client might not see anything, except for a RST sent by the actual server when the client continues to send data to a connection it perceives as open, while the server sees it as closed.
Try capturing the traffic on both the server and the client, see where the resets are coming from.
Related
I am working on a product which uses ZeroMQ (version 4.0.1).
The server and client communicate based on ZeroMQ ROUTER-socket.
To read socket events, server and client also create socket-monitor sockets (PAIR). There are three ports on which server binds and listens. Out of these three ports, one port is in a non-secured mode. Other two ports are using md5-authentication.
The issue I am facing is that, both the server and the client spontaneously receive socket disconnect for one of the secure port sockets (please see a log below). I have checked multiple times that server and client both have L3 reachability to each other.
What else I should check for?
What really triggers this error scenario?
zmq_print_callback:ZmQ: int zmq::stream_engine_t::read(void*, size_t):923
Stream engine recv():
TCP socket (187) to unknown:0 was disconnected
with error 107 [Transport endpoint is not connected]
Below sequence of events can trigger this error on server
Server receives ACCEPTED event for clientY and gets FD1.
Link-flap/network issue happens and clientY disconnects but server does not receive this disconnect.
Network recovers and clientY connects back to server.
Server receives ACCEPTED event for clientY and gets FD2. However, packets sent to this sockets does not go out of the server.
After 1 min or so, clientY receives "Transport endpoint is not connected error" for FD1.
Application can use this to treat as client disconnect.
I have a system with server/client applications. The client will send in socket connection request and the server will accept the socket connection when it's working correctly. However, in some situations (most likely due to ungraceful socket disconnection like system shutdown on client side or crash), the client will not be able to reconnect to the server application. The Wireshark capture shows the client will continue to try to connect; but after going through SYN->SYN,ACK->ACK, the server application will send RST. At this point, sometimes the netstat -an will show the connection is in CLOSE_WAIT state and other times would not show this connection. The capture shows 'Acknowledgment Number: Broken TCP. The ackowledge field is nonzero while the ACK flag is not set.
My questions is why the server application would send this RST?
On the server side: I am using epoll_wait to monitor the possible read IO on a TCP socket.
On the client side: I have a single threaded app to write to the socket that's connected to the server.
The problem is, sometimes epoll_wait doesn't recognize there is new IO to read even after a new message is sent from the client. (I confirmed the message is indeed received by the server using wireshark) So the client is hanging waiting on the response from server. BUT: if I kill the client connection, epoll_wait does get notified!
Originally I am using EPOLLET and thought it would be a problem. But this issue still exists after removing EPOLLET.
Is there any tool that I can use to debug this? (e.g, outside of server process, to confirm that there is IO on the server socket queue but epoll_wait doesn't process it?) Any thought or guidance on how to debug this would be appreciated.
I'm writing an application that attempts to do the following:
create a TCP server listening on an available port
create a TCP socket that connects to the server
have the server socket write data to the client
have the server socket close its end of the connection
have the client write a message to the server
Here's where the problem lies. When I attempt to run the application, the TCP exchange goes like this:
The first three packets establish the three-way handshake, and the fourth and fifth packets are the transmission of the data written by the server and its acknowledgement.
As expected, the server socket sends a packet with the FIN flag set to indicate that it is closing its end of the connection. The client acknowledges this and then attempts to write its data to the socket. The server immediately sends an RST packet, terminating the connection prematurely.
Why does this happen?
Note: the above capture was done on Windows 8.1.
The sender cannot send data after a [FIN]. Such an action will result in the receiver issuing an [RST].
The FIN probably indicates that the server has fully closed the connection in both directions. In this case if it receives any further data on the connection it will issue an RST. This suggests an application protocol error on your part. If the server sends a reply and then closes the socket, the client can't send anything else via that connection.
Possibly you need your server to call shutdown() with SHUT_WR and then read something else from the client before closing the socket. Or possibly you're just doing it wrong.
What is the meaning of the "connection reset by peer" error on a TCP connection? Is it a fatal error or just a notification or related to the network failure?
It's fatal. The remote server has sent you a RST packet, which indicates an immediate dropping of the connection, rather than the usual handshake. This bypasses the normal half-closed state transition. I like this description:
"Connection reset by peer" is the TCP/IP equivalent of slamming the phone back on the hook. It's more polite than merely not replying, leaving one hanging. But it's not the FIN-ACK expected of the truly polite TCP/IP converseur.
This means that a TCP RST was received and the connection is now closed. This occurs when a packet is sent from your end of the connection but the other end does not recognize the connection; it will send back a packet with the RST bit set in order to forcibly close the connection.
This can happen if the other side crashes and then comes back up or if it calls close() on the socket while there is data from you in transit, and is an indication to you that some of the data that you previously sent may not have been received.
It is up to you whether that is an error; if the information you were sending was only for the benefit of the remote client then it may not matter that any final data may have been lost. However you should close the socket and free up any other resources associated with the connection.
one of the reasons for seeing this error and having trouble connecting to the server is that you enabled the firewall in the UNIX machine and forgot to add a rule to accept ssh connection. search in your WPS provider and you will find a way to connect to you machine and add this rules:
ufw allow ssh && ufw allow 22