We are facing a problem where our client lets name it A. Is attempting to connect DB server (Cockroach) name B load balanced via ha-proxy
A < -- > haproxy < -- > B
Now at every, while our client A is receiving Broken Pipe error.
But I'm not able to understand why?
Cockroach server already has the below default value i.e 60 seconds.
COCKROACH_SQL_TCP_KEEP_ALIVE ## which is enabled to send for 60 second
Plus our haproxy config has the following setting.
defaults
mode tcp
# Timeout values should be configured for your specific use.
# See: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-timeout%20connect
timeout connect 10s
timeout client 1m
timeout server 1m
# TCP keep-alive on client side. Server already enables them.
option clitcpka
option clitcpka
So what is causing the TCP connection to drop when the keepalive is enabled on every end.
Keepalive is what makes connections go away if one of the end points has died without closing the connection. Investigate in that direction.
The only time keepalive actually keeps the connection alive is in connection with an ill-configured firewall that drops idle connections.
Related
I've Azure VM running Linux(ubuntu 18.06). I'm running Python socket server there. Now the problem is, any socket client which is not doing any activity for 4 minutes is getting disconnected. I've gone through https://github.com/wbuchwalter/azure-content/blob/master/includes/guidance-tcp-session-timeout-include.md and changed /etc/sysctl.conf on my linux instance, but it's not working. Now my question is,
1. Is it possible to change keepalive with default public ip of azre vm, because the link says "outbound using SNAT (Source NAT). This timeout is set to 4 minutes, and cannot be adjusted."
Inbound TCP timeout for Public IP can be controlled. For outbound, the default value is 4 minutes and cannot be changed. You an still keep the session active by sending keep-alive packets.
I am using haproxy 1.6.4 as TCP(not HTTP) proxy.
My clients are making TCP requests. They do not wait for any response, they just send the data and close the connection.
How haproxy behaves when all back-end nodes are down?
I see that (from the client point of view) haproxy is accepting incomming connections.
Haproxy statistics show that front-end has status OPEN, he is accepting connections.
Number of sessions and bytes-in increases for frontend, but not for back-end (he is DOWN).
Is haproxy buffering incoming TCP requests, and will pass them to the back-end once back-end is up?
If yes, it is possible to configure this buffer size? Where data is buffered (in memory, disk?)
Is this possible to turn off front-end (do not accept incoming TCP connections) when all back-end nodes are DOWN?
Edit:
when backend started, I see that
* backend in-bytes and sessions is equal to front-end number of sessions
* but my one and only back-end node has fever number of bytes-in, fever sessions and has errors.
So, it seems that in default configuration there is no tcp buffering.
Data is accepted by haproxy even if all backend nodes are down, but this data is lost.
I would prefer to turn off tcp front-end when there are no backend servers- so client connections would be rejected. Is that possible?
edit:
haproxy log is
Jul 15 10:02:32 172.17.0.2 haproxy[1]: 185.130.180.3:11319
[15/Jul/2016:10:02:32.335] tcp-in app/ -1/-1/0 0 SC \0/0/0/0/0
0/0 908
my log format is
%ci:%cp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ \%ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %U
What I understand from log:
there are no backeend servers
termination state SC translates to
S : the TCP session was unexpectedly aborted by the server, or the
server explicitly refused it.
C : the proxy was waiting for the CONNECTION to establish on the
server. The server might at most have noticed a connection attempt.
I don't think what you are looking for is possible. HAproxy handles the two sides of the connection (frontend, backend) separately. The incoming TCP connection is established first and then HAproxy looks for a matching destination for it.
We have a TCP application that receives connections in a protocol that we did not design and don’t control.
This protocol will assume that if it can establish a TCP connection, then it can send a message and that message is acknowledged.
This works ok if connecting directly to a machine, if the machine or application is down, the tcp connection will be refused or dropped and the client will attempt to redeliver the message.
When we use AWS elastic load balancer, ELB will establish a TCP connection with the client, regardless of whether there is an available back-end server to fulfil the request.
As a result if our application or server crashes then we lose messages.
ELB will close the TCP connection shortly thereafter, but its not good enough.
Is there a way to make ELB, only establish a connection if it can reach the back-end server?
What options do we have (within the AWS ecosystem), of balancing a TCP based service, while still refusing connections if they cannot be served.
I don't think that's achievable through ELB. By design a load balancer will manage 2 sets of connections (frontend - LB and LB - backend). The load balancer will attempt to minimize the time it takes to serve the traffic it receives. This means that the FE-LB connection will be established as the LB looks for a Backend connection to use / reuse. The case in which all of the Backend hosts are dead is such an edge case that you end up with the behavior you are seeing. Normally it's not a big deal as the requested will just get disconnected once the LB figures out that it cannot server the traffic.
Back to your protocol: to me it seem really weird that you would interpret the ability to establish a connection as equal to message delivery. It sounds like you're using TCP but not waiting for the confirmations that the message were actually received at the destination. To me that seems wrong and will get you in trouble eventually with or without a load balancer.
And not to sound too pessimistic (I do understand we are not living in an ideal world) what I would do in this specific scenario, if you can deploy additional software on the client, would be to use a tcp proxy on the client that would get disabled automatically whenever the load balancer is unhealthy/unable to serve traffic. Instruct the client to connect to this proxy. Far from ideal but it should do the trick.
You could create a health check from your ELB to verify if the backend EC2 instances respond on the TCP port. See ELB Health Checks
Then, you monitor the health status of the EC2 instances sent by the ELB to CloudWatch.
Once you determine that none of the EC2 instances are responding on the TCP port, you can remove the TCP listener from the ELB. See Delete ELB Listeners
Hopefully, at that point the ELB stops accepting TCP connections.
Note, I have not tested this solution.
We are using haproxy for thrift(rpc) server load balancing in tcp mode. But we've encountered one problem when backend server restarts.
When our thrift(rpc) server restarts, it first stop listening on the port which haproxy is configured to connect, but will still process running requests until they are all done(graceful restart).
So during restarting period, there are still connected sockets made from client to backend server via haproxy while backend server is not accepting any new connections, but haproxy still treats this backend server as healthy, and will dispatch new connections to this server. Any new connections dispatched to this server will take quite a long time to connect, and then timeout.
Is there any way to notify haproxy that server has stop listening and not to dispatch any connection to it?
I've tried following:
timeout connect set to very low + redispatch + retry 3
option tcp-check
Both not solve the problem.
I have a question about my haproxy config:
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 syslog emerg
maxconn 4000
quiet
user haproxy
group haproxy
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option abortonclose
option dontlognull
option httpclose
option httplog
option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is not found
timeout client 300000 # 5 min timeout for client
timeout server 300000 # 5 min timeout for server
stats enable
listen http_proxy localhost:81
balance roundrobin
option httpchk GET /empty.html
server server1 myip:80 maxconn 15 check inter 10000
server server2 myip:80 maxconn 15 check inter 10000
As you can see it is straight forward, but I am a bit confused about how the maxconn properties work.
There is the global one and the maxconn on the server, in the listen block. My thinking is this: the global one manages the total number of connections that haproxy, as a service, will queue or process at one time. If the number gets above that, it either kills the connection, or pools in some linux socket? I have no idea what happens if the number gets higher than 4000.
Then you have the server maxconn property set at 15. First off, I set that at 15 because my php-fpm, this is forwarding to on a separate server, only has so many child processes it can use, so I make sure I am pooling the requests here, instead of in php-fpm. Which I think is faster.
But back on the subject, my theory about this number is each server in this block will only be sent 15 connections at a time. And then the connections will wait for an open server. If I had cookies on, the connections would wait for the CORRECT open server. But I don't.
So questions are:
What happens if the global connections get above 4000? Do they die? Or pool in Linux somehow?
Are the global connection related to the server connections, other than the fact you can't have a total number of server connections greater than global?
When figuring out the global connections, shouldn't it be the amount of connections added up in the server section, plus a certain percentage for pooling? And obviously you have other restrains on the connections, but really it is how many you want to send to the proxies?
Thank you in advance.
Willy got me an answer by email. I thought I would share it. His answers are in bold.
I have a question about my haproxy config:
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 syslog emerg
maxconn 4000
quiet
user haproxy
group haproxy
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option abortonclose
option dontlognull
option httpclose
option httplog
option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is not found
timeout client 300000 # 5 min timeout for client
timeout server 300000 # 5 min timeout for server
stats enable
listen http_proxy localhost:81
balance roundrobin
option httpchk GET /empty.html
server server1 myip:80 maxconn 15 check inter 10000
server server2 myip:80 maxconn 15 check inter 10000
As you can see it is straight forward, but I am a bit confused about how the
maxconn properties work.
There is the global one and the maxconn on the server, in the listen block.
And there is also another one in the listen block which defaults to something
like 2000.
My thinking is this: the global one manages the total number of connections
that haproxy, as a service, will que or process at one time.
Correct. It's the per-process max number of concurrent connections.
If the number
gets above that, it either kills the connection, or pools in some linux
socket?
The later, it simply stops accepting new connections and they remain in the
socket queue in the kernel. The number of queuable sockets is determined
by the min of (net.core.somaxconn, net.ipv4.tcp_max_syn_backlog, and the
listen block's maxconn).
I have no idea what happens if the number gets higher than 4000.
The excess connections wait for another one to complete before being
accepted. However, as long as the kernel's queue is not saturated, the
client does not even notice this, as the connection is accepted at the
TCP level but is not processed. So the client only notices some delay
to process the request.
But in practice, the listen block's maxconn is much more important,
since by default it's smaller than the global one. The listen's maxconn
limits the number of connections per listener. In general it's wise to
configure it for the number of connections you want for the service,
and to configure the global maxconn to the max number of connections
you let the haproxy process handle. When you have only one service,
both can be set to the same value. But when you have many services,
you can easily understand it makes a huge difference, as you don't
want a single service to take all the connections and prevent the
other ones from working.
Then you have the server maxconn property set at 15. First off, I set that at
15 because my php-fpm, this is forwarding to on a separate server, only has
so many child processes it can use, so I make sure I am pooling the requests
here, instead of in php-fpm. Which I think is faster.
Yes, not only it should be faster, but it allows haproxy to find another
available server whenever possible, and also it allows it to kill the
request in the queue if the client hits "stop" before the connection is
forwarded to the server.
But back on the subject, my theory about this number is each server in this
block will only be sent 15 connections at a time. And then the connections
will wait for an open server. If I had cookies on, the connections would wait
for the CORRECT open server. But I don't.
That's exactly the principle. There is a per-proxy queue and a per-server
queue. Connections with a persistence cookie go to the server queue and
other connections go to the proxy queue. However since in your case no
cookie is configured, all connections go to the proxy queue. You can look
at the diagram doc/queuing.fig in haproxy sources if you want, it explains
how/where decisions are taken.
So questions are:
What happens if the global connections get above 4000? Do they die? Or
pool in Linux somehow?
They're queued in linux. Once you overwhelm the kernel's queue, then they're
dropped in the kernel.
Are the global connection related to the server connections, other than
the fact you can't have a total number of server connections greater than
global?
No, global and server connection settings are independant.
When figuring out the global connections, shouldn't it be the amount of
connections added up in the server section, plus a certain percentage for
pooling? And obviously you have other restrains on the connections, but
really it is how many you want to send to the proxies?
You got it right. If your server's response time is short, there is nothing
wrong with queueing thousands of connections to serve only a few at a time,
because it substantially reduces the request processing time. Practically,
establishing a connection nowadays takes about 5 microseconds on a gigabit
LAN. So it makes a lot of sense to let haproxy distribute the connections
as fast as possible from its queue to a server with a very small maxconn.
I remember one gaming site queuing more than 30000 concurrent connections
and running with a queue of 30 per server ! It was an apache server, and
apache is much faster with small numbers of connections than with large
numbers. But for this you really need a fast server, because you don't
want to have all your clients queued waiting for a connection slot because
the server is waiting for a database for instance.
Also something which works very well is to dedicate servers. If your site
has many statics, you can direct the static requests to a pool of servers
(or caches) so that you don't queue static requests on them and that the
static requests don't eat expensive connection slots.
Hoping this helps,
Willy