I'm not quite clear about the options http-request or http-response in HAProxy configuration.
Many of the parms seem to be used for modification of http request and response but what I found is it can be done using the regular option as well
What's the difference between
http-request set-src hdr(x-forwarded-for) #and
option forwardfor
Also what's the difference between:
connect timeout 5000
client timeout 5000
server timeout 5000 #And
http-request timeout 5000
I'm new to haproxy and from the documentation is written from configuration parameter perspective (like a api reference) instead of a use-case perspective (like a user-guide).
So If I asked an absurd question please do not mind and answer kindly. Thanks
What's the difference?
These first two are sort of opposites.
Configure HAProxy to use the contents of the X-Forward-For header to establish its internal concept of the source address of the request, instead of the actual IP address initiating the inbound connection:
http-request set-src hdr(x-forwarded-for)
Take the IP address of the inbound connection and add an X-Forwarded-For header for the benefit of downstream servers:
option forwardfor
Also what's the difference?
Let's take this one backwards.
First, this isn't valid in any version that I'm aware of:
http-request timeout 5000
I believe you mean this...
timeout http-request 5000
...which sets the timeout for the client to send complete, valid HTTP headers and an extra \r\n signifying end of headers. This timer doesn't usually apply to the body, if there is one -- only the request headers. If this timer fires, the transaction is aborted, a 408 Request Timeout is returned, and the client connection is forcibly closed. This timer stops once complete request headers have been received.
By default, this timeout only applies to the header part of the request,
and not to any data. As soon as the empty line is received, this timeout is
not used anymore.
http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4-timeout%20http-request
Note: http-request is something entirely different, and is used for manipulating the request during the request processing phase of the transaction lifecycle, before the request is sent to a back-end server.
Continuing, these aren't actually valid, either.
connect timeout 5000
client timeout 5000
server timeout 5000
It seems you've reversed the keywords. I believe you're thinking of these:
timeout connect 5000
That's the maximum time to wait for the back-end to accept our TCP connection by completing its share of the 3-way handshake. It has no correlation with timeout http-request, which is only timing the client sending the initial request. If this timer fires, the proxy will abort the transaction and return 503 Service Unavailable.
timeout client 5000
This one does overlap with timeout http-request, but not completely. If this timer is shorter than timeout http-request, the latter can never fire. This timer applies any time the server is expecting data from the client. The transaction aborts if this timer fires. I believe if this happens, the proxy just closes the connection.
timeout server 5000
This is time spent waiting for the server to send data. It also has no overlap with timeout http-request, since that window has already closed before this timer starts running. If we are waiting for the server to send data, and it is idle for this long, the transaction is aborted, a 504 Gateway Timeout error is by HAProxy, and the client connection is closed.
So, as you can see, the overlap here is actually pretty minimal between these three and timeout http-request.
You didn't really ask, but you'll find significant overlap between things like http-response set-header and rsp[i]rep, and http-request set-header and req[i]rep. The [req|rsp][i]rep keywords represent older functionality that is maintained for compatibility but largely obsoleted by the newer capabilities that have been introduced, and again, there's not as much overlap as there likely appears at first glance because the newer capabilities can do substantially more than the old.
I'm new to haproxy and from the documentation is written from configuration parameter perspective (like a api reference) instead of a use-case perspective (like a user-guide).
That seems like a fair point.
Related
I'm try to config haproxy to reuse connection(http keep alive) for reduce TCP establish connection cost
It's ok for normal request. But for health check, it's alway create new connection to check backend.
How can we re-use a connection for health check purpose.
Here is my config
balance roundrobin
mode http
option http-keep-alive
option httpchk GET /actuator/health HTTP/1.1\r\nHost:\ 10.1.1.2:8003
server micro-dev 10.1.1.2:8003 check maxconn 2000
Thanks
Few days agos, I have the same question about this header Connection: close which is unfortunately always added to the request.
The support replied that this behavior is normal.
In fact, the single request health check tells the server to close the connection once the response is returned.
It is not possible to change this behavior.
For more informations read this https://github.com/haproxy/haproxy/issues/560
How is HTTP Keep Alive implemented? Does it internally use TCP Keep Alive? If not, how does the server detect if the client is dead or alive?
I know this is an old question, but still:
HTTP Keep-Alive is a feature that allows HTTP client (usually browser) and server (webserver) to send multiple request/response pairs over the same TCP connection. This decreases latency for 2nd, 3rd,... HTTP request, decreases network traffic and similar.
TCP keepalive is a totally different beast. It keeps TCP connection opened by sending small packets. Additionally, when the packet is sent this serves as a check so the sender is notified as soon as connection drops (note that this is NOT the case otherwise - until we try to communicate through TCP connection we have no idea if it is ok or not).
To answer your questions about HTTP Keep-Alive:
How is HTTP Keep Alive implemented?
To put it simply, the HTTP server doesn't close the TCP connection after each response but waits some time if some other HTTP request will come over it too. After some timeout it closes it anyway.
Does it internally use TCP Keep Alive?
No, at least I see no point in it.
If not, how does the server detect if the client is dead or alive?
It doesn't - it doesn't need to. If a client sends a request, it will get the response. If the client doesn't send anything over TCP connection (maybe because the connection is dead) then a timeout will close the connection; client will of course notice this and will send request through another TCP connection if needed.
HTTP Keep-Alive is a feature of HTTP protocol. The web-server, implementing Keep-Alive Feature, has to check the connection/socket periodically (for incoming HTTP request) for the time span since it sent the last HTTP response (in case there was corresponding HTTP Request). If no HTTP request is received by the time of the configured keep-alive time (seconds) the web server closes the connection. No further HTTP request will be possible after the 'close' done by Web Server. On the other hand, TCP Keep-Alive is managed by OS in the TCP layer. HTTP Keep-Alive and TCP Keep-Alive is totally unrelated things.
HTTP keep-alive, a.k.a., HTTP persistent connection, is an instruction that allows a single TCP connection to remain open for multiple HTTP requests/responses.
By default, HTTP connections close after each request. When someone visits your site, their browser needs to create new connections to request each of the files that make up your web pages (e.g. images, Javascript, and CSS stylesheets), a process that can lead to high page load times.
Enabling the keep-alive header allows you to serve all web page resources over a single connection. Keep-alive also reduces both CPU and memory usage on your server.
Source: https://www.imperva.com/learn/performance/http-keep-alive/
http keep-alive is just making tcp living longer in order to transfer multi http request.After keep-alive timeout, the tcp connection will be closed.
tcp keep-alive is just a mechanism keeping the tcp connection,or check the tcp connection is not closed
Is there a reason why I should use application level heartbeating instead of TCP keepalives to detect stale connections, given that only Windows and Linux machines are involved in our setup?
It seems that the TCP keepalive parameters can't be set on a per-socket basis on Windows or OSX, that's why.
Edit: All parameters except the number of keepalive retransmissions can in fact be set on Windows (2000 onwards) too: http://msdn.microsoft.com/en-us/library/windows/desktop/dd877220%28v=vs.85%29.aspx
I was trying to do this with zeromq, but it just seems that zeromq does not support this on Windows?
From John Jefferies response : ZMQ Pattern Dealer/Router HeartBeating
"Heartbeating isn't necessary to keep the connection alive (there is a ZMQ_TCP_KEEPALIVE socket option for TCP sockets). Instead, heartbeating is required for both sides to know that the other side is still active. If either side does detect that the other is inactive, it can take alternative action."
TCP keepalives serve an entirely different function from application level heartbeating. A keepalive does just that, it keeps the TCP session active rather than allow it to time out after long periods of silence. This is important and good, and (if appropriate) you should use it in your application. But a TCP session dying due to inactivity is only one way that the connection can be severed between a pair of ZMQ sockets. One endpoint could lose power for 90 minutes and be offline, TCP keepalives wouldn't do squat for you in that scenario.
Application level heartbeating is not intended to keep the TCP session active, expecting you to rely on keepalives for that function if possible. Heartbeating is there to tell your application that the connection is in fact still active and the peer socket is still functioning properly. This would tell you that your peer is unavailable so you can behave appropriately, by caching messages, throwing an exception, sending an alert, etc etc etc.
In short:
a TCP keepalive is intended to keep the connection alive (but doesn't protect against all disconnection scenarios)
an app-level heartbeat is intended to tell your application if the connection is alive
I'm using LWP::UserAgent to communicate with webservices on several servers; the servers are contacted one at a time. Each response might take up to 30 minutes to finish, so I set the LWP timeout to 30 minutes.
Unfortunately the same timeout also applies, if the server is not reachable at all (e.g. the webserver is down). So my application waits 30 minutes for a server, which is not running.
Is it feasable to set two seperate timeouts?
a short one, which waits for the connection to be established.
a longer one, which waits for the response, once the connection has been established.
The same timeout doesn't "also apply" if the server is not reachable. The timeout option works in a very specific way:
The request is aborted if no activity on the connection to the server is
observed for timeout seconds. This means that the time it takes for the
complete transaction and the request() method to actually return might be
longer.
As long as data is being passed, the timeout won't be triggered. You can use callback functions (see the REQUEST METHODS section of the docs) to check how long data transfer has been going on, and to exit entirely if desired.
I have a question about my haproxy config:
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 syslog emerg
maxconn 4000
quiet
user haproxy
group haproxy
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option abortonclose
option dontlognull
option httpclose
option httplog
option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is not found
timeout client 300000 # 5 min timeout for client
timeout server 300000 # 5 min timeout for server
stats enable
listen http_proxy localhost:81
balance roundrobin
option httpchk GET /empty.html
server server1 myip:80 maxconn 15 check inter 10000
server server2 myip:80 maxconn 15 check inter 10000
As you can see it is straight forward, but I am a bit confused about how the maxconn properties work.
There is the global one and the maxconn on the server, in the listen block. My thinking is this: the global one manages the total number of connections that haproxy, as a service, will queue or process at one time. If the number gets above that, it either kills the connection, or pools in some linux socket? I have no idea what happens if the number gets higher than 4000.
Then you have the server maxconn property set at 15. First off, I set that at 15 because my php-fpm, this is forwarding to on a separate server, only has so many child processes it can use, so I make sure I am pooling the requests here, instead of in php-fpm. Which I think is faster.
But back on the subject, my theory about this number is each server in this block will only be sent 15 connections at a time. And then the connections will wait for an open server. If I had cookies on, the connections would wait for the CORRECT open server. But I don't.
So questions are:
What happens if the global connections get above 4000? Do they die? Or pool in Linux somehow?
Are the global connection related to the server connections, other than the fact you can't have a total number of server connections greater than global?
When figuring out the global connections, shouldn't it be the amount of connections added up in the server section, plus a certain percentage for pooling? And obviously you have other restrains on the connections, but really it is how many you want to send to the proxies?
Thank you in advance.
Willy got me an answer by email. I thought I would share it. His answers are in bold.
I have a question about my haproxy config:
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 syslog emerg
maxconn 4000
quiet
user haproxy
group haproxy
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option abortonclose
option dontlognull
option httpclose
option httplog
option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is not found
timeout client 300000 # 5 min timeout for client
timeout server 300000 # 5 min timeout for server
stats enable
listen http_proxy localhost:81
balance roundrobin
option httpchk GET /empty.html
server server1 myip:80 maxconn 15 check inter 10000
server server2 myip:80 maxconn 15 check inter 10000
As you can see it is straight forward, but I am a bit confused about how the
maxconn properties work.
There is the global one and the maxconn on the server, in the listen block.
And there is also another one in the listen block which defaults to something
like 2000.
My thinking is this: the global one manages the total number of connections
that haproxy, as a service, will que or process at one time.
Correct. It's the per-process max number of concurrent connections.
If the number
gets above that, it either kills the connection, or pools in some linux
socket?
The later, it simply stops accepting new connections and they remain in the
socket queue in the kernel. The number of queuable sockets is determined
by the min of (net.core.somaxconn, net.ipv4.tcp_max_syn_backlog, and the
listen block's maxconn).
I have no idea what happens if the number gets higher than 4000.
The excess connections wait for another one to complete before being
accepted. However, as long as the kernel's queue is not saturated, the
client does not even notice this, as the connection is accepted at the
TCP level but is not processed. So the client only notices some delay
to process the request.
But in practice, the listen block's maxconn is much more important,
since by default it's smaller than the global one. The listen's maxconn
limits the number of connections per listener. In general it's wise to
configure it for the number of connections you want for the service,
and to configure the global maxconn to the max number of connections
you let the haproxy process handle. When you have only one service,
both can be set to the same value. But when you have many services,
you can easily understand it makes a huge difference, as you don't
want a single service to take all the connections and prevent the
other ones from working.
Then you have the server maxconn property set at 15. First off, I set that at
15 because my php-fpm, this is forwarding to on a separate server, only has
so many child processes it can use, so I make sure I am pooling the requests
here, instead of in php-fpm. Which I think is faster.
Yes, not only it should be faster, but it allows haproxy to find another
available server whenever possible, and also it allows it to kill the
request in the queue if the client hits "stop" before the connection is
forwarded to the server.
But back on the subject, my theory about this number is each server in this
block will only be sent 15 connections at a time. And then the connections
will wait for an open server. If I had cookies on, the connections would wait
for the CORRECT open server. But I don't.
That's exactly the principle. There is a per-proxy queue and a per-server
queue. Connections with a persistence cookie go to the server queue and
other connections go to the proxy queue. However since in your case no
cookie is configured, all connections go to the proxy queue. You can look
at the diagram doc/queuing.fig in haproxy sources if you want, it explains
how/where decisions are taken.
So questions are:
What happens if the global connections get above 4000? Do they die? Or
pool in Linux somehow?
They're queued in linux. Once you overwhelm the kernel's queue, then they're
dropped in the kernel.
Are the global connection related to the server connections, other than
the fact you can't have a total number of server connections greater than
global?
No, global and server connection settings are independant.
When figuring out the global connections, shouldn't it be the amount of
connections added up in the server section, plus a certain percentage for
pooling? And obviously you have other restrains on the connections, but
really it is how many you want to send to the proxies?
You got it right. If your server's response time is short, there is nothing
wrong with queueing thousands of connections to serve only a few at a time,
because it substantially reduces the request processing time. Practically,
establishing a connection nowadays takes about 5 microseconds on a gigabit
LAN. So it makes a lot of sense to let haproxy distribute the connections
as fast as possible from its queue to a server with a very small maxconn.
I remember one gaming site queuing more than 30000 concurrent connections
and running with a queue of 30 per server ! It was an apache server, and
apache is much faster with small numbers of connections than with large
numbers. But for this you really need a fast server, because you don't
want to have all your clients queued waiting for a connection slot because
the server is waiting for a database for instance.
Also something which works very well is to dedicate servers. If your site
has many statics, you can direct the static requests to a pool of servers
(or caches) so that you don't queue static requests on them and that the
static requests don't eat expensive connection slots.
Hoping this helps,
Willy