I'm going through https://www.haproxy.com/blog/four-examples-of-haproxy-rate-limiting/ but unable to grok how to write a configuration which rate-limits based on the Host header. Does the following look alright?
frontend website
bind :80
stick-table type string size 100k expire 30s store http_req_rate(10s)
# What do I put here?
http-request track-sc0 request.header(Host)
# what does sc_http_req_rate(0) really mean?
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 20 }
default_backend servers
Also, what is an easy way to validate whether a rate-limiting configuration works as intended? (not simply test the syntactical validity of the the config)
I figured it out:
stick-table type string size 100k expire 300s store http_req_rate(60s)
http-request track-sc0 hdr(Host)
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 30 }
Related
I’m looking for the equivalent of this backend code block below for requests to www.example.com and example.com.
http-response set-header X-Target example.com
server web-servers site.example.com:80 check
I take all the requests to www.example.com but I want to serve them to site.example.com using haproxy. There are several variations of example.com so I would like to have a list of allowed domains and then if they're allowed I would like to have a backend code block like below where I could use %[req.hdr(Host)] as the value in the http-response X-Target statement.
http-response set-header X-Target %[req.hdr(Host)]
server web-servers site.%[req.hdr(Host),regsub(^www.,,)]:80 check
HA-Proxy version 2.1.4-273103-54 2020/05/07 - https://haproxy.org/
I’m getting this error when I try haproxy -c -f haproxy.test
[root#pm-prod-haproxy05 haproxy]# haproxy -c -f haproxy.test
[ALERT] 259/180932 (16116) : parsing [haproxy.test:40]: ‘http-response set-header’: sample fetch may not be reliably used here because it needs ‘HTTP request headers’ which is not available here.
[ALERT] 259/180932 (16116) : Error(s) found in configuration file : haproxy.test
[root#pm-prod-haproxy05 haproxy]#
I've also tried this:
http-request set-header X-Target %[req.hdr(Host)]
http-request set-header X-Hostname %[req.hdr(Host),regsub(^www.,site.,)]
http-request web-server do-lookup(hdr(X-Hostname))
server web-servers web-server:80 check
This is my full configuration.
global
log 127.0.0.1 local2 debug
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
option httplog
log global
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend frontend-http
bind *:80
bind *:443
acl redirect path_beg -i /rd
use_backend backend-tracking if redirect
default_backend backend-default
backend backend-default
option forwardfor
http-response set-header X-Publishing-system website
http-response set-header X-Target %[req.hdr(Host)]
server web-servers site.%[req.hdr(Host),regsub(^www.,,)]:80 check
backend backend-tracking
option forwardfor
http-response set-header X-Publishing-system redirect
http-request set-uri %[url,regsub(^/rd,/,)]
server web-hp www.trackingplatform.com:80 check
About Header manipulation
As the ALERT message say you can't use request header in the response. You should replace the following line.
Wrong line
http-response set-header X-Target %[req.hdr(Host)]
Right Line
http-request set-header X-Target %[req.hdr(Host)]
The Backend-Server should not remove this header. If you not want to send the Backend-Server the 'X-Target' host header then can you use a session variable to save the host header from the request to the response phase.
http-request set-var(txn.my_host) req.hdr(host),lower
http-response set-header X-Target %[var(txn.my_host)]
In the documentation are the set-var and set-header directive quite good explained.
http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-http-request
About the server manipulation
This line could not work because haproxy tries to resolve the target server at start time.
server web-servers site.%[req.hdr(Host),regsub(^www.,,)]:80 check
In newer version of haproxy. like 2.1, can you dynamically resolve and set the destination hosts.
http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-http-request%20do-resolve
I assume you want to change the host header for the target server that the right virtual server is used.
My suggestion to solve your issue is to change the host header and set the server name to a resolvable address.
backend backend-default
option forwardfor
http-response set-header X-Publishing-system website
http-request set-header X-Target %[req.hdr(Host)]
http-request replace-header Host ^www(.*) site.\1
http-request set-header X-NewTarget %[req.hdr(Host),regsub(^www.,,)]
server web-servers site.example.com:80 check
This backend config is only syntax checked.
About dynamic backend server
The server should be resolved dynamically.
For that solution is at least HAProxy 2.0 necessary.
I copy here some parts of the doc http-request do-resolve for this answer.
You will need to add a section resolvers to your config
resolvers mydns
# use here your prefered DNS Servers
nameserver local 127.0.0.53:53
nameserver google 8.8.8.8:53
timeout retry 1s
hold valid 10s
hold nx 3s
hold other 3s
hold obsolete 0s
accepted_payload_size 8192
frontend frontend-http
bind *:80
bind *:443
# define capture buffer for backend
declare capture request len 60
acl redirect path_beg -i /rd
use_backend backend-tracking if redirect
default_backend backend-default
# ... some more backends
backend backend-default
option forwardfor
http-response set-header X-Publishing-system website
http-request set-header X-Target %[req.hdr(Host)]
# replace www with site in host header
http-request replace-header Host ^www(.*) site.\1
# if necessary set X-NewTarget header
http-request set-header X-NewTarget %[req.hdr(Host),regsub(^www.,,)]
# do dynamic host resolving for dynamic
# server destination for
# the replaced Host Header above
http-request do-resolve(txn.myip,mydns,ipv4) hdr(Host),lower
# print the resolved IP in the log
http-request capture var(txn.myip) id 0
# rule to prevent HAProxy from reconnecting to services
# on the local network (forged DNS name used to scan the network)
# add the IP Range for the destination host here
http-request deny if { var(txn.myip) -m ip 127.0.0.0/8 10.0.0.0/8 }
http-request set-dst var(txn.myip)
server clear 0.0.0.0:0
Please take care about the note in the documentation
NOTE: Don't forget to set the "protection" rules to ensure HAProxy won't be used to scan the network or worst won't loop over itself...
HAProxy provides a built-in http_err_rate counter which “reports the average HTTP request error rate over that period.” This can be used in a stick table to rate-limit clients that are generating a lot of errors. That might look something like this:
frontend web
tcp-request content reject if { src_get_gpc0(Abuse) gt 0 }
acl scanner src_http_err_rate(Abuse) ge 10
http-request deny if scanner flag_abuser
backend Abuse
stick-table type ip size 1m expire 60m store gpc0,http_err_rate(20s)
What I'd like to do is track something like the http_err_rate, but only for 401 Unauthorized status codes. That way HAProxy would only be concerned with rate-limiting unauthorized requests, rather than all HTTP error codes.
Thanks!
What I'd like to do is track something like the http_err_rate, but only for 401 Unauthorized status codes.
You can use the General Purpose Counters together with an ACL matching on the status fetch. The following example configuration will track the rate of 404 errors for a given IP address [1] and deny requests with the 429 status if a rate of 10 requests per 10 seconds is exceeded:
frontend fe_http
mode http
bind *:8080
stick-table type ipv6 size 10k expire 300s store gpc0_rate(10s)
http-request track-sc0 src
http-request deny deny_status 429 if { sc0_gpc0_rate gt 10 }
# Relevant line below
http-response sc-inc-gpc0(0) if { status 404 }
default_backend be_http
backend be_http
mode http
server example example.com:80
[1] Note: I recommend to use ipv6 as the stick table key, it may contain both IPv4 and IPv6 addresses.
If you want to rate limit depending on their rate of 401 you need to change the 429 code by 401 in your config:
http-request deny deny_status 401 if { sc_http_req_cnt(0) gt 10 }
With both deny and tarpit you can add the deny_status flag to set a
custom response code instead of the default 403/500 that they use out
of the box. For example using http-request deny deny_status 429 will
cause HAProxy to respond to the client with the error 429: Too Many
Requests.
For more "general" information about acls and rate-limiting, you can see:
https://www.haproxy.com/blog/four-examples-of-haproxy-rate-limiting/
https://www.haproxy.com/blog/introduction-to-haproxy-acls/
I want to implement a rate-limit system using the sticky table of HAProxy. Consider that I have 100 servers, and a limit of 10 requests per server, the ACL would be
http-request track-sc0 int(1) table GlobalRequestsTracker
http-request deny deny_status 429 if { sc0_http_req_rate(GlobalRequestsTracker),div(100) gt 10 }
Now if I want to make this dynamic depending on the healthy servers count, I need to replace the hardcoded 100 per the nbsrv converter.
http-request track-sc0 int(1) table GlobalRequestsTracker
http-request deny deny_status 429 if { sc0_http_req_rate(GlobalRequestsTracker),div(nbsrv(MyBackend)) gt 10 }
But I'm getting the error:
error detected while parsing an 'http-request deny' condition : invalid args in converter 'div' : expects an integer or a variable name in ACL expression 'sc0_http_req_rate(GlobalRequestsTracker),div(nbsrv(MyBackend))'.
Is there a way to use nbsrv as a variable inside the div operator?
HAProxy does no allow for nested function calls as far as I know. But you could store the number of backend servers in a variable and use it in the division (see http-request set-var in the HAProxy documentation). I have not tested it or used personaly, but I guess it could look like:
frontend <fe>
http-request track-sc0 int(1) table <tbl>
http-request set-var(req.<var>) nbsrv(<be>)
http-request deny deny_status <code> if { sc0_http_req_rate(<tbl>),div(req.<var>) gt <val> }
See the HAProxy documentation.
I'm attempting to configure HAProxy to close a client TCP connection after it has been used to process N requests. My goal is to have our long-lived clients occasionally re-establish connections that are otherwise kept-alive by HTTP Keep-Alive.
Basically I'm trying to implement the equivalent of nginx's keepalive_requests (http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests).
I currently have something like:
frontend https-in
bind *:443 ssl crt /etc/ssl/private/cert.pem
stick-table type binary len 32 size 1000 expire 75s store gpc0
acl close_connection sc0_get_gpc0 gt 3
acl exceeded_connection sc0_get_gpc0 gt 4
http-response set-header Connection Keep-Alive unless close_connection
http-response set-header Keep-Alive timeout=75\ max=3 unless close_connection
http-response set-header Connection Close if close_connection
timeout http-keep-alive 75s
timeout client 75s
tcp-request content track-sc0 ssl_fc_session_id
tcp-request content reject if exceeded_connection
http-request sc-inc-gpc0
default_backend https
backend https
option httpchk GET /health
server localhost 127.0.0.1:8080 maxconn 1000
But some problems with this include:
The SSL session ID may be re-used across connections
This ends abruptly closing a connection from a client when they've exceeded the threshold (assuming they ignore the Connection: Close)
Are there any recommended approaches for something like this? Ideally I would like to:
Track the counter based on individual TCP connection (not src to avoid the case where the same IP has multiple connections established)
Close the connection on the final response (at the same time I send Connection: Close)
But I haven't been able to track down ways to do either of those.
Thanks!
Edit
I was able to devise a better way to track unique TCP connections by creating hashing a tuple of the src,src_port,dst,dst_port:
http-request set-header X-Unique-Id %[src]:%[src_port]:%[dst]:%[dst_port]
http-request set-header X-Unique-Id-SHA %[req.fhdr(X-Unique-Id),sha1]
http-request track-sc0 req.fhdr(X-Unique-Id-SHA)
I'm not crazy about having to create the dummy headers, but this seems to work.
I'm using HAProxy version 1.6.
How would I modify the config to redirect requests from:
localhost:8081/myapp
to:
localhost:8111/myapp
Thanks!
frontend weblb
bind *:8081
acl if is_seller url_beg /myapp
use_backend sellerserver if is_seller
backend sellerserver
balance source
server web1 127.0.0.1:8111 maxconn 1024 weight 3 check inter 2000 rise 2 fall 3
You could try using replace-value on the Host header:
http-request replace-value Host localhost:8081 localhost:8111
Which is nice because it also supports regexes:
http-request replace-value Host (.*):8081 \1:8111
something like :
listen weblb xxx.xxx.xxx.xxx:8081
balance leastconn
mode http
server web1 127.0.0.1:8111