Implement a rate-limit relating to the healthy servers count using haproxy - haproxy

I want to implement a rate-limit system using the sticky table of HAProxy. Consider that I have 100 servers, and a limit of 10 requests per server, the ACL would be
http-request track-sc0 int(1) table GlobalRequestsTracker
http-request deny deny_status 429 if { sc0_http_req_rate(GlobalRequestsTracker),div(100) gt 10 }
Now if I want to make this dynamic depending on the healthy servers count, I need to replace the hardcoded 100 per the nbsrv converter.
http-request track-sc0 int(1) table GlobalRequestsTracker
http-request deny deny_status 429 if { sc0_http_req_rate(GlobalRequestsTracker),div(nbsrv(MyBackend)) gt 10 }
But I'm getting the error:
error detected while parsing an 'http-request deny' condition : invalid args in converter 'div' : expects an integer or a variable name in ACL expression 'sc0_http_req_rate(GlobalRequestsTracker),div(nbsrv(MyBackend))'.
Is there a way to use nbsrv as a variable inside the div operator?

HAProxy does no allow for nested function calls as far as I know. But you could store the number of backend servers in a variable and use it in the division (see http-request set-var in the HAProxy documentation). I have not tested it or used personaly, but I guess it could look like:
frontend <fe>
http-request track-sc0 int(1) table <tbl>
http-request set-var(req.<var>) nbsrv(<be>)
http-request deny deny_status <code> if { sc0_http_req_rate(<tbl>),div(req.<var>) gt <val> }
See the HAProxy documentation.

Related

How to set host-header depending on choosen backend when using HAProxy as loadbalancer

Under certain circumstances it is required to modify the host-header based on the backend selected by HAProxy loadbalancing (an example is described here: https://www.claudiokuenzler.com/blog/919/haproxy-how-use-different-http-host-header-for-each-backend-server)
My internet research shows that there are basically two ways to achieve this:
a) use "http-send-name-header Host" (not recommended according the HAProxy docs: https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-http-send-name-header)
b) use " http-request set-header Host ... if { srv_id 1 }"
(the following article describes these two techniques: https://serverfault.com/questions/876871/configure-haproxy-to-include-host-headers-for-different-backends)
Option a) works as expected but it is discouraged to be use (see https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-http-send-name-header).
For this reason I am trying to option b). Unfortunately the config cowardly refuses to work:
backend loadbalanced-backends
mode http
balance roundrobin
option forwardfor
http-request set-header Host one.domain.com if { srv_id 1 }
http-request set-header Host two.domain.com if { srv_id 2 }
server one one.domain.com:8000
server two two.domain.com:8000
The warnings printed in the log:
[WARNING] (1) : config : parsing ....cfg:14] : anonymous acl will never match because it uses keyword 'srv_id' which is incompatible with 'backend http-request header rule'
[WARNING] (1) : config : parsing ....cfg:15] : anonymous acl will never match because it uses keyword 'srv_id' which is incompatible with 'backend http-request header rule'
Besids the warning the logged request-header show that the host is not changed to one.domain.com resp. two.domain.com.
What exactly means the warning "anonymous acl will never match..."? I do not see what is wrong with the configuration. Any ideas are welcome.

Setup rate-limits based on hostname

I'm going through https://www.haproxy.com/blog/four-examples-of-haproxy-rate-limiting/ but unable to grok how to write a configuration which rate-limits based on the Host header. Does the following look alright?
frontend website
bind :80
stick-table type string size 100k expire 30s store http_req_rate(10s)
# What do I put here?
http-request track-sc0 request.header(Host)
# what does sc_http_req_rate(0) really mean?
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 20 }
default_backend servers
Also, what is an easy way to validate whether a rate-limiting configuration works as intended? (not simply test the syntactical validity of the the config)
I figured it out:
stick-table type string size 100k expire 300s store http_req_rate(60s)
http-request track-sc0 hdr(Host)
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 30 }

How to rate limit by HTTP status code with HAProxy?

HAProxy provides a built-in http_err_rate counter which “reports the average HTTP request error rate over that period.” This can be used in a stick table to rate-limit clients that are generating a lot of errors. That might look something like this:
frontend web
tcp-request content reject if { src_get_gpc0(Abuse) gt 0 }
acl scanner src_http_err_rate(Abuse) ge 10
http-request deny if scanner flag_abuser
backend Abuse
stick-table type ip size 1m expire 60m store gpc0,http_err_rate(20s)
What I'd like to do is track something like the http_err_rate, but only for 401 Unauthorized status codes. That way HAProxy would only be concerned with rate-limiting unauthorized requests, rather than all HTTP error codes.
Thanks!
What I'd like to do is track something like the http_err_rate, but only for 401 Unauthorized status codes.
You can use the General Purpose Counters together with an ACL matching on the status fetch. The following example configuration will track the rate of 404 errors for a given IP address [1] and deny requests with the 429 status if a rate of 10 requests per 10 seconds is exceeded:
frontend fe_http
mode http
bind *:8080
stick-table type ipv6 size 10k expire 300s store gpc0_rate(10s)
http-request track-sc0 src
http-request deny deny_status 429 if { sc0_gpc0_rate gt 10 }
# Relevant line below
http-response sc-inc-gpc0(0) if { status 404 }
default_backend be_http
backend be_http
mode http
server example example.com:80
[1] Note: I recommend to use ipv6 as the stick table key, it may contain both IPv4 and IPv6 addresses.
If you want to rate limit depending on their rate of 401 you need to change the 429 code by 401 in your config:
http-request deny deny_status 401 if { sc_http_req_cnt(0) gt 10 }
With both deny and tarpit you can add the deny_status flag to set a
custom response code instead of the default 403/500 that they use out
of the box. For example using http-request deny deny_status 429 will
cause HAProxy to respond to the client with the error 429: Too Many
Requests.
For more "general" information about acls and rate-limiting, you can see:
https://www.haproxy.com/blog/four-examples-of-haproxy-rate-limiting/
https://www.haproxy.com/blog/introduction-to-haproxy-acls/

Setting a unique http request id with haproxy's http-request set-header

So I have some existing code that sets a unique request ID in our front-end load balancer:
unique-id-format %{+X}o\ %Ts_%ci_%cp_%fi_%fp_%rt_%pid
unique-id-header X-Request-Id
log-format %ci\ %r\ %ST\ %B\ %Tr\ %Tt\ %s\ %ID\ %hr
This works as expected -- X-Request-Id is created as expected, logged and passed to the backend. No problems. However, I'd like to make this request ID generation conditional. No problem -- I should just be able to use http-request set-header instead of unique-id-header:
unique-id-format %{+X}o\ %Ts_%ci_%cp_%fi_%fp_%rt_%pid
http-request set-header X-Request-Id %ID
log-format %ci\ %r\ %ST\ %B\ %Tr\ %Tt\ %s\ %ID\ %hr
(These are all in a front_end section). Maddeningly, however %ID seems to evaluate to empty when used this way. I can use a hardcoded value instead of using %ID and it works. I can also use another log field (like %Ts) and it works. It does not, however, work with %ID. Any clues would be helpful -- thanks in advance.
EDIT: Version is 1.6.11
I had the exact same issue in that I wanted to conditionally set the header if not present and %ID wasn't working as you'd expect. I found solutions suggesting to use %[unique-id] but it turns out that's only in version 1.7+. I have subsequently upgraded to 1.7 and now it works perfectly.
unique-id-format %{+X}o\ %pid%ci%cp%fi%fp%Ts%ms%rt
acl cid_exists req.hdr(X-Correlation-ID) -m found
http-request set-header X-Correlation-ID %[unique-id] unless cid_exists
http-request capture hdr(X-Correlation-ID) len 64
log-format "%ci:%cp [%tr] %ft %b/%s %Th/%Ti/%TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %{+Q}r %[capture.req.hdr(0)]"
The captured X-Correlation-ID header contains either a preexisting CID or one that this loadbalancer has created itself if missing.
The unique-id HTTP sample is referenced here.

How to fix an improper request in HAProxy

We have several (100+) clients in the field with a bug in the HTTP request. The request was previously working when directly routed to our Windows Server, but now with it fails with HAProxy v1.7 in front of it.
Here is an example request:
GET /index.aspx HTTP/1.1 \nHost: host\n\n
There is an extra space after the HTTP version before the \n.
Here is a snapshot of the relevant config.
frontend http_port_80
bind :80
mode http
reqrep (.)\ HTTP/1.1\ (.*) \1\ HTTP/1.1\2
option forwardfor
option accept-invalid-http-request
stats enable
use_backend cert_update if is_updater
use_backend getConsoleHTTP if is_getconsole
default_backend schedule_server
I have tried rewriting the request to remove the extra space and set the option accept-invalid-http-request to address the issue, but we still receive the same error.
{
type: haproxy,
timestamp: 1506545591,
termination_state: PR-,
http_status:400,
http_request:,
http_version:,
remote_addr:192.168.1.1,
bytes_read:187,
upstream_addr:-,
backend_name:http_port_80,
retries:0,
bytes_uploaded:92,
upstream_response_time:-1,
upstream_connect_time:-1,
session_duration:2382,
termination_state:PR
}
Does anyone have any ideas of how to fix the malformed request prior to haproxy rejecting it?