Setting a unique http request id with haproxy's http-request set-header - haproxy

So I have some existing code that sets a unique request ID in our front-end load balancer:
unique-id-format %{+X}o\ %Ts_%ci_%cp_%fi_%fp_%rt_%pid
unique-id-header X-Request-Id
log-format %ci\ %r\ %ST\ %B\ %Tr\ %Tt\ %s\ %ID\ %hr
This works as expected -- X-Request-Id is created as expected, logged and passed to the backend. No problems. However, I'd like to make this request ID generation conditional. No problem -- I should just be able to use http-request set-header instead of unique-id-header:
unique-id-format %{+X}o\ %Ts_%ci_%cp_%fi_%fp_%rt_%pid
http-request set-header X-Request-Id %ID
log-format %ci\ %r\ %ST\ %B\ %Tr\ %Tt\ %s\ %ID\ %hr
(These are all in a front_end section). Maddeningly, however %ID seems to evaluate to empty when used this way. I can use a hardcoded value instead of using %ID and it works. I can also use another log field (like %Ts) and it works. It does not, however, work with %ID. Any clues would be helpful -- thanks in advance.
EDIT: Version is 1.6.11

I had the exact same issue in that I wanted to conditionally set the header if not present and %ID wasn't working as you'd expect. I found solutions suggesting to use %[unique-id] but it turns out that's only in version 1.7+. I have subsequently upgraded to 1.7 and now it works perfectly.
unique-id-format %{+X}o\ %pid%ci%cp%fi%fp%Ts%ms%rt
acl cid_exists req.hdr(X-Correlation-ID) -m found
http-request set-header X-Correlation-ID %[unique-id] unless cid_exists
http-request capture hdr(X-Correlation-ID) len 64
log-format "%ci:%cp [%tr] %ft %b/%s %Th/%Ti/%TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %{+Q}r %[capture.req.hdr(0)]"
The captured X-Correlation-ID header contains either a preexisting CID or one that this loadbalancer has created itself if missing.
The unique-id HTTP sample is referenced here.

Related

How to set host-header depending on choosen backend when using HAProxy as loadbalancer

Under certain circumstances it is required to modify the host-header based on the backend selected by HAProxy loadbalancing (an example is described here: https://www.claudiokuenzler.com/blog/919/haproxy-how-use-different-http-host-header-for-each-backend-server)
My internet research shows that there are basically two ways to achieve this:
a) use "http-send-name-header Host" (not recommended according the HAProxy docs: https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-http-send-name-header)
b) use " http-request set-header Host ... if { srv_id 1 }"
(the following article describes these two techniques: https://serverfault.com/questions/876871/configure-haproxy-to-include-host-headers-for-different-backends)
Option a) works as expected but it is discouraged to be use (see https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-http-send-name-header).
For this reason I am trying to option b). Unfortunately the config cowardly refuses to work:
backend loadbalanced-backends
mode http
balance roundrobin
option forwardfor
http-request set-header Host one.domain.com if { srv_id 1 }
http-request set-header Host two.domain.com if { srv_id 2 }
server one one.domain.com:8000
server two two.domain.com:8000
The warnings printed in the log:
[WARNING] (1) : config : parsing ....cfg:14] : anonymous acl will never match because it uses keyword 'srv_id' which is incompatible with 'backend http-request header rule'
[WARNING] (1) : config : parsing ....cfg:15] : anonymous acl will never match because it uses keyword 'srv_id' which is incompatible with 'backend http-request header rule'
Besids the warning the logged request-header show that the host is not changed to one.domain.com resp. two.domain.com.
What exactly means the warning "anonymous acl will never match..."? I do not see what is wrong with the configuration. Any ideas are welcome.

HAProxy set authorization header from cookie

I have a backend that I don't control that works using magic links. If I use the magic link, I don't have to log on. I want to share the magic link externally without exposing the actual key as the key changes from time to time and I was hoping to use haproxy as a reverse proxy. I don't want the link to be open to the entire internet and would like to use basic auth as well. The problem I am facing is that the backend overwrites the Authorization header (it's required for the magic link to work) and I get stuck in a loop and need to log on every time
My workaround:
On first request I request the basic auth (works)
Then I write it to a cookie (this part works)
On each subsequent request, if the cookie exists, I read the cookie and set the Authorization header to the cookie value (this part does not work)
Then I run http_auth which in my mind should now work since I have overwritten the header
But it does not work. Any suggestions?
userlist auth-list
user myuser insecure-password mypass
frontend myfrontend
bind *:80
mode http
acl is-path path -i -m beg /publiclink
acl has_cookie req.hdr(X-MyAuth) -m found
http-request set-header Authorization %[req.hdr(X-MyAuth)] if has_cookie
http-request auth unless { http_auth(auth-list) }
http-request set-var(txn.myhostheader) req.hdr(Authorization) if { http_auth(auth-list) !has_cookie }
default_backend node
backend node
mode http
server dcnode1 192.168.0.1:8000 check
http-response set-header set-cookie "X-MyAuth=%[var(txn.myhostheader)]; Path=/" if { var(txn.myhostheader) -m found }
http-request replace-path /publiclink1(.*) /magiclink\1
http-request set-header Authorization "Key magiclink"
My answer is related to this point :
On each subsequent request, if the cookie exists, I read the cookie and set the Authorization header to the cookie value (this part does not work)
In your backend, you set the Set-Cookie: X-MyAuth=... header :
backend node
http-response set-header set-cookie "X-MyAuth=%[var(txn.myhostheader)]; Path=/" if { var(txn.myhostheader) -m found }
So the next request contains a Cookie header like this one : Cookie: .* X-MyAuth=.*.
But in your frontend, you use the X-MyAuth header (which probably does not exist) :
frontend myfrontend
acl has_cookie req.hdr(X-MyAuth) -m found
http-request set-header Authorization %[req.hdr(X-MyAuth)] if has_cookie
You may use the X-MyAuth cookie value like this :
frontend myfrontend
acl has_cookie cook(X-MyAuth) -m found
http-request set-header Authorization %[cook(X-MyAuth)] if has_cookie

How do I secure cookies in HAProxy 2.2+ using an `http-response` line?

I'm upgrading from HAProxy 1.8 to 2.2 and the command reqirep has been deprecated and removed. I used this previously to automatically add Secure to cookies that weren't previously secure. I want to use the new http-response syntax.
My old code looks like this:
rspirep ^(set-cookie:\ (?:(?!(\ Secure|ASPXAUTH=)).)*)$ \1;\ Secure
This adds ; Secure to any cookie header that doesn't contain Secure or ASPXAUTH=.
I'd like to do the same thing with one of the modern http-response commands.
Here's my initial translation:
http-request replace-header Set-Cookie (.*) %[src];\ Secure if { hdr_reg(Set-Cookie) -i (?!(\ Secure|ASPXAUTH=)) }
# Replace the "Set-Cookie" header
# That contains any value
# With the initial value with "; Secure" appended to the end
# If the cookie doesn't contain " Secure" or "ASPXAUTH=", ignoring case
Is this the right approach? Have you done this successfully?
We ended up with this as a solution. It's not perfect because it will only look for Secure modifier on the end of the Set-Cookie line but it works for what we need.
http-response replace-header Set-Cookie ^((?:.(?!\ [Ss]ecure))*)$ \1;\ Secure

Dynamic server name and header in HAProxy

I’m looking for the equivalent of this backend code block below for requests to www.example.com and example.com.
http-response set-header X-Target example.com
server web-servers site.example.com:80 check
I take all the requests to www.example.com but I want to serve them to site.example.com using haproxy. There are several variations of example.com so I would like to have a list of allowed domains and then if they're allowed I would like to have a backend code block like below where I could use %[req.hdr(Host)] as the value in the http-response X-Target statement.
http-response set-header X-Target %[req.hdr(Host)]
server web-servers site.%[req.hdr(Host),regsub(^www.,,)]:80 check
HA-Proxy version 2.1.4-273103-54 2020/05/07 - https://haproxy.org/
I’m getting this error when I try haproxy -c -f haproxy.test
[root#pm-prod-haproxy05 haproxy]# haproxy -c -f haproxy.test
[ALERT] 259/180932 (16116) : parsing [haproxy.test:40]: ‘http-response set-header’: sample fetch may not be reliably used here because it needs ‘HTTP request headers’ which is not available here.
[ALERT] 259/180932 (16116) : Error(s) found in configuration file : haproxy.test
[root#pm-prod-haproxy05 haproxy]#
I've also tried this:
http-request set-header X-Target %[req.hdr(Host)]
http-request set-header X-Hostname %[req.hdr(Host),regsub(^www.,site.,)]
http-request web-server do-lookup(hdr(X-Hostname))
server web-servers web-server:80 check
This is my full configuration.
global
log 127.0.0.1 local2 debug
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
option httplog
log global
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend frontend-http
bind *:80
bind *:443
acl redirect path_beg -i /rd
use_backend backend-tracking if redirect
default_backend backend-default
backend backend-default
option forwardfor
http-response set-header X-Publishing-system website
http-response set-header X-Target %[req.hdr(Host)]
server web-servers site.%[req.hdr(Host),regsub(^www.,,)]:80 check
backend backend-tracking
option forwardfor
http-response set-header X-Publishing-system redirect
http-request set-uri %[url,regsub(^/rd,/,)]
server web-hp www.trackingplatform.com:80 check
About Header manipulation
As the ALERT message say you can't use request header in the response. You should replace the following line.
Wrong line
http-response set-header X-Target %[req.hdr(Host)]
Right Line
http-request set-header X-Target %[req.hdr(Host)]
The Backend-Server should not remove this header. If you not want to send the Backend-Server the 'X-Target' host header then can you use a session variable to save the host header from the request to the response phase.
http-request set-var(txn.my_host) req.hdr(host),lower
http-response set-header X-Target %[var(txn.my_host)]
In the documentation are the set-var and set-header directive quite good explained.
http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-http-request
About the server manipulation
This line could not work because haproxy tries to resolve the target server at start time.
server web-servers site.%[req.hdr(Host),regsub(^www.,,)]:80 check
In newer version of haproxy. like 2.1, can you dynamically resolve and set the destination hosts.
http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-http-request%20do-resolve
I assume you want to change the host header for the target server that the right virtual server is used.
My suggestion to solve your issue is to change the host header and set the server name to a resolvable address.
backend backend-default
option forwardfor
http-response set-header X-Publishing-system website
http-request set-header X-Target %[req.hdr(Host)]
http-request replace-header Host ^www(.*) site.\1
http-request set-header X-NewTarget %[req.hdr(Host),regsub(^www.,,)]
server web-servers site.example.com:80 check
This backend config is only syntax checked.
About dynamic backend server
The server should be resolved dynamically.
For that solution is at least HAProxy 2.0 necessary.
I copy here some parts of the doc http-request do-resolve for this answer.
You will need to add a section resolvers to your config
resolvers mydns
# use here your prefered DNS Servers
nameserver local 127.0.0.53:53
nameserver google 8.8.8.8:53
timeout retry 1s
hold valid 10s
hold nx 3s
hold other 3s
hold obsolete 0s
accepted_payload_size 8192
frontend frontend-http
bind *:80
bind *:443
# define capture buffer for backend
declare capture request len 60
acl redirect path_beg -i /rd
use_backend backend-tracking if redirect
default_backend backend-default
# ... some more backends
backend backend-default
option forwardfor
http-response set-header X-Publishing-system website
http-request set-header X-Target %[req.hdr(Host)]
# replace www with site in host header
http-request replace-header Host ^www(.*) site.\1
# if necessary set X-NewTarget header
http-request set-header X-NewTarget %[req.hdr(Host),regsub(^www.,,)]
# do dynamic host resolving for dynamic
# server destination for
# the replaced Host Header above
http-request do-resolve(txn.myip,mydns,ipv4) hdr(Host),lower
# print the resolved IP in the log
http-request capture var(txn.myip) id 0
# rule to prevent HAProxy from reconnecting to services
# on the local network (forged DNS name used to scan the network)
# add the IP Range for the destination host here
http-request deny if { var(txn.myip) -m ip 127.0.0.0/8 10.0.0.0/8 }
http-request set-dst var(txn.myip)
server clear 0.0.0.0:0
Please take care about the note in the documentation
NOTE: Don't forget to set the "protection" rules to ensure HAProxy won't be used to scan the network or worst won't loop over itself...

How to fix an improper request in HAProxy

We have several (100+) clients in the field with a bug in the HTTP request. The request was previously working when directly routed to our Windows Server, but now with it fails with HAProxy v1.7 in front of it.
Here is an example request:
GET /index.aspx HTTP/1.1 \nHost: host\n\n
There is an extra space after the HTTP version before the \n.
Here is a snapshot of the relevant config.
frontend http_port_80
bind :80
mode http
reqrep (.)\ HTTP/1.1\ (.*) \1\ HTTP/1.1\2
option forwardfor
option accept-invalid-http-request
stats enable
use_backend cert_update if is_updater
use_backend getConsoleHTTP if is_getconsole
default_backend schedule_server
I have tried rewriting the request to remove the extra space and set the option accept-invalid-http-request to address the issue, but we still receive the same error.
{
type: haproxy,
timestamp: 1506545591,
termination_state: PR-,
http_status:400,
http_request:,
http_version:,
remote_addr:192.168.1.1,
bytes_read:187,
upstream_addr:-,
backend_name:http_port_80,
retries:0,
bytes_uploaded:92,
upstream_response_time:-1,
upstream_connect_time:-1,
session_duration:2382,
termination_state:PR
}
Does anyone have any ideas of how to fix the malformed request prior to haproxy rejecting it?