I am trying to create ACL in Haproxy to query Authorization from request header and route to backend based on AccessID. I have used map file which are populated with AccessID and backend server. I am sure that my ACL is not working and hence I am getting 503 for incoming requests. Any help is appreciated!
Config File:
frontend main
bind *:80
capture request header Authorization len 50
acl GET_calls method GET HEAD OPTIONS
acl PUT_calls method PUT
use_backend %[urlp,map_sub(/etc/haproxy/PUT_Header.map)] if PUT_calls
Map File:
# AccessID backend server
JMYQ get_s1
P2BH get_s1
WEA1 get_s2
I have captured the request header in log and I see AccessID.
Apr 8 10:10:29 localhost haproxy[79517]: 0.11.4.1:929 [08/Apr/2022:10:10:29.232] main main/<NOSRV> -1/-1/-1/-1/0 503 212 - - SC-- 0/0/0/0/0 0/0 {Credential=WEA1} "PUT /common/Demo2.file HTTP/1.1"
Related
This is the exact same question as http request to https request using haproxy
However, the accepted answer does not work for me and I dont understand why
haproxy.cfg:
global
daemon
maxconn 15
defaults
mode tcp
balance first
frontend google
bind *:10005
default_backend google-url
backend google-url
server xxx google.com:443 ssl verify none
when I call curl --location --request GET 'http://localhost:10005', I receive a response that comes from google but with a 404 status
The requested URL / was not found on this server. That’s all we know.
I tried both mode tcp and mode http, same result
If I activate the logs with
mode http
bind *:10005
default_backend google-url
option httplog
log stdout format raw local0
I have this
127.0.0.1:52588 [16/Jun/2022:08:24:49.976] google google-url/xxx 0/0/49/20/69 404 1884 - - ---- 2/2/0/0/0 0/0 "GET / HTTP/1.1"
127.0.0.1:52588 [16/Jun/2022:08:24:49.938] google google/<NOSRV> -1/-1/-1/-1/1038 400 0 - - CR-- 2/2/0/0/0 0/0 "<BADREQ>"
In case this has some impact, I'm running haproxy in kubernetes and then I "port-forward" 10005 (but this does not seem to be the issue because the logs demonstrate that haproxy is correctly receiving the request and using the correct backend...)
Your curent HAProxy configuration will accept your request:
curl --location --request GET 'http://localhost:10005'
(corresponds to the first log entry)
and proxy it to Google as:
curl --location -H 'Host: localhost' --request GET 'https://www.google.com/'
(note the Host header implied; I bet this is not what you'd expect).
Google will respond with 404 and HAProxy will log the BADREQ.
This happens because HAProxy can't infer that when client request's Host header is localhost it should re-write it to google.com (or better: www.google.com) simply because it proxies to a host with that name.
You need to configure:
backend google-url
server xxx google.com:443 ssl verify none
http-request set-header host www.google.com
i just want to save log for backend request and the restarting of haproxy service in haproxy,
this is the configuration of my haproxy.
[appadmin#sltxh5gvt4c rsyslog.d]$ cat /etc/rsyslog.conf
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# Save haproxy log
local3.* /var/log/haproxylog/haproxy.log
# Save keepalived log
local2.* /var/log/keepalived/keepalived.log
++++++++++ this is haproxy.cfg
defaults
option forwardfor
log global
option httplog
log 127.0.0.1 local3
frontend case3
bind :80
mode http
log global
option httpclose
timeout client 5000
acl sabrix path_beg -i /sabrix
acl geolink path_beg -i /axis2
use_backend sabrix_servers if sabrix
use_backend us_geolink if geolink
capture request header Host len 64
capture request header User-Agent len 128
capture request header X-Forwarded-For len 100
capture request header Referer len 200
capture response header Server len 40
capture response header Server-ID len 40
log-format %ci:%cp\ %si:%sp\ %ST\ %r\ %b\ %f\ %bi\
+++++++++++++++++++++++++ haproxy.log
2020-06-14T23:33:36+08:00 localhost haproxy[27891]: 10.12.12.12:42164 -:- 400 <BADREQ> case3 case3 -
2020-06-14T23:33:42+08:00 localhost haproxy[27891]: 10.12.12.12:42204 -:- 400 <BADREQ> case3 case3 -
2020-06-14T23:33:48+08:00 localhost haproxy[27891]: 10.12.12.12:42244 -:- 400 <BADREQ> case3 case3 -
i don't want to save such logs in my haproxy.log, how to change my configuration files ?
thanks
You can't do this in haproxy, the best way to discard this log lines is with rsyslog.
:msg, contains, "BADREQ" ~
There are more information about discarding unwanted messages in this document.
https://www.rsyslog.com/discarding-unwanted-messages/
This looks similar to this question.
rsyslog filtering and forwarding
i tried to edit /etc/rsyslog.conf
# ignore BADREQ in haproxy.log
:msg, contains, "BADREQ" /var/log/haproxylog/haproxy.log
or
# ignore BADREQ in haproxy.log
:msg, contains, "BADREQ" stop
both don't work.
i check haproxy.cfg, i got the answer, we just need add
option dontlognull
to haproxy.cfg
I tried to receive request and want to redirect it to other host using dns name and exposed with https protocol. For example, my server is http://8.8.8.8:10101/partnerA/getUser. I want haproxy redirect this to https://partner.com/partnerA/getUser (same path as the source).
I also want to filter by path for another redirect destination such as http://8.8.8.8:10101/partnerB/getMarketShare will redirected by HAProxy to https://subdomainb.differentpartner.com/partnerB/getMarketShare(notice the path also follow the same rule, but based on path it will give different host name.
I tried below haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main
bind *:10101
acl url_partnerA path_beg -i /partnerA
acl url_partnerB path_beg -i /partnerB
http-request redirect scheme https if url_partnerA
http-request redirect scheme https if url_partnerB
http-request redirect prefix https://partnerA.com if url_partnerA
http-request redirect prefix https://subdomainb.differentpartner.com/ if url_partnerA
default_backend app
#---------------------------------------------------------------------
# round robin balancing between the various backends
backend app
balance roundrobin
# server app1 127.0.0.1:11003 check
But everytime I access (I use http) POST http://8.8.8.8:10101/partnerA/getUser, the log from haproxy -f haproxy10101.cfg -d will give me this
00000000:main.accept(0005)=0009 from [8.8.8.8:48554] ALPN=<none>
00000000:main.clireq[0009:ffffffff]: POST /partnerA/getUser HTTP/1.1
00000000:main.clihdr[0009:ffffffff]: Host: 8.8.8.8:10101
00000000:main.clihdr[0009:ffffffff]: User-Agent: curl/7.47.0
00000000:main.clihdr[0009:ffffffff]: Accept: */*
00000000:main.clihdr[0009:ffffffff]: Authorization: Basic dGNhc2g6RzBqM2tmMHJsMWYzIQ==
00000000:main.clihdr[0009:ffffffff]: Content-Type: application/json
00000000:main.clihdr[0009:ffffffff]: Postman-Token: 45a236c-740a-4859-a13a-1c45195a99f2
00000000:main.clihdr[0009:ffffffff]: cache-control: no-cache
00000000:main.clihdr[0009:ffffffff]: Content-Length: 218
00000000:main.clicls[0009:ffffffff]
00000000:main.closed[0009:ffffffff]
Anything I miss to make it work? Thanks
I have a haproxy configuration as follows. (haproxy 1.7) We want to catch all OPTIONS request and respond directly to them instead of routing the requests to backends (which have basic auth enabled).
This was working fine when we developed it but now it seems to not be matching the rules in order (not sure what we have/haven't done which has caused this):
global
log 127.0.0.1 local1
tune.ssl.default-dh-param 2048
lua-load /etc/haproxy/cors.lua
stats socket /var/run/haproxy.sock mode 400
# Default certificate and key directories
ca-base /etc/ssl/private
crt-base /etc/ssl/private
# User lists used to enforce HTTP Basic Authentication
userlist ul_100123-2ovt9rsu
user app1 password $6$lCjf6VnWhI$kcjmpWdV.odeYf4psUhcVKs49ZtPk3MDhg5wtLNUx658A3EWdDHJQqs9xCD1d.7zG05M2nwOxdkC6o/MSpifv0
userlist ul_100123-9uvsclqr
user app1 password $6$DlcLoDMMu$wDm3O0W1eiQuk8gI.GmpzI1.jbBf.UYQ.KM73nHa1tGZJNfzkDpVnLUhh7v7C9yPHB1oo0cRrFnfOdeyAf/eU1
# Front-end for public services which have SSL termination at the router.
frontend term
bind *:443 accept-proxy ssl no-sslv3 crt router/fred-external.pem crt router/fred-external.ace.pem crt router
reqadd X-Forwarded-Proto:\ https
rspidel ^(Server|X-Powered-By):
option forwardfor
mode http
http-request use-service lua.cors-response if METH_OPTIONS { req.hdr(origin) -m found }
acl host_match_100123-2ovt9rsu ssl_fc_sni -i 2ovt9rsu.fredurl.com
use_backend b_term_100123-2ovt9rsu if host_match_100123-2ovt9rsu
......
If I curl -X OPTIONS to 2ovt9rsu.fredurl.com it matches the 2nd rule and forwards me to the b_term_100123-2ovt9rsu backend which then fails as I haven't provided auth creds.
If I curl -X OPTIONS to Anything.fredurl.com it matches the first http-request and responds with the cors response as expected.
Why does the 2ovt9rsu.fredurl.com not match the first http-request rule and then return the cors-response?
In the logs we can see
Nov 7 18:24:09 localhost haproxy[37302]: 94.45.23.22:49853 [07/Nov/2017:18:24:09.807] term~ b_term_100123-2ovt9rsu/<lua.cors-response> -1/-1/-1/-1/73 401 249 - - PR-- 0/0/0/0/3 0/0 "OPTIONS / HTTP/1.1"
when the request gets forwarded to the backend
http-request gets executed before use_backend, the config looks good to me, have you set origin header when you curl ?
I'm looking for a solution with redirects to another domain if the response from HTTP server was 404.
acl not_found status 404
acl found_ceph status 200
use_backend minio_s3 rsprep ^HTTP/1.1\ 404\ (.*)$ HTTP/1.1\ 302\ Found\nLocation:\ / if not_found
use_backend ceph if found_ceph
But still not working, this rule goes to minio_s3 backend.
Thank you for you advice.
When the response from this backend has status 404, first add a Location header that will send the browser to example.com with the original URI intact, then set the status code to 302 so the browser executes a redirect.
backend my-backend
mode http
server my-server 203.0.113.113:80 check inter 60000 rise 1 fall 2
http-response set-header Location http://example.com%[capture.req.uri] if { status eq 404 }
http-response set-status 302 if { status eq 404 }
Test:
$ curl -v http://example.org/pics/funny/cat.jpg
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to example.org (127.0.0.1) port 80 (#0)
> GET /pics/funny/cat.jpg HTTP/1.1
> User-Agent: curl/7.35.0
> Host: example.org
> Accept: */*
The actual back-end returns 404, but we don't see it. Instead...
< HTTP/1.1 302 Moved Temporarily
< Last-Modified: Thu, 04 Aug 2016 16:59:51 GMT
< Content-Type: text/html
< Content-Length: 332
< Date: Sat, 07 Oct 2017 00:03:22 GMT
< Location: http://example.com/pics/funny/cat.jpg
The response body from the back-end's 404 error page will still be sent to the browser, but -- as it turns out -- the browser will not display it, so no harm done. This requires HAProxy 1.6 or later.
#Michael's answer is rather good, but isno't working for me for two reasons:
Mainly because the %[capture.req.uri] tag resolves to empty (HA Proxy 1.7.9 Docker image)
Also due to the fact that the original assumptions are incomplete, due to the fact that the frontend section is missing...
So I struggled for a while, as you find all kinds of answers on the Internet, between those guys who swear the 404 logic should be put in the frontend, vs those who choose the backend, and any possible kind of tags...
This is my answer, which works for me.
My use case is that if an image is not found on the backend behind HA Proxy, then an S3 bucket is checked.
The entry point is: https://myhostname:8080/path/to/image.jpeg
defaults
mode http
global
log 127.0.0.1:514 local0 debug
frontend come_on_over_here
bind :8080
# The following two lines are here to save values while we have access to them. They won't be available in the backend section.
http-request set-var(txn.path) path
http-request set-var(txn.query) query
http-request replace-value Host localhost:8080 dev.local:80
default_backend onprems_or_s3_be
backend onprems_or_s3_be
log global
acl path_photos var(txn.path) -m beg /path/prefix/i/want/to/strip/off
acl p_ext_jpeg var(txn.path) -m end .jpeg
acl is404 status eq 404
http-response set-header Location https://mybucket.s3.eu-west-3.amazonaws.com"%[var(txn.path),regsub(^/path_prefix_i_want_to_strip_off/,/)]?%[var(txn.query)]" if path_photos p_ext_jpeg is404
http-response set-status 301 if is404
server onprems_server dev.local:80 check