HAProxy close front-end connection after N HTTP requests - haproxy

I'm attempting to configure HAProxy to close a client TCP connection after it has been used to process N requests. My goal is to have our long-lived clients occasionally re-establish connections that are otherwise kept-alive by HTTP Keep-Alive.
Basically I'm trying to implement the equivalent of nginx's keepalive_requests (http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests).
I currently have something like:
frontend https-in
bind *:443 ssl crt /etc/ssl/private/cert.pem
stick-table type binary len 32 size 1000 expire 75s store gpc0
acl close_connection sc0_get_gpc0 gt 3
acl exceeded_connection sc0_get_gpc0 gt 4
http-response set-header Connection Keep-Alive unless close_connection
http-response set-header Keep-Alive timeout=75\ max=3 unless close_connection
http-response set-header Connection Close if close_connection
timeout http-keep-alive 75s
timeout client 75s
tcp-request content track-sc0 ssl_fc_session_id
tcp-request content reject if exceeded_connection
http-request sc-inc-gpc0
default_backend https
backend https
option httpchk GET /health
server localhost 127.0.0.1:8080 maxconn 1000
But some problems with this include:
The SSL session ID may be re-used across connections
This ends abruptly closing a connection from a client when they've exceeded the threshold (assuming they ignore the Connection: Close)
Are there any recommended approaches for something like this? Ideally I would like to:
Track the counter based on individual TCP connection (not src to avoid the case where the same IP has multiple connections established)
Close the connection on the final response (at the same time I send Connection: Close)
But I haven't been able to track down ways to do either of those.
Thanks!
Edit
I was able to devise a better way to track unique TCP connections by creating hashing a tuple of the src,src_port,dst,dst_port:
http-request set-header X-Unique-Id %[src]:%[src_port]:%[dst]:%[dst_port]
http-request set-header X-Unique-Id-SHA %[req.fhdr(X-Unique-Id),sha1]
http-request track-sc0 req.fhdr(X-Unique-Id-SHA)
I'm not crazy about having to create the dummy headers, but this seems to work.

Related

Sometimes server connect failed via loadbalance

My server is load balanced to the backend server via gcp https lb and the backend server uses pm2 start -i options different ports and distribute them to those nodes using haproxy.
connection log
Log taken from server using code.
io.on('connection', (socket) => {
console.debug("connection!", socket.id);
}
Once every two times, the server connection fails.
Below is Log through haproxy -d.
fail
success
this log is success after failure
00000041:http-in.clireq[000a:ffffffff]: GET /socket.io/?EIO=3&transport=websocket&sid=rf3JvyUz2KCKV4KDAABS HTTP/1.1
00000041:http-in.clihdr[000a:ffffffff]: User-Agent: websocket-sharp/1.0
00000041:http-in.clihdr[000a:ffffffff]: Host: mydomain.com
00000041:http-in.clihdr[000a:ffffffff]: Upgrade: websocket
00000041:http-in.clihdr[000a:ffffffff]: Sec-WebSocket-Key: 9JCkV46YNC66nIUaZQZl9w==
00000041:http-in.clihdr[000a:ffffffff]: Sec-WebSocket-Version: 13
00000041:http-in.clihdr[000a:ffffffff]: X-Cloud-Trace-Context: 1210ae7f3bb6e56c817e7f5ad30e1d24/17748396103009389644
00000041:http-in.clihdr[000a:ffffffff]: Connection: Upgrade
00000041:http-in.clihdr[000a:ffffffff]: Via: 1.1 google
00000041:http-in.clihdr[000a:ffffffff]: X-Forwarded-For: source ip, dest ip
00000041:http-in.clihdr[000a:ffffffff]: X-Forwarded-Proto: https
00000041:websockets.srvrep[000a:000b]: HTTP/1.1 400 Bad Request
00000041:websockets.srvhdr[000a:000b]: Connection: close
00000041:websockets.srvhdr[000a:000b]: Content-type: text/html
00000041:websockets.srvhdr[000a:000b]: Content-Length: 18
00000041:websockets.srvcls[000a:adfd]
00000041:websockets.clicls[adfd:adfd]
00000041:websockets.closed[adfd:adfd]
00000042:http-in.accept(0007)=000a from [130.211.3.23:53189] ALPN=<none>
00000042:http-in.clireq[000a:ffffffff]: GET /socket.io/?EIO=3&transport=websocket&sid=xmZEqtHokEmBfs2QAABT HTTP/1.1
00000042:http-in.clihdr[000a:ffffffff]: User-Agent: websocket-sharp/1.0
00000042:http-in.clihdr[000a:ffffffff]: Host: mydomain.com
00000042:http-in.clihdr[000a:ffffffff]: Upgrade: websocket
00000042:http-in.clihdr[000a:ffffffff]: Sec-WebSocket-Key: 8PKkFxEv3c3KqIUW8dQbLA==
00000042:http-in.clihdr[000a:ffffffff]: Sec-WebSocket-Version: 13
00000042:http-in.clihdr[000a:ffffffff]: X-Cloud-Trace-Context: de0fa56d1689317bb9212879da8edfcb/11012650454108009103
00000042:http-in.clihdr[000a:ffffffff]: Connection: Upgrade
00000042:http-in.clihdr[000a:ffffffff]: Via: 1.1 google
00000042:http-in.clihdr[000a:ffffffff]: X-Forwarded-For: source ip, dest ip
00000042:http-in.clihdr[000a:ffffffff]: X-Forwarded-Proto: https
00000042:websockets.srvrep[000a:000b]: HTTP/1.1 101 Switching Protocols
00000042:websockets.srvhdr[000a:000b]: Upgrade: websocket
00000042:websockets.srvhdr[000a:000b]: Connection: Upgrade
00000042:websockets.srvhdr[000a:000b]: Sec-WebSocket-Accept: tW86o/zu95tHQayPP7IlGNXi96s=
There is no difference, but two results. i don't know why 400 bad request
haproxy.cfg
global
maxconn 4096
defaults
mode http
balance roundrobin
option redispatch
option forwardfor
timeout connect 5s
timeout queue 5s
timeout client 50s
timeout server 50s
frontend http-in
bind *:80
default_backend servers
# Any URL beginning with socket.io will be flagged as 'is_websocket'
acl is_websocket path_beg /socket.io
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
# The connection to use if 'is_websocket' is flagged
use_backend websockets if is_websocket
backend servers
server server1 10.168.0.50:80
# server server2 [Address]:[Port]
backend websockets
balance source
option http-server-close
option forceclose
cookie io prefix indirect nocache # using the `io` cookie set upon handshake
server ws-server1 10.168.0.50:5000 weight 1 maxconn 1024 cookie ws-server1 check
server ws-server2 10.168.0.50:5001 weight 1 maxconn 1024 cookie ws-server2 check
#server ws-server3 10.168.0.50:5002 weight 1 maxconn 1024 check
i use cookie SRVNAME insert options and server name SA, SB
but socket.io document read change cookie io prefix indirect nocache and server name ws-server1, ws-server2
My Tested:
client used long polling and websocket
testClient give option {transports: ['websocket']} always success.. but real client not use only websocket options
I don't know why it fails.
If only use ws-server1 the connection will always succeed. but ws-server2 use sometime connection failed. I guess sticky session problem. I try haproxy.cfg add cookie option but The problem is not solved.
How can I solve this problem?

Dynamic server name and header in HAProxy

I’m looking for the equivalent of this backend code block below for requests to www.example.com and example.com.
http-response set-header X-Target example.com
server web-servers site.example.com:80 check
I take all the requests to www.example.com but I want to serve them to site.example.com using haproxy. There are several variations of example.com so I would like to have a list of allowed domains and then if they're allowed I would like to have a backend code block like below where I could use %[req.hdr(Host)] as the value in the http-response X-Target statement.
http-response set-header X-Target %[req.hdr(Host)]
server web-servers site.%[req.hdr(Host),regsub(^www.,,)]:80 check
HA-Proxy version 2.1.4-273103-54 2020/05/07 - https://haproxy.org/
I’m getting this error when I try haproxy -c -f haproxy.test
[root#pm-prod-haproxy05 haproxy]# haproxy -c -f haproxy.test
[ALERT] 259/180932 (16116) : parsing [haproxy.test:40]: ‘http-response set-header’: sample fetch may not be reliably used here because it needs ‘HTTP request headers’ which is not available here.
[ALERT] 259/180932 (16116) : Error(s) found in configuration file : haproxy.test
[root#pm-prod-haproxy05 haproxy]#
I've also tried this:
http-request set-header X-Target %[req.hdr(Host)]
http-request set-header X-Hostname %[req.hdr(Host),regsub(^www.,site.,)]
http-request web-server do-lookup(hdr(X-Hostname))
server web-servers web-server:80 check
This is my full configuration.
global
log 127.0.0.1 local2 debug
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
option httplog
log global
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend frontend-http
bind *:80
bind *:443
acl redirect path_beg -i /rd
use_backend backend-tracking if redirect
default_backend backend-default
backend backend-default
option forwardfor
http-response set-header X-Publishing-system website
http-response set-header X-Target %[req.hdr(Host)]
server web-servers site.%[req.hdr(Host),regsub(^www.,,)]:80 check
backend backend-tracking
option forwardfor
http-response set-header X-Publishing-system redirect
http-request set-uri %[url,regsub(^/rd,/,)]
server web-hp www.trackingplatform.com:80 check
About Header manipulation
As the ALERT message say you can't use request header in the response. You should replace the following line.
Wrong line
http-response set-header X-Target %[req.hdr(Host)]
Right Line
http-request set-header X-Target %[req.hdr(Host)]
The Backend-Server should not remove this header. If you not want to send the Backend-Server the 'X-Target' host header then can you use a session variable to save the host header from the request to the response phase.
http-request set-var(txn.my_host) req.hdr(host),lower
http-response set-header X-Target %[var(txn.my_host)]
In the documentation are the set-var and set-header directive quite good explained.
http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-http-request
About the server manipulation
This line could not work because haproxy tries to resolve the target server at start time.
server web-servers site.%[req.hdr(Host),regsub(^www.,,)]:80 check
In newer version of haproxy. like 2.1, can you dynamically resolve and set the destination hosts.
http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-http-request%20do-resolve
I assume you want to change the host header for the target server that the right virtual server is used.
My suggestion to solve your issue is to change the host header and set the server name to a resolvable address.
backend backend-default
option forwardfor
http-response set-header X-Publishing-system website
http-request set-header X-Target %[req.hdr(Host)]
http-request replace-header Host ^www(.*) site.\1
http-request set-header X-NewTarget %[req.hdr(Host),regsub(^www.,,)]
server web-servers site.example.com:80 check
This backend config is only syntax checked.
About dynamic backend server
The server should be resolved dynamically.
For that solution is at least HAProxy 2.0 necessary.
I copy here some parts of the doc http-request do-resolve for this answer.
You will need to add a section resolvers to your config
resolvers mydns
# use here your prefered DNS Servers
nameserver local 127.0.0.53:53
nameserver google 8.8.8.8:53
timeout retry 1s
hold valid 10s
hold nx 3s
hold other 3s
hold obsolete 0s
accepted_payload_size 8192
frontend frontend-http
bind *:80
bind *:443
# define capture buffer for backend
declare capture request len 60
acl redirect path_beg -i /rd
use_backend backend-tracking if redirect
default_backend backend-default
# ... some more backends
backend backend-default
option forwardfor
http-response set-header X-Publishing-system website
http-request set-header X-Target %[req.hdr(Host)]
# replace www with site in host header
http-request replace-header Host ^www(.*) site.\1
# if necessary set X-NewTarget header
http-request set-header X-NewTarget %[req.hdr(Host),regsub(^www.,,)]
# do dynamic host resolving for dynamic
# server destination for
# the replaced Host Header above
http-request do-resolve(txn.myip,mydns,ipv4) hdr(Host),lower
# print the resolved IP in the log
http-request capture var(txn.myip) id 0
# rule to prevent HAProxy from reconnecting to services
# on the local network (forged DNS name used to scan the network)
# add the IP Range for the destination host here
http-request deny if { var(txn.myip) -m ip 127.0.0.0/8 10.0.0.0/8 }
http-request set-dst var(txn.myip)
server clear 0.0.0.0:0
Please take care about the note in the documentation
NOTE: Don't forget to set the "protection" rules to ensure HAProxy won't be used to scan the network or worst won't loop over itself...

haproxy close connecton after 1 minute

this config
frontend https_frontend
bind *:4055
mode tcp
maxconn 8192
use_backend https_web
backend https_web
mode tcp
balance roundrobin
option http-keep-alive
server haproxy2 xxx.xxx.xxx.xxx:4055 send-proxy-v2
new connection send keep-alive packets every 30 seconds. but connection drop after 1 minute
I think this is because you're using mode tcp, but option http-keep-alive is a mode http option. In this case, it would most likely be using whatever value you have for timeout client or timeout server before dropping the connection.
For more details about option http-keep-alive and mode http, see:
https://www.haproxy.com/documentation/aloha/7-5/traffic-management/lb-layer7/http-modes/#http-modes-in-haproxy
frontend https_frontend
bind *:4055
mode tcp
maxconn 8192
use_backend https_web
backend https_web
mode tcp
balance roundrobin
timeout client 600000
timeout server 600000
server haproxy2 147.78.65.172:4055 send-proxy-v2
now i send keep-alive packets and real data every 30 seconds
but steel drop after 2 minutes
its not http/https query. its sample tcp communication with rand data. maybe it problem?

How to link frontend to backend when the path request are different?

I have an Haproxy set with https offloadin, and I'm trying to correctly point the requests made to frontend to it's corresponding backend, but bumped into some obstacles.
I have a backend server on http://:9000/abc (NOT in root of the webserver) and when I set a frontend with https:///abc the pointing works as expected and I see the login page.
But I also have another backend server, which is on http://:8888 (IN the root of webserver, it makes it's own redirect to http://:8888/def) and I want it to be accessible by https:///def. But in this case the pointing doesn't work.
How can I make https:///def point to http://:8888 ? Heres is my .cfg
Using HAproxy 1.7
# Automaticaly generated, dont edit manually.
# Generated on: 2019-01-28 13:59
global
maxconn 1000
stats socket /tmp/haproxy.socket level admin
uid 80
gid 80
nbproc 1
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
server-state-file /tmp/haproxy_server_state
listen HAProxyLocalStats
bind 127.0.0.1:2200 name localstats
mode http
stats enable
stats refresh 10
stats admin if TRUE
stats show-legends
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000
frontend shared-frontend-merged
bind 200.129.168.14:443 name 200.129.168.14:443 no-sslv3 ssl crt-list /var/etc/haproxy/shared-frontend.crt_list
mode http
log global
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^ifamcmc\.ddns\.net(:([0-9]){1,5})?$
acl ACL1 var(txn.txnpath) -m sub -i abc
acl ACL2 var(txn.txnpath) -m sub -i def
http-request set-var(txn.txnhost) hdr(host)
http-request set-var(txn.txnpath) path
use_backend glpi_ipvANY if ACL1
use_backend ciweb_ipvANY if ACL2
frontend http-to-https
bind 200.129.168.14:80 name 200.129.168.14:80
mode http
log global
option http-keep-alive
timeout client 30000
http-request redirect scheme https
backend abc_ipvANY
mode http
id 102
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server abc 10.100.0.30:9000 id 103 check inter 1000
backend def_ipvANY
mode http
id 104
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server def 10.100.0.40:8888 id 105 check inter 1000
I expect that access to https:///def correctly points to the backend at http://:8888
https://<my.address.com>/abc ------> http://<internal_ip>:9000/abc (OK)
https://<my.address.com>/def ------> http://<internal_ip_2>:8888 (NOT OK)
Have your HAProxy system do initially forwarding based on ports, and then wildcards on your directory.
Please see below:
frontend a-frontend-conf
# Declare an ACL using path_beg (Path Begins)
acl path_images path_beg /images
# Use backend server1 if acl condition path_images is fulfilled
use_backend server1 if path_images
backend server1
[...]
Source: https://serverfault.com/questions/659793/haproxy-how-to-balance-traffic-within-directory-reached

HAProxy Multiport backends

I may be using the wrong terms to search, but I'm having trouble finishing connections to my backend servers though HAProxy. I am able to initially login to the servers, but then the application communicates through two others ports as well. I can go directly to one of the servers and it logs in and lunches the application correctly. Through HAProxy however, I can authenticate, and then I get a communication error. I'm thinking that HAProxy is not passing through data on the other ports. How can this be achieved?
My setup:
2 HAProxy server connected to 2 identical VM servers. HAProxy seems to be working correctly as far as telling when the service on the mcahines are running. It is passing through authentication on port 8443. But that's as far as it will go. It will not launch the VM which uses ports 3000 and 5432. Any ideas on the HAProxy setup?
Here is my configuration file:
global
ssl-server-verify none
tune.ssl.default-dh-param 2048
maxconn 256
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
timeout connect 5000
timeout client 10000
timeout server 10000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
listen vm-port-3000
bind *:5432
server qvd4 10.0.0.1:3000
server qvdnode02 10.0.0.2:3000
listen vm-port-5432
bind *:5432
server qvd4 10.0.0.1:5432
server qvdnode02 10.0.0.2:5432
listen stats
bind :1936
stats enable
stats hide-version
stats realm Loadbalanced\ Servers
stats uri /haproxy?stats
stats auth haproxy:haproxy
frontend vm-initial-conn
bind *:8443 ssl crt /etc/ssl/certs/qvd/haproxy.pem
default_backend vmConn
backend vmConn
option forwardfor
option httpchk GET /qvd/ping HTTP/1.1
http-check expect status 200
balance roundrobin
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Port %[dst_port]
server qvd4 10.0.0.1:8443 ssl verify none check
server qvdnode02 10.0.0.2:8443 ssl verify none check
In your vm-port-3000 you are actually binding to port 5432 instead of how it appears you intended to port 3000.
Thus, requests to port 5432 are randomly handled by either your vm-port-3000 or your vm-port-5432 listener while connections to port 3000 are not handled by HAProxy at all.