Currently, I am trying to configure a load balancer from where the traffic will be sent to a Kubernetes cluster. At the edge of the cluster, Istio ingress is serving the cluster's external request. HAProxy version 1.8
I can access the service using the below command from outside the cluster.
curl -k -HHost:httpbin.example.com --resolve httpbin.example.com:32009:192.168.50.10 https://httpbin.example.com:32009/status/418:
Below is my HAProxy configuration:
Frontend:
frontend https
bind *:443 ssl crt /etc/ssl/certs/site.pem
mode tcp
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend httpbin
Backend:
backend httpbin
balance roundrobin
mode tcp
acl httpbin_app req_ssl_sni -i httpbin.example.com
use-server master if httpbin_app
server master 192.168.50.10:32009 check ssl verify none
http-request set-header Host httpbin.example.com
http-request set-header X-Forwarded-For %[src]
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
Using HAProxy I am getting 503 always. Also during startup, HAProxy is saying the below line:
haproxy[14260]: backend httpbin has no server available!
Can you please help to find out the right configuration for backend?
Finally with the help of HAProxy community I have found the right configuration for Istio Ingress. This is very basic. Please update the settings as per your needs. Below is the link for the configuration.
https://gist.github.com/emamulandalib/2a613f5308c29518fcbcdc6b3bad3900
Related
How can I change kubernetes pod namespace dynamically in a backend of HA proxy? Basically, what I need is the reference to the 2nd matched pattern group of reqrep to be used as the namespace in server or extract the namespace from the Referer, say if the referer is https://example.name.net/abc/app/v1, the namespace should be abc.
backend My_Web_Server
balance roundrobin
option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
acl fail-validation req.fhdr(X-Haproxy-ACL) -m found
http-request deny if fail-validation
reqrep ^([^\\ ]*)\\ (.*)/app/(.*) \\1\\ /app/\\3
server <server-name> <kubernetes-pod-name>.<namespace>:<port>
I would like incoming queries in haproxy example: http://haproxy/test are redirected to one of the backends in a 301 redirection:
http://server-a/test or
http://server-b/test or
http://server-c/test or
…
my backend:
backend cluster_redirect
http mode
random balance
server server-a:80
server server-b:80
server server-c:80
my frontend:
front-end www
bind:80
http-request redirect code 301 location http://%bi%[capture.req.uri]
use_backend cluster_redirect
%bi is not interpreted!. URL for redirect is : “location: http:///test”
If i use a set-header for debug: http-response set-header X-Debug http://%bi%[capture.req.uri]
the header is : X-Debug: http://10.1.1.1/test.
10.1.1.1 is the IP for haproxy.
Thank you
I have an external LB using HAProxy in order to have a HA k8s cluster. My cluster is in K3s from Rancher and it's using Traefik LB internally.
I'm currently facing an issue where in my pods I'm getting the Traefik IP instead of the real client IP.
HAP Configuration:
# Ansible managed
defaults
maxconn 1000
mode http
log global
option dontlognull
log stdout local0 debug
option httplog
timeout http-request 5s
timeout connect 5000
timeout client 2000000
timeout server 2000000
frontend k8s
bind *:6443
bind *:80
bind *:443
mode tcp
option tcplog
use_backend masters-k8s
backend masters-k8s
mode tcp
balance roundrobin
server master01 master01.k8s.int.ntw
server master02 master02.k8s.int.ntw
# end Ansible managed
Traefik Service:
apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: traefik
meta.helm.sh/release-namespace: kube-system
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
labels:
app: traefik
app.kubernetes.io/managed-by: Helm
chart: traefik-1.81.0
heritage: Helm
release: traefik
spec:
clusterIP: 10.43.250.142
clusterIPs:
- 10.43.250.142
externalTrafficPolicy: Local
ports:
- name: http
nodePort: 32232
port: 80
protocol: TCP
targetPort: http
- name: https
nodePort: 30955
port: 443
protocol: TCP
targetPort: https
selector:
app: traefik
release: traefik
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 10.0.1.1
- ip: 10.0.1.11
- ip: 10.0.1.12
- ip: 10.0.1.2
With this configurations I never get my real IP on the pods, during some research I saw that people recommend to use send-proxy in the HAP like this:
server master01 master01.k8s.int.ntw check send-proxy-v2
server master02 master02.k8s.int.ntw check send-proxy-v2
But when I do so, all my cluster communication returns ERR_CONNECTION_CLOSED.
If I'm looking at it correctly, this means its going from the HAP to the cluster and the cluster is somewhere rejecting the traffic.
Any clues what I'm missing here?
Thanks
Well you have two options.
use proxy protocol
use X-Forwarded-For header
Option 1: proxy protocol
This option requires that both sites HAProxy and Traefik uses the proxy protocol, that's the reason why the people recommend "send-proxy-v2".
This requires also that all other clients which want to connect to Traefik MUST also use the proxy protocol, if the clients does not use the proxy protocol then you will get what you get a communication error.
As you configured HAProxy in TCP mode is this the only option to get the client ip to Traefik.
Option 2: X-Forwarded-For
I personally would use this option because it makes it possible to connect to traefik with any http(s) client.
This requires that you use the HTTP mode in haproxy and some more parameters like option forwardfor.
# Ansible managed
defaults
maxconn 1000
mode http
log global
option dontlognull
log stdout local0 debug
option httplog
timeout http-request 5s
timeout connect 5s
timeout client 200s
timeout server 200s
# send client ip in the x-forwarded-for header
option forwardfor
frontend k8s
bind *:6443 v4v6 alpn h2,http/1.1 ssl ca-file /etc/haproxy/letsencryptauthorityx3.pem crt /etc/ssl/haproxy/
bind *:80 v4v6 alpn h2,http/1.1 ssl ca-file /etc/haproxy/letsencryptauthorityx3.pem crt /etc/ssl/haproxy/
bind *:443 v4v6 alpn h2,http/1.1 ssl ca-file /etc/haproxy/letsencryptauthorityx3.pem crt /etc/ssl/haproxy/
use_backend masters-k8s
backend masters-k8s
balance roundrobin
server master01 master01.k8s.int.ntw check
server master02 master02.k8s.int.ntw check
# end Ansible managed
In the File "/etc/haproxy/letsencryptauthorityx3.pem" are the CA's for the backends, in the directory "/etc/ssl/haproxy/" are the certificates for the frontends.
Please take a look into the documentation about the crt keyword.
You have also to configure traefik to allow the header from haproxy forwarded-headers
If it's possible to disable https on some url, i try this, but it's not working.
I need a piece of my site without https and redirect
frontend http
bind *:80
mode http
acl folder path_beg -i ^/somefolder/subfolder/.* ^/somefolder/subfolder2/.*
redirect scheme https if !folder
option http-server-close
reqadd X-Forwarded-Proto:\ http
option forwardfor header X-Real-IP
default_backend nodes
frontend https
bind *:443
mode http
option http-server-close
reqadd X-Forwarded-Proto:\ https
option forwardfor header X-Real-IP
default_backend nodes
backend nodes
balance leastconn
server server1 10.10.10.7:80 cookie A check
server server2 10.10.10.8:80 cookie A check
Access list
acl folder path_dir -i /somefolder/subfolder/ /somefolder/subfolder2/
In backend you need rule
redirect scheme https if !folder !{ ssl_fc }
After that - all site has redirect to htts, but if uri contains /somefolder/subfolder/ or /somefolder/subfolder2/ it's be able to connect by http.
In nginx you need add some rules if you wanna redirect https to http
if ( $http_x_forwarded_proto = "https" ) {
rewrite ^/somefolder/subfolder2/ http://domain//somefolder/subfolder2/ permanent;
}
I am experiencing some problems, it seems I can't get acl's to work in tcp mode, everything works in http mode.
Here is my config.
frontend http *:80
acl http_test_acl path_beg -i /test
use_backend http_test if http_test_acl
default_backend http_default
backend http_test
balance roundrobin
server httptest 10.10.10.10:80 check
backend http_default
balance roundrobin
server httpdefault 10.10.10.10:80 check
############# HTTPS #################
frontend https *:443
mode tcp
acl https_test_acl path_beg -i /test
use_backend https_test if https_test_acl
default_backend https_default
backend https_test
mode tcp
balance roundrobin
server httpstest 10.10.10.10:443 check
backend https_default
mode tcp
balance roundrobin
server httpsdefault 10.10.10.10:443 check
Don't pay attention to ip 10.10.10.10 as I have hidden my orginal one. Could you please let me know why https is not working, http frontend/backend acl rules are working just fine.
cheers
Cause your https servers are in tcp mode (as they should be for ssl), so a layer 7 rule wont work.
for acl to work, disable tcp mode then set up ssl on the servers on your backend(hence the ssl keyword)
frontend https *:443
acl https_test_acl path_beg -i /test
use_backend https_test if https_test_acl
default_backend https_default
backend https_test
balance roundrobin
server httpstest 10.10.10.10:443 ssl check
backend https_default
balance roundrobin
server httpsdefault 10.10.10.10:443 ssl check
Alternatively instead of having to setup ssl on both your backend servers; use private IPS in the backend servers and make sure ports on the backend servers arent open to the world
backend https_test
balance roundrobin
server httpstest some_private_ip:8000 check