Haproxy SSL(https) health checks without terminating ssl - haproxy

so I can't figure out a proper way to do the SSL check, I am not using certificates, just need to check against a HTTPS websites url (google.com/ for example)
Trying multiple combinations at a time, without success. Maybe someone has a similar configuration,
backends using -
> check-sni google.com sni ssl_fc_sni
returns - reason: Layer7 wrong status, code: 301, info: "Moved Permanently"
check port 80 check-ssl -
reason: Layer6 invalid response, info: "SSL handshake failure"
All others just timing out. Here's the complete configuration file-
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-server-verify none
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
# https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend myfront
bind *:8000
mode tcp
tcp-request inspect-delay 5s
default_backend backend1
listen stats
bind :444
stats enable
stats uri /
stats hide-version
stats auth test:test
backend Backends
balance roundrobin
option forwardfor
option httpchk
http-check send hdr host google.com meth GET uri /
http-check expect status 200
#http-check connect
#http-check send meth GET uri / ver HTTP/1.1 hdr host haproxy.1wt.eu
#http-check expect status 200-399
#http-check connect port 443 ssl sni haproxy.1wt.eu
#http-check send meth GET uri / ver HTTP/1.1 hdr host haproxy.1wt.eu
#http-check expect status 200-399
#http-check connect port 443 ssl sni google.com
#http-check send meth GET uri / ver HTTP/1.1 hdr host google.com
default-server fall 10 rise 1
server Node1011 192.168.0.2:1011 check inter 15s check-ssl check port 443
server Node1012 192.168.0.2:1012 check inter 15s check-ssl check port 443
server Node1015 192.168.0.2:1015 check inter 15s check port 443
server Node1017 192.168.0.2:1017 check inter 15s check-ssl check-sni google.com sni ssl_fc_sni
server Node1018 192.168.0.2:1018 check inter 15s check-ssl check-sni google.com sni ssl_fc_sni
server Node1019 192.168.0.2:1019 check inter 15s check-sni google.com sni ssl_fc_sni
server Node1020 192.168.0.2:1020 check inter 15s check port 443 check-ssl
server Node1021 192.168.0.2:1021 check inter 15s check port 443 check-ssl
server Node1027 192.168.0.2:1027 check inter 15s check port 80
server Node1028 192.168.0.2:1028 check inter 15s check port 80
server Node1029 192.168.0.2:1029 check inter 15s check port 80
server Node1030 192.168.0.2:1030 check inter 15s check port 80 check-ssl
server Node1031 192.168.0.2:1031 check inter 15s check port 80 check-ssl
server Node1033 192.168.0.2:1033 check inter 15s check port 80 check-ssl verify none
server Node1034 192.168.0.2:1034 check inter 15s check port 80 check-ssl verify none
server Node1035 192.168.0.2:1035 check inter 15s check-ssl
server Node1036 192.168.0.2:1036 check inter 15s check-ssl
server Node1048 192.168.0.2:1048 check inter 15s check-ssl verify none
server Node1049 192.168.0.2:1049 check inter 15s check-ssl verify none
P.s Found a website, which explains just what I'm trying to do(https://hodari.be/posts/2020_09_04_configure_sni_for_haproxy_backends/), but that doesn't work either, my haproxy version is 2.2.3
P.s.s I am literally trying to check against www.google.com , just to be clear.
Thank you!

That's really not an error. If you do a curl to https://google.com it does do a 301 redirect to https://www.google.com/. I snipped out some protocol details below for brevity, but you get the idea.
Either change your expect to 301, or use www.google.com.
paul:~ $ curl -vv https://google.com
* Rebuilt URL to: https://google.com/
* Trying 172.217.1.206...
-[snip]-
> GET / HTTP/2
> Host: google.com
> User-Agent: curl/7.58.0
> Accept: */*
>
-[snip]-
< HTTP/2 301
< location: https://www.google.com/
< content-type: text/html; charset=UTF-8
< date: Mon, 18 Jan 2021 03:42:04 GMT
< expires: Wed, 17 Feb 2021 03:42:04 GMT
< cache-control: public, max-age=2592000
< server: gws
< content-length: 220
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< alt-svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
<
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
here.
</BODY></HTML>
So, if you want to avoid the 301, use the www.google.com value in your config as thus:
http-check send hdr host www.google.com meth GET uri /

Related

Issue with HAProxy not retrying on retry-on

We were having issues with Apache mod_proxy getting random 502/503 errors from our backend server (that we don’t control), so we decided to give HAProxy a shot in testing. We setup HAProxy and got the same errors, so decided to try the retry-on all-retryable-errors but keep getting the same errors. We would have thought that HAProxy would have attempted retries on this but it doesn’t seem to be happening.
For testing, we do a WGET every half a second for 10000 tries. out of the 10000 tries, we get about 10 errors.
Didn’t know if someone could look at our setup and logs to help us determine why the retry isn’t occuring.
haproxy.cfg
global
log 127.0.0.1 local2 debug
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend main
bind 127.0.0.1:5000
default_backend app
mode http
backend app
balance roundrobin
http-send-name-header Host
retry-on all-retryable-errors
retries 10
http-request disable-l7-retry if METH_POST
server srv1 backend-server:443 ssl verify none
server srv2 backend-server:443 ssl verify none
server srv3 backend-server:443 ssl verify none
haproxy.log (you see the 502 error in the middle of the log)
Apr 27 19:01:29 localhost haproxy[26058]: 127.0.0.1:58028 [27/Apr/2022:19:01:29.769] main app/srv1 0/0/5/111/116 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:30 localhost haproxy[12306]: 127.0.0.1:58032 [27/Apr/2022:19:01:30.430] main app/srv2 0/0/2/119/121 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:31 localhost haproxy[8726]: 127.0.0.1:58036 [27/Apr/2022:19:01:31.099] main app/srv2 0/0/6/114/120 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:33 localhost haproxy[26058]: 127.0.0.1:58040 [27/Apr/2022:19:01:31.764] main app/srv2 0/0/6/-1/1385 502 209 - - SH-- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:33 localhost haproxy[8726]: 127.0.0.1:58044 [27/Apr/2022:19:01:33.695] main app/srv3 0/0/10/112/122 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:34 localhost haproxy[26058]: 127.0.0.1:58048 [27/Apr/2022:19:01:34.362] main app/srv3 0/0/3/113/116 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:45 localhost haproxy[8726]: 127.0.0.1:58052 [27/Apr/2022:19:01:35.023] main app/srv1 0/0/16/10552/10568 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
output from the wget:
--2022-04-27 19:01:31-- http://localhost:5000/PBI_PBI1151/Login/RemoteInitialize/053103585
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:5000... connected.
HTTP request sent, awaiting response... 502 Bad Gateway
2022-04-27 19:01:33 ERROR 502: Bad Gateway.

Kubernetes service under default namespace is unreachable or throws connection refused error

I am trying to deploy the 'lighthouse' application in kubernetes cluster with 'v1.18.14' and the pods are up and running but the logs shows that there is a connection refused error on kubernettes service ip 10.233.0.1 on port 443.
Logs of lighthouse pod -
[centos#master elk_stack_6.x]$ kubectl logs lighthouse-webhooks-7f58c9897c-tbvfz -n jx
{"fields.level":"info","level":"info","msg":"setting the log level","time":"2021-01-21T03:35:52Z"}
{"level":"info","msg":"updating the Lighthouse core configuration","time":"2021-01-21T03:35:52Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2021-01-21T03:35:52Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2021-01-21T03:35:52Z"}
{"level":"info","msg":"updating the Lighthouse plugins configuration","time":"2021-01-21T03:35:52Z"}
{"level":"warning","msg":"not pushing metrics as there is no push_gateway defined in the config.yaml","time":"2021-01-21T03:35:52Z"}
{"level":"info","msg":"Lighthouse is now listening on path /hook and port 8080 for WebHooks","time":"2021-01-21T03:35:52Z"}
{"level":"info","msg":"Lighthouse is serving prometheus metrics on port 2112","time":"2021-01-21T03:35:52Z"}
E0129 07:23:34.030912 1 reflector.go:309] pkg/mod/k8s.io/client-go#v0.17.6/tools/cache/reflector.go:105: Failed to watch *v1.ConfigMap: Get https://10.233.0.1:443/api/v1/namespaces/jx/configmaps?watch=true: dial tcp 10.233.0.1:443: connect: connection refused
E0129 07:23:35.031560 1 reflector.go:309] pkg/mod/k8s.io/client-go#v0.17.6/tools/cache/reflector.go:105: Failed to watch *v1.ConfigMap: Get https://10.233.0.1:443/api/v1/namespaces/jx/configmaps?watch=true: dial tcp 10.233.0.1:443: connect: connection refused
Service under default namespace -
[centos#master elk_stack_6.x]$ kubectl get svc -n default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.233.0.1 <none> 443/TCP 12d
Endpoints of default namespace -
[centos#master elk_stack_6.x]$ kubectl get endpoints -n default
NAME ENDPOINTS AGE
kubernetes 167.254.204.56:6443 12d
Trying to connect kubernetes service using curl -
[centos#master ~]$ curl -kv https://10.233.0.1:443
* About to connect() to 10.233.0.1 port 443 (#0)
* Trying 10.233.0.1...
* Connected to 10.233.0.1 (10.233.0.1) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=kube-apiserver
* start date: Jan 20 13:49:12 2021 GMT
* expire date: Jan 20 13:49:12 2022 GMT
* common name: kube-apiserver
* issuer: CN=kubernetes
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.233.0.1
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Cache-Control: no-cache, private
< Content-Type: application/json
< X-Content-Type-Options: nosniff
< Date: Tue, 02 Feb 2021 07:57:21 GMT
< Content-Length: 233
<
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {
},
"code": 403
* Connection #0 to host 10.233.0.1 left intact
I am facing 403 forbidden error may be because of some missing RBAC policies. Any suggestions would be appreciated.
Kindly try the HTTPS endpoint as mentioned in the error instead of HTTP.
400 error would mean that the request is indeed reaching the server but not in a format that the server is expecting.
Updated to reflect updated question:-
403 means the user via which the cluster is being accessed is not authorised to use the IP for accessing the resource.
Detailed explanation can be found protecting your kubernetes api server
As per pod logs and Curl command output, it may be due to permission issue with Service Account the pod is running with. Check if Service Account of the pod has proper permission to watch/read configMap in the default namespace. If not there, then RBAC role and role binding K8S objects need to be created.

Nginx Controller in Kubernetes: Handshaking to upstream - peer closed connection in SSL handshake

On a local environment built with kubeadm. The cluster is made of Master and 2 worker nodes.
What it is failing:
Exposing Nginx-ingress Controller to external by Service type LoadBalancer, trying TLS termination to the Kubernetes cluster.
Here the exposed Service:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
nginx-ingress LoadBalancer 10.101.5.75 192.168.1.82 80:30745/TCP,443:30092/TCP
web-service ClusterIP 10.101.26.176 <none> 8080/TCP
What it's working:
Able to reach the web application with HTTP port 80 from external.
Here the pods in the cluster:
NAME READY STATUS RESTARTS AGE IP NODE
nginx-ingress-7c5588544d-4mw9d 1/1 Running 0 38m 10.44.0.2 node1
web-deployment-7d84778bc6-52pq7 1/1 Running 1 19h 10.44.0.1 node1
web-deployment-7d84778bc6-9wwmn 1/1 Running 1 19h 10.36.0.2 node2
Test result for TLS termination
Client side:
curl -k https://example.com -v
* Rebuilt URL to https://example.com/
* Trying 192.168.1.82...
* Connected to example.com (192.168.1.82) port 443 (#0)
* found 127 certificates in /etc/ssl/certs/ca-certificates.crt
* found 513 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification SKIPPED
* server certificate status verification SKIPPED
* common name: example.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #1
* subject: CN=example.com
* start date: Thu, 22 Oct 2020 16:33:49 GMT
* expire date: Fri, 22 Oct 2021 16:33:49 GMT
* issuer: CN=My Cert Authority
* compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 502 Bad Gateway
< Server: nginx/1.19.3
Server side (logs from nginx-ingress pod):
GET / HTTP/1.1
Connection: close
Host: example.com
X-Real-IP: 10.32.0.1
X-Forwarded-For: 10.32.0.1
X-Forwarded-Host: example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
User-Agent: curl/7.47.0
Accept: */*
2020/10/22 18:38:59 [error] 23#23: *12 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 10.32.0.1, server: example.com, request: "GET / HTTP/1.1", upstream: "https://10.44.0.1:8081/", host: "example.com"
2020/10/22 18:38:59 [warn] 23#23: *12 upstream server temporarily disabled while SSL handshaking to upstream, client: 10.32.0.1, server: example.com, request: "GET / HTTP/1.1", upstream: "https://10.44.0.1:8081/", host: "example.com"
HTTP/1.1 502 Bad Gateway
Server: nginx/1.19.3
What i checked
Generated a CA and Server certificate following the link:
https://kubernetes.github.io/ingress-nginx/examples/PREREQUISITES/#tls-certificates
Checked the server certificate and the server key under /etc/nginx/secrets/default of the nginx-ingress Pod and they look correct. Here the output of the resource VirtualServer:
NAME STATE HOST IP PORTS AGE
vs-example Valid example.com 192.168.1.82 [80,443] 121m
VirtualServer:
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: vs-example
namespace: nginx-ingress
spec:
host: example.com
tls:
secret: example-tls
upstreams:
- name: example
service: web-service
port: 8080
tls:
enable: true
routes:
- path: /v2
action:
pass: example
routes:
- path: /
action:
pass: example
Secret
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: nginx-ingress
data:
tls.crt: (..omitted..)
tls.key: (..omitted..)
type: kubernetes.io/tls
Nginx.conf
Here an extract of the nginx.conf taken from the running Pod:
server {
# required to support the Websocket protocol in VirtualServer/VirtualServerRoutes
set $default_connection_header "";
listen 80 default_server;
listen 443 ssl default_server;
ssl_certificate /etc/nginx/secrets/default;
ssl_certificate_key /etc/nginx/secrets/default;
server_name _;
server_tokens "on";
Cannot find what's in going on when trying to reach the web application with HTTPS.
several errors in the VirtualServer, edited with these changes solved the problem:
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: vs-example
namespace: nginx-ingress
spec:
host: example.com
tls:
secret: example-tls
upstreams:
- name: example
service: web-service
port: 8080
# tls: --> this caused the SSL issue in the upstream
# enable: true
routes:
- path: /
action:
pass: example
- path: /v1
action:
pass: example
accessing web application with HTTPS now works

proxy Memcache_Servers has no server available

proxy Memcache_Servers has no server available, when I start the haproxy.service:
[root#ha-node1 log]# systemctl restart haproxy.service
Message from syslogd#localhost at Aug 2 10:49:23 ...
haproxy[81665]: proxy Memcache_Servers has no server available!
The configuration in my haproxy.cfg:
listen Memcache_Servers
bind 45.117.40.168:11211
balance roundrobin
mode tcp
option tcpka
server ha-node1 ha-node1:11211 check inter 10s fastinter 2s downinter 2s rise 30 fall 3
server ha-node2 ha-node2:11211 check inter 10s fastinter 2s downinter 2s rise 30 fall 3
server ha-node3 ha-node3:11211 check inter 10s fastinter 2s downinter 2s rise 30 fall 3
At last, I found the ip in my hosts is like below:
[root#ha-node1 sysconfig]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.8.101 ha-node1 ha-node1.aa.com
192.168.8.102 ha-node2 ha-node2.aa.com
192.168.8.103 ha-node3 ha-node3.aa.com
45.117.40.168 ha-vhost devops.aa.com
192.168.8.104 nfs-backend backend.aa.com
But in my /etc/sysconfig/memcached, the ip is not the host ip before, so I changed to the ip in the hosts:
Now I restart the memcached and haproxy, it works normal now.

Haproxy doesnt check backend in tcp mode

there is a problem with checking health status in haproxy 1.5.
In sample-backend or sample-backend2 haproxy doesnt check status in tcp mode, it always checks in L7 mode even if i specify tcp mode. And it always UP.
Here is my config:
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
defaults
mode tcp
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend haproxy_in
mode http
bind 172.25.0.33:80
option forwardfor header X-Real-IP
acl host_static hdr_beg(host) -i sample.
use_backend sample-backend if host_static
acl host_static hdr_beg(host) -i af-ws.
use_backend sample-backend if host_static
default_backend haproxy_http
frontend haproxy_in_htpps
bind 172.25.0.33:443
mode tcp
use_backend haproxy_https
backend haproxy_https
balance roundrobin
mode tcp
option httpchk OPTIONS /manager/html
http-check expect status 401
option forwardfor
server web1 172.25.0.35:443 check addr 172.25.0.35 port 8085 inter 5000
server web2 172.25.0.36:443 check addr 172.25.0.36 port 8085 inter 5000
backend haproxy_http
balance roundrobin
mode http
option httpchk
option forwardfor
server web1 172.25.0.35:80 check
server web2 172.25.0.36:80 check backup
backend sample-backend
mode http
balance roundrobin
# option httpchk get /?action=Ping
# option forwardfor
option tcp-check
# server web4 172.25.0.38:80 check addr 172.25.0.38 port 8888 inter 5000
# server web3 172.25.0.37:80 check addr 172.25.0.37 port 8888 inter 5000
server test 10.41.41.240:8888 check addr 10.41.41.240 port 8888 inter 5000
server test1 172.25.0.37:9999 check addr 172.25.0.37 port 9999 inter 5000
backend sample-backend2
mode tcp
balance roundrobin
# option httpchk get /?action=Ping
# option forwardfor
option tcp-check
# server web4 172.25.0.38:80 check addr 172.25.0.38 port 8888 inter 5000
# server web3 172.25.0.37:80 check addr 172.25.0.37 port 8888 inter 5000
server test2 10.41.41.240:8888 check addr 10.41.41.240 port 8888 inter 5000
server test3 172.25.0.37:9999 check addr 172.25.0.37 port 9999 inter 5000
Where is my mistake ? Thanks!
My version was 1.5.4 , so it has bug in tcpcheck. Update to latest version and it works.