haproxy cannot detect services after reboot - kubernetes

Question

I have reids nodes:
NAME READY STATUS RESTARTS AGE
pod/redis-haproxy-deployment-65497cd78d-659tq 1/1 Running 0 31m
pod/redis-sentinel-node-0 3/3 Running 0 81m
pod/redis-sentinel-node-1 3/3 Running 0 80m
pod/redis-sentinel-node-2 3/3 Running 0 80m
pod/ubuntu 1/1 Running 0 85m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/redis-haproxy-balancer ClusterIP 10.43.92.106 <none> 6379/TCP 31m
service/redis-sentinel-headless ClusterIP None <none> 6379/TCP,26379/TCP 99m
service/redis-sentinel-metrics ClusterIP 10.43.72.97 <none> 9121/TCP 99m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/redis-haproxy-deployment 1/1 1 1 31m
NAME DESIRED CURRENT READY AGE
replicaset.apps/redis-haproxy-deployment-65497cd78d 1 1 1 31m
NAME READY AGE
statefulset.apps/redis-sentinel-node 3/3 99m
I connect to the master redis using the following command:
redis-cli -h redis-haproxy-balancer
redis-haproxy-balancer:6379> keys *
1) "sdf"
2) "sdf12"
3) "s4df12"
4) "s4df1"
5) "fsafsdf"
6) "!s4d!1"
7) "s4d!1"
Here is my configuration file haproxy.cfg:
global
daemon
maxconn 256
defaults REDIS
mode tcp
timeout connect 3s
timeout server 3s
timeout client 3s
frontend front_redis
bind 0.0.0.0:6379
use_backend redis_cluster
backend redis_cluster
mode tcp
option tcp-check
tcp-check comment PING\ phase
tcp-check send PING\r\n
tcp-check expect string +PONG
tcp-check comment role\ check
tcp-check send info\ replication\r\n
tcp-check expect string role:master
tcp-check comment QUIT\ phase
tcp-check send QUIT\r\n
tcp-check expect string +OK
server redis-0 redis-sentinel-node-0.redis-sentinel-headless:6379 maxconn 1024 check inter 1s
server redis-1 redis-sentinel-node-1.redis-sentinel-headless:6379 maxconn 1024 check inter 1s
server redis-2 redis-sentinel-node-2.redis-sentinel-headless:6379 maxconn 1024 check inter 1s
Here is the service I go to in order to get to the master redis - haproxy-service.yaml:
apiVersion: v1
kind: Service
metadata:
name: redis-haproxy-balancer
spec:
type: ClusterIP
selector:
app: redis-haproxy
ports:
- protocol: TCP
port: 6379
targetPort: 6379
here is a deployment that refers to a configuration file - redis-haproxy-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-haproxy-deployment
labels:
app: redis-haproxy
spec:
replicas: 1
selector:
matchLabels:
app: redis-haproxy
template:
metadata:
labels:
app: redis-haproxy
spec:
containers:
- name: redis-haproxy
image: haproxy:lts-alpine
volumeMounts:
- name: redis-haproxy-config-volume
mountPath: /usr/local/etc/haproxy/haproxy.cfg
subPath: haproxy.cfg
ports:
- containerPort: 6379
volumes:
- name: redis-haproxy-config-volume
configMap:
name: redis-haproxy-config
items:
- key: haproxy.cfg
path: haproxy.cfg
After restarting redis I cannot connect to it with redis-haproxy-balancer...
[NOTICE] (1) : New worker (8) forked
[NOTICE] (1) : Loading success.
[WARNING] (8) : Server redis_cluster/redis-0 is DOWN, reason: Layer7 timeout, info: " at step 6 of tcp-check (expect string 'role:master')", check duration: 1000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server redis_cluster/redis-1 is DOWN, reason: Layer7 timeout, info: " at step 6 of tcp-check (expect string 'role:master')", check duration: 1005ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server redis_cluster/redis-2 is DOWN, reason: Layer7 timeout, info: " at step 6 of tcp-check (expect string 'role:master')", check duration: 1001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] (8) : backend 'redis_cluster' has no server available!
It only works by connecting directly: redis-sentinel-node-0.redis-sentinel-headless
What is wrong with my haproxy?

Answer 1

You will need to add a resolver section and point it to the kubernetes dns.
Kubernetes: DNS for Services and Pods
HAProxy: Server IP address resolution using DNS
resolvers mydns
nameserver dns1 Kubernetes-DNS-Service-ip:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold other 30s
hold refused 30s
hold nx 30s
hold timeout 30s
hold valid 10s
hold obsolete 30s
backend redis_cluster
mode tcp
option tcp-check
... # your other settings
server redis-0 redis-sentinel-node-0.redis-sentinel-headless:6379 resolvers mydns maxconn 1024 check inter 1s
server redis-1 redis-sentinel-node-1.redis-sentinel-headless:6379 resolvers mydns maxconn 1024 check inter 1s