Nginx Ingress Controller not curling localhost on worker nodes - kubernetes

Question

CentOS 7, 3 VMs -- 1 master and 2 workers, Kubernetes 1.26 (kubelet is 1.25.5.0), cri-dockerd, Calico CNI
# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
rxxxx-vm1 Ready control-plane 4h48m v1.25.5 10.253.137.20 <none> CentOS Linux 7 (Core) 3.10.0-1160.80.1.el7.x86_64 docker://20.10.22
rxxxx-vm2 Ready <none> 4h27m v1.25.5 10.253.137.17 <none> CentOS Linux 7 (Core) 3.10.0-1160.80.1.el7.x86_64 docker://20.10.22
rxxxx-vm3 Ready <none> 4h27m v1.25.5 10.253.137.10 <none> CentOS Linux 7 (Core) 3.10.0-1160.80.1.el7.x86_64 docker://20.10.22
NGINX Ingress controller is deployed as a daemonset:
# kubectl get po -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-apiserver calico-apiserver-685568b969-b8mfr 1/1 Running 0 4h50m 172.17.0.6 rxxxx-vm1 <none> <none>
calico-apiserver calico-apiserver-685568b969-xrj2h 1/1 Running 0 4h50m 172.17.0.7 rxxxx-vm1 <none> <none>
calico-system calico-kube-controllers-67df98bdc8-2zdnj 1/1 Running 0 4h51m 172.17.0.4 rxxxx-vm1 <none> <none>
calico-system calico-node-498bb 1/1 Running 0 4h51m 10.253.137.20 rxxxx-vm1 <none> <none>
calico-system calico-node-sblv9 1/1 Running 0 4h30m 10.253.137.17 rxxxx-vm2 <none> <none>
calico-system calico-node-zkn28 1/1 Running 0 4h29m 10.253.137.10 rxxxx-vm3 <none> <none>
calico-system calico-typha-76c8f59f87-mq52d 1/1 Running 0 4h29m 10.253.137.10 rxxxx-vm3 <none> <none>
calico-system calico-typha-76c8f59f87-zk6jr 1/1 Running 0 4h51m 10.253.137.20 rxxxx-vm1 <none> <none>
kube-system coredns-787d4945fb-6mq5k 1/1 Running 0 4h51m 172.17.0.3 rxxxx-vm1 <none> <none>
kube-system coredns-787d4945fb-kmqcv 1/1 Running 0 4h51m 172.17.0.2 rxxxx-vm1 <none> <none>
kube-system etcd-rxxxx-vm1 1/1 Running 0 4h51m 10.253.137.20 rxxxx-vm1 <none> <none>
kube-system kube-apiserver-rxxxx-vm1 1/1 Running 0 4h51m 10.253.137.20 rxxxx-vm1 <none> <none>
kube-system kube-controller-manager-rxxxx-vm1 1/1 Running 0 4h51m 10.253.137.20 rxxxx-vm1 <none> <none>
kube-system kube-proxy-g9dbt 1/1 Running 0 4h29m 10.253.137.10 rxxxx-vm3 <none> <none>
kube-system kube-proxy-mnzks 1/1 Running 0 4h30m 10.253.137.17 rxxxx-vm2 <none> <none>
kube-system kube-proxy-n98xb 1/1 Running 0 4h51m 10.253.137.20 rxxxx-vm1 <none> <none>
kube-system kube-scheduler-rxxxx-vm1 1/1 Running 0 4h51m 10.253.137.20 rxxxx-vm1 <none> <none>
nginx-ingress nginx-ingress-2chhn 1/1 Running 0 4h29m 172.17.0.2 rxxxx-vm3 <none> <none>
nginx-ingress nginx-ingress-95h7s 1/1 Running 0 4h30m 172.17.0.2 rxxxx-vm2 <none> <none>
nginx-ingress nginx-ingress-wbxng 1/1 Running 0 4h51m 172.17.0.5 rxxxx-vm1 <none> <none>
play apple-app 1/1 Running 0 4h45m 172.17.0.8 rxxxx-vm1 <none> <none>
play banana-app 1/1 Running 0 4h45m 172.17.0.9 rxxxx-vm1 <none> <none>
tigera-operator tigera-operator-7795f5d79b-hmm5g 1/1 Running 0 4h51m 10.253.137.20 rxxxx-vm1 <none> <none>
Services:
# kubectl get svc -A -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
calico-apiserver calico-api ClusterIP 10.111.117.42 <none> 443/TCP 6h34m apiserver=true
calico-system calico-kube-controllers-metrics ClusterIP 10.99.121.254 <none> 9094/TCP 6h35m k8s-app=calico-kube-controllers
calico-system calico-typha ClusterIP 10.104.50.90 <none> 5473/TCP 6h35m k8s-app=calico-typha
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6h36m <none>
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 6h36m k8s-app=kube-dns
play apple-service ClusterIP 10.98.78.251 <none> 5678/TCP 6h29m app=apple
play banana-service ClusterIP 10.103.87.112 <none> 5678/TCP 6h29m app=banana
Service details:
# kubectl -n play describe svc apple-service
Name: apple-service
Namespace: play
Labels: <none>
Annotations: <none>
Selector: app=apple
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.98.78.251
IPs: 10.98.78.251
Port: <unset> 5678/TCP
TargetPort: 5678/TCP
Endpoints: 172.17.0.8:5678
Session Affinity: None
Events: <none>
Endpoints:
# kubectl get ep -A
NAMESPACE NAME ENDPOINTS AGE
calico-apiserver calico-api 172.17.0.6:5443,172.17.0.7:5443 6h39m
calico-system calico-kube-controllers-metrics 172.17.0.4:9094 6h39m
calico-system calico-typha 10.253.137.10:5473,10.253.137.20:5473 6h40m
default kubernetes 10.253.137.20:6443 6h40m
kube-system kube-dns 172.17.0.2:53,172.17.0.3:53,172.17.0.2:53 + 3 more... 6h40m
play apple-service 172.17.0.8:5678 6h34m
play banana-service 172.17.0.9:5678 6h34m
Endpoint details:
# kubectl -n play describe ep apple-service
Name: apple-service
Namespace: play
Labels: <none>
Annotations: endpoints.kubernetes.io/last-change-trigger-time: 2023-01-11T20:21:27Z
Subsets:
Addresses: 172.17.0.8
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 5678 TCP
Events: <none>
Ingress resource:
# kubectl get ing -A -o wide
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
play example-ingress nginx localhost 80 6h30m
Ingress details:
# kubectl -n play describe ing example-ingress
Name: example-ingress
Labels: <none>
Namespace: play
Address:
Ingress Class: nginx
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
localhost
/apple apple-service:5678 (172.17.0.8:5678)
/banana banana-service:5678 (172.17.0.9:5678)
Annotations: ingress.kubernetes.io/rewrite-target: /
Events: <none>
QUESTION:
While curl -kL http://localhost/apple on master node returns apple the same command produces below output on worker nodes:
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.23.2</center>
</body>
</html>
My understanding is that if the ingress controller pod is running on every node, then localhost should be resolved as this is defined as host in ingress resource definition. Is this understanding incorrect? If not, what am I doing wrong?
When I look at the corresponding node's ingress controller's pod's logs, I see this:
2023/01/12 03:02:05 [error] 68#68: *11 connect() failed (113: No route to host) while connecting to upstream, client: 172.17.0.1, server: localhost, request: "GET /apple HTTP/1.1", upstream: "http://172.17.0.8:5678/apple", host: "localhost"
172.17.0.1 - - [12/Jan/2023:03:02:05 +0000] "GET /apple HTTP/1.1" 502 157 "-" "curl/7.29.0" "-"