Nginx Ingress not working on k3s running on Raspberry Pi - kubernetes

Question

I have k3s installed on 4 Raspberry Pi's with traefik disabled.
I'm trying to run Home assistant on it using Nginx Ingress controller, kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.0/deploy/static/provider/baremetal/deploy.yaml.
But for some reason, I just cannot expose the service. The ingress assigned 192.168.0.57, which is one of the nodes' IP. Am I missing something?
root#rpi1:~# kubectl get ingress -n home-assistant home-assistant-ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
home-assistant-ingress nginx smart.home 192.168.0.57 80 20h
root#rpi1:~# curl http://192.168.0.57/
curl: (7) Failed to connect to 192.168.0.57 port 80: Connection refused
root#rpi1:~# curl http://smart.home/
curl: (7) Failed to connect to smart.home port 80: Connection refused
Please see following.
Pod:
root#rpi1:~# kubectl describe pod -n home-assistant home-assistant-deploy-7c4674b679-zbwn7
Name: home-assistant-deploy-7c4674b679-zbwn7
Namespace: home-assistant
Priority: 0
Node: rpi4/192.168.0.58
Start Time: Tue, 16 Aug 2022 20:31:28 +0100
Labels: app=home-assistant
pod-template-hash=7c4674b679
Annotations: <none>
Status: Running
IP: 10.42.3.7
IPs:
IP: 10.42.3.7
Controlled By: ReplicaSet/home-assistant-deploy-7c4674b679
Containers:
home-assistant:
Container ID: containerd://c7ec189112e9f2d085bd7f9cc7c8086d09b312e30771d7d1fef424685fcfbd07
Image: ghcr.io/home-assistant/home-assistant:stable
Image ID: ghcr.io/home-assistant/home-assistant#sha256:0555dc6a69293a1a700420224ce8d03048afd845465f836ef6ad60f5763b44f2
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 17 Aug 2022 18:06:16 +0100
Last State: Terminated
Reason: Unknown
Exit Code: 255
Started: Tue, 16 Aug 2022 20:33:33 +0100
Finished: Wed, 17 Aug 2022 18:06:12 +0100
Ready: True
Restart Count: 1
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n5tb7 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-n5tb7:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SandboxChanged 43m kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 43m kubelet Container image "ghcr.io/home-assistant/home-assistant:stable" already present on machine
Normal Created 43m kubelet Created container home-assistant
Normal Started 43m kubelet Started container home-assistant
The pod is listening at port 8123
root#rpi1:~# kubectl exec -it -n home-assistant home-assistant-deploy-7c4674b679-zbwn7 -- netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8123 0.0.0.0:* LISTEN 60/python3
tcp6 0 0 :::8123 :::* LISTEN 60/python3
Deployment:
root#rpi1:~# kubectl describe deployments.apps -n home-assistant
Name: home-assistant-deploy
Namespace: home-assistant
CreationTimestamp: Tue, 16 Aug 2022 20:31:28 +0100
Labels: app=home-assistant
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=home-assistant
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=home-assistant
Containers:
home-assistant:
Image: ghcr.io/home-assistant/home-assistant:stable
Port: <none>
Host Port: <none>
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: home-assistant-deploy-7c4674b679 (1/1 replicas created)
Events: <none>
Service with port set to 8080 and target port to 8123:
root#rpi1:~# kubectl describe svc -n home-assistant home-assistant-service
Name: home-assistant-service
Namespace: home-assistant
Labels: app=home-assistant
Annotations: <none>
Selector: app=home-assistant
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.248.75
IPs: 10.43.248.75
LoadBalancer Ingress: 192.168.0.53, 192.168.0.56, 192.168.0.57, 192.168.0.58
Port: <unset> 8080/TCP
TargetPort: 8123/TCP
NodePort: <unset> 31678/TCP
Endpoints: 10.42.3.7:8123
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal UpdatedIngressIP 20h svccontroller LoadBalancer Ingress IP addresses updated: 192.168.0.53, 192.168.0.56, 192.168.0.58
Normal UpdatedIngressIP 20h (x2 over 22h) svccontroller LoadBalancer Ingress IP addresses updated: 192.168.0.53, 192.168.0.56, 192.168.0.57, 192.168.0.58
Normal AppliedDaemonSet 20h (x19 over 22h) svccontroller Applied LoadBalancer DaemonSet kube-system/svclb-home-assistant-service-f2675711
Normal UpdatedIngressIP 47m svccontroller LoadBalancer Ingress IP addresses updated: 192.168.0.53, 192.168.0.56
Normal UpdatedIngressIP 47m svccontroller LoadBalancer Ingress IP addresses updated: 192.168.0.53, 192.168.0.56, 192.168.0.57
Normal UpdatedIngressIP 47m svccontroller LoadBalancer Ingress IP addresses updated: 192.168.0.53, 192.168.0.56, 192.168.0.57, 192.168.0.58
Normal AppliedDaemonSet 47m (x8 over 47m) svccontroller Applied LoadBalancer DaemonSet kube-system/svclb-home-assistant-service-f2675711
My Ingress:
root#rpi1:~# kubectl describe ingress -n home-assistant home-assistant-ingress
Name: home-assistant-ingress
Labels: <none>
Namespace: home-assistant
Address: 192.168.0.57
Ingress Class: nginx
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
smart.home
/ home-assistant-service:8080 (10.42.3.7:8123)
Annotations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 19h (x2 over 19h) nginx-ingress-controller Scheduled for sync
Normal Sync 49m (x3 over 50m) nginx-ingress-controller Scheduled for sync
root#rpi1:~# kubectl get ingress -n home-assistant home-assistant-ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
home-assistant-ingress nginx smart.home 192.168.0.57 80 19h
Can confirm I have Nginx ingress controller running:
root#rpi1:~# kubectl get pod -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-2thj7 0/1 Completed 0 22h
ingress-nginx-admission-patch-kwm4m 0/1 Completed 1 22h
ingress-nginx-controller-6dc865cd86-9h8wt 1/1 Running 2 (52m ago) 22h
Ingress Nginx Controller log
root#rpi1:~# kubectl logs -n ingress-nginx ingress-nginx-controller-6dc865cd86-9h8wt
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.3.0
Build: 2b7b74854d90ad9b4b96a5011b9e8b67d20bfb8f
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.10
-------------------------------------------------------------------------------
W0818 06:51:52.008386 7 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0818 06:51:52.009962 7 main.go:230] "Creating API client" host="https://10.43.0.1:443"
I0818 06:51:52.123762 7 main.go:274] "Running in Kubernetes cluster" major="1" minor="24" git="v1.24.3+k3s1" state="clean" commit="990ba0e88c90f8ed8b50e0ccd375937b841b176e" platform="linux/arm64"
I0818 06:51:52.594773 7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0818 06:51:52.691571 7 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0818 06:51:52.773089 7 nginx.go:258] "Starting NGINX Ingress controller"
I0818 06:51:52.807863 7 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"21ae6485-bb0e-447e-b098-c510e43b171e", APIVersion:"v1", ResourceVersion:"934", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0818 06:51:53.912887 7 store.go:429] "Found valid IngressClass" ingress="home-assistant/home-assistant-ingress" ingressclass="nginx"
I0818 06:51:53.913414 7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"home-assistant", Name:"home-assistant-ingress", UID:"eeb12441-9cd4-4571-b0da-5b2978ff3267", APIVersion:"networking.k8s.io/v1", ResourceVersion:"8719", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0818 06:51:53.975141 7 nginx.go:301] "Starting NGINX process"
I0818 06:51:53.975663 7 leaderelection.go:248] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I0818 06:51:53.976173 7 nginx.go:321] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0818 06:51:53.980492 7 controller.go:167] "Configuration changes detected, backend reload required"
I0818 06:51:54.025524 7 leaderelection.go:258] successfully acquired lease ingress-nginx/ingress-controller-leader
I0818 06:51:54.025924 7 status.go:84] "New leader elected" identity="ingress-nginx-controller-6dc865cd86-9h8wt"
I0818 06:51:54.039912 7 status.go:214] "POD is not ready" pod="ingress-nginx/ingress-nginx-controller-6dc865cd86-9h8wt" node="rpi3"
I0818 06:51:54.051540 7 status.go:299] "updating Ingress status" namespace="home-assistant" ingress="home-assistant-ingress" currentValue=[{IP:192.168.0.57 Hostname: Ports:[]}] newValue=[]
I0818 06:51:54.071502 7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"home-assistant", Name:"home-assistant-ingress", UID:"eeb12441-9cd4-4571-b0da-5b2978ff3267", APIVersion:"networking.k8s.io/v1", ResourceVersion:"14445", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0818 06:51:54.823911 7 controller.go:184] "Backend successfully reloaded"
I0818 06:51:54.824200 7 controller.go:195] "Initial sync, sleeping for 1 second"
I0818 06:51:54.824334 7 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-6dc865cd86-9h8wt", UID:"def1db3a-4766-4751-b611-ae3461911bc6", APIVersion:"v1", ResourceVersion:"14423", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W0818 06:51:57.788759 7 controller.go:1111] Service "home-assistant/home-assistant-service" does not have any active Endpoint.
I0818 06:52:54.165805 7 status.go:299] "updating Ingress status" namespace="home-assistant" ingress="home-assistant-ingress" currentValue=[] newValue=[{IP:192.168.0.57 Hostname: Ports:[]}]
I0818 06:52:54.190556 7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"home-assistant", Name:"home-assistant-ingress", UID:"eeb12441-9cd4-4571-b0da-5b2978ff3267", APIVersion:"networking.k8s.io/v1", ResourceVersion:"14590", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
Endpoints
root#rpi1:~# kubectl get endpoints -A
NAMESPACE NAME ENDPOINTS AGE
default kubernetes 192.168.0.53:6443 35h
kube-system kube-dns 10.42.0.12:53,10.42.0.12:53,10.42.0.12:9153 35h
home-assistant home-assistant-service 10.42.3.9:8123 35h
kube-system metrics-server 10.42.0.14:4443 35h
ingress-nginx ingress-nginx-controller-admission 10.42.2.13:8443 35h
ingress-nginx ingress-nginx-controller 10.42.2.13:443,10.42.2.13:80 35h
Can also confirm the Traefik Ingress controller is disabled
root#rpi1:~# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-admission-create-2thj7 0/1 Completed 0 22h
ingress-nginx ingress-nginx-admission-patch-kwm4m 0/1 Completed 1 22h
kube-system local-path-provisioner-7b7dc8d6f5-jcm4p 1/1 Running 1 (59m ago) 22h
kube-system svclb-home-assistant-service-f2675711-w88fv 1/1 Running 1 (59m ago) 22h
kube-system coredns-b96499967-rml6k 1/1 Running 1 (59m ago) 22h
kube-system svclb-home-assistant-service-f2675711-rv8rf 1/1 Running 1 (59m ago) 22h
kube-system svclb-home-assistant-service-f2675711-9qk8m 1/1 Running 2 (59m ago) 22h
kube-system svclb-home-assistant-service-f2675711-m62sl 1/1 Running 1 (59m ago) 22h
home-assistant home-assistant-deploy-7c4674b679-zbwn7 1/1 Running 1 (59m ago) 22h
kube-system metrics-server-668d979685-rp2wm 1/1 Running 1 (59m ago) 22h
ingress-nginx ingress-nginx-controller-6dc865cd86-9h8wt 1/1 Running 2 (59m ago) 22h
Ingress Nginx Controller Service:
root#rpi1:~# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.43.254.114 <none> 80:32313/TCP,443:31543/TCP 23h
ingress-nginx-controller-admission ClusterIP 10.43.135.213 <none> 443/TCP 23h
root#rpi1:~# kubectl describe svc -n ingress-nginx ingress-nginx-controller
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.3.0
Annotations: <none>
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.254.114
IPs: 10.43.254.114
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 32313/TCP
Endpoints: 10.42.2.10:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31543/TCP
Endpoints: 10.42.2.10:443
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Updated: added Ingress Nginx Controller service
Updated2: added Ingress Nginx Controller log and endpoints