Kubernetes on SuSE: NodePort Service Problems - kubernetes

Question

for days I'm trying to track down a weird behaviour concerning NodePort Services when running Kubernetes on openSUSE Leap 15.3.
For testing purposes on my own server I installed 3 VMs with openSUSE 15.3. With this article: How to install kubernetes in Suse Linux enterprize server 15 virtual machines? I set up this Kubernetes Cluster:
kubix01:~ # k get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubix01 Ready control-plane,master 25h v1.22.2 192.168.42.51 <none> openSUSE Leap 15.3 5.3.18-59.27-default docker://20.10.6-ce
kubix02 Ready <none> 25h v1.22.2 192.168.42.52 <none> openSUSE Leap 15.3 5.3.18-59.27-default docker://20.10.6-ce
kubix03 Ready <none> 25h v1.22.2 192.168.42.53 <none> openSUSE Leap 15.3 5.3.18-59.27-default docker://20.10.6-ce
For testing things out I made a new 3 Replica Deployment for a traefik/whoami Image with this yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: whoami
labels:
app: whoami
spec:
replicas: 3
selector:
matchLabels:
app: whoami
template:
metadata:
labels:
app: whoami
spec:
containers:
- name: whoami
image: traefik/whoami
ports:
- containerPort: 80
This results in three Pods spread over the 2 worker nodes as expected:
kubix01:~/k8s/whoami # k get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
whoami-8557b59f65-2qkvq 1/1 Running 2 (24h ago) 25h 10.244.2.7 kubix03 <none> <none>
whoami-8557b59f65-4wnmd 1/1 Running 2 (24h ago) 25h 10.244.1.6 kubix02 <none> <none>
whoami-8557b59f65-xhx5x 1/1 Running 2 (24h ago) 25h 10.244.1.7 kubix02 <none> <none>
After that I created a NodePort Service for making things available to the world with this yaml:
apiVersion: v1
kind: Service
metadata:
name: whoami
spec:
type: NodePort
selector:
app: whoami
ports:
- protocol: TCP
port: 8080
targetPort: 80
nodePort: 30080
This is the result:
kubix01:~/k8s/whoami # k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 25h
whoami NodePort 10.105.214.86 <none> 8080:30080/TCP 25h
kubix01:~/k8s/whoami # k describe svc whoami
Name: whoami
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=whoami
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.105.214.86
IPs: 10.105.214.86
Port: <unset> 8080/TCP
TargetPort: 80/TCP
NodePort: <unset> 30080/TCP
Endpoints: 10.244.1.6:80,10.244.1.7:80,10.244.2.7:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
So everything looks fine and I tested things out with curl:
curl on one Cluster Node to PodIP:PodPort
kubix01:~/k8s/whoami # curl 10.244.1.6
Hostname: whoami-8557b59f65-4wnmd
IP: 127.0.0.1
IP: 10.244.1.6
RemoteAddr: 10.244.0.0:50380
GET / HTTP/1.1
Host: 10.244.1.6
User-Agent: curl/7.66.0
Accept: */*
kubix01:~/k8s/whoami # curl 10.244.1.7
Hostname: whoami-8557b59f65-xhx5x
IP: 127.0.0.1
IP: 10.244.1.7
RemoteAddr: 10.244.0.0:36062
GET / HTTP/1.1
Host: 10.244.1.7
User-Agent: curl/7.66.0
Accept: */*
kubix01:~/k8s/whoami # curl 10.244.2.7
Hostname: whoami-8557b59f65-2qkvq
IP: 127.0.0.1
IP: 10.244.2.7
RemoteAddr: 10.244.0.0:43924
GET / HTTP/1.1
Host: 10.244.2.7
User-Agent: curl/7.66.0
Accept: */*
==> Everything works as expected
curl on Cluster Node to services ClusterIP:ClusterPort:
kubix01:~/k8s/whoami # curl 10.105.214.86:8080
Hostname: whoami-8557b59f65-xhx5x
IP: 127.0.0.1
IP: 10.244.1.7
RemoteAddr: 10.244.0.0:1106
GET / HTTP/1.1
Host: 10.105.214.86:8080
User-Agent: curl/7.66.0
Accept: */*
kubix01:~/k8s/whoami # curl 10.105.214.86:8080
Hostname: whoami-8557b59f65-4wnmd
IP: 127.0.0.1
IP: 10.244.1.6
RemoteAddr: 10.244.0.0:9707
GET / HTTP/1.1
Host: 10.105.214.86:8080
User-Agent: curl/7.66.0
Accept: */*
kubix01:~/k8s/whoami # curl 10.105.214.86:8080
Hostname: whoami-8557b59f65-2qkvq
IP: 127.0.0.1
IP: 10.244.2.7
RemoteAddr: 10.244.0.0:25577
GET / HTTP/1.1
Host: 10.105.214.86:8080
User-Agent: curl/7.66.0
Accept: */*
==> Everything fine, Traffic is LoadBalanced to the different pods.
curl on Cluster Node to NodeIP:NodePort
kubix01:~/k8s/whoami # curl 192.168.42.51:30080
Hostname: whoami-8557b59f65-2qkvq
IP: 127.0.0.1
IP: 10.244.2.7
RemoteAddr: 10.244.0.0:5463
GET / HTTP/1.1
Host: 192.168.42.51:30080
User-Agent: curl/7.66.0
Accept: */*
kubix01:~/k8s/whoami # curl 192.168.42.52:30080
^C [NoAnswer]
kubix01:~/k8s/whoami # curl 192.168.42.53:30080
^C [NoAnswer]
==> NodePort Service is only working at the same Node, no answer from the other nodes
curl from another Network Host to NodeIP:NodePort
user#otherhost:~$ curl 192.168.42.51:30080
^C [NoAnswer]
user#otherhost:~$ curl 192.168.42.52:30080
^C [NoAnswer]
user#otherhost:~$ curl 192.168.42.53:30080
^C [NoAnswer]
==> Service is not reachable from the outside at all, no answer on all nodes
Has anybody an idea what is going wrong here?
Thx in advance
T0mcat
PS:
Additionally here a little image for clearing things a bit more. Red curved arrows are the non - working connections, gree curved arrows the working ones:

Answer 1

FINALLY found the solution:
Weird behaviour of SUSE concerning ip_forward sysctl setting. During OS Installation the Option "IPv4 forwarding" was activated in the installer. After OS was installed, additionally added "net.ipv4.ip_forward = 1" and
"net.ipv4.conf.all.forwarding = 1" in /etc/sysctl.conf.
With "sysctl -l|grep ip_forward" checked, both options where activated but this obviously wasn't the truth.
Found the file "/etc/sysctl.d/70-yast.conf"... here both options where set to 0, which obviously was the truth. But manually setting these options to 1 in this file still weren't enough...
Using SUSEs YAST tool and entering "System/Network Settings" -> Routing one can see the Option "Enable IPv4 Forwarding", which was checked (remember: in 70-yast.conf these options where set to 0). After playing a bit with the options in YAST, so that the 70-yast.conf gets rewritten and both options where set to 1 in 70-yast.conf, NodePort services are working as expected now.
Conclusion:
IMHO this seems to be a Bug in SUSE... Activating "IPv4 Forwarding" during Installation leads to a 70-yast.conf with disabled forwarding options. And as a plus, "sysctl -l" displays wrong settings, so as YAST does...
The solution is to play around in YASTs network Settings, so that 70-yast.conf gets rewritten with activated ip_forwarding options.