Kubernetes pods have no outbound internet [closed] - kubernetes

Question

Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed last month.
Improve this question
I currently have a cluster with 2 nodes:
node1 - 192.168.1.20 (master, worker)
node2 - 192.168.1.30 (worker)
When running a pod on either of the nodes there is no outbound internet connection.
The problem seems to not be related to DNS since even pinging a public IP address of google.com does not work.
pod1-node1$ traceroute to google.com (216.58.215.46), 30 hops max, 46 byte packets
1 192-168-1-20.kubernetes.default.svc.home-cloud.local (192.168.1.20) 0.008 ms 0.009 ms 0.009 ms
2 *
Nodes have internet access and everything works there
node1$ ping 216.58.215.46 -> OK
node2$ ping 216.58.215.46 -> OK
Ip tables on master node:
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-N DOCKER-INGRESS
-N KUBE-FIREWALL
-N KUBE-KUBELET-CANARY
-N KUBE-LOAD-BALANCER
-N KUBE-MARK-DROP
-N KUBE-MARK-MASQ
-N KUBE-NODE-PORT
-N KUBE-POSTROUTING
-N KUBE-SERVICES
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -o docker_gwbridge -m addrtype --src-type LOCAL -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 9000 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p udp -m udp --dport 34197 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p tcp -m tcp --dport 27015 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i docker_gwbridge -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 9000 -j DNAT --to-destination 172.17.0.2:9000
-A DOCKER ! -i docker0 -p udp -m udp --dport 34197 -j DNAT --to-destination 172.17.0.3:34197
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 27015 -j DNAT --to-destination 172.17.0.3:27015
-A DOCKER-INGRESS -p tcp -m tcp --dport 3306 -j DNAT --to-destination 172.18.0.2:3306
-A DOCKER-INGRESS -p tcp -m tcp --dport 9980 -j DNAT --to-destination 172.18.0.2:9980
-A DOCKER-INGRESS -j RETURN
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SERVICES -m comment --comment "Kubernetes service lb portal" -m set --match-set KUBE-LOAD-BALANCER dst,dst -j KUBE-LOAD-BALANCER
-A KUBE-SERVICES ! -s 10.233.64.0/18 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
-A KUBE-SERVICES -m set --match-set KUBE-LOAD-BALANCER dst,dst -j ACCEPT
From within a pod I get a strange looking ip route result. My docker overlay network is '10.233.64.0/18' configured with calico.
pod1-node1$ ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
I also have metallb installed and it installs a kube-proxy but not sure how exactly it works or if it can be related to the problem.
Any advice is greatly appreciated.
Cheers Alex
EDIT: Cluster is bare metal, installed with kubespray, I haven't installed any CNI manually

Answer 1

It turned out to be a bug in the CNI - Calico.
I manually set the calico version in my kubespray to 3.22.1 and the problem was solved.
I then encountered other connectivity issues between my nodes and opted to use flannel.
Since then everything works fine.
My advise to others after spending a week of research myself :
Container Networking is HARD
Take it slow and do not cut corners ! If you are missing some
knowledge about a topic - stop and learn at least the basics of the
said topic before continuing.
Diagnose your problem - most commonly it is DNS or Packet Routing:
Get the IP of google.com on another machine and ping the IP from within a pod - if it works you have a DNS problem if it doesn't then probably CNI related (packet routing).
Learn how resolved works and how it is configured
This will allow you to check what DNS is actually used on your node and within your pod.
Learn how kubernetes DNS works and why it is needed:
It allows pod to service communication to work among other things. It also needs upstream DNS addresses so that it can resolve things outside your cluster.
Learn how IP tables work and how masquerading works
It allows you to understand how network packets are routed between your node and your pods, using the bridge interface.
Learn what a bridge interface is
The docker0 bridge interface allows all your containers (and pods by extension) to talk to outside world.
Read this very informational article on how calico interacts with your IP tables to create nat masquerading rules on the fly:
https://medium.com/#bikramgupta/pod-network-troubleshooting-while-using-calico-on-kubernetes-ee78b731d4d8