I am trying to resolve a Cluster internal DNS from a Node in the Cluster.
Example: dig #10.96.0.10 kubernetes.local (10.96.0.10 being the Service IP of the DNS)
I am expecting to get the IP of the Service (10.96.0.1 in this case), however the Connection times out.
This Problem only happens if I try connecting from a Host in the Cluster to a Service via UDP, while the Pods of the Service are not hosted on the Node I am connecting from.
If I try to connect from a Pod running on the same Node, it works as expected.
If I try to connect to the Pods directly instead of the Service, it works as expected.
If I try to connect to the Service via TCP instead of UDP, it works as expected.
If I try to connect to the Service when the Pods are running on the same Node as I am connecting from, it works as expected.
I am running Kubernetes v1.17 (Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}) with the flannel pod network, running on Debian Buster.
So far I have looked at the iptables rules, the service and pod rules seem correct.
The relevant sections of iptables-save:
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-SBQ7D3CPOXKXY6NJ
-A KUBE-SVC-TCOU7JCQXEZGVUNU -j KUBE-SEP-CDWMPIYNA34YYC2O
-A KUBE-SEP-CDWMPIYNA34YYC2O -s 10.244.1.218/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-CDWMPIYNA34YYC2O -p udp -m udp -j DNAT --to-destination 10.244.1.218:53
-A KUBE-SEP-SBQ7D3CPOXKXY6NJ -s 10.244.1.217/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-SBQ7D3CPOXKXY6NJ -p udp -m udp -j DNAT --to-destination 10.244.1.217:53
I also used the sudo tcpdump -i flannel.1 udp command on sender and receiver and found out, that the packages get send, but not received.
When I address the Pods directly, for example via dig #10.244.1.218 kubernetes.local, the package gets send and received properly.
Related
Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed last month.
Improve this question
I currently have a cluster with 2 nodes:
node1 - 192.168.1.20 (master, worker)
node2 - 192.168.1.30 (worker)
When running a pod on either of the nodes there is no outbound internet connection.
The problem seems to not be related to DNS since even pinging a public IP address of google.com does not work.
pod1-node1$ traceroute to google.com (216.58.215.46), 30 hops max, 46 byte packets
1 192-168-1-20.kubernetes.default.svc.home-cloud.local (192.168.1.20) 0.008 ms 0.009 ms 0.009 ms
2 *
Nodes have internet access and everything works there
node1$ ping 216.58.215.46 -> OK
node2$ ping 216.58.215.46 -> OK
Ip tables on master node:
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-N DOCKER-INGRESS
-N KUBE-FIREWALL
-N KUBE-KUBELET-CANARY
-N KUBE-LOAD-BALANCER
-N KUBE-MARK-DROP
-N KUBE-MARK-MASQ
-N KUBE-NODE-PORT
-N KUBE-POSTROUTING
-N KUBE-SERVICES
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -o docker_gwbridge -m addrtype --src-type LOCAL -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 9000 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p udp -m udp --dport 34197 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p tcp -m tcp --dport 27015 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i docker_gwbridge -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 9000 -j DNAT --to-destination 172.17.0.2:9000
-A DOCKER ! -i docker0 -p udp -m udp --dport 34197 -j DNAT --to-destination 172.17.0.3:34197
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 27015 -j DNAT --to-destination 172.17.0.3:27015
-A DOCKER-INGRESS -p tcp -m tcp --dport 3306 -j DNAT --to-destination 172.18.0.2:3306
-A DOCKER-INGRESS -p tcp -m tcp --dport 9980 -j DNAT --to-destination 172.18.0.2:9980
-A DOCKER-INGRESS -j RETURN
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SERVICES -m comment --comment "Kubernetes service lb portal" -m set --match-set KUBE-LOAD-BALANCER dst,dst -j KUBE-LOAD-BALANCER
-A KUBE-SERVICES ! -s 10.233.64.0/18 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
-A KUBE-SERVICES -m set --match-set KUBE-LOAD-BALANCER dst,dst -j ACCEPT
From within a pod I get a strange looking ip route result. My docker overlay network is '10.233.64.0/18' configured with calico.
pod1-node1$ ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
I also have metallb installed and it installs a kube-proxy but not sure how exactly it works or if it can be related to the problem.
Any advice is greatly appreciated.
Cheers Alex
EDIT: Cluster is bare metal, installed with kubespray, I haven't installed any CNI manually
It turned out to be a bug in the CNI - Calico.
I manually set the calico version in my kubespray to 3.22.1 and the problem was solved.
I then encountered other connectivity issues between my nodes and opted to use flannel.
Since then everything works fine.
My advise to others after spending a week of research myself :
Container Networking is HARD
Take it slow and do not cut corners ! If you are missing some
knowledge about a topic - stop and learn at least the basics of the
said topic before continuing.
Diagnose your problem - most commonly it is DNS or Packet Routing:
Get the IP of google.com on another machine and ping the IP from within a pod - if it works you have a DNS problem if it doesn't then probably CNI related (packet routing).
Learn how resolved works and how it is configured
This will allow you to check what DNS is actually used on your node and within your pod.
Learn how kubernetes DNS works and why it is needed:
It allows pod to service communication to work among other things. It also needs upstream DNS addresses so that it can resolve things outside your cluster.
Learn how IP tables work and how masquerading works
It allows you to understand how network packets are routed between your node and your pods, using the bridge interface.
Learn what a bridge interface is
The docker0 bridge interface allows all your containers (and pods by extension) to talk to outside world.
Read this very informational article on how calico interacts with your IP tables to create nat masquerading rules on the fly:
https://medium.com/#bikramgupta/pod-network-troubleshooting-while-using-calico-on-kubernetes-ee78b731d4d8
I have a freshly installed pi4, and after running curl -L https://install.pivpn.io | bash to install pivpn (wireguard), the clients connect but have no internet access.
From my Pi I can ping the clients. Using tcpdump I can also see the clients are connected, but still no internet access
My Iptables is set like this:
#iptables -S
-P INPUT DROP
-P FORWARD DROP
-P OUTPUT ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 51820 -m comment --comment wireguard-input-rule -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -m comment --comment ssh -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 443 -m comment --comment ssh -j ACCEPT
-A FORWARD -d 10.6.0.0/24 -i eth0 -o wg0 -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment wireguard-forward-rule -j ACCEPT
-A FORWARD -s 10.6.0.0/24 -i wg0 -o eth0 -m comment --comment wireguard-forward-rule -j ACCEPT
in my /etc/sysctl.conf the net.ipv4.ip_forward=1
and the rest of the settings of ipvpn are just standard.
In the wireguard settings I just use google DNS: 8.8.8.8
I am running out of ideas.
Can anyone please help ?
Sounds like the only thing you're missing is IP Forwarding:
You can enable it on the fly with: sysctl -w net.ipv4.ip_forward=1
And you can make it permanent by editing /etc/sysctl.conf and setting
net.ipv4.ip_forward = 1
Sorry, I'm a noob in iptables.
I have a VPN app which binds on local port 1080, while it goes to destination port 1194 (openvpn). The app does not support privileged port binding (which needs root, of which I have). I want the app to bind on local port 25. I have browsed Google and the answer seems to be iptables. I have seen many posts, of which many say the SNAT target is the one I should use.
I have tried this code:
iptables -I POSTROUTING -o wlan0 -t nat -p tcp --destination 195.123.216.159 -m tcp --dport 1194 -j SNAT --to-source 192.168.43.239:25
And these:
iptables -I FORWARD -p tcp -d 192.168.43.239 -m tcp --dport 25 -j ACCEPT
iptables -I FORWARD -p tcp -s 192.168.43.239 -m tcp --sport 25 -j ACCEPT
iptables -I OUTPUT -o wlan0 -p tcp -m tcp --sport 25 -j ACCEPT
iptables -I INPUT -i wlan0 -p tcp -m tcp --dport 25 -j ACCEPT
What I want is to make the output to be something like this when I run the netstat command:
tcp 0 0 192.168.43.239:25 195.123.216.159:1194 ESTABLISHED
But instead, after running all the codes, the output to netstat becomes this:
tcp 0 0 192.168.43.239:1080 195.123.216.159:5000 ESTABLISHED
Is it impossible to change binding port using iptables? Please help me to understand the concepts of networking.
Turns out iptables was just doing its job correctly. Translated packets turn out to not be tracked by netstat. I was lost and completely didnt understand that iptables doesnt alter ip v6 traffic of which the app was using. And the forward rules where not necessary since the chain policy was to accept the packets.
For every service in k8s cluster, kubernetes do snat for request packets. The iptables rules are:
-A KUBE-SERVICES ! -s 10.254.0.0/16 -d 10.254.186.26/32 -p tcp -m comment --comment "policy-demo/nginx: cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.254.186.26/32 -p tcp -m comment --comment "policy-demo/nginx: cluster IP" -m tcp --dport 80 -j KUBE-SVC-3VXIGVIYYFN7DHDA
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
It works well in most circumstances, but not networkpolicy. Caclico uses ipset to implement networkpolicy and the matched set only contains pod ip.
So when the service pod runs on node1, and access pod runs on node2. The networkpolicy will DROP the request because the src ip of the request is node2's ip or flannel.1 ip.
I think there might be a method to close snat for clusterip service.But I can't find it anywhere, could anyone help me?
Thank you very much!
The problem has been resolved.
I changed --cluster-cidr=10.254.0.0/16 for kube-proxy to --cluster-cidr=172.30.0.0/16. And then it worked well.
The kube-proxy cluster-cidr needs to match the one used on the controller manager, also the one used by calico.
So, I was experimenting and added a Service with an ExternalIP set to the end hosts ipv6 address.
This is a configuration that's accepted, but not supported, in Kubernetes, and causes the kube-proxy to do fail.
From the kube-proxy logs:
-A KUBE-SERVICES -m comment --comment "default/frontend: external IP" -m tcp -p tcp -d 207.154.225.168/32 --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m comment --comment "default/frontend: external IP" -m tcp -p tcp -d 207.154.225.168/32 --dport 80 -m physdev ! --physdev-is-in -m addrtype ! --src-type LOCAL -j KUBE-SVC-GYQQTB6TY565JPRW
-A KUBE-SERVICES -m comment --comment "default/frontend: external IP" -m tcp -p tcp -d 207.154.225.168/32 --dport 80 -m addrtype --dst-type LOCAL -j KUBE-SVC-GYQQTB6TY565JPRW
-A KUBE-SERVICES -m comment --comment "default/frontend: external IP" -m tcp -p tcp -d 2a03:b0c0:3:d0::43bb:4001/32 --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m comment --comment "default/frontend: external IP" -m tcp -p tcp -d 2a03:b0c0:3:d0::43bb:4001/32 --dport 80 -m physdev ! --physdev-is-in -m addrtype ! --src-type LOCAL -j KUBE-SVC-GYQQTB6TY565JPRW
E0502 07:38:39.913815 1 proxier.go:1312] Failed to execute iptables-restore: exit status 2 (iptables-restore v1.4.21: host/network 2a03:b0c0:3:d0::43bb:4001' not found
Error occurred at line: 53
Tryiptables-restore -h' or 'iptables-restore --help' for more information.
This is the problem:
There is no "default/frontend" service defined. I have explicitly deleted it,
kubectl get services --namespace=default
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.3.0.1 443/TCP 3d
However, kube-proxy on all my nodes still attempts to add these, which causes all services created after this misstep to break.
So, the question is, how do I purge this ghost of a service from my cluster?
One solution seems to be to reboot the API server/Kubernetes master. Not a pleasant one, but services work again at least.