Kubernetes kube-proxy iptables rules seem to be redundant - kubernetes

Context
Maybe there is unnecessary redundancy in the iptables rules generated by kubeadm init for kube-proxy:
iptables -t filter -S
output:
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N KUBE-EXTERNAL-SERVICES
-N KUBE-FIREWALL
-N KUBE-FORWARD
-N KUBE-SERVICES
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
The 10.244.0.0/16 range corresponds to the pod overlay network.
Lets focus on the FORWARD chain.
-P FORWARD DROP
-N KUBE-FORWARD
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
Question:
Why the KUBE-FORWARD accepts packages within the overlay network, when their connection state is RELATED or ESTABLISHED if FORWARD chain will accept all packet traffic within the overlay network regardless their connection state?
Note:
The kubernetes cluster works fine.

This duplication is there, because the default FORWARD policy can be disabled for some reason, and Kubernetes still wants to forward packets that either:
Are marked with "masqueradeMark" (those can start new connections)
Are part of already established connection
You can try reading the comments in the k8s source: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go#L1325
In general, one should expect some duplication in iptables rules when those rules are automatically managed. It makes it easier to code the automation.

Related

Kubernetes pods have no outbound internet [closed]

Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed last month.
Improve this question
I currently have a cluster with 2 nodes:
node1 - 192.168.1.20 (master, worker)
node2 - 192.168.1.30 (worker)
When running a pod on either of the nodes there is no outbound internet connection.
The problem seems to not be related to DNS since even pinging a public IP address of google.com does not work.
pod1-node1$ traceroute to google.com (216.58.215.46), 30 hops max, 46 byte packets
1 192-168-1-20.kubernetes.default.svc.home-cloud.local (192.168.1.20) 0.008 ms 0.009 ms 0.009 ms
2 *
Nodes have internet access and everything works there
node1$ ping 216.58.215.46 -> OK
node2$ ping 216.58.215.46 -> OK
Ip tables on master node:
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-N DOCKER-INGRESS
-N KUBE-FIREWALL
-N KUBE-KUBELET-CANARY
-N KUBE-LOAD-BALANCER
-N KUBE-MARK-DROP
-N KUBE-MARK-MASQ
-N KUBE-NODE-PORT
-N KUBE-POSTROUTING
-N KUBE-SERVICES
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -o docker_gwbridge -m addrtype --src-type LOCAL -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 9000 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p udp -m udp --dport 34197 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p tcp -m tcp --dport 27015 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i docker_gwbridge -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 9000 -j DNAT --to-destination 172.17.0.2:9000
-A DOCKER ! -i docker0 -p udp -m udp --dport 34197 -j DNAT --to-destination 172.17.0.3:34197
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 27015 -j DNAT --to-destination 172.17.0.3:27015
-A DOCKER-INGRESS -p tcp -m tcp --dport 3306 -j DNAT --to-destination 172.18.0.2:3306
-A DOCKER-INGRESS -p tcp -m tcp --dport 9980 -j DNAT --to-destination 172.18.0.2:9980
-A DOCKER-INGRESS -j RETURN
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SERVICES -m comment --comment "Kubernetes service lb portal" -m set --match-set KUBE-LOAD-BALANCER dst,dst -j KUBE-LOAD-BALANCER
-A KUBE-SERVICES ! -s 10.233.64.0/18 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
-A KUBE-SERVICES -m set --match-set KUBE-LOAD-BALANCER dst,dst -j ACCEPT
From within a pod I get a strange looking ip route result. My docker overlay network is '10.233.64.0/18' configured with calico.
pod1-node1$ ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
I also have metallb installed and it installs a kube-proxy but not sure how exactly it works or if it can be related to the problem.
Any advice is greatly appreciated.
Cheers Alex
EDIT: Cluster is bare metal, installed with kubespray, I haven't installed any CNI manually
It turned out to be a bug in the CNI - Calico.
I manually set the calico version in my kubespray to 3.22.1 and the problem was solved.
I then encountered other connectivity issues between my nodes and opted to use flannel.
Since then everything works fine.
My advise to others after spending a week of research myself :
Container Networking is HARD
Take it slow and do not cut corners ! If you are missing some
knowledge about a topic - stop and learn at least the basics of the
said topic before continuing.
Diagnose your problem - most commonly it is DNS or Packet Routing:
Get the IP of google.com on another machine and ping the IP from within a pod - if it works you have a DNS problem if it doesn't then probably CNI related (packet routing).
Learn how resolved works and how it is configured
This will allow you to check what DNS is actually used on your node and within your pod.
Learn how kubernetes DNS works and why it is needed:
It allows pod to service communication to work among other things. It also needs upstream DNS addresses so that it can resolve things outside your cluster.
Learn how IP tables work and how masquerading works
It allows you to understand how network packets are routed between your node and your pods, using the bridge interface.
Learn what a bridge interface is
The docker0 bridge interface allows all your containers (and pods by extension) to talk to outside world.
Read this very informational article on how calico interacts with your IP tables to create nat masquerading rules on the fly:
https://medium.com/#bikramgupta/pod-network-troubleshooting-while-using-calico-on-kubernetes-ee78b731d4d8

How to delete iptables rules added by kube-proxy?

I want to manually delete iptables rules for debugging.
I have several rules created by kube-proxy based on service nettools:
# kubectl get endpoints nettools
NAME ENDPOINTS AGE
nettools 172.16.27.138:7493 1h
And its iptables rules:
# iptables-save|grep nettools
-A KUBE-SEP-6DFMUWHMXOYMFWKG -s 172.16.27.138/32 -m comment --comment "default/nettools:web" -j KUBE-MARK-MASQ
-A KUBE-SEP-6DFMUWHMXOYMFWKG -p tcp -m comment --comment "default/nettools:web" -m tcp -j DNAT --to-destination 172.16.27.138:7493
-A KUBE-SERVICES -d 10.0.1.2/32 -p tcp -m comment --comment "default/nettools:web cluster IP" -m tcp --dport 7493 -j KUBE-SVC-INDS3KD6I5PFKUWF
-A KUBE-SVC-INDS3KD6I5PFKUWF -m comment --comment "default/nettools:web" -j KUBE-SEP-6DFMUWHMXOYMFWKG
However,I cannot delete those rules:
# iptables -D KUBE-SVC-INDS3KD6I5PFKUWF -m comment --comment "default/nettools:web" -j KUBE-SEP-6DFMUWHMXOYMFWKG
iptables v1.4.21: Couldn't load target `KUBE-SEP-6DFMUWHMXOYMFWKG':No such file or directory
# iptables -D KUBE-SERVICES -d 10.0.1.2/32 -p tcp -m comment --comment "default/nettools:web cluster IP" -m tcp --dport 7493 -j KUBE-SVC-INDS3KD6I5PFKUWF
iptables v1.4.21: Couldn't load target `KUBE-SVC-INDS3KD6I5PFKUWF':No such file or directory
There are multiple tables in play when dealing with iptables. filter table is the default if nothing is specified. The rules that you are trying to delete are part of the nat table.
Just add -t nat to your rules to delete those rules.
Example:
# iptables -t nat -D KUBE-SVC-INDS3KD6I5PFKUWF -m comment --comment "default/nettools:web" -j KUBE-SEP-6DFMUWHMXOYMFWKG

when and where a service ip created for kubernetes static pods

I'm using kubernetes 1.6.7 cluster, 3 nodes as master (node1, node2 and node3).
The apiserver cluster ip as bellow:
root#node1:/etc/kubernetes/manifests# kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.233.0.1 <none> 443/TCP 14d
root#node1:/etc/kubernetes/manifests# kubectl get endpoints
kubernetes 172.16.10.11:6443,172.16.10.12:6443,172.16.10.13:6443 14d
The problem is when node2 was down, tiller pod can't access 10.233.0.1:443, logs as bellow:
[storage] 2018/08/28 06:46:34 listing all releases with filter
[storage/driver] 2018/08/28 06:46:34 list: failed to list: Get https://10.233.0.1:443/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%3DTILLER: dial tcp 10.233.0.1:443: connect: no route to host
and the service kubernetes still has 3 endpoints, but i think the 172.16.10.12 should be delete from the endpoint list. iptables as bellow:
-A KUBE-SVC-LRLUS54FOYJDJ5GT -m comment --comment "default/wangxj35-nginx:nginx" -j KUBE-SEP-GGIOR777CPKVSIB7
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-CS4Z4M5BFAAWBFXQ --mask 255.255.255.255 --rsource -j KUBE-SEP-CS4Z4M5BFAAWBFXQ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-MQZA2ZXHX6GRRY22 --mask 255.255.255.255 --rsource -j KUBE-SEP-MQZA2ZXHX6GRRY22
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-ZX77NMWHNJWLXYWF --mask 255.255.255.255 --rsource -j KUBE-SEP-ZX77NMWHNJWLXYWF
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-CS4Z4M5BFAAWBFXQ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-MQZA2ZXHX6GRRY22
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-ZX77NMWHNJWLXYWF
-A KUBE-SEP-CS4Z4M5BFAAWBFXQ -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-CS4Z4M5BFAAWBFXQ --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.10.11:6443
-A KUBE-SEP-MQZA2ZXHX6GRRY22 -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-MQZA2ZXHX6GRRY22 --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.10.12:6443
-A KUBE-SEP-ZX77NMWHNJWLXYWF -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-ZX77NMWHNJWLXYWF --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.10.13:6443
Could anyone help me where the static pods service created by kubernetes source code?
Thanks.
Controller.go creates the kubernetes service:
// UpdateKubernetesService attempts to update the default Kube service.
func (c *Controller) UpdateKubernetesService(reconcile bool)> error {
// Update service & endpoint records.
Also, to work around this problem, kubernetes1.9.0 has added --endpoint-reconciler-type=lease to allow apiserver to track the alive apiserver, see issue51698, while for release below 1.9.0, you can unset the --apiserver-count flag to let alive apiserver to preempt the only one endpoint position.

how to avoid snat when using service type clusterip on kubernetes?

For every service in k8s cluster, kubernetes do snat for request packets. The iptables rules are:
-A KUBE-SERVICES ! -s 10.254.0.0/16 -d 10.254.186.26/32 -p tcp -m comment --comment "policy-demo/nginx: cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.254.186.26/32 -p tcp -m comment --comment "policy-demo/nginx: cluster IP" -m tcp --dport 80 -j KUBE-SVC-3VXIGVIYYFN7DHDA
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
It works well in most circumstances, but not networkpolicy. Caclico uses ipset to implement networkpolicy and the matched set only contains pod ip.
So when the service pod runs on node1, and access pod runs on node2. The networkpolicy will DROP the request because the src ip of the request is node2's ip or flannel.1 ip.
I think there might be a method to close snat for clusterip service.But I can't find it anywhere, could anyone help me?
Thank you very much!
The problem has been resolved.
I changed --cluster-cidr=10.254.0.0/16 for kube-proxy to --cluster-cidr=172.30.0.0/16. And then it worked well.
The kube-proxy cluster-cidr needs to match the one used on the controller manager, also the one used by calico.

kubernetes: cannot access NodePort from other machines

I am trying to install kubernetes.
There was a problem when checking the container executed on kubernetes.
I set the type of service to Nodeport, but I could not access from a node other than the one on which the container is running.
Want to make it accessible from other computers, please tell me where it is different.
I tried externalIPs and LoadBarancer, but it was impossible.
Enviroment
OS:Ubuntu 16.04 LTS
Kubernetes:1.8
Docker:17.09.0-ce
etcd:3.2.8
flannel:0.9.0
Network
Physical:10.1.1.0/24
flannel:172.16.0.0/16
docker:192.168.0.0/16
Machines
Master Node(2nodes):10.1.1.24,10.1.1.25
Worker Node(2nodes):10.1.1.26,10.1.1.27
kubectl describe svc nginx-cluster
Name: nginx-cluster
Namespace: default
Labels: app=nginx-demo
Annotations: <none>
Selector: app=nginx-demo
Type: ClusterIP
IP: 172.16.236.159
Port: <unset> 8090/TCP
TargetPort: 80/TCP
Endpoints: 192.168.24.2:80
Session Affinity: None
Events: <none>
kubectl describe svc nginx-service
Name: nginx-service
Namespace: default
Labels: app=nginx-demo
Annotations: <none>
Selector: app=nginx-demo
Type: NodePort
IP: 172.16.199.69
Port: <unset> 8090/TCP
TargetPort: 80/TCP
NodePort: <unset> 31659/TCP
Endpoints: 192.168.24.2:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
running container worker node(10.1.1.27)
curl 10.1.1.27:31659
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
worker node(10.1.1.26)
curl 10.1.1.27:31659
curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.
other machine(10.1.1.XX)
curl 10.1.1.27:31659
curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
echoserver-848b75d85-9fx7r 1/1 Running 3 6d 192.168.70.2 k8swrksv01
nginx-demo-85cc49574c-wv2b9 1/1 Running 3 6d 192.168.2.2 k8swrksv02
kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
clusterip ClusterIP 172.16.39.77 <none> 80/TCP 6d run=echoserver
kubernetes ClusterIP 172.16.0.1 <none> 443/TCP 10d <none>
nginx-cluster ClusterIP 172.16.236.159 <none> 8090/TCP 6d app=nginx-demo
nginx-service NodePort 172.16.199.69 <none> 8090:31659/TCP 6d app=nginx-demo
nodeport NodePort 172.16.38.40 <none> 80:31317/TCP 6d run=echoserver
netstat -ntlp
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 1963/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 2202/kube-proxy
tcp 0 0 127.0.0.1:4243 0.0.0.0:* LISTEN 1758/dockerd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 996/sshd
tcp6 0 0 :::4194 :::* LISTEN 1963/kubelet
tcp6 0 0 :::10250 :::* LISTEN 1963/kubelet
tcp6 0 0 :::31659 :::* LISTEN 2202/kube-proxy
tcp6 0 0 :::10255 :::* LISTEN 1963/kubelet
tcp6 0 0 :::10256 :::* LISTEN 2202/kube-proxy
tcp6 0 0 :::31317 :::* LISTEN 2202/kube-proxy
tcp6 0 0 :::22 :::* LISTEN 996/sshd
iptables-save
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-AZ4EGFEAU4RTSLJO - [0:0]
:KUBE-SEP-C7HQKKO26GIFOZZM - [0:0]
:KUBE-SEP-EWKNS2YCPXGJCXDC - [0:0]
:KUBE-SEP-LQVPUPFGW6BWATIP - [0:0]
:KUBE-SEP-OMMOFZ27GPKZ4OPA - [0:0]
:KUBE-SEP-UD3HOGDD5NDLNY74 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-CQNAS6RSUGJF2C2D - [0:0]
:KUBE-SVC-GKN7Y2BSGW4NJTYL - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-XP7QDA4CRQ2QA33W - [0:0]
:KUBE-SVC-Z5P6OMNAEVLAQUTS - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 192.168.2.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 192.168.0.0/16 -d 192.168.0.0/16 -j RETURN
-A POSTROUTING -s 192.168.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.2.0/24 -j RETURN
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.0.0/16 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -s 192.168.70.2/32 -m comment --comment "default/clusterip:" -j KUBE-MARK-MASQ
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -p tcp -m comment --comment "default/clusterip:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-C7HQKKO26GIFOZZM -s 192.168.70.2/32 -m comment --comment "default/nodeport:" -j KUBE-MARK-MASQ
-A KUBE-SEP-C7HQKKO26GIFOZZM -p tcp -m comment --comment "default/nodeport:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-EWKNS2YCPXGJCXDC -s 10.1.1.25/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-EWKNS2YCPXGJCXDC -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.25:6443
-A KUBE-SEP-LQVPUPFGW6BWATIP -s 192.168.2.2/32 -m comment --comment "default/nginx-service:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LQVPUPFGW6BWATIP -p tcp -m comment --comment "default/nginx-service:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -s 10.1.1.24/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.24:6443
-A KUBE-SEP-UD3HOGDD5NDLNY74 -s 192.168.2.2/32 -m comment --comment "default/nginx-cluster:" -j KUBE-MARK-MASQ
-A KUBE-SEP-UD3HOGDD5NDLNY74 -p tcp -m comment --comment "default/nginx-cluster:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SERVICES -d 172.16.236.159/32 -p tcp -m comment --comment "default/nginx-cluster: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-Z5P6OMNAEVLAQUTS
-A KUBE-SERVICES -d 172.16.199.69/32 -p tcp -m comment --comment "default/nginx-service: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-SERVICES -d 172.16.38.40/32 -p tcp -m comment --comment "default/nodeport: cluster IP" -m tcp --dport 80 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-SERVICES -d 172.16.39.77/32 -p tcp -m comment --comment "default/clusterip: cluster IP" -m tcp --dport 80 -j KUBE-SVC-CQNAS6RSUGJF2C2D
-A KUBE-SERVICES -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-CQNAS6RSUGJF2C2D -m comment --comment "default/clusterip:" -j KUBE-SEP-AZ4EGFEAU4RTSLJO
-A KUBE-SVC-GKN7Y2BSGW4NJTYL -m comment --comment "default/nginx-service:" -j KUBE-SEP-LQVPUPFGW6BWATIP
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-XP7QDA4CRQ2QA33W -m comment --comment "default/nodeport:" -j KUBE-SEP-C7HQKKO26GIFOZZM
-A KUBE-SVC-Z5P6OMNAEVLAQUTS -m comment --comment "default/nginx-cluster:" -j KUBE-SEP-UD3HOGDD5NDLNY74
COMMIT
*filter
:INPUT ACCEPT [40:14606]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [42:6275]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:DOCKER-USER - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A DOCKER-USER -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT
It's caused by default DROP on FORWARD chain (which in turn is caused by docker).
You can see it working again if you add iptables -A FORWARD -j ACCEPT rule to your node.
The k8s issue is here: https://github.com/kubernetes/kubernetes/issues/39823 but the actual fix is here https://github.com/kubernetes/kubernetes/pull/52569 (expected to be in 1.9).
additional information.
Something is supposed to block the port, but that is unknown...
pod running node
nmap 10.1.1.27 -p31000-32000
Not shown: 999 closed ports
PORT STATE SERVICE
31317/tcp open unknown
31659/tcp open unknown
other node
nmap 10.1.1.27 -p31000-32000
Not shown: 999 closed ports
PORT STATE SERVICE
31317/tcp filtered unknown
31659/tcp filtered unknown
Farcaller is right. We hit the same on centos running firewalld.
Until we upgrade to k8s 1.9 we added the following firewalld rule. The rule is similar to the one created by kube-proxy in k8s 1.9
#!/bin/bash
# follows https://github.com/kubernetes/kubernetes/pull/52569 introduced in k8s 1.9
# required to support nodeport services routing from all nodes in the cluster when the firewall is turned on.
# KUBE-MARK-MASQ corresponds to kube-proxy --iptables-masquerade-bit=14, which is the default.
KUBE_MARK_MASQ="0x4000/0x4000"
firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 3 -m mark --mark "$KUBE_MARK_MASQ" -j ACCEPT
you can first check the node port whether the port is open,
use
netstat -ntlp
to check, if yes , that maybe some problem with the iptable or routing,
if no , please check the firewall or other question
Good luck