kube-proxy doesn't create DNAT-rules on nodes for services registered on master.
everything works fine beside this service-ip-to-pod-ip NATing.
my setup:
kubernetes master: 10.98.99.176/24 (running: api-srv, scheduler, controller-manager)
kubernetes node1: 10.98.99.136/24 with CIDR 10.116.0.0/24 (running: kubelet, kube-proxy)
kubernetes node2: 10.98.99.137/24 with CIDR 10.116.1.0/24 (running: kubelet, kube-proxy)
CIDRs are configured on master via Node.spec.PodCIDR
cbr0-bridge is created on kubelet start.
routing is set up an works.
all fine.
i can ping the kube-dns-container running on node1 from a node2 container via its container ip (10.116.0.2) but not via its service ip (10.0.0.10).
i see this icmp-packets moving out on eth0 to default-gw (10.116.1.2 -> 10.0.0.10)
so when i have a look at iptables-save output:
# Generated by iptables-save v1.4.21 on Fri Oct 28 14:08:56 2016
*filter
:INPUT ACCEPT [40:11192]
:FORWARD ACCEPT [3:180]
:OUTPUT ACCEPT [77:31363]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -j KUBE-FIREWALL
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-SERVICES -d 10.0.0.10/32 -p udp -m comment --comment "default/kube-dns:dns has no endpoints" -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.0.0.10/32 -p tcp -m comment --comment "default/kube-dns:dns-tcp has no endpoints" -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Fri Oct 28 14:08:56 2016
# Generated by iptables-save v1.4.21 on Fri Oct 28 14:08:56 2016
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-SM34KKATJ2TS55C5 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-D376NYSDDVFPF2KN - [0:0]
:KUBE-SVC-N6R7PS4OMIK6NEO2 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING ! -d 10.0.0.0/8 -m addrtype ! --dst-type LOCAL -j MASQUERADE
-A POSTROUTING ! -d 10.0.0.0/8 -m addrtype ! --dst-type LOCAL -j MASQUERADE
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-SM34KKATJ2TS55C5 -s 10.98.99.176/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-SM34KKATJ2TS55C5 -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-SM34KKATJ2TS55C5 --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.98.99.176:6443
-A KUBE-SERVICES -d 10.0.0.10/32 -p udp -m comment --comment "default/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-D376NYSDDVFPF2KN
-A KUBE-SERVICES -d 10.0.0.10/32 -p tcp -m comment --comment "default/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-N6R7PS4OMIK6NEO2
-A KUBE-SERVICES -d 10.0.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 180 --reap --name KUBE-SEP-SM34KKATJ2TS55C5 --mask 255.255.255.255 --rsource -j KUBE-SEP-SM34KKATJ2TS55C5
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-SM34KKATJ2TS55C5
COMMIT
# Completed on Fri Oct 28 14:08:56 2016
there are DNAT-rules missing for something like 10.0.0.10 -> 10.116.0.2, isn't it?
here the kube-proxy-debug-log-output while starting:
systemd[1]: Stopping Kubernetes Kube-Proxy Server...
systemd[1]: Starting Kubernetes Kube-Proxy Server...
systemd[1]: Started Kubernetes Kube-Proxy Server.
kube-proxy[29691]: I1028 13:37:36.485645 29691 server.go:155] setting OOM scores is unsupported in this build
kube-proxy[29691]: I1028 13:37:36.487545 29691 server.go:202] Using iptables Proxier.
kube-proxy[29691]: I1028 13:37:36.491592 29691 server.go:214] Tearing down userspace rules.
kube-proxy[29691]: I1028 13:37:36.491628 29691 iptables.go:380] running iptables -C [OUTPUT -t nat -m comment --comment handle ClusterIPs; NOTE: this must be before the NodePort rules -j KUBE-PORTALS-HOST]
kube-proxy[29691]: I1028 13:37:36.492721 29691 iptables.go:380] running iptables -C [PREROUTING -t nat -m comment --comment handle ClusterIPs; NOTE: this must be before the NodePort rules -j KUBE-PORTALS-CONTAINER]
kube-proxy[29691]: I1028 13:37:36.493751 29691 iptables.go:380] running iptables -C [OUTPUT -t nat -m addrtype --dst-type LOCAL -m comment --comment handle service NodePorts; NOTE: this must be the last rule in the chain -j KUBE-NODEPORT-HOST]
kube-proxy[29691]: I1028 13:37:36.494708 29691 iptables.go:380] running iptables -C [PREROUTING -t nat -m addrtype --dst-type LOCAL -m comment --comment handle service NodePorts; NOTE: this must be the last rule in the chain -j KUBE-NODEPORT-CONTAINER]
kube-proxy[29691]: I1028 13:37:36.495659 29691 iptables.go:380] running iptables -C [INPUT -t filter -m comment --comment Ensure that non-local NodePort traffic can flow -j KUBE-NODEPORT-NON-LOCAL]
kube-proxy[29691]: I1028 13:37:36.496493 29691 iptables.go:380] running iptables -F [KUBE-PORTALS-CONTAINER -t nat]
kube-proxy[29691]: I1028 13:37:36.497316 29691 iptables.go:380] running iptables -F [KUBE-PORTALS-HOST -t nat]
kube-proxy[29691]: I1028 13:37:36.498291 29691 iptables.go:380] running iptables -F [KUBE-NODEPORT-HOST -t nat]
kube-proxy[29691]: I1028 13:37:36.499014 29691 iptables.go:380] running iptables -F [KUBE-NODEPORT-CONTAINER -t nat]
kube-proxy[29691]: I1028 13:37:36.500045 29691 iptables.go:380] running iptables -F [KUBE-NODEPORT-NON-LOCAL -t filter]
kube-proxy[29691]: I1028 13:37:36.500861 29691 reflector.go:202] Starting reflector *api.Service (15m0s) from pkg/proxy/config/api.go:30
kube-proxy[29691]: I1028 13:37:36.500897 29691 reflector.go:202] Starting reflector *api.Endpoints (15m0s) from pkg/proxy/config/api.go:33
kube-proxy[29691]: I1028 13:37:36.500985 29691 conntrack.go:40] Setting nf_conntrack_max to 131072
kube-proxy[29691]: I1028 13:37:36.501009 29691 reflector.go:253] Listing and watching *api.Service from pkg/proxy/config/api.go:30
kube-proxy[29691]: I1028 13:37:36.501233 29691 reflector.go:253] Listing and watching *api.Endpoints from pkg/proxy/config/api.go:33
kube-proxy[29691]: I1028 13:37:36.501372 29691 conntrack.go:57] Setting conntrack hashsize to 32768
kube-proxy[29691]: I1028 13:37:36.502921 29691 config.go:208] Calling handler.OnServiceUpdate()
kube-proxy[29691]: I1028 13:37:36.502938 29691 proxier.go:440] Adding new service "default/kube-dns:dns" at 10.0.0.10:53/UDP
kube-proxy[29691]: I1028 13:37:36.503016 29691 proxier.go:453] added serviceInfo(default/kube-dns:dns): (*iptables.serviceInfo)(0xc82049b9a0)({
kube-proxy[29691]: clusterIP: (net.IP) (len=16 cap=16) 10.0.0.10,
kube-proxy[29691]: port: (int) 53,
kube-proxy[29691]: protocol: (api.Protocol) (len=3) "UDP",
kube-proxy[29691]: nodePort: (int) 0,
kube-proxy[29691]: loadBalancerStatus: (api.LoadBalancerStatus) {
kube-proxy[29691]: Ingress: ([]api.LoadBalancerIngress) {
kube-proxy[29691]: }
kube-proxy[29691]: },
kube-proxy[29691]: sessionAffinityType: (api.ServiceAffinity) (len=4) "None",
kube-proxy[29691]: stickyMaxAgeSeconds: (int) 180,
kube-proxy[29691]: externalIPs: ([]string) <nil>,
kube-proxy[29691]: loadBalancerSourceRanges: ([]string) <nil>
kube-proxy[29691]: })
kube-proxy[29691]: I1028 13:37:36.503030 29691 proxier.go:440] Adding new service "default/kube-dns:dns-tcp" at 10.0.0.10:53/TCP
kube-proxy[29691]: I1028 13:37:36.503055 29691 proxier.go:453] added serviceInfo(default/kube-dns:dns-tcp): (*iptables.serviceInfo)(0xc82049ba40)({
kube-proxy[29691]: clusterIP: (net.IP) (len=16 cap=16) 10.0.0.10,
kube-proxy[29691]: port: (int) 53,
kube-proxy[29691]: protocol: (api.Protocol) (len=3) "TCP",
kube-proxy[29691]: nodePort: (int) 0,
kube-proxy[29691]: loadBalancerStatus: (api.LoadBalancerStatus) {
kube-proxy[29691]: Ingress: ([]api.LoadBalancerIngress) {
kube-proxy[29691]: }
kube-proxy[29691]: },
kube-proxy[29691]: sessionAffinityType: (api.ServiceAffinity) (len=4) "None",
kube-proxy[29691]: stickyMaxAgeSeconds: (int) 180,
kube-proxy[29691]: externalIPs: ([]string) <nil>,
kube-proxy[29691]: loadBalancerSourceRanges: ([]string) <nil>
kube-proxy[29691]: })
kube-proxy[29691]: I1028 13:37:36.503062 29691 proxier.go:440] Adding new service "default/kubernetes:https" at 10.0.0.1:443/TCP
kube-proxy[29691]: I1028 13:37:36.503087 29691 proxier.go:453] added serviceInfo(default/kubernetes:https): (*iptables.serviceInfo)(0xc82049bae0)({
kube-proxy[29691]: clusterIP: (net.IP) (len=16 cap=16) 10.0.0.1,
kube-proxy[29691]: port: (int) 443,
kube-proxy[29691]: protocol: (api.Protocol) (len=3) "TCP",
kube-proxy[29691]: nodePort: (int) 0,
kube-proxy[29691]: loadBalancerStatus: (api.LoadBalancerStatus) {
kube-proxy[29691]: Ingress: ([]api.LoadBalancerIngress) {
kube-proxy[29691]: }
kube-proxy[29691]: },
kube-proxy[29691]: sessionAffinityType: (api.ServiceAffinity) (len=8) "ClientIP",
kube-proxy[29691]: stickyMaxAgeSeconds: (int) 180,
kube-proxy[29691]: externalIPs: ([]string) <nil>,
kube-proxy[29691]: loadBalancerSourceRanges: ([]string) <nil>
kube-proxy[29691]: })
kube-proxy[29691]: I1028 13:37:36.503123 29691 proxier.go:674] Not syncing iptables until Services and Endpoints have been received from master
kube-proxy[29691]: I1028 13:37:36.503128 29691 proxier.go:670] syncProxyRules took 18.524µs
kube-proxy[29691]: I1028 13:37:36.503135 29691 proxier.go:400] OnServiceUpdate took 201.564µs for 2 services
kube-proxy[29691]: I1028 13:37:36.503508 29691 config.go:99] Calling handler.OnEndpointsUpdate()
kube-proxy[29691]: I1028 13:37:36.503534 29691 proxier.go:516] Setting endpoints for "default/kubernetes:https" to [10.98.99.176:6443]
kube-proxy[29691]: I1028 13:37:36.503566 29691 proxier.go:677] Syncing iptables rules
kube-proxy[29691]: I1028 13:37:36.503571 29691 iptables.go:380] running iptables -N [KUBE-SERVICES -t filter]
kube-proxy[29691]: I1028 13:37:36.504519 29691 iptables.go:380] running iptables -N [KUBE-SERVICES -t nat]
kube-proxy[29691]: I1028 13:37:36.505365 29691 iptables.go:380] running iptables -C [OUTPUT -t filter -m comment --comment kubernetes service portals -j KUBE-SERVICES]
kube-proxy[29691]: I1028 13:37:36.506177 29691 iptables.go:380] running iptables -C [OUTPUT -t nat -m comment --comment kubernetes service portals -j KUBE-SERVICES]
kube-proxy[29691]: I1028 13:37:36.506976 29691 iptables.go:380] running iptables -C [PREROUTING -t nat -m comment --comment kubernetes service portals -j KUBE-SERVICES]
kube-proxy[29691]: I1028 13:37:36.507794 29691 iptables.go:380] running iptables -N [KUBE-POSTROUTING -t nat]
kube-proxy[29691]: I1028 13:37:36.508626 29691 iptables.go:380] running iptables -C [POSTROUTING -t nat -m comment --comment kubernetes postrouting rules -j KUBE-POSTROUTING]
kube-proxy[29691]: I1028 13:37:36.509438 29691 iptables.go:299] running iptables-save [-t filter]
kube-proxy[29691]: I1028 13:37:36.510575 29691 iptables.go:299] running iptables-save [-t nat]
kube-proxy[29691]: I1028 13:37:36.511985 29691 proxier.go:1096] Restoring iptables rules: *filter
kube-proxy[29691]: :KUBE-SERVICES - [0:0]
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns has no endpoints" -m udp -p udp -d 10.0.0.10/32 --dport 53 -j REJECT
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns-tcp has no endpoints" -m tcp -p tcp -d 10.0.0.10/32 --dport 53 -j REJECT
kube-proxy[29691]: COMMIT
kube-proxy[29691]: *nat
kube-proxy[29691]: :KUBE-SERVICES - [0:0]
kube-proxy[29691]: :KUBE-NODEPORTS - [0:0]
kube-proxy[29691]: :KUBE-POSTROUTING - [0:0]
kube-proxy[29691]: :KUBE-MARK-MASQ - [0:0]
kube-proxy[29691]: :KUBE-SVC-D376NYSDDVFPF2KN - [0:0]
kube-proxy[29691]: :KUBE-SVC-N6R7PS4OMIK6NEO2 - [0:0]
kube-proxy[29691]: :KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
kube-proxy[29691]: :KUBE-SEP-SM34KKATJ2TS55C5 - [0:0]
kube-proxy[29691]: -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x00004000/0x00004000 -j MASQUERADE
kube-proxy[29691]: -A KUBE-MARK-MASQ -j MARK --set-xmark 0x00004000/0x00004000
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns cluster IP" -m udp -p udp -d 10.0.0.10/32 --dport 53 -j KUBE-SVC-D376NYSDDVFPF2KN
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns-tcp cluster IP" -m tcp -p tcp -d 10.0.0.10/32 --dport 53 -j KUBE-SVC-N6R7PS4OMIK6NEO2
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kubernetes:https cluster IP" -m tcp -p tcp -d 10.0.0.1/32 --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
kube-proxy[29691]: -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -m recent --name KUBE-SEP-SM34KKATJ2TS55C5 --rcheck --seconds 180 --reap -j KUBE-SEP-SM34KKATJ2TS55C5
kube-proxy[29691]: -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -j KUBE-SEP-SM34KKATJ2TS55C5
kube-proxy[29691]: -A KUBE-SEP-SM34KKATJ2TS55C5 -m comment --comment default/kubernetes:https -s 10.98.99.176/32 -j KUBE-MARK-MASQ
kube-proxy[29691]: -A KUBE-SEP-SM34KKATJ2TS55C5 -m comment --comment default/kubernetes:https -m recent --name KUBE-SEP-SM34KKATJ2TS55C5 --set -m tcp -p tcp -j DNAT --to-destination 10.98.99.176:6443
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
kube-proxy[29691]: COMMIT
kube-proxy[29691]: I1028 13:37:36.512060 29691 iptables.go:359] running iptables-restore [--noflush --counters /tmp/kube-temp-iptables-restore-332107562]
kube-proxy[29691]: I1028 13:37:36.517470 29691 conntrack.go:62] Setting nf_conntrack_tcp_timeout_established to 86400
kube-proxy[29691]: I1028 13:37:36.519918 29691 iptables.go:380] running iptables -C [POSTROUTING -t nat -m comment --comment kubernetes service traffic requiring SNAT -m mark --mark 0x4d415351 -j MASQUERADE]
kube-proxy[29691]: I1028 13:37:36.521108 29691 proxier.go:670] syncProxyRules took 17.541464ms
kube-proxy[29691]: I1028 13:37:36.521129 29691 proxier.go:478] OnEndpointsUpdate took 17.613002ms for 2 endpoints
kube-proxy[29691]: I1028 13:38:06.517827 29691 proxier.go:677] Syncing iptables rules
kube-proxy[29691]: I1028 13:38:06.517876 29691 iptables.go:380] running iptables -N [KUBE-SERVICES -t filter]
kube-proxy[29691]: I1028 13:38:06.519393 29691 iptables.go:380] running iptables -N [KUBE-SERVICES -t nat]
kube-proxy[29691]: I1028 13:38:06.520329 29691 iptables.go:380] running iptables -C [OUTPUT -t filter -m comment --comment kubernetes service portals -j KUBE-SERVICES]
kube-proxy[29691]: I1028 13:38:06.521251 29691 iptables.go:380] running iptables -C [OUTPUT -t nat -m comment --comment kubernetes service portals -j KUBE-SERVICES]
kube-proxy[29691]: I1028 13:38:06.522293 29691 iptables.go:380] running iptables -C [PREROUTING -t nat -m comment --comment kubernetes service portals -j KUBE-SERVICES]
kube-proxy[29691]: I1028 13:38:06.523397 29691 iptables.go:380] running iptables -N [KUBE-POSTROUTING -t nat]
kube-proxy[29691]: I1028 13:38:06.524257 29691 iptables.go:380] running iptables -C [POSTROUTING -t nat -m comment --comment kubernetes postrouting rules -j KUBE-POSTROUTING]
kube-proxy[29691]: I1028 13:38:06.525331 29691 iptables.go:299] running iptables-save [-t filter]
kube-proxy[29691]: I1028 13:38:06.526562 29691 iptables.go:299] running iptables-save [-t nat]
kube-proxy[29691]: I1028 13:38:06.528202 29691 proxier.go:1096] Restoring iptables rules: *filter
kube-proxy[29691]: :KUBE-SERVICES - [0:0]
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns has no endpoints" -m udp -p udp -d 10.0.0.10/32 --dport 53 -j REJECT
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns-tcp has no endpoints" -m tcp -p tcp -d 10.0.0.10/32 --dport 53 -j REJECT
kube-proxy[29691]: COMMIT
kube-proxy[29691]: *nat
kube-proxy[29691]: :KUBE-SERVICES - [0:0]
kube-proxy[29691]: :KUBE-NODEPORTS - [0:0]
kube-proxy[29691]: :KUBE-POSTROUTING - [0:0]
kube-proxy[29691]: :KUBE-MARK-MASQ - [0:0]
kube-proxy[29691]: :KUBE-SVC-D376NYSDDVFPF2KN - [0:0]
kube-proxy[29691]: :KUBE-SVC-N6R7PS4OMIK6NEO2 - [0:0]
kube-proxy[29691]: :KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
kube-proxy[29691]: :KUBE-SEP-SM34KKATJ2TS55C5 - [0:0]
kube-proxy[29691]: -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x00004000/0x00004000 -j MASQUERADE
kube-proxy[29691]: -A KUBE-MARK-MASQ -j MARK --set-xmark 0x00004000/0x00004000
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns cluster IP" -m udp -p udp -d 10.0.0.10/32 --dport 53 -j KUBE-SVC-D376NYSDDVFPF2KN
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kube-dns:dns-tcp cluster IP" -m tcp -p tcp -d 10.0.0.10/32 --dport 53 -j KUBE-SVC-N6R7PS4OMIK6NEO2
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "default/kubernetes:https cluster IP" -m tcp -p tcp -d 10.0.0.1/32 --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
kube-proxy[29691]: -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -m recent --name KUBE-SEP-SM34KKATJ2TS55C5 --rcheck --seconds 180 --reap -j KUBE-SEP-SM34KKATJ2TS55C5
kube-proxy[29691]: -A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment default/kubernetes:https -j KUBE-SEP-SM34KKATJ2TS55C5
kube-proxy[29691]: -A KUBE-SEP-SM34KKATJ2TS55C5 -m comment --comment default/kubernetes:https -s 10.98.99.176/32 -j KUBE-MARK-MASQ
kube-proxy[29691]: -A KUBE-SEP-SM34KKATJ2TS55C5 -m comment --comment default/kubernetes:https -m recent --name KUBE-SEP-SM34KKATJ2TS55C5 --set -m tcp -p tcp -j DNAT --to-destination 10.98.99.176:6443
kube-proxy[29691]: -A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
kube-proxy[29691]: COMMIT
kube-proxy[29691]: I1028 13:38:06.528286 29691 iptables.go:359] running iptables-restore [--noflush --counters /tmp/kube-temp-iptables-restore-616375937]
kube-proxy[29691]: I1028 13:38:06.530293 29691 iptables.go:380] running iptables -C [POSTROUTING -t nat -m comment --comment kubernetes service traffic requiring SNAT -m mark --mark 0x4d415351 -j MASQUERADE]
kube-proxy[29691]: I1028 13:38:06.532051 29691 proxier.go:670] syncProxyRules took 14.232833ms
the kube-proxy receives the services but no rules are generate for them (maybe because the api-server does not provide the pod-ips)
here are my additional systemd startup-flags:
docker: --bridge=cbr0 --iptables=false --ip-masq=false
api-server: --service-cluster-ip-range=10.0.0.0/24
kubelet: --configure-cbr0=true --cluster-dns=10.0.0.10
kube-proxy: --proxy-mode=iptables
SOLVED:
as i investigated in this problem i queried the api to get more information, there i found this:
http://localhost:8080/api/v1/namespaces/default/endpoints:
...
notReadyAddresses: [
{
ip: "10.116.0.2",
targetRef: {
kind: "Pod",
namespace: "default",
name: "kube-dns-v10-kdhaf",
uid: "83d266e7-9ceb-11e6-bf42-5254009edb97",
resourceVersion: "535855"
}
}
],
...
the problem was that the dns-pod didn't start healthy since i activated ServiceAccount at admissioncontrol with some error-msg like ...missing serviceaccount-cert ...
after fixing that it runs like a charm
Related
Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed last month.
Improve this question
I currently have a cluster with 2 nodes:
node1 - 192.168.1.20 (master, worker)
node2 - 192.168.1.30 (worker)
When running a pod on either of the nodes there is no outbound internet connection.
The problem seems to not be related to DNS since even pinging a public IP address of google.com does not work.
pod1-node1$ traceroute to google.com (216.58.215.46), 30 hops max, 46 byte packets
1 192-168-1-20.kubernetes.default.svc.home-cloud.local (192.168.1.20) 0.008 ms 0.009 ms 0.009 ms
2 *
Nodes have internet access and everything works there
node1$ ping 216.58.215.46 -> OK
node2$ ping 216.58.215.46 -> OK
Ip tables on master node:
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-N DOCKER-INGRESS
-N KUBE-FIREWALL
-N KUBE-KUBELET-CANARY
-N KUBE-LOAD-BALANCER
-N KUBE-MARK-DROP
-N KUBE-MARK-MASQ
-N KUBE-NODE-PORT
-N KUBE-POSTROUTING
-N KUBE-SERVICES
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -o docker_gwbridge -m addrtype --src-type LOCAL -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 9000 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p udp -m udp --dport 34197 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p tcp -m tcp --dport 27015 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i docker_gwbridge -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 9000 -j DNAT --to-destination 172.17.0.2:9000
-A DOCKER ! -i docker0 -p udp -m udp --dport 34197 -j DNAT --to-destination 172.17.0.3:34197
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 27015 -j DNAT --to-destination 172.17.0.3:27015
-A DOCKER-INGRESS -p tcp -m tcp --dport 3306 -j DNAT --to-destination 172.18.0.2:3306
-A DOCKER-INGRESS -p tcp -m tcp --dport 9980 -j DNAT --to-destination 172.18.0.2:9980
-A DOCKER-INGRESS -j RETURN
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SERVICES -m comment --comment "Kubernetes service lb portal" -m set --match-set KUBE-LOAD-BALANCER dst,dst -j KUBE-LOAD-BALANCER
-A KUBE-SERVICES ! -s 10.233.64.0/18 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
-A KUBE-SERVICES -m set --match-set KUBE-LOAD-BALANCER dst,dst -j ACCEPT
From within a pod I get a strange looking ip route result. My docker overlay network is '10.233.64.0/18' configured with calico.
pod1-node1$ ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
I also have metallb installed and it installs a kube-proxy but not sure how exactly it works or if it can be related to the problem.
Any advice is greatly appreciated.
Cheers Alex
EDIT: Cluster is bare metal, installed with kubespray, I haven't installed any CNI manually
It turned out to be a bug in the CNI - Calico.
I manually set the calico version in my kubespray to 3.22.1 and the problem was solved.
I then encountered other connectivity issues between my nodes and opted to use flannel.
Since then everything works fine.
My advise to others after spending a week of research myself :
Container Networking is HARD
Take it slow and do not cut corners ! If you are missing some
knowledge about a topic - stop and learn at least the basics of the
said topic before continuing.
Diagnose your problem - most commonly it is DNS or Packet Routing:
Get the IP of google.com on another machine and ping the IP from within a pod - if it works you have a DNS problem if it doesn't then probably CNI related (packet routing).
Learn how resolved works and how it is configured
This will allow you to check what DNS is actually used on your node and within your pod.
Learn how kubernetes DNS works and why it is needed:
It allows pod to service communication to work among other things. It also needs upstream DNS addresses so that it can resolve things outside your cluster.
Learn how IP tables work and how masquerading works
It allows you to understand how network packets are routed between your node and your pods, using the bridge interface.
Learn what a bridge interface is
The docker0 bridge interface allows all your containers (and pods by extension) to talk to outside world.
Read this very informational article on how calico interacts with your IP tables to create nat masquerading rules on the fly:
https://medium.com/#bikramgupta/pod-network-troubleshooting-while-using-calico-on-kubernetes-ee78b731d4d8
Context
Maybe there is unnecessary redundancy in the iptables rules generated by kubeadm init for kube-proxy:
iptables -t filter -S
output:
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N KUBE-EXTERNAL-SERVICES
-N KUBE-FIREWALL
-N KUBE-FORWARD
-N KUBE-SERVICES
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
The 10.244.0.0/16 range corresponds to the pod overlay network.
Lets focus on the FORWARD chain.
-P FORWARD DROP
-N KUBE-FORWARD
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
Question:
Why the KUBE-FORWARD accepts packages within the overlay network, when their connection state is RELATED or ESTABLISHED if FORWARD chain will accept all packet traffic within the overlay network regardless their connection state?
Note:
The kubernetes cluster works fine.
This duplication is there, because the default FORWARD policy can be disabled for some reason, and Kubernetes still wants to forward packets that either:
Are marked with "masqueradeMark" (those can start new connections)
Are part of already established connection
You can try reading the comments in the k8s source: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go#L1325
In general, one should expect some duplication in iptables rules when those rules are automatically managed. It makes it easier to code the automation.
I want to manually delete iptables rules for debugging.
I have several rules created by kube-proxy based on service nettools:
# kubectl get endpoints nettools
NAME ENDPOINTS AGE
nettools 172.16.27.138:7493 1h
And its iptables rules：
# iptables-save|grep nettools
-A KUBE-SEP-6DFMUWHMXOYMFWKG -s 172.16.27.138/32 -m comment --comment "default/nettools:web" -j KUBE-MARK-MASQ
-A KUBE-SEP-6DFMUWHMXOYMFWKG -p tcp -m comment --comment "default/nettools:web" -m tcp -j DNAT --to-destination 172.16.27.138:7493
-A KUBE-SERVICES -d 10.0.1.2/32 -p tcp -m comment --comment "default/nettools:web cluster IP" -m tcp --dport 7493 -j KUBE-SVC-INDS3KD6I5PFKUWF
-A KUBE-SVC-INDS3KD6I5PFKUWF -m comment --comment "default/nettools:web" -j KUBE-SEP-6DFMUWHMXOYMFWKG
However，I cannot delete those rules:
# iptables -D KUBE-SVC-INDS3KD6I5PFKUWF -m comment --comment "default/nettools:web" -j KUBE-SEP-6DFMUWHMXOYMFWKG
iptables v1.4.21: Couldn't load target `KUBE-SEP-6DFMUWHMXOYMFWKG':No such file or directory
# iptables -D KUBE-SERVICES -d 10.0.1.2/32 -p tcp -m comment --comment "default/nettools:web cluster IP" -m tcp --dport 7493 -j KUBE-SVC-INDS3KD6I5PFKUWF
iptables v1.4.21: Couldn't load target `KUBE-SVC-INDS3KD6I5PFKUWF':No such file or directory
There are multiple tables in play when dealing with iptables. filter table is the default if nothing is specified. The rules that you are trying to delete are part of the nat table.
Just add -t nat to your rules to delete those rules.
Example:
# iptables -t nat -D KUBE-SVC-INDS3KD6I5PFKUWF -m comment --comment "default/nettools:web" -j KUBE-SEP-6DFMUWHMXOYMFWKG
I'm using kubernetes 1.6.7 cluster, 3 nodes as master (node1, node2 and node3).
The apiserver cluster ip as bellow:
root#node1:/etc/kubernetes/manifests# kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.233.0.1 <none> 443/TCP 14d
root#node1:/etc/kubernetes/manifests# kubectl get endpoints
kubernetes 172.16.10.11:6443,172.16.10.12:6443,172.16.10.13:6443 14d
The problem is when node2 was down, tiller pod can't access 10.233.0.1:443, logs as bellow:
[storage] 2018/08/28 06:46:34 listing all releases with filter
[storage/driver] 2018/08/28 06:46:34 list: failed to list: Get https://10.233.0.1:443/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%3DTILLER: dial tcp 10.233.0.1:443: connect: no route to host
and the service kubernetes still has 3 endpoints, but i think the 172.16.10.12 should be delete from the endpoint list. iptables as bellow:
-A KUBE-SVC-LRLUS54FOYJDJ5GT -m comment --comment "default/wangxj35-nginx:nginx" -j KUBE-SEP-GGIOR777CPKVSIB7
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-CS4Z4M5BFAAWBFXQ --mask 255.255.255.255 --rsource -j KUBE-SEP-CS4Z4M5BFAAWBFXQ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-MQZA2ZXHX6GRRY22 --mask 255.255.255.255 --rsource -j KUBE-SEP-MQZA2ZXHX6GRRY22
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-ZX77NMWHNJWLXYWF --mask 255.255.255.255 --rsource -j KUBE-SEP-ZX77NMWHNJWLXYWF
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-CS4Z4M5BFAAWBFXQ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-MQZA2ZXHX6GRRY22
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-ZX77NMWHNJWLXYWF
-A KUBE-SEP-CS4Z4M5BFAAWBFXQ -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-CS4Z4M5BFAAWBFXQ --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.10.11:6443
-A KUBE-SEP-MQZA2ZXHX6GRRY22 -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-MQZA2ZXHX6GRRY22 --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.10.12:6443
-A KUBE-SEP-ZX77NMWHNJWLXYWF -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-ZX77NMWHNJWLXYWF --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.10.13:6443
Could anyone help me where the static pods service created by kubernetes source code?
Thanks.
Controller.go creates the kubernetes service:
// UpdateKubernetesService attempts to update the default Kube service.
func (c *Controller) UpdateKubernetesService(reconcile bool)> error {
// Update service & endpoint records.
Also, to work around this problem, kubernetes1.9.0 has added --endpoint-reconciler-type=lease to allow apiserver to track the alive apiserver, see issue51698, while for release below 1.9.0, you can unset the --apiserver-count flag to let alive apiserver to preempt the only one endpoint position.
I am trying to install kubernetes.
There was a problem when checking the container executed on kubernetes.
I set the type of service to Nodeport, but I could not access from a node other than the one on which the container is running.
Want to make it accessible from other computers, please tell me where it is different.
I tried externalIPs and LoadBarancer, but it was impossible.
Enviroment
OS:Ubuntu 16.04 LTS
Kubernetes:1.8
Docker:17.09.0-ce
etcd:3.2.8
flannel:0.9.0
Network
Physical:10.1.1.0/24
flannel:172.16.0.0/16
docker:192.168.0.0/16
Machines
Master Node(2nodes):10.1.1.24,10.1.1.25
Worker Node(2nodes):10.1.1.26,10.1.1.27
kubectl describe svc nginx-cluster
Name: nginx-cluster
Namespace: default
Labels: app=nginx-demo
Annotations: <none>
Selector: app=nginx-demo
Type: ClusterIP
IP: 172.16.236.159
Port: <unset> 8090/TCP
TargetPort: 80/TCP
Endpoints: 192.168.24.2:80
Session Affinity: None
Events: <none>
kubectl describe svc nginx-service
Name: nginx-service
Namespace: default
Labels: app=nginx-demo
Annotations: <none>
Selector: app=nginx-demo
Type: NodePort
IP: 172.16.199.69
Port: <unset> 8090/TCP
TargetPort: 80/TCP
NodePort: <unset> 31659/TCP
Endpoints: 192.168.24.2:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
running container worker node(10.1.1.27)
curl 10.1.1.27:31659
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
worker node(10.1.1.26)
curl 10.1.1.27:31659
curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.
other machine(10.1.1.XX)
curl 10.1.1.27:31659
curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
echoserver-848b75d85-9fx7r 1/1 Running 3 6d 192.168.70.2 k8swrksv01
nginx-demo-85cc49574c-wv2b9 1/1 Running 3 6d 192.168.2.2 k8swrksv02
kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
clusterip ClusterIP 172.16.39.77 <none> 80/TCP 6d run=echoserver
kubernetes ClusterIP 172.16.0.1 <none> 443/TCP 10d <none>
nginx-cluster ClusterIP 172.16.236.159 <none> 8090/TCP 6d app=nginx-demo
nginx-service NodePort 172.16.199.69 <none> 8090:31659/TCP 6d app=nginx-demo
nodeport NodePort 172.16.38.40 <none> 80:31317/TCP 6d run=echoserver
netstat -ntlp
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 1963/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 2202/kube-proxy
tcp 0 0 127.0.0.1:4243 0.0.0.0:* LISTEN 1758/dockerd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 996/sshd
tcp6 0 0 :::4194 :::* LISTEN 1963/kubelet
tcp6 0 0 :::10250 :::* LISTEN 1963/kubelet
tcp6 0 0 :::31659 :::* LISTEN 2202/kube-proxy
tcp6 0 0 :::10255 :::* LISTEN 1963/kubelet
tcp6 0 0 :::10256 :::* LISTEN 2202/kube-proxy
tcp6 0 0 :::31317 :::* LISTEN 2202/kube-proxy
tcp6 0 0 :::22 :::* LISTEN 996/sshd
iptables-save
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-AZ4EGFEAU4RTSLJO - [0:0]
:KUBE-SEP-C7HQKKO26GIFOZZM - [0:0]
:KUBE-SEP-EWKNS2YCPXGJCXDC - [0:0]
:KUBE-SEP-LQVPUPFGW6BWATIP - [0:0]
:KUBE-SEP-OMMOFZ27GPKZ4OPA - [0:0]
:KUBE-SEP-UD3HOGDD5NDLNY74 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-CQNAS6RSUGJF2C2D - [0:0]
:KUBE-SVC-GKN7Y2BSGW4NJTYL - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-XP7QDA4CRQ2QA33W - [0:0]
:KUBE-SVC-Z5P6OMNAEVLAQUTS - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 192.168.2.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 192.168.0.0/16 -d 192.168.0.0/16 -j RETURN
-A POSTROUTING -s 192.168.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.2.0/24 -j RETURN
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.0.0/16 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -s 192.168.70.2/32 -m comment --comment "default/clusterip:" -j KUBE-MARK-MASQ
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -p tcp -m comment --comment "default/clusterip:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-C7HQKKO26GIFOZZM -s 192.168.70.2/32 -m comment --comment "default/nodeport:" -j KUBE-MARK-MASQ
-A KUBE-SEP-C7HQKKO26GIFOZZM -p tcp -m comment --comment "default/nodeport:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-EWKNS2YCPXGJCXDC -s 10.1.1.25/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-EWKNS2YCPXGJCXDC -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.25:6443
-A KUBE-SEP-LQVPUPFGW6BWATIP -s 192.168.2.2/32 -m comment --comment "default/nginx-service:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LQVPUPFGW6BWATIP -p tcp -m comment --comment "default/nginx-service:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -s 10.1.1.24/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.24:6443
-A KUBE-SEP-UD3HOGDD5NDLNY74 -s 192.168.2.2/32 -m comment --comment "default/nginx-cluster:" -j KUBE-MARK-MASQ
-A KUBE-SEP-UD3HOGDD5NDLNY74 -p tcp -m comment --comment "default/nginx-cluster:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SERVICES -d 172.16.236.159/32 -p tcp -m comment --comment "default/nginx-cluster: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-Z5P6OMNAEVLAQUTS
-A KUBE-SERVICES -d 172.16.199.69/32 -p tcp -m comment --comment "default/nginx-service: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-SERVICES -d 172.16.38.40/32 -p tcp -m comment --comment "default/nodeport: cluster IP" -m tcp --dport 80 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-SERVICES -d 172.16.39.77/32 -p tcp -m comment --comment "default/clusterip: cluster IP" -m tcp --dport 80 -j KUBE-SVC-CQNAS6RSUGJF2C2D
-A KUBE-SERVICES -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-CQNAS6RSUGJF2C2D -m comment --comment "default/clusterip:" -j KUBE-SEP-AZ4EGFEAU4RTSLJO
-A KUBE-SVC-GKN7Y2BSGW4NJTYL -m comment --comment "default/nginx-service:" -j KUBE-SEP-LQVPUPFGW6BWATIP
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-XP7QDA4CRQ2QA33W -m comment --comment "default/nodeport:" -j KUBE-SEP-C7HQKKO26GIFOZZM
-A KUBE-SVC-Z5P6OMNAEVLAQUTS -m comment --comment "default/nginx-cluster:" -j KUBE-SEP-UD3HOGDD5NDLNY74
COMMIT
*filter
:INPUT ACCEPT [40:14606]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [42:6275]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:DOCKER-USER - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A DOCKER-USER -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT
It's caused by default DROP on FORWARD chain (which in turn is caused by docker).
You can see it working again if you add iptables -A FORWARD -j ACCEPT rule to your node.
The k8s issue is here: https://github.com/kubernetes/kubernetes/issues/39823 but the actual fix is here https://github.com/kubernetes/kubernetes/pull/52569 (expected to be in 1.9).
additional information.
Something is supposed to block the port, but that is unknown...
pod running node
nmap 10.1.1.27 -p31000-32000
Not shown: 999 closed ports
PORT STATE SERVICE
31317/tcp open unknown
31659/tcp open unknown
other node
nmap 10.1.1.27 -p31000-32000
Not shown: 999 closed ports
PORT STATE SERVICE
31317/tcp filtered unknown
31659/tcp filtered unknown
Farcaller is right. We hit the same on centos running firewalld.
Until we upgrade to k8s 1.9 we added the following firewalld rule. The rule is similar to the one created by kube-proxy in k8s 1.9
#!/bin/bash
# follows https://github.com/kubernetes/kubernetes/pull/52569 introduced in k8s 1.9
# required to support nodeport services routing from all nodes in the cluster when the firewall is turned on.
# KUBE-MARK-MASQ corresponds to kube-proxy --iptables-masquerade-bit=14, which is the default.
KUBE_MARK_MASQ="0x4000/0x4000"
firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 3 -m mark --mark "$KUBE_MARK_MASQ" -j ACCEPT
you can first check the node port whether the port is open,
use
netstat -ntlp
to check, if yes , that maybe some problem with the iptable or routing,
if no , please check the firewall or other question
Good luck