Cannot access pod network through master node - kubernetes

Question

Follow the tutorial https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ to deploy a single-node kubernetes with canal network plugin.
# kubeadm init --pod-network-cidr 10.244.0.0/16 --kubernetes-version stable-1.9
kube-dns container are not all running.
# kubectl -n kube-system get pod
NAME READY STATUS RESTARTS AGE
canal-mpzrt 3/3 Running 0 6h
etcd-gavin-k8s 1/1 Running 0 6h
kube-apiserver-gavin-k8s 1/1 Running 0 6h
kube-controller-manager-gavin-k8s 1/1 Running 0 6h
kube-dns-6f4fd4bdf-fc8pd 2/3 Running 0 53s
kube-proxy-vj2r9 1/1 Running 0 2h
kube-scheduler-gavin-k8s 1/1 Running 0 6h
kubectl -n kube-system logs kube-dns-6f4fd4bdf-fc8pd kubedns
I0425 08:40:41.303524 1 dns.go:48] version: 1.14.6-3-gc36cb11
I0425 08:40:41.304274 1 server.go:69] Using configuration read from directory: /kube-dns-config with period 10s
I0425 08:40:41.304308 1 server.go:112] FLAG: --alsologtostderr="false"
I0425 08:40:41.304316 1 server.go:112] FLAG: --config-dir="/kube-dns-config"
I0425 08:40:41.304326 1 server.go:112] FLAG: --config-map=""
I0425 08:40:41.304330 1 server.go:112] FLAG: --config-map-namespace="kube-system"
I0425 08:40:41.304334 1 server.go:112] FLAG: --config-period="10s"
I0425 08:40:41.304340 1 server.go:112] FLAG: --dns-bind-address="0.0.0.0"
I0425 08:40:41.304343 1 server.go:112] FLAG: --dns-port="10053"
I0425 08:40:41.304349 1 server.go:112] FLAG: --domain="cluster.local."
I0425 08:40:41.304354 1 server.go:112] FLAG: --federations=""
I0425 08:40:41.304359 1 server.go:112] FLAG: --healthz-port="8081"
I0425 08:40:41.304363 1 server.go:112] FLAG: --initial-sync-timeout="1m0s"
I0425 08:40:41.304367 1 server.go:112] FLAG: --kube-master-url=""
I0425 08:40:41.304372 1 server.go:112] FLAG: --kubecfg-file=""
I0425 08:40:41.304376 1 server.go:112] FLAG: --log-backtrace-at=":0"
I0425 08:40:41.304382 1 server.go:112] FLAG: --log-dir=""
I0425 08:40:41.304386 1 server.go:112] FLAG: --log-flush-frequency="5s"
I0425 08:40:41.304391 1 server.go:112] FLAG: --logtostderr="true"
I0425 08:40:41.304394 1 server.go:112] FLAG: --nameservers=""
I0425 08:40:41.304398 1 server.go:112] FLAG: --stderrthreshold="2"
I0425 08:40:41.304401 1 server.go:112] FLAG: --v="2"
I0425 08:40:41.304405 1 server.go:112] FLAG: --version="false"
I0425 08:40:41.304411 1 server.go:112] FLAG: --vmodule=""
I0425 08:40:41.304482 1 server.go:194] Starting SkyDNS server (0.0.0.0:10053)
I0425 08:40:41.304700 1 server.go:213] Skydns metrics enabled (/metrics:10055)
I0425 08:40:41.304709 1 dns.go:146] Starting endpointsController
I0425 08:40:41.304715 1 dns.go:149] Starting serviceController
I0425 08:40:41.308584 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0425 08:40:41.308603 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0425 08:40:41.804866 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0425 08:40:42.304875 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0425 08:40:42.804873 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0425 08:40:43.304871 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0425 08:40:43.804868 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0425 08:40:44.304880 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0425 08:40:44.804873 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0425 08:40:45.304869 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0425 08:40:45.804863 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0425 08:40:46.304833 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0425 08:40:46.804868 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0425 08:40:47.304876 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0425 08:40:47.804878 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I found the root cause of kube-dns failure is container in pod cannot access my machine's physical ip.
Master node run at 192.168.80.167
# kubectl cluster-info
Kubernetes master is running at https://192.168.80.167:6443
KubeDNS is running at https://192.168.80.167:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
196.18.80.167 is an address of physical network bridge on my machine.
# ifconfig br0
br0 Link encap:Ethernet HWaddr 24:5E:BE:0C:C5:92
inet addr:192.168.80.167 Bcast:192.168.81.255 Mask:255.255.254.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4661901 errors:0 dropped:191628 overruns:0 frame:0
TX packets:317984 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1116345980 (1.0 GiB) TX bytes:56761158 (54.1 MiB)
# brctl show br0
bridge name bridge id STP enabled interfaces
br0 8000.245ebe0cc592 no eth0
kubedns container cannot access the physical bridge ip of my machine, then it failed.
# kubectl -n kube-system exec -it kube-dns-6f4fd4bdf-fc8pd --container kubedns -- sh
/ # ping 192.168.80.167
PING 192.168.80.167 (192.168.80.167): 56 data bytes
^C
--- 192.168.80.167 ping statistics ---
16 packets transmitted, 0 packets received, 100% packet loss
The strange thing is kubedns can access other machines in LAN. It cannot access my machine which running the pod only.
/ # ping 192.168.80.107
PING 192.168.80.107 (192.168.80.107): 56 data bytes
64 bytes from 192.168.80.107: seq=0 ttl=63 time=0.361 ms
64 bytes from 192.168.80.107: seq=1 ttl=63 time=0.342 ms
64 bytes from 192.168.80.107: seq=2 ttl=63 time=4.112 ms
^C
--- 192.168.80.107 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.342/1.605/4.112 ms
Analyze network traffic by tcpdump, the traffic comes from calic0b238d4ce2 is not forward into br0, so no one answer the traffic.
# tcpdump -i caliec0efa8668a -Q inout | grep ICMP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on caliec0efa8668a, link-type EN10MB (Ethernet), capture size 262144 bytes
09:05:31.950671 IP 10.244.0.3 > Gavin-K8S: ICMP echo request, id 34560, seq 54, length 64
09:05:32.950733 IP 10.244.0.3 > Gavin-K8S: ICMP echo request, id 34560, seq 55, length 64
09:05:33.950794 IP 10.244.0.3 > Gavin-K8S: ICMP echo request, id 34560, seq 56, length 64
# tcpdump -i br0 -Q inout | grep ICMP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
P.S: Every user pods are the same situation with kubedns: pods cannot access the node running it, but they can access other machines.
On host(master node), check the routing table
# ip route show
default via 192.168.80.254 dev br0 proto static metric 100
10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1
10.0.5.0/24 dev docker0 proto kernel scope link src 10.0.5.1 dead linkdown
10.244.0.4 dev calic0b238d4ce2 scope link
10.244.0.6 dev cali45026c409f9 scope link
10.244.0.7 dev caliec0efa8668a scope link
169.254.0.0/16 dev docker_gwbridge proto kernel scope link src 169.254.8.151
192.168.80.0/23 dev br0 proto kernel scope link src 192.168.80.167
# ip route get 192.168.80.167
local 192.168.80.167 dev lo src 192.168.80.167
cache <local>
On container, check the routing table
/ # ip route show
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0
/ # ip route get 192.168.80.167
192.168.80.167 via 169.254.1.1 dev eth0 src 10.244.0.7
Result of iptable-save
# Generated by iptables-save v1.6.0 on Wed Apr 25 21:25:22 2018
*raw
:PREROUTING ACCEPT [5988958:1384538104]
:OUTPUT ACCEPT [4321136:929267397]
:cali-OUTPUT - [0:0]
:cali-PREROUTING - [0:0]
:cali-failsafe-in - [0:0]
:cali-failsafe-out - [0:0]
:cali-from-host-endpoint - [0:0]
:cali-to-host-endpoint - [0:0]
-A PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" -j cali-PREROUTING
-A OUTPUT -m comment --comment "cali:tVnHkvAo15HuiPy0" -j cali-OUTPUT
-A cali-OUTPUT -m comment --comment "cali:WX1xZBEtmbS0Rhjs" -j MARK --set-xmark 0x0/0xf000000
-A cali-OUTPUT -m comment --comment "cali:iE00ZyllJNXfrlg_" -j cali-to-host-endpoint
-A cali-OUTPUT -m comment --comment "cali:Asois4hxp1rUxwJS" -m mark --mark 0x1000000/0x1000000 -j ACCEPT
-A cali-PREROUTING -m comment --comment "cali:zatSDPVUhhPCk6Iy" -j MARK --set-xmark 0x0/0xf000000
-A cali-PREROUTING -i cali+ -m comment --comment "cali:-ES4EW0vxFmM81t8" -j MARK --set-xmark 0x4000000/0x4000000
-A cali-PREROUTING -m comment --comment "cali:VE1J3S_1t9q8GAsm" -m mark --mark 0x0/0x4000000 -j cali-from-host-endpoint
-A cali-PREROUTING -m comment --comment "cali:VX8l4jKL9w89GXz5" -m mark --mark 0x1000000/0x1000000 -j ACCEPT
-A cali-failsafe-in -p tcp -m comment --comment "cali:wWFQM43tJU7wwnFZ" -m multiport --dports 22 -j ACCEPT
-A cali-failsafe-in -p udp -m comment --comment "cali:LwNV--R8MjeUYacw" -m multiport --dports 68 -j ACCEPT
-A cali-failsafe-out -p tcp -m comment --comment "cali:73bZKoyDfOpFwC2T" -m multiport --dports 2379 -j ACCEPT
-A cali-failsafe-out -p tcp -m comment --comment "cali:QMFuWo6o-d9yOpNm" -m multiport --dports 2380 -j ACCEPT
-A cali-failsafe-out -p tcp -m comment --comment "cali:Kup7QkrsdmfGX0uL" -m multiport --dports 4001 -j ACCEPT
-A cali-failsafe-out -p tcp -m comment --comment "cali:xYYr5PEqDf_Pqfkv" -m multiport --dports 7001 -j ACCEPT
-A cali-failsafe-out -p udp -m comment --comment "cali:nbWBvu4OtudVY60Q" -m multiport --dports 53 -j ACCEPT
-A cali-failsafe-out -p udp -m comment --comment "cali:UxFu5cDK5En6dT3Y" -m multiport --dports 67 -j ACCEPT
COMMIT
# Completed on Wed Apr 25 21:25:22 2018
# Generated by iptables-save v1.6.0 on Wed Apr 25 21:25:22 2018
*nat
:PREROUTING ACCEPT [16:2103]
:INPUT ACCEPT [14:1981]
:OUTPUT ACCEPT [5:677]
:POSTROUTING ACCEPT [4:617]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-JPEBCQ2YOSKQPXKG - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
:SYSDOCKER - [0:0]
:SYSNAT - [0:0]
:VPNNAT - [0:0]
:cali-OUTPUT - [0:0]
:cali-POSTROUTING - [0:0]
:cali-PREROUTING - [0:0]
:cali-fip-dnat - [0:0]
:cali-fip-snat - [0:0]
:cali-nat-outgoing - [0:0]
-A PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" -j cali-PREROUTING
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "cali:tVnHkvAo15HuiPy0" -j cali-OUTPUT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "cali:O3lYWMrLQYEMJtB5" -j cali-POSTROUTING
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
-A POSTROUTING -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j RETURN
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-JPEBCQ2YOSKQPXKG -s 192.168.80.167/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-JPEBCQ2YOSKQPXKG -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-JPEBCQ2YOSKQPXKG --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 192.168.80.167:6443
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-JPEBCQ2YOSKQPXKG --mask 255.255.255.255 --rsource -j KUBE-SEP-JPEBCQ2YOSKQPXKG
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-JPEBCQ2YOSKQPXKG
-A cali-OUTPUT -m comment --comment "cali:GBTAv2p5CwevEyJm" -j cali-fip-dnat
-A cali-POSTROUTING -m comment --comment "cali:Z-c7XtVd2Bq7s_hA" -j cali-fip-snat
-A cali-POSTROUTING -m comment --comment "cali:nYKhEzDlr11Jccal" -j cali-nat-outgoing
-A cali-PREROUTING -m comment --comment "cali:r6XmIziWUJsdOK6Z" -j cali-fip-dnat
-A cali-nat-outgoing -m comment --comment "cali:Wd76s91357Uv7N3v" -m set --match-set cali4-masq-ipam-pools src -m set ! --match-set cali4-all-ipam-pools dst -j MASQUERADE
COMMIT
# Completed on Wed Apr 25 21:25:23 2018
# Generated by iptables-save v1.6.0 on Wed Apr 25 21:25:23 2018
*mangle
:PREROUTING ACCEPT [1727587:391808161]
:INPUT ACCEPT [5150922:1211808224]
:FORWARD ACCEPT [1062:89161]
:OUTPUT ACCEPT [4321182:929275109]
:POSTROUTING ACCEPT [4331603:931649202]
:VPNCUSSETMARK - [0:0]
:VPNDEFSETMARK - [0:0]
:cali-PREROUTING - [0:0]
:cali-failsafe-in - [0:0]
:cali-from-host-endpoint - [0:0]
-A PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" -j cali-PREROUTING
-A PREROUTING -j VPNCUSSETMARK
-A PREROUTING -m mark --mark 0x0/0xffff -j VPNDEFSETMARK
-A VPNCUSSETMARK -m set --match-set vpnbr0 src -j MARK --set-xmark 0x900/0xff00
-A VPNCUSSETMARK -m set --match-set vpndocker0 src -j MARK --set-xmark 0xa00/0xff00
-A VPNCUSSETMARK -m set --match-set vpnlxcbr0 src -j MARK --set-xmark 0xc00/0xff00
-A cali-PREROUTING -m comment --comment "cali:6BJqBjBC7crtA-7-" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-PREROUTING -m comment --comment "cali:nE3PUa5RSRqBBvwx" -m mark --mark 0x1000000/0x1000000 -j ACCEPT
-A cali-PREROUTING -i cali+ -m comment --comment "cali:qgFofvzQe6yJPouQ" -j ACCEPT
-A cali-PREROUTING -m comment --comment "cali:o178eO5vvpj8e65z" -j cali-from-host-endpoint
-A cali-PREROUTING -m comment --comment "cali:5TQcm-i_T8rVGEEa" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x1000000/0x1000000 -j ACCEPT
-A cali-failsafe-in -p tcp -m comment --comment "cali:wWFQM43tJU7wwnFZ" -m multiport --dports 22 -j ACCEPT
-A cali-failsafe-in -p udp -m comment --comment "cali:LwNV--R8MjeUYacw" -m multiport --dports 68 -j ACCEPT
COMMIT
# Completed on Wed Apr 25 21:25:23 2018
# Generated by iptables-save v1.6.0 on Wed Apr 25 21:25:23 2018
*filter
:INPUT ACCEPT [3389:699050]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [2944:635600]
:DOCKER-USER - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-SERVICES - [0:0]
:SYSDOCKER - [0:0]
:SYSDOCKER-ISOLATION - [0:0]
:cali-FORWARD - [0:0]
:cali-INPUT - [0:0]
:cali-OUTPUT - [0:0]
:cali-failsafe-in - [0:0]
:cali-failsafe-out - [0:0]
:cali-from-host-endpoint - [0:0]
:cali-from-wl-dispatch - [0:0]
:cali-fw-cali45026c409f9 - [0:0]
:cali-fw-calic0b238d4ce2 - [0:0]
:cali-fw-caliec0efa8668a - [0:0]
:cali-pri-k8s_ns.default - [0:0]
:cali-pri-k8s_ns.kube-system - [0:0]
:cali-pro-k8s_ns.default - [0:0]
:cali-pro-k8s_ns.kube-system - [0:0]
:cali-to-host-endpoint - [0:0]
:cali-to-wl-dispatch - [0:0]
:cali-tw-cali45026c409f9 - [0:0]
:cali-tw-calic0b238d4ce2 - [0:0]
:cali-tw-caliec0efa8668a - [0:0]
:cali-wl-to-host - [0:0]
-A INPUT -m comment --comment "cali:Cz_u1IQiXIMmKD4c" -j cali-INPUT
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "cali:wUHhoiAYhphO9Mso" -j cali-FORWARD
-A FORWARD -m comment --comment "kubernetes forward rules" -j KUBE-FORWARD
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT
-A FORWARD -i br0 -o caliec0efa8668a -j ACCEPT
-A FORWARD -i caliec0efa8668a -o br0 -j ACCEPT
-A OUTPUT -m comment --comment "cali:tVnHkvAo15HuiPy0" -j cali-OUTPUT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-USER -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns has no endpoints" -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp has no endpoints" -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A SYSDOCKER-ISOLATION -j RETURN
-A cali-FORWARD -i cali+ -m comment --comment "cali:X3vB2lGcBrfkYquC" -j cali-from-wl-dispatch
-A cali-FORWARD -o cali+ -m comment --comment "cali:UtJ9FnhBnFbyQMvU" -j cali-to-wl-dispatch
-A cali-FORWARD -i cali+ -m comment --comment "cali:Tt19HcSdA5YIGSsw" -j ACCEPT
-A cali-FORWARD -o cali+ -m comment --comment "cali:9LzfFCvnpC5_MYXm" -j ACCEPT
-A cali-FORWARD -m comment --comment "cali:7AofLLOqCM5j36rM" -j MARK --set-xmark 0x0/0xe000000
-A cali-FORWARD -m comment --comment "cali:QM1_joSl7tL76Az7" -m mark --mark 0x0/0x1000000 -j cali-from-host-endpoint
-A cali-FORWARD -m comment --comment "cali:C1QSog3bk0AykjAO" -j cali-to-host-endpoint
-A cali-FORWARD -m comment --comment "cali:DmFiPAmzcisqZcvo" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x1000000/0x1000000 -j ACCEPT
-A cali-INPUT -m comment --comment "cali:i7okJZpS8VxaJB3n" -m mark --mark 0x1000000/0x1000000 -j ACCEPT
-A cali-INPUT -i cali+ -m comment --comment "cali:JaoDb6CLdcGw8g0Y" -g cali-wl-to-host
-A cali-INPUT -m comment --comment "cali:c5eKVW2VdKQ_LiSM" -j MARK --set-xmark 0x0/0xf000000
-A cali-INPUT -m comment --comment "cali:hwQKYSlSCkpE_9uN" -j cali-from-host-endpoint
-A cali-INPUT -m comment --comment "cali:ttp8-serzKCP-bKZ" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x1000000/0x1000000 -j ACCEPT
-A cali-OUTPUT -m comment --comment "cali:YQSSJIsRcHjFbXaI" -m mark --mark 0x1000000/0x1000000 -j ACCEPT
-A cali-OUTPUT -o cali+ -m comment --comment "cali:KRjBsKsBcFBYKCEw" -j RETURN
-A cali-OUTPUT -m comment --comment "cali:3VKAQBcyUUW5kS_j" -j MARK --set-xmark 0x0/0xf000000
-A cali-OUTPUT -m comment --comment "cali:Z1mBCSH1XHM6qq0k" -j cali-to-host-endpoint
-A cali-OUTPUT -m comment --comment "cali:N0jyWt2RfBedKw3L" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x1000000/0x1000000 -j ACCEPT
-A cali-failsafe-in -p tcp -m comment --comment "cali:wWFQM43tJU7wwnFZ" -m multiport --dports 22 -j ACCEPT
-A cali-failsafe-in -p udp -m comment --comment "cali:LwNV--R8MjeUYacw" -m multiport --dports 68 -j ACCEPT
-A cali-failsafe-out -p tcp -m comment --comment "cali:73bZKoyDfOpFwC2T" -m multiport --dports 2379 -j ACCEPT
-A cali-failsafe-out -p tcp -m comment --comment "cali:QMFuWo6o-d9yOpNm" -m multiport --dports 2380 -j ACCEPT
-A cali-failsafe-out -p tcp -m comment --comment "cali:Kup7QkrsdmfGX0uL" -m multiport --dports 4001 -j ACCEPT
-A cali-failsafe-out -p tcp -m comment --comment "cali:xYYr5PEqDf_Pqfkv" -m multiport --dports 7001 -j ACCEPT
-A cali-failsafe-out -p udp -m comment --comment "cali:nbWBvu4OtudVY60Q" -m multiport --dports 53 -j ACCEPT
-A cali-failsafe-out -p udp -m comment --comment "cali:UxFu5cDK5En6dT3Y" -m multiport --dports 67 -j ACCEPT
-A cali-from-wl-dispatch -i cali45026c409f9 -m comment --comment "cali:QTLwRyKNiscc-kE7" -g cali-fw-cali45026c409f9
-A cali-from-wl-dispatch -i calic0b238d4ce2 -m comment --comment "cali:7mRUmkMzCXKDHDzk" -g cali-fw-calic0b238d4ce2
-A cali-from-wl-dispatch -i caliec0efa8668a -m comment --comment "cali:vI_cBpGlZQpakzSQ" -g cali-fw-caliec0efa8668a
-A cali-from-wl-dispatch -m comment --comment "cali:y5WqyrGI7OWfnqNM" -m comment --comment "Unknown interface" -j DROP
-A cali-fw-cali45026c409f9 -m comment --comment "cali:OTJIDsP3TegJFYqm" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-fw-cali45026c409f9 -m comment --comment "cali:uvhYBVFYvBcMfF1E" -m conntrack --ctstate INVALID -j DROP
-A cali-fw-cali45026c409f9 -m comment --comment "cali:N9Pier8knvEySzpb" -j MARK --set-xmark 0x0/0x1000000
-A cali-fw-cali45026c409f9 -m comment --comment "cali:6ctr2BZXeRQITWs2" -j cali-pro-k8s_ns.kube-system
-A cali-fw-cali45026c409f9 -m comment --comment "cali:Juq9dxqhxLUhudVk" -m comment --comment "Return if profile accepted" -m mark --mark 0x1000000/0x1000000 -j RETURN
-A cali-fw-cali45026c409f9 -m comment --comment "cali:o7CTzqIS9bu5DymV" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-fw-calic0b238d4ce2 -m comment --comment "cali:2dB9gQ0XK7ky-okg" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-fw-calic0b238d4ce2 -m comment --comment "cali:ywcP6SMI-Q-GlUyW" -m conntrack --ctstate INVALID -j DROP
-A cali-fw-calic0b238d4ce2 -m comment --comment "cali:wroMotnj-PmPY-A1" -j MARK --set-xmark 0x0/0x1000000
-A cali-fw-calic0b238d4ce2 -m comment --comment "cali:nOL8WwmNyRPNDCRb" -j cali-pro-k8s_ns.default
-A cali-fw-calic0b238d4ce2 -m comment --comment "cali:r1XYAvTJ5M_XMUux" -m comment --comment "Return if profile accepted" -m mark --mark 0x1000000/0x1000000 -j RETURN
-A cali-fw-calic0b238d4ce2 -m comment --comment "cali:8-iYoFbdlSboxtvI" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-fw-caliec0efa8668a -m comment --comment "cali:NvFOTdFzvt46kQfQ" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-fw-caliec0efa8668a -m comment --comment "cali:jxl0wYR8pO3dsQLg" -m conntrack --ctstate INVALID -j DROP
-A cali-fw-caliec0efa8668a -m comment --comment "cali:VlVHHstfJPnNr3LI" -j MARK --set-xmark 0x0/0x1000000
-A cali-fw-caliec0efa8668a -m comment --comment "cali:DlqVod2qRMSGS4t4" -j cali-pro-k8s_ns.default
-A cali-fw-caliec0efa8668a -m comment --comment "cali:LluPSlt2p5-XuwUs" -m comment --comment "Return if profile accepted" -m mark --mark 0x1000000/0x1000000 -j RETURN
-A cali-fw-caliec0efa8668a -m comment --comment "cali:23YDqnq73LBpscup" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-pri-k8s_ns.default -m comment --comment "cali:6MWuUqsVPzpSgE3L" -j MARK --set-xmark 0x1000000/0x1000000
-A cali-pri-k8s_ns.default -m comment --comment "cali:UGCdoOXoPRcONGv8" -m mark --mark 0x1000000/0x1000000 -j RETURN
-A cali-pri-k8s_ns.kube-system -m comment --comment "cali:plMTf6GGo5FLt-zw" -j MARK --set-xmark 0x1000000/0x1000000
-A cali-pri-k8s_ns.kube-system -m comment --comment "cali:d_ypsHpl3J96oOpx" -m mark --mark 0x1000000/0x1000000 -j RETURN
-A cali-pro-k8s_ns.default -m comment --comment "cali:DTsGE7pFaKbRuEBg" -j MARK --set-xmark 0x1000000/0x1000000
-A cali-pro-k8s_ns.default -m comment --comment "cali:4bIByWXruQ1DMcbo" -m mark --mark 0x1000000/0x1000000 -j RETURN
-A cali-pro-k8s_ns.kube-system -m comment --comment "cali:lDQGDZg5UANF5wIK" -j MARK --set-xmark 0x1000000/0x1000000
-A cali-pro-k8s_ns.kube-system -m comment --comment "cali:wn_dnW-P0COWnhhy" -m mark --mark 0x1000000/0x1000000 -j RETURN
-A cali-to-wl-dispatch -o cali45026c409f9 -m comment --comment "cali:c75T2Dgm3k-jJrbE" -g cali-tw-cali45026c409f9
-A cali-to-wl-dispatch -o calic0b238d4ce2 -m comment --comment "cali:qDV3G3z8-XF7ASpj" -g cali-tw-calic0b238d4ce2
-A cali-to-wl-dispatch -o caliec0efa8668a -m comment --comment "cali:0KGW9LSlkHoj3Pth" -g cali-tw-caliec0efa8668a
-A cali-to-wl-dispatch -m comment --comment "cali:jDu3duVnwTVndWys" -m comment --comment "Unknown interface" -j DROP
-A cali-tw-cali45026c409f9 -m comment --comment "cali:T8ds95eQAxnZl6cA" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-tw-cali45026c409f9 -m comment --comment "cali:sBFjo942EoAZxbwi" -m conntrack --ctstate INVALID -j DROP
-A cali-tw-cali45026c409f9 -m comment --comment "cali:7mrDpuB_JSOiwD-w" -j MARK --set-xmark 0x0/0x1000000
-A cali-tw-cali45026c409f9 -m comment --comment "cali:SZ7jptebHBWtu0ut" -j cali-pri-k8s_ns.kube-system
-A cali-tw-cali45026c409f9 -m comment --comment "cali:XZUosCvhE-CFRBZf" -m comment --comment "Return if profile accepted" -m mark --mark 0x1000000/0x1000000 -j RETURN
-A cali-tw-cali45026c409f9 -m comment --comment "cali:UPdmXt0SUq5GpdCk" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-tw-calic0b238d4ce2 -m comment --comment "cali:k8kHsWO63lPZ_T5S" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-tw-calic0b238d4ce2 -m comment --comment "cali:WcRO5jfNEyBl-P8e" -m conntrack --ctstate INVALID -j DROP
-A cali-tw-calic0b238d4ce2 -m comment --comment "cali:qgZ3s3ojXF7_0v41" -j MARK --set-xmark 0x0/0x1000000
-A cali-tw-calic0b238d4ce2 -m comment --comment "cali:l9FROf8cQyfmubvU" -j cali-pri-k8s_ns.default
-A cali-tw-calic0b238d4ce2 -m comment --comment "cali:i1mW8rmxu9TCd-T4" -m comment --comment "Return if profile accepted" -m mark --mark 0x1000000/0x1000000 -j RETURN
-A cali-tw-calic0b238d4ce2 -m comment --comment "cali:EOs-JJ221Us5p0EP" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-tw-caliec0efa8668a -m comment --comment "cali:_7y3hRmp6EU47Y0s" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-tw-caliec0efa8668a -m comment --comment "cali:lqljOLOQn5ZkCC2p" -m conntrack --ctstate INVALID -j DROP
-A cali-tw-caliec0efa8668a -m comment --comment "cali:AGwqz_dfQJPaIJOa" -j MARK --set-xmark 0x0/0x1000000
-A cali-tw-caliec0efa8668a -m comment --comment "cali:IQNHtVteTcEbbzLF" -j cali-pri-k8s_ns.default
-A cali-tw-caliec0efa8668a -m comment --comment "cali:zFjCvYL15RsUfNaU" -m comment --comment "Return if profile accepted" -m mark --mark 0x1000000/0x1000000 -j RETURN
-A cali-tw-caliec0efa8668a -m comment --comment "cali:-GRpWsx8gV1ZNLvl" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-wl-to-host -m comment --comment "cali:Ee9Sbo10IpVujdIY" -j cali-from-wl-dispatch
-A cali-wl-to-host -m comment --comment "cali:nSZbcOoG1xPONxb8" -m comment --comment "Configured DefaultEndpointToHostAction" -j ACCEPT
COMMIT
# Completed on Wed Apr 25 21:25:23 2018

Answer 1

This is just a guess, but I think I know what the problem is.
Kubernetes uses iptables to manage traffic between pods and process requests to services, including some NAT rules.
When you call the service on the node, your request also processed by iptables, which includes NAT rules based on the source IP.
But, looks like when you call the service from the same node, your packets do not match the NAT rule of the Service and they are not processed correctly.
Calico has NatOutgoing option which enables masquerading for all packets with destinations outside the Pool.
With that option, Calico will masquerade packages (replace the source IP with the IP of the node) and it will be routed as the package from the node itself and will be caught by the right service's NAT rule.
Looks like it might help.

Answer 2

The ip rules of my machine, it will block container network traffic from going to my physical ip. After I delete the ip rule, the issue is resolved.