kube-proxy unhappy after setting IPALLOC_RANGE in weave configuration - kubernetes

Question

I’m installing kubernetes v1.5.6 + weave using kubeadm-1.6.0-0.alpha.0.2074.a092d8e0f95f52.x86_64.rpm on CentOS 7.3. Since my network IP range for host machine is 10.41.30.xx and it overlaps with internal weave IP range. I configured weave to use IPALLOC_RANGE as 172.30.0.0/16.
After setup, I’m not able to connect to kubernetes services. Kube-proxy complains about connecting to kubernetes master.
E0728 18:04:47.201682 1 server.go:421] Can't get Node "ctdpc001571.ctd.khi.com", assuming iptables proxy, err: Get https://10.41.30.50:6443/api/v1/nodes/ctdpc001571.ctd.khi.co.jp: dial tcp 10.41.30.50:6443: getsockopt: connection refused
I0728 18:04:47.204522 1 server.go:215] Using iptables Proxier.
W0728 18:04:47.205022 1 server.go:468] Failed to retrieve node info: Get https://10.41.30.50:6443/api/v1/nodes/ctdpc001571.ctd.khi.com: dial tcp 10.41.30.50:6443: getsockopt: connection refused
W0728 18:04:47.205325 1 proxier.go:249] invalid nodeIP, initialize kube-proxy with 127.0.0.1 as nodeIP
W0728 18:04:47.205347 1 proxier.go:254] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0728 18:04:47.205394 1 server.go:227] Tearing down userspace rules.
I0728 18:04:47.238324 1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_max' to 1048576
I0728 18:04:47.239243 1 conntrack.go:66] Setting conntrack hashsize to 262144
I0728 18:04:47.242492 1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0728 18:04:47.242640 1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
E0728 18:04:47.260748 1 reflector.go:188] pkg/proxy/config/api.go:33: Failed to list *api.Endpoints: Get https://10.41.30.50:6443/api/v1/endpoints?resourceVersion=0: dial tcp 10.41.30.50:6443: getsockopt: connection refused
E0728 18:04:47.260931 1 reflector.go:188] pkg/proxy/config/api.go:30: Failed to list *api.Service: Get https://10.41.30.50:6443/api/v1/services?resourceVersion=0: dial tcp 10.41.30.50:6443: getsockopt: connection refused
E0728 18:04:47.265569 1 event.go:208] Unable to write event: 'Post https://10.41.30.50:6443/api/v1/namespaces/default/events: dial tcp 10.41.30.50:6443: getsockopt: connection refused' (may retry after sleeping)
E0728 18:04:48.262006 1 reflector.go:188] pkg/proxy/config/api.go:33: Failed to list *api.Endpoints: Get https://10.41.30.50:6443/api/v1/endpoints?resourceVersion=0: dial tcp 10.41.30.50:6443: getsockopt: connection refused
Steps I followed:
$ yum -y install \
yum-versionlock \
docker-1.12.6-11.el7.centos \
kubectl-1.5.4-0 \
kubelet-1.5.4-0 \
kubernetes-cni-0.3.0.1-0.07a8a2 \
https://storage.googleapis.com/falkonry-k8-installer/kubeadm-1.6.0-0.alpha.0.2074.a092d8e0f95f52.x86_64.rpm
$ yum versionlock add kubectl kubelet kubernetes-cni kubeadm
$ systemctl enable docker && systemctl start docker
$ systemctl enable kubelet && systemctl start kubelet
$ kubeadm init --use-kubernetes-version=v1.5.6
Set `IPALLOC_RANGE` as `172.30.0.0/16` in https://git.io/weave-kube
$ kubectl apply -f weave-kube-config
$ kubectl run -i --tty busybox --image=busybox -- sh
$ nslookup kubernetes
After this, I cannot connect to kubernetes or any of the other services.
iptable
# Generated by iptables-save v1.4.21 on Sat Jul 29 04:19:10 2017
*filter
:INPUT ACCEPT [1350:566634]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1344:579110]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
:WEAVE-NPC - [0:0]
:WEAVE-NPC-DEFAULT - [0:0]
:WEAVE-NPC-INGRESS - [0:0]
-A INPUT -j KUBE-FIREWALL
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o weave -j WEAVE-NPC
-A FORWARD -o weave -m state --state NEW -j NFLOG --nflog-group 86
-A FORWARD -o weave -j DROP
-A FORWARD -i weave ! -o weave -j ACCEPT
-A FORWARD -o weave -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A WEAVE-NPC -m state --state RELATED,ESTABLISHED -j ACCEPT
-A WEAVE-NPC -d 224.0.0.0/4 -j ACCEPT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-DEFAULT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-INGRESS
-A WEAVE-NPC -m set ! --match-set weave-local-pods dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-k?Z;25^M}|1s7P3|H9i;*;MhG dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-iuZcey(5DeXbzgRFs8Szo]+#p dst -j ACCEPT
COMMIT
# Completed on Sat Jul 29 04:19:10 2017
# Generated by iptables-save v1.4.21 on Sat Jul 29 04:19:10 2017
*nat
:PREROUTING ACCEPT [2:148]
:INPUT ACCEPT [2:148]
:OUTPUT ACCEPT [24:1452]
:POSTROUTING ACCEPT [24:1452]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-DYFWWILLIC32NPM5 - [0:0]
:KUBE-SEP-GX7UKBANGEPIDZWU - [0:0]
:KUBE-SEP-YXLAFMRH4ZX57Y3W - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
:WEAVE - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A POSTROUTING -j WEAVE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-DYFWWILLIC32NPM5 -s 172.30.0.5/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-DYFWWILLIC32NPM5 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 172.30.0.5:53
-A KUBE-SEP-GX7UKBANGEPIDZWU -s 172.30.0.5/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-GX7UKBANGEPIDZWU -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 172.30.0.5:53
-A KUBE-SEP-YXLAFMRH4ZX57Y3W -s 10.41.30.50/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-YXLAFMRH4ZX57Y3W -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-YXLAFMRH4ZX57Y3W --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.41.30.50:6443
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-DYFWWILLIC32NPM5
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-YXLAFMRH4ZX57Y3W --mask 255.255.255.255 --rsource -j KUBE-SEP-YXLAFMRH4ZX57Y3W
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-YXLAFMRH4ZX57Y3W
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-GX7UKBANGEPIDZWU
-A WEAVE -s 172.30.0.0/16 -d 224.0.0.0/4 -j RETURN
-A WEAVE ! -s 172.30.0.0/16 -d 172.30.0.0/16 -j MASQUERADE
-A WEAVE -s 172.30.0.0/16 ! -d 172.30.0.0/16 -j MASQUERADE
COMMIT
# Completed on Sat Jul 29 04:19:10 2017
# Generated by iptables-save v1.4.21 on Sat Jul 29 04:19:10 2017
*mangle
:PREROUTING ACCEPT [323631:155761830]
:INPUT ACCEPT [323586:155756413]
:FORWARD ACCEPT [26:1880]
:OUTPUT ACCEPT [317539:144236316]
:POSTROUTING ACCEPT [317582:144241373]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Sat Jul 29 04:19:10 2017
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 eno1d1
10.41.30.0 0.0.0.0 255.255.255.0 U 100 0 0 eno1
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.30.0.0 0.0.0.0 255.255.0.0 U 0 0 0 weave
192.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 eno1d1
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0