Can't join cluster: connection refused during kubeadm join - kubernetes

Question

I am getting the following error when I try to join the cluster
sudo kubeadm join 10.1.1.150:6443 --token ypcdg7.w6pun0nd31c4q5c2 --discovery-token-ca-cert-hash sha256:1ac79447f8dee9d90d592a3ead3d6c54ce7046dcd0b3854917c93cf6bbff7894
[preflight] running pre-flight checks
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support
I0710 13:39:56.080260 2415 kernel_validator.go:81] Validating kernel version
I0710 13:39:56.080423 2415 kernel_validator.go:96] Validating kernel config
[discovery] Trying to connect to API Server "10.1.1.150:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.1.1.150:6443"
[discovery] Failed to request cluster info, will try again: [Get https://10.1.1.150:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.1.1.150:6443: connect: connection refused]
I get the same error with curl on the machine that I want to join the cluster.
But if I run this
curl -k https://10.1.1.150:6443/api/v1/namespaces/kube-public/configmaps/cluster-info
in any machine in the local net (10.1.1.0/24) I get a good JSON response.
Some usefull info:
I can ping my master node (10.1.1.150)
6443 is open on my master node, this is netstat on my master
jp#tensor3:~$ sudo netstat -tulpn
Connessioni Internet attive (solo server)
Proto CodaRic CodaInv Indirizzo locale Indirizzo remoto Stato PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1110/sshd
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 6783/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 7375/kube-proxy
tcp 0 0 127.0.0.1:10251 0.0.0.0:* LISTEN 7120/kube-scheduler
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 7065/etcd
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 7058/kube-controlle
tcp 0 0 127.0.0.1:2380 0.0.0.0:* LISTEN 7065/etcd
tcp 0 0 127.0.0.1:37491 0.0.0.0:* LISTEN 6783/kubelet
tcp6 0 0 :::22 :::* LISTEN 1110/sshd
tcp6 0 0 :::10250 :::* LISTEN 6783/kubelet
tcp6 0 0 :::6443 :::* LISTEN 7130/kube-apiserver
tcp6 0 0 :::10256 :::* LISTEN 7375/kube-proxy
udp 0 0 0.0.0.0:68 0.0.0.0:* 980/dhclient
udp 0 0 0.0.0.0:8472 0.0.0.0:* -
I'm using flannel for network communication
ifconfig and route on my master
jp#tensor3:~$ ifconfig -a
cni0 Link encap:Ethernet IndirizzoHW 0a:58:0a:f4:00:01
indirizzo inet:10.244.0.1 Bcast:0.0.0.0 Maschera:255.255.255.0
indirizzo inet6: fe80::ec67:aeff:fe90:4fbc/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:31641 errors:0 dropped:0 overruns:0 frame:0
TX packets:31925 errors:0 dropped:0 overruns:0 carrier:0
collisioni:0 txqueuelen:1000
Byte RX:2032440 (2.0 MB) Byte TX:11726575 (11.7 MB)
docker0 Link encap:Ethernet IndirizzoHW 02:42:44:f4:4f:cb
indirizzo inet:172.17.0.1 Bcast:0.0.0.0 Maschera:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisioni:0 txqueuelen:0
Byte RX:0 (0.0 B) Byte TX:0 (0.0 B)
ens160 Link encap:Ethernet IndirizzoHW 00:50:56:3d:73:84
indirizzo inet:10.1.1.150 Bcast:10.1.1.255 Maschera:255.255.255.0
indirizzo inet6: fe80::250:56ff:fe3d:7384/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:294081 errors:0 dropped:0 overruns:0 frame:0
TX packets:74724 errors:0 dropped:0 overruns:0 carrier:0
collisioni:0 txqueuelen:1000
Byte RX:297445831 (297.4 MB) Byte TX:5900453 (5.9 MB)
flannel.1 Link encap:Ethernet IndirizzoHW 06:a7:4c:1d:6f:cd
indirizzo inet:10.244.0.0 Bcast:0.0.0.0 Maschera:255.255.255.255
indirizzo inet6: fe80::4a7:4cff:fe1d:6fcd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:8 overruns:0 carrier:0
collisioni:0 txqueuelen:0
Byte RX:0 (0.0 B) Byte TX:0 (0.0 B)
lo Link encap:Loopback locale
indirizzo inet:127.0.0.1 Maschera:255.0.0.0
indirizzo inet6: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:1219503 errors:0 dropped:0 overruns:0 frame:0
TX packets:1219503 errors:0 dropped:0 overruns:0 carrier:0
collisioni:0 txqueuelen:1
Byte RX:271937625 (271.9 MB) Byte TX:271937625 (271.9 MB)
vethd451ffcc Link encap:Ethernet IndirizzoHW 8a:f1:f0:05:80:f3
indirizzo inet6: fe80::88f1:f0ff:fe05:80f3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:8799 errors:0 dropped:0 overruns:0 frame:0
TX packets:8860 errors:0 dropped:0 overruns:0 carrier:0
collisioni:0 txqueuelen:0
Byte RX:687962 (687.9 KB) Byte TX:3256670 (3.2 MB)
vethed2073fb Link encap:Ethernet IndirizzoHW ea:9e:43:5e:4e:30
indirizzo inet6: fe80::e89e:43ff:fe5e:4e30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:8800 errors:0 dropped:0 overruns:0 frame:0
TX packets:8887 errors:0 dropped:0 overruns:0 carrier:0
collisioni:0 txqueuelen:0
Byte RX:688228 (688.2 KB) Byte TX:3258477 (3.2 MB)
jp#tensor3:~$ route -n
Tabella di routing IP del kernel
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.1.1.29 0.0.0.0 UG 0 0 0 ens160
10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens160
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
ifconfig and route on my node
jp#tensor2:~$ ifconfig
docker0 Link encap:Ethernet IndirizzoHW 02:42:4e:2f:0e:97
indirizzo inet:172.17.0.1 Bcast:0.0.0.0 Maschera:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisioni:0 txqueuelen:0
Byte RX:0 (0.0 B) Byte TX:0 (0.0 B)
ens160 Link encap:Ethernet IndirizzoHW 00:50:56:8c:0a:cd
indirizzo inet:10.1.1.151 Bcast:10.1.1.255 Maschera:255.255.255.0
indirizzo inet6: fe80::250:56ff:fe8c:acd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:17470 errors:0 dropped:0 overruns:0 frame:0
TX packets:913 errors:0 dropped:0 overruns:0 carrier:0
collisioni:0 txqueuelen:1000
Byte RX:1511461 (1.5 MB) Byte TX:101801 (101.8 KB)
lo Link encap:Loopback locale
indirizzo inet:127.0.0.1 Maschera:255.0.0.0
indirizzo inet6: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:160 errors:0 dropped:0 overruns:0 frame:0
TX packets:160 errors:0 dropped:0 overruns:0 carrier:0
collisioni:0 txqueuelen:1
Byte RX:11840 (11.8 KB) Byte TX:11840 (11.8 KB)
jp#tensor2:~$ route -n
Tabella di routing IP del kernel
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.1.1.29 0.0.0.0 UG 0 0 0 ens160
10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens160
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
and this is the status of my cluster
jp#tensor3:~$ sudo kubectl get all --namespace=kube-system
[sudo] password di jp:
NAME READY STATUS RESTARTS AGE
pod/coredns-78fcdf6894-bblcs 1/1 Running 0 1h
pod/coredns-78fcdf6894-wrmj4 1/1 Running 0 1h
pod/etcd-tensor3 1/1 Running 0 1h
pod/kube-apiserver-tensor3 1/1 Running 0 1h
pod/kube-controller-manager-tensor3 1/1 Running 0 1h
pod/kube-flannel-ds-amd64-p7jmq 1/1 Running 0 1h
pod/kube-proxy-qg7jj 1/1 Running 0 1h
pod/kube-scheduler-tensor3 1/1 Running 0 1h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 1h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/kube-flannel-ds-amd64 1 1 1 1 1 beta.kubernetes.io/arch=amd64 1h
daemonset.apps/kube-flannel-ds-arm 0 0 0 0 0 beta.kubernetes.io/arch=arm 1h
daemonset.apps/kube-flannel-ds-arm64 0 0 0 0 0 beta.kubernetes.io/arch=arm64 1h
daemonset.apps/kube-flannel-ds-ppc64le 0 0 0 0 0 beta.kubernetes.io/arch=ppc64le 1h
daemonset.apps/kube-flannel-ds-s390x 0 0 0 0 0 beta.kubernetes.io/arch=s390x 1h
daemonset.apps/kube-proxy 1 1 1 1 1 beta.kubernetes.io/arch=amd64 1h
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/coredns 2 2 2 2 1h
NAME DESIRED CURRENT READY AGE
replicaset.apps/coredns-78fcdf6894 2 2 2 1h
On the same machine I'm using:
docker version 1.13.1
k8s version 1.11

Answer 1

I met the similar behaviour on Ubuntu 16.04 after reboot.
I did
sudo kubeadm join ...
It crashed like described.
Then I did:
sudo su
kubeadm init
exit
I tried again and that second time it was successful.