Why Kubernetes cluster on VM instances Google Cloud can't connect host with NodePort? - kubernetes

Question

I have installed a kubernetes cluster using this tutorial.
When I've set it up on VM Virtual Box - my host can connect with NodePort normally. When I've tried it on Compute Engine Virtual Machine instance, the Kubernetes cluster can't connect host with NodePort?
I have attached two pictures.
Thank you for your support.
Kubernetes cluster (bare metal) on Local VM Virtual Box
Kubernetes cluster (bare metal) on Google cloud Platform VM Instances

Answer 1

This took me a while to test but I finally have a result. So it turns out the reason of your issue is Calico and GCP firewall. To be more specific you have to add firewall rules before you can be successful with the connectivity.
Following this document on installing Calico for GCE:
GCE blocks traffic between hosts by default; run the following command
to allow Calico traffic to flow between containers on different hosts
(where the source-ranges parameter assumes you have created your
project with the default GCE network parameters - modify the address
range if yours is different):
So you need to allow the traffic to flow between containers:
gcloud compute firewall-rules create calico-ipip --allow 4 --network "default" --source-ranges "10.128.0.0/9"
Note that this IP should be changed. You can use for test purposes 10.0.0.0/8 but this is way to wide range so please narrow it down to your needs.
Then proceed with setting up instances for master and nodes.
You can actually skip most of the steps from the tutorial you posted, as connectivity is resolved by cloud provider. Here is a really simple script I use for Kubeadm on VM's. You can also perform this step by step.
#!/bin/bash
swapoff -a
echo net/bridge/bridge-nf-call-ip6tables = 1 >> /etc/ufw/sysctl.conf
echo net/bridge/bridge-nf-call-iptables = 1 >> /etc/ufw/sysctl.conf
echo net/bridge/bridge-nf-call-arptables = 1 >> /etc/ufw/sysctl.conf
apt-get install -y ebtables ethtool
apt-get update
apt-get install -y docker.io
apt-get install -y apt-transport-https
apt-get install -y curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
kubeadm init --pod-network-cidr=192.168.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
kubectl taint nodes --all node-role.kubernetes.io/master-
In my case I used simple Redis application from Kubernetes documentation
root#calico-master:/home/xxx# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29m
redis-master ClusterIP 10.107.41.117 <none> 6379/TCP 26m
root#calico-master:/home/xxx# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
redis-master-57fc67768d-5lx92 1/1 Running 0 27m 192.168.1.4 calico <none>
root#calico-master:/home/xxx# ping 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=63 time=1.48 ms
Before the firewall rules and regular Calico installation I was not able to ping, nor wgetfrom the service after that there is no problem with pinging the IP or hostname and also wget works:
> root#calico-master:/home/xxx# wget http://10.107.41.117:6379
> --2018-10-24 13:24:43-- http://10.107.41.117:6379/ Connecting to 10.107.41.117:6379... connected. HTTP request sent, awaiting response... 200 No headers, assuming HTTP/0.9 Length: unspecified
> Saving to: ‘index.html.2’
Steps above were also tested for type: NodePort and it works as well.
Another way is to use Flannel which I also tested and it worked out of the box for the needs of testing your issue. Be sure to read more about CNI’s so you can choose one that will suit your needs.
Hope this solves your problem.

Answer 2

That's because in minikube there's only one node for everything and the VM is that node. So if you are in the VM you can connect to the NodePort locally or on localhost.
In the case of GCP, you don't usually run work the master(s) and nodes on the same VMs. So you need to will get a reply from one of the nodes (VMs) where your pod is listening.
To get the list of nodes on your cluster you can simply run:
kubectl get nodes -o=wide
You should see for example an Internal IP for your nodes. Then you can try
curl http://<Internal IP>:<NodePort>

Answer 3

You can also get the IP details using,
kubectl describe nodes
Then ping trying to connect to your host using the corresponding IP.