We have started a cluser with /16 subnet, and flannel as our networking overlay. The pods are getting created on the 2 nodes running sock-shop demo application. But what we are noticing is that pods in different nodes cannot establish connection between them. We do see the routing entries for the pods using flannel.1 interface. Even ping fails. Any pointers to debug information would be appreciated.
You could try to check if the docker bridge ip (--bip= option) is in the same network as flannel interface.
Also you can check ETCD network settings in /coreos.com/network/ with etcdctl command.
Related
I'm trying to launch a prometheus pod in order to scrape the etcd metrics from within our kubernetes cluster.
I was trying to reproduce the solution proposed here: Access etcd metrics for Prometheus
Unfortunately, the etcd containers seem to be unavailable from the cluster.
# nc -vz etcd1 2379
nc: getaddrinfo for host "etcd1" port 2379: Name or service not known
In a way, this seems logical since no etcd container appear in the cluster:
kubectl get pods -A | grep -i etcd does not return anything.
However, when I connect onto the machine hosting the master nodes, I can find the containers using the docker ps command.
The cluster has been deployed using Kubespray.
Do you know if there is a way to reach the etcd containers from the cluster pods?
Duh… the etcd container is configured with the host network. Therefore, the metrics endpoint is directly accessible on the node.
I have 2 different IP sets in the same network. My kubeadm is in a different IP range than my other nodes. How shall I set the property here: kubeadm init --pod-network-cidr=
cat /etc/hosts
#kubernetes slaves ebdp-ch2-d587p.sys.***.net 172.26.0.194, ebdp-ch2-d588p.sys.***.net 172.26.0.195
10.248.43.214 kubemaster
172.26.0.194 kube2
172.26.0.195 kube3
--pod-network-cidr is for IPs of the pods that kubernetes will manage. It is not related with nodes of the cluster.
For nodes, the requirement is (from Kubernetes doc):
Full network connectivity between all machines in the cluster (public
or private network is fine)
In addition to #Yavuz Sert answer, --pod-network-cidr flag identifies Container Network Interface (CNI) IP pool for Pods communication purpose within a Kubernetes cluster. You have to choose some separate IP subnet for Pod networking, it has to be different against your current given network sets. Since --pod-network-cidr has successfully applied kube-proxy reflects Pod IP subnet and add appropriate routes for network communication between Pods through cluster overlay network. Indeed you can find clusterCIDR flag withing kube-proxy configmap which corresponds to --pod-network-cidr.
After initializing kubernetes multimaster cluster I realized that my --service-cluster-ip-range overlaps with actual hosts subnet. A lot of services IPs are overlaping with actual kube node hosts IPs. Now because of that I see a lot of issues in kubedns pods like below:
getsockopt: no route to host
My LAN is: 10.100.0.0/24
Kube service subnet is: 10.96.0.0/12
Now I want to change this in the kube-api pods yamls after removing all the services, but it won't allow me saying that specific section is not a subject o be changed. Is there a way to fix this?
I have a kubernetes master and node setup in two centos VMs on my Win 10.
I used flannel for CNI and deployed ambassador as an API gateway.
As the ambassador routes did not work, I analysed further to understand that the DNS (ip-10.96.0.10) is not accessible from busybox pod which means that none of the service names can be accessed. Could I get any suggestion please.
1. You should use newest version of Flannel.
Flannel does not setup service IPs but kube-proxy does, you should look at kube-proxy on your nodes and ensure they are not reporting errors.
I'd suggest taking a look at https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#tabs-pod-install-4 and ensure you have met the requirements stated there.
Similar issue but with Calico plugin you can find here: https://github.com/projectcalico/calico/issues/1798
2. Check if you have open port 8285, flannel uses UDP port 8285 for sending encapsulated IP packets. Make sure to enable this traffic to pass between the hosts.
3. Ambassador includes an integrated diagnostics service to help with troubleshooting, this may be useful for you. By default, this is not exposed to the Internet. To view it, we'll need to get the name of one of the Ambassador pods:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
ambassador-3655608000-43x86 1/1 Running 0 2m
ambassador-3655608000-w63zf 1/1 Running 0 2m
Forwarding local port 8877 to one of the pods:
kubectl port-forward ambassador-3655608000-43x86 8877
will then let us view the diagnostics at http://localhost:8877/ambassador/v0/diag/.
First spot should solve your problem, if not, try remainings.
I hope this helps.
I've been running my kubernetes masters separate from my kubernetes nodes. So I have kube-apiserver, kube-scheduler and kube-controllermanager running on a server without kubelet, kube-proxy or flannel.
So far this has worked perfectly. However, today I attempted to set up the Web UI and access it through an API server. I got the the following error when accessing http://kube-master-0:8080/ui:
Error: 'dial tcp 172.16.72.12:9090: getsockopt: connection timed out'
Trying to reach: 'http://172.16.72.12:9090/'
This suggests to me that the API server is trying to connect to the pod IP, as we don't have flannel or kube-proxy running on this host, the 172.16.72.12 IP will not be routed.
Am I expected to run kube-proxy and flannel on my API servers? Is there another way to let the API server proxy the UI?
It's not required, but it will certainly make your life easier.
The reason this isn't working is because kube-proxy isn't directing traffic to the service. Try kube-node:8080/ui (assuming you have exposed it as with NodePort configuration
In theory, Kube apiserver does not expect the presence of kube-proxy.
This means kube apiserver will run correctly, receives requests and handles them(mostly reads from and writes to etcd).
But if you want the whole cluster working, you will need other components running, for example:
if you want pods or deployments to be scheduled, kube-scheduler should be running
if you want pods and containers be running in nodes, kubelet has to be running
if you want replications can be guarded, controller-manager should be runing
As for kube-proxy and flannel, they are critical parts to make sure networking is working. Load Balance, service, across-hosts pod communication etc all depends on them.