Get https://[10.96.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.96.0.1:443: i/o timeout] - kubernetes

Question

my pod stucks in ContainerCreating status with this massage :
Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "483590313b7fd092fe5eeec92356152721df3e971d942174464ac5a3f1529898" network for pod "my-nginx": networkPlugin cni failed to set up pod "my-nginx_default" network: CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "483590313b7fd092fe5eeec92356152721df3e971d942174464ac5a3f1529898", failed to clean up sandbox container "483590313b7fd092fe5eeec92356152721df3e971d942174464ac5a3f1529898" network for pod "my-nginx": networkPlugin cni failed to teardown pod "my-nginx_default" network: error getting ClusterInformation: Get https://[10.96.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.96.0.1:443: i/o timeout]
the state of worker node is Ready .
but the output of kubectl get pods -n kube-system seems to have issues :
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-6dfcd885bf-ktbhb 1/1 Running 0 22h
calico-node-4fs2v 0/1 Init:RunContainerError 1 22h
calico-node-l9qvc 0/1 Running 0 22h
coredns-f9fd979d6-8pzcd 1/1 Running 0 23h
coredns-f9fd979d6-v4cq8 1/1 Running 0 23h
etcd-k8s-master 1/1 Running 1 23h
kube-apiserver-k8s-master 1/1 Running 128 23h
kube-controller-manager-k8s-master 1/1 Running 4 23h
kube-proxy-bwtwj 0/1 CrashLoopBackOff 342 23h
kube-proxy-stq7q 1/1 Running 1 23h
kube-scheduler-k8s-master 1/1 Running 4 23h
and the resualt of command kubectl -n kube-system logs kube-proxy-bwtwj the resulst was :
failed to try resolving symlinks in path "/var/log/pods/kube-system_kube-proxy-bwtwj_1a0f4b93-cc6f-46b9-bf29-125feba593cb/kube-proxy/348.log": lstat /var/log/pods/kube-system_kube-proxy-bwtwj_1a0f4b93-cc6f-46b9-bf29-125feba593cb/kube-proxy/348.log: no such file or directory

Answer 1

I see two topics here:
The default --pod-network-cidr for calico is 192.168.0.0/16. You can use a different one but always make sure that there are no overlays. However, I have tested with the default one and my cluster runs with no problems. In order to start over with a proper config, you should Remove the node and Clean up the control plane. Than proceed with:
kubeadm init --pod-network-cidr=192.168.0.0/16
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml
After that, join your worker nodes with kubeadm join
Use sudo where/if needed. All necessary details can be found in the official documentation.
The failed to try resolving symlinks error means that kubelet is looking for the pod logs in a wrong directory. In order to fix it you need to pass the --log-dir=/var/log flag to kubelet. After adding the flag you have run systemctl daemon-reload so the kubelet would be restarted. This has to be done on all of your nodes.

Answer 2

Make sure you deploy calico before joining other nodes to your cluster. When you have other nodes in your cluster calico-kube-controllers sometimes gets push to a worker node. This can lead to issues

Answer 3

You need to carefully check logs for calico-node pods.
In my case i have some other network interfaces and the autodetection mechanism in calico was detecting wrong interface (ip address).
You need to consult this documentation https://projectcalico.docs.tigera.io/reference/node/configuration.
What i did in my case, was simply:
kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=cidr=172.16.8.0/24
cidr is my "working network".
After this, all calico-nodes restarted and suddenly everything was fine.