Pods unable to resolve hostnames in Kubernetes cluster - kubernetes

Question

I'm working on aws eks, and I'm having issues with networking because none of the pods can resolve hostnames.
Checking the kube-config pods, I found this:
NAME READY STATUS RESTARTS AGE
alb-ingress-controller-68b7d9dd9-4mpnc 1/1 Running 0 3d20h
alb-ingress-controller-68b7d9dd9-kqmsl 1/1 Running 8 17d
alb-ingress-controller-68b7d9dd9-q4m87 1/1 Running 11 26d
aws-node-m6ncq 1/1 Running 34 31d
aws-node-r8bs5 1/1 Running 31 28d
aws-node-vfvjn 1/1 Running 34 31d
coredns-f47955f89-4bfb5 1/1 Running 11 26d
coredns-f47955f89-7mqp7 1/1 Running 8 17d
kube-flannel-ds-chv64 0/1 CrashLoopBackOff 1814 28d
kube-flannel-ds-fct45 0/1 CrashLoopBackOff 1831 31d
kube-flannel-ds-zs8z4 0/1 CrashLoopBackOff 1814 28d
kube-proxy-6lcst 1/1 Running 18 31d
kube-proxy-9qfkg 1/1 Running 17 28d
kube-proxy-l5qvd 1/1 Running 18 31d
and kube-flannel logs are this:
I0208 21:01:23.542634 1 main.go:218] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: help:false version:false autoDetectIPv4:false autoDetectIPv6:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true subnetFile:/run/flannel/subnet.env subnetDir: publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 charonExecutablePath: charonViciUri: iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0208 21:01:23.542713 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0208 21:01:23.647398 1 kube.go:120] Waiting 10m0s for node controller to sync
I0208 21:01:23.647635 1 kube.go:378] Starting kube subnet manager
I0208 21:01:24.647775 1 kube.go:127] Node controller sync successful
I0208 21:01:24.647801 1 main.go:238] Created subnet manager: Kubernetes Subnet Manager - ip-10-23-21-240.us-east-2.compute.internal
I0208 21:01:24.647811 1 main.go:241] Installing signal handlers
I0208 21:01:24.647887 1 main.go:460] Found network config - Backend type: vxlan
I0208 21:01:24.647916 1 main.go:652] Determining IP address of default interface
I0208 21:01:24.648449 1 main.go:699] Using interface with name eth0 and address 10.23.21.240
I0208 21:01:24.648476 1 main.go:721] Defaulting external address to interface address (10.23.21.240)
I0208 21:01:24.648482 1 main.go:734] Defaulting external v6 address to interface address (<nil>)
I0208 21:01:24.648543 1 vxlan.go:137] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
E0208 21:01:24.648871 1 main.go:326] Error registering network: failed to acquire lease: node "ip-10-23-21-240.us-east-2.compute.internal" pod cidr not assigned
W0208 21:01:24.649006 1 reflector.go:424] github.com/flannel-io/flannel/subnet/kube/kube.go:379: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
I0208 21:01:24.649060 1 main.go:440] Stopping shutdownHandler...
Any idea what could be the issue or what to try?
Thanks!

Answer 1

Amazon VPC CNI and Flannel cannot co-exist on EKS. Note Flannel is not on the suggested alternate compatible CNI. To get an idea what does it take to use Flannel on EKS checkout this excellent blog.