what is ` i/o timeout` issue in kubernete pods? - kubernetes

Question

I deployed a kubernete cluster with two EC2 instances.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-172-31-21-12 Ready control-plane,master 36h v1.20.2
ip-172-31-21-62 Ready <none> 12h v1.20.2
There is an error pod in the cluster:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
es-cluster-0 0/1 Error 141 12h
I see this error in the log of this pod:
$ kubectl logs pods/es-cluster-0
Error from server: Get "https://172.31.21.62:10250/containerLogs/default/es-cluster-0/elasticsearch": dial tcp 172.31.21.62:10250: i/o timeout
It is talking about having error with the node 172.31.21.62:10250. how can I fix the issue? I am not sure that this error mean?
Below command is from master node:
kubectl describe pods/es-cluster-0 command output:
$ kubectl describe pods/es-cluster-0
Name: es-cluster-0
Namespace: default
Priority: 0
Node: ip-172-31-21-62/172.31.21.62
Start Time: Wed, 17 Feb 2021 10:08:04 +0000
Labels: controller-revision-hash=es-cluster-55b6944c56
name=elasticsearch
statefulset.kubernetes.io/pod-name=es-cluster-0
Annotations: <none>
Status: Running
IP: 10.32.0.2
IPs:
IP: 10.32.0.2
Controlled By: StatefulSet/es-cluster
Containers:
elasticsearch:
Container ID: docker://838e02ff6fba31234656e68b804f49e86ec7fea0053e5d1062abdd9d24b728b9
Image: elasticsearch:7.10.1
Image ID: docker-pullable://elasticsearch#sha256:7cd88158f6ac75d43b447fdd98c4eb69483fa7bf1be5616a85fe556262dc864a
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 78
Started: Thu, 18 Feb 2021 01:40:23 +0000
Finished: Thu, 18 Feb 2021 01:40:39 +0000
Ready: False
Restart Count: 178
Environment: <none>
Mounts:
/usr/share/elasticsearch/config/elasticsearch.yml from elasticsearch-config (rw,path="elasticsearch.yml")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ntgwr (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
elasticsearch-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: es-config
Optional: false
default-token-ntgwr:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-ntgwr
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 42s (x4082 over 15h) kubelet Back-off restarting failed container
Below command is from master node:
$ netstat -tunlp| grep 10250
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::10250 :::* LISTEN -
Below command is from worker node:
$ kubectl describe pods/es-cluster-0
Name: es-cluster-0
Namespace: default
Priority: 0
Node: ip-172-31-21-62/172.31.21.62
Start Time: Wed, 17 Feb 2021 10:08:04 +0000
Labels: controller-revision-hash=es-cluster-55b6944c56
name=elasticsearch
statefulset.kubernetes.io/pod-name=es-cluster-0
Annotations: <none>
Status: Running
IP: 10.32.0.2
IPs:
IP: 10.32.0.2
Controlled By: StatefulSet/es-cluster
Containers:
elasticsearch:
Container ID: docker://0efa71da7b70725b248cfe8dfae5560c2fc95aaf40e3a3a899970e8579e7ac27
Image: elasticsearch:7.10.1
Image ID: docker-pullable://elasticsearch#sha256:7cd88158f6ac75d43b447fdd98c4eb69483fa7bf1be5616a85fe556262dc864a
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 78
Started: Thu, 18 Feb 2021 05:31:55 +0000
Finished: Thu, 18 Feb 2021 05:32:12 +0000
Ready: False
Restart Count: 221
Environment: <none>
Mounts:
/usr/share/elasticsearch/config/elasticsearch.yml from elasticsearch-config (rw,path="elasticsearch.yml")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ntgwr (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
elasticsearch-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: es-config
Optional: false
default-token-ntgwr:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-ntgwr
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 55m (x211 over 19h) kubelet Container image "elasticsearch:7.10.1" already present on machine
Warning BackOff 54s (x5087 over 19h) kubelet Back-off restarting failed container
Below command is from worker node:
$ netstat -tunlp| grep 10250
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::10250 :::* LISTEN -