Pods not getting scheduled to node with matching labels - kubernetes

Question

I'm getting this error when exec'ing into my pod. Error from server (BadRequest): pod es-master-5cb49c68cc-w6dxv does not have a host assigned
It seemed to be related to my nodeAffinity but I don't see anything immediately wrong with it. I can't seem to get my deployment to attach its pod to any of my nodes. I don't have any taints or tolerations setup on the node or pod. I've tried switching to labels that are automatically generated that are on every node, but nothing seems to work. I've even tried removing my affinity section entirely, and also tried adding nodeSelector to spec by itself.
Here is my deployment config and output from kubectl describe pod -n elasticsearch
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: elasticsearch
role: master
name: es-master
namespace: elasticsearch
spec:
replicas: 3
selector:
matchLabels:
component: elasticsearch
role: master
template:
metadata:
labels:
component: elasticsearch
role: master
annotations:
iam.amazonaws.com/role: {REDACTED}
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
component: elasticsearch
role: master
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/region
operator: In
values:
- us-east-2
Name: es-master-866f7fb558-298ht
Namespace: elasticsearch
Priority: 0
Node: <none>
Labels: component=elasticsearch
pod-template-hash=866f7fb558
role=master
Annotations: iam.amazonaws.com/role: {REDACTED}
kubernetes.io/psp: eks.privileged
Status: Pending
IP:
Controlled By: ReplicaSet/es-master-866f7fb558
Init Containers:
init-sysctl:
Image: busybox:1.27.2
Port: <none>
Host Port: <none>
Command:
sysctl
-w
vm.max_map_count=262144
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xflv6 (ro)
Containers:
elasticsearch:
Image: amazon/opendistro-for-elasticsearch:0.9.0
Ports: 9300/TCP, 9200/TCP, 9600/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Limits:
cpu: 2
memory: 12Gi
Requests:
cpu: 2
memory: 12Gi
Liveness: tcp-socket :transport delay=60s timeout=1s period=10s #success=1 #failure=3
Environment:
CLUSTER_NAME: logs
NUMBER_OF_MASTERS: 3
NODE_MASTER: true
NODE_INGEST: false
NODE_DATA: false
NETWORK_HOST: 0.0.0.0
TRANSPORT_TLS_PEM_PASS:
HTTP_TLS_PEM_PASS:
NODE_NAME: es-master-866f7fb558-298ht (v1:metadata.name)
DISCOVERY_SERVICE: elasticsearch-discovery
KUBERNETES_NAMESPACE: elasticsearch (v1:metadata.namespace)
PROCESSORS: 2 (limits.cpu)
ES_JAVA_OPTS: -Xms6g -Xmx6g
Mounts:
/usr/share/elasticsearch/config/admin-crt.pem from certs (ro,path="admin-crt.pem")
/usr/share/elasticsearch/config/admin-key.pem from certs (ro,path="admin-key.pem")
/usr/share/elasticsearch/config/admin-root-ca.pem from certs (ro,path="admin-root-ca.pem")
/usr/share/elasticsearch/config/elasticsearch.yml from config (rw,path="elasticsearch.yml")
/usr/share/elasticsearch/config/elk-crt.pem from certs (ro,path="elk-crt.pem")
/usr/share/elasticsearch/config/elk-key.pem from certs (ro,path="elk-key.pem")
/usr/share/elasticsearch/config/elk-root-ca.pem from certs (ro,path="elk-root-ca.pem")
/usr/share/elasticsearch/config/logging.yml from config (rw,path="logging.yml")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xflv6 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: elasticsearch
Optional: false
certs:
Type: Secret (a volume populated by a Secret)
SecretName: elasticsearch-tls-data
Optional: false
default-token-xflv6:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xflv6
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 59s (x3 over 3m44s) default-scheduler 0/8 nodes are available: 8 Insufficient cpu.
All nodes are m5a.large ec2 instances.

Answer 1

The error is pretty clear 0/8 nodes are available: 8 Insufficient cpu which means nodes don't have 2 cpu cores free as specified in requests. Solution is to either provision nodes with more cpu or reduce the cpu requests in pod spec.