Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 18m (x145 over 3h19m) default-scheduler 0/2 nodes are available: 1 node(s) didn't match Pod's node affinity, 1 node(s) had taint {node-role.kubernetes.io/controlplane: true}, that the pod didn't tolerate.
Please provide me a solution to deploy the pods on worker server.
It seems that your pod is not getting scheduled to a Node. Can you try to run the below command ?
kubectl taint nodes <name-node-master> node-role.kubernetes.io/control-plane:NoSchedule-
To find the name of your node please use
kubectl get nodes
Related
I have created a new EKS cluster with 1 worker node in a public subnet. I am able to query node, connect to the cluster, and run pod creation command, however, when I am trying to create a pod it fails with the below error got by describing the pod. Please guide.
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 81s default-scheduler 0/1 nodes are available: 1 Too many pods. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
Warning FailedScheduling 16m default-scheduler 0/2 nodes are available: 2 Too many pods, 2 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }, 2 node(s) were unschedulable. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
Warning FailedScheduling 16m default-scheduler 0/3 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }, 2 node(s) were unschedulable, 3 Too many pods. preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling.
Warning FailedScheduling 14m (x3 over 22m) default-scheduler 0/2 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }, 1 node(s) were unschedulable, 2 Too many pods. preemption: 0/2 nodes are available: 1 No preemption victims found for incoming pod, 1 Preemption is not helpful for scheduling.
Warning FailedScheduling 12m default-scheduler 0/2 nodes are available: 1 Too many pods, 2 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }, 2 node(s) were unschedulable. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
Warning FailedScheduling 7m14s default-scheduler no nodes available to schedule pods
Warning FailedScheduling 105s (x5 over 35m) default-scheduler 0/1 nodes are available: 1 Too many pods. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
I am able to get status of the node and it looks ready:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-12-61.ec2.internal Ready <none> 15m v1.24.7-eks-fb459a0
While troubleshooting I tried below options:
recreate the complete demo cluster - still the same error
try recreating pods with different images - still the same error
trying to increase to instance type to t3.micro - still the same error
reviewed security groups and other parameters in a cluster - Couldnt come to RCA
it's due to the node's POD limit or IP limit on Nodes.
So if we see official Amazon doc, t3.micro maximum 2 interface you can use and 2 private IP. Roughly you might be getting around 4 IPs to use and 1st IP get used by Node etc, There will be also default system PODs running as Daemon set and so.
Add new instance or upgrade to larger instance who can handle more pods.
When running this command for resources ( deployment, ReplicaSet ...) other than Pod
$ kubectl describe deployment xxx-deployment
---- ------ ------
Events: <none>
I have deployed several resources, but I haven't seen the event yet except for Pod.
What type of event will occur if events occur in other resources?
Could you recommend any materials to refer to?
Good explanation what is event in Kubernetes you can find in Types of Kubernetes Events article. Author also mentioned about types of events.
Kubernetes events are a resource type in Kubernetes that are automatically created when other resources have state changes, errors, or other messages that should be broadcast to the system. While there is not a lot of documentation available for events, they are an invaluable resource when debugging issues in your Kubernetes cluster.
You can describe not only pod, deployment or replicaset but almost all resources in kubernetes.
Examples:
kubectl describe job pi -n test
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 12s job-controller Created pod: pi-5rgbz
kubectl describe node ubuntu
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning MissingClusterDNS 22h (x98 over 23h) kubelet, ubuntu-18 kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
Normal Starting 22h kubelet, ubuntu-18 Starting kubelet.
Warning InvalidDiskCapacity 22h kubelet, ubuntu-18 invalid capacity 0 on image filesystem
Normal NodeHasSufficientMemory 22h kubelet, ubuntu-18 Node ubuntu-18 status is now: NodeHasSufficientMemory
Normal NodeHasSufficientPID 22h
To list all resources events you can use
$ kubectl get events --all-namespaces
$ kubectl get events --all-namespaces
NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE
default 50m Normal Starting node/gke-cluster-1-default-pool-XXXXXXXXXXXXX Starting kubelet.
default 50m Normal NodeHasSufficientMemory node/gke-cluster-1-default-pool-XXXXXXXXXXXXX Node gke-cluster-1-default-pool-XXXXXXXXXXXXX status is now: NodeHasSufficientMemory
default 2m47s Normal SuccessfulCreate job/pi Created pod: pi-5rgbz
kube-system 50m Normal ScalingReplicaSet deployment/fluentd-gcp-scaler Scaled up replica set fluentd-gcp-scaler-6855f55bcc to 1
In Object column you resource type.
If you would like more detailed information you can use -o wide flag - $ kubectl get events --all-namespaces -o wide
$ kubectl get events -o wide
LAST SEEN TYPE REASON OBJECT SUBOBJECT SOURCE MESSAGE
FIRST SEEN COUNT NAME
20m Normal Scheduled pod/hello-world-86d6c6f84d-8qz9d default-scheduler Successfully assigned default/hello-world-86d
6c6f84d-8qz9d to ubuntu-18
Possibly root cause.
I wasn't able to create deployment without any event at the beginning I would guess that you have set --event-ttl which is described in Kube-apiserver docs.
--event-ttl duration Default: 1h0m0s
Amount of time to retain events.
It was also mentioned in Github thread.
In short, all events will disappear after 1 hour if you have this flag set.
To check if you have this flag set in kube-apiserver you can check this StackOverflow thread.
If this didn't help you please edit your question with informations like your configuration YAMLs, what version of K8s are you using, steps to reproduce etc.
Well yes deployment do have events. But keep that in mind events only available for around 1 hr.
you can also filter by labels with --labelsfor describe all resources
When scheduling a kubernetes Job and Pod, if the Pod can't be placed the explanation available from kubectl describe pods PODNAME looks like:
Warning FailedScheduling <unknown> default-scheduler 0/172 nodes are available:
1 Insufficient pods, 1 node(s) were unschedulable, 11 Insufficient memory,
30 Insufficient cpu, 32 node(s) didn't match node selector, 97 Insufficient nvidia.com/gpu.
That's useful but a little too vague. I'd like more detail than that.
Specifically can I list all nodes with the reason the pod wasn't scheduled to each particular node?
I was recently changing labels and the node selector and want to determine if I made a mistake somewhere in that process or if the nodes I need really are just busy.
You can find more details related to problems with scheduling particular Pod in kube-scheduler logs. If you set up your cluster with kubeadm tool, kube-scheduler as well as other key components of the cluster is deployed as a system Pod. You can list such Pods with the following command:
kubectl get pods -n kube-system
which will show you among others your kube-scheduler Pod:
NAME READY STATUS RESTARTS AGE
kube-scheduler-master-ubuntu-18-04 1/1 Running 0 2m37s
Then you can check its logs. In my example the command will look as follows:
kubectl logs kube-scheduler-master-ubuntu-18-04 -n kube-system
You should find there the information you need.
One more thing...
If you've already verified it, just ignore this tip
Let's start from the beginning...
I've just created a simple job from the example you can find here:
kubectl apply -f https://k8s.io/examples/controllers/job.yaml
job.batch/pi created
If I run:
kubectl get jobs
it shows me:
NAME COMPLETIONS DURATION AGE
pi 0/1 17m 17m
Hmm... completions 0/1 ? Something definitely went wrong. Let's check it.
kubectl describe job pi
tells me basically nothing. In it's events I can see only:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 18m job-controller Created pod: pi-zxp4p
as if everything went well... but we already know it didn't. So let's investigate further. As you probably know, job-controller creates Pods that run to completion to perform certain task. From the perspective of the job-controller everything went well (we've just seen it in it's events):
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 23m job-controller Created pod: pi-zxp4p
It did it's part of the task and reported that everything went fine. But it's just part of the whole task. It passed actual Pod creation task further to the kube-scheduler controller as being just a job-controller it isn't responsible (and doesn't even have enough privileges) to schedule the actual Pod on particular node. If we run:
kubectl get pods
we can see one Pod in a Pending state:
NAME READY STATUS RESTARTS AGE
pi-zxp4p 0/1 Pending 0 30m
Let's describe it:
kubectl describe pod pi-zxp4p
In events we can see some very important and specific info:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 20s (x24 over 33m) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
so now we know the actual reason why our Pod couldn't be scheduled.
Pay attention to different fields of the event:
From: default-scheduler - it means that the message was originated from our kube-scheduler.
Type: Warning, which isn't as important as Critical or Error so chances are that it may not appear in kube-scheduler logs if the last one was started with the default level of log verbosity.
You can read here that:
As per the comments, the practical default level is V(2). Developers
and QE environments may wish to run at V(3) or V(4). If you wish to
change the log level, you can pass in -v=X where X is the desired
maximum level to log.
I have the micro cluster on GCloud running smallest imaginable Node.js application.
Eventhough app is going down all the time. Here is output of kubernate describe pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 1m (x2 over 1m) default-scheduler 0/3 nodes are available: 1 Insufficient cpu, 2 node(s) were not ready, 2 node(s) were out of disk space.
Warning FailedScheduling 55s (x9 over 2m) default-scheduler 0/3 nodes are available: 1 node(s) were not ready, 1 node(s) were out of disk space, 2 Insufficient cpu.
I suspect maybe there is some garbage left from previous versions of an app. How can I check what causes problems and get rid of unnecessary processes and data?
It seems a deployment has gotten stuck. How can I diagnose this further?
kubectl rollout status deployment/wordpress
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
It's stuck on that for ages already. It is not terminating the two older pods:
kubectl get pods
NAME READY STATUS RESTARTS AGE
nfs-server-r6g6w 1/1 Running 0 2h
redis-679c597dd-67rgw 1/1 Running 0 2h
wordpress-64c944d9bd-dvnwh 4/4 Running 3 3h
wordpress-64c944d9bd-vmrdd 4/4 Running 3 3h
wordpress-f59c459fd-qkfrt 0/4 Pending 0 22m
wordpress-f59c459fd-w8c65 0/4 Pending 0 22m
And the events:
kubectl get events --all-namespaces
NAMESPACE LAST SEEN FIRST SEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
default 25m 2h 333 wordpress-686ccd47b4-4pbfk.153408cdba627f50 Pod Warning FailedScheduling default-scheduler No nodes are available that match all of the predicates: Insufficient cpu (1), Insufficient memory (2), MatchInterPodAffinity (1).
default 25m 2h 337 wordpress-686ccd47b4-vv9dk.153408cc8661c49d Pod Warning FailedScheduling default-scheduler No nodes are available that match all of the predicates: Insufficient cpu (1), Insufficient memory (2), MatchInterPodAffinity (1).
default 22m 22m 1 wordpress-686ccd47b4.15340e5036ef7d1c ReplicaSet Normal SuccessfulDelete replicaset-controller Deleted pod: wordpress-686ccd47b4-4pbfk
default 22m 22m 1 wordpress-686ccd47b4.15340e5036f2fec1 ReplicaSet Normal SuccessfulDelete replicaset-controller Deleted pod: wordpress-686ccd47b4-vv9dk
default 2m 22m 72 wordpress-f59c459fd-qkfrt.15340e503bd4988c Pod Warning FailedScheduling default-scheduler No nodes are available that match all of the predicates: Insufficient cpu (1), Insufficient memory (2), MatchInterPodAffinity (1).
default 2m 22m 72 wordpress-f59c459fd-w8c65.15340e50399a8a5a Pod Warning FailedScheduling default-scheduler No nodes are available that match all of the predicates: Insufficient cpu (1), Insufficient memory (2), MatchInterPodAffinity (1).
default 22m 22m 1 wordpress-f59c459fd.15340e5039d6c622 ReplicaSet Normal SuccessfulCreate replicaset-controller Created pod: wordpress-f59c459fd-w8c65
default 22m 22m 1 wordpress-f59c459fd.15340e503bf844db ReplicaSet Normal SuccessfulCreate replicaset-controller Created pod: wordpress-f59c459fd-qkfrt
default 3m 23h 177 wordpress.1533c22c7bf657bd Ingress Normal Service loadbalancer-controller no user specified default backend, using system default
default 22m 22m 1 wordpress.15340e50356eaa6a Deployment Normal ScalingReplicaSet deployment-controller Scaled down replica set wordpress-686ccd47b4 to 0
default 22m 22m 1 wordpress.15340e5037c04da6 Deployment Normal ScalingReplicaSet deployment-controller Scaled up replica set wordpress-f59c459fd to 2
You can use describe kubectl describe po wordpress-f59c459fd-qkfrt but from the message the pods cannot be scheduled in any of the nodes.
Provide more capacity, like try to add a node, to allow the pods to be scheduled.
The new deployment had a replica count of 3 while the previous had 2. I assumed I could set a high value for replica count and it would try to deploy as many replicas as it could before it reaches it's resource capacity. However this does not seem to be the case...