kubernetes UnexpectedAdmissionError after rollout - kubernetes

Question

I had a service failing to reply to some HTTP requests, digging it's logs it seemed to be some sort of DNS failure on reaching a proxy service
'proxy' failed to resolve 'proxy.default.svc.cluster.local' after 2 queries
So I could not find anything wrong and tried kubectl rollout restart deployment/backend.
Just after that these appeared in the pods list:
backend-54769cbb4-xkwf2 0/1 UnexpectedAdmissionError 0 4h6m
backend-54769cbb4-xlpgf 0/1 UnexpectedAdmissionError 0 4h4m
backend-54769cbb4-xmnr5 0/1 UnexpectedAdmissionError 0 4h7m
backend-54769cbb4-xmq5n 0/1 UnexpectedAdmissionError 0 4h7m
backend-54769cbb4-xphrw 0/1 UnexpectedAdmissionError 0 4h5m
backend-54769cbb4-xrmrq 0/1 UnexpectedAdmissionError 0 4h1m
backend-54769cbb4-xrmw8 0/1 UnexpectedAdmissionError 0 4h6m
backend-54769cbb4-xt4ck 0/1 UnexpectedAdmissionError 0 4h4m
backend-54769cbb4-xws8r 0/1 UnexpectedAdmissionError 0 4h6m
backend-54769cbb4-xx6r4 0/1 UnexpectedAdmissionError 0 4h5m
backend-54769cbb4-xxpfd 0/1 UnexpectedAdmissionError 0 4h6m
backend-54769cbb4-xzjql 0/1 UnexpectedAdmissionError 0 4h4m
backend-54769cbb4-xzzlk 0/1 UnexpectedAdmissionError 0 4h7m
backend-54769cbb4-z46ms 0/1 UnexpectedAdmissionError 0 4h5m
backend-54769cbb4-z4sl7 0/1 UnexpectedAdmissionError 0 4h6m
backend-54769cbb4-z6jpj 0/1 UnexpectedAdmissionError 0 4h5m
backend-54769cbb4-z6ngq 0/1 UnexpectedAdmissionError 0 4h6m
backend-54769cbb4-z8w4h 0/1 UnexpectedAdmissionError 0 4h5m
backend-54769cbb4-z9jqb 0/1 UnexpectedAdmissionError 0 4h3m
backend-54769cbb4-zbvqm 0/1 UnexpectedAdmissionError 0 4h2m
backend-54769cbb4-zcfxg 0/1 UnexpectedAdmissionError 0 4h3m
backend-54769cbb4-zcvqm 0/1 UnexpectedAdmissionError 0 4h6m
backend-54769cbb4-zf2f8 0/1 UnexpectedAdmissionError 0 4h2m
backend-54769cbb4-zgnkh 0/1 UnexpectedAdmissionError 0 4h7m
backend-54769cbb4-zhdr8 0/1 UnexpectedAdmissionError 0 4h2m
backend-54769cbb4-zhx6g 0/1 UnexpectedAdmissionError 0 4h7m
backend-54769cbb4-zj8f2 0/1 UnexpectedAdmissionError 0 4h3m
backend-54769cbb4-zjbwp 0/1 UnexpectedAdmissionError 0 4h5m
backend-54769cbb4-zjc8g 0/1 UnexpectedAdmissionError 0 4h3m
backend-54769cbb4-zjdcp 0/1 UnexpectedAdmissionError 0 4h4m
backend-54769cbb4-zkcrb 0/1 UnexpectedAdmissionError 0 4h7m
backend-54769cbb4-zlpll 0/1 UnexpectedAdmissionError 0 4h2m
backend-54769cbb4-zm2cx 0/1 UnexpectedAdmissionError 0 4h6m
backend-54769cbb4-zn7mr 0/1 UnexpectedAdmissionError 0 4h6m
backend-54769cbb4-znjkp 0/1 UnexpectedAdmissionError 0 4h3m
backend-54769cbb4-zpnk7 0/1 UnexpectedAdmissionError 0 4h2m
backend-54769cbb4-zrrl7 0/1 UnexpectedAdmissionError 0 4h2m
backend-54769cbb4-zsdsz 0/1 UnexpectedAdmissionError 0 4h4m
backend-54769cbb4-ztdx8 0/1 UnexpectedAdmissionError 0 4h2m
backend-54769cbb4-ztln6 0/1 UnexpectedAdmissionError 0 4h2m
backend-54769cbb4-ztplg 0/1 UnexpectedAdmissionError 0 4h6m
backend-54769cbb4-ztzfh 0/1 UnexpectedAdmissionError 0 4h2m
backend-54769cbb4-zvb8g 0/1 UnexpectedAdmissionError 0 4h6m
backend-54769cbb4-zwsr8 0/1 UnexpectedAdmissionError 0 4h7m
backend-54769cbb4-zwvxr 0/1 UnexpectedAdmissionError 0 4h5m
backend-54769cbb4-zwx6h 0/1 UnexpectedAdmissionError 0 4h6m
backend-54769cbb4-zz4bf 0/1 UnexpectedAdmissionError 0 4h1m
backend-54769cbb4-zzq6t 0/1 UnexpectedAdmissionError 0 4h2m
(and many more of these)
So I added two more nodes, and now everything seems fine except for this big list of pods in an error state which I don't understand. What is this UnexpectedAdmissionError, and what should I do about it?
Note: this is a DigitalOcean cluster
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T12:38:36Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
The following seems important: kubectl describe one_failed_pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m51s default-scheduler Successfully assigned default/backend-549f576d5f-xzdv4 to std-16gb-g7mo
Warning UnexpectedAdmissionError 2m51s kubelet, std-16gb-g7mo Update plugin resources failed due to failed to write checkpoint file "kubelet_internal_checkpoint": write /var/lib/kubelet/device-plugins/.543592130: no space left on device, which is unexpected.

Answer 1

I had the same issue, while describing one of the pods with UnexpectedAdmissionError I saw the following:
Update plugin resources failed due to failed to write deviceplugin checkpoint file "kubelet_internal_checkpoint": write /var/lib/kubelet/device-plugins/.525608957: no space left on device, which is unexpected.
when doing describing node:
OutOfDisk Unknown Tue, 30 Jun 2020 14:07:04 -0400 Tue, 30 Jun 2020 14:12:05 -0400 NodeStatusUnknown Kubelet stopped posting node status.
I resolved this by rebooting node

Answer 2

Because the pod was not even started you can't actually check the logs. However describing the pod provided me with the error. We had some disk/cpu/memory utilization issues with worker5 node.
kubectl get pods | grep -i err
kube-system coredns-autoscaler-79599b9dc6-6l8s8 0/1 UnexpectedAdmissionError 0 10h <none> worker5 <none> <none>
kube-system coredns-autoscaler-79599b9dc6-kzt9z 0/1 UnexpectedAdmissionError 0 10h <none> worker5 <none> <none>
kube-system coredns-autoscaler-79599b9dc6-tgkrc 0/1 UnexpectedAdmissionError 0 10h <none> worker5 <none> <none>
kubectl describe pod -n kube-system coredns-autoscaler-79599b9dc6-kzt9z
Reason: UnexpectedAdmissionError
Message: Pod Allocate failed due to failed to write checkpoint file "kubelet_internal_checkpoint": mkdir /var: file exists, which is unexpected
First step was rebooting the node which fixed the issue. The reason was we had restored some backups to the new cluster and restore process caused this issue.
For the pods because they were a part of replica set, they got spawned on other worker nodes. Therefore we deleted the pods.
A quick way to delete a lot of pods, you can use:
kubectl get pods -n namespace | grep -i Error | cut -d' ' -f 1 | xargs kubectl delete pod
To delete all the erroraneous pods in entire cluster
kubectl get pods -A | grep -i Error | awk '{print $2}' | xargs kubectl delete pod
You can use flag -A/--all-namespaces to get pods from all namespaces in the cluster.
However if they are not getting spawned automatically which would be weird, you can run kubectl replace
kubectl get pod coredns-autoscaler-79599b9dc6-6l8s8 -n kube-system -o yaml | kubectl replace --force -f -
For further a verbose read please refer kubectl replace --help and the following blog