I tried to set worker node of kubernetes on board using arm64 arch.
This worker node was not changed to Ready status from NotReady status.
I checked Conditions log using below command:
$ kubectl describe nodes
...
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Wed, 02 Dec 2020 14:37:46 +0900 Wed, 02 Dec 2020 14:34:35 +0900 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 02 Dec 2020 14:37:46 +0900 Wed, 02 Dec 2020 14:34:35 +0900 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 02 Dec 2020 14:37:46 +0900 Wed, 02 Dec 2020 14:34:35 +0900 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Wed, 02 Dec 2020 14:37:46 +0900 Wed, 02 Dec 2020 14:34:35 +0900 KubeletNotReady [container runtime status check may not have completed yet, runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized, missing node capacity for resources: ephemeral-storage]
...
Capacity:
cpu: 8
memory: 7770600Ki
pods: 110
Allocatable:
cpu: 8
memory: 7668200Ki
pods: 110
...
This worker node seems not have ephemeral-storage resource, so
this log seems be created
"[container runtime status check may not have completed yet, runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized, missing node capacity for resources: ephemeral-storage]"
but root filesystem is mounted on / like follows,
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 23602256 6617628 15945856 30% /
devtmpfs 3634432 0 3634432 0% /dev
tmpfs 3885312 0 3885312 0% /dev/shm
tmpfs 3885312 100256 3785056 3% /run
tmpfs 3885312 0 3885312 0% /sys/fs/cgroup
tmpfs 524288 25476 498812 5% /tmp
tmpfs 524288 212 524076 1% /var/volatile
tmpfs 777060 0 777060 0% /run/user/1000
/dev/sde4 122816 49088 73728 40% /firmware
/dev/sde5 65488 608 64880 1% /bt_firmware
/dev/sde7 28144 20048 7444 73% /dsp
How can detect ephemeral-storage resource on worker node of kubernetes?
=======================================================================
I added full log of $ kubectl get nodes and $ kubectl describe nodes
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
raas-linux Ready master 6m25s v1.19.4
robot-dd9f6aaa NotReady <none> 5m16s v1.16.2-dirty
$
$ kubectl describe nodes
Name: raas-linux
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=raas-linux
kubernetes.io/os=linux
node-role.kubernetes.io/master=
Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"a6:a1:0b:43:38:29"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 192.168.3.106
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Fri, 04 Dec 2020 09:54:49 +0900
Taints: node-role.kubernetes.io/master:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: raas-linux
AcquireTime: <unset>
RenewTime: Fri, 04 Dec 2020 10:00:19 +0900
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Fri, 04 Dec 2020 09:55:14 +0900 Fri, 04 Dec 2020 09:55:14 +0900 FlannelIsUp Flannel is running on this node
MemoryPressure False Fri, 04 Dec 2020 09:55:19 +0900 Fri, 04 Dec 2020 09:54:45 +0900 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Fri, 04 Dec 2020 09:55:19 +0900 Fri, 04 Dec 2020 09:54:45 +0900 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Fri, 04 Dec 2020 09:55:19 +0900 Fri, 04 Dec 2020 09:54:45 +0900 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Fri, 04 Dec 2020 09:55:19 +0900 Fri, 04 Dec 2020 09:55:19 +0900 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.3.106
Hostname: raas-linux
Capacity:
cpu: 8
ephemeral-storage: 122546800Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 8066548Ki
pods: 110
Allocatable:
cpu: 8
ephemeral-storage: 112939130694
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7964148Ki
pods: 110
System Info:
Machine ID: 5aa3b32d7e9e409091929e7cba2d558b
System UUID: a930a228-a79a-11e5-9e9a-147517224400
Boot ID: 4e6dd5d2-bcc4-433b-8c4d-df56c33a9442
Kernel Version: 5.4.0-53-generic
OS Image: Ubuntu 18.04.5 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.10
Kubelet Version: v1.19.4
Kube-Proxy Version: v1.19.4
PodCIDR: 10.244.0.0/24
PodCIDRs: 10.244.0.0/24
Non-terminated Pods: (8 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system coredns-f9fd979d6-h7hd5 100m (1%) 0 (0%) 70Mi (0%) 170Mi (2%) 5m9s
kube-system coredns-f9fd979d6-hbkbl 100m (1%) 0 (0%) 70Mi (0%) 170Mi (2%) 5m9s
kube-system etcd-raas-linux 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5m20s
kube-system kube-apiserver-raas-linux 250m (3%) 0 (0%) 0 (0%) 0 (0%) 5m20s
kube-system kube-controller-manager-raas-linux 200m (2%) 0 (0%) 0 (0%) 0 (0%) 5m20s
kube-system kube-flannel-ds-k8b2d 100m (1%) 100m (1%) 50Mi (0%) 50Mi (0%) 5m9s
kube-system kube-proxy-wgn4l 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5m9s
kube-system kube-scheduler-raas-linux 100m (1%) 0 (0%) 0 (0%) 0 (0%) 5m20s
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 850m (10%) 100m (1%)
memory 190Mi (2%) 390Mi (5%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 5m20s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 5m20s kubelet Node raas-linux status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 5m20s kubelet Node raas-linux status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 5m20s kubelet Node raas-linux status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 5m20s kubelet Updated Node Allocatable limit across pods
Normal Starting 5m8s kube-proxy Starting kube-proxy.
Normal NodeReady 5m kubelet Node raas-linux status is now: NodeReady
Name: robot-dd9f6aaa
Roles: <none>
Labels: beta.kubernetes.io/arch=arm64
beta.kubernetes.io/os=linux
kubernetes.io/arch=arm64
kubernetes.io/hostname=robot-dd9f6aaa
kubernetes.io/os=linux
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Fri, 04 Dec 2020 09:55:58 +0900
Taints: node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/not-ready:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: robot-dd9f6aaa
AcquireTime: <unset>
RenewTime: Fri, 04 Dec 2020 10:00:16 +0900
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Fri, 04 Dec 2020 09:55:58 +0900 Fri, 04 Dec 2020 09:55:58 +0900 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Fri, 04 Dec 2020 09:55:58 +0900 Fri, 04 Dec 2020 09:55:58 +0900 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Fri, 04 Dec 2020 09:55:58 +0900 Fri, 04 Dec 2020 09:55:58 +0900 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Fri, 04 Dec 2020 09:55:58 +0900 Fri, 04 Dec 2020 09:55:58 +0900 KubeletNotReady [container runtime status check may not have completed yet, runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized, missing node capacity for resources: ephemeral-storage]
Addresses:
InternalIP: 192.168.3.102
Hostname: robot-dd9f6aaa
Capacity:
cpu: 8
memory: 7770620Ki
pods: 110
Allocatable:
cpu: 8
memory: 7668220Ki
pods: 110
System Info:
Machine ID: de6c58c435a543de8e13ce6a76477fa0
System UUID: de6c58c435a543de8e13ce6a76477fa0
Boot ID: d0999dd7-ab7d-4459-b0cd-9b25f5a50ae4
Kernel Version: 4.9.103-sda845-smp
OS Image: Kairos - Smart Machine Platform 1.0
Operating System: linux
Architecture: arm64
Container Runtime Version: docker://19.3.2
Kubelet Version: v1.16.2-dirty
Kube-Proxy Version: v1.16.2-dirty
PodCIDR: 10.244.1.0/24
PodCIDRs: 10.244.1.0/24
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system kube-flannel-ds-9xc6n 100m (1%) 100m (1%) 50Mi (0%) 50Mi (0%) 4m21s
kube-system kube-proxy-4dk7f 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4m21s
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 100m (1%) 100m (1%)
memory 50Mi (0%) 50Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 4m22s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 4m21s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 4m21s kubelet Node robot-dd9f6aaa status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 4m21s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientPID
Normal Starting 4m10s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 4m10s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 4m10s kubelet Node robot-dd9f6aaa status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 4m10s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientPID
Normal Starting 3m59s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 3m59s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 3m59s kubelet Node robot-dd9f6aaa status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 3m59s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientPID
Normal Starting 3m48s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 3m48s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal NodeHasSufficientPID 3m48s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientPID
Normal NodeHasNoDiskPressure 3m48s kubelet Node robot-dd9f6aaa status is now: NodeHasNoDiskPressure
Normal Starting 3m37s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 3m36s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 3m36s kubelet Node robot-dd9f6aaa status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 3m36s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientPID
Normal Starting 3m25s kubelet Starting kubelet.
Normal Starting 3m14s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 3m3s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal Starting 3m3s kubelet Starting kubelet.
Normal Starting 2m52s kubelet Starting kubelet.
Normal Starting 2m40s kubelet Starting kubelet.
Normal Starting 2m29s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 2m29s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 2m29s kubelet Node robot-dd9f6aaa status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 2m29s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientPID
Normal NodeHasSufficientMemory 2m18s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 2m18s kubelet Node robot-dd9f6aaa status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 2m18s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientPID
Normal Starting 2m18s kubelet Starting kubelet.
Normal Starting 2m7s kubelet Starting kubelet.
Normal Starting 115s kubelet Starting kubelet.
Normal NodeHasNoDiskPressure 104s kubelet Node robot-dd9f6aaa status is now: NodeHasNoDiskPressure
Normal Starting 104s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 104s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal Starting 93s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 93s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 93s kubelet Node robot-dd9f6aaa status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 93s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientPID
Normal Starting 82s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 82s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal Starting 71s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 70s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 70s kubelet Node robot-dd9f6aaa status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 70s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientPID
Normal Starting 59s kubelet Starting kubelet.
Normal Starting 48s kubelet Starting kubelet.
Normal Starting 37s kubelet Starting kubelet.
Normal Starting 26s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 25s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal Starting 15s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 14s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal Starting 3s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 3s kubelet Node robot-dd9f6aaa status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 3s kubelet Node robot-dd9f6aaa status is now: NodeHasNoDiskPressure
Delete /etc/docker/daemon.json file and reboot
Install CNI plugins binaries in /opt/cni/bin directory
https://github.com/containernetworking/plugins/releases/download/v0.8.7/cni-plugins-linux-arm64-v0.8.7.tgz
Step 1: kubectl get mutatingwebhookconfigurations -oyaml > mutating.txt
Step 2: Kubectl delete -f mutating.txt
Step 3: Restart the node
Step 4: You should be able to see the node is ready
Step 5: Install the mutatingwebhookconfiguration back
Related
We are facing a problem with one of our kubernetes clusters which uses only preemtible nodes for our staging environment. Essentially nodes change state to NotReady randomly which causes pods to get stuck on rescheduling. We started experiencing this after we upgraded kubernetes to v1.15. We didn't had this issue before. We tried switching off different deployments to pint point if its caused maybe by a certain pod, but so far didn't found any correlation. Any advice would be very helpful at least where to look for cause of NotReady state in the first place. Here is a typical output when checking node states:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-staging-default-pool-3de3436f-0xqr NotReady <none> 27m v1.15.11-gke.9
gke-staging-default-pool-3de3436f-9cb8 Ready <none> 27m v1.15.11-gke.9
gke-staging-default-pool-3de3436f-jrt6 Ready <none> 27m v1.15.11-gke.9
gke-staging-default-pool-3de3436f-ls7p NotReady <none> 17m v1.15.11-gke.9
gke-staging-default-pool-3de3436f-r8xr Ready <none> 14m v1.15.11-gke.9
gke-staging-default-pool-3de3436f-z01r NotReady <none> 27m v1.15.11-gke.9
gke-staging-default-pool-3de3436f-nw2g NotReady <none> 100m v1.15.11-gke.9
evaldas#evaldas-pc:~$ kubectl describe node gke-staging-default-pool-3de3436f-nw2g
Name: gke-staging-default-pool-3de3436f-nw2g
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/fluentd-ds-ready=true
beta.kubernetes.io/instance-type=n1-standard-4
beta.kubernetes.io/os=linux
cloud.google.com/gke-nodepool=default-pool
cloud.google.com/gke-os-distribution=cos
cloud.google.com/gke-preemptible=true
failure-domain.beta.kubernetes.io/region=europe-west1
failure-domain.beta.kubernetes.io/zone=europe-west1-d
kubernetes.io/arch=amd64
kubernetes.io/hostname=gke-staging-default-pool-3de3436f-nw2g
kubernetes.io/os=linux
pool-designation=main-pool
Annotations: container.googleapis.com/instance_id: 100599290254760530
node.alpha.kubernetes.io/ttl: 0
node.gke.io/last-applied-node-labels:
beta.kubernetes.io/fluentd-ds-ready=true,cloud.google.com/gke-nodepool=default-pool,cloud.google.com/gke-os-distribution=cos,cloud.google....
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 15 Jun 2020 09:01:06 +0300
Taints: cloud.google.com/impending-node-termination:NoSchedule
node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: gke-staging-default-pool-3de3436f-nw2g
AcquireTime: <unset>
RenewTime: Mon, 15 Jun 2020 10:36:08 +0300
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
ReadonlyFilesystem False Mon, 15 Jun 2020 10:35:57 +0300 Mon, 15 Jun 2020 10:35:56 +0300 FilesystemIsNotReadOnly Filesystem is not read-only
CorruptDockerOverlay2 False Mon, 15 Jun 2020 10:35:57 +0300 Mon, 15 Jun 2020 10:35:56 +0300 NoCorruptDockerOverlay2 docker overlay2 is functioning properly
FrequentUnregisterNetDevice False Mon, 15 Jun 2020 10:35:57 +0300 Mon, 15 Jun 2020 10:35:56 +0300 NoFrequentUnregisterNetDevice node is functioning properly
FrequentKubeletRestart False Mon, 15 Jun 2020 10:35:57 +0300 Mon, 15 Jun 2020 10:35:56 +0300 NoFrequentKubeletRestart kubelet is functioning properly
FrequentDockerRestart False Mon, 15 Jun 2020 10:35:57 +0300 Mon, 15 Jun 2020 10:35:56 +0300 NoFrequentDockerRestart docker is functioning properly
FrequentContainerdRestart False Mon, 15 Jun 2020 10:35:57 +0300 Mon, 15 Jun 2020 10:35:56 +0300 NoFrequentContainerdRestart containerd is functioning properly
KernelDeadlock False Mon, 15 Jun 2020 10:35:57 +0300 Mon, 15 Jun 2020 10:35:56 +0300 KernelHasNoDeadlock kernel has no deadlock
NetworkUnavailable False Mon, 15 Jun 2020 09:01:20 +0300 Mon, 15 Jun 2020 09:01:20 +0300 RouteCreated RouteController created a route
MemoryPressure Unknown Mon, 15 Jun 2020 10:36:09 +0300 Mon, 15 Jun 2020 10:36:52 +0300 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Mon, 15 Jun 2020 10:36:09 +0300 Mon, 15 Jun 2020 10:36:52 +0300 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Mon, 15 Jun 2020 10:36:09 +0300 Mon, 15 Jun 2020 10:36:52 +0300 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Mon, 15 Jun 2020 10:36:09 +0300 Mon, 15 Jun 2020 10:36:52 +0300 NodeStatusUnknown Kubelet stopped posting node status.
Addresses:
InternalIP: 10.132.0.60
ExternalIP: 35.205.52.80
InternalDNS: gke-staging-default-pool-3de3436f-nw2g.c.stacktome-prod.internal
Hostname: gke-staging-default-pool-3de3436f-nw2g.c.stacktome-prod.internal
Capacity:
attachable-volumes-gce-pd: 127
cpu: 4
ephemeral-storage: 98868448Ki
hugepages-2Mi: 0
memory: 15397596Ki
pods: 110
Allocatable:
attachable-volumes-gce-pd: 127
cpu: 3920m
ephemeral-storage: 47093746742
hugepages-2Mi: 0
memory: 12698332Ki
pods: 110
System Info:
Machine ID: b5baa1dbc20255ae27033ed23ab4e10f
System UUID: b5baa1db-c202-55ae-2703-3ed23ab4e10f
Boot ID: bdebb6f0-9267-469e-a9fb-ca330e50f249
Kernel Version: 4.19.104+
OS Image: Container-Optimized OS from Google
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.1
Kubelet Version: v1.15.11-gke.9
Kube-Proxy Version: v1.15.11-gke.9
PodCIDR: 10.4.119.0/24
ProviderID: gce://stacktome-prod/europe-west1-d/gke-staging-default-pool-3de3436f-nw2g
Non-terminated Pods: (4 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system fluentd-gcp-v3.1.1-9z74z 100m (2%) 1 (25%) 200Mi (1%) 500Mi (4%) 100m
kube-system kube-proxy-gke-staging-default-pool-3de3436f-nw2g 100m (2%) 0 (0%) 0 (0%) 0 (0%) 5m15s
kube-system node-termination-handler-jwvqw 150m (3%) 150m (3%) 30Mi (0%) 30Mi (0%) 100m
kube-system prometheus-to-sd-jctnk 1m (0%) 3m (0%) 20Mi (0%) 37Mi (0%) 100m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 351m (8%) 1153m (29%)
memory 250Mi (2%) 567Mi (4%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-gce-pd 0 0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 39m kubelet, gke-staging-default-pool-3de3436f-nw2g Starting kubelet.
Normal NodeAllocatableEnforced 39m kubelet, gke-staging-default-pool-3de3436f-nw2g Updated Node Allocatable limit across pods
Normal NodeHasSufficientMemory 39m (x2 over 39m) kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 39m (x2 over 39m) kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 39m (x2 over 39m) kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g status is now: NodeHasSufficientPID
Warning Rebooted 39m kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g has been rebooted, boot id: 893432db-0107-4781-a9a7-05c88def8887
Normal NodeReady 39m kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g status is now: NodeReady
Normal Starting 39m kube-proxy, gke-staging-default-pool-3de3436f-nw2g Starting kube-proxy.
Normal NodeReady 24m kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g status is now: NodeReady
Normal NodeAllocatableEnforced 24m kubelet, gke-staging-default-pool-3de3436f-nw2g Updated Node Allocatable limit across pods
Normal NodeHasSufficientMemory 24m (x2 over 24m) kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 24m (x2 over 24m) kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g status is now: NodeHasNoDiskPressure
Normal Starting 24m kubelet, gke-staging-default-pool-3de3436f-nw2g Starting kubelet.
Warning Rebooted 24m kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g has been rebooted, boot id: 1632075d-b6b6-4646-979b-88622c79bcd0
Normal NodeHasSufficientPID 24m (x2 over 24m) kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g status is now: NodeHasSufficientPID
Normal Starting 24m kube-proxy, gke-staging-default-pool-3de3436f-nw2g Starting kube-proxy.
Normal Starting 5m19s kubelet, gke-staging-default-pool-3de3436f-nw2g Starting kubelet.
Normal NodeAllocatableEnforced 5m19s kubelet, gke-staging-default-pool-3de3436f-nw2g Updated Node Allocatable limit across pods
Warning Rebooted 5m19s kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g has been rebooted, boot id: bdebb6f0-9267-469e-a9fb-ca330e50f249
Normal NodeHasSufficientMemory 5m18s (x2 over 5m19s) kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 5m18s (x2 over 5m19s) kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 5m18s (x2 over 5m19s) kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g status is now: NodeHasSufficientPID
Normal NodeReady 5m18s kubelet, gke-staging-default-pool-3de3436f-nw2g Node gke-staging-default-pool-3de3436f-nw2g status is now: NodeReady
Normal Starting 5m16s kube-proxy, gke-staging-default-pool-3de3436f-nw2g Starting kube-proxy.
kubelet logs:
Default
2020-06-15 09:54:02.097 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-termination-handleraudit_log, method: "io.k8s.core.v1.nodes.update", principal_email: "system:serviceaccount:kube-system:node-termination-handler"
Default
2020-06-15 09:55:00.326 EEST
k8s.ioio.k8s.core.v1.nodes.patchcore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-controlleraudit_log, method: "io.k8s.core.v1.nodes.patch", principal_email: "system:serviceaccount:kube-system:node-controller"
Default
2020-06-15 10:02:06.111 EEST
k8s.ioio.k8s.core.v1.nodes.createcore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:node:gke-staging-default-pool-3de3436f-nw2gConflict
Default
2020-06-15 10:02:06.187 EEST
k8s.ioio.k8s.core.v1.nodes.patchcore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-controlleraudit_log, method: "io.k8s.core.v1.nodes.patch", principal_email: "system:serviceaccount:kube-system:node-controller"
Default
2020-06-15 10:02:06.413 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:gcp-controller-managerConflict
Default
2020-06-15 10:02:06.690 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:gcp-controller-manageraudit_log, method: "io.k8s.core.v1.nodes.update", principal_email: "system:gcp-controller-manager"
Default
2020-06-15 10:02:39.685 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-termination-handleraudit_log, method: "io.k8s.core.v1.nodes.update", principal_email: "system:serviceaccount:kube-system:node-termination-handler"
Default
2020-06-15 10:08:33.254 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-termination-handleraudit_log, method: "io.k8s.core.v1.nodes.update", principal_email: "system:serviceaccount:kube-system:node-termination-handler"
Default
2020-06-15 10:09:20.685 EEST
k8s.ioio.k8s.core.v1.nodes.patchcore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-controlleraudit_log, method: "io.k8s.core.v1.nodes.patch", principal_email: "system:serviceaccount:kube-system:node-controller"
Default
2020-06-15 10:16:28.608 EEST
k8s.ioio.k8s.core.v1.nodes.createcore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:node:gke-staging-default-pool-3de3436f-nw2gConflict
Default
2020-06-15 10:16:28.680 EEST
k8s.ioio.k8s.core.v1.nodes.patchcore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-controlleraudit_log, method: "io.k8s.core.v1.nodes.patch", principal_email: "system:serviceaccount:kube-system:node-controller"
Default
2020-06-15 10:16:28.979 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:gcp-controller-managerConflict
Default
2020-06-15 10:16:29.215 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:gcp-controller-manageraudit_log, method: "io.k8s.core.v1.nodes.update", principal_email: "system:gcp-controller-manager"
Default
2020-06-15 10:17:01.589 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-termination-handleraudit_log, method: "io.k8s.core.v1.nodes.update", principal_email: "system:serviceaccount:kube-system:node-termination-handler"
Default
2020-06-15 10:28:05.346 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-termination-handleraudit_log, method: "io.k8s.core.v1.nodes.update", principal_email: "system:serviceaccount:kube-system:node-termination-handler"
Default
2020-06-15 10:28:51.815 EEST
k8s.ioio.k8s.core.v1.nodes.patchcore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-controlleraudit_log, method: "io.k8s.core.v1.nodes.patch", principal_email: "system:serviceaccount:kube-system:node-controller"
Default
2020-06-15 10:35:58.960 EEST
k8s.ioio.k8s.core.v1.nodes.createcore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:node:gke-staging-default-pool-3de3436f-nw2gConflict
Default
2020-06-15 10:35:59.032 EEST
k8s.ioio.k8s.core.v1.nodes.patchcore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-controlleraudit_log, method: "io.k8s.core.v1.nodes.patch", principal_email: "system:serviceaccount:kube-system:node-controller"
Default
2020-06-15 10:35:59.250 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:gcp-controller-managerConflict
Default
2020-06-15 10:35:59.479 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:gcp-controller-manageraudit_log, method: "io.k8s.core.v1.nodes.update", principal_email: "system:gcp-controller-manager"
Default
2020-06-15 10:36:52.169 EEST
k8s.ioio.k8s.core.v1.nodes.patchcore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-controlleraudit_log, method: "io.k8s.core.v1.nodes.patch", principal_email: "system:serviceaccount:kube-system:node-controller"
Default
2020-06-15 10:44:14.663 EEST
k8s.ioio.k8s.core.v1.nodes.createcore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:node:gke-staging-default-pool-3de3436f-nw2gConflict
Default
2020-06-15 10:44:14.744 EEST
k8s.ioio.k8s.core.v1.nodes.patchcore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-controlleraudit_log, method: "io.k8s.core.v1.nodes.patch", principal_email: "system:serviceaccount:kube-system:node-controller"
Default
2020-06-15 10:44:15.013 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:gcp-controller-managerConflict
Default
2020-06-15 10:44:15.306 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:gcp-controller-manageraudit_log, method: "io.k8s.core.v1.nodes.update", principal_email: "system:gcp-controller-manager"
Default
2020-06-15 10:44:48.316 EEST
k8s.ioio.k8s.core.v1.nodes.updatecore/v1/nodes/gke-staging-default-pool-3de3436f-nw2gsystem:serviceaccount:kube-system:node-termination-handleraudit_log, method: "io.k8s.core.v1.nodes.update", principal_email: "system:servic
I am very begginer on kubernetes. Sorry if this a is dumb question.
I am using minikube and kvm2(5.0.0). Here is the info about minikube and kubectl version
Minikube status output
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
kubectl cluster-info output:
Kubernetes master is running at https://127.0.0.1:32768
KubeDNS is running at https://127.0.0.1:32768/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
I am trying to deploy a pod using kubectl apply -f client-pod.yaml. Here is my client-pod.yaml configuration
apiVersion: v1
kind: Pod
metadata:
name: client-pod
labels:
component: web
spec:
containers:
- name: client
image: stephengrider/multi-client
ports:
- containerPort: 3000
This is the kubectl get pods output:
NAME READY STATUS RESTARTS AGE
client-pod 0/1 Pending 0 4m15s
kubectl describe pods output:
Name: client-pod
Namespace: default
Priority: 0
Node: <none>
Labels: component=web
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"component":"web"},"name":"client-pod","namespace":"default"},"spec...
Status: Pending
IP:
IPs: <none>
Containers:
client:
Image: stephengrider/multi-client
Port: 3000/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-z45bq (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
default-token-z45bq:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-z45bq
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
Warning FailedScheduling <unknown> default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
I have been searching for a way to see which taints is stopping to pod to initialize wihout luck.
Is there a way to see the taint that is failing?
kubectl get nodes output:
NAME STATUS ROLES AGE VERSION
m01 Ready master 11h v1.17.3
-- EDIT --
kubectl describe nodes output:
Name: home-pc
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=home-pc
kubernetes.io/os=linux
minikube.k8s.io/commit=eb13446e786c9ef70cb0a9f85a633194e62396a1
minikube.k8s.io/name=minikube
minikube.k8s.io/updated_at=2020_03_17T22_51_28_0700
minikube.k8s.io/version=v1.8.2
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 17 Mar 2020 22:51:25 -0500
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: home-pc
AcquireTime: <unset>
RenewTime: Tue, 17 Mar 2020 22:51:41 -0500
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 17 Mar 2020 22:51:41 -0500 Tue, 17 Mar 2020 22:51:21 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 17 Mar 2020 22:51:41 -0500 Tue, 17 Mar 2020 22:51:21 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 17 Mar 2020 22:51:41 -0500 Tue, 17 Mar 2020 22:51:21 -0500 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 17 Mar 2020 22:51:41 -0500 Tue, 17 Mar 2020 22:51:41 -0500 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.0.12
Hostname: home-pc
Capacity:
cpu: 12
ephemeral-storage: 227688908Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 8159952Ki
pods: 110
Allocatable:
cpu: 12
ephemeral-storage: 209838097266
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 8057552Ki
pods: 110
System Info:
Machine ID: 339d426453b4492da92f75d06acc1e0d
System UUID: 62eedb55-444f-61ce-75e9-b06ebf3331a0
Boot ID: a9ae9889-d7cb-48c5-ae75-b2052292ac7a
Kernel Version: 5.0.0-38-generic
OS Image: Ubuntu 19.04
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.5
Kubelet Version: v1.17.3
Kube-Proxy Version: v1.17.3
Non-terminated Pods: (7 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system coredns-6955765f44-mbwqt 100m (0%) 0 (0%) 70Mi (0%) 170Mi (2%) 10s
kube-system coredns-6955765f44-sblf2 100m (0%) 0 (0%) 70Mi (0%) 170Mi (2%) 10s
kube-system etcd-home-pc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 13s
kube-system kube-apiserver-home-pc 250m (2%) 0 (0%) 0 (0%) 0 (0%) 13s
kube-system kube-controller-manager-home-pc 200m (1%) 0 (0%) 0 (0%) 0 (0%) 13s
kube-system kube-proxy-lk7xs 0 (0%) 0 (0%) 0 (0%) 0 (0%) 10s
kube-system kube-scheduler-home-pc 100m (0%) 0 (0%) 0 (0%) 0 (0%) 12s
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 750m (6%) 0 (0%)
memory 140Mi (1%) 340Mi (4%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 24s kubelet, home-pc Starting kubelet.
Normal NodeHasSufficientMemory 23s (x4 over 24s) kubelet, home-pc Node home-pc status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 23s (x3 over 24s) kubelet, home-pc Node home-pc status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 23s (x3 over 24s) kubelet, home-pc Node home-pc status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 23s kubelet, home-pc Updated Node Allocatable limit across pods
Normal Starting 13s kubelet, home-pc Starting kubelet.
Normal NodeHasSufficientMemory 13s kubelet, home-pc Node home-pc status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 13s kubelet, home-pc Node home-pc status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 13s kubelet, home-pc Node home-pc status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 13s kubelet, home-pc Updated Node Allocatable limit across pods
Normal Starting 9s kube-proxy, home-pc Starting kube-proxy.
Normal NodeReady 3s kubelet, home-pc Node home-pc status is now: NodeReady
You have some taints on the node which is stopping the scheduler from deploying the pod.Either remove the taint from master node or add tolerations in the pod spec.
I just installed Kubernetes on a cluster of Ubuntu's using the explanation of digital ocean with Ansible. Everything seems to be fine; but when verifying the cluster, the master node is in a not ready status:
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
jwdkube-master-01 NotReady master 44m v1.12.2
jwdkube-worker-01 Ready <none> 44m v1.12.2
jwdkube-worker-02 Ready <none> 44m v1.12.2
This is the version:
# kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
When I check the master node, the kube-proxy is hanging in a starting mode:
# kubectl describe nodes jwdkube-master-01
Name: jwdkube-master-01
Roles: master
...
LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
OutOfDisk False Thu, 08 Nov 2018 10:24:45 +0000 Thu, 08 Nov 2018 09:36:10 +0000 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 08 Nov 2018 10:24:45 +0000 Thu, 08 Nov 2018 09:36:10 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 08 Nov 2018 10:24:45 +0000 Thu, 08 Nov 2018 09:36:10 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 08 Nov 2018 10:24:45 +0000 Thu, 08 Nov 2018 09:36:10 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Thu, 08 Nov 2018 10:24:45 +0000 Thu, 08 Nov 2018 09:36:10 +0000 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
InternalIP: 104.248.207.107
Hostname: jwdkube-master-01
Capacity:
cpu: 1
ephemeral-storage: 25226960Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 1008972Ki
pods: 110
Allocatable:
cpu: 1
ephemeral-storage: 23249166298
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 906572Ki
pods: 110
System Info:
Machine ID: 771c0f669c0a40a1ba7c28bf1f05a637
System UUID: 771c0f66-9c0a-40a1-ba7c-28bf1f05a637
Boot ID: 2532ae4d-c08c-45d8-b94c-6e88912ed627
Kernel Version: 4.18.0-10-generic
OS Image: Ubuntu 18.10
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.6.1
Kubelet Version: v1.12.2
Kube-Proxy Version: v1.12.2
PodCIDR: 10.244.0.0/24
Non-terminated Pods: (5 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system etcd-jwdkube-master-01 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system kube-apiserver-jwdkube-master-01 250m (25%) 0 (0%) 0 (0%) 0 (0%)
kube-system kube-controller-manager-jwdkube-master-01 200m (20%) 0 (0%) 0 (0%) 0 (0%)
kube-system kube-proxy-p8cbq 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system kube-scheduler-jwdkube-master-01 100m (10%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 550m (55%) 0 (0%)
memory 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasSufficientDisk 48m (x6 over 48m) kubelet, jwdkube-master-01 Node jwdkube-master-01 status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 48m (x6 over 48m) kubelet, jwdkube-master-01 Node jwdkube-master-01 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 48m (x6 over 48m) kubelet, jwdkube-master-01 Node jwdkube-master-01 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 48m (x5 over 48m) kubelet, jwdkube-master-01 Node jwdkube-master-01 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 48m kubelet, jwdkube-master-01 Updated Node Allocatable limit across pods
Normal Starting 48m kube-proxy, jwdkube-master-01 Starting kube-proxy.
update
running kubectl get pods -n kube-system:
NAME READY STATUS RESTARTS AGE
coredns-576cbf47c7-8p7k2 1/1 Running 0 4h47m
coredns-576cbf47c7-s5tlv 1/1 Running 0 4h47m
etcd-jwdkube-master-01 1/1 Running 1 140m
kube-apiserver-jwdkube-master-01 1/1 Running 1 140m
kube-controller-manager-jwdkube-master-01 1/1 Running 1 140m
kube-flannel-ds-5bzrx 1/1 Running 0 4h47m
kube-flannel-ds-bfs9k 1/1 Running 0 4h47m
kube-proxy-4lrzw 1/1 Running 1 4h47m
kube-proxy-57x28 1/1 Running 0 4h47m
kube-proxy-j8bf5 1/1 Running 0 4h47m
kube-scheduler-jwdkube-master-01 1/1 Running 1 140m
tiller-deploy-6f6fd74b68-5xt54 1/1 Running 0 112m
It seems to be a problem of Flannel v0.9.1 compatibility with Kubernetes cluster v1.12.2. Once you replace URL in master configuration playbook it should help:
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/
kube-flannel.yml
To enforce this solution on the current cluster:
On the master node delete relevant objects for Flannel v0.9.1:
kubectl delete clusterrole flannel -n kube-system
kubectl delete clusterrolebinding flannel-n kube-system
kubectl delete clusterrolebinding flannel -n kube-system
kubectl delete serviceaccount flannel -n kube-system
kubectl delete configmap kube-flannel-cfg -n kube-system
kubectl delete daemonset.extensions kube-flannel-ds -n kube-system
Proceed also with Flannel Pods deletion:
kubectl delete pod kube-flannel-ds-5bzrx -n kube-system
kubectl delete pod kube-flannel-ds-bfs9k -n kube-system
And check whether no more objects related to Flannel exist:
kubectl get all --all-namespaces
Install the latest Flannel version to your cluster:
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
For me it works, however if you discover any further problems write a comment below this answer.
Can somebody explain why the following command shows that there have been no restarts but the age is 2 hours when it was started 17 days ago
kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
api-depl-nm-xxx 1/1 Running 0 17d xxx.xxx.xxx.xxx ip-xxx-xxx-xxx-xxx.eu-west-1.compute.internal
ei-depl-nm-xxx 1/1 Running 0 2h xxx.xxx.xxx.xxx ip-xxx-xxx-xxx-xxx.eu-west-1.compute.internal
jenkins-depl-nm-xxx 1/1 Running 0 2h xxx.xxx.xxx.xxx ip-xxx-xxx-xxx-xxx.eu-west-1.compute.internal
The deployments have been running for 17 days:
kubectl get deploy -o wide
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE CONTAINER(S) IMAGE(S) SELECTOR
api-depl-nm 1 1 1 1 17d api-depl-nm xxx name=api-depl-nm
ei-depl-nm 1 1 1 1 17d ei-depl-nm xxx name=ei-depl-nm
jenkins-depl-nm 1 1 1 1 17d jenkins-depl-nm xxx name=jenkins-depl-nm
The start time was 2 hours ago:
kubectl describe po ei-depl-nm-xxx | grep Start
Start Time: Tue, 24 Jul 2018 09:07:05 +0100
Started: Tue, 24 Jul 2018 09:10:33 +0100
The application logs show it restarted.
So why is the restarts 0?
Updated with more information as a response to answer.
I may be wrong but I don't think the deployment was updated or scaled it certainly was not done be me and no one else has access to the system.
kubectl describe deployment ei-depl-nm
...
CreationTimestamp: Fri, 06 Jul 2018 17:06:24 +0100
Labels: name=ei-depl-nm
...
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
...
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: ei-depl-nm-xxx (1/1 replicas created)
Events: <none>
I may be wrong but I don't think the worker node was restarted or shut down
kubectl describe nodes ip-xxx.eu-west-1.compute.internal
Taints: <none>
CreationTimestamp: Fri, 06 Jul 2018 16:39:40 +0100
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Fri, 06 Jul 2018 16:39:45 +0100 Fri, 06 Jul 2018 16:39:45 +0100 RouteCreated RouteController created a route
OutOfDisk False Wed, 25 Jul 2018 16:30:36 +0100 Fri, 06 Jul 2018 16:39:40 +0100 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Wed, 25 Jul 2018 16:30:36 +0100 Wed, 25 Jul 2018 02:23:01 +0100 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 25 Jul 2018 16:30:36 +0100 Wed, 25 Jul 2018 02:23:01 +0100 KubeletHasNoDiskPressure kubelet has no disk pressure
Ready True Wed, 25 Jul 2018 16:30:36 +0100 Wed, 25 Jul 2018 02:23:11 +0100 KubeletReady kubelet is posting ready status
......
Non-terminated Pods: (4 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
default ei-depl-nm-xxx 100m (5%) 0 (0%) 0 (0%) 0 (0%)
default jenkins-depl-nm-xxx 100m (5%) 0 (0%) 0 (0%) 0 (0%)
kube-system kube-dns-xxx 260m (13%) 0 (0%) 110Mi (1%) 170Mi (2%)
kube-system kube-proxy-ip-xxx.eu-west-1.compute.internal 100m (5%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
560m (28%) 0 (0%) 110Mi (1%) 170Mi (2%)
Events: <none>
There are two things that might happen:
The deployment was updated or scaled:
age of deployment does not change
new ReplicaSet is created, old ReplicaSet is deleted. You can check it by running
$ kubectl describe deployment <deployment_name>
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 1m deployment-controller Scaled up replica set testdep1-75488876f6 to 1
Normal ScalingReplicaSet 1m deployment-controller Scaled down replica set testdep1-d4884df5f to 0
pods created by old ReplicaSet are terminated, new ReplicaSet created brand new pod with restarts 0 and age 0 sec.
Worker node was restarted or shut down.
Pod on old worker node disappears
Scheduler creates a brand new pod on the first available node (it can be the same node after reboot) with restarts 0 and age 0 sec.
You can check the node start events by running
kubectl describe nodes <node_name>
...
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 32s kubelet, <node-name> Starting kubelet.
Normal NodeHasSufficientPID 31s (x5 over 32s) kubelet, <node-name> Node <node-name> status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 31s kubelet, <node-name> Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 30s (x6 over 32s) kubelet, <node-name> Node <node-name> status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 30s (x6 over 32s) kubelet, <node-name> Node <node-name> status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 30s (x6 over 32s) kubelet, <node-name> Node <node-name> status is now: NodeHasNoDiskPressure
Normal Starting 10s kube-proxy, <node-name> Starting kube-proxy.
I have a k8s cluster deployed on premise on ubuntu 16.04.3 VMs.
Cluster is composed by one master and 3 nodes. Overlay network is flannel.
# kubectl get no
NAME STATUS ROLES AGE VERSION
buru Ready <none> 70d v1.8.4
fraser Ready,SchedulingDisabled <none> 2h v1.8.4
tasmania Ready <none> 1d v1.8.4
whiddy Ready,SchedulingDisabled master 244d v1.8.4
Despite being configured exactly the same way, 2 of my nodes (buru and tasmania) work correctly, while the third one (fraser) simply doesn't want to collaborate.
If I ssh in the fraser server, I can correctly reach the overlay network:
root#fraser:~# ifconfig flannel.1
flannel.1 Link encap:Ethernet HWaddr 52:4a:da:84:8a:7b
inet addr:10.244.3.0 Bcast:0.0.0.0 Mask:255.255.255.255
inet6 addr: fe80::504a:daff:fe84:8a7b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:11 errors:0 dropped:0 overruns:0 frame:0
TX packets:11 errors:0 dropped:8 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:756 (756.0 B) TX bytes:756 (756.0 B)
root#fraser:~# ping 10.244.0.1
PING 10.244.0.1 (10.244.0.1) 56(84) bytes of data.
64 bytes from 10.244.0.1: icmp_seq=1 ttl=64 time=0.764 ms
^C
--- 10.244.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.764/0.764/0.764/0.000 ms
root#fraser:~# ping 10.244.0.1
PING 10.244.0.1 (10.244.0.1) 56(84) bytes of data.
64 bytes from 10.244.0.1: icmp_seq=1 ttl=64 time=0.447 ms
64 bytes from 10.244.0.1: icmp_seq=2 ttl=64 time=1.20 ms
64 bytes from 10.244.0.1: icmp_seq=3 ttl=64 time=0.560 ms
^C
--- 10.244.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.447/0.736/1.203/0.334 ms
But the pods cannot reach the overlay network apparently:
# kubectl --all-namespaces=true get po -o wide | grep fraser
kube-system test-fraser 1/1 Running 0 20m 10.244.3.7 fraser
# kubectl -n kube-system exec -ti test-fraser ash
/ # ping 10.244.0.1
PING 10.244.0.1 (10.244.0.1): 56 data bytes
^C
--- 10.244.0.1 ping statistics ---
12 packets transmitted, 0 packets received, 100% packet loss
The test-fraser pod is just an alpine static pod I am using for troubleshooting.
The same pod, deployed the same way in another node (buru) works just fine.
Since overlay network works on the host itself, I'd say that flannel is working just fine here.
For some reason, though, networking inside pods isn't working.
Other notes
All server have no firewall enabled
Docker version is the same (1.13.1)
All nodes are up to date in terms of ubuntu updates
Can anybody help me troubleshoot this problem?
edit
kubectl describe no fraser
Name: fraser
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=fraser
Annotations: flannel.alpha.coreos.com/backend-data={"VtepMAC":"52:4a:da:84:8a:7b"}
flannel.alpha.coreos.com/backend-type=vxlan
flannel.alpha.coreos.com/kube-subnet-manager=true
flannel.alpha.coreos.com/public-ip=80.211.157.110
node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: <none>
CreationTimestamp: Thu, 07 Dec 2017 12:51:22 +0100
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
OutOfDisk False Thu, 07 Dec 2017 15:27:27 +0100 Thu, 07 Dec 2017 12:51:22 +0100 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 07 Dec 2017 15:27:27 +0100 Thu, 07 Dec 2017 14:47:57 +0100 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 07 Dec 2017 15:27:27 +0100 Thu, 07 Dec 2017 14:47:57 +0100 KubeletHasNoDiskPressure kubelet has no disk pressure
Ready True Thu, 07 Dec 2017 15:27:27 +0100 Thu, 07 Dec 2017 14:48:07 +0100 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 80.211.157.110
Hostname: fraser
Capacity:
cpu: 4
memory: 8171244Ki
pods: 110
Allocatable:
cpu: 4
memory: 8068844Ki
pods: 110
System Info:
Machine ID: cb102c57fd539a2fb8ffab52578f27bd
System UUID: 423E50F4-C4EF-23F0-F300-B568F4B4B8B1
Boot ID: ca80d640-380a-4851-bab0-ee1fffd20bb2
Kernel Version: 4.4.0-92-generic
OS Image: Ubuntu 16.04.3 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://1.13.1
Kubelet Version: v1.8.4
Kube-Proxy Version: v1.8.4
PodCIDR: 10.244.3.0/24
ExternalID: fraser
Non-terminated Pods: (5 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system filebeat-mghqx 100m (2%) 0 (0%) 100Mi (1%) 200Mi (2%)
kube-system kube-flannel-ds-gvw4s 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system kube-proxy-62vts 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system test-fraser 0 (0%) 0 (0%) 0 (0%) 0 (0%)
prometheus prometheus-prometheus-node-exporter-mwq67 0 (0%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
100m (2%) 0 (0%) 100Mi (1%) 200Mi (2%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 48m kubelet, fraser Starting kubelet.
Normal NodeAllocatableEnforced 48m kubelet, fraser Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 48m kubelet, fraser Node fraser status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 48m kubelet, fraser Node fraser status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 48m kubelet, fraser Node fraser status is now: NodeHasNoDiskPressure
Normal NodeNotReady 48m kubelet, fraser Node fraser status is now: NodeNotReady
Normal NodeNotSchedulable 48m kubelet, fraser Node fraser status is now: NodeNotSchedulable
Normal NodeReady 48m kubelet, fraser Node fraser status is now: NodeReady
Normal NodeNotSchedulable 48m kubelet, fraser Node fraser status is now: NodeNotSchedulable
Normal NodeAllocatableEnforced 48m kubelet, fraser Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 48m kubelet, fraser Node fraser status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 48m kubelet, fraser Node fraser status is now: NodeHasSufficientMemory
Normal Starting 48m kubelet, fraser Starting kubelet.
Normal NodeNotReady 48m kubelet, fraser Node fraser status is now: NodeNotReady
Normal NodeHasNoDiskPressure 48m kubelet, fraser Node fraser status is now: NodeHasNoDiskPressure
Normal NodeReady 48m kubelet, fraser Node fraser status is now: NodeReady
Normal Starting 39m kubelet, fraser Starting kubelet.
Normal NodeAllocatableEnforced 39m kubelet, fraser Updated Node Allocatable limit across pods
Normal NodeHasSufficientDisk 39m kubelet, fraser Node fraser status is now: NodeHasSufficientDisk
Normal NodeHasSufficientMemory 39m (x2 over 39m) kubelet, fraser Node fraser status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 39m (x2 over 39m) kubelet, fraser Node fraser status is now: NodeHasNoDiskPressure
Normal NodeNotReady 39m kubelet, fraser Node fraser status is now: NodeNotReady
Normal NodeNotSchedulable 39m kubelet, fraser Node fraser status is now: NodeNotSchedulable
Normal NodeReady 39m kubelet, fraser Node fraser status is now: NodeReady
Normal Starting 39m kube-proxy, fraser Starting kube-proxy.
The issue has been answered in the comment sections.
for debugging k8s Node, we need to make sure following components (Kubelet,Docker,Kube-proxy and IPtables) work perfectly.
we can get comprehensive information by the following command
kubectl get nodes
kubectl describe nodes NODE-NAME
By the comprehensive result, we can check kube-proxy, kubelet , docker and CNI plugin (flannel) are running perfectly.
If its networking issues we will check IPtables
iptables -L -v