CoreDNS logs report unauthorized - kubernetes

Question

We deployed new Kubernetes cluster, and it has 2 pods for Coredns.
$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-74ff55c5b-7v9bd 0/1 Running 0 7h22m
coredns-74ff55c5b-tfpqb 0/1 Running 0 7h23m
There suppose to be 2 replicas, but 0 READY.
When I check the logs to find the reason for not running, I see there are many Unauthorized errors.
$ for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --tail 20 --namespace=kube-system $p; done
E0323 00:58:04.393710 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:58:34.184217 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:58:51.873269 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:59:00.966217 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:59:23.151006 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:59:47.362409 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
E0323 00:59:48.563791 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Unauthorized
E0323 00:59:56.278764 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:58:07.504557 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:58:24.948534 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:58:33.605013 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:58:56.471477 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:59:20.436808 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
E0323 00:59:21.200346 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
E0323 00:59:29.597663 1 reflector.go:178] pkg/mod/k8s.io/client-go#v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
When I try to find some help on net, I found out, it use coredns service user. I check for all that roles and bindings.
SERVICE ACCOUNT
$ kubectl get sa coredns -n kube-system -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2021-03-03T15:17:38Z"
name: coredns
namespace: kube-system
resourceVersion: "297"
uid: 13633498-2e6b-4ac4-bb34-f2d5c9e4d262
secrets:
- name: coredns-token-sg7p9
TOKEN SECRET
$ kubectl get secret coredns-token-sg7p9 -n kube-system
NAME TYPE DATA AGE
coredns-token-sg7p9 kubernetes.io/service-account-token 3 19d
CLUSTER ROLE
$ kubectl get clusterrole system:coredns -n kube-system -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2021-03-03T15:17:38Z"
managedFields:
- apiVersion: rbac.authorization.k8s.io/v1
fieldsType: FieldsV1
fieldsV1:
f:rules: {}
manager: kubeadm
operation: Update
time: "2021-03-03T15:17:38Z"
name: system:coredns
resourceVersion: "292"
uid: 35adc9a3-7415-4498-81b2-a4eab50882b1
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
CLUSTER ROLE BINDINGS
$ kubectl get clusterrolebindings system:coredns -n kube-system -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: "2021-03-03T15:17:38Z"
managedFields:
- apiVersion: rbac.authorization.k8s.io/v1
fieldsType: FieldsV1
fieldsV1:
f:roleRef:
f:apiGroup: {}
f:kind: {}
f:name: {}
f:subjects: {}
manager: kubeadm
operation: Update
time: "2021-03-03T15:17:38Z"
name: system:coredns
resourceVersion: "293"
uid: 2d47c2cb-6641-4a62-b867-8a598ac3923a
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:
- kind: ServiceAccount
name: coredns
namespace: kube-system
From unauthorized error, I can predict it might be related to token, like token expired and not renewed. I was trying to find help on net, for how to renew token for Coredns, but didn't find any help.
I might be doing something wrong, but can't find that.
There is help available if Pod is not in Running state, but not for unauthorized after running pod.