kube-dns remains in "ContainerCreating" - kubernetes

Question

I manually installed k8s-1.6.6 and I deployed calico-2.3(uses etcd-3.0.17 with kube-apiserver) and kube-dns on baremetal(ubuntu 16.04).
It dosen't have any problems without RBAC.
But, after applying RBAC by adding "--authorization-mode=RBAC" to kube-apiserver.
I couldn't apply kube-dns whose status remains in "ContainerCreating".
I checked "kubectl describe pod kube-dns.."
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
10m 10m 1 default-scheduler Normal Scheduled Successfully assigned kube-dns-1759312207-t35t3 to work01
9m 9m 1 kubelet, work01 Warning FailedSync Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: 8c2585b1b3170f220247a6abffb1a431af58060f2bcc715fe29e7c2144d19074
8m 8m 1 kubelet, work01 Warning FailedSync Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: c6962db6c5a17533fbee563162c499631a647604f9bffe6bc71026b09a2a0d4f
7m 7m 1 kubelet, work01 Warning FailedSync Error syncing pod, skipping: failed to "KillPodSandbox" for "f693931a-7335-11e7-aaa2-525400aa8825" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"kube-dns-1759312207-t35t3_kube-system\" network: CNI failed to retrieve network namespace path: Error: No such container: 9adc41d07a80db44099460c6cc56612c6fbcd53176abcc3e7bbf843fca8b7532"
5m 5m 1 kubelet, work01 Warning FailedSync Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: 4c2d450186cbec73ea28d2eb4c51497f6d8c175b92d3e61b13deeba1087e9a40
4m 4m 1 kubelet, work01 Warning FailedSync Error syncing pod, skipping: failed to "KillPodSandbox" for "f693931a-7335-11e7-aaa2-525400aa8825" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"kube-dns-1759312207-t35t3_kube-system\" network: CNI failed to retrieve network namespace path: Error: No such container: 12df544137939d2b8af8d70964e46b49f5ddec1228da711c084ff493443df465"
3m 3m 1 kubelet, work01 Warning FailedSync Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: c51c9d50dcd62160ffe68d891967d118a0f594885e99df3286d0c4f8f4986970
2m 2m 1 kubelet, work01 Warning FailedSync Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: 94533f19952c7d5f32e919c03d9ec5147ef63d4c1f35dd4fcfea34306b9fbb71
1m 1m 1 kubelet, work01 Warning FailedSync Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: 166a89916c1e6d63e80b237e5061fd657f091f3c6d430b7cee34586ba8777b37
16s 12s 2 kubelet, work01 Warning FailedSync (events with common reason combined)
10m 2s 207 kubelet, work01 Warning FailedSync Error syncing pod, skipping: failed to "CreatePodSandbox" for "kube-dns-1759312207-t35t3_kube-system(f693931a-7335-11e7-aaa2-525400aa8825)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-dns-1759312207-t35t3_kube-system(f693931a-7335-11e7-aaa2-525400aa8825)\" failed: rpc error: code = 2 desc = NetworkPlugin cni failed to set up pod \"kube-dns-1759312207-t35t3_kube-system\" network: the server does not allow access to the requested resource (get pods kube-dns-1759312207-t35t3)"
10m 1s 210 kubelet, work01 Normal SandboxChanged Pod sandbox changed, it will be killed and re-created.
my kubelet
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=/bin/mkdir -p /var/log/containers
ExecStartPre=/bin/mkdir -p /etc/cni/net.d
ExecStartPre=/bin/mkdir -p /opt/cni/bin
ExecStart=/usr/local/bin/kubelet \
--api-servers=http://127.0.0.1:8080 \
--allow-privileged=true \
--pod-manifest-path=/etc/kubernetes/manifests \
--kubeconfig=/var/lib/kubelet/kubeconfig \
--cluster-dns=10.3.0.10 \
--cluster-domain=cluster.local \
--register-node=true \
--network-plugin=cni \
--cni-conf-dir=/etc/cni/net.d \
--cni-bin-dir=/opt/cni/bin \
--container-runtime=docker
my kube-apiserver
apiVersion: v1
kind: Pod
metadata:
name: kube-apiserver
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-apiserver
image: kube-apiserver:v1.6.6
command:
- kube-apiserver
- --bind-address=0.0.0.0
- --etcd-servers=http://127.0.0.1:2379
- --allow-privileged=true
- --service-cluster-ip-range=10.3.0.0/16
- --secure-port=6443
- --advertise-address=172.30.1.10
- --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota
- --tls-cert-file=/srv/kubernetes/apiserver.pem
- --tls-private-key-file=/srv/kubernetes/apiserver-key.pem
- --client-ca-file=/srv/kubernetes/ca.pem
- --service-account-key-file=/srv/kubernetes/apiserver-key.pem
- --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP
- --anonymous-auth=false
- --authorization-mode=RBAC
- --token-auth-file=/srv/kubernetes/known_tokens.csv
- --basic-auth-file=/srv/kubernetes/basic_auth.csv
- --storage-backend=etcd3
livenessProbe:
httpGet:
host: 127.0.0.1
port: 8080
path: /healthz
scheme: HTTP
initialDelaySeconds: 15
timeoutSeconds: 15
ports:
- name: https
hostPort: 6443
containerPort: 6443
- name: local
hostPort: 8080
containerPort: 8080
volumeMounts:
- name: srvkube
mountPath: "/srv/kubernetes"
readOnly: true
- name: etcssl
mountPath: "/etc/ssl"
readOnly: true
volumes:
- name: srvkube
hostPath:
path: "/srv/kubernetes"
- name: etcssl
hostPath:
path: "/etc/ssl"

Answer 1

I found the cause.
This issue is not related to kube-dns.
I just missed out applying ClusterRole/ClusterRoleBinding, before deplying calico