How to let kubelet communicate with apiserver by using HTTPS? v0.19 - kubernetes

Question

I deployed apiserver on master node (core01) with following conf:
core01> /opt/bin/kube-apiserver \
--insecure_bind_address=127.0.0.1 \
--insecure_port=8080 \
--kubelet_port=10250 \
--etcd_servers=http://core01:2379,http://core02:2379,http://core03:2379 \
--service-cluster-ip-range=10.1.0.0/16 \
--allow_privileged=false \
--logtostderr=true \
--v=5 \
--tls-cert-file="/var/run/kubernetes/apiserver_36kr.pem" \
--tls-private-key-file="/var/run/kubernetes/apiserver_36kr.key" \
--client-ca-file="/var/run/kubernetes/cacert.pem" \
--kubelet-certificate-authority="/var/run/kubernetes/cacert.pem" \
--kubelet-client-certificate="/var/run/kubernetes/kubelet_36kr.pem" \
--kubelet-client-key="/var/run/kubernetes/kubelet_36kr.key"
On minion node (core02), I can call api from HTTPS:
core02> curl https://core01:6443/api/v1/nodes --cert /var/run/kubernetes/kubelet_36kr.pem --key /var/run/kubernetes/kubelet_36kr.key
> GET /api/v1/nodes HTTP/1.1
> Host: core01:6443
> User-Agent: curl/7.42.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Sat, 27 Jun 2015 15:33:50 GMT
< Content-Length: 1577
<
{
"kind": "NodeList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/nodes",
"resourceVersion": "510078"
}, ....
However, I can not start kubelet on this minion. It always complain no credentials.
How can I make it work? Is there any doc on master <-> minion communication authentication? Could you please give me the best practice?
FYI, The command is following:
core02> /opt/bin/kubelet \
--logtostderr=true \
--v=0 \
--api_servers=https://core01:6443 \
--address=127.0.0.1 \
--port=10250 \
--allow-privileged=false \
--tls-cert-file="/var/run/kubernetes/kubelet_36kr.pem" \
--tls-private-key-file="/var/run/kubernetes/kubelet_36kr.key"
kubelet log is following:
W0627 23:34:03.646311 3004 server.go:460] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead.
W0627 23:34:03.646520 3004 server.go:422] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults.
I0627 23:34:03.646710 3004 manager.go:127] cAdvisor running in container: "/system.slice/sshd.service"
I0627 23:34:03.647292 3004 fs.go:93] Filesystem partitions: map[/dev/sda9:{mountpoint:/ major:0 minor:30} /dev/sda4:{mountpoint:/usr major:8 minor:4} /dev/sda6:{mountpoint:/usr/share/oem major:8 minor:6}]
I0627 23:34:03.648234 3004 manager.go:156] Machine: {NumCores:1 CpuFrequency:2399996 MemoryCapacity:1046294528 MachineID:29f94a4fad8b31668bd219ca511bdeb0 SystemUUID:4F4AF929-8BAD-6631-8BD2-19CA511BDEB0 BootID:fa1bea28-675e-4989-ad86-00797721a794 Filesystems:[{Device:/dev/sda9 Capacity:18987593728} {Device:/dev/sda4 Capacity:1031946240} {Device:/dev/sda6 Capacity:113229824}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:21474836480 Scheduler:cfq} 8:16:{Name:sdb Major:8 Minor:16 Size:1073741824 Scheduler:cfq}] NetworkDevices:[{Name:eth0 MacAddress:52:54:71:f6:fc:b8 Speed:0 Mtu:1500} {Name:flannel0 MacAddress: Speed:10 Mtu:1472}] Topology:[{Id:0 Memory:1046294528 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}]}
I0627 23:34:03.649934 3004 manager.go:163] Version: {KernelVersion:4.0.5 ContainerOsVersion:CoreOS 695.2.0 DockerVersion:1.6.2 CadvisorVersion:0.15.1}
I0627 23:34:03.651758 3004 plugins.go:69] No cloud provider specified.
I0627 23:34:03.651855 3004 docker.go:289] Connecting to docker on unix:///var/run/docker.sock
I0627 23:34:03.652877 3004 server.go:659] Watching apiserver
E0627 23:34:03.748954 3004 reflector.go:136] Failed to list *api.Pod: the server has asked for the client to provide credentials (get pods)
E0627 23:34:03.750157 3004 reflector.go:136] Failed to list *api.Node: the server has asked for the client to provide credentials (get nodes)
E0627 23:34:03.751666 3004 reflector.go:136] Failed to list *api.Service: the server has asked for the client to provide credentials (get services)
I0627 23:34:03.758158 3004 plugins.go:56] Registering credential provider: .dockercfg
I0627 23:34:03.856215 3004 server.go:621] Started kubelet
E0627 23:34:03.858346 3004 kubelet.go:662] Image garbage collection failed: unable to find data for container /
I0627 23:34:03.869739 3004 kubelet.go:682] Running in container "/kubelet"
I0627 23:34:03.869755 3004 server.go:63] Starting to listen on 127.0.0.1:10250
E0627 23:34:03.899877 3004 event.go:185] Server rejected event '&api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba23275ceda25", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"starting", Message:"Starting kubelet.", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016043, nsec:856189989, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016043, nsec:856189989, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
I0627 23:34:04.021297 3004 factory.go:226] System is using systemd
I0627 23:34:04.021790 3004 factory.go:234] Registering Docker factory
I0627 23:34:04.022241 3004 factory.go:89] Registering Raw factory
I0627 23:34:04.144065 3004 manager.go:946] Started watching for new ooms in manager
I0627 23:34:04.144655 3004 oomparser.go:183] oomparser using systemd
I0627 23:34:04.145379 3004 manager.go:243] Starting recovery of all containers
I0627 23:34:04.293020 3004 manager.go:248] Recovery completed
I0627 23:34:04.343829 3004 status_manager.go:56] Starting to sync pod status with apiserver
I0627 23:34:04.343928 3004 kubelet.go:1683] Starting kubelet main sync loop.
E0627 23:34:04.457765 3004 event.go:185] Server rejected event '&api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba232995c8213", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeReady", Message:"Node core02 status is now: NodeReady", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:452676115, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:452676115, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
E0627 23:34:04.659874 3004 event.go:185] Server rejected event '&api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba232a599cf8c", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeReady", Message:"Node core02 status is now: NodeReady", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:658020236, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:658020236, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)

Answer 1

The first two lines of the kubelet log file actually point to the underlying problem -- you aren't specifying any client credentials for the kubelet to connect to the master.
The --tls-cert-file and --tls-private-key-file arguments for the kubelet are used to configure the http server on the kubelet (if not specified, the kubelet will generate a self-signed certificate for its https endpoint). This certificate / key pair are not used as the client certificate presented to the master for authentication.
To specify credentials, there are two options: a kubeconfig file and a kubernetes_auth file. The later is deprecated, so I would recommend using a kubeconfig file.
Inside the kubeconfig file you need to specify either a bearer token or a client certificate that the kubelet should present to the apiserver. You can also specify the CA certificate for the apiserver (if you want the connection to be secure) or tell the kubelet to skip checking the certificate presented by the apiserver. Since you have certificates for the apiserver, I'd recommend adding the CA certificate to the kubeconfig file.
The kubeconfig file should look like:
apiVersion: v1
kind: Config
users:
- name: kubelet
user:
client-certificate-data: <base64-encoded-cert>
client-key-data: <base64-encoded-key>
clusters:
- name: local
cluster:
certificate-authority-data: <base64-encoded-ca-cert>
contexts:
- context:
cluster: local
user: kubelet
name: service-account-context
current-context: service-account-context
To generate the base64 encoded client cert, you should be able to run something like cat /var/run/kubernetes/kubelet_36kr.pem | base64. If you don't have the CA certificate handy, you can replace the certificate-authority-data: <base64-encoded-ca-cert> with insecure-skip-tls-verify: true.
If you put this file at /var/lib/kubelet/kubeconfig it should get picked up automatically. Otherwise, you can use the --kubeconfig argument to specify a custom location.

Answer 2

All credit to jnoller as he specifies the below commands on this issue.
He just made a typo because he runs kubectl config set-credentials twice
This is similar to Robert Bailey's accepted answer except you don't need to base64 anything and it makes it easy to script.
kubectl config set-cluster default-cluster --server=https://${MASTER} \
--certificate-authority=/path/to/ca.pem
kubectl config set-credentials default-admin \
--certificate-authority=/path/to/ca.pem \
--client-key=/path/to/admin-key.pem \
--client-certificate=/path/to/admin.pem
kubectl config set-context default-system --cluster=default-cluster --user=default-admin
kubectl config use-context default-system
The resulting config generated in ~/.kube/config looks like this:
apiVersion: v1
clusters:
- cluster:
certificate-authority: /etc/kubernetes/certs/ca.crt
server: https://kubernetesmaster
name: default-cluster
contexts:
- context:
cluster: default-cluster
user: default-admin
name: default-system
current-context: default-system
kind: Config
preferences: {}
users:
- name: default-admin
user:
client-certificate: /etc/kubernetes/certs/server.crt
client-key: /etc/kubernetes/certs/server.key

Answer 3

As Robert Bailey said, the main problem has to do with the client credentials to connect the master ("Could not load kubeconfig file /var/lib/kubelet/kubeconfig...").
Instead of create the kubeconfig file manually, I chose to generate it using kubectl tool.
Example from docs:
$ kubectl config set-credentials myuser --username=myusername --password=mypassword
$ kubectl config set-cluster local-server --server=http://localhost:8080
$ kubectl config set-context default-context --cluster=local-server --user=myuser
$ kubectl config use-context default-context
$ kubectl config set contexts.default-context.namespace mynamespace
Those commands will generate a config file in ~/.kube/config
Check the result with:
$ kubectl config view
Then I just created a symbolic link inside /var/lib/kubelet (default place) to my config file:
$ cd /var/lib/kubelet
$ sudo ln -s ~/.kube/config kubeconfig
This worked for me. I hope it works for you too.