How to diagnose Kubernetes not responding on API - kubernetes

Question

Aug 01 12:49:49 master kubelet[18344]: E0801 12:49:49.534129 18344 eviction_manager.go:243] eviction manager: failed to get get summary stats: failed to get node info: node "master" not found
Aug 01 12:49:49 master kubelet[18344]: I0801 12:49:49.925152 18344 kubelet_node_status.go:269] Setting node annotation to enable volume controller attach/detach
Aug 01 12:49:49 master kubelet[18344]: I0801 12:49:49.927988 18344 kubelet_node_status.go:79] Attempting to register node master
Aug 01 12:49:49 master kubelet[18344]: E0801 12:49:49.928908 18344 kubelet_node_status.go:103] Unable to register node "master" with API server: Post https://192.168.0.33:6443/api/v1/nodes: dial tcp 192.168.
Aug 01 12:49:50 master kubelet[18344]: E0801 12:49:50.004760 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://192.168.0.33:6443/api/v1/nodes?fieldSel
Aug 01 12:49:50 master kubelet[18344]: E0801 12:49:50.006130 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://192.168.0.33:6443/api/v1/services?li
Aug 01 12:49:50 master kubelet[18344]: E0801 12:49:50.008020 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.0.33:6443/api/v1/pods?fi
Aug 01 12:49:50 master kubelet[18344]: I0801 12:49:50.729713 18344 kubelet_node_status.go:269] Setting node annotation to enable volume controller attach/detach
Aug 01 12:49:50 master kubelet[18344]: I0801 12:49:50.733513 18344 kubelet_node_status.go:79] Attempting to register node master
Aug 01 12:49:50 master kubelet[18344]: E0801 12:49:50.734866 18344 kubelet_node_status.go:103] Unable to register node "master" with API server: Post https://192.168.0.33:6443/api/v1/nodes: dial tcp 192.168.
Aug 01 12:49:51 master kubelet[18344]: E0801 12:49:51.006313 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://192.168.0.33:6443/api/v1/nodes?fieldSel
Aug 01 12:49:51 master kubelet[18344]: E0801 12:49:51.009443 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://192.168.0.33:6443/api/v1/services?li
Aug 01 12:49:51 master kubelet[18344]: E0801 12:49:51.010510 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.0.33:6443/api/v1/pods?fi
lines 22914-22948/22948
How can I diagnose this further?
telnet 192.168.0.33 6443
Trying 192.168.0.33...
telnet: Unable to connect to remote host: Connection refused
systemctl status kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Wed 2018-08-01 12:49:48 EDT; 3min 47s ago
Docs: http://kubernetes.io/docs/
Main PID: 18344 (kubelet)
Tasks: 13 (limit: 4915)
Memory: 39.4M
CPU: 4.091s
CGroup: /system.slice/kubelet.service
└─18344 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cn
Aug 01 12:53:33 master kubelet[18344]: E0801 12:53:33.522282 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://192.168.0.33:6443/api/v1/services?limit=500&resourceVersion=0
Aug 01 12:53:33 master kubelet[18344]: E0801 12:53:33.527787 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://192.168.0.33:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmas
Aug 01 12:53:33 master kubelet[18344]: E0801 12:53:33.537549 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.0.33:6443/api/v1/pods?fieldSelector=spec.nodeName
Aug 01 12:53:34 master kubelet[18344]: I0801 12:53:34.051830 18344 kubelet_node_status.go:269] Setting node annotation to enable volume controller attach/detach
Aug 01 12:53:34 master kubelet[18344]: E0801 12:53:34.523429 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://192.168.0.33:6443/api/v1/services?limit=500&resourceVersion=0
Aug 01 12:53:34 master kubelet[18344]: E0801 12:53:34.530208 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://192.168.0.33:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmas
Aug 01 12:53:34 master kubelet[18344]: E0801 12:53:34.538744 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.0.33:6443/api/v1/pods?fieldSelector=spec.nodeName
Aug 01 12:53:35 master kubelet[18344]: E0801 12:53:35.524380 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://192.168.0.33:6443/api/v1/services?limit=500&resourceVersion=0
Aug 01 12:53:35 master kubelet[18344]: E0801 12:53:35.531218 18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://192.168.0.33:6443/api/v1/nodes?fieldSelecto
Think I broke something...

Answer 1

I managed to find the issue by looking at the logs of the apiserver. In my particular case I found them via the below:
osboxes#master:/var/log/pods$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
84ec4c4de5b2 k8s.gcr.io/pause:3.1 "/pause" About an hour ago Up About an hour k8s_POD_kube-apiserver-master_kube-system_fdb932ada5768a1891d839f8cf2306a9_0
ea84510e26be 272b3a60cd68 "kube-scheduler --..." About an hour ago Up About an hour k8s_kube-scheduler_kube-scheduler-master_kube-system_537879acc30dd5eff5497cb2720a6d64_1
d36af3896c3b 52096ee87d0e "kube-controller-m..." About an hour ago Up About an hour k8s_kube-controller-manager_kube-controller-manager-master_kube-system_f31b9af1b177e27c1d4ace1fa4d37d83_1
a53569da6f29 f0fad859c909 "/opt/bin/flanneld..." 4 hours ago Up 4 hours k8s_kube-flannel_kube-flannel-ds-94xm7_kube-system_7e32bed6-9585-11e8-a2f7-080027a08edc_0
9a31712003ac k8s.gcr.io/pause:3.1 "/pause" 4 hours ago Up 4 hours k8s_POD_kube-flannel-ds-94xm7_kube-system_7e32bed6-9585-11e8-a2f7-080027a08edc_0
276b6107a4b3 d5c25579d0ff "/usr/local/bin/ku..." 4 hours ago Up 4 hours k8s_kube-proxy_kube-proxy-2hcfx_kube-system_e7bebd1e-9584-11e8-a2f7-080027a08edc_0
87d2c5657240 k8s.gcr.io/pause:3.1 "/pause" 4 hours ago Up 4 hours k8s_POD_kube-proxy-2hcfx_kube-system_e7bebd1e-9584-11e8-a2f7-080027a08edc_0
d04c7669f27c b8df3b177be2 "etcd --advertise-..." 4 hours ago Up 4 hours k8s_etcd_etcd-master_kube-system_d3a295b6d0da8bbfe30c134cab4d030b_0
6c174ea2f877 k8s.gcr.io/pause:3.1 "/pause" 4 hours ago Up 4 hours k8s_POD_kube-scheduler-master_kube-system_537879acc30dd5eff5497cb2720a6d64_0
e5603c531a1c k8s.gcr.io/pause:3.1 "/pause" 4 hours ago Up 4 hours k8s_POD_kube-controller-manager-master_kube-system_f31b9af1b177e27c1d4ace1fa4d37d83_0
cf6ee3a78089 k8s.gcr.io/pause:3.1 "/pause" 4 hours ago Up 4 hours k8s_POD_etcd-master_kube-system_d3a295b6d0da8bbfe30c134cab4d030b_0
osboxes#master:/var/log/pods$ tail -f fdb932ada5768a1891d839f8cf2306a9/kube-apiserver/29.log
tail: cannot open 'fdb932ada5768a1891d839f8cf2306a9/kube-apiserver/29.log' for reading: Permission denied
tail: no files remaining
osboxes#master:/var/log/pods$ tail -f fdb932ada5768a1891d839f8cf2306a9/kube-apiserver/29.log
tail: cannot open 'fdb932ada5768a1891d839f8cf2306a9/kube-apiserver/29.log' for reading: Permission denied
tail: no files remaining
osboxes#master:/var/log/pods$ sudo tail -f fdb932ada5768a1891d839f8cf2306a9/kube-apiserver/29.log
{"log":" --tls-private-key-file string File containing the default x509 private key matching --tls-cert-file.\n","stream":"stderr","time":"2018-08-01T16:56:17.315279517Z"}
{"log":" --tls-sni-cert-key namedCertKey A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: \"example.crt,example.key\" or \"foo.crt,foo.key:*.foo.com,foo.com\". (default [])\n","stream":"stderr","time":"2018-08-01T16:56:17.315282242Z"}
{"log":" --token-auth-file string If set, the file that will be used to secure the secure port of the API server via token authentication.\n","stream":"stderr","time":"2018-08-01T16:56:17.31528618Z"}
{"log":" -v, --v Level log level for V logs\n","stream":"stderr","time":"2018-08-01T16:56:17.315288985Z"}
{"log":" --version version[=true] Print version information and quit\n","stream":"stderr","time":"2018-08-01T16:56:17.31529159Z"}
{"log":" --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging\n","stream":"stderr","time":"2018-08-01T16:56:17.315296449Z"}
{"log":" --watch-cache Enable watch caching in the apiserver (default true)\n","stream":"stderr","time":"2018-08-01T16:56:17.315299265Z"}
{"log":" --watch-cache-sizes strings List of watch cache sizes for every resource (pods, nodes, etc.), comma separated. The individual override format: resource[.group]#size, where resource is lowercase plural (no version), group is optional, and size is a number. It takes effect when watch-cache is enabled. Some resources (replicationcontrollers, endpoints, nodes, pods, services, apiservices.apiregistration.k8s.io) have system defaults set by heuristics, others default to default-watch-cache-size\n","stream":"stderr","time":"2018-08-01T16:56:17.31530237Z"}
{"log":"\n","stream":"stderr","time":"2018-08-01T16:56:17.315306007Z"}
{"log":"error: loading audit policy file: failed to read file path \"/etc/kubernetes/audit.yaml\": open /etc/kubernetes/audit.yaml: no such file or directory\n","stream":"stderr","time":"2018-08-01T16:56:17.315480703Z"}
q^C