k8s dashboard: Metric client health check failed - kubernetes

Question

I install the k8s dashboard use the following command:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.4/aio/deploy/recommended.yaml
then I watch the log of dashboard pod:
$ kubectl -n kubernetes-dashboard logs -f kubernetes-dashboard-665f4c5ff-wcrj9
2020/09/12 04:19:10 Metric client health check failed: an error on the server ("unknown") has prevented the request from succeeding (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2020/09/12 04:19:43 Metric client health check failed: an error on the server ("unknown") has prevented the request from succeeding (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2020/09/12 04:20:17 Metric client health check failed: an error on the server ("unknown") has prevented the request from succeeding (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2020/09/12 04:20:50 Metric client health check failed: an error on the server ("unknown") has prevented the request from succeeding (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2020/09/12 04:21:23 Metric client health check failed: an error on the server ("unknown") has prevented the request from succeeding (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2020/09/12 04:21:56 Metric client health check failed: an error on the server ("unknown") has prevented the request from succeeding (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2020/09/12 04:22:29 Metric client health check failed: an error on the server ("unknown") has prevented the request from succeeding (get services dashboard-metrics-scraper). Retrying in 30 seconds.
kubeadm version: 1.19
kubectl version: 1.19
Can anyone help me?

Answer 1

To give a bit of background information: once you install the Kubernetes Dashboard you install a Pod that provides the Dashboard as well as a Pod that is in charge of scraping Metrics from the Kubernetes Metrics API, the Dashboard Metrics Scraper. The dashboard delegates to the scraper, expecting to address it via its K8s Service: "dashboard-metrics-scraper".
In your case, this service can't be found. Do a "kubectl get service -n kubernetes-dashboard" to see whether the scraper service was deleted or renamed. If it was deleted, reapply the Dashboard installation yamls to recreate it.

Answer 2

I was unable to replicate your issue but here are some steps you can try to debug the problem:
Metric client health check failed: ... Retrying in 30 seconds error appears only one time in the dashboard's source code, when Health check fails.
HealthCheck itself is a proxy request to api-server.
Use following command to test if proxy is working correctly.
$ kubectl get --raw "/api/v1/namespaces/kubernetes-dashboard/services/dashboard-metrics-scraper/proxy/healthz"
it should return: URL: /healthz. If didn't, there is most probably sth wrong with the dashboard-metrics-scraper service or the pod. Make sure that service exists and the pod is running and ready.
If it's working for you (from cli), but it is still not working for kubernetes-dashboard, this mean that you should check kubernetes-dashboard's RBAC permissions. Make sure that kubernetes-dashboard has permissions to proxy.
The second error you are seeing:
{"level":"error","msg":"Error scraping node metrics: the server could not find the requested resource (get nodes.metrics.k8s.io)","time":"2020-09-13T02:52:38Z"}
indicates that you don't have a metrics server deployed in your cluster. Check metrics-server github repo for more information.

Answer 3

I'm on kubernetes 1.20.1-00 ubuntu 20.04. I got the
{"level":"error","msg":"Error scraping node metrics: the server could not find the requested resource (get nodes.metrics.k8s.io)","time":"2020-09-13T02:52:38Z"}
error because I deployed kubernetes dashboard with metric scraper prior to deploying metric server. After a day of running in that configuration I was still getting the "Error scraping node..." in my metric scraper pod logs.
I resolved it by scaling the the metric scraper deployment to 0 (zero) and then scaling it back to the desired no of pods (in my case 3).
The error message in the logs went away immediately once the metric scraper pods had spun up.
I'm not implying that this is the correct fix just an observation from seeing an identical error. It could caused by simply deploying metric server and Kubernetes dashboard in the wrong order as I did.