HPA not able to fetch metrics from Prometheus in Kubernetes - kubernetes

Question

I have a two node Kubernetes cluster i.e one master node and two worker nodes. For monitoring purpose, I have deployed Prometheus and Grafana. Now, I want to autoscale pods based on CPU usage. But even after configuring Grafana and Prometheus, I am getting the following error ---
Name: php-apache
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Mon, 17 Jun 2019 12:33:01 +0530
Reference: Deployment/php-apache
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 50%
Min replicas: 1
Max replicas: 10
Deployment pods: 1 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 112s (x12408 over 2d4h) horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Can anybody let me know why Kubernetes is not fetching metrics from Prometheus ?

Answer 1

Kubernetes retrieves metrics from either the metrics.k8s.io API (normally implemented by the metrics-server which can be seperatly installed) or the custom.metrics.k8s.io API (which can be any type of metric and is normally provided by third parties). To use prometheus in HPA for kubernetes the Prometheus Adapter for the custom metrics API needs to be installed.
A walkthrough for the setup can be found here.

Answer 2

heapster is now depracted : https://github.com/kubernetes-retired/heapster
To enable auto-scaling on your cluster you can use HPA(horizontal pod auto-scaler) and you can also install metrics server to check all metrics.
To install metrics server on kubernetes you can follow this guide also :
amazon : https://docs.aws.amazon.com/eks/latest/userguide/metrics-server.html
https://github.com/kubernetes-incubator/metrics-server
https://medium.com/#cagri.ersen/kubernetes-metrics-server-installation-d93380de008

Answer 3

You don't need custom metrics to use HPA for auto-scaling pods based on their CPU usage.
As #Blokje5 mentioned earlier, you just need to install 'kube-state-metrics'.
The most convenient way to do it is with a dedicated helm chart (kube-state-metrics).
Hint: use override parameters with 'helm install' to create ServiceMonitor object for 'kube-state-metrics' Pod, to allow Prometheus to discover a new target for metrics scraping, e.g.:
helm install stable/kube-state-metrics --set prometheus.monitor.enabled=true
Remark: Pay attention to the 'serviceMonitorSelector' defined in your existing Prometheus resource object/configuration, so that it matches the ServiceMonitor definition for 'kube-state-metrics'. This is to make available Pods' metrics in Prometheus console.