I am working on a use case related to Horizontal Pod Autoscaling. I am able to fetch memory and CPU usage from the metrics server in order to decide on scale out (found this after reading multiple blogs).
I wish to know if any of the other standard metrics such as throughput, disk usage, resource consumption etc. can be fetched from the metrics server. Have not been able to find anything on the same.
You can find all available metrics from Documentation of kube-state-metrics for all available resources.
Also, as mentioned in Horizontal Pod Autoscaler documentation you can use custom metrics
The Horizontal Pod Autoscaler automatically scales the number of pods
in a replication controller, deployment or replica set based on
observed CPU utilization (or, with custom metrics support, on some
other application-provided metrics). Note that Horizontal Pod
Autoscaling does not apply to objects that can’t be scaled, for
example, DaemonSets.
Related
I have:
deployments of services A and B in k8s
Prometheus stack
I wanna scale service A when metric m1 of service B is changed.
Solutions which I found and not suitable more or less:
I can define HPA for service A with the following part of spec:
- type: Object
object:
metric:
name: m1
describedObject:
apiVersion: api/v1
kind: Pod
name: certain-pod-of-service-B
current:
value: 10k
Technically, it will work. But it's not suitable for dynamic nature of k8s.
Also I can't use pods metric (metrics: - type: Pods pods:) in HPA cause it will request m1 metric for pods of service A (which obviously doesn't have this)
Define custom metric in prometheus-adapter which query m1 metric from pods of service B. It's more suitable, but looks like workaround cause I already have a metric m1
The same for external metrics
I feel that I miss something cause it doesn't seem like a non realistic case :)
So, advise me please how to scale one service by metric of another in k8s?
I decided to provide a Community Wiki answer that may help other people facing a similar issue.
The Horizontal Pod Autoscaler is a Kubernetes feature that allows to scale applications based on one or more monitored metrics.
As we can find in the Horizontal Pod Autoscaler documentation:
The Horizontal Pod Autoscaler automatically scales the number of Pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics).
There are three groups of metrics that we can use with the Horizontal Pod Autoscaler:
resource metrics: predefined resource usage metrics (CPU and
memory) of pods and nodes.
custom metrics: custom metrics associated with a Kubernetes
object.
external metrics: custom metrics not associated with a
Kubernetes object.
Any HPA target can be scaled based on the resource usage of the pods (or containers) in the scaling target. The CPU utilization metric is a resource metric, you can specify other resource metrics besides CPU (e.g. memory). This seems to be the easiest and most basic method of scaling, but we can use more specific metrics by using custom metrics or external metrics.
There is one major difference between custom metrics and external metrics (see: Custom and external metrics for autoscaling workloads):
Custom metrics and external metrics differ from each other:
A custom metric is reported from your application running in Kubernetes.
An external metric is reported from an application or service not running on your cluster, but whose performance impacts your Kubernetes application.
All in all, in my opinion it is okay to use custom metrics in the case above,
I did not find any other suitable way to accomplish this task.
I'm new to Kubernetes, and I have a doubt about horizontal pod autoscaling. Can I apply HPA with just one node ? If so, what are the benefits of HPA using one node only ?
If I use the metrics below, the target says averageUtilization 50% of cpu. Does that imply that I need a new node after the value is reached ?
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
Any advice ?
Here are some notes that might help you to sort things out:
Yes, you can use horizontal pod autoscaling on one node only.
The benefit of running multiple pods is parallelism: More instances of your app can handle more load - in that regard it doesn't matter if you run the pods on one or several nodes.
But if you have more pods of your application, you might end up in a situation where you need additional nodes to handle the load.
To determine out how many pods can run on one node, kubernetes uses the concept of resource limits and requests.
HPA will spawn new pods if the actual utilization of your pod hits the target utilization - but it doesn't take care that your node can handle more pods - you need to configure this using resource limits and requests.
Scaling up the nodes of your cluster is not handled by HPA, you need to use the kubernetes cluster autoscaler for that.
I like to see how my services will work on kubernethes so I can optimize my code and set good values for request/limit on both CPU and memory.
To do that I have tried kubectl top bit ot only gives me the current usage.
kubectl top pod podname
How do I get the init, min and max usage?
If it is not possible to get all those values, is there any way to get max usage?
In order to see stats you may want to use one of these monitoring tools:
cAdvisor
Container Advisor is a great monitoring tool that provides
container-level metrics and exposes resource usage and performance
data from running containers. It provides quick insight into CPU
usage, memory usage, and network receive/transmit of running
containers. cAdvisor is embedded into the kubelet, hence you can
scrape the kubelet to get container metrics, store the data in a
persistent time-series store like Prometheus/InfluxDB, and then
visualize it via Grafana.
Metrics Server
Metrics Server is a cluster-wide aggregator of resource usage data and
collects basic metrics like CPU and memory usage for Kubernetes nodes,
pods, and containers. It’s used by Horizontal Pod Autoscaler and the
Kubernetes dashboard itself, and users can access these metrics
directly by using the kubectl top command. Metrics Server replaces
Heapster as the primary metrics aggregator in the cluster, which has
been marked as deprecated in the newer version of Kubernetes.
Node Exporter
Node Exporter is the Prometheus exporter for hardware and operating
system metrics. It allows you to monitor node-level metrics such as
CPU, memory, filesystem space, network traffic, and other monitoring
metrics, which Prometheus scraps from a running node exporter
instance. You can then visualize these metrics in Grafana.
Kube-State-Metrics
Kube-state-metrics is an add-on agent that listens to the Kubernetes
API server. It generates metrics about the state of the Kubernetes
objects inside the cluster like deployments, replica sets, nodes, and
pods.
Metrics generated by kube-state-metrics are different from resource
utilization metrics, which are primarily geared more towards CPU,
memory, and network usage. Kube-state-metrics expose critical metrics
about the condition of your Kubernetes cluster:
Resource requests and limits
Number of objects–nodes, pods, namespaces, services, deployments
Number of pods in a running/terminated/failed state
Prometheus
Prometheus is a free software application used for event monitoring
and alerting. It records real-time metrics in a time series database
built using a HTTP pull model, with flexible queries and real-time
alerting
You can visualize Prometheus monitoring data with Grafana
and its dashboard collection.
You can find detailed Monitor Your Kubernetes Cluster With Prometheus and Grafana instruction how to use them together
I am newbie to Kubernetes world. Please excuse if I am getting anything wrong.
I understand that pod replication is handled by k8s itself. We can also set cpu and memory usage for pods. But is it possible to change replication criteria based on memory and cpu usage? For example if I want to a pod to replicate when its memory/cpu usage reaches 70%.
Can we do it using metrics collected by Prometheus etc ?
You can use horizontal pod autoscaler. From the docs
The Horizontal Pod Autoscaler automatically scales the number of Pods
in a replication controller, deployment, replica set or stateful set
based on observed CPU utilization (or, with custom metrics support, on
some other application-provided metrics). Note that Horizontal Pod
Autoscaling does not apply to objects that can't be scaled, for
example, DaemonSets.
The Horizontal Pod Autoscaler is implemented as a Kubernetes API
resource and a controller. The resource determines the behavior of the
controller. The controller periodically adjusts the number of replicas
in a replication controller or deployment to match the observed
average CPU utilization to the target specified by user
An example from the doc
The following command will create a Horizontal Pod Autoscaler that maintains between 1 and 10 replicas of the Pods. HPA will increase and decrease the number of replicas to maintain an average CPU utilization across all Pods of 50%.
kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
I have a pod running with two containers. The actual application is running in one of the containers (container-app) and the other one is the proxy container (container-proxy). I enabled the Horizontal Pod Autoscaler (HPA) for CPU usage percentage but as it states in HPA documentation, both of the container metrics are put in the calculation.
I want to exclude the CPU metrics of container-proxy from HPA calculation because I want only application container to be the scaling element for the pod.
Is there any way to exclude some containers metrics from HPA calculation for multi-container pods?
The cluster autoscaler works on a per-node pool basis. Horizontal Pod Autoscaler monitors CPU utilization of the pods and scales the number of replicas automatically. It provides immediate efficiency and capacity when needed, operates within user-defined minimum/maximum bounds, and allows users to set it and forget it. The design of the horizontal autoscaler is for pods not for the individual container.
HPA calculates pod cpu utilization as total cpu usage of all containers in pod divided by total request. It does not exclude containers metrics from HPA calculation if multiple containers are inside the pod.
Kubernetes 1.20+ supports container metrics, so as to target the utilisation per container, which would allow excluding a specific container of a pod from being considered.
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#container-resource-metrics
type: ContainerResource
containerResource:
name: cpu
container: application
target:
type: Utilization
averageUtilization: 60
Its an alpha feature though, so not available without turning on alpha features in Kubernetes.