I am currently trying to set up a GKE cluster and to configure an HorizontalPodAutoscaler based on a custom metric (GPU consumption).
I have two node-pools and I want to horizontally scale them based on the average GPU consumption of each node_pool. I have configured two identical HPA like this:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: ner
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ner
minReplicas: 1
maxReplicas: 10
metrics:
- type: External
external:
metric:
name: kubernetes.io|container|accelerator|duty_cycle
target:
type: AverageValue
averageValue: 60
where I only replace the scaleTargetRef but it turns out that this metric seems to be aggregated at a cluster level. I have double checked that the scaleTargetRef are properly defined.
Is there a way to filter the metrics by container_name or node_pool? Any other suggestion would be awesome !
So I think you are looking for metrics for your k8 cluster especially by container_name or node_pool.
You have five types of metrics you can use in an HPA object(autoscaling/v2beta2)
k explain HorizontalPodAutoscaler.spec.metrics.type --api-version=autoscaling/v2beta2
Edit update
ContainerResource
External # Use this if the metrics not related to Kubernetes objects.
Object
Pods
Resource
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: ner
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ner
minReplicas: 1
maxReplicas: 10
metrics:
- type: ContainerResource
containerResource:
name: gpu
container: your-application-container
target:
type: Utilization
averageUtilization: 60
Edit Update
For GKP Autoscaling Deployments with Cloud Monitoring metrics
Related
I have three node mongodb cluster in GCP and it was deployed using MongoDB Community Operator. It is working fine. I need to setup auto scaling feature. I tried it with HPA Kubernetes object.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: mongodb-hpa
spec:
maxReplicas: 5
minReplicas: 3
scaleTargetRef:
apiVersion: apps/v1
kind: StatefulSet
name: mongodb-dev
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 80
HPA is collect stats and try to scale up/down. But created pod suddenly delete in scale up and again change to 3.
Is this done by operator ?
How I achieve this auto scaling feature?
I'm trying to understand how the Kubernetes HorizontalPodAutoscaler works.
Until now, I have used the following configuration:
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: my-deployment
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-deployment
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 50
This uses the targetCPUUtilizationPercentage parameter but I would like to use a metric for the memory percentage used, but I was not able to find any example.
Any hint?
I found also that there is this type of configuration to support multiple metrics, but the apiVersion is autoscaling/v2alpha1. Can this be used in a production environment?
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2alpha1
metadata:
name: WebFrontend
spec:
scaleTargetRef:
kind: ReplicationController
name: WebFrontend
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80
- type: Object
object:
target:
kind: Service
name: Frontend
metricName: hits-per-second
targetValue: 1k
Here is a manifest example for what you need, that includes Memory Metrics:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: web-servers
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: web-servers
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 20
- type: Resource
resource:
name: memory
target:
type: AverageValue
averageValue: 30Mi
An important thing to notice is that, as you can see, it uses the autoscaling/v2beta2 API version, so you need to follow all the previous instructions listed here.
Regarding the possibility to use the autoscaling/v2alpha1, yes, you can use it, as it includes support for scaling on memory and custom metrics as this URL specifies, but keep in mind that alpha versions are released for testing, as they are not final versions.
For more autoscaling/v2beta2 YAML’s examples and a deeper look into memory metrics, you can take a look at this thread.
I'm using a k8s HPA template for CPU and memory like below:
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: {{.Chart.Name}}-cpu
labels:
app: {{.Chart.Name}}
chart: {{.Chart.Name}}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{.Chart.Name}}
minReplicas: {{.Values.hpa.min}}
maxReplicas: {{.Values.hpa.max}}
targetCPUUtilizationPercentage: {{.Values.hpa.cpu}}
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: {{.Chart.Name}}-mem
labels:
app: {{.Chart.Name}}
chart: {{.Chart.Name}}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{.Chart.Name}}
minReplicas: {{.Values.hpa.min}}
maxReplicas: {{.Values.hpa.max}}
metrics:
- type: Resource
resource:
name: memory
target:
type: Utilization
averageValue: {{.Values.hpa.mem}}
Having two different HPA is causing any new pods spun up for triggering memory HPA limit to be immediately terminated by CPU HPA as the pods' CPU usage is below the scale down trigger for CPU.
It always terminates the newest pod spun up, which keeps the older pods around and triggers the memory HPA again, causing an infinite loop.
Is there a way to instruct CPU HPA to terminate pods with higher usage rather than nascent pods every time?
As per the suggestion in comments, using a single HPA solved my issue. I just had to move CPU HPA to same apiVersion as memory HPA.
Autoscaling based on multiple metrics/Custom metrics:-
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: nginx
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: nginx
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
- type: Resource
resource:
name: memory
target:
type: AverageValue
averageValue: 100Mi
When created, the Horizontal Pod Autoscaler monitors the nginx Deployment for average CPU utilization, average memory utilization, and (if you uncommented it) the custom packets_per_second metric. The Horizontal Pod Autoscaler autoscales the Deployment based on the metric whose value would create the larger autoscale event.
https://cloud.google.com/kubernetes-engine/docs/how-to/horizontal-pod-autoscaling#kubectl-apply
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: test
spec:
behavior:
scaleDown:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 10
periodSeconds: 15
scaleUp:
stabilizationWindowSeconds: 0
policies:
#-type: Percent
#value: 100
#periodSeconds: 15
- type: Pods
value: 5
periodSeconds: 15
maxReplicas: 30
minReplicas: 2
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: test
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
As per the Kubernetes official doc the HPA behavior is available for Kubernetes version v1.18 but GKE has it's own versioning. also it has api version "autoscaling/v2beta2" but the behavior is not supported.
GKE VERSION: 1.16.13-gke.1
Am I the only one to face this issue ?
Yes, you are right. GKE have it's own versioning. You can find more details here.
Note: The Kubernetes API is versioned separately from Kubernetes itself. Refer to the Kubernetes API documentation for information about Kubernetes API versioning.
Unfortunately, GKE is not supporting behavior parameter in apiVersion: autoscaling/v2beta2.
error: error validating "hpa.yaml": error validating data: ValidationError(HorizontalPodAutoscaler.spec): unknown field "behavior" in io.k8s.api.autoscaling.v2beta2.HorizontalPodAutoscalerSpec; if you choose to ignore these errors, turn validation off with --validate=false
However, it can be freely used with Kubeadm and Minikube with Kubernetes 1.18+.
There is already a Public Issue Tracker related to this issue. You can add yourself to CC in this PIT to get new updates related to this issue.
if you are on GKE and facing issue where enabled API are
autoscaling/v1
autoscaling/v2beta1
while GKE version is around 1.12 to 1.14 you wont be able to apply manifest of autoscaling/v2beta2 however you can apply same thing something like
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: core-deployment
namespace: default
spec:
maxReplicas: 9
minReplicas: 5
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: core-deployment
metrics:
- type: Resource
resource:
name: cpu
targetAverageValue: 500m
if you want based on utilization
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: core-deployment
namespace: default
spec:
maxReplicas: 9
minReplicas: 5
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: core-deployment
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80
Im currently trying to run an autoscaling demo using prometheus and the prometheus adapter, and i was wondering if there is a way to autoscale one of my deployments based on metrics that prometheus scrapes from another deployment.
What i have right now are 2 different deployments, kafka-consumer-application (which i want to scale) and kafka-exporter (which exposes the kafka metrics that I'll be using for scaling). I know that if I have both of them as containers in the same deployment the autoscaling works, but the issue is that the kafka-exporter also gets autoscaled and its not ideal, so i want to separate them. I tried with the following HPA but i could not get it to work:
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: consumer-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: kafka-consumer-application
minReplicas: 1
maxReplicas: 10
metrics:
- type: object
object:
target: kafka-exporter
metricName: "kafka_consumergroup_lag"
targetValue: 5
Im not sure if im doing something wrong or if this is just not something i can do, so any advice is appreciated.
Thanks!
Note: im running the adapter with this config:
rules:
default: false
resource: {}
custom:
- seriesQuery: 'kafka_consumergroup_lag'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
name:
matches: "kafka_consumergroup_lag"
as: "kafka_consumergroup_lag"
metricsQuery: 'avg_over_time(kafka_consumergroup_lag{topic="my-topic",consumergroup="we-consume"}[1m])'
``
In kubernetes documentation you can read:
Autoscaling on metrics not related to Kubernetes objects
Applications running on Kubernetes may need to autoscale based on metrics that don’t have an obvious relationship to any object in the Kubernetes cluster, such as metrics describing a hosted service with no direct correlation to Kubernetes namespaces. In Kubernetes 1.10 and later, you can address this use case with external metrics
So using external metrics, your HPA yaml could look like following:
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta2
metadata:
name: consumer-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: kafka-consumer-application
minReplicas: 1
maxReplicas: 10
metrics:
- type: External
external:
metric:
name: kafka_consumergroup_lag
#selector:
# matchLabels:
# topic: "my-topic"
target:
type: AverageValue
averageValue: 5
If you have more than one kafka-exporter you can use selector to filter it (source):
selector is the string-encoded form of a standard kubernetes label selector for the given metric When set, it is passed as an additional parameter to the metrics server for more specific metrics scoping. When unset, just the metricName will be used to gather metrics
Also have a look at this Stack question.