This is my hpa yaml file:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: mysql-hpa
spec:
maxReplicas: 2
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: StatefulSet
name: mysql
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
The problem is that while i send requests to my app with jmeter, hpa creates a 2nd pod but doesn't share the traffic to both pods, except a few times!
You can see it to the photos below..
Ιf i create a pod with 2 replicas (by yaml file) without hpa, traffic is devided normally!
Any idea?
i have another pod with 12 containers and the hpa works fine.
Related
I have three node mongodb cluster in GCP and it was deployed using MongoDB Community Operator. It is working fine. I need to setup auto scaling feature. I tried it with HPA Kubernetes object.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: mongodb-hpa
spec:
maxReplicas: 5
minReplicas: 3
scaleTargetRef:
apiVersion: apps/v1
kind: StatefulSet
name: mongodb-dev
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 80
HPA is collect stats and try to scale up/down. But created pod suddenly delete in scale up and again change to 3.
Is this done by operator ?
How I achieve this auto scaling feature?
I am using prometheus kubernetes adapter to scale up application.
I have service nginx running at port 8080 displaying metric requests.
I use hpa.yaml as follows:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: nginx
minReplicas: 1
maxReplicas: 10
metrics:
- type: Pods
pods:
metricName: requests
targetAverageValue: 10
This is working... But I want to scale up different deployment using requests as metric so I change my hpa to
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: nginx2
minReplicas: 1
maxReplicas: 10
metrics:
- type: Object
object:
target:
kind: Service
name: nginx
metricName: requests
targetValue: 10
But it is giving error as " invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric requests: no metrics returned from custom metrics API"
"the HPA was unable to compute the replica count: unable to get metric requests: no metrics returned from custom metrics API"
So I can not scale up other application base on nginx. I want to use nginx service to scale up other deployments.
Any Suggestions... Where I am going wrong?
I am currently trying to set up a GKE cluster and to configure an HorizontalPodAutoscaler based on a custom metric (GPU consumption).
I have two node-pools and I want to horizontally scale them based on the average GPU consumption of each node_pool. I have configured two identical HPA like this:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: ner
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ner
minReplicas: 1
maxReplicas: 10
metrics:
- type: External
external:
metric:
name: kubernetes.io|container|accelerator|duty_cycle
target:
type: AverageValue
averageValue: 60
where I only replace the scaleTargetRef but it turns out that this metric seems to be aggregated at a cluster level. I have double checked that the scaleTargetRef are properly defined.
Is there a way to filter the metrics by container_name or node_pool? Any other suggestion would be awesome !
So I think you are looking for metrics for your k8 cluster especially by container_name or node_pool.
You have five types of metrics you can use in an HPA object(autoscaling/v2beta2)
k explain HorizontalPodAutoscaler.spec.metrics.type --api-version=autoscaling/v2beta2
Edit update
ContainerResource
External # Use this if the metrics not related to Kubernetes objects.
Object
Pods
Resource
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: ner
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ner
minReplicas: 1
maxReplicas: 10
metrics:
- type: ContainerResource
containerResource:
name: gpu
container: your-application-container
target:
type: Utilization
averageUtilization: 60
Edit Update
For GKP Autoscaling Deployments with Cloud Monitoring metrics
I'm using a k8s HPA template for CPU and memory like below:
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: {{.Chart.Name}}-cpu
labels:
app: {{.Chart.Name}}
chart: {{.Chart.Name}}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{.Chart.Name}}
minReplicas: {{.Values.hpa.min}}
maxReplicas: {{.Values.hpa.max}}
targetCPUUtilizationPercentage: {{.Values.hpa.cpu}}
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: {{.Chart.Name}}-mem
labels:
app: {{.Chart.Name}}
chart: {{.Chart.Name}}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{.Chart.Name}}
minReplicas: {{.Values.hpa.min}}
maxReplicas: {{.Values.hpa.max}}
metrics:
- type: Resource
resource:
name: memory
target:
type: Utilization
averageValue: {{.Values.hpa.mem}}
Having two different HPA is causing any new pods spun up for triggering memory HPA limit to be immediately terminated by CPU HPA as the pods' CPU usage is below the scale down trigger for CPU.
It always terminates the newest pod spun up, which keeps the older pods around and triggers the memory HPA again, causing an infinite loop.
Is there a way to instruct CPU HPA to terminate pods with higher usage rather than nascent pods every time?
As per the suggestion in comments, using a single HPA solved my issue. I just had to move CPU HPA to same apiVersion as memory HPA.
Autoscaling based on multiple metrics/Custom metrics:-
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: nginx
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: nginx
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
- type: Resource
resource:
name: memory
target:
type: AverageValue
averageValue: 100Mi
When created, the Horizontal Pod Autoscaler monitors the nginx Deployment for average CPU utilization, average memory utilization, and (if you uncommented it) the custom packets_per_second metric. The Horizontal Pod Autoscaler autoscales the Deployment based on the metric whose value would create the larger autoscale event.
https://cloud.google.com/kubernetes-engine/docs/how-to/horizontal-pod-autoscaling#kubectl-apply
I am running an EKS cluster and I have a HorizontalPodAutoscaler created for autoscaling number of pods based on average CPU utilisation.
How to do the same for Average memory utilization?
Suppose all of the pods running in an EKS clusters, have used average of 70% of memory they are allocated (using resources), then the deployment should be autoscaled.
How to do this? Is creating a custom metric in CloudWatch the only way?
Even if cloudWatch is the only way, how to do that? Is there a specific documentation or tutorial or blog that does this?
Please try the below HPA configuration object.
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-hpa
namespace: default
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: nginx
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: memory
targetAverageUtilization: 70
and apply the object using kubectl apply