I used the following metrics inside hpa
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: app-svc-hpa
namespace: default
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: app-svc
minReplicas: 1
maxReplicas: 1000
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
- type: Pods
pods:
metric:
name: packets-per-second
target:
type: AverageValue
averageValue: 1k
But the hpa is unable to get the metrics
Warning FailedGetPodsMetric 14s (x6 over 1m) horizontal-pod-autoscaler unable to get metric packets-per-second: unable to fetch metrics from custom metrics API: the server could not find the descriptor for metric custom.googleapis.com/packets-per-second: googleapi: Error 403: Permission monitoring.metricDescriptors.get denied (or the resource may not exist)., forbidden
I am running the pods on a dedicated nodepool and each nodes is running under a service account.
The service account does have these iam roles
Monitoring Viewere,
Monitoring Metrics Writer
Unsure how to fix this error. Any pointers are greatly appreciated. Thanks.
I had a cluster with workload identity enabled. Apparently when a cluster has workload identity enabled, the metrics fetch was failing.
1) I had to install the custom stack driver adapter and create the custom metric as pointed by David Kruk in his comments
2) I had to add the hostNetwork:true in the custom stackdriver adapter deployment pod spec. The issue is mentioned here in github repository for csa
With these two updates, the autoscaler works as expected.
Related
suppose we have below HPA(HorizontalPodAutoscaler) deployed in the demo namespace, and multiple pods (POD-A,POD-B) in this demo namespace have the same metric "istio_requests_per_second", How does the HPA determine the metric "istio_requests_per_second" from which pod should be used? Or every POD with this metric will be evaluate against the HPA target?
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: httpbin
spec:
minReplicas: 1
maxReplicas: 5
metrics:
- type: Pods
pods:
metric:
name: istio_requests_per_second
target:
type: AverageValue
averageValue: "10"
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: httpbin
test...
If you're using prometheus then its the adapter thats correlating between k8's pod name and what metric value to return. Basically the HPA is asking the prometheus adapter for metric istio_requests_per_second. By calling /apis/custom.metrics.k8s.io/v1beta1/namespaces/myNamespace/pods/mypod the adapter takes that and looks at its rules configured for what it should query for.
https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/config-walkthrough.md
Based on my test, I think HPA uses the 'scaleTargetRef' to determine which POD's metrics should be used, and pull these metrics from the metrics server and evaluate them against the target config.
As per Kubernetes documentation:
For object metrics and external metrics, a single metric is fetched, which describes the object in question. This metric is compared to the target value, to produce a ratio as above. In the autoscaling/v2 API version, this value can optionally be divided by the number of Pods before the comparison is made.
It will calculate the ratio based on the mean across the target pods.
References:
1.-https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#how-does-a-horizontalpodautoscaler-work
I am trying to autoscale using custom metrics, with metric type "http_request". My following command is showing correct output:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/" | jq
Below is my hpa.yaml file:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: podinfo
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
minReplicas: 2
maxReplicas: 10
metrics:
- type: Pods
pods:
metricName: http_requests
targetAverageValue: 1
but my scaling is failing due to
the HPA was unable to compute the replica count:
unable to get metric http_requests: unable to fetch metrics from custom metrics API: an error on the server`
("Internal Server Error: \"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%!A(MISSING)/http_requests?labelSelector=app%!D(MISSING)podinfo\": the server could not find the requested resource")
has prevented the request from succeeding (get pods.custom.metrics.k8s.io *)
Please help me out in this :)
Seems like you are missing pods in your cluster that match the provided deployment specification. Can you check if your podinfo deployment is running? And that it has healthy pods in it?
The command works because you're only checking the availability of the metrics endpoint. This simply implies that the endpoint is live to start providing metrics, doesn't guarantee that you will receive metrics (without any resources).
I am trying to auto-scale my pods based on CloudSQL instance response time. We are using cloudsql-proxy for secure connection.
Deployed the Custom Metrics Adapter.
https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: application_name
spec:
minReplicas: 1
maxReplicas: 5
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: application_name
minReplicas: 1
maxReplicas: 5
metrics:
- type: Pods
pods:
metric:
name: custom-metric-stackdriver-adapter
target:
type: AverageValue
averageValue: 20
I deployed the application and create HPA for that, But i am seeing error.
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetPodsMetric the HPA was unable to compute the replica count: unable to get metric custom-metric: unable to fetch metrics from custom metrics API: the server could not find the descriptor for metric custom.googleapis.com/custom-metric: googleapi: Error 404: Could not find descriptor for metric 'custom.googleapis.com/custom-metric'., notFound
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetPodsMetric 4m22s (x10852 over 47h) horizontal-pod-autoscaler unable to get metric custom-metric: unable to fetch metrics from custom metrics API: the server could not find the descriptor for metric custom.googleapis.com/custom-metric: googleapi: Error 404: Could not find descriptor for metric 'custom.googleapis.com/custom-metric'., notFound
Please refer to the link below to deploy a HorizontalPodAutoscaler (HPA) resource to scale your application based on Cloud Monitoring metrics.
https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#custom-metric_4
Looks like the custom metric name is different in the app and hpa deployment configuration files(yaml). Metric and application names should be the same in both app and hpa deployment configuration files.
In the hpa deployment yaml file,
a. Replace custom-metric-stackdriver-adapter with custom-metric (Or change the
metric name to custom-metric-stackdriver-adapter in the app deployment yaml
file).
b. Add “namespace: default” next to the application name at metadata.Also
ensure you are adding the namespace in the app deployment configuration
file.
c. Delete the duplicate lines 6 & 7 (minReplicas: 1, maxReplicas: 5).
d. Go to Cloud Console->Kubernetes Engine->Workloads. Delete the workloads (application-name & custom-metrics-stackdriver-adapter) created by app deployment yaml and adapter_new_resource_model.yaml files.
e. Now apply configurations to resource model, app and hpa (yaml files).
What I am trying to achieve is creating a Horizontal Pod Autoscaler able to scale worker pods according to a custom metric produced by a controller pod.
I already have Prometheus scraping, Prometheus Adapater, Custom Metric Server fully operational and scaling the worker deployment with a custom metric my_controller_metric produced by the worker pods already works.
Now my workerpods don't produce this metric anymore, but the controller does.
It seems that the API autoscaling/v1 does not support this feature. I am able to specify the HPA with the autoscaling/v2beta1 API if necessary though.
Here is my spec for this HPA:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: my-worker-hpa
namespace: work
spec:
maxReplicas: 10
minReplicas: 1
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: my-worker-deployment
metrics:
- type: Object
object:
target:
kind: Deployment
name: my-controller-deployment
metricName: my_controller_metric
targetValue: 1
When the configuration is applied with kubectl apply -f my-worker-hpa.yml I get the message:
horizontalpodautoscaler "my-worker-hpa" configured
Though this message seems to be OK, the HPA does not work. Is this spec malformed?
As I said, the metric is available in the Custom Metric Server with a kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . | grep my_controller_metric.
This is the error message from the HPA:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetObjectMetric the HPA was unable to compute the replica count: unable to get metric my_controller_metric: Deployment on work my-controller-deployment/unable to fetch metrics from custom metrics API: the server could not find the metric my_controller_metric for deployments
Thanks!
In your case problem is HPA configuration: spec.metrics.object.target should also specify API version.
Putting apiVersion: extensions/v1beta1 under spec.metrics.object.target should fix it.
In addition, there is an open issue about better config validation in HPA: https://github.com/kubernetes/kubernetes/issues/60511
I'm trying to configure a Horizontal Pod Autoscaler (HPA) on Google Kubernetes Engine (GKE) using External Metrics from an Ingress LoadBalancer, basing the configuration on instructions such as
https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling
and
https://blog.doit-intl.com/autoscaling-k8s-hpa-with-google-http-s-load-balancer-rps-stackdriver-metric-92db0a28e1ea
With an HPA like
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: my-api
namespace: production
spec:
minReplicas: 1
maxReplicas: 20
metrics:
- external:
metricName: loadbalancing.googleapis.com|https|request_count
metricSelector:
matchLabels:
resource.labels.forwarding_rule_name: k8s-fws-production-lb-my-api--63e2a8ddaae70
targetAverageValue: "1"
type: External
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-api
the autoscaler does kick in when the request count rises - but placing heavy load on the service, like 100 simultaneous requests per second, doesn't increase the external metric request_count much beyond 6 RPS, while the observed backend_latencies metric in Stackdriver does increase significantly; so I'd like to utilise that metric by adding to the HPA configuration, like so:
- external:
metricName: loadbalancing.googleapis.com|https|backend_latencies
metricSelector:
matchLabels:
resource.labels.forwarding_rule_name: k8s-fws-production-lb-my-api--63e2a8ddaae70
targetValue: "3000"
type: External
but that results in the error:
...unable to fetch metrics from external metrics API: googleapi: Error 400: Field aggregation.perSeriesAligner had an invalid value of "ALIGN_RATE": The aligner cannot be applied to metrics with kind DELTA and value type DISTRIBUTION., badRequest
which can be observed with the command
$ kubectl describe hpa -n production
or by visiting
http://localhost:8080/apis/external.metrics.k8s.io/v1beta1/namespaces/default/loadbalancing.googleapis.com%7Chttps%7Cbackend_latencies
after setting up a proxy with
$ kubectl proxy --port=8080
Are https/backend_latencies or https/total_latencies not supported as External Stackdriver Metrics in an HPA configuration for GKE?
Maybe someone would find this helpful, though the question is old.
My working config looks like next:
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 95
- type: External
external:
metric:
name: loadbalancing.googleapis.com|https|backend_latencies
selector:
matchLabels:
resource.labels.backend_name: frontend
metric.labels.proxy_continent: Europe
reducer: REDUCE_PERCENTILE_95
target:
type: Value
value: "79.5"
type: Value used because it's the only way to not divide the metric value by the replica number.
reducer: REDUCE_PERCENTILE_95 used to work only with a single value of the distribution (source).
Also, I edited custom-metrics-stackdriver-adapter deployment to look like this:
- image: gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.12.2-gke.0
imagePullPolicy: Always
name: pod-custom-metrics-stackdriver-adapter
command:
- /adapter
- --use-new-resource-model=true
- --fallback-for-container-metrics=true
- --enable-distribution-support=true
The thing is this key enable-distribution-support=true, which enables working with distribution kind of metrics.