kubernetes Autoscaler - Cannot obtain loadbalancing.googleapis.com|https|request_count - kubernetes

Question

I'm trying to define an Horizontal Pod Autoscaler for two Kubernetes services.
The Autoscaler strategy relies in 3 metrics:
cpu
pubsub.googleapis.com|subscription|num_undelivered_messages
loadbalancing.googleapis.com|https|request_count
CPU and num_undelivered_messages are correctly obtained, but no matter what i do, i cannot get the request_count metric.
The first service is a backend service (Service A), and the other (Service B) is an API that uses an Ingress to manage the external access to the service.
The Autoscaling strategy is based on Google documentation: Autoscaling Deployments with External Metrics.
For service A, the following defines the metrics used for Autoscaling:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: ServiceA
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: ServiceA
minReplicas: 1
maxReplicas: 3
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80
- external:
metricName: pubsub.googleapis.com|subscription|num_undelivered_messages
metricSelector:
matchLabels:
resource.labels.subscription_id: subscription_id
targetAverageValue: 100
type: External
For service B, the following defines the metrics used for Autoscaling:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: ServiceB
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: ServiceB
minReplicas: 1
maxReplicas: 3
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80
- external:
metricName: loadbalancing.googleapis.com|https|request_count
metricSelector:
matchLabels:
resource.labels.forwarding_rule_name: k8s-fws-default-serviceb--3a908157de956ba7
targetAverageValue: 100
type: External
As defined in the above article, the metrics server is running, and the metrics server adapter is deployed:
$ kubectl get apiservices |egrep metrics
v1beta1.custom.metrics.k8s.io custom-metrics/custom-metrics-stackdriver-adapter True 2h
v1beta1.external.metrics.k8s.io custom-metrics/custom-metrics-stackdriver-adapter True 2h
v1beta1.metrics.k8s.io kube-system/metrics-server True 2h
v1beta2.custom.metrics.k8s.io custom-metrics/custom-metrics-stackdriver-adapter True 2h
For service A, all metrics, CPU and num_undelivered_messages, are correctly obtained:
$ kubectl get hpa ServiceA
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
ServiceA Deployment/ServiceA 0/100 (avg), 1%/80% 1 3 1 127m
For service B, HPA cannot obtain the Request Count:
$ kubectl get hpa ServiceB
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
ServiceB Deployment/ServiceB <unknown>/100 (avg), <unknown>/80% 1 3 1 129m
When accessing the Ingress, i get this warning:
unable to get external metric default/loadbalancing.googleapis.com|https|request_count/&LabelSelector{MatchLabels:map[string]string{resource.labels.forwarding_rule_name: k8s-fws-default-serviceb--3a908157de956ba7,},MatchExpressions:[],}: no metrics returned from external metrics API
The metricSelector for the forwarding-rule is correct, as confirmed when describing the ingress (only the relevant information is show):
$ kubectl describe ingress serviceb
Annotations:
ingress.kubernetes.io/https-forwarding-rule: k8s-fws-default-serviceb--3a908157de956ba7
I've tried to use a different metric selector, for example, using url_map_name, to no avail, i've got a similar error.
I've followed the exact guidelines on Google Documentation, and checked with a few online tutorials that refer the exact same process, but i haven't been able to understand what i'm missing.
I'm probably lacking some configuration, or some specific detail, but i cannot find it documented anywhere.
What am i missing, that explains why i'm not being able to obtain the loadbalancing.googleapis.com|https|request_count metric?

Answer 1

It seems the metric that you're defining isn't available in the External Metrics API. To find out what's going on, you can inspect the External Metrics API directly:
kubectl get --raw="/apis/external.metrics.k8s.io/v1beta1" | jq
Is the loadbalancing.googleapis.com|https|request_count metric reported in the output?
You can then dig deeper by making requests of the following form:
kubectl get --raw="/apis/external.metrics.k8s.io/v1beta1/namespaces/<namespace_name>/<metric_name>?labelSelector=<selector>" | jq
And see what's returned given your metric name and a specific metric selector.
These are precisely the requests that the Horizontal Pod Autoscaler also makes at runtime. By replicating them manually, you should be able to pinpoint the source of the problem.
Comments about additional information:
1) 83m is the Kubernetes way of writing 0.083 (read as 83 "milli-units").
2) In your HorizontalPodAutoscaler definition, you use a targetAverageValue. So, if there exist multiple targets with this metric, the HPA calculates their average. So, 83m might be an average of multiple targets. To make sure, you use only the metric of a single target, you can use the targetValue field (see API reference).
3) Not sure why the items: [] array in the API response is empty. The documentation mentions that after sampling, the data is not visible for 210 seconds... You could try making the API request when the HPA is not running.

Answer 2

Thank you very much for your detailed response.
When using the metricSelector to select the specific forwarding_rule_name, we need to use the exact forwarding_rule_name as defined by the ingress:
metricSelector:
matchLabels:
resource.labels.forwarding_rule_name: k8s-fws-default-serviceb--3a908157de956ba7
$ kubectl describe ingress
Name: serviceb
...
Annotations:
ingress.kubernetes.io/https-forwarding-rule: k8s-fws-default-serviceb--9bfb478c0886702d
...
kubernetes.io/ingress.allow-http: false
kubernetes.io/ingress.global-static-ip-name: static-ip
The problem, is that the suffix of the forwarding_rule_name (3a908157de956ba7) changes for every deployment, and is created dynamically on Ingress creation:
k8s-fws-default-serviceb--3a908157de956ba7
We have a fully automated deployment using Helm, and, as such, when the HPA is created, we don't know what the forwarding_rule_name will be.
And, it seems that the matchLabels does not accept regular expressions, or else we would simply do something like:
metricSelector:
matchLabels:
resource.labels.forwarding_rule_name: k8s-fws-default-serviceb--*
I've tried several approaches, all without success:
Use Annotations to force the forwarding_rule_name
Use a different machLabel, as backend_target_name
Obtain the forwarding_rule_name using a command, so i can insert it later in the yaml file.
Use Annotations to force the forwarding_rule_name:
When creating the ingress, i can use specific annotations to change the default behavior, or define specific values, for example, on Ingress.yaml:
annotations:
kubernetes.io/ingress.global-static-ip-name: static-ip
I tried to use the https-forwarding-rule annotation to force a specific "static" name, but this didn't work:
annotations:
ingress.kubernetes.io/https-forwarding-rule: some_name
annotations:
kubernetes.io/https-forwarding-rule: some_name
Use a different machLabel, as backend_target_name
metricSelector:
matchLabels:
resource.labels.backend_target_name: serviceb
Also failed.
Obtain the forwarding_rule_name using a command
When executing the following command, i get the list of Forwarding Rules, but for all the clusters. And according to the documentation, is not possible to filter by cluster:
gcloud compute forwarding-rules list
NAME P_ADDRESS IP_PROTOCOL TARGET
k8s-fws-default-serviceb--4e1c268b39df8462 xx TCP k8s-tps-default-serviceb--4e1c268b39df8462
k8s-fws-default-serviceb--9bfb478c0886702d xx TCP k8s-tps-default-serviceb--9bfb478c0886702d
Is there any way to allow me to select the resource i need, in order to get the Requests count metric?

Answer 3

It seems everything was OK with my code, but, there is a time delay (aprox. 10m), before the request_count metric is available. After this period, the metric is now computed and available:
$ kubectl get hpa ServiceB
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
ServiceB Deployment/ServiceB 83m/100 (avg), 1%/80% 1 3 1 18m
Now, regarding the loadbalancing.googleapis.com|https|request_count metric, i'm not understanding how its being presented. What does 83m means?
According to Google documentation for Load balancing metrics:
https/request_bytes_count Request bytes
DELTA, INT64, By
GA
The number of requests served by HTTP/S load balancer. Sampled every 60
seconds. After sampling, data is not visible for up to 210 seconds.
According to Metric Details:
In a DELTA metric, each data point represents the change in a value
over the time interval. For example, the number of service requests
received since the previous measurement would be a delta metric.
I've made one single request to the service, so i was expecting a value of 1, and i can't understand what the 83m means.
Another possibility, could be that i'm not using the correct metric.
I've selected the loadbalancing.googleapis.com|https|request_count metric, assuming it would provide the number of requests that were executed by the service, via the loadbalancer.
Isn't exactly this information that the loadbalancing.googleapis.com|https|request_count metric provides?
Regarding the above comment, when executing:
kubectl get --raw="/apis/external.metrics.k8s.io/v1beta1/namespaces/default/pubsub.googleapis.com|subscription|num_undelivered_messages" | jq
i get the correct data:
...
{
"metricName": "pubsub.googleapis.com|subscription|num_undelivered_messages",
"metricLabels": {
"resource.labels.project_id": "project-id",
"resource.labels.subscription_id": "subscription_id",
"resource.type": "pubsub_subscription"
},
"timestamp": "2019-10-22T15:39:58Z",
"value": "4"
}
...
but, when executing:
kubectl get --raw="/apis/external.metrics.k8s.io/v1beta1/namespaces/default/loadbalancing.googleapis.com|https|request_count" | jq
i get nothing back:
{ "kind": "ExternalMetricValueList", "apiVersion":
"external.metrics.k8s.io/v1beta1", "metadata": {
"selfLink": >"/apis/external.metrics.k8s.io/v1beta1/namespaces/default/loadbalancing.googleapis.com%7Chttps%7Crequest_count"
}, "items": [] }