Kubernetes HPA not scaling with custom metric using prometheus adapter on istio - kubernetes

Question

I have two deployments running a v1 and v2 of the same service in istio. I have set up a custom-metric 'istio-total-requests' to be collected through prometheus adapater.
I have set up an HPA to scale the v2 deployment and can see the target value increasing when I send requests, but what is not happening is the HPA is not scaling the number of pods/replicas.
I am running kubernetes v1.19 on minikube v1.13.1, and cannot understand why its not scaling.
prometheus:
url: http://prometheus.istio-system.svc.cluster.local
rules:
default: false
custom:
# this rule matches cumulative cAdvisor metrics measured in seconds
- seriesQuery: 'istio_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
seriesFilters: []
resources:
# template: <<.Resource>>
# skip specifying generic resource<->label mappings, and just
# attach only pod and namespace resources by mapping label names to group-resources
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
# specify that the `container_` and `_seconds_total` suffixes should be removed.
# this also introduces an implicit filter on metric family names
name:
# we use the value of the capture group implicitly as the API name
# we could also explicitly write `as: "$1"`
matches: "^(.*)_total"
as: "${1}_per_second"
# matches: ""
# as: ""
# specify how to construct a query to fetch samples for a given series
# This is a Go template where the `.Series` and `.LabelMatchers` string values
# are available, and the delimiters are `<<` and `>>` to avoid conflicts with
# the prometheus query language
metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)"
HPA YAML
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: translate-deployment-v2-hpa
spec:
minReplicas: 1
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: istio_requests_per_second
# selector: {matchLabels: {destination_version: 0.0.2}}
target:
type: AverageValue
averageValue: 10
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: translate-deployment-v2
See the HPA pulling and measuring the metric but not scaling, the window below it shows the prometheus-adapter getting queried for the metric successfully.
HPA Description
One final item which I'm not clear on, is what is the purpose of the selector in the hpa definition above ? Is it to select specific values from the data range which is queried by prometheus ?
For example I know the field I'm querying is output by default by envoy as follows:
istio_requests_total{app="istio-ingressgateway",chart="gateways",connection_security_policy="unknown",destination_app="translate-pod",destination_canonical_revision="0.0.1",destination_canonical_service="translate-pod",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_service="translate-service.default.svc.cluster.local",destination_service_name="translate-service",destination_service_namespace="default",destination_version="0.0.1",destination_workload="translate-deployment",destination_workload_namespace="default",heritage="Tiller",install_operator_istio_io_owning_resource="unknown",instance="172.17.0.5:15020",istio="ingressgateway",istio_io_rev="default",job="kubernetes-pods",kubernetes_namespace="istio-system",kubernetes_pod_name="istio-ingressgateway-6cfd75fc57-flmhp",operator_istio_io_component="IngressGateways",pod_template_hash="6cfd75fc57",release="istio",reporter="source",request_protocol="http",response_code="200",response_flags="-",service_istio_io_canonical_name="istio-ingressgateway",service_istio_io_canonical_revision="latest",source_app="istio-ingressgateway",source_canonical_revision="latest",source_canonical_service="istio-ingressgateway",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",source_version="unknown",source_workload="istio-ingressgateway",source_workload_namespace="istio-system"}
Does the selector allow you to further filter the series data, if not what is the purpose and how do you use it ?

Answer 1

According to your screenshot, HPA works as expected because your metric value is lower than the threshold. If the value does not go over your threshold value, HPA will not trigger the scale up. Instead, it may trigger the scale down in your case.
The metric you are using right now is istio_requests_per_second. That is calculated by the total request per second. The first screenshot show the average value is 200m, which will be 0.2. Your threshold is 10 so HPA definitely would not scale up in this case.
For the selector, it gives you the ability to select your target label under the metrics. For example, if you want to scale the instances against the GET method only. You can define the selector matching the GET method label in this case.