Scale deployment based on custom metric - kubernetes

Question

I'm trying to scale a deployment based on a custom metric coming from a custom metric server. I deployed my server and when I do
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/kubernetes/test-metric"
I get back this JSON
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/kubernetes/test-metric"
},
"items": [
{
"describedObject": {
"kind": "Service",
"namespace": "default",
"name": "kubernetes",
"apiVersion": "/v1"
},
"metricName": "test-metric",
"timestamp": "2019-01-26T02:36:19Z",
"value": "300m",
"selector": null
}
]
}
Then I created my hpa.yml using this
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: test-all-deployment
namespace: default
spec:
maxReplicas: 10
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: test-all-deployment
metrics:
- type: Object
object:
target:
kind: Service
name: kubernetes
apiVersion: custom.metrics.k8s.io/v1beta1
metricName: test-metric
targetValue: 200m
but it doesn't scale and I'm not sure what is wrong. running get hpa returns
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
test-all-deployment Deployment/test-all-deployment <unknown>/200m 1 10 1 9m
The part I'm not sure about is the target object in the metrics collection in the hpa definition. Looking at the doc here https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
It has
describedObject:
apiVersion: extensions/v1beta1
kind: Ingress
name: main-route
target:
kind: Value
value: 10k
but that gives me a validation error for API v2beta1. and looking at the actual object here https://github.com/kubernetes/api/blob/master/autoscaling/v2beta1/types.go#L296 it doesn't seem to match. I don't know how to specify that with the v2beta1 API.

Answer 1

It looks like there is a mistake in the documentation. In the same example two diffierent API version are used.
autoscaling/v2beta1 notation:
- type: Pods
pods:
metric:
name: packets-per-second
targetAverageValue: 1k
autoscaling/v2beta2 notation:
- type: Resource
resource:
name: cpu
target:
type: AverageUtilization
averageUtilization: 50
There is a difference between autoscaling/v2beta1 and autoscaling/v2beta2 APIs:
kubectl get hpa.v2beta1.autoscaling -o yaml --export > hpa2b1-export.yaml
kubectl get hpa.v2beta2.autoscaling -o yaml --export > hpa2b2-export.yaml
diff -y hpa2b1-export.yaml hpa2b2-export.yaml
#hpa.v2beta1.autoscaling hpa.v2beta2.autoscaling
#-----------------------------------------------------------------------------------
apiVersion: v1 apiVersion: v1
items: items:
- apiVersion: autoscaling/v2beta1 | - apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler kind: HorizontalPodAutoscaler
metadata: metadata:
creationTimestamp: "2019-03-21T13:17:47Z" creationTimestamp: "2019-03-21T13:17:47Z"
name: php-apache name: php-apache
namespace: default namespace: default
resourceVersion: "8441304" resourceVersion: "8441304"
selfLink: /apis/autoscaling/v2beta1/namespaces/default/ho | selfLink: /apis/autoscaling/v2beta2/namespaces/default/ho
uid: b8490a0a-4bdb-11e9-9043-42010a9c0003 uid: b8490a0a-4bdb-11e9-9043-42010a9c0003
spec: spec:
maxReplicas: 10 maxReplicas: 10
metrics: metrics:
- resource: - resource:
name: cpu name: cpu
targetAverageUtilization: 50 | target:
> averageUtilization: 50
> type: Utilization
type: Resource type: Resource
minReplicas: 1 minReplicas: 1
scaleTargetRef: scaleTargetRef:
apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1
kind: Deployment kind: Deployment
name: php-apache name: php-apache
status: status:
conditions: conditions:
- lastTransitionTime: "2019-03-21T13:18:02Z" - lastTransitionTime: "2019-03-21T13:18:02Z"
message: recommended size matches current size message: recommended size matches current size
reason: ReadyForNewScale reason: ReadyForNewScale
status: "True" status: "True"
type: AbleToScale type: AbleToScale
- lastTransitionTime: "2019-03-21T13:18:47Z" - lastTransitionTime: "2019-03-21T13:18:47Z"
message: the HPA was able to successfully calculate a r message: the HPA was able to successfully calculate a r
resource utilization (percentage of request) resource utilization (percentage of request)
reason: ValidMetricFound reason: ValidMetricFound
status: "True" status: "True"
type: ScalingActive type: ScalingActive
- lastTransitionTime: "2019-03-21T13:23:13Z" - lastTransitionTime: "2019-03-21T13:23:13Z"
message: the desired replica count is increasing faster message: the desired replica count is increasing faster
rate rate
reason: TooFewReplicas reason: TooFewReplicas
status: "True" status: "True"
type: ScalingLimited type: ScalingLimited
currentMetrics: currentMetrics:
- resource: - resource:
currentAverageUtilization: 0 | current:
currentAverageValue: 1m | averageUtilization: 0
> averageValue: 1m
name: cpu name: cpu
type: Resource type: Resource
currentReplicas: 1 currentReplicas: 1
desiredReplicas: 1 desiredReplicas: 1
kind: List kind: List
metadata: metadata:
resourceVersion: "" resourceVersion: ""
selfLink: "" selfLink: ""
Here is how the object definition is supposed to look like:
#hpa.v2beta1.autoscaling hpa.v2beta2.autoscaling
#-----------------------------------------------------------------------------------
type: Object type: Object
object: object:
metric: metric:
name: requests-per-second name: requests-per-second
describedObject: describedObject:
apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1
kind: Ingress kind: Ingress
name: main-route name: main-route
targetValue: 2k target:
type: Value
value: 2k