How to check external metrics data in Kubernetes? - kubernetes

Question

I am using DirectXMan12/k8s-prometheus-adapte to push the external metric from Prometheus to Kubernetes.
After pushing the external metric how can I verify the data is k8s?
When I hit kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq I got the following result but after that, I do not have an idea how to fetch actual metrics value
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "external.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "subscription_back_log",
"singularName": "",
"namespaced": true,
"kind": "ExternalMetricValueList",
"verbs": [
"get"
]
}]
}

Answer 1

actual metric value is fetched per instance, for example, the metric you attached is namespaced: true, assuming the metric is for pods, you can access the actual data at
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/wanted_namepsace/pods/*/subscription_back_log" | jq '.'
(or specify the pod name instead of *)
If you want HPA to read you metric, the configurations are (for example)
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: your-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: your-pod
minReplicas: 1
maxReplicas: 10
metrics:
- pods:
metricName: subscription_back_log
targetAverageValue: 10000
type: Pods

Answer 2

The metric is namespaced, so you will need to add the namespace into the URL. Contrary to what the other answer suggests, I believe you don't need to include pods into the URL. This is an external metric. External metrics are not associated to any kubernetes object, so only the namespace should suffice:
/apis/external.metrics.k8s.io/v1beta1/namespaces/<namespace>/<metric_name>
Here's an example that works for me, using an external metric in my setup:
$ kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "external.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "redis_key_size",
"singularName": "",
"namespaced": true,
"kind": "ExternalMetricValueList",
"verbs": [
"get"
]
}
]
}
$ kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/default/redis_key_size
{
"kind": "ExternalMetricValueList",
"apiVersion": "external.metrics.k8s.io/v1beta1",
"metadata": {},
"items": [
{
"metricName": "redis_key_size",
"metricLabels": {
"key": "..."
},
"timestamp": "2021-10-07T09:00:01Z",
"value": "0"
},
...
]
}