Retrieve custom `custom.metrics.k8s.io` value using curl - kubernetes

Question

I can list all the custom.metrics available, but I don't know how to query an individual value. For example I have tried:
curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/ | jq .
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "custom.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "deployments.apps/aws_sqs_approximate_number_of_messages_visible_average",
"singularName": "",
"namespaced": false,
"kind": "MetricValueList",
"verbs": [
"get"
]
}
]
}
But if I try this:
curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/deployments.apps/aws_sqs_approximate_number_of_messages_visible_average | jq .
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "the server could not find the requested resource",
"reason": "NotFound",
"details": {
},
"code": 404
}
I get a 404. I've seen this issue which shows how to get a namespaced metric, but mine does not have a namespace? Is there a definition for how to use this API?

Answer 1

Just like Resource Metrics, Custom Metrics are bound to Kubernetes objects too.
What you're missing in your URL is the resource you want the metric to relate to.
For example the Pod the custom metric is related to, but the same is true for Deployments.
Try to adjust this url to your needs:
kubectl get --raw \
'/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pod/podinfo-67c9fd95d-fqk4g/http_requests_per_second' \
| jq .
Here are the slides for the talk we gave at FOSDEM 2019 on the Prometheus Adapter: https://speakerdeck.com/metalmatze/kubernetes-metrics-api?slide=26
I'll update this answer, once the video is available too.

Answer 2

Since I'm using DirectXMan12/k8s-prometheus-adapter there are a few things to know:
I think it can only work with namespaced metrics.
If a query does not return a metric for a particular time period in prometheus k8s-prometheus-adapter will report it as non-existent.
This is my actual problem.
Using the custom metrics API is very simple:
kubectl proxy to open a proxy to your kubernetes API
curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/ to list all custom metrics available.
For example you may see:
{
"name": "deployments.extensions/kube_deployment_status_replicas_available",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
}
We know it is namespaced from namespaced: true and beneath the namespace we can select via deployment from the name field.
So we would build our query like so:
curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/namespace/$NAMESPACE/deployments.extensions/$DEPLOYMENT/kube_deployment_status_replicas_available
At least I think that's how it should work, although if you do the same query without deployments.extensions section it will show the value for the namespace:
curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/namespace/$NAMESPACE/kube_deployment_status_replicas_available
Perhaps this is due to how the query executes in prometheus.