I was following this walk through (partially, am using EKS.
https://itnext.io/horizontal-pod-autoscale-with-custom-metrics-8cb13e9d475
I manage to get scaled up one deployment with this http_requests_total metric.
Now, I am trying to add a new metric. I have prometheus server, and it already scrapes cloudwatch and I have aws_sqs_approximate_age_of_oldest_message_maximum value there for many of my queues.
In the similar manner to the mentioned tutorial, I am adding the definition of a metric:
- seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
name:
matches: "^(.*)_total"
as: "${1}_per_second"
metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
vs
- seriesQuery: 'aws_sqs_approximate_age_of_oldest_message_maximum{queue_name!=""}'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
metricsQuery: '<<.Series>>{<<.LabelMatchers>>}'
Or some version of the bottom one. But, I can never see it in:
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq
No matter what I try.
Any ideas how to move forward?
Thanks!
If you don't see the metric in /apis/custom.metrics.k8s.io/v1beta1 it means that the Prometheus Adapter couldn't discover it.
The Prometheus Adapter discovers metrics by using the value of your seriesQuery field for an /api/v1/series request to Prometheus (done periodically with a frequency defined by the relist interval).
Things to try:
What do you get if you make the following request to Prometheus?
http://<prometheus-ip>:9090/api/v1/series? match[]=aws_sqs_approximate_age_of_oldest_message_maximum{queue_name!=""}&start=<current-timestamp-sec>
What do you get if you drop the following in the query text box of the Prometheus UI and press Execute?
aws_sqs_approximate_age_of_oldest_message_maximum{queue_name!=""}
If you get no data back in either case, then you just don't have any time series in Prometheus that match your seriesQuery specification.
Related
I want to do Horizontal pod autoscaling on the basis of input request. I made query for prometheus which returns aggregated input tps.
sum(sum by (path) (rate(dapr_http_server_request_count{app_id="governor",path=~"/v1.0/invoke/app/method/interceptor/.*"}[10s])))
I want to use this output in kubernetes HPA.
I am using prometheus-adapter for the same. Prometheus configuration is as follows:
default: true
custom:
- seriesQuery: '{__name__=~"dapr_http_server_request_avg_.*",namespace!="",pod!=""}'
resources:
overrides:
namespace:
resource: namespace
pod:
resource: pod
name:
matches: "^(.*)_total"
as: "${1}_per_second"
metricsQuery: 'sum(sum by (path) (rate(dapr_http_server_request_count{app_id="governor",path=~"/v1.0/invoke/app/method/interceptor/.*"}[10s])))'
When i try to get the output in custom api of kubernetes it returns.
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/app/pods/*/dapr_http_server_request_avg_total" | jq .
Error from server (NotFound): the server could not find the metric dapr_http_server_request_avg_total for pods
You should check your prometheus-adapter config file, which makes me confused.
seriesQuery: dapr_http_server_request_avg_.*
metricsQuery: dapr_http_server_request_count
It is possible to get metrics dapr_http_server_request_count from above series?
name:
matches: "^(.*)_total"
as: "${1}_per_second"
and here take care, your metric name seems to be renamed, you should find the right metric name for you query.
try this: kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
you will see what your K8s Api-server actually get from Prometheus Adapter.
I have two deployments running a v1 and v2 of the same service in istio. I have set up a custom-metric 'istio-total-requests' to be collected through prometheus adapater.
I have set up an HPA to scale the v2 deployment and can see the target value increasing when I send requests, but what is not happening is the HPA is not scaling the number of pods/replicas.
I am running kubernetes v1.19 on minikube v1.13.1, and cannot understand why its not scaling.
prometheus:
url: http://prometheus.istio-system.svc.cluster.local
rules:
default: false
custom:
# this rule matches cumulative cAdvisor metrics measured in seconds
- seriesQuery: 'istio_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
seriesFilters: []
resources:
# template: <<.Resource>>
# skip specifying generic resource<->label mappings, and just
# attach only pod and namespace resources by mapping label names to group-resources
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
# specify that the `container_` and `_seconds_total` suffixes should be removed.
# this also introduces an implicit filter on metric family names
name:
# we use the value of the capture group implicitly as the API name
# we could also explicitly write `as: "$1"`
matches: "^(.*)_total"
as: "${1}_per_second"
# matches: ""
# as: ""
# specify how to construct a query to fetch samples for a given series
# This is a Go template where the `.Series` and `.LabelMatchers` string values
# are available, and the delimiters are `<<` and `>>` to avoid conflicts with
# the prometheus query language
metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)"
HPA YAML
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: translate-deployment-v2-hpa
spec:
minReplicas: 1
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: istio_requests_per_second
# selector: {matchLabels: {destination_version: 0.0.2}}
target:
type: AverageValue
averageValue: 10
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: translate-deployment-v2
See the HPA pulling and measuring the metric but not scaling, the window below it shows the prometheus-adapter getting queried for the metric successfully.
HPA Description
One final item which I'm not clear on, is what is the purpose of the selector in the hpa definition above ? Is it to select specific values from the data range which is queried by prometheus ?
For example I know the field I'm querying is output by default by envoy as follows:
istio_requests_total{app="istio-ingressgateway",chart="gateways",connection_security_policy="unknown",destination_app="translate-pod",destination_canonical_revision="0.0.1",destination_canonical_service="translate-pod",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_service="translate-service.default.svc.cluster.local",destination_service_name="translate-service",destination_service_namespace="default",destination_version="0.0.1",destination_workload="translate-deployment",destination_workload_namespace="default",heritage="Tiller",install_operator_istio_io_owning_resource="unknown",instance="172.17.0.5:15020",istio="ingressgateway",istio_io_rev="default",job="kubernetes-pods",kubernetes_namespace="istio-system",kubernetes_pod_name="istio-ingressgateway-6cfd75fc57-flmhp",operator_istio_io_component="IngressGateways",pod_template_hash="6cfd75fc57",release="istio",reporter="source",request_protocol="http",response_code="200",response_flags="-",service_istio_io_canonical_name="istio-ingressgateway",service_istio_io_canonical_revision="latest",source_app="istio-ingressgateway",source_canonical_revision="latest",source_canonical_service="istio-ingressgateway",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",source_version="unknown",source_workload="istio-ingressgateway",source_workload_namespace="istio-system"}
Does the selector allow you to further filter the series data, if not what is the purpose and how do you use it ?
According to your screenshot, HPA works as expected because your metric value is lower than the threshold. If the value does not go over your threshold value, HPA will not trigger the scale up. Instead, it may trigger the scale down in your case.
The metric you are using right now is istio_requests_per_second. That is calculated by the total request per second. The first screenshot show the average value is 200m, which will be 0.2. Your threshold is 10 so HPA definitely would not scale up in this case.
For the selector, it gives you the ability to select your target label under the metrics. For example, if you want to scale the instances against the GET method only. You can define the selector matching the GET method label in this case.
I have an existing deployment of prometheus operator on my k8s cluster. Now, I want to add a scrape target for a custom exporter I created. I created my prometheus.yaml file, but can't find how to apply this to the existing prometheus operator.
I have no idea how and where you create/modify your prometheus.yaml file, but will show you the convenient way of managing it.
First of all you I would recommend you store prometheus configuration file prometheus.yaml as a configmap. This is super useful plus changes you will do in CM will be automatically without your involving spreading/propagating to the pod that work with/consume this CM.
So after you make changes with new scrapes in CM - it will take some time to propagate changes. Total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period (1 minute by default) + ttl of ConfigMaps cache (1 minute by default) in kubelet.
Now time to make changes live..
Prometheus has an option to reload its configuration on flight. You have 2 options how to do that. More info you can find opening Reloading Prometheus’ Configuration url.
I will stop here only on one solution:
You can send a HTTP POST to the Prometheus web server:
curl -X POST http://localhost:9090/-/reload
Note that as of Prometheus 2.0, the --web.enable-lifecycle command
line flag must be passed for HTTP reloading to work.
If the reload is successful Prometheus will log that it has updated its targets:
INFO[0248] Loading configuration file prometheus.yml source=main.go:196
INFO[0248] Stopping target manager... source=targetmanager.go:203
INFO[0248] Target manager stopped. source=targetmanager.go:216
INFO[0248] Starting target manager... source=targetmanager.go:114
And cherry on cake for you :)
There is a great Prometheus, ConfigMaps and Continuous Deployment that explain how to monitor Prometheus with prometheus(Maybe it will be applicable somehow in your another question). The main stuff I want to show you is that you can automate POST request.
So basically you need a tiny sidecar container that will check CM changes and send POST in case of new changes. This sidecar should be in the same pod with prometheus
Copy pasting example from article here for future reference
spec:
containers:
- name: prometheus
...
volumeMounts:
- name: config-volume
mountPath: /etc/prometheus
- name: watch
image: weaveworks/watch:master-5b2a6e5
imagePullPolicy: IfNotPresent
args: ["-v", "-t", "-p=/etc/prometheus", "curl", "-X", "POST", "--fail", "-o", "-", "-sS", "http://localhost:80/-/reload"]
volumeMounts:
- name: config-volume
mountPath: /etc/prometheus
volumes:
- name: config-volume
configMap:
name: prometheus-config
Documentation for the operator: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md
But you have to use additionalScrapeConfigsSecret:
additionalScrapeConfigsSecret:
enabled: true
name: additional-scrape-configs
key: prometheus-additional.yaml
Else you get the Error cannot unmarshal !!map into []yaml.MapSlice
Here is a better documentation:
https://github.com/prometheus-community/helm-charts/blob/8b45bdbdabd9b54766c4beb3c562b766b268a034/charts/kube-prometheus-stack/values.yaml#L2691
According to this, you could add scrape configs without packaging into a secret like this:
additionalScrapeConfigs: |
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
So I am trying to figure out how can I configure an Horizontal Pod Autoscaler from a custom metric reading from Prometheus that returns CPU usage with percentile 0.95
I have everything set up to use custom metrics with prometheus-adapter, but I don't understand how to create the rule in Prometheus. For example, if I go to Grafana to check some of the Graphs that comes by default I see this metric:
sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace="api", pod_name="api-xxxxx9b-bdxx", container_name!="POD", cluster=""}) by (container_name)
But how can I modify that to be percentile 95? I tried with histogram_quantile function but it says no datapoints found:
histogram_quantile(0.95, sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace="api", pod_name="api-xxxxx9b-bdxx", container_name!="POD", cluster=""}) by (container_name))
But even if that works, will the pod name and namespace be filled by prometheus-adapter or prometheus when using custom metrics?
And every example I find using custom metrics are not related with CPU. So... other question I have is how people is using autoscaling metrics in production? I'm used to scale based on percentiles but I don't understand how is this managed in Kubernetes.
If I understand you correctly you don't have to use custom metrics in order to horizontally autoscale your pods. By default, you can automatically scale the number of Kubernetes pods based on the observed CPU utilization.
Here is the official documentation with necessary details.
The Horizontal Pod Autoscaler automatically scales the number of pods
in a replication controller, deployment or replica set based on
observed CPU utilization (or, with custom metrics support, on some
other application-provided metrics).
The Horizontal Pod Autoscaler is implemented as a Kubernetes API
resource and a controller. The resource determines the behavior of the
controller. The controller periodically adjusts the number of replicas
in a replication controller or deployment to match the observed
average CPU utilization to the target specified by user.
And here you can find the walkthrough of how to set it up.
Also, here is the kubectl autoscale command documentation.
Example: kubectl autoscale rc foo --max=5 --cpu-percent=80
Auto scale a replication controller "foo", with the number of pods between 1 and 5, target CPU utilization at 80%
I believe that it is the easiest way so no need to complicate it with some custom metrics.
Please let me know if that helped.
If you want to add HPA based on custom metrics you can use Prometheus adapter.
Prometheus adapter helps you in exposing custom metrics to HPA.
Helm Chart - https://github.com/helm/charts/tree/master/stable/prometheus-adapter
Prometheus adapter - https://github.com/DirectXMan12/k8s-prometheus-adapter
Note - You have to enable 6443 port from public to cluster, because prometheus doesn’t provide override option.
https://github.com/helm/charts/blob/master/stable/prometheus-adapter/templates/custom-metrics-apiserver-deployment.yaml#L34
Make sure that Prometheus is getting custom metrics data
Install Prometheus adapter on the same kubernetes cluster where you want apply hpa
helm install --name my-release stable/prometheus-adapter -f values.yaml
Pass the following config file to helm - values.yaml
prometheus-adapter:
enabled: true
prometheus:
url: http://prometheus.namespace.svc.cluster.local
rules:
default: true
custom:
- seriesQuery: '{__name__="cpu",namespace!="",pod!="",service="svc_name"}'
seriesFilters: []
resources:
overrides:
namespace: {resource: "namespace"}
pod: {resource: "pod"}
name:
matches: "cpu"
as: "cpu_95"
metricsQuery: "histogram_quantile(0.95, sum(irate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>,le))"
Above config will expose
cpu metrics as cpu_95 to HPA.
To Verify, if data is exposed properly run following command -
Fetch the data curl raw query command - kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/namespace_name/pods/\*/cpu_95 | jq .
HPA config -
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: test-cpu-manual
labels:
app: app_name
spec:
scaleTargetRef:
apiVersion: apps/v1beta2
kind: Deployment
name: app_name
minReplicas: 1
maxReplicas: 15
metrics:
- type: Pods
pods:
metricName: cpu_95
targetAverageValue: 75
I have a K8S cluster which is also managing VMs via virtlet. This K8S cluster is running K8S v1.13.2, with prometheus and the prometheus-adapter, and a custom-metrics server. I have written a custom metrics exporter for libvirtd which pulls in VM metrics and have configured prometheus to scrape that exporter for those VM metrics -- this is working and working well.
What I need to do next, is to have the prometheus-adapter push those metrics into K8S. Nothing I have done is working. Funny thing is, I can see the metrics in prometheus, but I am unable to present them to the custom metrics API.
Example metric visible in prometheus:
libvirt_cpu_stats_cpu_time_nanosecs{app="prometheus-lex",domain="virtlet-c91822c8-5e82-beta-deflect",instance="192.168.2.32:9177",job="kubernetes-pods",kubernetes_namespace="default",kubernetes_pod_name="prometheus-lex-866694b884-9z8v6",name="prometheus-lex",pod_template_hash="866694b884"}
Prometheus Adapter configuration for this metric:
- seriesQuery: 'libvirt_cpu_stats_cpu_time_nanosecs{job="kubernetes-pods", app="prometheus-lex"}'
seriesFilters: []
resource:
overrides:
kubernetes_pod_name:
resource: pod
kubernetes_namespace:
resource: namespace
name:
matches: libvirt_cpu_stats_cpu_time_nanosecs
as: libvirt_cpu_stats_cpu_time_rate
metricsQuery: rate(libvirt_cpu_stats_cpu_time_nanosecs{job="kubernetes-pods", app="prometheus-lex", <<.LabelMatchers>>}[5m])
When I query the custom metrics API, I do not see what I am looking for:
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1|grep libvirt
returns nothing
Additionally, I can see the prometheus-adapter is able to query the series from prometheus. So I know that side of the adapter is working. I am just trying to figure out why it's not presenting them to the custom metrics server.
From the prometheus-adapter
I0220 19:12:58.442937 1 api.go:74] GET http://prometheus-server.default.svc.cluster.local:80/api/v1/series?match%5B%5D=libvirt_cpu_stats_cpu_time_nanosecs%7Bkubernetes_namespace%21%3D%22%22%2Ckubernetes_pod_name%21%3D%22%22%7D&start=1550689948.392 200 OK
Any ideas what I am missing here?
Update::
I have also tried the following new configuration, and it's still not working.
- seriesQuery: 'libvirt_cpu_stats_cpu_time_nanosecs{kubernetes_namespace!="",kubernetes_pod_name!=""}'
seriesFilters: []
resource:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
name:
matches: 'libvirt_cpu_stats_cpu_time_nanosecs'
as: 'libvirt_cpu_stats_cpu_time_rate'
metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
It actually depends on how you install the Prometheus Adapter. If you install via helm and use the YAML as configuration to the rules. You need to follow this README https://github.com/helm/charts/blob/master/stable/prometheus-adapter/README.md and and declare the rules like
rules:
custom:
- seriesQuery: '{__name__=~"^some_metric_count$"}'
resources:
template: <<.Resource>>
name:
matches: ""
as: "my_custom_metric"
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
Pay attention to the custom keyword. If you miss it, the number won't be available via custom metrics.