I have a Kubernetes Pod which serves metrics for prometheus.
Once in a while I update the release and thus the pod gets restarted.
Prometheus safes the metrics but labels it according to the new pod name:
this is by prometheus' design, so its ok.
but if I display this data with grafana, Im getting this (the pods ahve been redeployed twice):
So for example the metric "Registered Users" now has 3 different colors because the source from it comes from 3 diffferent pods
I have some options. Maybe disregard the pod name in prometheus, but I consider that bad practise because I dont want to lose data.
So I think I have to consolidate this in grafana. But how I can I tell Grafana that I want to merge all data with container-name api-gateway-narkuma and disregard the label pods?
You can do something like
max(users) without (instance, pod)
I have used custom metrics API and have successfully auto scaled the service based on some metrics.But here is the point the auto scaling works in way that the minikube's HPA(horizontal pod autoscaler) will check the particular URL/API and try to find out the metric value repetitively with some polling period.For example HPA will check for the value for every 15 seconds.So this is continuous polling from HPA to the URL/API to fetch the value of that metric.After that it will simply compare the value with the target reference value given and try to scale.
What I want is, the API/URL should trigger the minikube HPA whenever needed, it's like HPA should work as a interrupt here in simple words.
call for autoscale should be from Service/API to HPA not from HPA to Service !
is this feature available in Kubernetes ? or do you have any comments in this scenario ? please share your view on this I am currently in last stage and this question is stopping my progress!
Google publishes a tutorial for using custom metrics to drive the HorizontalPodAutoscaler here, and this tutorial contains instructions for:
Using a Kubernetes manifest to deploy the custom metrics adapter into a custom-metrics namespace.
Deploying a dummy application to generate metrics.
Configuring the HPA to use custom metrics.
We are deploying into a default cluster without any special VPC rules, and we have roughly followed the tutorial's guidance, with a few exceptions:
We're using Helm v2, and rather than grant cluster admin role to Tiller, we have granted all of the necessary cluster roles and role bindings to allow the custom-metrics-adapter-deploying Kubernetes manifest to work. We see no issues there; at least the custom metrics adapter spins up and runs.
We have defined some custom metrics that are based upon data extracted from a jsonPayload in Stackdriver logs.
We have deployed a minute-by-minute CronJob that reads the above metrics and publishes a derived metric, which is the value we want to use to drive the autoscaler. The CronJob is working, and we can see the metric in the derived metric, on a per-Pod basis, in the log metric explorer:
We're configuring the HPA to scale based on the average of the derived metric across all of the pods belonging to a stateful set (The HPA has a metrics entry with type Pods). However, the HPA is unable to read our derived metric. We see this error message:
failed to get object metric value: unable to get metric xxx_scaling_metric: no metrics returned from custom metrics API
Update
We were seeing DNS errors, but these were apparently false alarms, perhaps in the log while the cluster was spinning up.
We restarted the Stackdriver metrics adapter with the command line option --v=5 to get some more verbose debugging. We see log entries like these:
I0123 20:23:08.069406 1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1/namespaces/defaults/pods/%2A/xxx_scaling_metric: (56.16652ms) 200 [kubectl/v1.13.11 (darwin/amd64) kubernetes/2e298c7 10.44.1.1:36286]
I0123 20:23:12.997569 1 translator.go:570] Metric 'xxx_scaling_metric' not found for pod 'xxx-0'
I0123 20:23:12.997775 1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta2/namespaces/default/pods/%2A/xxx_scaling_metric?labelSelector=app%3Dxxx: (98.101205ms) 200 [kube-controller-manager/v1.13.11 (linux/amd64) kubernetes/56d8986/system:serviceaccount:kube-system:horizontal-pod-autoscaler 10.44.1.1:36286]
So it looks to us as if the HPA is making the right query for pods-based custom metrics. If we ask the custom metrics API what data it has, and filter with jq to our metric of interest, we see:
{"kind":"MetricValueList",
"apiVersion":"custom.metrics.k8s.io/v1beta1",
"metadata: {"selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/xxx_scaling_metric"},
"items":[]}
That the items array is empty is troubling. Again, we can see data in the metrics explorer, so we're left to wonder if our CronJob app that publishes our scaling metric is supplying the right fields in order for the data to be saved in Stackdriver or exposed through the metrics adapter.
For what it's worth the resource.labels map for the time series that we're publishing in our CronJob looks like:
{'cluster_name': 'test-gke',
'zone': 'us-central1-f',
'project_id': 'my-project-1234',
'container_name': '',
'instance_id': '1234567890123456789',
'pod_id': 'xxx-0',
'namespace_id': 'default'}
We finally solved this. Our CronJob that's publishing the derived metric we want to use is getting its raw data from two other metrics that are extracted from Stackdriver logs, and calculating a new value that it publishes back to Stackdriver.
We were using the resource labels that we saw from those metrics when publishing our derived metric. The POD_ID resource label value in the "input" Stackdriver metrics we are reading is the name of the pod. However, the stackdriver custom metrics adapter at gcr.io/google-containers/custom-metrics-stackdriver-adapter:v0.10.0 is enumerating pods in a namespace and asking stackdriver for data associated with pods' UIDs, not their names. (Read the adapter's source code to figure this out...)
So our CronJob now builds a map of pod names to pod UIDs (which requires it to have RBAC pod list and get roles), and publishes the derived metric we use for HPA with the POD_ID set to the pod's UID instead of its name.
The reason that published examples of custom metrics for HPA (like this) work is that they use the Downward API to get a pod's UID, and provide that value as "POD_ID". In retrospect, that should have been obvious, if we had looked at how the "dummy" metrics exporters got their pod id values, but there are certainly examples (as in Stackdriver logging metrics) where POD_ID ends up being a name and not a UID.
I have an EKS kubernetes cluster i setup on HPA to be able to scaleup if there is any traffic, but there is unexpected behavior happens with every Deployment. HPA scales up to the maximum number of pods then after 5 minites it scales down again
After so many searches I found that there is a cpu spike happens after the app is redeployed again and this spike takes only mili seconds that’s why it might scale.
So, Do you have an Idea how to prevent this spike from happening or just I want to disable the HPA while deploying or delaying the controller manger to scaleup for example after 1 minute no the default value
Thanks
This is a known issue. There is a bug with hpa implementation. You can check it out here -
Issues - https://github.com/kubernetes/kubernetes/issues/78712 and https://github.com/kubernetes/kubernetes/issues/72775
The Fix is rolled out in version 1.16+ - https://github.com/kubernetes/kubernetes/pull/79035
I was facing the same issue. I have implemented a workaround for the issue. The small script which works like a charm. The steps are -
Set the HPA max to current replicas
Set the image
Wait for deployment to be completed or specified time
Set the max HPA to the original number
https://gist.github.com/shukla2112/035d4e633dded6441f5a9ba0743b4f08
it's bash script, if you are using the Jenkins for deployment, you can simply integrate it or use it independently by accommodating within your deployment workflow.
The fix would be rolled out in - 1.16+https://github.com/kubernetes/kubernetes/pull/79035version #79035 (comment)
I use prometheus to monitor kuernetes cluster. When i use sum(container_fs_reads_total), the result is 0 . How can I find pod's filesystem reads per seconds
Prometheus graphing dashboard may or may not be getting the values for that metric.
Since this is part of cadvisor and this
Verify the k8s pods associated with cadvisor are up and running.
Check to see that your cadvisor web site has data under /containers for the metric.
Verify in the config map for Prometheus that you are scraping/containers inside the scrape_config.
Once you have the Prometheus Dashboard up, go to the Graph tab and see if the metric has any values for the last couple of days or so.
Then check the targets tab and make sure the cadvisor host is a target and is up.
Those are some suggestions to narrow down your search for verifying the data is being collected and scraped.