I have added some labels to kubernetes namespace metadata, now I want to scrape namespace as well as those labels using prometheus, Actually I am trying to create a grafana dashboard and want to categorise namespaces based on labels. I tried using kubernetes_sd_configs, I am able to get namespace but unable to get labels of those namespaces. Does anyone know of any way to scrape labels along with namespace.
In case somebody is also looking for answer, we can use kube-state-metrics
It exposes kube_namespace_labels metric which has labels as well as namespace.
Related
I have a Kubernetes Pod which serves metrics for prometheus.
Once in a while I update the release and thus the pod gets restarted.
Prometheus safes the metrics but labels it according to the new pod name:
this is by prometheus' design, so its ok.
but if I display this data with grafana, Im getting this (the pods ahve been redeployed twice):
So for example the metric "Registered Users" now has 3 different colors because the source from it comes from 3 diffferent pods
I have some options. Maybe disregard the pod name in prometheus, but I consider that bad practise because I dont want to lose data.
So I think I have to consolidate this in grafana. But how I can I tell Grafana that I want to merge all data with container-name api-gateway-narkuma and disregard the label pods?
You can do something like
max(users) without (instance, pod)
There is a label like envName=vnc in kubernetes pod yaml file, the original metrics collected by prometheus is:
container_memory_usage_bytes{container_name="vncserver-vscode",id="......",image="......",name="......",namespace="kube-system",pod_name="vncserver-vscode-59878ff5cf-l2dd9"} 6.28785152e+08
And I want add the label to the metrics, like this:
container_memory_usage_bytes{container_name="vncserver-vscode",id="......",image="......",name="......",namespace="kube-system",pod_name="vncserver-vscode-59878ff5cf-l2dd9", envName="vnc"} 6.28785152e+08
Does prometheus support that?
Unfortunately there's no way to do this. You can not change/add labels in metrics that already exists.
You will need to create new scrape with desired labels.
I am scraping the kubernetes metrics from prometheus and would need to extract the number of running pods.
I can see container_last_seen metrics but how should i get no of pods running. Can someone help on this?
If you need to get number of running pods, you can use a metric from the list of pods metrics https://github.com/kubernetes/kube-state-metrics/blob/master/docs/pod-metrics.md for that (To get the info purely on pods, it'd make sens to use pod-specific metrics).
For example if you need to get the number of pods per namespace, it'll be:
count(kube_pod_info{namespace="$namespace_name"}) by (namespace)
To get the number of all pods running on the cluster, then just do:
count(kube_pod_info)
Assuming you want to display that in Grafana according to your question tags, from this Kubernetes App Metrics dashboard for example:
count(count(container_memory_usage_bytes{container_name="$container", namespace="$namespace"}) by (pod_name))
You can just import the dashboard and play with the queries.
Depending on your configuration/deployment, you can adjust the variables container_name and namespace, grouping by (pod_name) and count'ing it does the trick. Some other label than pod_name can be used as long as it's shared between the pods you want to count.
If you want to see only the number of "deployed" pods in some namespace, you can use the solutions in previous answers.
My use case was to see the current running pods in some namespace and below is my solution:
'min_over_time(sum(group(kube_pod_container_status_ready{namespace="BC_NAME"}) by (pod,uid)) [5m:1m]) OR on() vector(0)'
Please replace BC_NAME with your namespace name.
The timespan provides you fine the data.
If no data found - no pod currently running it returns '0'
I have setup Prometheus and Grafana for monitoring my kubernetes cluster and everything works fine. Then I have created custom dashboard in Grafana for my application.The metrics available in Prometheus is as follows and i have added the same in grafana as metrics:
sum(irate(container_cpu_usage_seconds_total{namespace="test", pod_name="my-app-65c7d6576b-5pgjq", container_name!="POD"}[1m])) by (container_name)
The issue is, my application is running as pod in kubernetes,so when the pod is deleted or recreated, then the name of the pod will change and it will be different than the pod name specified in the above metrics "my-app-65c7d6576b-5pgjq". So the data for the above metrics will not work anymore. and I have to add new metrics again in Grafana. Please let me know How can I overcome this situation.
Answer was provided by manu thankachan:
I have done it. Made some change in the query as follow:
sum(irate(container_cpu_usage_seconds_total{namespace="test",
container_name="my-app", container_name!="POD"}[1m])) by
(container_name)
If pod is created directly(not a part of deployment) then only pod name is same as we mentioned.
If pod is part of Deployment the it will have unique string from replicaset and also ends with random 5 characters to maintain unique name.
So always try to use container_name label or if your Kubernetes version is > v1.16.0 then use container label
I am creating a grafana dashboard to show metrices.
Is there any keyword which i can use in my query to check my service is running or not.
I am using prometheus to retrive data from my api for the metrics creation.
You can create a query like so
count_scalar(container_last_seen{name=<container_name>})
That will give you the count of how many containers are running with that name