I have Prometheus setup via Helm from Terraform and it's is configured to connect to my Kubernetes cluster. I open my Prometheus but I am not sure which metric to choose from the list to be able to view the CPU/MEM of running pods/jobs.
Here are all the pods running with the command (test1 is the kube namespace):
kubectl -n test1 get pods
podsrunning
When, I am on Prometheus, I see many metrics related to CPU, but not sure which one to choose:
prom1
I tried to choose one, but the namespace = prometheus and it uses prometheus-node-exporter and I don't see my cluster or my namespace test1 anywhere here.
prom2
Could you please help me? Thank you very much in advance.
UPDATE SCREENSHOT
UPDATE SCREENSHOT
I need to concentrate on this specific namespace, normally with the command:
kubectl get pods --all-namespaces | grep hermatwin
I see the first line with namespace = jobs I think this is namespace.
No result when set calendar to last Friday:
UPDATE SCREENSHOT April 20
I tried to select 2 days with starting date on last Saturday 17 April but I don't see any result:
ANd, if I remove (namespace="jobs") condition, I don't see any result either:
I tried to rerun the job (simulation jobs) again just now and tried to execute the prometheus query while the job was still running mode but I don't get any result :-( Here you can see my jobs where running.
I don't get any result:
When using simple filter, just container_cpu_usage_seconds_total, I can see the namespace="jobs"
node_cpu_seconds_total is a metric from node-exporter, the exporter that brings machine statistics and its metrics are prefixed with node_. You need metrics from cAdvisor, this one produces metrics related to containers and they are prefixed with container_:
container_cpu_usage_seconds_total
container_cpu_load_average_10s
container_memory_usage_bytes
container_memory_rss
Here are some basic queries for you to get started. Be ready that they may require tweaking (you may have different label names):
CPU Utilisation Per Pod
sum(irate(container_cpu_usage_seconds_total{container!="POD", container=~".+"}[2m])) by (pod)
RAM Usage Per Pod
sum(container_memory_usage_bytes{container!="POD", container=~".+"}) by (pod)
In/Out Traffic Rate Per Pod
Beware that pods with host network mode (not isolated) show traffic rate for the whole node. * 8 is to convert bytes to bits for convenience (MBit/s, GBit/s, etc).
# incoming
sum(irate(container_network_receive_bytes_total[2m])) by (pod) * 8
# outgoing
sum(irate(container_network_transmit_bytes_total[2m])) by (pod) * 8
Related
I have a bunch of Rancher clusters I take care of and on some of them developers use PriorityClasses to ensure that some of the more important workloads get scheduled. The 3 PriorityClasses are in 3 digits range so they will not interfere with the default ones. However, at present none of the PriorityClasses is set as default and neither is the preemptionPolicy set so it defaults to PreemptLowerPriority.
None of the rancher, longhorn, prometheus, grafana, etc., workloads have priorityClassName set.
Long story short, I believe this causes havoc on the cluster when resources are in short supply.
Before I take my opinion to the developers I would like to collect some data to back up my story.
The question: How do I detect if the pod was Terminated due to Preemption?
I tried to google the subject but couldn't find anything. I was hoping kube state metrics would have something but I didn't find anything.
Any help would be greatly appreciated.
You can try to look for convincing data like the pod termination reason with help of kubectl.
You can see the last restart logs of a container using the following command:
kubectl logs podname -c containername --previous
You can also use the following command to check the lifecycle events sent by the kubelet to the apiserver about the pod.
kubectl describe pod podname
Finally, You can also write a final message to /dev/termination-log, and this will show up as described in the docs.
To use kubectl commands with rancher kindly refer to this documentation page.
Given a kubernetes cluster with:
prometheus
node-exporter
kube-state-metrics
I like to use the metric container_memory_usage_bytes but select by deployment_name instead of pod.
Selectors like container_memory_usage_bytes{pod_name=~"foo-.+"} if the deployment_name=foo are great as long there is not a deployment with deployment_name=foo-bar.
The same I'd like to achieve with the metric kube_pod_container_resource_limits_memory_bytes.
Is there a way to achieve this?
TL;DR
There is no straightforward way to query prometheus by a deployment-name.
You can query memory usage of a specific deployment by using deployment's labels.
Used query:
sum(
kube_pod_labels{label_app=~"ubuntu.*"} * on (pod) group_right(label_app) container_memory_usage_bytes{namespace="memory-testing", container=""}
)
by (label_app)
There is an awesome article which explains the concepts behind this query. I encourage you to read it:
Medium.com: Amimahloof: Kubernetes promql prometheus cpu aggregation walkthrough
I've included an explanation with example below.
The selector mentioned in the question:
container_memory_usage_bytes{pod_name=~"foo-.+"}
.+ - match any string but not an empty string
with pods like:
foo-12345678-abcde - will match (deployment foo)
foo-deployment-98765432-zyxzy - will match (deployment foo-deployment)
As shown above it will match for both pods and for both deployments.
For more reference:
Prometheus.io: Docs: Prometheus: Querying: Basics
As mentioned earlier, you can use labels from your deployment to pinpoint the resource used by your specific deployment.
Assuming that:
There are 2 deployments in memory-testing namespace:
ubuntu with 3 replicas
ubuntu-additional with 3 replicas
Above deployments have labels the same as their names (they can be different):
app: ubuntu
app: ubuntu-additional
Kubernetes cluster version 1.18.X
Why do I specify Kubernetes version?
Kubernetes 1.16 will remove the duplicate pod_name and container_name metric labels from cAdvisor metrics. For the 1.14 and 1.15 release all pod, pod_name, container and container_name were available as a grace period.
Github.com: Kubernetes: Metrics Overhaul
This means that you will need to substitute the parameters like:
pod with pod_name
container with container_name
To deploy Prometheus and other tools to monitor the cluster I used: Github.com: Coreos: Kube-prometheus
The pods in ubuntu deployment are configured to generate artificial load (stress-ng). This is done to show how to avoid situation where the used resources are doubled.
The resources used by pods in memory-testing namespace:
$ kubectl top pod --namespace=memory-testing
NAME CPU(cores) MEMORY(bytes)
ubuntu-5b5d6c56f6-cfr9g 816m 280Mi
ubuntu-5b5d6c56f6-g6vh9 834m 278Mi
ubuntu-5b5d6c56f6-sxldj 918m 288Mi
ubuntu-additional-84bdf9b7fb-b9pxm 0m 0Mi
ubuntu-additional-84bdf9b7fb-dzt72 0m 0Mi
ubuntu-additional-84bdf9b7fb-k5z6w 0m 0Mi
If you were to query Prometheus with below query:
container_memory_usage_bytes{namespace="memory-testing", pod=~"ubuntu.*"}
You would get output similar to one below (it's cut to show only one pod for example purposes, by default it would show all pods with ubuntu in name and in memory-testing namespace):
container_memory_usage_bytes{endpoint="https-metrics",id="/kubepods/besteffort/podb96dea39-b388-471e-a789-8c74b1670c74",instance="192.168.0.117:10250",job="kubelet",metrics_path="/metrics/cadvisor",namespace="memory-testing",node="node1",pod="ubuntu-5b5d6c56f6-cfr9g",service="kubelet"} 308559872
container_memory_usage_bytes{container="POD",endpoint="https-metrics",id="/kubepods/besteffort/podb96dea39-b388-471e-a789-8c74b1670c74/312980f90e6104d021c12c376e83fe2bfc524faa4d4cee6553182d0fa2e007a1",image="k8s.gcr.io/pause:3.2",instance="192.168.0.117:10250",job="kubelet",metrics_path="/metrics/cadvisor",name="k8s_POD_ubuntu-5b5d6c56f6-cfr9g_memory-testing_b96dea39-b388-471e-a789-8c74b1670c74_0",namespace="memory-testing",node="node1",pod="ubuntu-5b5d6c56f6-cfr9g",service="kubelet"} 782336
container_memory_usage_bytes{container="ubuntu",endpoint="https-metrics",id="/kubepods/besteffort/podb96dea39-b388-471e-a789-8c74b1670c74/1b93889a3e7415ad3fa040daf89f3f6bc77e569d85069de518267666ede8e21c",image="ubuntu#sha256:55cd38b70425947db71112eb5dddfa3aa3e3ce307754a3df2269069d2278ce47",instance="192.168.0.117:10250",job="kubelet",metrics_path="/metrics/cadvisor",name="k8s_ubuntu_ubuntu-5b5d6c56f6-cfr9g_memory-testing_b96dea39-b388-471e-a789-8c74b1670c74_0",namespace="memory-testing",node="node1",pod="ubuntu-5b5d6c56f6-cfr9g",service="kubelet"} 307777536
In this point you will need to choose which metric you will be using. In this example I used the first one. For more of a deep dive please take a look on this articles:
Blog.freshtracks.io: A deep dive into kubernetes metrics part 3 container resource metrics
Ianlewis.org: Almighty pause container
If we were to aggregate this metrics with sum (QUERY) by (pod) we would have in fact doubled our reported used resources.
Dissecting the main query:
container_memory_usage_bytes{namespace="memory-testing", container=""}
Above query will output records with used memory metric for each pod. The container=""parameter is used to get only one record (mentioned before) which does not have container parameter.
kube_pod_labels{label_app=~"ubuntu.*"}
Above query will output record with pods and it's labels with regexp of ubuntu.*
kube_pod_labels{label_app=~"ubuntu.*"} * on (pod) group_right(label_app) container_memory_usage_bytes{namespace="memory-testing", container=""}
Above query will match the pod from kube_pod_labels with pod of container_memory_usage_bytes and add the label_app to each of the records.
sum (kube_pod_labels{label_app=~"ubuntu.*"} * on (pod) group_right(label_app) container_memory_usage_bytes{namespace="memory-testing", container=""}) by (label_app)
Above query will sum the records by label_app.
After that you should be able to get the query that will sum the used memory by a label (and in fact a Deployment).
As for:
The same I'd like to achieve with the metric
kube_pod_container_resource_limits_memory_bytes.
You can use below query to get the memory limit for the deployment tagged with labels as in previous example, assuming that each pod in a deployment is having the same limits:
kube_pod_labels{label_app="ubuntu-with-limits"} * on (pod) group_right(label_app) kube_pod_container_resource_limits_memory_bytes{namespace="memory-testing", pod=~".*"}
You can apply functions like avg(),mean(),max() on this query to get the single number that will be your memory limit:
avg(kube_pod_labels{label_app="ubuntu-with-limits"} * on (pod) group_right(label_app) kube_pod_container_resource_limits_memory_bytes{namespace="memory-testing", pod=~".*"}) by (label_app)
Your memory limits can vary if you use VPA. In that situation you could show all of them simultaneously or use the avg() to get the average for all of the "deployment".
As a workaround to above solutions you could try to work with regexp like below:
container_memory_usage_bytes{pod=~"^ubuntu-.{6,10}-.{5}"}
The following PromQL query should return per-deployment memory usage in Kubernetes:
sum(
label_replace(
container_memory_usage_bytes{pod!=""},
"deployment", "$1", "pod", "(.+)-[^-]+-[^-]+"
)
) by (namespace,deployment)
The query works in the following way:
The container_memory_usage_bytes{pod!=""} time series selector selects all the time series with the name container_memory_usage_bytes and with non-empty pod label. We need to filter out time series without pod label, since such time series account for cgroups hierarchy, which isn't needed in this query. See this answer for more details on this.
The inner label_replace() extracts deployment name from pod label and puts it to deployment label. It expects that pod names are constructed with the following pattern: <deployment>-<some_suffix_1>-<some_suffix_2>.
The outer sum() sums pod memory usage per each group with identical namespace and deployment labels.
I have an application running in Kubernetes (Azure AKS) in which each pod contains two containers. I also have Grafana set up to display various metrics some of which are coming from Prometheus. I'm trying to troubleshoot a separate issue and in doing so I've noticed that some metrics don't appear to match up between data sources.
For example, kube_deployment_status_replicas_available returns the value 30 whereas kubectl -n XXXXXXXX get pod lists 100 all of which are Running, and kube_deployment_status_replicas_unavailable returns a value of 0. Also, if I get the deployment in question using kubectl I'm seeing the expected value.
$ kubectl get deployment XXXXXXXX
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
XXXXXXXX 100 100 100 100 49d
There are other applications (namespaces) in the same cluster where all the values correlate correctly so I'm not sure where the fault may be or if there's any way to know for sure which value is the correct one. Any guidance would be appreciated. Thanks
Based on having the kube_deployment_status_replicas_available metric I assume that you have Prometheus scraping your metrics from kube-state-metrics. It sounds like there's something quirky about its deployment. It could be:
Cached metric data
And/or simply it can't pull current metrics from the kube-apiserver
I would:
Check the version that you are running for kube-state-metrics and see if it's compatible with your K8s version.
Restart the kube-state-metrics pod.
Check the logs kubectl logskube-state-metrics`
Check the Prometheus logs
If you don't see anything try starting Prometheus with the --log.level=debug flag.
Hope it helps.
I am scraping the kubernetes metrics from prometheus and would need to extract the number of running pods.
I can see container_last_seen metrics but how should i get no of pods running. Can someone help on this?
If you need to get number of running pods, you can use a metric from the list of pods metrics https://github.com/kubernetes/kube-state-metrics/blob/master/docs/pod-metrics.md for that (To get the info purely on pods, it'd make sens to use pod-specific metrics).
For example if you need to get the number of pods per namespace, it'll be:
count(kube_pod_info{namespace="$namespace_name"}) by (namespace)
To get the number of all pods running on the cluster, then just do:
count(kube_pod_info)
Assuming you want to display that in Grafana according to your question tags, from this Kubernetes App Metrics dashboard for example:
count(count(container_memory_usage_bytes{container_name="$container", namespace="$namespace"}) by (pod_name))
You can just import the dashboard and play with the queries.
Depending on your configuration/deployment, you can adjust the variables container_name and namespace, grouping by (pod_name) and count'ing it does the trick. Some other label than pod_name can be used as long as it's shared between the pods you want to count.
If you want to see only the number of "deployed" pods in some namespace, you can use the solutions in previous answers.
My use case was to see the current running pods in some namespace and below is my solution:
'min_over_time(sum(group(kube_pod_container_status_ready{namespace="BC_NAME"}) by (pod,uid)) [5m:1m]) OR on() vector(0)'
Please replace BC_NAME with your namespace name.
The timespan provides you fine the data.
If no data found - no pod currently running it returns '0'
I have 3 nodes in a k8s cluster and I need exactly 2 pods to be scheduled in each node, so I would end up having 3 nodes with 2 pods each (6 replicas).
I found that k8s have Pod Affinity/Anti-Affinity feature and that seems to be the correct way of doing.
My problem is: I want to run 2 pods per node but I often use kubectl apply to upgrade my docker image version, and in this case k8s should've be able to schedule 2 new images in each node before terminating the old ones - will the newer images be scheduled if I use Pod Affinity/Anti-Affinity to allow only 2 pods per node?
How can I do this in my deployment configuration? I cannot get it to work.
I believe it is part of kubelet's setting, so you would have to look into kubelet's --max-pods flag, depending on what your cluster configuration is.
The following links could be useful:
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/#kubelet
and
https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/