I have 10 Kubernetes nodes (consider them as VMs) which have between 7 and 14 allocatable CPU cores which can be requested by Kubernetes pods. Therefore I'd like to show cluster CPU usage.
This is my current query
sum(kube_pod_container_resource_requests_cpu_cores{node=~"$node"}) / sum(kube_node_status_allocatable_cpu_cores{node=~"$node"})
This query shows strange results, for example over 400%.
I would like to add filter to only calculate this for nodes that have Running pods, since there might be some old node definitions which are not user. I have inherited this setup, so it is not that easy for me to wrap my head around it.
Any suggestions with a query that I can try?
Your current query is summing up CPU utilization of each nodes so it might show invalid data.
You can check CPU utilization of all pods in the cluster by running:
sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name!=""}[5m]))
If you want to check CPU usage of each running pod you can use using:
sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name!=""}[5m])) by (pod_name)
Related
I notice the cpu utilization of pods in same hpa varies from 31m to 1483m. Is this expected and normal? See below for the cpu utilization of 8 pods which are of the same hpa.
NAME CPU(cores)
myapp-svc-pod1 31m
myapp-svc-pod2 87m
myapp-svc-pod3 1061m
myapp-svc-pod4 35m
myapp-svc-pod5 523m
myapp-svc-pod6 1483m
myapp-svc-pod7 122m
myapp-svc-pod8 562m
HPA main goal is to spawn more pods to keep average load for a group of pods on specified level.
HPA is not responsible for Load Balancing and equal connection distribution.
For equal connection distribution is responsible k8s service, which works by deafult in iptables mode and - according to k8s docs - it picks pods by random.
Your uneven cpu load distribution is most probably caused by the data it processes. To make sure it's not the issue with k8s service, I'd recomment you to export some metrics like number of connections and time it takes to process one request. After you gathered this data, have a look at it and see if a pattern emerges.
Now to answer your question:
Is this expected and normal?
It depends what you consider as normal, but if you were expecting more equal cpu load distribution then you may want to rethink your design. It's hard to say what you can do to make it more equal because I don't know what myapp-svc-pods do, but as I already mentioned, it may be best to have a look at the metrics.
How to figure out how much min and max resources to allocate for each application deployment? I'm setting up a cluster and I haven't setup any resources and letting it run freely.
I guess I could use top command to figure out the load during the peak time and work on that but still top says like 6% or 10% but then I'm not sure how to calculate them to produce something like 0.5 cpu or 100 MB. Is there a method/formula to determine max and min based on top command usage?
I'm running two t3.medium nodes and I have the following pods httpd and tomcat in namespace1, mysql in namepsace2, jenkins and gitlab in namespace3. Is there any guide to minimum resources it needs? or Do I have to figure it based on top or some other method?
There are few things to discuss here:
Unix top and kubectl top are different:
Unix top uses proc virtual filesystem and reads /proc/meminfo file to get an actual information about the current memory usage.
kubectl top shows metrics information based on reports from cAdvisor, which collects the resource usage. For example: kubectl top pod POD_NAME --containers: Show metrics for a given pod and its containers or kubectl top node NODE_NAME: Show metrics for a given node.
You can use the metrics-server to get the CPU and memory usage of the pods. With it you will be able to Assign CPU Resources to Containers and Pods.
Optimally, your pods should be using exactly the amount of resources you requested but that's almost impossible to achieve. If the usage is lower than your request, you are wasting resources. If it's higher, you are risking performance issues. Consider a 25% margin up and down the request value as a good starting point. Regarding limits, achieving a good setting would depend on trying and adjusting. There is no optimal value that would fit everyone as it
depends on many factors related to the application itself, the
demand model, the tolerance to errors etc.
As a supplement I recommend going through the Managing Resources for Containers docs.
I'm trying to figure out why a GKE "Workload" CPU usage is not equivalent to the sum of cpu usage of its pods.
Following image shows a Workload CPU usage.
Service Workload CPU Usage
Following images show pods CPU usage for the above Workload.
Pod #1 CPU Usage
Pod #2 CPU Usage
For example, at 9:45, the Workload cpu usage was around 3.7 cores, but at the same time Pod#1 CPU usage was around 0.9 cores and Pod#2 CPU usage was around 0.9 cores too. It means, the service Workload CPU Usage should have been around 1.8 cores, but it wasn't.
Does anyone have an idea of this behavior?
Thanks.
On your VM, the node managed by Kubernetes, you have the deployed pods (that you manage) but also several services that run on it for the supervision, the management, the logs ingestion,... A basic description here
You can see all these basic services by performing this command kubctl get all --namespace kube-system.
If you have installed additional components, like Istio or Knative, you have additional services and namespaces. All of these get a part of the resources of the node.
Danny,
The CPU chart on the Workloads page is an aggregate of CPU usage for managed pods. The values are taken from the Stackdriver Monitoring metric container/cpu/usage_time, check this link. That metric represents "Cumulative CPU usage on all cores in seconds. This number divided by the elapsed time represents usage as a number of cores, regardless of any core limit that might be set."
Please let me know if you have further questions in regard to this.
I suspect this is a bug in the UI. There is no actual metric for deployment CPU usage. Stackdriver Monitoring only collects data on container, pod, and node level metrics thus the only really reliable metrics in this case are the ones for pod CPU usage.
The graph for the total deployment CPU usage is likely meant to be a sum of all the pods metrics calculated and then presented to you. It is not as reliable as the pod or container metrics since it is not a direct metric.
If you are seeing this discrepancy consistently, I recommend opening a UI bug report through the Google Public Issue Tracker to report this to the GCP Engineers.
I am trying to assign CPU resources for services running in kubernetes pods. Services are mostly nodejs based REST endpoints with some DB operations.
During the load test, tried different combinations between 100m and 1000m for pods.
For the expected number of requests/second, when value is less than < 500m more pods are getting spawned as part of HPA than when the value is >500m. I am using CPU based trigger for HPA.
I couldn't figure out depending on what I shall be selecting particular CPU resource value. Can someone help me in this regard?
Two points:
If you configured the HPA to autoscale based on CPU utilisation, it makes sense that there are more replicas if the CPU request is 500m than if it's 1000m. This is because the target utilisation that you define for the HPA is relative to the CPU request of the Pods.
For example, if your target utilisation is 80% and the CPU request of the Pods is 500m, then the HPA scales your app up if the actual CPU usage exceeds 400m. On the other hand, if the CPU requests are 1000m, then the HPA only scales up if the CPU usage exceeds 800m.
Selecting resource requests (e.g. CPU) for a container is very important, but it's also an art in itself. The CPU is the minimum amount of CPU that your container needs for running reliably. What you could do to find out this value is running your app locally and try to evaluate how much CPU it is actually using, for example, with ps or top.
I've set up Prometheus and Grafana for tracking and monitoring of my Kubernetes cluster.
I've set up 3 Nodes for my cluster.
I have 26 pods running (mostly monitoring namespace).
I have one major Node app (deployment) running and right now there isn't any load.
I'm trying to understand these graph metrics. However I can't understand why there's such high CPU cores usage despite there being no load on the app.
Here's a grafana screenshot
24% memory usage I can understand as there are Kubernetes processes running as well such as kube-system etc.
And it's also telling me my cluster can support 330 pods (currently 26). I'm only worried about high cpu cores. Can anybody explain it.
82% is not the CPU usage of the processes but the ratio of requested to allocatable resources (2.31 / 2.82 = 0.819 --> ~82%).
This means that out of your 2.82 available (allocatable CPUs), you requested (allocated) about 82% for pods in the monitoring namespace but that does not mean they actually use that much CPU.
To see actual CPU usage, look at metrics like container_cpu_usage_seconds_total (per container CPU usage) or maybe even process_cpu_seconds_total (per process CPU usage).