Visualizing K8S CPU resources Prometheus and Grafana - kubernetes

Question

I'm new with monitoring tools like Prometheus and Grafana and I would like to create dashboard which represents current requests and limits resources and usage for a pod. In addition, this pod has 2 containers inside.
My resources for first container looks like:
resources:
requests:
cpu: "3800m"
memory: "9500Mi"
limits:
cpu: "6500m"
memory: "9500Mi"
and for second container:
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 50m
memory: 50Mi
When executing this query in Prometheus:
rate(container_cpu_usage_seconds_total{pod=~"MY_POD"}[5m])
I get:
And to be honest I dont know how this returned data are valid with resources. On Grafana it looks like this:
In addition I would like to add informations about requests and limits to dashboard, but I don't know how to scale dashboard to show all data.
When I execute this query: kube_pod_container_resource_requests{pod=~"MY_POD"} I get:
And this looks valid compared to my resources. I have proper value for limits too, but I would like to represent all this data (usage, requests, limits) on ona dashboard. Could somebody give my any tips how to achieve this?

Answer 1

Simple, just add 2 more queries
if you don't know where to add, below Metrics Browser and Options tab you
can find + symbol to add more queries
and container_cpu_usage_seconds_total this metrics is a counter and
this will give you the CPU Usage in Cores.
kube_pod_container_resource_requests{pod=~"MY_POD"} and kube_pod_container_resource_limits{pod=~"MY_POD"}. If it's only one Pod means no issues. But if you're having mutliple Pods means try to use Sum
Query A: sum(rate(container_cpu_usage_seconds_total{pod=~"MY_POD"}[5m]))
Query B: sum(kube_pod_container_resource_requests{pod=~"MY_POD"})
Query C: sum(kube_pod_container_resource_limits{pod=~"MY_POD"})
This will looks good without too much details, for more details like container wise data just create three more Panel for Requests, Limits and Usage by Containers and add by(container) after the every Query
Another Approach:
Create Variables for Pod and Container so that you can select the Container which you want to see and add 3 queries in single panel so that the Panel looks more Dynamic and less Noise