I'm trying to import some pre-built Grafana dashboards for Kubernetes monitoring but I don't get why some metrics seem to be duplicated or multiplied.
For example, this metric is yielding 6 nodes:
sum(kube_node_info{node=~"$node"})
Which is double than the what the cluster actually has:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-agentpool-vmss000000 Ready agent 57d v1.23.5
aks-agentpool-vmss000001 Ready agent 57d v1.23.5
aks-agentpool-vmss000007 Ready agent 35d v1.23.5
Another example:
This metrics is yielding a total of 36 cores, when in reality there are only 12 (3 nodes x 4 cores each)
sum (machine_cpu_cores{kubernetes_io_hostname=~"^$Node$"})
Capacity:
cpu: 4
ephemeral-storage: 129900528Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 16393292Ki
pods: 110
If I filter que query by system_uuid, each of the 3 uuids yield 12 cores.
The same goes for memory usage, filesystem storage and so on. Any idea why the metrics are multiplied?
The dashboard in question is this: https://grafana.com/grafana/dashboards/10000
Related
I have one strange issue.
The error that I'm getting is:
unschedulable 0/1 nodes are available insufficient ephemeral-storage
My requests per workflow that I run in kubernetes are:
resources:
requests:
ephemeral-storage: 50Gi
memory: 8Gi
And my node capacity is 100GiB per node.
When I run kubectl describe node <node-name> I get the following result:
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 125m (3%) 0 (0%)
memory 8Gi (55%) 0 (0%)
ephemeral-storage 50Gi (56%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Do ephemeral-storage and memory overlap? What can be the issue? I cannot resolve.
In the kubectl describe node output, Kubernetes believes it's used 50 GiB of disk ("ephemeral storage") and that's 56% of the available resources. That implies there's about 89 GiB of usable disk space, and about 39 GiB left, so less space than your container claims it needs.
If the node has a 100 GiB disk, space required by the operating system, Kubernetes, and any pulled images counts against that disk capacity before being considered available for ephemeral storage. You probably will never be able to run two Pods that both require 50 GiB of disk; with the OS overhead they will not both fit at the same time.
(It's also not impossible that the node has 100 GB and not 100 GiB storage. 100 * 10^9 is only 93 * 2^30, which would make that overhead about 4 GiB, which feels a little more typical to me.)
The easiest and "most Kubernetes" option here is to get another node, maybe via the cluster autoscaler. If you do control the node configuration, changing nodes to more like 120 GB storage would make these Pods fit better. Especially in an AWS/EKS context, current Kubernetes also supports generic ephemeral volumes which would let you get per-pod storage backed by a volume (on AWS, most likely an EBS volume) rather than fixed-size local disk.
I'm running a 4vCPU 8GB Node Pool, but all of my nodes report this for Capacity:
Capacity:
cpu: 4
ephemeral-storage: 165103360Ki
hugepages-2Mi: 0
memory: 8172516Ki
pods: 110
I'd expect it to show 8388608Ki (the equivalent of 8192Mi/8Gi).
How come?
Memory can be reserved for both system services (system-reserved) and the Kubelet itself (kube-reserved). https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ has details but DO is probably setting it up for you.
I am a newbie and I may ask a stupid question, but I could not find answers on Kind or on stackoverflow, so I dare asking:
I run kind (Kubernestes-in-Docker) on a Ubuntu machine, with 32GB memory and 120 GB disk.
I need to run a Cassandra cluster on this Kind cluster, and each node needs at least 0.5 CPU and 1GB memory.
When I look at the node, it gives this:
Capacity:
cpu: 8
ephemeral-storage: 114336932Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32757588Ki
pods: 110
Allocatable:
cpu: 8
ephemeral-storage: 114336932Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32757588Ki
pods: 110
so in theory, there is more than enough resources to go. However, when I try to deploy the cassandra deployment, the first Pod keeps in a status 'Pending' because of a lack of resources. And indeed, the Node resources look like this:
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 100m (1%) 100m (1%)
memory 50Mi (0%) 50Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
The node does not get actually access to the available resources: it stays limited at 10% of a CPU and 50MB memory.
So, reading the exchange above and having read #887, I understand that I need to actually configure Docker on my host machine in order for Docker to allow the containers simulating the Kind nodes to grab more resources. But then... how can give such parameters to Kind so that they are taken into account when creating the cluster ?
\close
Sorry for this post: I finally found out that the issue was related to the storageclass not being properly configured in the spec of the Cassandra cluster, and not related to the dimensioning of the nodes.
I changed the cassandra-statefulset.yaml file to indicate the 'standard' storageclass: this storageclass is provisionned by default on a KinD cluster since version 0.7. And it works fine.
Since Cassandra is resource hungry, and depending on the machine, you may have to increase the timeout parameters so that the Pods would not be considered faulty during the deployment of the Cassandra cluster. I had to increase the timouts from respectively 15 and 5s, to 25 and 15s.
This topic should be closed.
I get this error when i tried to deploy one deployment with ten replicas.
0/2 nodes are available: 1 Insufficient memory, 1 node(s) had taints that the pod didn't tolerate.
I don't understand why two node. Is the same node and just the same problem.
I have a lot of RAM (1GB) free.
How can i fix this error with out add another node.
I have in deployment yaml file this for resources:
limits:
cpu: 1000m
memory: 1000Mi
requests:
cpu: 100m
memory: 200Mi
Server:
Master:
CPU: 2
RAM: 2 - 1 Free
Slave:
CPU: 2
RAM: 2 - 1 Free
I think you have multiple issues here.
First to the format of the error message you get
0/2 nodes are available: 1 Insufficient memory, 1 node(s) had taints that the pod didn't tolerate.
The first thing is clear you have 2 nodes in total an could not schedule to any of them. Then comes a list of conditions which prevent the scheduling on that node. One node can be affected by multiple issues. For example, low memory and insufficient CPU. So, the numbers can add up to more than what you have on total nodes.
The second issue is that the requests you write into your YAML file apply per replica. If you instantiate the same pod with 100M Memory 5 times they need 500M in total. You want to run 10 pods which request each 200Mi memory. So, you need 2000Mi free memory.
Your error message already implies that there is not enough memory on one node. I would recommend you inspect both nodes via kubectl describe node <node-name> to find out how much free memory Kubernetes "sees" there. Kubernetes always blocks the full amount of memory a pod requests regardless how much this pod uses.
The taints in your error message tells that the other node, possibly the master, has a taint which is not tolerated by the deployment. For more about taints and tolerations see the documentation. In short find out which taint on the node prevents the scheduling and remove it via kubectl taint nodes <node-name> <taint-name>-.
When I create a Pod that cannot be scheduled because there are no nodes with sufficient CPU to meet the Pod's CPU request, the events output from kubectl describe pod/... contain a message like No nodes are available that match all of the following predicates:: Insufficient cpu (3).
What does the (3) in Insufficient cpu (3) mean?
For example, if I try to create a pod that requests 24 CPU when all of my nodes only have 4 CPUs:
$ kubectl describe pod/large-cpu-request
Name: large-cpu-request
Namespace: default
Node: /
Labels: <none>
Annotations: <none>
Status: Pending
IP:
Controllers: <none>
Containers:
cpuhog:
...
Requests:
cpu: 24
...
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
23m 30s 84 default-scheduler Warning FailedScheduling No nodes are available that match all of the following predicates:: Insufficient cpu (3).
At other times I have seen event messages like No nodes are available that match all of the following predicates:: Insufficient cpu (2), PodToleratesNodeTaints (1) when a pod's resource requests were too high, so the 3 does not seem like a constant number - nor does it seem related to my 24 CPU request either.
It means that your Pod doesn't fit on 3 nodes because of Insufficient CPU and 1 node because of taints (likely the master).
A pod can't be scheduled when it requests more cpu than you have in your cluster. For example, if you have 8 Kubernetes CPU (see this page to calculate how many kubernetes cpu you have) in total and if your existing pods have already consumed that much cpu then you can't schedule more pods unless some of your existing pods are killed by the time you request to schedule a new pod. Here is a simple equation can be followed in Horizontal Pod Autoscaler (HPA):
RESOURCE REQUEST CPU * HPA MAX PODS <= Total Kubernetes CPU
You can always tune up these numbers. In my case, I adjusted my manifest file for the RESOURCE REQUEST CPU. It can be 200m, or 1000m (= 1 kubernetes cpu).