Is there a known method / keyword/ topic to solve how to decide auto scale threshold value?
Take K8s HPA for example below, I only know I can install some monitoring tools then check memory usage showing on the graph by my eyes to decide a proper threshold value 100Mi. But why not to set it 99Mi, why not to set it 101Mi? I think this method is too manual.
- type: Resource
resource:
name: memory
target:
type: AverageValue
averageValue: 100Mi
As I am not mastering in computer science, I want to ask
Is there a known method on solving this kind of problem?
Or what kind of course will cover this problem?
Or what is the keyword to search from academic article?
In order to display this information without any graph you can use metrics server. Running it in your cluster makes it possible to get usage for nodes and individual pods through the kubectl top command.
Here`s an example where I'm checking the node resouces:
➜ ~ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
minikube 580m 28% 1391Mi 75%
And for a pod:
➜ ~ kubectl top pod
NAME CPU(cores) MEMORY(bytes)
front-end 0m 28Mi
You can also see resource usages across individual containers instead of pods using the --containers option.
I assume that if you use HPA you have this already installed but it's worth to know that If you use minikube you can easily enable metrics server with minikube addons enable metrics-server. If you bootstrap your server using kubeadm then you have to install it and configure with all of it`s requirements in order to run correctly.
Lastly you can always check manually your pod usage with exec into it:
kubectl exec -it <name_of_the_pod> top
You can here for more prod information about autoscalers.
Related
We have a cluster built via Rancher, I have monitoring enabled and I can see the monitoring showing exactly the double of used memory as the "kubectl top pod" command, and this is causing memory used higher than limit and impossible to deploy an HorizontalPodAutoScaler on memory usage...
Anyone has ever been in this problem or knows where should I star looking for the problem?
# kubectl top pod xxxxxxx-api-66f8446df9-drccw
NAME CPU(cores) MEMORY(bytes)
xxxxxxx-api-66f8446df9-drccw 113m 310Mi
And Rancher UI:
Rancher v2.3.5
User Interface v2.3.36
Helm v2.14.3-rancher1
Machine v0.15.0-rancher29
I'm new in Kubernetes and currenlty I'm researching about profiling in Kubernetes. I want to log deployment process in Kubernetes (creating pod, restart pod, etc) and want to know the time and resources(RAM, CPU) needed in each process (for example when downloading image, building deployment, pod, etc).
Is there a way or tool for me to log this process? Thank you!
I am not really sure you can achieve the outcome you want without extensive knowledge about certain components and some deep dive coding.
What can be retrieved from Kubernetes:
Information about events
Like pod creation, termination, allocation with timestamps:
$ kubectl get events --all-namespaces
Even in the json format there is nothing about CPU/RAM usage in this events.
Information about pods
$ kubectl get pods POD_NAME -o json
No information about CPU/RAM usage.
$ kubectl describe pods POD_NAME
No information about CPU/RAM usage either.
Information about resource usage
There is some tools to monitor and report basic resource usage:
$ kubectl top node
With output:
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
MASTER 90m 9% 882Mi 33%
WORKER1 47m 5% 841Mi 31%
WORKER2 37m 3% 656Mi 24%
$ kubectl top pods --all-namespaces
With output:
NAMESPACE NAME CPU(cores) MEMORY(bytes)
default nginx-local-84ddb99b55-2nzdb 0m 1Mi
default nginx-local-84ddb99b55-nxfh5 0m 1Mi
default nginx-local-84ddb99b55-xllw2 0m 1Mi
There is CPU/RAM usage but in basic form.
Information about deployments
$ kubectl describe deployment deployment_name
Provided output gives no information about CPU/RAM usage.
Getting information about resources
Getting resources like CPU/RAM usage specific to some actions like pulling the image or scaling the deployment could be problematic. Not all processes are managed by Kubernetes and additional tools at OS level might be needed to fetch that information.
For example pulling an image for deployment engages the kubelet agent as well as the CRI to talk to Docker or other Container Runtime your cluster is using. Adding to that, the Container Runtime not only downloads the image, it does other actions that are not directly monitored by Kubernetes.
For another example HPA (Horizontal Pod Autoscaler) is Kubernetes abstraction and getting it's metrics would be highly dependent on how the metrics are collected in the cluster in order to determine the best way to fetch them.
I would highly encourage you to share what exactly (case by case) you want to monitor.
You can find these in the events feed for the pod, check kubectl describe pod.
I have a list of namespaces created under the same k8s cluster and I'd like to find out the resource (CPU, memory) usage per namespace. Is there any command I can use?
Yes. You can use
$ kubectl -n <nampespace> top pod
For example:
$ kubectl top pod -n kube-system
NAME CPU(cores) MEMORY(bytes)
calico-node-xxxxx 17m 166Mi
coredns-xxxxxxxxxx-xxxxx 2m 11Mi
coredns-xxxxxxxxxx-xxxxx 3m 11Mi
etcd-ip-x-x-x-x.us-west-2.compute.internal 19m 149Mi
kube-apiserver-ip-x-x-x-x.us-west-2.compute.internal 39m 754Mi
kube-controller-manager-ip-x-x-x-x.us-west-2.compute.internal 20m 138Mi
kube-proxy-xxxxx 5m 12Mi
kube-scheduler-ip-x-x-x-x.us-west-2.compute.internal 6m 17Mi
metrics-server-xxxxxxxxxx-xxxxx 0m 15Mi
You need to add up all the entries on the CPU and MEMORY columns if you want the total.
Note that for kubectl top to work you need to have the metrics-server set up and configured appropriately. (Older clusters use the heapster)
Write a shell script to get all namespaces in the cluster. Iterate through each namespace. Run kubectl top pod.
Add up the cpu and memory of all pods in the namespace.
Thanks Rico, the answer is good but just as an addition:
You can specify resource quotas and then view them as specified here.
Other than that, there are external monitoring tools like Prometheus. Also, there is a Resource Explorer which can:
Display historical statistical resource usage from StackDriver.
https://github.com/kubernetes/kubernetes/issues/55046
List resource QoS allocation to pods in a cluster. Inspired by:
https://github.com/kubernetes/kubernetes/issues/1751
The case is still open on GitHub, but it seems there should be some changes eventually as one of the contributors states there is a plan to remove kubectl top and using some native solutions so I advise to follow this thread.
I'd like to know the current limit on the RAM. (No limit/request was explicitly configured.)
How do I see the current configuration of an existing pod?
[Edit] That configuration would include not only how much memory is now in use, but also the max-limit, the point at which it would be shut down.
(If I blow up the heap with huge strings, I see a limit of approx 4 GB, and the Google Cloud Console shows a crash at 5.4 GB (which of course includes more than the Python interpreter), but I don't know where this comes from. The Nodes have up to 10 GB.)
I tried kubectl get pod id-for-the-pod -o yaml, but it shows nothing about memory.
I am using Google Container Engine.
Use kubectl top command
kubectl top pod id-for-the-pod
kubectl top --help
Display Resource (CPU/Memory/Storage) usage.
The top command allows you to see the resource consumption for nodes
or pods.
This command requires Heapster to be correctly configured and working
on the server.
Available Commands: node Display Resource
(CPU/Memory/Storage) usage of nodes pod Display Resource
(CPU/Memory/Storage) usage of pods
Usage: kubectl top [flags] [options]
The edit in the question asks how to see the max memory limit for an existing pod. This shold do:
kubectl -n <namespace> exec <pod-name> cat /sys/fs/cgroup/memory/memory.limit_in_bytes
Reference: https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
With QoS class of BestEffort (seen in the output from kubectl -n <namespace> get pod <pod-name> -o yaml or kubectl -n <namespace> describe pod <pod-name>), there may be no limits (other than the available memory on the node where the pod is running) so the value returned can be a large number (e.g. 9223372036854771712 - see here for an explanation).
You can use
kubectl top pod POD_NAME
It will show you memory and CPU usage.
[Edit: See comment for more]
As already answered by the community, you can run "kubectl top pod POD_NAME" to get how much memory your pod is using. The max limit actually depends on the available memory of nodes (You may get an idea of CPU Requests and CPU Limits of nodes by running "kubectl describe nodes"). Furthermore, the max limit of the pod also depends on its memory requests and limits as defined in the pod's configuration ("requests" and "limits" specs under "resources"). You can also read this relevant link.
Deploy Metrics Server in Kubernetes Cluster (Heapster is deprecated) and then use
kubectl top POD_NAME
to get pod CPU and memory usages.
Answer from comment from #Artem Timchenko: kubectl -n NAMESPACE describe pod POD_NAME | grep -A 2 "Limits"
I'm trying to deploy Magento on a GCE n1-standard-1 machine, but I keep getting the following error message.
pod (magento-magento-1486272877-zd34d) failed to fit in any node fit failure summary on nodes : Insufficient cpu (1)
I'm using the official Magento helm chart, and I've configured the values.yml file to contain very low CPU requests: cpu: 25m
When I look at the node details on the kubernetes dashboard, I see that my CPU is already spinning at 0.728 (72.80%) while it's not even doing anything besides the system containers. Also see image below:
Does this mean I have 1 - 0.728 = 0.272m left for container requests? Then why is kubernetes still telling me that it has insufficient CPU when specifying 0.25m?
Thanks for your help.
I didn't see that the CPU limits were 0.248 according to the picture in my post, so I put cpu: 20m and it worked.
There is a nifty kubectl command to get information about your nodes resources...
kubectl top nodes
And pods...
kubectl top pods
Pods with containers
kubectl top pods --containers=true