Does anyone know how to get a sorted list of pods on a given node based on their disk space consumption?
The command below helps me in listing pods based on a given node but my requirement is to list those pods which are causing high disk space usage as part of investigating and solving the DiskPressure eviction state.
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<NODE_NAME>
I was able to find commands to list CPU and memory usage of nodes (ref: https://github.com/kubernetes/kubernetes/issues/17512#issuecomment-317757388) but nothing related to node disk usage.
alias util='kubectl get nodes --no-headers | awk '\''{print $1}'\'' | xargs -I {} sh -c '\''echo {} ; kubectl describe node {} | grep Allocated -A 5 | grep -ve Event -ve Allocated -ve percent -ve -- ; echo '\'''
# Get CPU request total (we x20 because because each m3.large has 2 vcpus (2000m) )
alias cpualloc='util | grep % | awk '\''{print $1}'\'' | awk '\''{ sum += $1 } END { if (NR > 0) { print sum/(NR*20), "%\n" } }'\'''
# Get mem request total (we multiply by 75 because because each m3.large has 7.5G ram )
alias memalloc='util | grep % | awk '\''{print $5}'\'' | awk '\''{ sum += $1 } END { if (NR > 0) { print sum/(NR*75), "%\n" } }'\'''
found an answer to my requirement - list pods disk usage running in a given node
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.pods[0] | "PodName: ", .podRef.name, "usedBytes: ", .containers[].rootfs.usedBytes'
Some other stats:
get node filesystem usage
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.fs.usedBytes'
get node imageFs usage
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.runtime.imageFs.usedBytes'
get node iNodes stats
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.fs.inodesFree'
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.runtime.imageFs.inodesFree'
Unfortunately you won't be able to achieve this with kubectl as it does not have this kind of functionality. One way to go is login to node and the use docker ps -s.
Another way that I can think of is to expose kubelet/cadvisor metrics and use it with metrics scraper like Prometheus.
The kubelet exposes all of it’s runtime metrics, and all of the cAdvisor metrics, on a /metrics endpoint in the Prometheus exposition format. Note that kubelet also exposes metrics in /metrics/cadvisor, /metrics/resource and /metrics/probes endpoints. Disk metrics are generally available on the /stats/summary endpoint.
Additional info:
Stack case with great example how to curl kubelet's endpoint.
Kubernetes metrics documents at /cluster-administration/system-metrics/
Related
Using this command, I am able to get the container name that have restarted.
kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[?(#.restartCount>0)].name}'
Is there a way to get the pod name as well in the same command?
It's much easier to get json with kubectl and then process it with jq :
#!/usr/bin/env bash
kubectl get pods -o=json |
jq -r '.items[] |
"\(.metadata.name) \(.status.containerStatuses[]|select(.restartCount>0).name)"'
I've not tried this.
If (!) it works, it's not quite what you want as it should give you every Pod name and then a list of Container names that match the predicate.
I think you can't use kubectl --output=jsonpath alone to filter only the Pod names that have a Container with restarts.
FILTER='
{range .items[*]}
{.metadata.name}
{"\t"}
[
{.status.containerStatuses[?(#.restartCount>0)].name}
]
{"\n"}
{end}
'
kubectl get pods \
--output=jsonpath="${FILTER}"
In kubectl, describe and get -o can be used to get the details of a resource, can we get a describe of only a list of selected pods (different labels), my case is to details of selected pod names for analysis. I'm using describe but unable to get details of specific pods, there are quite a few to do it manually.
If it was events/ specific labels, i could do the below but need more specific details of certain pods out of 100 of them
kubectl get describe po -l app2=test > desc.txt
Kubectl get events -n test|grep -E 'app2|app3|hello1' > events.txt
Tried the below it returns the events of all pods in the namespace (doesnt help), any simpler way or may be this cannot be done or write a script to loop through the o/p?
kubectl get po -n test |grep -E 'app2|app3|hello1' |awk '{print $1}'|k describe po -n test > desc.txt
Thanks for the help!
Are you saying you can't use labels?
Not sure I understand correctly but if you need to choose pods by name - this will work:
kubectl get pod -oname | grep -E 'app2|app3|hello' | xargs kubectl describe
I've this command to list the Kubernetes pods that are not running:
sudo kubectl get pods -n my-name-space | grep -v Running
Is there a command that will return a count of pods that are not running?
If you add ... | wc -l to the end of that command, it will print the number of lines that the grep command outputs. This will probably include the title line, but you can suppress that.
kubectl get pods -n my-name-space --no-headers \
| grep -v Running \
| wc -l
If you have a JSON-processing tool like jq available, you can get more reliable output (the grep invocation will get an incorrect answer if an Evicted pod happens to have the string Running in its name). You should be able to do something like (untested)
kubectl get pods -n my-namespace -o json \
| jq '.items | map(select(.status.phase != "Running")) | length'
If you'll be doing a lot of this, writing a non-shell program using the Kubernetes API will be more robust; you will generally be able to do an operation like "get pods" using an SDK call and get back a list of pod objects that you can filter.
You can do it without any external tool:
kubectl get po \
--field-selector=status.phase!=Running \
-o go-template='{{len .items}}'
the filtering is done with field-selectors
the counting is done with go-template: {{ len .items }}
newbie here . I am trying to automate the way to get the nodes that are ready using the following script:
until kubectl get nodes | grep -m 2 "Ready"; do sleep 1 ; done
Is there a better way to do this, specifically i am looking for a way to do this without having to specify the node number?
To get the names of all Ready nodes, use
$ kubectl get nodes -o json | jq -r '.items[] | select(.status.conditions[].type=="Ready") | .metadata.name '
master-0
node-1
node-3
node-x
Basing on the official documentation
You can check your ready nodes by using this command:
JSONPATH='{range .items[*]}{#.metadata.name}:{range #.status.conditions[*]}{#.type}={#.status};{end}{end}' \
&& kubectl get nodes -o jsonpath="$JSONPATH" | grep "Ready=True"
You don't need to specify number of nodes in your command, just use it like this:
until kubectl get nodes | grep -i "Ready"; do sleep 1 ; done
I want to get the name of the oldest pod as part of a script. It seems like I should be able to run kubectl get po --no-headers=true, sort-by AGE, and then just pipe to head -n1|awk '{print $1}', but I can't seem to get sort-by working. I'm running kubectl 1.7.9.
The AGE times are in an irregular format (23m, 2d) that’s hard to sort on, but you can ask kubectl to write out the time a pod started instead. The times will come out in the very sortable ISO 8601 format. This recipe to get the single oldest pod might work for you:
kubectl get pods \
--no-headers \
--output=custom-columns=START:.status.startTime,NAME:.metadata.name \
| sort \
| head -1 \
| awk '{print $2}'
The kubectl command asks to only print out the start time and name, in that order, for each pod.
Also consider kubectl get pods -o json, which will give you a very large very detailed JSON record. If you have a preferred full-featured scripting language you can pick that apart there, or use a command-line tool like jq to try digesting it further. Any field path can also be inserted into the custom-columns output spec.
This can be accomplished with:
kubectl get pods --sort-by=.metadata.creationTimestamp -o=name | head -1
I'm not sure what version this started work in, but I'm using it in kubectl 1.12.