newbie here . I am trying to automate the way to get the nodes that are ready using the following script:
until kubectl get nodes | grep -m 2 "Ready"; do sleep 1 ; done
Is there a better way to do this, specifically i am looking for a way to do this without having to specify the node number?
To get the names of all Ready nodes, use
$ kubectl get nodes -o json | jq -r '.items[] | select(.status.conditions[].type=="Ready") | .metadata.name '
master-0
node-1
node-3
node-x
Basing on the official documentation
You can check your ready nodes by using this command:
JSONPATH='{range .items[*]}{#.metadata.name}:{range #.status.conditions[*]}{#.type}={#.status};{end}{end}' \
&& kubectl get nodes -o jsonpath="$JSONPATH" | grep "Ready=True"
You don't need to specify number of nodes in your command, just use it like this:
until kubectl get nodes | grep -i "Ready"; do sleep 1 ; done
Related
I am running a deployment called mydeployment that manages several pods/replicas for a certain service. I want to search all the service pods/instances/replicas of that deployment for a certain keyword. The command below defaults to one replica and returns the keyword matching in this replica only.
Kubectl logs -f deploy/mydeployment | grep "keyword"
Is it possible to customize the above command to return all the matching keywords out of all instances/pods of the deployment mydeployment? Any hint?
Save this to a file fetchLogs.sh file, and if you are using Linux box, use sh fetchLogs.sh
#!/bin/sh
podName="key-word-from-pod-name"
keyWord="actual-log-search-keyword"
nameSpace="name-space-where-the-pods-or-running"
echo "Running Script.."
for podName in `kubectl get pods -A -o name | grep -i ${podName} | cut -d'/' -f2`;
do
echo "searching pod ${podName}"
kubectl -n ${nameSpace} logs pod/${podName} | grep -i ${keyWord}
done
I used the pods, if you want to use deployment, the idea is same change the kubectl command accordingly.
Using this command, I am able to get the container name that have restarted.
kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[?(#.restartCount>0)].name}'
Is there a way to get the pod name as well in the same command?
It's much easier to get json with kubectl and then process it with jq :
#!/usr/bin/env bash
kubectl get pods -o=json |
jq -r '.items[] |
"\(.metadata.name) \(.status.containerStatuses[]|select(.restartCount>0).name)"'
I've not tried this.
If (!) it works, it's not quite what you want as it should give you every Pod name and then a list of Container names that match the predicate.
I think you can't use kubectl --output=jsonpath alone to filter only the Pod names that have a Container with restarts.
FILTER='
{range .items[*]}
{.metadata.name}
{"\t"}
[
{.status.containerStatuses[?(#.restartCount>0)].name}
]
{"\n"}
{end}
'
kubectl get pods \
--output=jsonpath="${FILTER}"
Does anyone know how to get a sorted list of pods on a given node based on their disk space consumption?
The command below helps me in listing pods based on a given node but my requirement is to list those pods which are causing high disk space usage as part of investigating and solving the DiskPressure eviction state.
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<NODE_NAME>
I was able to find commands to list CPU and memory usage of nodes (ref: https://github.com/kubernetes/kubernetes/issues/17512#issuecomment-317757388) but nothing related to node disk usage.
alias util='kubectl get nodes --no-headers | awk '\''{print $1}'\'' | xargs -I {} sh -c '\''echo {} ; kubectl describe node {} | grep Allocated -A 5 | grep -ve Event -ve Allocated -ve percent -ve -- ; echo '\'''
# Get CPU request total (we x20 because because each m3.large has 2 vcpus (2000m) )
alias cpualloc='util | grep % | awk '\''{print $1}'\'' | awk '\''{ sum += $1 } END { if (NR > 0) { print sum/(NR*20), "%\n" } }'\'''
# Get mem request total (we multiply by 75 because because each m3.large has 7.5G ram )
alias memalloc='util | grep % | awk '\''{print $5}'\'' | awk '\''{ sum += $1 } END { if (NR > 0) { print sum/(NR*75), "%\n" } }'\'''
found an answer to my requirement - list pods disk usage running in a given node
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.pods[0] | "PodName: ", .podRef.name, "usedBytes: ", .containers[].rootfs.usedBytes'
Some other stats:
get node filesystem usage
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.fs.usedBytes'
get node imageFs usage
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.runtime.imageFs.usedBytes'
get node iNodes stats
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.fs.inodesFree'
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.runtime.imageFs.inodesFree'
Unfortunately you won't be able to achieve this with kubectl as it does not have this kind of functionality. One way to go is login to node and the use docker ps -s.
Another way that I can think of is to expose kubelet/cadvisor metrics and use it with metrics scraper like Prometheus.
The kubelet exposes all of it’s runtime metrics, and all of the cAdvisor metrics, on a /metrics endpoint in the Prometheus exposition format. Note that kubelet also exposes metrics in /metrics/cadvisor, /metrics/resource and /metrics/probes endpoints. Disk metrics are generally available on the /stats/summary endpoint.
Additional info:
Stack case with great example how to curl kubelet's endpoint.
Kubernetes metrics documents at /cluster-administration/system-metrics/
To simplify, I want to list all the docker images defined in a helm chart.
For eg, let's say I have the following set of templates:
$ helm template jenkins/jenkins
https://charts.jenkins.io/
Then, I want to somehow use kubectl to parse this result so I can apply a filter such as:
kubectl get pods -l k8s-app=kube-dns -o jsonpath={.items[*].spec.containers[*].name}
To return me the list. However, that command is to get pods. Any clue?
EDIT: I found a way:
❯ helm template jenkins/jenkins | kubectl apply -f - --dry-run=client -o jsonpath="{..image}" | tr -s '[[:space:]]' '\n' | sort | uniq
bats/bats:1.2.1
jenkins/jenkins:2.263.1
kiwigrid/k8s-sidecar:0.1.275
With one drawback: kubectl needs to be connected to a cluster. I would like to prevent that.
You can use a different jsonpath to get all images:
kubectl get pods -A -o jsonpath="{..image}"
If you just want unique images: kubectl get pods -A -o jsonpath="{..image}" | tr -s '[[:space:]]' '\n' | sort -u.
Substituting -A for the namespace of your chart or manifests.
If you have the manifests on your machine and not deployed, of course, you can just grep: grep 'image: ' *.yml
You can also use Go template syntax:
kubectl get pods -A -o go-template --template="{{range .items}}{{range .spec.containers}}{{.image}} {{end}}{{end}}"
If you have more than one chart in a given namespace, I think grepping would be the best way: helm template some-chart | grep 'image:'
EDIT:
Since this will be running in CI, it would be better to use a little bit of code to avoid potential false positives. This Python script does the trick:
#!/usr/bin/env python3
import sys
import yaml # pip install PyYAML
from nested_lookup import nested_lookup # pip install nested-lookup
template = ""
for line in sys.stdin:
template += line
parts = template.split('---')
for p in parts:
y = yaml.safe_load(p)
matches = nested_lookup("image", y)
if (len(matches)):
print("\n".join(matches))
Usage: helm template jenkins/jenkins | ./this-script.py. It prints out each occurrence of images, so if you only want unique images you'd need to throw all the matches in a list, then unique that before printing (or pipe it to sort -u).
I've this command to list the Kubernetes pods that are not running:
sudo kubectl get pods -n my-name-space | grep -v Running
Is there a command that will return a count of pods that are not running?
If you add ... | wc -l to the end of that command, it will print the number of lines that the grep command outputs. This will probably include the title line, but you can suppress that.
kubectl get pods -n my-name-space --no-headers \
| grep -v Running \
| wc -l
If you have a JSON-processing tool like jq available, you can get more reliable output (the grep invocation will get an incorrect answer if an Evicted pod happens to have the string Running in its name). You should be able to do something like (untested)
kubectl get pods -n my-namespace -o json \
| jq '.items | map(select(.status.phase != "Running")) | length'
If you'll be doing a lot of this, writing a non-shell program using the Kubernetes API will be more robust; you will generally be able to do an operation like "get pods" using an SDK call and get back a list of pod objects that you can filter.
You can do it without any external tool:
kubectl get po \
--field-selector=status.phase!=Running \
-o go-template='{{len .items}}'
the filtering is done with field-selectors
the counting is done with go-template: {{ len .items }}