Is it possible to delete POD in kubernetes based on creation time or age?
Example : I would like to delete all PODs which are older than 1 day. These PODs are orphaned , therefore no new PODs will be created.
This command will delete all PODs older than one day :
kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}} {{.metadata.creationTimestamp}}{{"\n"}}{{end}}' | awk '$2 <= "'$(date -d 'yesterday' -Ins --utc | sed 's/+0000/Z/')'" { print $1 }' | xargs --no-run-if-empty kubectl delete pod
This command will delete all PODs older than 4 hours :
kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}} {{.metadata.creationTimestamp}}{{"\n"}}{{end}}' | awk '$2 <= "'$(date -d'now-4 hours' -Ins --utc | sed 's/+0000/Z/')'" { print $1 }' | xargs --no-run-if-empty kubectl delete pod
We could do this with awk by doing a regex [0-9]+d directly on the AGE ($5, 5th column) column and then printing the corresponding NAME ($1, first column) column
kubectl delete pod $(kubectl get pod | awk 'match($5,/[0-9]+d/) {print $1}')
Test first to see what's matching:
kubectl get pod | awk 'match($5,/[0-9]+d/) {print $0}'
$0 means all columns
You can either add a liveness probe to track how long the pod alive and kill it when it's longer a certain period. Or you can schedule a CronJob
modified to work on mac:
# try the gnu versions: gxargs
brew install findutils coreutils
kubectl get pods -o go-template -n gui2 --template '{{range .items}}{{.metadata.name}} {{.metadata.creationTimestamp}}{{"\n"}}{{end}}' | awk '$2 <= "'$(gdate -d '21 days ago' -Ins --utc | sed 's/+0000/Z/')'" { print $1 }' | gxargs --no-run-if-empty kubectl delete pod
Here is an alternative answer that uses kubectl, jq, and xargs.
It works locally for me on Mac, but I also uses it in a Kubernetes cronjob that runs debian-slim, so I think it works on both Mac and Linux.
The time is set in seconds (86400s here for 1 day).
kubectl get pod -o json | jq -r --argjson timestamp 86400 '.items[] | select (.metadata.creationTimestamp | sub("\\..*";"Z") | sub("\\s";"T") | fromdate < now - $timestamp).metadata.name' | xargs -r -L1 kubectl delete pod ;
Here's a variant of the answer by dansl1982 above that will delete all pods with a particular label in all namespaces.
kubectl get pods -A -l app=my-api -o go-template --template '{{range .items}}{{.metadata.namespace}} {{.metadata.name}} {{.metadata.creationTimestamp}}{{"\n"}}{{end}}' | awk '$3 <= "'$(date -d'now-4 hours' -Ins --utc | sed 's/+0000/Z/')'" { print $1 " " $2 }' | xargs -L 1 --no-run-if-empty kubectl delete pod -n $1 $2
Related
BAD_PODS=`kubectl get pods --context ${EKS_CLUSTER_NAME} | grep "0/1" | awk {'print $1'}`
if [ -z "$BAD_PODS" ]; then
log "No Pod in Not Ready state"
else
for pod in $BAD_PODS
do
duration=`kubectl get pod --context ${EKS_CLUSTER_NAME} $pod | grep "0/1" | awk -F' ' {'print $5'}`
if [ `echo $duration | egrep "h|d"` ]; then
log "Pod not running since more than an hour. Deleting it."
kubectl delete pod --context ${EKS_CLUSTER_NAME} $pod
elif [ `echo $duration | awk -F'm' {'print $1'}` -gt 20 ]; then
log "Pod not running since more than 20 minutes. Deleting it."
kubectl delete pod --context ${EKS_CLUSTER_NAME} $pod
fi
done
fi
this however does not seem to work when I deploy to my cluster. I keep getting a crash loop back off error. Would like some feedback in this
Try command like
kubectl --context ${EKS_CLUSTER_NAME} delete pod $pod
whole script
BAD_PODS=`kubectl get pods --context ${EKS_CLUSTER_NAME} | grep "0/1" | awk {'print $1'}`
if [ -z "$BAD_PODS" ]; then
log "No Pod in Not Ready state"
else
for pod in $BAD_PODS
do
duration=`kubectl get pod --context ${EKS_CLUSTER_NAME} $pod | grep "0/1" | awk -F' ' {'print $5'}`
if [ `echo $duration | egrep "h|d"` ]; then
log "Pod not running since more than an hour. Deleting it."
kubectl --context ${EKS_CLUSTER_NAME} delete pod $pod
elif [ `echo $duration | awk -F'm' {'print $1'}` -gt 20 ]; then
log "Pod not running since more than 20 minutes. Deleting it."
kubectl --context ${EKS_CLUSTER_NAME} delete pod $pod
fi
done
fi
I have a problem with minikube
when I want to create deployment manifest files, I receive an error
when I write this code:
minikube kubectl create -f .
I got this Error:
Error: unknown shorthand flag: 'f' in -f
See 'minikube kubectl --help' for usage.
but not only this, I try to write another command but again same error happening
minikube kubectl delete daemonsets,replicasets,services,deployments,pods,rc,pvc --all
Error: unknown flag: --all
See 'minikube kubectl --help' for usage.
please help me.
thanks
AFAIK, --all is not a valid flag. Valid flag is --all-namespaces or just -A.
However, "kubectl delete" does not take -A as it needs the resource name for deletion.
To accomplish what you are trying to do you will have to write a loop to delete the objects 1 by one using
kubectl get daemonsets,replicasets,services,deployments,pods,rc,pvc -A --no-headers | while read line; do
namespace=$(echo $line | awk '{print $1}')
resource=$(echo $line | awk '{print $2}')
kubectl delete ${resource} -n ${namespace}
done
Execution - BE extremely careful with this as it will delete all queried resources in all namespaces including those in kube-system namespace:
controlplane $ kubectl get daemonsets,replicasets,services,deployments,pods,rc,pvc -A --no-headers | while read line; do
> namespace=$(echo $line | awk '{print $1}')
> resource=$(echo $line | awk '{print $2}')
> kubectl delete ${resource} -n ${namespace}
> done
daemonset.extensions "kube-keepalived-vip" deleted
daemonset.extensions "kube-proxy" deleted
daemonset.extensions "weave-net" deleted
replicaset.extensions "coredns-fb8b8dccf" deleted
replicaset.extensions "katacoda-cloud-provider-d5cb9d656" deleted
service "kubernetes" deleted
service "kube-dns" deleted
deployment.extensions "coredns" deleted
deployment.extensions "katacoda-cloud-provider" deleted
How to list the pods running in a particular zone?
And suppose my cluster is configured into multiple zones, how to ensure pods are distributed into every zone?
Topology key zone value is a label applied on nodes and so you can get the nodes with 'zone-value' and list pods of those nodes.
Somethig like below,
kubectl get nodes -l zone:<zone value> -o json | jq '.items[].metadata.name' | xargs -I worker sh -c 'kubectl get pods -o wide | grep worker'
kubectl get nodes --show-labels|awk 'NR == 1 {next} {print $1,$6}'|awk -F "/" '{print $1,$NF}'|awk '{print $1,$NF}'|awk -F "=" '{print $1,$NF}'|awk '{print $1,"\t",$NF;}'
I'm trying to delete Kubernetes evicted pods which are older than 30 days from all namespaces, but the command throws an error :
Error from server(NotFound): pods (xxxxxxxxxxxxx) not found
Command:
kubectl delete pod $(kubectl get pods --all-namespaces | grep Evicted | sed 's#d$##' | awk '$6 > 30 {print $2}')"
Any ideas?
i think it's due to when you are listing down the pod you are listing all pod and deleting all those
but command
kubectl delete pod not getting the namespace from which to delete the specific pod
kubectl delete pod $(kubectl get pods --all-namespaces | grep Evicted | awk '$6 > 30 {print $2}') -n $(kubectl get pods --all-namespaces | grep Evicted | awk '$6 > 30 {print $1}')
maybe you try something like or simplify it better way.
This assumes the GNU coreutils version of date and a bourne-like shell:
One-liner:
onemonthago=$(date -u -d '-1 month' +"%FT%TZ"); kubectl get pod --field-selector='status.phase=Failed' -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATUS:.status.reason,STARTTIME:.status.startTime -A | while read ns pod status starttime; do if [ "$status" = "Evicted" ]; then if [ "$starttime" \> "$onemonthago" ]; then kubectl -n $ns delete pod $pod --wait=false; fi; fi; done
More readable formatting:
onemonthago=$(date -u -d '-1 month' +"%FT%TZ");
kubectl get pod --field-selector='status.phase=Failed' -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATUS:.status.reason,STARTTIME:.status.startTime -A | while read ns pod status starttime; do
if [ "$status" = "Evicted" ]; then
if [ "$starttime" \> "$onemonthago" ]; then
kubectl -n $ns delete pod $pod --wait=false;
fi;
fi;
done
The details:
onemonthago=$(date -u -d '-1 month' +"%FT%TZ"); gets the cutoff time in the same format as what kubectl returns and stores it in a shell variable.
The kubectl then gets all the failed pods and outputs the columns that we need for further steps.
It then gets piped to a while loop that reads the fields into shell variables.
The if statements then check whether the pod was evicted or is failed for another reason (The status.reason field is not supported for --field-selector) and that it is older than a month. If it matches that criteria, it deletes it.
(It is likely possible to optimise it to gather a list of all pods in a namespace and pass that to the minimum number of kubectls using xargs in one go) (Both checks can be combined, either in the shell or in AWK as well) (Replacing the entire while with an elaborate AWK script would likely yield the best perfomance (collecting a list of pos to delete per namespace and running / outputting the delete command from the END pattern)
I'm looking for a command like "gcloud config get-value project" that retrieves a project's name, but for a pod (it can retrieve any pod name that is running). I know you can get multiple pods with "kubectl get pods", but I would just like one pod name as the result.
I'm having to do this all the time:
kubectl get pods # add one of the pod names in next line
kubectl logs -f some-pod-frontend-3931629792-g589c some-app
I'm thinking along the lines of "gcloud config get-value pod". Is there a command to do that correctly?
There are many ways, here are some examples of solutions:
kubectl get pods -o name --no-headers=true
kubectl get pods -o=name --all-namespaces | grep kube-proxy
kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'
For additional reading, please take a look to these links:
kubernetes list all running pods name
Kubernetes list all container id
https://kubernetes.io/docs/tasks/access-application-cluster/list-all-running-container-images/
You can use the grep command to filter any output on stdout. So to get pods matching a specified pattern you can use a command like this:
> kubectl get pods --all-namespaces|grep grafana
Output:
monitoring kube-prometheus-grafana-57d5b4d79f-smkz6 2/2 Running 0 1h
To only output the pod name, you can use the awk command with a parameter of '{print $2}', which displays the second column of the previous output:
kubectl get pods --all-namespaces|grep grafana|awk '{print $2}'
To only display one line you can use the head command like so:
kubectl get pods --all-namespaces|grep grafana|awk '{print $2}'|head -n 1
This will output the last pod name :
kubectl get pods -o go-template --template ' {{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | awk '{print $1}' | tail -n 1
if you want to fetch logs for a pod:
kubectl logs -f kubectl get pods -o go-template --template ' {{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | awk '{print $1}' | tail -n 1
just run the command with a single quotes.
Links:
kubernetes list all running pods name
https://kubernetes.io/docs/tasks/access-application-cluster/list-all-running-container-images/
Hope this helps!!