In my namespace, I have several pods named with the same prefix, followed by the random string. There are also other pods, named differently. The result of kubectl get pods would look something like this:
service-job-12345abc
service-job-abc54321
other-job-54321cba
I need to find the nameof the most recently created pod starting with "service-job-".
I found this thread, which helps getting the name of the most recent pod in general. This one gets me the complete names of pods starting with a specific prefix.
What I struggle with is combining these two methods. With each one, I seem to lose the information I need to perform the other one.
Note: I am not an administrator of the cluster, so I cannot change anything about the naming etc. of the pods. The pods could also be in any possible state.
This works as you expect:
kubectl get pods --sort-by=.metadata.creationTimestamp --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep service-job- | head -1
Related
Is there a simple method (that won't require googling at every use) to get names of all deployment configs with no running pods (scaled to 0) in Kubernetes / Openshift? Methods without JSON tokens and awk please.
The docs of oc get dc --help are way too long to decipher for the occasional need.
The only CLI arg for advanced filtering without working with JSON is a --field-selector, but it has a limited scope which not include spec.replicas field.
So, there will be some magic around JSON with other flag - jsonpath.
Here is a command to filter and print names of all deployments which are scaled to 0:
kubectl get deployments --all-namespaces -o=jsonpath='{range .items[?(#.spec.replicas==0)]}{.metadata.name}{"\n"}{end}'
Jsonpath reference is here.
Why do pod names have 5 random alphanumeric characters appended to their name when created through a kubernetes deployment? Is it possible to get rid of them so that the pods names don't change? I am frequently deleting and creating deployments and would prefer that pod names don't change.
Update: I would like to have the same name because I am constantly deleting/recreating the same deployment and if the name doesn't change, then I can quickly reuse old commands to exec into/see the logs of the containers.
Reason for having random alphanumeric in pod names:
When we create a deployment, it will not directly create pods(to match the replica count).
It will create a replicaset (with name = deployname_name + 10 digit aplhanumeric). But why extra alphanumeric ? When we do upgrade of deployment, new replicaset is create with new alphanumeric and old is kept as it is. This old replicaset is used for rollbacks.
The created replicaset will create pods (with name = replicaset_name + 5 digit alphanumeric). But why extra alphanumeric? We cannot have two pods with same name.
If your usecase is to use the old commands frequently, then going for Statefulset is not the good solution. Statefulsets are heavy weight(ordered deployment, ordered termination, unique network names) and they are specially designed to preserve state across restart (in combination with persistent volume).
There are few tools which you can use:
stern
kube-fzf
Lightweight solution to your problem:
You can use labels to get the same pod across deployments:
kubectl get pods -l app=my_app,app_type=server
NAME READY STATUS RESTARTS AGE
my-app-5b7644f7f6-4hb8s 1/1 Running 0 22h
my-app-5b7644f7f6-72ssz 1/1 Running 0 22h
after this we can use some bash magic get what we want like below
Final command:
kubectl get pods -l app=my_app,app_type=server -o name | rg "pod/" -r "" | head -n 1 | awk '{print "kubectl logs " $0}' | bash
Explanation:
get list of pod names
kubectl get pods -l app=my_app,app_type=server -o namenames
pod/my-app-5b7644f7f6-4hb8s
pod/my-app-5b7644f7f6-72ssz
replace pod/ using ripgrep or sed (rg "pod/" -r "")
take only one pod using head -n 1
use awk to print exec/see_log command
pipe it to bash to execute
This is how deployments works, every time pod dies, ReplcaSet create pod with different name to match desired state and random number attached to pod name to give them unique names.
Whatever you are trying to achieve is not possible with deployment object as they are intended for stateless applications. As you want to preserve state( name) of application this is certainly possible with StatefulSet.
So if you use StatefulSet object to manage replicas, every pod will be created with certain name convention, e. g. POD_NAME-1, POD_NAME-2 etc i. e. Index will be appeneded to pod name. Also when pod dies, new pod will created with same name.
Ao you want to achieve is ideal use case of StatefulSet. Go for it.
If you deploy a pod from deployment object, kind:Deployment, then the deployment controller appends a unique name to pod that is part of specific deployment.
This is how the deployment controller looks up all the relevant pods of respective deployment. This is needed for rolling upgrade, rollback functions
Here is a transcript:
LANELSON$ kubectl --kubeconfig foo get -a jobs
No resources found.
OK, fine; even with the -a option, no jobs exist. Cool! Oh, let's just be paranoid and check for one that we know was created. Who knows? Maybe we'll learn something:
LANELSON$ kubectl --kubeconfig foo get -a job emcc-poc-emcc-broker-mp-populator
NAME DESIRED SUCCESSFUL AGE
emcc-poc-emcc-broker-mp-populator 1 0 36m
Er, um, what?
In this second case, I just happen to know the name of a job that was created, so I ask for it directly. I would have thought that kubectl get -a jobs would have returned it in its output. Why doesn't it?
Of course what I'd really like to do is get the logs of one of the pods that the job created, but kubectl get -a pods doesn't show any of that job's terminated pods either, and of course I don't know the name of any of the pods that the job would have spawned.
What is going on here?
Kubernetes 1.7.4 if it matters.
The answer is that Istio automatic sidecar injection happened to be "on" in the environment (I had no idea, nor should I have). When this happens, you can opt out of it, but otherwise all workloads are affected by default (!). If you don't opt out of it, and Istio's presence causes your Job not to be created for any reason, then your Job is technically uninitialized. If a resource is uninitialized, then it does not show up in kubectl get lists. To make an uninitialized resource show up in kubectl get lists, you need to include the --include-uninitialized option to get. So once I issued kubectl --kubeconfig foo get -a --include-uninitialized jobs, I could see the failed jobs.
My higher-level takeaway is that the initializer portion of Kubernetes, currently in alpha, is not at all ready for prime time yet.
The problem
It seems that deleting a pod by UID used to be possible, but has been removed (https://github.com/kubernetes/kubernetes/issues/40121).
It also seems one cannot get a pod by its resource either (https://github.com/kubernetes/kubernetes/issues/20572).
(Why did they remove deleting by UID, what is the use of the UID then?)
Why do I want to use UID in the first place, and not, say, name?
Because I need something like replicas and none of the controllers (like Job, Deployment etc) suit my case.
The good thing about UIDs is that kubernetes generates them -- I don't have to worry about differentiating the pods then, I just save the returned UID. So it would be great if I could use it later to delete that particular pod ("replica").
The question
Is it really not possible to use the UID to get/delete?
Is my only option then to take care of differentiating the pods myself (e.g. keep a counter or generate a unique id myself and set it as the name or label)?
This works as a get by UID:
Use kubectl custom-columns
List all PodName along with its UID of a namespace:
$ kubectl get pods -n <namespace> -o custom-columns=PodName:.metadata.name,PodUID:.metadata.uid
source: How do I get the pod ID in Kubernetes?
Using awk and xargs and you can delete with custom-columns too.
I assume you are writing a controller to manage a set of replicas using a TPR or CRD
The short answer is that you should not track pods by UID or name but by using a selector much like Deployments/ReplicaSets/Services do. If pods match that selector, you can check the Pod's metadata or Pod spec to see if it needs to be
deleted for some reason.
When a Kubernetes pod goes into CrashLoopBackOff state, you will fix the underlying issue. How do you force it to be rescheduled?
For apply new configuration the new pod should be created (the old one will be removed).
If your pod was created automatically by Deployment or DaemonSet resource, this action will run automaticaly each time after you update resource's yaml.
It is not going to happen if your resource have spec.updateStrategy.type=OnDelete.
If problem was connected with error inside docker image, that you solved, you should update pods manually, you can use rolling-update feature for this purpose, In case when new image have same tag, you can just remove broken pod. (see below)
In case of node failure, the pod will recreated on new node after few time, the old pod will be removed after full recovery of broken node. worth noting it is not going to happen if your pod was created by DaemonSet or StatefulSet.
Any way you can manual remove crashed pod:
kubectl delete pod <pod_name>
Or all pods with CrashLoopBackOff state:
kubectl delete pod `kubectl get pods | awk '$3 == "CrashLoopBackOff" {print $1}'`
If you have completely dead node you can add --grace-period=0 --force options for remove just information about this pod from kubernetes.
Generally a fix requires you to change something about the configuration of the pod (the docker image, an environment variable, a command line flag, etc), in which case you should remove the old pod and start a new pod. If your pod is running under a replication controller (which it should be), then you can do a rolling update to the new version.
5 Years later, unfortunately, this scenario seems to still be the case.
#kvaps answer above suggested an alternative (rolling updates), that essentially updates(overwrites) instead of deleting a pod -- the current working link of rolling updates
The alternative to being able to delete a pod, was NOT to create a pod but instead create a deployment, and delete the deployment that contains the pod, subject to deletion.
$ kubectl get deployments -A
$ kubectl delete -n <NAMESPACE> deployment <DEPLOYMENT>
# When on minikube or using docker for development + testing
$ docker system prune -a
The first command displays all deployments, alongside their respective namespaces. This helped me reduce the error of deleting deployments that share the same name(name collision) but from two different namespaces.
The second command deletes a deployment that is exactly located underneath a namespace.
The last command helps when working in development mode. Essentially, removing all unused images, which is not required but helps clean up and save some disk-space.
Another great tip, is to try to understand the reasons why a Pod is failing. The problem may be relying completely somewhere else, and k8s does a good deal of documenting. For that one of the following may help:
$ kubectl logs -f <POD NAME>
$ kubectl get events
Other reference here on StackOveflow:
https://stackoverflow.com/a/55647634/132610
For anyone interested I wrote a simple helm chart and python script which watches the current namespace and deletes any pod that enters CrashLoopBackOff.
The chart is at https://github.com/timothyclarke/helm-charts/tree/master/charts/dr-abc.
This is a sticking plaster. Fixing the problem is always the best option. In my specific case getting the historic apps into K8s so the development teams have a common place to work and strangle the old applications with new ones is preferable to fixing all the bugs in the old apps. Having this in the namespace to keep the illusion of everything running buys that time.
This command will delete all pods that are in any of (CrashLoopBackOff, Init:CrashLoopBackOff, etc.) states. You can use grep -i <keyword> to match different states and then delete the pods that match the state. In your case it should be:
kubectl get pod -n <namespace> --no-headers | grep -i crash | awk '{print $1}' | while read line; do; kubectl delete pod -n <namespace> $line; done