We have a argo-rollout for one of the service. I used the cmd to update the image.
kubectl-argo-rollouts -n ddash5 set image detector detector=starry-academy-177207/detector:deepak-detector-8
I was expecting this to update the pod, but it created a new one.
NAME READY STATUS RESTARTS AGE
detector-5d96bc8456-h2x7p 1/1 Running 0 35m
detector-68f89d8b45-j465j 0/1 Running 0 35m
Even if I delete detector-5d96bc8456-h2x7p, pod gets recreated with the older image.
and detector-68f89d8b45-j465j stays in 0/1 state.
I am new to kube, Can someone give me insights to this?
Thanks!!!
Deepak
You are using argo rollout, where rolling updates allow deployment updates pods instances with new ones. The new Pods will be scheduled on Nodes with available resources.That is the reason new pods are getting created by replacing existing pods.
Instead you can use kubectl set image command which is used to update images of existing deployment, it will update images without recreating the deployment.Use the following command.
kubectl set image deployment/<deployment-name> <container-name>=<image>:<tag>
In your case:
kubectl set image deployment/detector detector=starry-academy-177207/detector:deepak-detector-8
This will update existing deployment, try it and let me know if this works.Found ArgoCD Image updater you can check it.
Related
In my kubernetes cluster, there are multiple deployments in a namespace.
For a specific deployment, there is a need to not allow "CrashLoopBackoff" pods to exist.
So basically, when any pod gets to this state, I would want it to be deleted and later a new pod to be created which is already handled by the ReplicaSet.
I tried with custom controllers, with the thought that the SharedInformer would alert about the state of Pod and then I would delete it from that loop.
However, this brings dependency on the pod on which the custom controller would run.
I also tried searching for any option to be configured in the manifest itself, but could not find any.
I am pretty new to Kuberenetes, so need help in the implementation of this behaviour.
Firstly, you should address the reason why the pod has entered the CrashLoopBackOff state rather than just delete it. If you do this, you'll potentially just recreate the problem again and you'll be deleting pods repeatedly. For example, if your pod is trying to access an external DB and that DB is down, it'll CrashLoop, and deleting and restarting the pod won't help fix that.
Secondly, if you want to do this deleting in an automated manner, an easy way would be to run a CronJob resource that goes through your deployment and deletes the CrashLooped pods. You could set the cronjob to run once an hour or whatever schedule you wish.
Deleting the POD and waiting for the New one is like restarting the deployment or POD.
Kubernetes will auto restart your CrashLoopBackoff POD if failing, you can check the Restart count.
NAME READY STATUS RESTARTS AGE
te-pod-1 0/1 CrashLoopBackOff 2 1m44s
This restarts will be similar to what you have mentioned
when any pod gets to this state, I would want it to be deleted and
later a new pod to be created which is already handled by the
ReplicaSet.
If you want to remove Crashing the POD fully and not look for new POD to come up, you have to rollback the deployment.
If there is any issue with your Replicaset and your POD is crashing it would be useless, any number of times you delete and restart the POD it will crash all time, unless you check logs & debug to solve the real issue in replicaset(Deployment).
I created a pod that I want to delete but it does not seem te work.
I did all ready tried a lot of different things.
I do not have deployments or replica sets, as a lot of people suggest to delete those.
When i type:
kubectl get all
Only the pod and the service is visible, both of which will return upon deleting them.
When i type:
kubectl delete pods <pod-name> --grace-period=0 --force
The pod will come back again.
Is there something else I can do?
If pod not part of the deployment, rs, sts, ds, then it must be part of the static pod. Static pod name combines with node (i.e staticpod-node01). By default location of the static path is /etc/Kubernetes/manifest/.
Doc: https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/
Apparently, it was some sort of invisible job that started the pod.
It was not showing when entering kubectl get jobs
So when finding this job and deleting it the pod disappeared!
I have a requirement I want to know every part of the time spent of a Pod.
how much time to pull a docker image? Maybe a Pod has multiple initContainers and containers. I want to know every part of them.
Maybe I can analysis the Events using
'kubectl describe pod-name...'
how much time a Pod get ready? From being created and get readiness ready.
For a bare Pod, I can know the startTime of the Pod and which time it is finished. Then I can calculate the duration.
But for pods that created by Deployment,StatefulSet,DaemonSet, I cannot find any time flag that indicating the first time that the Pod becomes readiness ready.
I want to know how much time spent to get the Pod ready. Not the age of the Pod.
The easiest method would be to subscribe to api-server to notify you if some changes occur in your cluster.
For example, I issued:
$ kubectl get pods --output-watch-events --watch
and then created a new pod. Here is the output:
EVENT NAME READY STATUS RESTARTS AGE
ADDED example-pod 0/1 Pending 0 0s
MODIFIED example-pod 0/1 ContainerCreating 0 0s
MODIFIED example-pod 0/1 Running 0 19s
MODIFIED example-pod 1/1 Running 0 23s
and here is a little explanation:
As you can see first event is ADDED and it is in Pending state which means that pod object just got created.
Second event is MODIFIED with ContainerCreating status, age 0 which means it took less than 1s time to assing/schedule the pod to a node. Now kubelet starts downloading continer image.
Third event has Running status meaning continer is started running. Looking at age column you can see it took 19s since previous event so it took around 19s to download the image and start the container. If you take a look at READY column you can see 0/1 value, so container is running but it is not yet in ready state.
Fourth event has READY column set to 1/1 so readiness probe has passed successfully. If you now look at the age column, you can see it took around 4s (32-19) to check readiness probe and change pod status.
If this information in not enough you can use --output= parameter to receive full pod specification on every change.
You can also play with kubectl get events to receive some more events. And of course, by adding --watch flag you can watch events in real time.
If you want higher level of felxibility, use dedicated kubernetes clinet libraries instead of kuebctl to receive this information and to process it.
I am working on two servers, each with a number of pods. The first server is Validation Env and I can use kubernetes commands, but the second server is on Prod Env I do not have full rights. It is out of the question to get full rights on the last one.
So, I am doing a platform stability statistic and I need info about the last reset of pods. I can see the "Age" but I cannot use a screenshot in my statistic, so I need a command that outputs every pods age or the last reset.
P.S. Every night at 00:00 the pods are saved and archived in a separate folder.
Get pods already gives you that info:
$ kubectl get po
NAME READY STATUS RESTARTS AGE
nginx-7cdbd8cdc9-8pnzq 1/1 Running 0 36s
$ kubectl delete po nginx-7cdbd8cdc9-8pnzq
pod "nginx-7cdbd8cdc9-8pnzq" deleted
$ kubectl get po
NAME READY STATUS RESTARTS AGE
nginx-7cdbd8cdc9-67l8l 1/1 Running 0 4s
I found a solution:
command:
zgrep "All subsystems started successfully" 201911??/*ota*
response:
23:23:37,429 [INFO ] main c.o.c.a.StartUp - All subsystems started successfully
P.S. "ota" is my pod's name.
A new GKE cluster created at v1.1.1 and using latest kubectl (from gcloud components update) when deleting resources (say a pod) sometimes kubectl get pods in a 'Terminating' state and other times they delete (are removed from kubectl get pods output) right away.
NAME READY STATUS RESTARTS AGE
cassandra 1/1 Terminating 0 44s
Is this new behavior of kubectl? I don't recall it doing this at my prior levels.
Yes, it is new behavior in v1.1.1. PR #9165 added graceful deletion of pods, which causes them to appear in the "Terminating" state for a short amount of time. Issue #1535 has some more background discussion.
You can explicitly set TerminationGracePeriodSeconds to zero in the PodSpec to obtain the old behavior.