Old ReplicaSet not getting replaced by new ReplicaSet after an kubectl edit - kubernetes

Question

I am creating a deployment using this yaml file. It creates a replica of 4 busybox pods. All fine till here.
But when I edit this deployment using the command kubectl edit deployment my-dep2, only changing the version of busybox image to 1.31 (a downgrade but still an update from K8s point of view), the ReplicaSet is not completely replaced.
The output of kubectl get all --selector app=my-dep2 post the edit is:
NAME READY STATUS RESTARTS AGE
pod/my-dep2-55f67b974-5k7t9 0/1 ErrImagePull 2 5m26s
pod/my-dep2-55f67b974-wjwfv 0/1 CrashLoopBackOff 2 5m26s
pod/my-dep2-dcf7978b7-22khz 0/1 CrashLoopBackOff 6 12m
pod/my-dep2-dcf7978b7-2q5lw 0/1 CrashLoopBackOff 6 12m
pod/my-dep2-dcf7978b7-8mmvb 0/1 CrashLoopBackOff 6 12m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/my-dep2 0/4 2 0 12m
NAME DESIRED CURRENT READY AGE
replicaset.apps/my-dep2-55f67b974 2 2 0 5m27s
replicaset.apps/my-dep2-dcf7978b7 3 3 0 12m
As you can see from the output above, there are 2 ReplicaSet which are existing in parallel. I expect the old ReplicaSet to be completely replaced by new ReplicaSet (containing the 1.31 version of busybox). But this is not happening. What am I missing here?

Answer 1

You are ignoring the errors ErrImagePull and CrashLoopBackOff. Those are telling you it is not being possible to run new containers (the image was not found in the docker registry), so old ones are kept to ensure the service runs (blue-green default/rolling update).
Edit
Also, your Busybox containers start and run nothing (as far as I can remember) and then finish, which causes Kubernetes to restart it and never arrive to an alive state. Maybe you'd better run some sleep 300 to it's entrypoint?

Answer 2

This is totally normal, expected result, related with Rolling Update mechanism in kubernetes
Take a quick look at the following working example, in which I used sample nginx Deployment. Once it's deployed, I run:
kubectl edit deployments.apps nginx-deployment
and removed the image tag which actually equals to performing an update to nginx:latest. Immediatelly after applying the change you can see the following:
$ kubectl get all --selector=app=nginx
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-574b87c764-bvmln 0/1 Terminating 0 2m6s
pod/nginx-deployment-574b87c764-zfzmh 1/1 Running 0 2m6s
pod/nginx-deployment-574b87c764-zskkk 1/1 Running 0 2m7s
pod/nginx-deployment-6fcf476c4-88fdm 0/1 ContainerCreating 0 1s
pod/nginx-deployment-6fcf476c4-btvgv 1/1 Running 0 3s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx-deployment ClusterIP 10.3.247.159 <none> 80/TCP 6d4h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 3/3 2 3 2m7s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-574b87c764 2 2 2 2m7s
replicaset.apps/nginx-deployment-6fcf476c4 2 2 1 3s
As you can see, at certain point in time there are running pods in both replicas. It's because of the mentioned rolling update mechanism, which ensures your app availability when it is being updated.
When the update process is ended replicas count in the old replicaset is reduced to 0 so there are no running pods, managed by this replicaset as the new one achieved its desired state:
$ kubectl get all --selector=app=nginx
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-6fcf476c4-88fdm 1/1 Running 0 10s
pod/nginx-deployment-6fcf476c4-btvgv 1/1 Running 0 12s
pod/nginx-deployment-6fcf476c4-db5z7 1/1 Running 0 8s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx-deployment ClusterIP 10.3.247.159 <none> 80/TCP 6d4h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 3/3 3 3 2m16s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-574b87c764 0 0 0 2m16s
replicaset.apps/nginx-deployment-6fcf476c4 3 3 3 12s
You may ask yourself: why is it still there ? why it is not deleted immediatelly after the new one becomes ready. Try the following:
$ kubectl rollout history deployment nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
1 <none>
2 <none>
As you can see, there are 2 revisions of our rollout for this deployment. So now we may want to simply undo this recent change:
$ kubectl rollout undo deployment nginx-deployment
deployment.apps/nginx-deployment rolled back
Now, when wee look at our replicas we can observe a reverse process:
$ kubectl get all --selector=app=nginx
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-574b87c764-6j7l5 0/1 ContainerCreating 0 1s
pod/nginx-deployment-574b87c764-m7956 1/1 Running 0 4s
pod/nginx-deployment-574b87c764-v2r75 1/1 Running 0 3s
pod/nginx-deployment-6fcf476c4-88fdm 0/1 Terminating 0 3m25s
pod/nginx-deployment-6fcf476c4-btvgv 1/1 Running 0 3m27s
pod/nginx-deployment-6fcf476c4-db5z7 0/1 Terminating 0 3m23s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx-deployment ClusterIP 10.3.247.159 <none> 80/TCP 6d4h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 3/3 3 3 5m31s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-574b87c764 3 3 2 5m31s
replicaset.apps/nginx-deployment-6fcf476c4 1 1 1 3m27s
Note, that there is no need to create a 3rd replicaset, as there is still the old one which can be used to undo our recent change. The final result looks as follows:
$ kubectl get all --selector=app=nginx
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-574b87c764-6j7l5 1/1 Running 0 40s
pod/nginx-deployment-574b87c764-m7956 1/1 Running 0 43s
pod/nginx-deployment-574b87c764-v2r75 1/1 Running 0 42s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx-deployment ClusterIP 10.3.247.159 <none> 80/TCP 6d4h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 3/3 3 3 6m10s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-574b87c764 3 3 3 6m10s
replicaset.apps/nginx-deployment-6fcf476c4 0 0 0 4m6s
I hope that the above example helped you to realize why this old replicaset isn't immediatelly removed and what it can be still useful for.

Answer 3

as #emi said busybox and alpine etc don't do anything in case you give an explicit command. Kubernetes try to keep running but the default container does not perform any action and at the end, Kubernetes say okay, something is wrong no need to try to restart the container again and again. For test purposes, it might look as below.
kind: Pod
apiVersion: v1
metadata:
name: my-test-pod
spec:
containers:
- image: nginx
name: enginx
- image: alpine
name: alpine
command: ["sleep", "3600"]