Why the k8s rollback (rollout undo) is not working? - kubernetes

Question

After a successful
kubectl rollout restart deployment/foo
the
kubectl rollout undo deployment/foo
or
kubectl rollout undo deployment/foo --to-revision=x
are not having effect. I mean, the pods are replaced by new ones and a new revision is created which can be checked with
kubectl rollout history deployment foo
but when I call the service, the rollback had no effect.
I also tried to remove the imagePullPolicy: Always, guessing that it was always pulling even in the rollback, with no success because probably one thing is not related to the other.
Edited: The test is simple, I change the health check route of the http api to return something different in the json, and it doesn't.
Edited:
Maybe a typo, but not: I was executing with ... undo deployment/foo ..., and now tried with ... undo deployment foo .... It also gives me deployment.apps/foo rolled back, but no changes in the live system.
More tests: I changed again my api route to test what would happen if I executed a rollout undo to every previous revision one by one. I applied the last 10 revisions, and nothing.

Answer 1

To be able to rollback to a previous version don't forget to append the --record parameter to your kubectl command, for example:
kubectl apply -f DEPLOYMENT.yaml --record
Then you should be able to see the history as you know with:
kubectl rollout history deployment DEPLOYMENT_NAME
And your rollback will work properly
kubectl rollout undo deployment DEPLOYMENT_NAME --to-revision=CHOOSEN_REVISION_NUMBER
Little example:
consider my nginx deployment manifest "nginx-test.yaml" here:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
lets create it:
❯ kubectl apply -f nginx-test.yaml --record
Flag --record has been deprecated, --record will be removed in the future
deployment.apps/nginx-deployment created
lets check the image of this deployment, as expected from the manifest:
❯ k get pod nginx-deployment-74d589986c-k9whj -o yaml | grep image:
- image: nginx
image: docker.io/library/nginx:latest
now lets modify the image of this deployment to "nginx:1.21":
#"nginx=" correspond to the name of the container inside the pod create by the deployment.
❯ kubectl set image deploy nginx-deployment nginx=nginx:1.21.6
deployment.apps/nginx-deployment image updated
we can optionnaly check the rollout status:
❯ kubectl rollout status deployment nginx-deployment
deployment "nginx-deployment" successfully rolled out
we can check the rollout history with:
❯ kubectl rollout history deploy nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
1 kubectl apply --filename=nginx-test.yaml --record=true
2 kubectl apply --filename=nginx-test.yaml --record=true
lets check the image of this deployment, as expected:
❯ k get pod nginx-deployment-66dcfc79b5-4pk7w -o yaml | grep image:
- image: nginx:1.21.6
image: docker.io/library/nginx:1.21.6
Oh, no, i don't like this image ! Lets rollback:
❯ kubectl rollout undo deployment nginx-deployment --to-revision=1
deployment.apps/nginx-deployment rolled back
creating:
> kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-deployment-66dcfc79b5-4pk7w 1/1 Running 0 3m41s 10.244.3.4 so-cluster-1-worker3 <none> <none>
pod/nginx-deployment-74d589986c-m2htr 0/1 ContainerCreating 0 13s <none> so-cluster-1-worker2 <none> <none>
after few seconds:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-deployment-74d589986c-m2htr 1/1 Running 0 23s 10.244.4.10 so-cluster-1-worker2 <none> <none>
as you can see it worked:
❯ k get pod nginx-deployment-74d589986c-m2htr -o yaml | grep image:
- image: nginx
image: docker.io/library/nginx:latest
lets recheck the history:
❯ kubectl rollout history deploy nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
1 kubectl apply --filename=nginx-test.yaml --record=true
2 kubectl apply --filename=nginx-test.yaml --record=true
you can change the rollout history's CHANGE-CAUSE with the "kubernetes.io/change-cause" annotation:
❯ kubectl annotate deploy nginx-deployment kubernetes.io/change-cause="update image from 1.21.6 to latest" --reco
rd
Flag --record has been deprecated, --record will be removed in the future
deployment.apps/nginx-deployment annotated
lets recheck the history:
❯ kubectl rollout history deploy nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
2 kubectl apply --filename=nginx-test.yaml --record=true
3 update image from 1.21.6 to latest
lets describe the deployment:
❯ kubectl describe deploy nginx-deploy
Name: nginx-deployment
Namespace: so-tests
CreationTimestamp: Fri, 06 May 2022 00:56:09 -0300
Labels: app=nginx
Annotations: deployment.kubernetes.io/revision: 3
kubernetes.io/change-cause: update image from latest to latest
...
hope this has helped you, bguess.

Answer 2

I experienced a similar situation and it was more an own mistake rather than a configuration issue with my K8s manifests.
At my Docker image creation workflow, I forgot to update the version of my Docker image. I have a GitHub Action workflow that pushes the image to DockerHub. Not updating the image version, overwrote the current image with latest changes in the application.
Then my kubectl rollout undo command was pulling the correct image but that image had the most recent changes in the application. In other words, image 1.1 was the same as 1.0. Running undo had no effect on the application state.
Stupid mistake but that was my experience in case helps someone.