Currently, I'm updating the version of an image to be deployed using the set image command:
$ kubectl set image deployments myapp myapp=caarlos0/myapp:v2
And then I watch the changes with rollout status:
$ kubectl rollout status deployments myapp
The problems I found while doing it this way are:
some times, it seems that a deployment is not triggered at all, and when I call rollout status, I get errors like this:
$ kubectl rollout status deployments myapp
Waiting for deployment spec update to be observed...
error: timed out waiting for the condition
The rollout history command show the CHANGE-CAUSE as <none>, and I can't find a way of making it show anything useful in there.
So, am I doing something wrong (or not in the best way)? How can I improve this workflow?
You're doing the right thing. Within the Updating a deployment documentation you'll find this:
Note: a Deployment’s rollout is triggered if and only if the Deployment’s pod template (i.e. .spec.template) is changed, e.g. updating labels or container images of the template. Other updates, such as scaling the Deployment, will not trigger a rollout.
So running $ kubectl set image deployments/app <image> will only trigger a rollout if <image> is not already configured for your containers.
The change cause can be used to record the command which was used to trigger the rollout by appending the --record flag to your commands (see Checking rollout history).
Deployment is observed by the controlplane component Deployment controller, make sure kube-controller-manager is running. for example below controller-manager is not running so it will not rollout deployments.
once controller is up and running it will start rollout example below
Related
A pod can be created by Deployment or ReplicaSet or DaemonSet, if I am updating a pod's container specs, is it OK for me to simply modify the yaml file that created the pod? Would it be erroneous once I have done that?
Brief Question:
Is kubectl apply -f xxx.yml the silver bullet for all pod update?
...if I am updating a pod's container specs, is it OK for me to simply modify the yaml file that created the pod?
The fact that the pod spec is part of the controller spec (eg. deployment, daemonset), to update the container spec you naturally start with the controller spec. Also, a running pod is largely immutable, there isn't much you can change directly unless you do a replace - which is what the controller already doing.
you should not make changes to the pods directly, but update the spec.template.spec section of the deployment used to create the pods.
reason for this is that the deployment is the controller that manages the replicasets and therefore the pods that are created for your application. that means if you apply changes to the pods manifest directly, and something like a pod rescheduling/restart happens, the changes made to the pod will be lost because the replicaset will recreate the pod according to its own specification and not the specification of the last running pod.
you are safe to use kubectl apply to apply changes to existing resources but if you are unsure, you can always extract the current state of the deployment from kubernetes and pipe that output into a yaml file to create a backup:
kubectl get deploy/<name> --namespace <namespace> -o yaml > deploy.yaml
another option is to use the internal rollback mechanism of kubernetes to restore a previous revision of your deployment. see https://learnk8s.io/kubernetes-rollbacks for more infos on that.
When deciding on update strategy for a kubernetes application, there is an option to use Recreate strategy.
How would this be different from just uninstalling and installing the app?
I assume that by "just uninstalling and installing the app" you mean complete deletion of your deployment e.g.:
kubectl delete deployment nginx-deployment
and creating it again:
kubectl apply -f nginx-deployment.yaml
Note that when using Recreate strategy there is no complete deletion of the deployment so there is fundamental difference here. By choosing this strategy you only inform kubernetes that all the pods managed by your deployment should be deleted and recreated when you update them (e.g. you update the image version of the container) rather than deleting and recreating their new versions one at a time what takes place when using RollingUpdate strategy. This way you make sure that certain number of pods serving an old version of the application are still available when the update occurs and pods with a new version of the image appear.
When you delete your deployment and create a new one, your new deployment has nothing to do with the old one. In other words, completely new Deployment resource is created and no history of the changes you made is preserved.
I believe the best way of explaining things is always an example. So let's move on to the following one.
Let's say you've created a new nginx deployment based on your yaml manifest:
kubectl apply -f nginx-deployment.yaml
and then you decided to update the image version, either by editing nginx-deployment.yaml manifest and re-applying it or this way:
kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 --record=true
In either case you will be able to check rollout history by running:
kubectl rollout history deployment nginx-deployment
and you should see something like this:
$ kubectl rollout history deployment nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
1 kubectl apply --filename=nginx-deployment.yaml --record=true
2 kubectl set image deployment nginx-deployment nginx=nginx:1.16.1 --record=true
When you have rollout history you're able to undo your latest change and go back to the previous revision:
kubectl rollout undo deployment.v1.apps/nginx-deployment
Now your rollout history for this deployment may look like this:
$ kubectl rollout history deployment nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
2 kubectl set image deployment nginx-deployment nginx=nginx:1.16.1 --record=true
3 kubectl apply --filename=nginx-deployment.yaml --record=true
When you simply delete your deployment and recreate it again you will have nothing in rollout history for newly created deployment and you won't be able to roll it back to some older revision in such an easy way.
Recreate strategy will delete your Pods and then add new Pods - you will get short downtime, but on the other side you will not use much extra resources during upgrade.
You typically want RollingUpgrade since that takes a few Pods at a time and you can deploy stateless applications without downtime.
I'm trying to get all the deployments of a namespace to be restarted for implementation reasons.
I'm using "kubectl rollout -n restart deploy" and it works perfectly, but I'm not sure it that command causes downtime or if it works as the "rollout update", applying the restart one by one, keeping my services up.
Does anyone know?
In the documentation I can only find this:
Operation
Syntax
Description
rollout
kubectl rollout SUBCOMMAND [options]
Manage the rollout of a resource. Valid resource types include: deployments, daemonsets and statefulsets.
But I can't find details about the specific "rollout restart deploy".
I need to make sure it doesn't cause downtime. Right now is very hard to tell, because the restart process is very quick.
Update: I know that for one specific deployment (kubectl rollout restart deployment/name), it works as expected and doesn't cause downtime, but I need to apply it to all the namespace (without specifying the deployment) and that's the case I'm not sure about.
kubectl rollout restart deploy -n namespace1 will restart all deployments in specified namespace with zero downtime.
Restart command will work as follows:
After restart it will create new pods for a each deployments
Once new pods are up (running and ready) it will terminate old pods
Add readiness probes to your deployments to configure initial delays.
#pcsutar 's answer is almost correct. kubectl rollout restart $resourcetype $resourcename restarts your deployment, daemonset or stateful set according to the its update strategy. so if it is set to rollingUpdate it will behave exactly as the above answer:
After restart it will create new pods for a each deployments
Once new pods are up (running and ready) it will terminate old pods
Add readiness probes to your deployments to configure initial delays.
However, if the strategy for example is type: recreate all the currently running pods belonging to the deployment will be terminated before new pods will be spun up!
I have a Pod - I want to update to latest version of container image, if it fails then want to revert it back to earlier version of container image.
Which approach of below will be good to update to latest or earlier container image.
1> using kubectl patch is good option?
2> deleting the pod and again creating the pod is good option?
what are benefits/disadvantages of both approach.
kubectl patch pod test-pod -p '{"spec":{"containers":[{"name":"test","image":"1.0"}]}}'
kubectl delete pod test-pod and kubectl apply -f testpod.yaml
From the docs here one of the use cases of deployment
Rollback to an earlier Deployment revision if the current state of the
Deployment is not stable. Each rollback updates the revision of the
Deployment.
So use deployment to perform rolling update and rollback when necessary
kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 --record=true
Check rollout history
kubectl rollout history deployment.v1.apps/nginx-deployment
Rollback to a previous version
kubectl rollout undo deployment.v1.apps/nginx-deployment
based on the scenario put up by you, you can choose to look for doing canary deployments by using the rolling update strategy, if the new deployment's containers fail to serve the load then the old ones will not be destroyed and load will continuously be served by pre-existing containers.
The other way can be to keep deployment history, you can define the number of manifests you want to be saved by kubernetes and then you can manually rollback to your desired version
And if you are not aware about what version carries which image then you can go for patch command, as portrayed by you in the question
I usually restart my applications by:
kubectl scale deployment my-app --replicas=0
Followed by:
kubectl scale deployment my-app --replicas=1
which works fine. I also have another running application but when I look at its replicaset I see:
$ kubectl get rs
NAME DESIRED CURRENT READY AGE
another-app 2 2 2 2d
So to restart that correctly I would of course need to:
kubectl scale deployment another-app --replicas=0
kubectl scale deployment another-app --replicas=2
But is there a better way to do this so I don't have to manually look at the repliasets before scaling/restarting my application (that might have replicas > 1)?
You can restart pods by using level
kubectl delete pods -l name=myLabel
You can rolling restart of all pods for a deployments, so that you don't take the service down
kubectl patch deployment your_deployment_name -p \
"{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"
And After kubernetes version 1.15 you can
kubectl rollout restart deployment your_deployment_name
To make changes in your current deployment you can use kubectl rollout pause deployment/YOUR_DEPLOYMENT. This way the deployment will be marked as paused and won't be reconciled by the controller. After it's paused you can make necessary changes to your configuration and then resume it by using kubectl rollout resume deployment/YOUR_DEPLOYMENT. This way it will create a new replicaset with updated configuration.
Pod with new configuration will be started and when it's in running status, pod with old configuration will be terminated.
Using this method you will be able to rollout the deployment to previous version by using:
kubectl rollout history deployment/YOUR_DEPLOYMENT
to check history of the rollouts and then execute following command to rollback:
kubectl rollout undo deployment/YOUR_DEPLOYMENT --to-revision=REVISION_NO