How to do a canary upgrade to existing istio customised setup.
Requirements:
We have existing customised setup of istio 1.7.3 (installed using istoctl method and no revision set for this) for AKS 1.18.14.
Now we need to upgrade to istio 1.8 with no downtime or minimal.
The upgrade should be safer and it wont break our prod environemnt in any ways.
How we installed the current istio customised environment:
created manifest.
istioctl manifest generate --set profile=default -f /manifests/overlay/overlay.yaml > $HOME/generated-manifest.yaml
installed istio.
istioctl install --set profile=default -f /manifests/overlay/overlay.yaml
Verified istio against the deployed manifest.
istioctl verify-install -f $HOME/generated-manifest.yaml
Planned upgrade process Reference
Precheck for upgrade.
istioctl x precheck
export the current used configuration of istio using below command to a yaml file.
kubectl -n istio-system get iop installed-state-install -o yaml > /tmp/iop.yaml
Download istio 1.8 binary and extract the directory and navigate the directory to where we have the 1.8 version istioctl binary.
cd istio1.8\istioctl1.8
from the new version istio directory, create a new controlplane for istio(1.8) with proper revision set and use the previously exported installed-state "iop.yaml".
./istioctl1.8 install --set revision=1-8 --set profile=default -f /tmp/iop.yaml
Expect that it will create new control plane with existing
costamised configuration and now we will have two control plane
deployments and services running side-by-side:
kubectl get pods -n istio-system -l app=istiod
NAME READY STATUS RESTARTS AGE
istiod-786779888b-p9s5n 1/1 Running 0 114m
istiod-1-7-6956db645c-vwhsk 1/1 Running 0 1m
After this, we need to change the existing label of all our cluster namespaces where we need to inject the istio proxy containers. Need to remove the old "istio-injection" label, and add the istio.io/rev label to point to the canary revision 1-8.
kubectl label namespace test-ns istio-injection- istio.io/rev=1-8
Hope, at this point also the environment is stable with old istio configurations and we can make decision on which app pods can be restarted to make the new control plane changes as per our downtime, and its allowed to run some apps with older control plane and another with new controller plane configs t this point. eg:
kubectl rollout restart deployment -n test-ns (first)
kubectl rollout restart deployment -n test-ns2 (later)
kubectl rollout restart deployment -n test-ns3 (again after sometieme later)
Once we planed for downtime and restarted the deployments as we decided, confirm all the pods are now using dataplane proxy injector of version 1.8 only
kubectl get pods -n test-ns -l istio.io/rev=1-8
To verify that the new pods in the test-ns namespace are using the istiod-canary service corresponding to the canary revision
istioctl proxy-status | grep ${pod_name} | awk '{print $7}'
After upgrading both the control plane and data plane, can uninstall the old control plane
istioctl x uninstall -f /tmp/iop.yaml.
Need to clear below points before upgrade.
Are all the steps prepared for the upgrade above are good to proceed for highly used Prod environment?
By exporting the installed state iop is enough to get all customised step to proceed the canary upgrade? or is there any chance of braking the upgrade or missing any settings?
Whether the step 4 above will create the 1.8 istio control plane with all the customization as we already have without any break or missing something?
after the step 4, do we need to any any extra configuration related to istiod service configuration> the followed document is not clear about that,
for the step 5 above, how we can identy all the namespaces, where we have the istio-injection enabled and only modify those namespace alone and leave others as it was before?
so for the step 8 above, how to ensure we are uninstalling old control plane only ? we have to get the binary for old controlplane say (1.7 in my case)and use that binary with same exported /tmp/iop.yaml?
No Idea about how to rollback any issues happened in between.. before or after the old controlplane deleted
No. You should go through changelog and upgrade notes. See what's new, what's changed, depracted etc. Adjust your configs accordingly.
In theory - yes, in practice - no. See above. That's why you should always check upgarde notes/changelog and plan accordingly. There is always a slim chance something will go wrong.
It should, but again, be prepared that something may break (One more time - go through changelog/upgrade notes, this is important).
No.
You can find all namespaces with Istio injection enabled with:
kubectl get namespaces -l=istio-injection=enabled
Istio upgrade process should only modify namespaces with injection enabled (and istio-system namespace).
If your old control plane does not have a revision label, you have to uninstall it using its original installation options (old yaml file)
istioctl x uninstall -f /path/to/old/config.yaml
If it does have revision label:
istioctl x uninstall --revision <revision>
You can just uninstall new control plane with
istioctl x uninstall revision=1-8
This will revert to the old control plane, assuming you have not yet uninstalled it. However, you will have to reinstall gateways for the old version manually, as the uninstall command does not revert them automatically.
I would strongly recommend creating a temporary test environment. Recreating existing cluster on test env. Performing upgrade there, and adjusting the process to meet your needs.
This way you will avoid catastrofic failures on your production environment.
Related
In k8s you can roll back a deployment. Can you also roll back a service?
Rolling back a service may be helpful if there was an erroneous update done to a service resource.
rollback / rollout undo is not available for service resource:
kubectl rollout
Manage the rollout of a resource.
Valid resource types include:
* deployments
* daemonsets
* statefulsets
There is no option to roll back the service as answered by confused genius, however just adding my 50 cents.
If you are using the Helm chart for deployment you could implement some way to roll back all resources if your deployment fails.
So while upgrading the helm release version you can use the --atomic which will auto roll back the resources if your deployment fails.
$ helm upgrade --atomic -f myvalues.yaml -f override.yaml redis ./redis
--atomic if set, upgrade process rolls back changes made in case of failed upgrade. The --wait flag will be set
automatically if --atomic is used
Read more about the atomic helm
But again there is no default support for SVC to roll back like deployment.
I use a kubernetes manifest file to deploy my code. My manifest typically has a number of things like Deployment, Service, Ingress, etc.. How can I perform a type of "rollout" or "restart" of everything that was applied with my manifest?
I know I can update my deployment say by running
kubectl rollout restart deployment <deployment name>
but what if I need to update all resources like ingress/service? Can it all be done together?
I would recommend you to store your manifests, e.g. Deployment, Service and Ingress in a directory, e.g. <your-directory>
Then use kubectl apply to "apply" those files to Kubernetes, e.g.:
kubectl apply -f <directory>/
See more on Declarative Management of Kubernetes Objects Using Configuration Files.
When your Deployment is updated this way, your pods will be replaced with the new version during a rolling deployment (you can configure to use another deployment strategy).
This is a Community Wiki answer so feel free to edit it and add any additional details you consider important.
As Burak Serdar has already suggested in his comment, you can simply use:
kubectl apply -f your-manifest.yaml
and it will apply all the changes you made in your manifest file to the resources, which are already deployed.
However note that running:
kubectl rollout restart -f your-manifest.yaml
makes not much sense as this file contains definitions of resources such as Services to which kubectl rollout restart cannot be applied. In consequence you'll see the following error:
$ kubectl rollout restart -f deployment-and-service.yaml
deployment.apps/my-nginx restarted
error: services "my-nginx" restarting is not supported
So as you can see it is perfectly possible to run kubectl rollout restart against a file that contains definitions of both resources that support this operation and those which do not support it.
Running kubectl apply instead will result in update of all the resources which definition has changed in your manifest:
$ kubectl apply -f deployment-and-service.yaml
deployment.apps/my-nginx configured
service/my-nginx configured
I was doing a practice exam on the website killer.sh , and ran into a question I feel I did a hacky solution to. Given a deployment that has had a bad rollout, revert to the last revision that didn't have any issues. If I check a deployment's rollout history, for example with the command:
kubectl rollout history deployment mydep
I get a small table with version numbers, and "change-cause" commands. Is there any way to check the changes made to the deployment's yaml file for a specific revision? Because I was stumped in figuring out which specific revision didn't have the error inside of it.
Behind the scenes a Deployment creates a ReplicaSet that has its metadata.generation set to the REVISION you see in kubectl rollout history deployment mydep, so you can look at and diff old ReplicaSets associated with the Deployment.
On the other hand, being an eventually-consistent system, kubernetes has no notion of "good" or "bad" state, so it can't know what was the last successful deployment, for example; that's why deployment tools like helm, kapp etc. exist.
Kubernetes does not store more than what is necessary for it to operate and most of the time that is just the desired state because kubernetes is not a version control system.
This is preciously why you need to have a version control system coupled with tools like helm or kustomize where you store the deployment yamls and apply them to the cluster with a new version of the software. This helps in going back in history to dig out details when things break.
You can record the last executed command that changed the deployment with --record option. When using --record you would see the last executed command executed(change-cause) to the deployment metadata.annotations. You will not see this in your local yaml file but when you try to export the deployment as yaml then you will notice the change.
--record option like below
kubectl create deployment <deployment name> --image=<someimage> > testdelpoyment.yaml
kubectl create -f testdeployment.yaml --record
or
kubectl set image deployment/<deploymentname> imagename=newimagename:newversion --record
We are developing k8s CSI driver
Currently in order to upgrade driver we delete the installed operator pods, cdrs and roles and recreate them from new version images.
What is suggested way to do upgrade? Or is uninstall/install is the suggested method?
I couldn't find any relevant information
We also have support of installing from OpenShift. Is there any difference regarding upgrade from OpenShift?
You should start from this documentation:
This page describes to CSI driver developers how to deploy their
driver onto a Kubernetes cluster.
Especially:
Deploying a CSI driver onto Kubernetes is highlighted in detail in
Recommended Mechanism for Deploying CSI Drivers on Kubernetes.
Also, you will find there all the necessary info with an example.
Your question lacks some details regarding your use case but I strongly recommend starting from the guide I have presented you.
Please, let me know if that helps.
CSI drivers can differ, but I believe the best approach is to do rolling update of your plugin's DaemonSet. It will happen automatically once you apply the new DaemonSet configuration, e.g. newer docker image.
For more details, see https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/
For example:
kubectl get -n YOUR-NAMESPACE daemonset YOUR-DAEMONSET --export -o yaml > plugin.yaml
vi plugin.yaml # Update your image tag(s)
kubectl apply -n YOUR-NAMESPACE -f plugin.yaml
A shorted way to update just the image:
kubectl set image ds/YOUR-DAEMONSET-NAME YOUR-CONTAINER-NAME=YOUR-IMAGE-URL:YOUR-TAG -n YOUR-NAMESPACE
Note: I found that I also needed to restart (kill) the pod with the external provisioner. There's probably a more elegant way to handle this, but it works in a pinch.
kubectl delete pod -n YOUR-NAMESPACE YOUR-EXTERNAL-PROVISIONER-POD
Is there any way for me to replicate the behavior I get on cloud.docker where a service can be redeployed either manually with the latest image or automatically when the repository image is updated?
Right now I'm doing something like this manually in a shell script with my controller and service files:
kubectl delete -f ./ticketing-controller.yaml || true
kubectl delete -f ./ticketing-service.yaml || true
kubectl create -f ./ticketing-controller.yaml
kubectl create -f ./ticketing-service.yaml
Even that seems a bit heavy handed, but works fine. I'm really missing the autoredeploy feature I have on cloud.docker.
Deleting the controller yaml file itself won't delete the actual controller in kubernetes unless you have a special configuration to do so. If you have more than 1 instance running, deleting the controller probably isn't what you would want because it would delete all the instances of your running application. What you really want to do is perform a rolling update of your application that incrementally replaces containers running the old image with containers running the new one.
You can do this manually by:
For a Deployment controller update the yaml file image and execute kubectl apply.
For a ReplicationController update the yaml file and execute kubectl rollingupdate. See: http://kubernetes.io/docs/user-guide/rolling-updates/
With v1.3 you will be able to use kubectl set image
Alternatively you could use a PaaS to automatically push the image when it is updated in the repo. Here is an incomplete list of a few Paas options:
Red Hat OpenShift
Spinnaker
Deis Workflow
According to Kubernetes documentation:
Let’s say you were running version 1.7.9 of nginx:
$ kubectl run my-nginx --image=nginx:1.7.9 --replicas=3
deployment "my-nginx" created
To update to version 1.9.1, simply change
.spec.template.spec.containers[0].image from nginx:1.7.9 to
nginx:1.9.1, with the kubectl commands.
$ kubectl edit deployment/my-nginx
That’s it! The Deployment will declaratively update the deployed nginx
application progressively behind the scene. It ensures that only a
certain number of old replicas may be down while they are being
updated, and only a certain number of new replicas may be created
above the desired number of pods.