In k8s you can roll back a deployment. Can you also roll back a service?
Rolling back a service may be helpful if there was an erroneous update done to a service resource.
rollback / rollout undo is not available for service resource:
kubectl rollout
Manage the rollout of a resource.
Valid resource types include:
* deployments
* daemonsets
* statefulsets
There is no option to roll back the service as answered by confused genius, however just adding my 50 cents.
If you are using the Helm chart for deployment you could implement some way to roll back all resources if your deployment fails.
So while upgrading the helm release version you can use the --atomic which will auto roll back the resources if your deployment fails.
$ helm upgrade --atomic -f myvalues.yaml -f override.yaml redis ./redis
--atomic if set, upgrade process rolls back changes made in case of failed upgrade. The --wait flag will be set
automatically if --atomic is used
Read more about the atomic helm
But again there is no default support for SVC to roll back like deployment.
Related
I am deploying a service through argocd with
SYNC POLICY
AUTOMATED = True
PRUNE RESOURCES = True
SELF HEAL = True
I am using wave https://github.com/wave-k8s/wave to monitor my service deployment with secret as volume mount. When this secret change(which will), wave triggers rolling upgrade for the deployment which will then get the latest secret.
My question is
Does the rolling upgrade happen? If yes, then will argocd auto sync's it to old state after sometime? Or will argocd ignore the rollouts done manually on cluster?
Doing a rolling update is fine and Argo will not fight this. you can validate this by doing a rolling update manually on an application that is managed by Argo (the update should progress w/ no issues)
With that sync policy, you need to make sure that Argo is NOT managing the secret, or that you have an ignoreDifferences setting for it - https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/.
How to do a canary upgrade to existing istio customised setup.
Requirements:
We have existing customised setup of istio 1.7.3 (installed using istoctl method and no revision set for this) for AKS 1.18.14.
Now we need to upgrade to istio 1.8 with no downtime or minimal.
The upgrade should be safer and it wont break our prod environemnt in any ways.
How we installed the current istio customised environment:
created manifest.
istioctl manifest generate --set profile=default -f /manifests/overlay/overlay.yaml > $HOME/generated-manifest.yaml
installed istio.
istioctl install --set profile=default -f /manifests/overlay/overlay.yaml
Verified istio against the deployed manifest.
istioctl verify-install -f $HOME/generated-manifest.yaml
Planned upgrade process Reference
Precheck for upgrade.
istioctl x precheck
export the current used configuration of istio using below command to a yaml file.
kubectl -n istio-system get iop installed-state-install -o yaml > /tmp/iop.yaml
Download istio 1.8 binary and extract the directory and navigate the directory to where we have the 1.8 version istioctl binary.
cd istio1.8\istioctl1.8
from the new version istio directory, create a new controlplane for istio(1.8) with proper revision set and use the previously exported installed-state "iop.yaml".
./istioctl1.8 install --set revision=1-8 --set profile=default -f /tmp/iop.yaml
Expect that it will create new control plane with existing
costamised configuration and now we will have two control plane
deployments and services running side-by-side:
kubectl get pods -n istio-system -l app=istiod
NAME READY STATUS RESTARTS AGE
istiod-786779888b-p9s5n 1/1 Running 0 114m
istiod-1-7-6956db645c-vwhsk 1/1 Running 0 1m
After this, we need to change the existing label of all our cluster namespaces where we need to inject the istio proxy containers. Need to remove the old "istio-injection" label, and add the istio.io/rev label to point to the canary revision 1-8.
kubectl label namespace test-ns istio-injection- istio.io/rev=1-8
Hope, at this point also the environment is stable with old istio configurations and we can make decision on which app pods can be restarted to make the new control plane changes as per our downtime, and its allowed to run some apps with older control plane and another with new controller plane configs t this point. eg:
kubectl rollout restart deployment -n test-ns (first)
kubectl rollout restart deployment -n test-ns2 (later)
kubectl rollout restart deployment -n test-ns3 (again after sometieme later)
Once we planed for downtime and restarted the deployments as we decided, confirm all the pods are now using dataplane proxy injector of version 1.8 only
kubectl get pods -n test-ns -l istio.io/rev=1-8
To verify that the new pods in the test-ns namespace are using the istiod-canary service corresponding to the canary revision
istioctl proxy-status | grep ${pod_name} | awk '{print $7}'
After upgrading both the control plane and data plane, can uninstall the old control plane
istioctl x uninstall -f /tmp/iop.yaml.
Need to clear below points before upgrade.
Are all the steps prepared for the upgrade above are good to proceed for highly used Prod environment?
By exporting the installed state iop is enough to get all customised step to proceed the canary upgrade? or is there any chance of braking the upgrade or missing any settings?
Whether the step 4 above will create the 1.8 istio control plane with all the customization as we already have without any break or missing something?
after the step 4, do we need to any any extra configuration related to istiod service configuration> the followed document is not clear about that,
for the step 5 above, how we can identy all the namespaces, where we have the istio-injection enabled and only modify those namespace alone and leave others as it was before?
so for the step 8 above, how to ensure we are uninstalling old control plane only ? we have to get the binary for old controlplane say (1.7 in my case)and use that binary with same exported /tmp/iop.yaml?
No Idea about how to rollback any issues happened in between.. before or after the old controlplane deleted
No. You should go through changelog and upgrade notes. See what's new, what's changed, depracted etc. Adjust your configs accordingly.
In theory - yes, in practice - no. See above. That's why you should always check upgarde notes/changelog and plan accordingly. There is always a slim chance something will go wrong.
It should, but again, be prepared that something may break (One more time - go through changelog/upgrade notes, this is important).
No.
You can find all namespaces with Istio injection enabled with:
kubectl get namespaces -l=istio-injection=enabled
Istio upgrade process should only modify namespaces with injection enabled (and istio-system namespace).
If your old control plane does not have a revision label, you have to uninstall it using its original installation options (old yaml file)
istioctl x uninstall -f /path/to/old/config.yaml
If it does have revision label:
istioctl x uninstall --revision <revision>
You can just uninstall new control plane with
istioctl x uninstall revision=1-8
This will revert to the old control plane, assuming you have not yet uninstalled it. However, you will have to reinstall gateways for the old version manually, as the uninstall command does not revert them automatically.
I would strongly recommend creating a temporary test environment. Recreating existing cluster on test env. Performing upgrade there, and adjusting the process to meet your needs.
This way you will avoid catastrofic failures on your production environment.
I was doing a practice exam on the website killer.sh , and ran into a question I feel I did a hacky solution to. Given a deployment that has had a bad rollout, revert to the last revision that didn't have any issues. If I check a deployment's rollout history, for example with the command:
kubectl rollout history deployment mydep
I get a small table with version numbers, and "change-cause" commands. Is there any way to check the changes made to the deployment's yaml file for a specific revision? Because I was stumped in figuring out which specific revision didn't have the error inside of it.
Behind the scenes a Deployment creates a ReplicaSet that has its metadata.generation set to the REVISION you see in kubectl rollout history deployment mydep, so you can look at and diff old ReplicaSets associated with the Deployment.
On the other hand, being an eventually-consistent system, kubernetes has no notion of "good" or "bad" state, so it can't know what was the last successful deployment, for example; that's why deployment tools like helm, kapp etc. exist.
Kubernetes does not store more than what is necessary for it to operate and most of the time that is just the desired state because kubernetes is not a version control system.
This is preciously why you need to have a version control system coupled with tools like helm or kustomize where you store the deployment yamls and apply them to the cluster with a new version of the software. This helps in going back in history to dig out details when things break.
You can record the last executed command that changed the deployment with --record option. When using --record you would see the last executed command executed(change-cause) to the deployment metadata.annotations. You will not see this in your local yaml file but when you try to export the deployment as yaml then you will notice the change.
--record option like below
kubectl create deployment <deployment name> --image=<someimage> > testdelpoyment.yaml
kubectl create -f testdeployment.yaml --record
or
kubectl set image deployment/<deploymentname> imagename=newimagename:newversion --record
I'm running GKE cluster and there is a deployment that uses image which I push to Container Registry on GCP, issue is - even though I build the image and push it with latest tag, the deployment keeps on creating new pods with the old one cached - is there a way to update it without re-deploying (aka without destroying it first)?
There is a known issue with the kubernetes that even if you change configmaps the old config remains and you can either redeploy or workaround with
kubectl patch deployment $deployment -n $ns -p \
"{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"
is there something similar with cached images?
I think you're looking for kubectl set or patch which I found there in kubernetes documentation.
To update image of deployment you can use kubectl set
kubectl set image deployment/name_of_deployment name_of_deployment=image:name_of_image
To update image of your pod you can use kubectl patch
kubectl patch pod name_of_pod -p '{"spec":{"containers":[{"name":"name_of_pod_from_yaml","image":"name_of_image"}]}}'
You can always use kubectl edit to edit which allows you to directly edit any API resource you can retrieve via the command line tool.
kubectl edit deployment name_of_deployment
Let me know if you have any more questions.
1) You should change the way of your thinking. Destroying pod is not bad. Application downtime is what is bad. You should always plan your deployments in such a way that it can tolerate one pod death. Use multiple replicas for stateless apps and use clusters for stateful apps. Use Kubernetes rolling update for any changes to your deployments. Rolling updates have many extremely important settings which directly influence the uptime of your apps. Read it carefully.
2) The reason why Kubernetes launches old image is that by default it uses
imagePullPolicy: IfNotPresent. Use imagePullPolicy: Always and it will always try to pull latest version on redeploy.
Is there some tool available that could tell me whether a K8s YAML configuration (to-be-supplied to kubectl apply) is valid for the target Kubernetes version without requiring a connection to a Kubernetes cluster?
One concrete use-case here would be to detect incompatibilities before actual deployment to a cluster, just because some already-deprecated label has been finally dropped in a newer Kubernetes version, e.g. as has happened for Helm and the switch to Kubernetes 1.16 (see Helm init fails on Kubernetes 1.16.0):
Dropped:
apiVersion: extensions/v1beta1
New:
apiVersion: apps/v1
I want to check these kind of incompatibilities within a CI system, so that I can reject it before even attempting to deploy it.
just run below command to validate the syntax
kubectl create -f <yaml-file> --dry-run
In fact the dry-run option is to validate the YAML syntax and the object schema. You can grab the output into a variable and if there is no error then rerun the command without dry-run
You could use kubeval
https://kubeval.instrumenta.dev/
I don't think kubectl support client-side only validation yet (02/2022)