I am using azure Devops for my CICD. I get the below error in release pipeline - Helm upgrade stage. I am not getting any clues as to what causes this. I am not able to go past this stage from the past one day. I do not know what is the "another operation (install/upgrade/rollback) is in progress"
Any help to resolve this would greatly be appreciated, thanks.
I manage to resolve the issue using
helm delete <chart name>
I faced the same issue. I was able to solve it by running this command:
helm rollback <chart name>
I had the same issue while releasing using the ADO pipeline. Finally, I have figured it out using the below steps. I was not even able to list the release under this usual command.
helm list -n <name-space>
this was responding empty. So it's funny behavior from the helm.
kubectl config get-contexts
make sure your context is set for the correct Kubernetes cluster. Then next step is
helm history <release> -n <name-space> --kube-context <kube-context-name>
try applying the rollback to above command.
helm rollback <release> <revision> -n <name-space> --kube-context <kube-context-nam>
Related
I use gitlab + kubernetes.
I use this command:
helm secrets -d vault upgrade --install --atomic --namespace NAMESPACE --values VALUES.yml --set image.tag="WHATEVER" DEPLOYMENT_NAME FILE_TO_THIS_DEPLOYMENT
the moment the CI pipeline fails i cannot restart it again, because of some Kubernetes/Helm errors:
another operation (install/upgrade/rollback) is in progress
I know that i can just fix this inside kubernetes and then i can rerun, but this is just a shitty experience for people who dont know that much about Kubernetes/Helm.
Is there a one-shot command which is just really deploying a new version and if the old version was somehow in a failing state, delete/fix it beforehand?
I really just want to execute the same commands again and again and just expect it to work without manually fixing kubernetes state everytime anything happens.
Yesterday I stopped a helm upgrade when it was running on a release pipeline in Azure DevOps and the followings deployments got it failure.
I tried to see the chart that has failed with the aim of delete it but the chart of the microservice ("auth") doesn't appear. I used the command «helm list -n [namespace_of_AKS]» and it doesn't appear.
What can i do to solve this problem?
Error in Azure Release Pipeline
2022-03-24T08:01:39.2649230Z Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
2022-03-24T08:01:39.2701686Z ##[error]Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
Helm List
This error can happen for few reasons, but it most commonly occurs when there is an interruption during the upgrade/install process as you already mentioned.
To fix this one may need to, first rollback to another version, then reinstall or helm upgrade again.
Try below command to list
helm ls --namespace <namespace>
but you may note that when running that command ,it may not show any columns with information
Try to check the history of the previous deployment
helm history <release> --namespace <namespace>
This provides with information mostly like the original installation was never completed successfully and is pending state something like STATUS: pending-upgrade state.
To escape from this state, use the rollback command:
helm rollback <release> <revision> --namespace <namespace>
revision is optional, but you should try to provide it.
You may then try to issue your original command again to upgrade or reinstall.
helm ls -a -n {namespace} will list all releases within a namespace, regardless of status.
You can also use helm ls -aA instead to list all releases in all namespaces -- in case you actually deployed the release to a different namespace (I've done that before)
Try deleting the latest helm secret for the deployment and re-run your helm apply command.
kubectl get secret -A | grep <app-name>
kubectl delete secret <secret> -n <namespace>
I was doing a practice exam on the website killer.sh , and ran into a question I feel I did a hacky solution to. Given a deployment that has had a bad rollout, revert to the last revision that didn't have any issues. If I check a deployment's rollout history, for example with the command:
kubectl rollout history deployment mydep
I get a small table with version numbers, and "change-cause" commands. Is there any way to check the changes made to the deployment's yaml file for a specific revision? Because I was stumped in figuring out which specific revision didn't have the error inside of it.
Behind the scenes a Deployment creates a ReplicaSet that has its metadata.generation set to the REVISION you see in kubectl rollout history deployment mydep, so you can look at and diff old ReplicaSets associated with the Deployment.
On the other hand, being an eventually-consistent system, kubernetes has no notion of "good" or "bad" state, so it can't know what was the last successful deployment, for example; that's why deployment tools like helm, kapp etc. exist.
Kubernetes does not store more than what is necessary for it to operate and most of the time that is just the desired state because kubernetes is not a version control system.
This is preciously why you need to have a version control system coupled with tools like helm or kustomize where you store the deployment yamls and apply them to the cluster with a new version of the software. This helps in going back in history to dig out details when things break.
You can record the last executed command that changed the deployment with --record option. When using --record you would see the last executed command executed(change-cause) to the deployment metadata.annotations. You will not see this in your local yaml file but when you try to export the deployment as yaml then you will notice the change.
--record option like below
kubectl create deployment <deployment name> --image=<someimage> > testdelpoyment.yaml
kubectl create -f testdeployment.yaml --record
or
kubectl set image deployment/<deploymentname> imagename=newimagename:newversion --record
If I run kubectl apply -f <some statefulset>.yaml separately, is there a way to bind the stateful set to a previous helm release? (eg by specifying some tags in the yaml file)
As far as I know - you cannot do it.
Yes, you can always create resources via templates before installing the Helm chart.
However, I have never seen a solution for your question.
Apologize if this comes under basic. I am using stable/rabbitmq community helm chart to install. But whenever I changed release name using --name option in install like helm install --name rabbitmq stable/rabbitmq, it gives this error: 0/7 nodes are available: 1 PodToleratesNodeTaints, 7 MatchNodeSelector and hangs indefinitely. And if default/auto generated name is used then it creates release. I don't want to use default release name as it creates unpredictable service name as well. Can someone please help me understanding this? Or is there a way to solve it.
Thanks in advance.