I'm attempting to run a pulumi up command but it's failing because some of the resources that're being replaced were already manually deleted in the GCP Kubernetes dashboard. Is there a way to ignore the fact that these resources have already been deleted and continue with the update?
Run pulumi refresh which will reach out to GCP resources and will update the current state file in Pulumi
Use pulumi state delete <urn> to inform pulumi that the resource has been deleted.
Alternatively, pulumi refresh to attempt an automatic resync.
Finally, you can manually edit the state with pulumi stack export > mystack and pulumi stack import --file mystack.
Related
When I run kubectl delete deployment.yaml, It is displayed on cli that the deployment is deleted. The pod also gets into terminating state. But a new pod is again created with the same deployment and replica-set.
On further digging in I found out that deployment and RS are not being removed. Any reason why deployment and RS wouldn't be removed? Why would the be terminated if deployment isn't removed?
Any leads are appreciated.
As OP confirmed in the comments that they are running argocd then the recreation of the resources is expected behaviour if argocd is running in auto sync mode for the impacted namespace.
Here is a short snippet from the document
Argo CD has the ability to automatically sync an application when it detects differences between the desired manifests in Git, and the live state in the cluster. A benefit of automatic sync is that CI/CD pipelines no longer need direct access to the Argo CD API server to perform the deployment. Instead, the pipeline makes a commit and push to the Git repository with the changes to the manifests in the tracking Git repo.
Solution: you can disable autosync and monitor the delta and approve sync manually. This is something decided at project level. you can read about it here.
I want to redeploy to a new cluster using the terraform destroy command, but I want to keep the state resources managed outside the cluster like "google_storage_bucket".
How can I exclude some resources from being deleted?
Or is there a better practice to do this?
When you use destroy command, all resources from terraform state are planned to be destroyed.
I assume you are talking about the bucket that is also a part of terraform code, but not a part of a cluster that was created using this code.
The best option would be to delete only the resources you want to be destroyed using:
terraform apply -destroy -target=resource.to_be_destroyed
# example:
terraform apply -destroy -target=google_container_cluster.main
which means the destroy plan is going to be only about those specific resource (you can add a couple at the same time using more -target flags).
The other option is to use -replace option. This marks a module or resource to be recreated in next apply, example:
terraform apply -replace=module.gke
Sources:
https://developer.hashicorp.com/terraform/cli/commands/plan#replace-address
https://developer.hashicorp.com/terraform/cli/commands/plan#resource-targeting
I have a node.js app running in AKS and needs to access a Key Vault. I have used Deployment center in the K8s service to set up DevOps. By mistake I did the setup in Deployment center twice which lead to two copies of .yml files (deploytoAksCluster.yml and deploytoAksCluster-1.yml). I have fixed this, but when I run the following command to enable pod identity I get an error.
az aks update -g $resource_group -n $k8s_name --enable-pod-identity
Error:
(BadRequest) Tag name cannot be hidden-DevOpsInfo:GH:my-GithubOrg/myApplication:main:deploytoAksCluster-1.yml:deploytoAksCluster-1.yml:59a0dfdb:my-akscluster:1646402646541.43;GH:my-GithubOrg/myApplication:main:deploytoAksCluster-1.yml:deploytoAksCluster-1.yml:13350477:my-akscluster:1646924094935.21; or be longer than 512 characters. Please see https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-group-using-tags for more details.
Currently I have only one workflow in GitHub (deploytoAksCluster.yml), but the error with reference to deploytoAksCluster-1.yml never goes away.
I have used this sample as inspiration: https://learn.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity#run-a-sample-application
What I have tried
removed the duplicate files
reintroduce the duplicate files
delete the deployment
This is how AKS Deployment center looks.
[1]: https://i.stack.imgur.com/ziFEk.png
Update
59a0dfdb referers to a git commit. This commit resulted in a failed workflow. The workflow has been fixed and everything deploys nicely to K8s, but --enable-pod-identity keeps complaining with the above error. I have removed the commit from github history.
I have even removed the repository in github.
Must be a git history somewhere in k8s that --enable-pod-identity is hung up on somehow?
Please retry deleting the deployment cluster again as in some cases trying twice removed the required cluster which is not required .
Also check if corresponding resource group is deleted and clear the cache .
Try update the git version depending on the type of OS you are using.
Also
NOTE : If you're using existing resources when you're creating a new cluster, such as an IP address or route table, az aks create
overwrites the set of tags. If you delete that cluster later, any tags
set by the cluster will be removed.
from Azure tags on an AKS cluster
To update the tags on an existing cluster, we need to run az aks update with the --tags parameter.
Reference
errors when trying to create update scale delete or upgrade cluster
The tag name of the cluster was auto generated to "hidden-DevOpsInfo:GH:my-GithubOrg/myApplication:main:deploytoAksCluster-1.yml:deploytoAksCluster-1.yml:59a0dfdb:my-akscluster:1646402646541.43;GH:my-GithubOrg/myApplication:main:deploytoAksCluster-1.yml:deploytoAksCluster-1.yml:13350477:my-akscluster:1646924094935.21".
The solution was in the error message: "Tag name cannot be..."
I got the tags by running:
az aks show -g $resource_group -n $k8s_name --query '[tags]'
and updated the tag with:
az aks update --resource-group $resource_group --name $k8s_name --tags "key"="Value"
Is it possible to undo "gcloud container clusters delete" command?
Unfortunately not: Deleting a Cluster
All the source volumes and data (that are not persistent) are removed, and unless you made a conscious choice to take a backup of the cluster, it would be a permanent operation.
If a backup does exist, it would be a restore from backup rather than a revert on the delete command.
I suggest reading a bit more into the Administration of a cluster on Gcloud for more info: Administration of Clusters Overview
Unfortunately if you will delete cluster it is impossible to undo this.
In the GCP documentation you can check what will be deleted after gcloud container clusters delete and what will remain after this command.
One of the things which will remain is Persistent disk volumes. It means that if your ClaimPolicy was set to Retain and your PV status is Released you will be able to get data from PersistentVolume. To do that you will have to create PersistentVolumeClain. More info about ReclaimPolicyhere.
Run $ kubectl get pv to check if it is still bound and check ReclaimPolicy. Similar case can be found in this github thread.
In this documentation you can find step by stop how to connect pod to specific PV.
In addition, please note that you can backup your cluster. To do this you can use for example Ark.
As I'm playing around with K8s deployment and Gitlab CI my deployment got stuck with the state ContainerStarting.
To reset that, I deleted the K8s namespace using kubectl delete namespaces my-namespace.
Now my Gitlab runner shows me
$ ensure_namespace
Checking namespace [MASKED]-docker-3
error: the server doesn't have a resource type "namespace"
error: You must be logged in to the server (Unauthorized)
I think that has something to do with RBAC and most likely Gitlab created that namespace with some arguments and permissions (but I don't know exactly when and how that happens), which are missing now because of my deletion.
Anybody got an idea on how to fix this issue?
In my case I had to delete the namespace in Gitlab database, so gitlab would readd service account and namespace:
On the gitlab machine or task runner enter the PostgreSQL console:
gitlab-rails dbconsole -p
Then select the database:
\c gitlabhq_production
Next step is to find the namespace that was deleted:
SELECT id, namespace FROM clusters_kubernetes_namespaces;
Take the id of the namespace to delete it:
DELETE FROM clusters_kubernetes_namespaces WHERE id IN (6,7);
Now you can restart the pipeline and the namespace and service account will be readded.
Deleting the namespace manually caused the necessary secrets from Gitlab to get removed. It seems they get autocreated on the first ever deployment and it's impossible to repeat that process.
I had to create a new repo and push to it. Now everything works.
Another solution is removing the cluster from Gitlab (under operations/kubernetes in your repo) and re-adding it.
From GitLab 12.6 you can simply clear the cluster cache.
To clear the cache:
Navigate to your project’s Operations > Kubernetes page, and select your cluster.
Expand the Advanced settings section.
Click Clear cluster cache.
This avoids losing secrets and potentially affecting other applications.