Unable to delete perconaxtradbclusters which are in errored state - kubernetes

Question

I've installed Percona XtraDB on kubernetes using 1.3.0 operator.
After using it, I wanted to delete the namespace. So I deleted them in the order which I applied them.
Everything is deleted and nothing is visible in svc, pods but there are two resources which are in errored state and cannot be deleted.
~ kubectl get perconaxtradbclusters -n pxc
NAME ENDPOINT STATUS PXC PROXYSQL AGE
cluster1 Error 0 0 4h1m
cluster2 Error 0 0 3h34m
I am not able to delete both of them and due to this I'm not able to create a cluster with the same name.
When I run the delete command, it gets stuck forever
~ kubectl delete perconaxtradbclusters -n pxc cluster1
perconaxtradbcluster.pxc.percona.com "cluster1" deleted
The command execution is never completed.
The yaml of the object
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"pxc.percona.com/v1-3-0","kind":"PerconaXtraDBCluster"}
creationTimestamp: "2020-04-21T18:06:13Z"
deletionGracePeriodSeconds: 0
deletionTimestamp: "2020-04-21T18:38:33Z"
finalizers:
- delete-pxc-pods-in-order
generation: 2
name: cluster2
namespace: pxc
resourceVersion: "5445879"
selfLink: /apis/pxc.percona.com/v1/namespaces/pxc/perconaxtradbclusters/cluster2
uid: 8c100840-b7a8-40d1-b976-1f80c469622b
spec:
allowUnsafeConfigurations: false
backup:
image: percona/percona-xtradb-cluster-operator:1.3.0-backup
schedule:
- keep: 3
name: sat-night-backup
schedule: 0 0 * * 6
storageName: s3-us-west
- keep: 5
name: daily-backup
schedule: 0 0 * * *
storageName: fs-pvc
serviceAccountName: percona-xtradb-cluster-operator
storages:
fs-pvc:
type: filesystem
volume:
persistentVolumeClaim:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 6Gi
s3-us-west:
s3:
bucket: S3-BACKUP-BUCKET-NAME-HERE
credentialsSecret: my-cluster-name-backup-s3
region: us-west-2
type: s3
pmm:
enabled: false
image: percona/percona-xtradb-cluster-operator:1.3.0-pmm
serverHost: monitoring-service
serverUser: pmm
proxysql:
affinity:
antiAffinityTopologyKey: kubernetes.io/hostname
enabled: true
gracePeriod: 30
image: percona/percona-xtradb-cluster-operator:1.3.0-proxysql
podDisruptionBudget:
maxUnavailable: 1
resources:
requests:
cpu: 600m
memory: 1G
size: 3
volumeSpec:
persistentVolumeClaim:
resources:
requests:
storage: 2Gi
pxc:
affinity:
antiAffinityTopologyKey: kubernetes.io/hostname
gracePeriod: 600
image: percona/percona-xtradb-cluster-operator:1.3.0-pxc
podDisruptionBudget:
maxUnavailable: 1
resources:
requests:
cpu: 600m
memory: 4G
size: 3
volumeSpec:
persistentVolumeClaim:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 60Gi
storageClassName: local-storage
secretsName: my-cluster-secrets
sslInternalSecretName: my-cluster-ssl-internal
sslSecretName: my-cluster-ssl
status:
conditions:
- lastTransitionTime: "2020-04-21T18:06:13Z"
message: 'wrong PXC options: set version: new version: Malformed version: '
reason: ErrorReconcile
status: "True"
type: Error
message:
- 'Error: wrong PXC options: set version: new version: Malformed version: '
proxysql:
ready: 0
pxc:
ready: 0
state: Error
How can I get rid of them

Answer 1

Your perconaxtradbclusters yaml example mentions pvc resources, so you'll probably have to delete the associated pvc first, if you haven't already done so.
Can you edit the resources to remove the finalizer blocks, and try delete them again?
kubectl edit perconaxtradbclusters cluster1 -n pxc
and delete
finalizers:
- delete-pxc-pods-in-order
If there's nothing left relying on those resources, that is.
Edit:
I would generally only use this method if I've exhausted all other possibilities and I can't find the hanging resources that are blocking the deletion. I did some digging around. This comment here describes other steps to take before resorting to removing the finalizers.
- Check that the API services are available
- Find any lingering resources that still exist and delete them.