RookIO AttachVolume.Attach failed for volume - kubernetes

Question

I have Kubernetes 1.18 with rookio setup, this pod was running for sometime. one of the node went out of Ready status for some reason. I rebooted the node, now its in Ready status.
But Pod stuck on ContainerCreating status. its waiting to mount the rookio PVC.
Pod status
# kgp |grep -v Running
NAME READY STATUS RESTARTS AGE
redis-slave-0 0/1 ContainerCreating 0 14h
PodEvents
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 23m (x69 over 13h) kubelet, node05 Unable to attach or mount volumes: unmounted volumes=[redis-data], unattached volumes=[redis-data config redis-tmp-conf default-token-lqpgm health]: timed out waiting for the condition
Warning FailedMount 19m (x95 over 13h) kubelet, node05 Unable to attach or mount volumes: unmounted volumes=[redis-data], unattached volumes=[health redis-data config redis-tmp-conf default-token-lqpgm]: timed out waiting for the condition
Warning FailedMount 14m (x79 over 13h) kubelet, node05 Unable to attach or mount volumes: unmounted volumes=[redis-data], unattached volumes=[default-token-lqpgm health redis-data config redis-tmp-conf]: timed out waiting for the condition
Warning FailedMount 5m45s (x66 over 13h) kubelet, node05 Unable to attach or mount volumes: unmounted volumes=[redis-data], unattached volumes=[config redis-tmp-conf default-token-lqpgm health redis-data]: timed out waiting for the condition
Warning FailedAttachVolume 2m44s (x101 over 6h32m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-e854eee7-0a36-4a92-ba61-f9e6e976f64c" : attachdetachment timeout for volume 0001-0009-rook-ceph-0000000000000002-0c4a5173-e8a7-11ea-9bd1-0637030c9151
PVC attach status set to false
kubectl get volumeattachment |grep -v true
NAME ATTACHER PV NODE ATTACHED AGE
csi-3424d1bdc5212aeef30e681c9d99df38dd68fdabb47e5f820125c90d54d61d7b rook-ceph.rbd.csi.ceph.com pvc-e854eee7-0a36-4a92-ba61-f9e6e976f64c node05 false 14h
I try to move the pod to different node, still same issue.
PV and PVC status
# k describe pv pvc-e854eee7-0a36-4a92-ba61-f9e6e976f64c
Name: pvc-e854eee7-0a36-4a92-ba61-f9e6e976f64c
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: rook-ceph.rbd.csi.ceph.com
Finalizers: [kubernetes.io/pv-protection]
StorageClass: rook-ceph-block
Status: Bound
Claim: default/redis-data-redis-slave-0
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 287Mi
Node Affinity: <none>
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: rook-ceph.rbd.csi.ceph.com
FSType: ext4
VolumeHandle: 0001-0009-rook-ceph-0000000000000002-0c4a5173-e8a7-11ea-9bd1-0637030c9151
ReadOnly: false
VolumeAttributes: clusterID=rook-ceph
imageFeatures=layering
imageFormat=2
imageName=csi-vol-0c4a5173-e8a7-11ea-9bd1-0637030c9151
journalPool=replicapool
pool=replicapool
radosNamespace=
storage.kubernetes.io/csiProvisionerIdentity=1598460149789-8081-rook-ceph.rbd.csi.ceph.com
k describe pvc redis-data-redis-slave-0
Name: redis-data-redis-slave-0
Namespace: default
StorageClass: rook-ceph-block
Status: Bound
Volume: pvc-e854eee7-0a36-4a92-ba61-f9e6e976f64c
Labels: app=redis
component=slave
heritage=Helm
release=redis
role=slave
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 287Mi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: redis-slave-0
Events: <none>
How to fix this mount issue?
Thanks
SR

Answer 1

Please check if you have the PV and Node in the same zone(availability zones in AWS). If they are in different zones the PV will not attach to the node.
To resolve this simply delete the PV and PVC and recreate it, it will create the new PV in the same zone as node.