ceph shared file system mounting error: permission denied - kubernetes

Question

I am facing this issue when mounting a static Ceph volume to K8s.
MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph │
│ 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/test1-pv/gl │
│ obalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-1586083215,mds_namespace=myfs,_netdev] stderr: mount error 13 = Permission denied
But I'm not sure if this is the right thing to do. Please point me to the right direction. Here is my use case: I want to setup a shared file system that can be accessed from all Pods in all namespaces. Concurrent write operations are not a big concern, as most of the Pods will read from this shared location, such as Python packages etc.
It is not possible by re-using the same PVC as it is a namespaced object.
What I did was to create a static volume in Ceph under a SubVolumeGroup, and create one pv-pvc pair for each namespace, and expect it will access the same files in the Ceph volume.
Here is the volume that I mounted to a Pod:
ubuntu#host1:~$ ls -l /mnt/ceph/volumes/sharedvg/sharedvolume/
total 0
drwxrwxrwx 2 root root 0 Mar 9 11:22 8a370586-60e6-4ec7-9d5b-c8c7ce7786c6
ubuntu#host1:~$ ls -l /mnt/ceph/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6/
total 0
ubuntu#host1:~$
Here is the PV and PVC yaml file. I copied the adminID and adminKey values in the secret 'rook-csi-cephfs-provisioner' of rook-ceph namespace.
apiVersion: v1
kind: Secret
metadata:
name: rook-csi-cephfs-static-provisioner
type: Opaque
data:
userID: "XXX"
userKey: "XXX"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: test1-pv
namespace: default
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 128Gi
csi:
driver: rook-ceph.cephfs.csi.ceph.com
nodeStageSecretRef:
name: rook-csi-cephfs-static-provisioner
namespace: default
volumeAttributes:
clusterID: rook-ceph
fsName: "myfs"
staticVolume: "true"
rootPath: /volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6
volumeHandle: test1-pv
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
claimRef:
name: test-pvc-1
namespace: default
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-pvc-1
namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 128Gi
storageClassName: ""
volumeMode: Filesystem
volumeName: test1-pv
This is the busybox deployment yaml file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-deployment
namespace: default
labels:
app: test
spec:
replicas: 1
strategy:
type: RollingUpdate
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- name: test
image: busybox
imagePullPolicy: IfNotPresent
volumeMounts:
- name: test1
mountPath: /test1
command: ['sh', '-c', 'echo Container 1 is Running ; sleep 3600']
volumes:
- name: test1
persistentVolumeClaim:
claimName: test-pvc-1
This is the log of the Pod:
Warning FailedMount 29m kubelet MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32) │
│ occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kube │
│ let/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-973267258,mds_namespace=myfs,_netdev] stderr: mount error 13 = │
│ Permission denied │
│ Warning FailedMount 27m kubelet MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32) │
│ occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kube │
│ let/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-2348945139,mds_namespace=myfs,_netdev] stderr: mount error 13 │
│ = Permission denied │
│ Warning FailedMount 25m kubelet MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32) │
│ occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kube │
│ let/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-3861388178,mds_namespace=myfs,_netdev] stderr: mount error 13 │
│ = Permission denied │
│ Warning FailedMount 23m kubelet MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal desc = an error (exit status 32) │
│ occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7-9d5b-c8c7ce7786c6 /var/lib/kube │
│ let/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-4165129570,mds_namespace=myfs,_netdev] stderr: mount error 13 │
│ = Permission denied │
│ Warning FailedMount 7m14s (x10 over 34m) kubelet Unable to attach or mount volumes: unmounted volumes=[test1], unattached volumes=[test1 kube-api-access-fwr79]: tim │
│ ed out waiting for the condition │
│ Warning FailedMount 3m3s (x13 over 21m) kubelet (combined from similar events): MountVolume.MountDevice failed for volume "test1-pv" : rpc error: code = Internal d │
│ esc = an error (exit status 32) occurred while running mount args: [-t ceph 10.107.127.65:6789,10.98.28.166:6789,10.96.128.54:6789:/volumes/sharedvg/sharedvolume/8a370586-60e6-4ec7 │
│ -9d5b-c8c7ce7786c6 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/test1-pv/globalmount -o name=csi-cephfs-provisioner,secretfile=/tmp/csi/keys/keyfile-2075406143,mds_namespace=myfs, │
│ _netdev] stderr: mount error 13 = Permission denied │
│ Warning FailedMount 27s (x2 over 29m) kubelet Unable to attach or mount volumes: unmounted volumes=[test1], unattached volumes=[kube-api-access-fwr79 test1]: tim │
│ ed out waiting for the condition
The Pod is in 'ContainerCreating' state.
Any suggestions? Thanks