How do you install mayastor for openebs with microk8s to use as PV/SC? - kubernetes

Question

I have a 3 node microk8s cluster running on virtualbox Ubuntu vms. And I am trying to get mayastor for openebs working to use with PVCs. I have followed the steps in this guide:
https://mayastor.gitbook.io/introduction/quickstart/preparing-the-cluster
https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor
https://mayastor.gitbook.io/introduction/quickstart/configure-mayastor
An example of my MayastorPool from step 3 looks like this:
apiVersion: "openebs.io/v1alpha1"
kind: MayastorPool
metadata:
name: pool-on-node1-n2
namespace: mayastor
spec:
node: node1
disks: [ "/dev/nvme0n2" ]
And my StorageClass looks like this:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: mayastor-3
provisioner: io.openebs.csi-mayastor
parameters:
repl: '3'
protocol: 'nvmf'
ioTimeout: '60'
local: 'true'
volumeBindingMode: WaitForFirstConsumer
All the checks seem fine according to guide, but when I try to creating a PVC and using it according to this https://mayastor.gitbook.io/introduction/quickstart/deploy-a-test-application the the test application fio pod doesn't come up. When I look at it with describe I see the following:
$ kubectl describe pods fio -n mayastor
Name: fio
Namespace: mayastor
Priority: 0
Node: node2/192.168.40.12
Start Time: Wed, 02 Jun 2021 22:56:03 +0000
Labels: <none>
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
fio:
Container ID:
Image: nixery.dev/shell/fio
Image ID:
Port: <none>
Host Port: <none>
Args:
sleep
1000000
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-l6cdf (ro)
/volume from ms-volume (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
ms-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: ms-volume-claim
ReadOnly: false
kube-api-access-l6cdf:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: openebs.io/engine=mayastor
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 44m default-scheduler Successfully assigned mayastor/fio to node2
Normal SuccessfulAttachVolume 44m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-ec6ce101-fb3e-4a5a-8d61-1d228f8f8199"
Warning FailedMount 24m (x4 over 40m) kubelet Unable to attach or mount volumes: unmounted volumes=[ms-volume], unattached volumes=[kube-api-access-l6cdf ms-volume]: timed out waiting for the condition
Warning FailedMount 13m (x23 over 44m) kubelet MountVolume.SetUp failed for volume "pvc-ec6ce101-fb3e-4a5a-8d61-1d228f8f8199" : rpc error: code = Internal desc = Failed to find parent dir for mountpoint /var/snap/microk8s/common/var/lib/kubelet/pods/b1166af6-1ade-4a3a-9b1d-653151418695/volumes/kubernetes.io~csi/pvc-ec6ce101-fb3e-4a5a-8d61-1d228f8f8199/mount, volume ec6ce101-fb3e-4a5a-8d61-1d228f8f8199
Warning FailedMount 4m3s (x13 over 42m) kubelet Unable to attach or mount volumes: unmounted volumes=[ms-volume], unattached volumes=[ms-volume kube-api-access-l6cdf]: timed out waiting for the condition
Any ideas where to look or what to do to get mayastor working with microk8s? Happy to post more information.

Answer 1

Thanks to Kiran Mova's comments and Niladri from the openebs slack channel:
Replace the step:
https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor#csi-node-plugin
kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/csi-daemonset.yaml
with
curl -fSs https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/csi-daemonset.yaml | sed "s|/var/lib/kubelet|/var/snap/microk8s/common/var/lib/kubelet|g" - | kubectl apply -f -
So replace the path with the microk8s installation specific path. Even though there is a symlink things don't seem to work out right without this change.