Unable to mount local PersistentVolume in k8s 1.13 - kubernetes

Question

I'm trying to deploy a stateful set with a persistent volume claim on a bare metal kubernetes cluster (v1.13) but the pod times out when trying to mount the volume.
I have a local-storage storage class defined:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local-storage
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
I have a PV defined:
kind: PersistentVolume
apiVersion: v1
metadata:
name: cassandradev1
labels:
app: cassandra
environment: dev
spec:
storageClassName: local-storage
capacity:
storage: 1Ti
accessModes:
- ReadWriteOnce
local:
path: "/data1/cassandradev1"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- my-node1
And I have a stateful set that issues a claim (truncated):
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: cassandra-set
spec:
...
volumeClaimTemplates:
- metadata:
name: cassandra-data
spec:
selector:
matchLabels:
app: "cassandra"
environment: "dev"
accessModes: [ "ReadWriteOnce" ]
storageClassName: "local-storage"
resources:
requests:
storage: 1Ti
When I try to apply the stateful set, the Pod gets scheduled but times out:
Normal Scheduled 2m13s default-scheduler Successfully assigned default/cassandra-set-0 to my-node1
Warning FailedMount 13s kubelet, my-node1 Unable to mount volumes for pod "cassandra-set-0 (dd252f77-fda3-11e8-96d3-1866dab905dc)": timeout expired waiting for volumes to attach or mount for pod "default"/"cassandra-set-0". list of unmounted volumes=[cassandra-data]. list of unattached volumes=[cassandra-data default-token-t2dg8]
If I look at the logs for the controller I see an error message for no volume plugin matched:
kubectl logs pod/kube-controller-manager -n kube-system
W1212 00:51:24.218114 1 plugins.go:845] FindExpandablePluginBySpec(cassandradev1) -> err:no volume plugin matched
Any ideas on where to look next?

Answer 1

First of all PV definition is incorrect, there is no hostPath in local-storage class. This is how you should define your local storage PV:
kind: PersistentVolume
apiVersion: v1
metadata:
name: cassandradev1
labels:
app: cassandra
environment: dev
spec:
storageClassName: local-storage
capacity:
storage: 1Ti
accessModes:
- ReadWriteOnce
local:
path: "/data1/cassandradev1"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- my-node1
Also keep in mind unlike hostPath, /data1/cassandradev1 should exist on my-node1, local-storage doesn't automatically create the path and when you deploy statefulset and path is not there, it will give error related to mounting.
This should resolve your issue. Hope this helps.
EDIT: So, I setup the cassandra statefulset with local-storage by following yaml files. I have omitted some config maps and so it will not work as it is. Could you please try to check what is different there:
apiVersion: apps/v1
kind: StatefulSet
metadata:
generation: 1
labels:
state: cassandra
name: cassandra
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: cassandra
serviceName: cassandra
template:
metadata:
annotations:
pod.alpha.kubernetes.io/initialized: "true"
creationTimestamp: null
labels:
app: cassandra
spec:
containers:
- args:
- chmod -R 777 /logs/; /on_start.sh
command:
- /bin/sh
- -c
image: <image>
imagePullPolicy: Always
name: cassandra
ports:
- containerPort: 9042
protocol: TCP
resources: {}
volumeMounts:
- mountPath: /data
name: data
imagePullSecrets:
- name: gcr-imagepull-json-key
volumeClaimTemplates:
- metadata:
creationTimestamp: null
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: local-storage
PV.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
labels:
type: local
name: cassandra-data-vol-0
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: data-cassandra-0
namespace: default
local:
path: /data/cassandra-0
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- ip-10-0-1-91.ec2.internal
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
Make sure /data/cassandra-0 exists before you create PV. Let me know if you face any issues.