Binding statefulset to local persistent volumes - volume node affinity conflict error - kubernetes

Question

I have 3-node kubernetes, host names are host_1, host_2, host_3.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
host_1 Ready master 134d v1.10.1
host_2 Ready <none> 134d v1.10.1
host_3 Ready <none> 134d v1.10.1
I have defined 3 local persistent volumes of size 100M, mapped to a local directory on each node. I used the following descriptor 3 times where <hostname> is one of: host_1, host_2, host_3:
apiVersion: v1
kind: PersistentVolume
metadata:
name: test-volume-<hostname>
spec:
capacity:
storage: 100M
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /opt/jnetx/volumes/test-volume
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- <hostname>
After applying three such yamls, I have the following:
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
test-volume-host_1 100M RWO Delete Available local-storage 58m
test-volume-host_2 100M RWO Delete Available local-storage 58m
test-volume-host_3 100M RWO Delete Available local-storage 58m
Now, I have a very simple container that writes to a file. The file should be located on the local persistent volume. I deploy it as a statefulset with 1 replica and map volumes via statefulset's volumeClaimTemplates:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: filewriter
spec:
serviceName: filewriter
...
replicas: 1
template:
spec:
containers:
- name: filewriter
...
volumeMounts:
- mountPath: /test/data
name: fw-pv-claim
volumeClaimTemplates:
- metadata:
name: fw-pv-claim
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-storage
resources:
requests:
storage: 100M
The volume claim seems to have been created ok and bound to pv on the first host:
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
test-volume-host_1 100M RWO Delete Bound default/fw-pv-claim-filewriter-0 local-storage 1m
test-volume-host_2 100M RWO Delete Available local-storage 1h
test-volume-host_3 100M RWO Delete Available local-storage 1h
But, the pod hangs in Pending state:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
filewriter-0 0/1 Pending 0 4s
If we describe, we can see the following errors:
$ kubectl describe pod filewriter-0
Name: filewriter-0
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2s (x8 over 1m) default-scheduler 0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 node(s) had volume node affinity conflict.
Can you help me figure out what is wrong? Why can't it just create the pod?

Answer 1

It seems that the one node where the PV is available has a taint that your StatefulSet does not have toleration for.

Answer 2

I had a very similar case to the above and observed the same symptom (node affinity conflict). In my case the issue was that I had 2 volumes attached to 2 different nodes but was trying to use them within 1 pod.
I detected this by using kubectl describe pvc name-of-pvc and noting the selected-node annotation. Once I set the pod to use volumes that were both on one node, I no longer had issues.