Error while running the MongoDB as a Statefulset set in Kubernetes - mongodb

Question

I am trying to run the mongodb as a statefulset in the minikube Kubernetes cluster. I have 3 replicas but I have the following problem - which is, one replica (mongo-0) is up and running without any issue but the second replica (mongo-1) is forever in the pending state. I tried to describe the pod and I get the following output:
kubectl describe pod mongo-1 -n ng-mongo
. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 17m (x70 over 6h9m) default-scheduler 0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
As per the above error, it says it cannot find the persistent volume, but there is one already.
Please find my YAML definitions for this:
apiVersion: v1
kind: Service
metadata:
name: mongodb-service
labels:
name: mongo
spec:
ports:
- port: 27017
targetPort: 27017
clusterIP: None
selector:
role: mongo
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local-storage
namespace: ng-mongo
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-pv
namespace: ng-mongo
spec:
capacity:
storage: 10Gi
# volumeMode field requires BlockVolume Alpha feature gate to be enabled.
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /tmp
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- minikube
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: local-claim
namespace: ng-mongo
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
storageClassName: local-storage
---
apiVersion: v1
kind: Service
metadata:
name: mongo
namespace: ng-mongo
labels:
name: mongo
spec:
ports:
- port: 27017
targetPort: 27017
clusterIP: None
selector:
role: mongo
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
namespace: ng-mongo
spec:
serviceName: "mongo"
replicas: 3
selector:
matchLabels:
role: mongo
template:
metadata:
labels:
role: mongo
environment: test
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongo
image: mongo
command:
- mongod
- "--bind_ip"
- "0.0.0.0"
- "--replSet"
- rs0
resources:
requests:
cpu: 0.2
memory: 200Mi
ports:
- containerPort: 27017
volumeMounts:
- name: localvolume
mountPath: /data/db
- name: mongo-sidecar
image: cvallance/mongo-k8s-sidecar
env:
- name: MONGO_SIDECAR_POD_LABELS
value: "role=mongo,environment=test"
# volumes:
# - name: localvolume
# persistentVolumeClaim:
# claimName: local-claim
volumeClaimTemplates:
- metadata:
name: localvolume
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "local-storage"
resources:
requests:
storage: 2Gi
Can someone help me find the issue here?

Answer 1

You are using the node affinity while creating the PV, that need to be configured correctly:
It will inform Kubernetes my disk will attach to this type of node. Due to affinity your PV is attached to one type of specific node only.
when you are deploying the deployment it's not getting scheduled on that specific node and your POD is not getting that PV or PVC.
If you are adding node affinity to PVC add it to deployment also. So both PVC and pod get scheduled on the same node.
Resolution steps:
Make sure both deployment and pvc schedule with the same node add the node affinity to deployment also so deployment schedule on the respective node.
or else
Remove the node affinity rule from PV and create a new PV and PVC and use it.
here is the place where you have mentioned the node affinity rule
Note: In your node affinity you have mentioned as minikube, verify the node by
kubectl get nodes make changes if required.