NFS based mount fails in kubernetes - kubernetes

Question

I'm using Kubernetes - v1.24.7 on Ubuntu 18.04.6 LTS and facing problem with the NFS - Persistent Volume mount. When i tried to deploy my Jenkins deployment file it always fails with below errors.
$ kubectl describe pod jenkins-6786789d5d-m26zw -n jenkins
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 25m default-scheduler Successfully assigned jenkins/jenkins-6786789d5d-m26zw to worker-3
Warning FailedMount 5m31s (x2 over 14m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-access-65npd data]: timed out waiting for the condition
Warning FailedMount 3m17s (x8 over 23m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-65npd]: timed out waiting for the condition
Warning FailedMount 3m6s (x19 over 25m) kubelet MountVolume.SetUp failed for volume "pv-nfs" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs -o nfsvers=4.1 192.168.72.136:/mnt/nfs/stg/jenkins /var/lib/kubelet/pods/853c44ed-bf2b-4e6a-b666-c1adab7f7f4b/volumes/kubernetes.io~nfs/pv-nfs
Output: mount.nfs: mounting 192.168.72.136:/mnt/nfs/stg/jenkins failed, reason given by server: No such file or directory
The below External NFS mount path provided by our IT-Storage Administrator.
192.168.72.136:/nfs-volume
The below packages have already been installed on master and nodes.
apt install nfs-common
apt install cifs-utils
apt install nfs-kernel-server
In my master and workers(Host Machine) i have added below in /etc/fstab and i could mount the nfs volume.
192.168.72.136:/nfs-volume /mnt/nfs/stg/ nfs defaults 0 0
However still same problem persisting while Kubernetes application deployment, Also tried with below option in /etc/fstab but same result.
192.168.72.136:/nfs-volume /mnt/nfs/stg/ nfs rw,hard,intr 0 0
My pv & pvc volume status.
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv-nfs 100Gi RWX Retain Bound jenkins/pvc-nfs nfs 11s
$ kubectl get pvc -n jenkins
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-nfs Bound pv-nfs 100Gi RWX nfs 17s
My PersistentVolume and Deployment yml as follows.
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-nfs
labels:
type: pv-nfs
spec:
storageClassName: nfs
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
mountOptions:
- nfsvers=4.0
nfs:
server: 192.168.72.136
path: "/mnt/nfs/stg/jenkins"
readOnly: false
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: jenkins
labels:
app: jenkins
spec:
replicas: 1
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
securityContext:
fsGroup: 0
runAsUser: 0
serviceAccountName: admin
containers:
- name: jenkins
image: jenkins/jenkins:latest
securityContext:
privileged: true
runAsUser: 0
ports:
- containerPort: 8080
volumeMounts:
- name: data
mountPath: /var/jenkins_home
volumes:
- name: data
persistentVolumeClaim:
claimName: pvc-nfs
Directory /mnt/nfs/stg/jenkins existing in NFS. Please let me know what I'm missing here?
Thanks for helping.

Answer 1

When the storage IT administrator has exported NFS share: /nfs-volume from 192.168.72.136, then in the PersistentVolume spec, the path should be /nfs-volume.