Need support on using git-sync and persistent volumes - kubernetes

I am trying to use git-sync to write data from a gitlab repo to a persistent volume, then pull that data into another pod (trim_load) and perform a job. Here are the manifests I have set up. I am new to this and developing locally, and I could use all the direction I can get!
I am getting an error that the directory doesn't exist, but it does on my local machine, but not on the kind cluster that I am using. How do I create a directory on the kind cluster?
apiVersion: v1
kind: PersistentVolume
metadata:
name: dbt-pv
spec:
capacity:
storage: 2Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /Users/my_user/k8s/pv1
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- kind-control-plane
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dbt-pvc
spec:
storageClassName: local-storage
accessModes:
- ReadWriteMany
volumeName: dbt-pv
resources:
requests:
storage: 1Gi
apiVersion: v1
kind: Pod
metadata:
name: gitsync-sidecar
spec:
containers:
- name: git-sync
# This container pulls git data and publishes it into volume
# "content-from-git". In that volume you will find a symlink
# "current" (see -dest below) which points to a checked-out copy of
# the master branch (see -branch) of the repo (see -repo).
# NOTE: git-sync already runs as non-root.
image: k8s.gcr.io/git-sync/git-sync:v3.3.4
args:
- --repo= <the git repo I wanna copy HTTPS link>
- --branch=master
- --depth=1
- --period=60
- --link=current
- --root=/git # I don't know what this means
volumeMounts:
- name: dbt-pv
mountPath: /git # I don't know what this means
volumes:
- name: dbt-pv
persistentVolumeClaim:
claimName: dbt-pvc
apiVersion: v1
kind: Pod
metadata:
name: trim-pod
spec:
containers:
- name: trim-pod-cont
image: <my docker container to run the code>
volumeMounts:
- name: dbt-pv
mountPath: /tmp/dbt
volumes:
- name: dbt-pv
persistentVolumeClaim:
claimName: dbt-pvc

I'm not familiar with kind, but from what i gather on their website, it works like minikube. Which means your cluster is inside a container itself.
The folder on your local machine exists, but your volume is looking for a folder inside the host machine, which is the container running your cluster.
You have to open a shell in your cluster container and create the folder there.
If you want to access this folder on your local machine, you then have to mount your local folder in your kind cluster.

Related

Why local persistent volumes not visible in EKS?

In order to test if I can get self written software deployed in amazon using docker images,
I have a test eks cluster.
I have written a small test script that reads and writes a file to see if I understand how to deploy. I have successfully deployed it in minikube, using three replica's. The replica's all use a shared directory on my local file system, and in minikube that is mounted into the pods with a volume
The next step was to deploy that in the eks cluster. However, I cannot get it working in eks. The problem is that the pods don't see the contents of the mounted directory.
This does not completely surprise me, since in minikube I had to create a mount first to a local directory on the server. I have not done something similar on the eks server.
My question is what I should do to make this working (if possible at all).
I use this yaml file to create a pod in eks:
apiVersion: v1
kind: PersistentVolume
metadata:
name: "pv-volume"
spec:
storageClassName: local-storage
capacity:
storage: "1Gi"
accessModes:
- "ReadWriteOnce"
hostPath:
path: /data/k8s
type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: "pv-claim"
spec:
storageClassName: local-storage
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "500M"
---
apiVersion: v1
kind: Pod
metadata:
name: ruudtest
spec:
containers:
- name: ruud
image: MYIMAGE
volumeMounts:
- name: cmount
mountPath: "/config"
volumes:
- name: cmount
persistentVolumeClaim:
claimName: pv-claim
So what I expect is that I have a local directory, /data/k8s, that is visible in the pods as path /config.
When I apply this yaml, I get a pod that gives an error message that makes clear the data in the /data/k8s directory is not visible to the pod.
Kubectl gives me this info after creation of the volume and claim
[rdgon#NL013-PPDAPP015 probeer]$ kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pv-volume 1Gi RWO Retain Available 15s
persistentvolume/pvc-156edfef-d272-4df6-ae16-09b12e1c2f03 1Gi RWO Delete Bound default/pv-claim gp2 9s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/pv-claim Bound pvc-156edfef-d272-4df6-ae16-09b12e1c2f03 1Gi RWO gp2 15s
Which seems to indicate everything is OK. But it seems that the filesystem of the master node, on which I run the yaml file to create the volume, is not the location where the pods look when they access the /config dir.
On EKS, there's no storage class named 'local-storage' by default.
There is only a 'gp2' storage class, which is also used when you don't specify a storageClassName.
The 'gp2' storage class creates a dedicated EBS volume and attaches it your Kubernetes Node when required, so it doesn't use a local folder. You also don't need to create the pv manually, just the pvc:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: "pv-claim"
spec:
storageClassName: gp2
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "500M"
---
apiVersion: v1
kind: Pod
metadata:
name: ruudtest
spec:
containers:
- name: ruud
image: MYIMAGE
volumeMounts:
- name: cmount
mountPath: "/config"
volumes:
- name: cmount
persistentVolumeClaim:
claimName: pv-claim
If you want a folder on the Node itself, you can use a 'hostPath' volume, and you don't need a pv or pvc for that:
apiVersion: v1
kind: Pod
metadata:
name: ruudtest
spec:
containers:
- name: ruud
image: MYIMAGE
volumeMounts:
- name: cmount
mountPath: "/config"
volumes:
- name: cmount
hostPath:
path: /data/k8s
This is a bad idea, since the data will be lost if another node starts up, and your pod is moved to the new node.
If it's for configuration only, you can also use a configMap, and put the files directly in your kubernetes manifest files.
apiVersion: v1
kind: ConfigMap
metadata:
name: ruud-config
data:
ruud.properties: |
my ruud.properties file content...
---
apiVersion: v1
kind: Pod
metadata:
name: ruudtest
spec:
containers:
- name: ruud
image: MYIMAGE
volumeMounts:
- name: cmount
mountPath: "/config"
volumes:
- name: cmount
configMap:
name: ruud-config
Please check whether the pv got created and its "bound" to PVC by running below commands
kubectl get pv
kubectl get pvc
Which will give information whether the objects are created properly
The local path you refer to is not valid. Try:
apiVersion: v1
kind: Pod
metadata:
name: ruudtest
spec:
containers:
- name: ruud
image: MYIMAGE
volumeMounts:
- name: cmount
mountPath: /config
volumes:
- name: cmount
hostPath:
path: /data/k8s
type: DirectoryOrCreate # <-- You need this since the directory may not exist on the node.

How to use block device attached to host, as block device within the pod

Question: How can I use raw devices attached to the host within the pod as block device.
I tried using "hostPath" with type "BlockDevice"
volumes:
- my-data:
hostPath:
path: /dev/nvme1n2
type: BlockDevice
containers:
.....
volumeDevices:
- name: my-data
devicePath: /dev/sda
This configuration gives me the below error.
Invalid value: "my-data": can only use volume source type of PersistentVolumeClaim for block mode
Can I achieve this using PersistentVolume and PersistentVolumeClaim ? Can someone help me with an example config. Appreciate the help.
Support for Block devices in K8s allows user and admins to use PVs & PVCs for raw block devices to be mounted in Pods. Excerpts below show a small use-case.
Create a PV which refers the Raw device on host say /dev/xvdf.
kind: PersistentVolume
apiVersion: v1
metadata:
name: local-raw-pv
spec:
volumeMode: Block
capacity:
storage: 100Gi
local:
path: /dev/xvdf
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
Create a PVC claiming the block device for applications
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: block-pvc
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
resources:
requests:
storage: 10Gi
Create pod with the above claim which will mount the host device /dev/xvdf inside pod at path /dev/xvda
apiVersion: v1
kind: Pod
metadata:
name: pod-with-block-volume
spec:
containers:
- name: some-container
image: ubuntu
command: ["/bin/sh", "-c"]
args: [ "tail -f /dev/null" ]
volumeDevices:
- name: data
devicePath: /dev/xvda
volumes:
- name: data
persistentVolumeClaim:
claimName: block-pvc

How to have multiple pods access an existing NFS folder in Kubernetes?

I have a folder of TFRecords on a network that I want to expose to multiple pods. The folder has been exported via NFS.
I have tried creating a Persistent Volume, followed by a Persistent Volume Claim. However, that just creates a folder inside the NFS mount, which I don't want. Instead, I want to Pod to access the folder with the TFRecords.
I have listed the manifests for the PV and PVC.
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-tfrecord-pv
spec:
capacity:
storage: 30Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
nfs:
path: /media/veracrypt1/
server: 1.2.3.4
readOnly: false
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs-tfrecord-pvc
namespace: default
spec:
accessModes:
- ReadWriteMany
storageClassName: nfs-tfrecord
resources:
requests:
storage: 1Gi
I figured it out. The issue was I was looking at the problem the wrong way. I didn't need any provisioning. Instead, what was need was to simply mount the NFS volume within the container:
kind: Pod
apiVersion: v1
metadata:
name: pod-using-nfs
spec:
containers:
- name: app
image: alpine
volumeMounts:
- name: data
mountPath: /mnt/data
command: ["/bin/sh"]
args: ["-c", "sleep 500000"]
volumes:
- name: data
nfs:
server: 1.2.3.4
path: /media/foo/DATA

Nodemon takes a while to pick up nfs changes

This may be configurable in the nfs share or within nodemon, not sure...
Basically, for local dev of production k8s system using minikube with a persistent volume claim where the volume is a locally defined nfs share.
/etc/exports:
/path/to/nfs/share -alldirs -mapall=501:20 192.168.99.100
volume.yml:
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-volume
spec:
capacity:
storage: 15Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: standard
nfs:
server: 192.168.99.1
path: /path/to/nfs/share
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: cr-volume
spec:
storageClassName: standard
accessModes:
- ReadWriteMany
resources:
requests:
storage: 15Gi
deployment.yml:
...(relevant section to mounting nfs share into container)
containers:
- name: user
image: image_name
imagePullPolicy: Always
command:
- npm
args:
- run
- dev-start
ports:
- containerPort: 5600
- containerPort: 5898
volumeMounts: # mount host directory for local development
- mountPath: /usr/src/app
name: user-mount
subPath: backend/services/user
volumes:
- name: user-mount
persistentVolumeClaim:
claimName: cr-volume
...
The command here (npm run dev-start) is . ./config/development/export-env.sh && DEBUG=user:* node_modules/nodemon/bin/nodemon.js --inspect=0.0.0.0:5898 --legacy-watch -L ./bin/www as opposed to the prod npm start of node ./bin/www
So this all works and mimics a prod env locally and allows me to make changes on the local file system which make their way through to the containers and reload. My only issue is that nodemon generally won't pick up the changes to the file for between 30 to 60 seconds.
I don't know enough about nfs shares and how the daemon broadcasts changes or how nodemon with it's legacy-watch flag watches to know where to go from here, but I'd love to be able to fix this so changes are reloaded faster.

Setup with Kubernetes hostPath but file doesn't show up in container

I'm trying to set up hostPath to share a file between pods.
I'm following this guide Configure a Pod to Use a PersistentVolume for Storage.
Here are the configuration files for pv,pvc,pod.
PV:
kind: PersistentVolume
apiVersion: v1
metadata:
name: task-pv-volume
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/tmp/data"
PVC:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: task-pv-claim
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
POD:
kind: Pod
apiVersion: v1
metadata:
name: task-pv-pod
spec:
volumes:
- name: task-pv-storage
persistentVolumeClaim:
claimName: task-pv-claim
containers:
- name: task-pv-container
image: nginx
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: task-pv-storage
I'm adding a file to /tmp/data and but I can't see it in the container.
When I check the status of pv,pvc,pod, the result is as following:
Can some give me a clue on why I can't see the file? Any suggestion or command on how to debug this kind of issue is welcome.
MINIKUBE specific
I was getting the same error when running a local cluster on minikube for Mac.
Minikube actually creates a VM and then runs your containers on it. So the hostPath actually refers to paths inside that VM and not on your local machine. That is why all mounts show up as empty folders.
Solution:
Map your local path to minikube's VM by same name. That way you can refer it as is in you kubernetes Manifests.
minikube mount <source directory>:<target directory>
In this case:
minikube mount /tmp/data:/tmp/data
This should do the trick.
SOURCE: https://minikube.sigs.k8s.io/docs/handbook/mount/
I think I figure it out.
HostPath is only suitable for a one-node cluster. my cluster have 2 nodes. so the physical storage the PV use is on another computer.
when I first go through the documentation, I don't pay attention this:
"You need to have a Kubernetes cluster that has only one Node"
did you put a local file in /tmp/data in the first place ?