Is NFS hard- or soft-mounted when making a Pod in Kubernetes with an NFS-volume?
As I understand this might have an impact on how it handles a timeout?
Example yaml:
apiVersion: v1
kind: Pod
metadata:
name: nfs-web
spec:
containers:
- name: web
image: nginx
ports:
- name: web
containerPort: 80
volumeMounts:
- name: nfs
mountPath: "/usr/share/nginx/html"
volumes:
- name: nfs
nfs:
server: nfs-server.default.kube.local
path: "/"
I believe that the NFS mounts inside of a POD will use the defaults provided by the implementation of NFS in the container OS. I can't be 100% certain (I'm not deeply familiar with the code), but in my experience the NFS mounts are mounted with the hard option, which is default in most implementations of NFS (see man nfs for more details on your OS; soft is often considered dangerous.)
The NFSVolumeSource struct doesn't appear to have the ability to know about mount settings (except read-only) and I don't see any hard-coded options in the NFS volume code.
You can check on your own PODs with something like this to gather the NFS options in use:
$ kubectl exec nfs-web-<XXXXX> -c web -- mount|grep nfs
its always "hard" mounted by default...however you can change it to soft by explicitly passing the additional annotation in PV definition.
volume.beta.kubernetes.io/mount-options: soft
Related
I'm pretty stuck with this learning step of Kubernetes named PV and PVC.
What I'm trying to do here is understand how to handle shared read-write volume on multiple pods.
What I understood here is that a PVC cannot be shared between pods unless a NFS-like storage class has been configured.
I'm still with my hostPath Storage Class and I tried the following (Docker Desktop and 3 nodes microK8s cluster) :
This PVC with dynamic Host Path provisionning
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-desktop
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
Deployment with 3 replicated pods writing on the same PVC.
apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox
spec:
replicas: 3
selector:
matchLabels:
app: busybox
template:
metadata:
labels:
app: busybox
spec:
containers:
- name: busybox
image: library/busybox:stable
command: ["/bin/sh"]
args:
["-c", 'while true; do echo "1: $(hostname)" >> /root/index.html; sleep 2; done;',]
volumeMounts:
- mountPath: /root
name: vol-desktop
volumes:
- name: vol-desktop
persistentVolumeClaim:
claimName: pvc-desktop
Nginx server for serving volume content
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:stable
volumeMounts:
- mountPath: /usr/share/nginx/html
name: vol-desktop
ports:
- containerPort: 80
volumes:
- name: vol-desktop
persistentVolumeClaim:
claimName: pvc-desktop
Following what I understood on the documentation, this could not be possible, but in reality everything run pretty smoothly and my Nginx server displayed the up to date index.html file pretty well.
It actually worked on a single-node cluster and multi-node cluster.
What am I not getting here? Why this thing works?
Is every pod mounting is own host path volume on start?
How can a hostPath storage works between multiple nodes?
EDIT: For the multi-node case, a network folder has been created between the same storage path of each machine this is why everything has been replicated successfully. I didn't understand that the same host path is created on each node with that PVC mounted.
To anyone with the same problem: each node mounting this hostpath PVC will have is own folder created at the PV path.
So without network replication between nodes, only pods of the same node will share the same folder.
This is why it's discouraged on a multi-node cluster due to the unpredictable location of a pod on the cluster.
Thanks!
how to handle shared read-write volume on multiple pods.
Redesign your application to avoid it. It tends to be fragile and difficult to manage multiple writers safely; you depend on both your application correctly performing things like file locking, the underlying shared filesystem implementation handling things properly, and the system being tolerant of any sort of network hiccup that might happen.
The example you give is something that frequently appears in Docker Compose setups: have an application with a mix of backend code and static files, and then try to publish the static files at runtime through a volume to a reverse proxy. Instead, you can build an image that copies the static files at build time:
FROM nginx
ARG app_version=latest
COPY --from=my/app:${app_version} /app/static /usr/share/nginx/html
Have your CI system build this and push it immediately after the backend image is built. The resulting image serves the corresponding static files, but doesn't require a shared volume or any manual management of the volume contents.
For other types of content, consider storing data in a database, or use an object-storage service that maintains its own backing store and can handle the concurrency considerations. Then most of your pods can be totally stateless, and you can manage the data separately (maybe even outside Kubernetes).
How can a hostPath storage works between multiple nodes?
It doesn't. It's an instruction to Kubernetes, on whichever node the pod happens to be scheduled on, to mount that host directory into the container. There's no management of any sort of the directory content; if two pods get scheduled on the same node, they'll share the directory, and if not, they won't; and if your pod's Deployment is updated and the pod is deleted and recreated somewhere else, it might not be the same node and might not have the same data.
With some very specific exceptions you shouldn't use hostPath volumes at all. The exceptions are things like log collectors run as DaemonSets, where there is exactly one pod on every node and you're interested in picking up the host-directory content that is different on each node.
In your specific setup either you're getting lucky with where the data producers and consumers are getting colocated, or there's something about your MicroK8s setup that's causing the host directories to be shared. It is not in general reliable storage.
I want to share my non-empty local directory with kind cluster.
Based on answer here: How to reference a local volume in Kind (kubernetes in docker)
I tried few variations of the following:
Kind Cluster yaml:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraMounts:
- hostPath: /Users/xyz/documents/k8_automation/data/manual/
containerPath: /host_manual
extraPortMappings:
- containerPort: 30000
hostPort: 10000
Pod yaml:
apiVersion: v1
kind: Pod
metadata:
name: manual
spec:
serviceAccountName: manual-sa
containers:
- name: tools
image: tools:latest
imagePullPolicy: Never
command:
- bash
tty: true
volumeMounts:
- mountPath: /home/jenkins/agent/data
name: data
volumes:
- name: data
hostPath:
path: /host_manual
type: Directory
---
I see that the directory /home/jenkins/agent/data does exist when the pod gets created. However, the folder is empty.
kinds documentation here: https://kind.sigs.k8s.io/docs/user/configuration/#extra-mounts
It should be the case that whatever is in the local machine at hostpath (/Users/xyz/documents/k8_automation/data/manual/) in extraMounts in the cluster yaml be available to the node at containerPath (/host_manual), which then gets mounted at container volume mounthPath (/home/jenkins/agent/data).
I should add that even if I change the hostPath in the cluster yaml file to a non-existent folder, the empty "data" folder still gets mounted in the container, so I think it's the connection from my local to kind cluster that's the issue.
Why am I not getting the contents of /Users/xyz/documents/k8_automation/data/manual/ with it's many files also available at /home/jenkins/agent/data in the container?
How can I fix this?
Any alternatives if there is no fix?
Turns out these yaml configuration was just fine.
The reason the directory was not showing up in the container was related with docker settings. And because "kind is a tool for running local Kubernetes clusters using Docker container “nodes”", it matters.
It seems docker restricts resource sharing and allows only specific directories to be bind mounted into Docker containers by default. Once I added the specific directory I wanted to show up in the container to the list of directories under Preferences -> Resources -> File sharing, it worked!
I'm trying to monitor Kubernetes PVC disk usage. I need the memory that is in use for Persistent Volume Claim. I found the command:
kubectl get --raw / api / v1 / persistentvolumeclaims
Return:
"status":{
"phase":"Bound",
"accessModes":[
"ReadWriteOnce"
],
"capacity":{
"storage":"1Gi"
}
}
But it only brings me the full capacity of the disk, and as I said I need the used one
Does anyone know which command could return this information to me?
I don't have a definitive anwser, but I hope this will help you. Also, I would be interested if someone has a better anwser.
Get current usage
The PersistentVolume subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed.
-- Persistent Volume | Kubernetes
As stated in the Kubernetes documentation, PV (PersistentVolume) and PVC (PersistentVolumeClaim) are abstractions over storage. As such, I do not think you can inspect PV or PVC, but you can inspect the storage medium.
To get the usage, create a debugging pod which will use your PVC, from which you will check the usage. This should work depending on your storage provider.
# volume-size-debugger.yaml
kind: Pod
apiVersion: v1
metadata:
name: volume-size-debugger
spec:
volumes:
- name: debug-pv
persistentVolumeClaim:
claimName: <pvc-name>
containers:
- name: debugger
image: busybox
command: ["sleep", "3600"]
volumeMounts:
- mountPath: "/data"
name: debug-pv
Apply the above manifest with kubectl apply -f volume-size-debugger.yaml, and run a shell inside it with kubectl exec -it volume-size-debugger sh. Inside the shell run du -sh to get the usage in a human readable format.
Monitoring
As I am sure you have noticed, this is not especially useful for monitoring. It may be useful for a one-time check from time to time, but not for monitoring or low disk space alerts.
One way to setup monitoring would be to have a similar sidecar pod like ours above and gather our metrics from there. One such example seems to be the node_exporter.
Another way would be to use CSI (Container Storage Interface). I have not used CSI and do not know enough about it to really explain more. But here are a couple of related issues and related Kubernetes documentation:
Monitoring Kubernetes PersistentVolumes - prometheus-operator
Volume stats missing - csi-digitalocean
Storage Capacity | Kubernetes
+1 to touchmarine's answer however I'd like to expand it a bit and add also my three cents.
But it only brings me the full capacity of the disk, and as I said I
need the used one
PVC is an abstraction which represents a request for a storage and simply doesn't store such information as disk usage. As a higher level abstraction it doesn't care at all how the underlying storage is used by its consumer.
#touchmarine, Instead of using a Pod whose only function is to sleep and every time you need to check the disk usage you need to attach to it maually, I would propose to use something like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
volumes:
- name: media
persistentVolumeClaim:
claimName: media
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/data"
name: media
- name: busybox
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do du -sh /data; sleep 10;done"]
volumeMounts:
- mountPath: "/data"
name: media
It can be of course a single-container busybox Pod as in #touchmarine's example but here I decided to to show also how it can be used as a sidecar running next to nginx container within a single Pod.
As it runs a simple bash script - an infinite while loop, which prints out current disk usage to the standard output it can be read with kubectl logs without a need of using kubectl exec and attaching to the Pod:
$ kubectl logs nginx-deployment-56bb5c87f6-dqs5h busybox
20.0K /data
20.0K /data
20.0K /data
I guess it can be also used more effectively to configure some sort of monitoring of disk usage.
I'm currently mounting a gcePersistentDisk to each pod in my kubernetes deployment. Since I want multiple pods to read from the disk, I have to mount it as read only. My deployment yaml file looks like this:
apiVersion: extensions/v1beta1
kind: Deployment
spec:
replicas: 1
...
...
template:
...
...
spec:
containers:
- image: ...
...
...
volumeMounts:
- mountPath: /my-volume
name: my-volume
readOnly: true
...
...
volumes:
- name: my-storage
gcePersistentDisk:
pdName: my-disk
fsType: ext4
readOnly: true
Right now, in order to write new stuff to the disk, I need to scale the deployment to 0, then start a kubernetes job that mounts the disk to a single pod that has read / write access, write to the disk and then scale the deployment up again.
Is there a way I can do this without taking down all my pods?
Is it possible/recommended to do something like "hot-swapping" persistent disks in kubernetes deployments?
Looking at the requirements:
1)- No other choice with the current use-case. Pods need to be scaled down every time.
2)- You can use a different type of PV, then use ReadWriteMany access mode [1] & [2].
3)- hot-swap: meaning changing the deployment (kubectl apply)? Not sure, need clarification.
4)- Another option is to use NFS [2], but that obviously is a whole different approach.
[1] https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes#access_modes
[2] Access Modes https://kubernetes.io/docs/concepts/storage/persistent-volumes/
Using Kubernetes on bare metal and trying to figure out how to mount a external bloc storage volume from an OpenStack cloud provider.
I understand I need to use the Cinder plugin.
https://github.com/kubernetes/kubernetes/tree/master/pkg/volume/cinder
I modified an example I found to build a test pod, the volume is simply defined as the following, in the pod definition:
apiVersion: v1
kind: Pod
metadata:
name: test
labels:
name: test
spec:
containers:
- image: busybox
name: busybox
command:
- "sleep"
- "3600"
volumeMounts:
- name: persistent-storage
mountPath: /var/lib/storage
volumes:
- name: persistent-storage
cinder:
volumeID: bd82f7e2-wece-4c01-a505-4acf60b07f4a
fsType: ext4
I have a volumeID I got from the OpenStack volume API.
I put it there, but I am not sure the volume is actually being mounted:
I am not sure how to check actually, but I would guess that df -h would show a remote volume being mounted on the host and in the container, but I don't see any.
I would think Kubernetes would send me an error if the volume was not mounted, the pod would fail or something... but it runs.
So, the question is: how do I verify the volume is mounted? and as I believe it is not mounted, what should I do to make this cinder plugin work?
The conclusion of my search on this was that the nodes using the block storage also need to be on the same OpenStack cluster.
That is, it is not (easily/standard) possible to mount Cinder block storage into a cluster of nodes that is not on the Open Stack cluster.
See:
Kubernetes: using OpenStack Cinder from one cloud provider while nodes on another