Does Kubernetes mount an emtpyDir volume on the host? - kubernetes

Kubernetes features quite a few types of volumes, including emptyDir:
An emptyDir volume is first created when a Pod is assigned to a Node, and exists as long as that Pod is running on that node. As the name says, it is initially empty. Containers in the pod can all read and write the same files in the emptyDir volume, though that volume can be mounted at the same or different paths in each container. When a Pod is removed from a node for any reason, the data in the emptyDir is deleted forever.
...
By default, emptyDir volumes are stored on whatever medium is backing the node.
Is the emtpyDir actually mounted on the node, and accessible to a container outside the pod, or to the node FS itself?

Yes it is also accessible on the node. It is bind mounted into the container (sort of). The source directories are under /var/lib/kubelet/pods/PODUID/volumes/kubernetes.io~empty-dir/VOLUMENAME
You can find the location on the host like this:
sudo ls -l /var/lib/kubelet/pods/`kubectl get pod -n mynamespace mypod -o 'jsonpath={.metadata.uid}'`/volumes/kubernetes.io~empty-dir

You can list all emptyDir volumes on the host using this command
df
To view only volumes mapped to a specific volume
df | grep -i cache-volume
where cache-volume is the volume name in your pod definition
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /cache
name: cache-volume
volumes:
- name: cache-volume
emptyDir: {}

Related

Kubernetes PV/PVC not mounting to the correct volumeMounts in the pods

I'm running into a weird issue with my newer deployments where volumes aren't mounting correctly.
Example..
There are PV/PVCs for three NFS directories that relate to one deployment:
NFS/in
NFS/out
NFS/config
In the deployment, those PVCs mount to the corresponding volumeMounts
volumeMounts/in
volumeMounts/out
volumeMounts/config
With my older deployments, this works as expected. With the new deployments, the NFS directories are mounting to the incorrect mount points... The contents of NFS/in are mounted in volumeMounts/config. NFS/config are mounted in volumeMounts/in.
This is vanilla Kubernetes on a bare metal node. The only configuration change from default that has been made was yanking PVC protection due to PVCs not being deleted on request:
kubectl patch pvc PVC_NAME -p '{"metadata":{"finalizers": []}}' --type=merge
Any ideas on what causes the directories to mount in the incorrect volumeMounts?
You have to set ClaimName though your deployment or statefulset :
...
apiVersion: apps/v1
kind: StatefulSet
.......
containers:
- name: container-name
image: container-image:container-tag
volumeMounts:
- name: claim1
mountPath: /path/to/directory
volumes:
- name: claim1
persistentVolumeClaim:
claimName: PVC_NAME
...

K8s mongodb container cannot use the EBS volume mount

I am having the below Pod definition.
apiVersion: v1
kind: Pod
metadata:
name: mongodb
spec:
volumes:
- name: mongodb-data
awsElasticBlockStore:
volumeID: vol-0c0d9800c22f8c563
fsType: ext4
containers:
- image: mongo
name: mongodb
volumeMounts:
- name: mongodb-data
mountPath: /data/db
ports:
- containerPort: 27017
protocol: TCP
I have created volumne in AWS and tried to mount to the container. The container is not starting.
kubectl get po
NAME READY STATUS RESTARTS AGE
mongodb 0/1 ContainerCreating 0 6m57s
When I created the volume and assigned it to a Availability zone where the node is running and and the pod was scheduled on that node, the volume was mounted successfully. If the pod is not scheduled on the node, the mount fails. How can I make sure that the volume can be accessed by all the nodes
According to the documentation:
There are some restrictions when using an awsElasticBlockStore volume:
the nodes on which Pods are running must be AWS EC2 instances
those instances need to be in the same region and availability-zone as the EBS volume
EBS only supports a single EC2 instance mounting a volume
Make sure all of the above are met. If your nodes are in different zones than you might need to create additional EBS volumes, for example:
aws ec2 create-volume --availability-zone=eu-west-1a --size=10 --volume-type=gp2
Please let me know if that helped.

Volume is already exclusively attached to one node and can't be attached to another

I have a pretty simple Kubernetes pod. I want a stateful set and want the following process:
I want to have an initcontainer download and uncompress a tarball from s3 into a volume mounted to the initcontainer
I want to mount that volume to my main container to be used
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: app
namespace: test
labels:
name: app
spec:
serviceName: app
replicas: 1
selector:
matchLabels:
app: app
template:
metadata:
labels:
app: app
spec:
initContainers:
- name: preparing
image: alpine:3.8
imagePullPolicy: IfNotPresent
command:
- "sh"
- "-c"
- |
echo "Downloading data"
wget https://s3.amazonaws.com/.........
tar -xvzf xxxx-........ -C /root/
volumeMounts:
- name: node-volume
mountPath: /root/data/
containers:
- name: main-container
image: ecr.us-west-2.amazonaws.com/image/:latest
imagePullPolicy: Always
volumeMounts:
- name: node-volume
mountPath: /root/data/
volumeClaimTemplates:
- metadata:
name: node-volume
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: gp2-b
resources:
requests:
storage: 80Gi
I continue to get the following error:
At first I run this and I can see the logs flowing of my tarball being downloaded by the initcontainer. About half way done it terminates and gives me the following error:
Multi-Attach error for volume "pvc-faedc8" Volume is
already exclusively attached to one node and can't be
attached to another
Looks like you have a dangling PVC and/or PV that is attached to one of your nodes. You can ssh into the node and run a df or mount to check.
If you look at this the PVCs in a StatefulSet are always mapped to their pod names, so it may be possible that you still have a dangling pod(?)
If you have a dangling pod:
$ kubectl -n test delete pod <pod-name>
You may have to force it:
$ kubectl -n test delete pod <pod-name> --grace-period=0 --force
Then, you can try deleting the PVC and it's corresponding PV:
$ kubectl delete pvc pvc-faedc8
$ kubectl delete pv <pv-name>
I had the same issue right now and the problem was, that the node on which the pod is usually running on was down and another one took over (which didn't work as expected for whatever reason). Had the "node down" scenario a few times before already and it never caused any issues. Couldn't get the StatefulSet and Deployment back up and running without booting the node that was down. But as soon as the node was up and running again the StatefulSet and Deployment immediately came back to life as well.
I had a similar error:
The volume pvc-2885ea01-f4fb-11eb-9528-00505698bd8b
cannot be attached to the node node1 since it is already attached to the node node2*
I use longhorn as a storage provisioner and manager. So I just detached this pv in the error and restarted the stateful set. It automatically was able to attach to the pv correctly this time.
I'll add an answer that will prevent this from happening again.
Short answer
Access modes: Switch from ReadWriteOnce to ReadWriteMany.
In a bit more details
You're usng a StatefulSet where each replica has its own state, with a unique persistent volume claim (PVC) created for each pod.
Each PVC is referring to a Persistent Volume where you decided that the access mode is ReadWriteOnce.
Which as you can see from here:
ReadWriteOnce
the volume can be mounted as read-write by a single
node. ReadWriteOnce access mode still can allow multiple pods to
access the volume when the pods are running on the same node.
So in case K8S Scheduler (due to priorities or resource calculations or due to a Cluster autoscaler which decided to shift the pod to a different node) - you will receive an error that the volume is already exclusively attached to one node and can't be
attached to another node.
Please consider using ReadWriteMany where the volume can be mounted as read-write by many nodes.

Creating a pod/container in kubernetes - how to copy a bunch of files into it

Sorry if this is a noob question:
I am creating a pod in a kubernetes cluster using a pod defintion yaml file.
This pod defines just one container. I'd like to ... copy a few files to a particular directory in the container.
sort of like in docker-compose:
volumes:
- ./testHelpers/certs:/var/private/ssl/certs
Is it possible to do that at this point (point of defining the pod?)
If not, what could my alternatives be?
PS - I understand that the sample from docker-compose is very different since this maps local directory to a directory in container
It's better to use volumes in pod definition.
Initialize the pod before the container runs
Apart from this, you can also use ConfigMap to store certs and other config files you needed and than can access them in the container as volumes.
More details here
You should create a config map, you can do it from files or a directory.
kubectl create configmap my-config --from-file=configuration/
And then mount the config map as directory:
apiVersion: v1
kind: Pod
metadata:
name: configmap-pod
spec:
containers:
- name: test
image: busybox
volumeMounts:
- name: my-config
mountPath: /etc/config
volumes:
- name: my-config
configMap:
name: my-config

How to use the cinder plugin in Kubernetes

Using Kubernetes on bare metal and trying to figure out how to mount a external bloc storage volume from an OpenStack cloud provider.
I understand I need to use the Cinder plugin.
https://github.com/kubernetes/kubernetes/tree/master/pkg/volume/cinder
I modified an example I found to build a test pod, the volume is simply defined as the following, in the pod definition:
apiVersion: v1
kind: Pod
metadata:
name: test
labels:
name: test
spec:
containers:
- image: busybox
name: busybox
command:
- "sleep"
- "3600"
volumeMounts:
- name: persistent-storage
mountPath: /var/lib/storage
volumes:
- name: persistent-storage
cinder:
volumeID: bd82f7e2-wece-4c01-a505-4acf60b07f4a
fsType: ext4
I have a volumeID I got from the OpenStack volume API.
I put it there, but I am not sure the volume is actually being mounted:
I am not sure how to check actually, but I would guess that df -h would show a remote volume being mounted on the host and in the container, but I don't see any.
I would think Kubernetes would send me an error if the volume was not mounted, the pod would fail or something... but it runs.
So, the question is: how do I verify the volume is mounted? and as I believe it is not mounted, what should I do to make this cinder plugin work?
The conclusion of my search on this was that the nodes using the block storage also need to be on the same OpenStack cluster.
That is, it is not (easily/standard) possible to mount Cinder block storage into a cluster of nodes that is not on the Open Stack cluster.
See:
Kubernetes: using OpenStack Cinder from one cloud provider while nodes on another