I am working with kubernetes for a long time and I have a question that I am not sure what the answer is.
If I create a PersistentVolume and couple of PersistentVolumeClaims that uses that PersistentVolume.
I mapped different PersistentVolumeClaims to different pods/containers.
Is There a way that one pod could read data that the other pod/container written?
TL;DR: By mistake - no. By a chain of mistakes - yes.
First, PV Access Mode need to be set to ReadWriteMany.
Then, a specific Volume Plugin have to be used. While Kuberentes supports a lot of different volumes, only few support ReadWriteMany Access Mode.
Those volume types that do support ReadWriteMany can be found in this table (scroll down a bit).
As far as a container is concerned, PersistentVolume is just a volume mounted in container's filesystem. If you mount it to multiple pods, they will be able to read and write to the same files.
Related
I've been working with Kubernetes for quite a while, but still often got confused about Volume, PersistentVolume and PersistemtVolumeClaim. It would be nice if someone could briefly summarize the difference of them.
Volume - For a pod to reference a storage that is external , it needs volume spec. This volume can be from configmap, from secrets, from persistantvolumeclaim, from hostpath etc
PeristentVolume - It is representation of a storage that is made avaliable. The plugins for cloud provider enable to create this resource.
PeristentVolumeClaim - This claims specific resources and if the persistent volume is avaliable in namespaces match the claim requirement, then claim get tied to that Peristentvolume
At this point this PVC/PV aren't used. Then in Pod spec, pod makes use of claim as volumes and then the storage is attached to Pod
These are all in a Kubernetes application context. Too keep applications portable between different Kubernetes platforms, it is good to abstract away the infrastructure from the application. Here I will explain the Kubernetes objects that belongs to Application config and also to the Platform config. If your application runs on both e.g. GCP and AWS, you will need two sets of platform configs, one for GCP and one for AWS.
Application config
Volume
A pod may mount volumes. The source for volumes can be different things, e.g. a ConfigMap, Secret or a PersistentVolumeClaim
PersistentVolumeClaim
A PersistentVolumeClaim represents a claim of a specific PersistentVolume instance. For portability this claim can be for a specific StorageClass, e.g. SSD.
Platform config
StorageClass
A StorageClass represents PersistentVolume type with specific properties. It can be e.g. SSD. But the StorageClass is different on each platform, e.g. one definition on AWS, Azure, another on GCP or on Minikube.
PersistentVolume
This is a specific volume on the platform. And it may be different on platforms, e.g. awsElasticBlockStore or gcePersistentDisk. This is the instance that holds the actual data.
Minikube example
See Configure a Pod to Use a PersistentVolume for Storage for a full example on how to use PersistentVolume, StorageClass and Volume for a Pod using Minikube and a hostPath.
As per this official document, Kubernetes Persistent Volumes support three types of access modes.
ReadOnlyMany
ReadWriteOnce
ReadWriteMany
The given definitions of them in the document is very high-level. It would be great if someone can explain them in little more detail along with some examples of different use cases where we should use one vs other.
You should use ReadWriteX when you plan to have Pods that will need to write to the volume, and not only read data from the volume.
You should use XMany when you want the ability for Pods to access the given volume while those workloads are running on different nodes in the Kubernetes cluster. These Pods may be multiple replicas belonging to a Deployment, or may be completely different Pods. There are many cases where it's desirable to have Pods running on different nodes, for instance if you have multiple Pod replicas for a single Deployment, then having them run on different nodes can help ensure some amount of continued availability even if one of the nodes fails or is being updated.
If you don't use XMany, but you do have multiple Pods that need access to the given volume, that will force Kubernetes to schedule all those Pods to run on whatever node the volume gets mounted to first, which could overload that node if there are too many such pods, and can impact the availability of Deployments whose Pods need access to that volume as explained in the previous paragraph.
So putting all that together:
If you need to write to the volume, and you may have multiple Pods needing to write to the volume where you'd prefer the flexibility of those Pods being scheduled to different nodes, and ReadWriteMany is an option given the volume plugin for your K8s cluster, use ReadWriteMany.
If you need to write to the volume but either you don't have the requirement that multiple pods should be able to write to it, or ReadWriteMany simply isn't an available option for you, use ReadWriteOnce.
If you only need to read from the volume, and you may have multiple Pods needing to read from the volume where you'd prefer the flexibility of those Pods being scheduled to different nodes, and ReadOnlyMany is an option given the volume plugin for your K8s cluster, use ReadOnlyMany.
If you only need to read from the volume but either you don't have the requirement that multiple pods should be able to read from it, or ReadOnlyMany simply isn't an available option for you, use ReadWriteOnce. In this case, you want the volume to be read-only but the limitations of your volume plugin have forced you to choose ReadWriteOnce (there's no ReadOnlyOnce option). As a good practice, consider the containers.volumeMounts.readOnly setting to true in your Pod specs for volume mounts corresponding to volumes that are intended to be read-only.
ReadOnlyMany – the volume can be mounted read-only by many nodes
By this method, multiple pods running on multiple nodes can use a single volume and read data.
If a pod mounts a volume with ReadOnlyMany access mode, other pod can mount it and perform only read operation. Right now GCP is not supporting this method.
This means a volume can be mounted on one or many nodes of your Kubernetes cluster and you can only perform read operation.
You have one pod is running on node and you are reading stored file from volume. While on same volume you can not perform writes.
As it's ReadOnlyMany, if your pod is scheduled to another node, then also volume and data will be available to perform read operation.
ReadWriteMany – the volume can be mounted as read-write by many nodes
By this method, multiple pods running on multiple nodes can use a single volume and read/write data.
If a pod mounts a volume with ReadWriteMany access mode, other pod can also mount it.
This means the volume can be mounted on one or many node of your Kubernetes cluster and you can perform both, read and write operation.
You have one pod running on a node and you are reading & writing the stored file from the volume.
As it's ReadWriteMany if your pod schedule to another node then also the volume and data will be available there to perform read/write operation.
for this, you can use NFS (MinIO, GlusterFS) or EFS filesystem also.
ReadWriteOnce – the volume can be mounted as read-write by a single node
If a pod mounts a volume with ReadWriteOnce access mode, no other pod can mount it. In GCE (Google Compute Engine) the only allowed modes are ReadWriteOnce and ReadOnlyMany. So either one pod mounts the volume ReadWrite, or one or more pods mount the volume ReadOnlyMany.
This means the volume can be mounted on only one node of your kubernetes cluster and you can only perform read operation.
You have one pod running on node and you are reading stored file from volume. While on same volume you cannot perform writes.
As it's ReadWriteOnce if your pod is scheduled to another node then may mossible volume will be attached to the node and you can not get access of data there.
In Kubernetes you provision storage either statically(using a storage class) or dynamically (Persistent Volume). Once the storage is available to bound and claimed, you need to configure it in what way your Pods or Nodes are connecting to the storage (a persistent volume). That could be configured in below four modes.
ReadOnlyMany (ROX)
In this mode multiple pods running on different Nodes could connect to the storage and carry out read operation.
ReadWriteMany (RWX)
In this mode multiple pods running on different Nodes could connect to the storage and carry out read and write operation.
ReadWriteOnce (RWO)
In this mode multiple pods running in only one Node could connect to the storage and carry out read and write operation.
ReadWriteOncePod (RWOP)
In this mode the volume can be mounted as read-write by a single Pod. Use ReadWriteOncePod access mode if you want to ensure that only one pod across whole cluster can read that PVC or write to it. This is only supported for CSI volumes and Kubernetes version 1.22+.
Folow the documentation to get more insight.
Can we use single volume across multiple Nodes in read-only manner and that way save on the storage required for hosting Configuration in kubernetes.
If yes then which is the best sutaible volume type for same?
Not sure what do you mean by single volume but, you can use persistentVolumeClaim which should be created with Access Modes ReadOnlyMany
ReadOnlyMany – the volume can be mounted read-only by many nodes
You need to keep in mind that not all Types of Persistent Volumes are available in that Access Mode
The best way to manage configuration in Kubernetes is by creating a ConfigMap objects and mounting them as volumes/folders into pods.
Mounting ConfigMap as volumes:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#add-configmap-data-to-a-volume
Good luck.
I am trying to understand different access modes for Persistent Volume Claims in Openshift. Found the following information from here
Access Mode CLI Abbreviation Description
ReadWriteOnce RWO The volume can be mounted as read-write by a single node.
ReadOnlyMany ROX The volume can be mounted read-only by many nodes.
ReadWriteMany RWX The volume can be mounted as read-write by many nodes.
I know that PVC are bound to single project/namespace and can be extended to different projects as well.
But the confusion here is, what does "single node" or "many nodes" mean here. For example, in RWO mode, "The volume can be mounted as read-write by a single node". What node it is referring to.
Can someone give me the significance of these modes with respect to single project/namespace. Does the storage with RWO can have write permission for only one application or all the applications within the project?
The whole RWO vs RWX concept is related to the issue of mounting the same filesystem on multiple hosts, which requires support for things like ie. distributed locking. There are specific implementations that can handle this like ie. NFS, Ceph, GlusterFS etc. generally network/cluster oriented filesystems. Other filesystems are unable to operate correctly if you try to mount them on different servers at the same time (usually they will just not allow this).
So, node, in this case, means particular kubernetes cluster node (be it baremetal server or vm). But, by extension, you should think about it in scope of POD as well, cause in most cases pods can spin up on different nodes, meaning they could not use the same volume or you can not assume that this volume will have coherent shared state, as would happen ie. using HostPath volumes that are unique per every node in cluster.
To clarify for the question below :
RWO volumes have 1:1 relation to pod in general. While in some cases you can define RWO volumes to point to the same physical resource like hostPath, technically they will always be tightly coupled to exclusively one POD. This is specially visible if you use PhysicalVolumes / PhysicalVolumeClaims objects, which will take into account these restrictions for binding PV to PVC. Only RWX volumes give you a storage shared by multiple pods with all pods being able to write to it.
It seems that Kubernetes supports 3 kinds of access mode for persistent volume: ReadWriteOnce, ReadOnlyMany, ReadWriteMany.
I'm really curious about the scheduler strategy for a pod which uses the ReadWriteOnce mode volume. For example, I created an RC which have pod num=2, I guess the two pods will be scheduled into the same host because they use the volume that has ReadWriteOnce mode?
I really want to know the source code of this part.
I think the upvoted answer is wrong. As per Kubernetes docs on Access Modes
The access modes are:
ReadWriteOnce -- the volume can be mounted as read-write by a single node
ReadOnlyMany -- the volume can be mounted read-only by many nodes
ReadWriteMany -- the volume can be mounted as read-write by many nodes
So AccessModes as defined today, only describe node attach (not pod mount) semantics, and doesn't enforce anything.
So to prevent two pods mount the same PVC even if they are scheduled to be run on the same node you can use pod anti-affinity. It is not the same as not to mount one volume to 2 pods scheduled on the same node. But anti-affinity can be used to ask scheduler not to run 2 pods on the same node. Therefore it prevents mounting one volume into 2 pods.
If a pod mounts a volume with ReadWriteOnce access mode, no other pod can mount it. In GCE (Google Compute Engine) the only allowed modes are ReadWriteOnce and ReadOnlyMany. So either one pod mounts the volume ReadWrite, or one or more pods mount the volume ReadOnlyMany.
The scheduler (code here) will not allow a pod to schedule if it uses a GCE volume that has already been mounted read-write.
(Documentation reference for those who didn't understand the question: persistent volume access modes)
In Kubernetes you provision storage either statically(using a storage class) or dynamically (Persistent Volume). Once the storage is available to bound and claimed, you need to configure it in what way your Pods or Nodes are connecting to the storage (a persistent volume). That could be configured in below four modes.
ReadOnlyMany (ROX)
In this mode multiple pods running on different Nodes could connect to the storage and carry out read operation.
ReadWriteMany (RWX)
In this mode multiple pods running on different Nodes could connect to the storage and carry out read and write operation.
ReadWriteOnce (RWO)
In this mode multiple pods running in only one Node could connect to the storage and carry out read and write operation.
ReadWriteOncePod (RWOP)
In this mode the volume can be mounted as read-write by a single Pod. Use ReadWriteOncePod access mode if you want to ensure that only one pod across whole cluster can read that PVC or write to it. This is only supported for CSI volumes and Kubernetes version 1.22+.
Follow the documentation to get more insight.