I am currently looking into setting up Kubernetes pods for a project on GCP.
The problem - I need to set a persistent shared volume which will be used by multiple nodes. I need all nodes to be able to read from the volume and only one node must be able to write on the volume. So I need some advice what's the best way to achieve that?
I have checked the Kubernetes documentation and know that GCEPersistentDisks does not support ReadWriteMany but anyway this access mode I think will be an overkill. Regarding the ReadOnlyMany I get that nodes can read from the PV but I don't understand how or what can actually modify the PV in this case. Currently my best bet is setting up NFS with GCE persistent disk.
Also the solution should be able to run on the cloud or on premise. Any advice will be appreciated :)
According to official documentation:
A PVC to PV binding is a one-to-one mapping.
A volume can only be mounted using one access mode at a time, even if
it supports many. For example, a GCEPersistentDisk can be mounted as
ReadWriteOnce by a single node or ReadOnlyMany by many nodes, but not
at the same time.
So I am afraid that this would be impossible to do it how you described.
However, you may want to try task queue
Please let me know if that helped.
Assuming there is some NFS space availalbe, it could be possible to create two persistent volume claims (PVCs) for it: one readonly, one readwrite.
Then you could have two persistent volumes binding one to one to each of the PVCs.
Now create two deployments. One of them describes the pod with the writing application, and you ensure you have one replica. The other deployment describes the reading application, and you can scale it to whatever amount you like.
Related
I have a GKE cluster, with almost 6-7 micro-services deployed. I need a Postgres DB to be installed inside GKE (not Cloudsql as cost). When checked the different types of persistent volumes i can see that if multiple micro-service accessing the same DB, should i go using NFS or PVC with normal disk would be enough not anyway local storage.
Request your thought on this.
Everything depends from your scenario. In general you should follow AccessMode when you are considering which Volume Plugin you want to use.
A PersistentVolume can be mounted on a host in any way supported by the resource provider. As shown in the table below, providers will have different capabilities and each PV's access modes are set to the specific modes supported by that particular volume.
In this documentation below, you will find table with different Volume Plugins and supported Access Modes.
According to update form your comment, you have only one node. With that setup, you can use almost every Volume which supports RWO Access mode.
ReadWriteOnce -- the volume can be mounted as read-write by a single node.
There are 2 other Access Modes which should be consider if would like to use it for more than 1 node.
ReadOnlyMany -- the volume can be mounted read-only by many nodes
ReadWriteMany -- the volume can be mounted as read-write by many nodes
So in your case you can use gcePersistentDisk as it supports (ReadWriteOnce and ReadOnlyMany).
Using NFS would benefit if you would like to access this PV from many nodes.
NFS can support multiple read/write clients, but a specific NFS PV might be exported on the server as read-only. Each PV gets its own set of access modes describing that specific PV's capabilities.
Just as addition, if this is for learning puropse, you can also check Local Persistent Volume. Example can be found in this tutorial, however it would require few updates like image or apiVersion.
I need to build an application that has many bare-metal nodes joined in a Kubernetes cluster and I need a shared persistent file system between those nodes. The nodes should be able to read-write in this file system simultaneously.
Bonus: is there a way to keep the file system alive even if the cluster crashes?
I read this article but cant find answers to this question.
This problem is very important to me because it is a requirement to my undergraduate paper.
Yes it does. What you're looking for is to set your AccessMode to ReadWriteMany.
Note that not all Volume Plugins provide ReadWriteMany.
Multiple pods might be reading/writing to the Volume plugin at the same time. If a node/pod were to restart, you would still have access to the volume.
To get a full list of what which Volume Plugin supports that, refer to the official documentation.
I've previously used both types, I've also read through the docs at:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/
https://kubernetes.io/docs/concepts/storage/volumes/
However it's still not clear what the difference is, both seem to support the same storage types, the only thing that comes to mind is there seems to be a 'provisioning' aspect to persistent volumes.
What is the practical difference?
Are there advantages / disadvantages between the two - or for what use case would one be better suited to than the other?
Is it perhaps just 'synctactic sugar'?
For example NFS could be mounted as a volume, or a persistent volume. Both require a NFS server, both will have it's data 'persisted' between mounts. What difference would be had in this situation?
Volume decouples the storage from the Container. Its lifecycle is coupled to a pod. It enables safe container restarts and sharing data between containers in a pod.
Persistent Volume decouples the storage from the Pod. Its lifecycle is independent. It enables safe pod restarts and sharing data between pods.
A volume exists in the context of a pod, that is, you can't create a volume on its own. A persistent volume on the other hand is a first class object with its own lifecycle, which you can either manage manually or automatically.
The way I understand it is that the concept of a Persistent Volumes builds on that of a Volume and that the difference is that a Persistent Volume is more decoupled from Pods using it. Or as expressed in the introduction of the documentation page about Persistent Volumes:
PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV.
A Volume's lifecycle on the other hand depends on the lifecycle of the Pod using it:
A Kubernetes volume [...] has an explicit lifetime - the same as the Pod that encloses it.
NFS is not really relevant here. Both Volumes and Persistent Volumes are Kubernetes resources. They provide an abstraction of a data storage facility. So for using the cluster, it doesn't matter which concrete operating system resource is behind that abstraction. That's in a way the whole point of Kubernetes.
It might also be relevant here to keep in mind that Kubernetes and its API are still evolving. The Kubernetes developers might sometimes choose to introduce new concepts/resources that differ only subtly from existing ones. I presume one reason for this is to maintain backwards compatibility while still being able to fine tune basic API concepts. Another example for this are Replication Controllers and Replica Sets, which conceptually largely overlap and are therefore redundant to some extent. Although, what's different to the Volume/Persitent Volume matter is that Replication Controllers are explicitly deprecated now.
Volumes ≠ Persistent Volumes
Volumes and Persistent Volumes are related, but very different!
Volumes:
appear in Pod specifications
do not exist as API resources (cannot do kubectl get volumes)
Persistent Volumes:
are API resources (can do kubectl get persistentvolumes)
correspond to concrete volumes (e.g. on a SAN, EBS, etc.)
cannot be associated with a Pod directly
(they need a Persistent Volume Claim)
They are two different implementations which can provide some similar common functionality (hence a lot of confusion).
Persistent volumes:
Support storage provisioned via StorageClass
Does not support emptyDir volume type (https://github.com/kubernetes/kubernetes/issues/75378)
Volumes:
Are bound to a pod
Are simpler to define (less Kubernetes resources required)
I have a question regarding what is the best approach with K8S in AWS.
the way I see it that either I use the EBS directly for PV AND PVC or that I mount the EBS as a regular folder in my EC2 and then use those mounted folders for my PV and PVC.
what approach is better in your opinion?
it is important to notice that I want my K8s to Cloud agnostic so maybe forcing EBS configuration is less better that using a folder so the ec2 does not care what is the origin of the folder.
many thanks
what approach is better in your opinion?
Without question: using the PV and PVC. Half the reason will go here, and the other half below. By declaring those as managed resources, kubernetes will cheerfully take care of attaching the volumes to the Node it is scheduling the Pod upon, and detaching it from the Node when the Pod is unscheduled. That will matter in a huge way if a Node reboots, for example, because the attach-detach cycle will happen transparently, no Pager Duty involved. That will not be true if you are having to coordinate amongst your own instances who is alive and should have the volume attached at this moment.
it is important to notice that I want my K8s to Cloud agnostic so maybe forcing EBS configuration is less better that using a folder so the ec2 does not care what is the origin of the folder.
It still will be cloud agnostic, because what you have told kubernetes -- declaratively, I'll point out, using just text in a yaml file -- is that you wish for some persistent storage to be volume mounted into your container(s) before they are launched. Only drilling down into the nitty gritty will surface the fact that it's provided by an AWS EBS volume. I would almost guarantee you could move those descriptors over to GKE (or Azure's thing) with about 90% of the text exactly the same.
The problem seems to have been solved a long time ago, as the answer and the comments does not provide real solutions, I would like to get some help from experienced users
The error is the following (when describing the pod, which keeps on the ContainerCreating state) :
Multi-Attach error for volume "pvc-xxx" Volume is already exclusively attached to one node and can't be attached to another
This all run on GKE. I had a previous cluster, and the problem never occured. I have reused the same disk when creating this new cluster — not sure if it is related
Here is the full yaml config files (I'm leaving the concerned code part commented as to highlight it; it is not when in effective use)
Thanks in advance if obvious workarounds
Based on your description what you are experiencing is exactly what is supposed to happen.
You are using gcePersistentDisk in your PV/PVC definition. The accessMode is ReadWriteOnce - this means that this PV can only be attached to a single Node (stressing Node here, there can be multiple Pods running on the same Node using the same PV). There is not much you can do about this; gcePersistentDisk is like a remote block device, it's not possible to mount it on multiple Nodes simultaneously (unless read only).
There is a nice table that shows which PVs support ReadWriteMany (that is, write access on multiple Nodes at the same time):
Important! A volume can only be mounted using one access mode at a time, even if it supports many. For example, a GCEPersistentDisk can be mounted as ReadWriteOnce by a single node or ReadOnlyMany by many nodes, but not at the same time.
Your Deployment yaml shows 5 replicas, which will not work with GCE PD in ReadWriteOnce mode. GCE PD can only be attached to multiple nodes in ReadOnlyMany mode.
If you need shared, writable storage across all replicas, then you should look into a multi-writer solution like NFS or Gluster.
If you want each replica to have its own disk, then you can use StatefulSets, which will have a PVC per replica.