I'm currently trying to configure a Logstash cluster with Kubernetes and I would like to have each of the logstash nodes mount a volume as read-only with the pipelines. This same volume would then be mounted as read/write on a single management instance where I could edit the configs.
Is this possible with K8s and GCEPersistentDisk?
By Logstash I believe you mean an ELK cluster. Logstash is just a log forwarder and not an endpoint for storage.
Not really. It's not possible with a GCEPersistentDisk. This is more of GCE limitation where you can only mount a volume on an instance at a time.
Also, as you can see in the docs supports the ReadWriteOnce and the ReadOnlyMany but not at the same time.
Important! A volume can only be mounted using one access mode at a time, even if it supports many. For example, a GCEPersistentDisk can be mounted as ReadWriteOnce by a single node or ReadOnlyMany by many nodes, but not at the same time.
You could achieve this but just using a single volume on a single K8s node and then partition your volume to be used by different Elasticsearch pods on the same node but this would defeat the purpose of having a distributed cluster.
Elasticsearch works fine if you have your nodes in different Kubernetes nodes and each of them has a separate volume.
Related
In a Kubernetes cluster on Oracle cloud, I have a pod with an Apache server.
This pod needs a persistent volume so I used a persistentVolumeClaim and the cloud provider is able to automatically create an associated volume (Oracle Block Volume).
The access mode used by the PVC is readWriteOnce and therefore the volume created has the same access mode.
Everything work great.
Now I want to backup this volume using borg backup and borgmatic by starting a new pod regularly with a cronJob.
This backup pod needs to mount the volume in read only.
Question:
Can I use the previously defined PVC?
Do I need to create a new PVC with readOnly access mode?
As per documentation:
ReadWriteOnce:
the volume can be mounted as read-write by a single node. ReadWriteOnce access mode still can allow multiple pods to access the volume when the pods are running on the same node.
That means if you make a strict rule for deploying your pods to the same node, you can use the same PVC, here's the INSTRUCTION
Can we use kubernetes volumes for deployments? If yes than that will be mutliple pods sharing the same volume?
If that is possible then what happens when all the pods for the deployment are on different host machines?
Especially when using Amazon EBS where an ebs volume cannot be shared across multiple hosts.
Yes, you can use a persistent volume for deployments
Such a volume will be mounted to your desired location in all the pods
If you use EBS block storage, all your pods will need to be scheduled on the same node where you have attached your volume. This may not work if you have many replicas
You will have to use a network file storage, such as EFS, GlusterFS, Portworx, etc. with ReadWriteMany if you want your pods to be spun up on different nodes
EBS will give you the best performance with the aforementioned single node limitation
I was going through in depth of K8s volumes and one of the concept messed me up. It would be great if someone could explain me this.
These are the modes in K8s volumes.
ReadWriteOnce -- the volume can be mounted as read-write by a single node
ReadOnlyMany -- the volume can be mounted read-only by many nodes
ReadWriteMany -- the volume can be mounted as read-write by many nodes
What if I run two mysql pods on same node and the mode is ReadWriteOnce ? How my both pods will write?
What if I run two mysql pods on two different nodes and the mode is ReadWriteOnce ? How my both pods will write?
What if I run two mysql pods on same node and the mode is ReadWriteMany ? How my both pods will write?
What if I run two mysql pods on two different nodes and the mode is ReadWriteMany ? How my both pods will write?
All have same pod replica.
What if I run two mysql pods on same node and the node is ReadWriteOnce ? How my both pods will write?
If you run two DBMS systems (e.g. MySQL or PostgreSQL) in Pods, they are both independent and they should have two independent PersistentVolumeClaims. Only run single-node DBMS for test purposes. For production consider a distributed database that replicates the data.
What if I run two mysql pods on two different node and the node is ReadWriteOnce ? How my both pods will write?
As I recommended above, the two should be independent (unless you use a distributed database like cockroachDB) so running on different nodes is perfectly fine.
What if I run two mysql pods on same different node and the node is ReadWriteMany ? How my both pods will write?
MySQL is a single-node DBMS system, it should not share volumes/files with other instances unless you run a clustered system.
What if I run two mysql pods on two different node and the node is ReadWriteMany ? How my both pods will write?
Same as above, single-node system should not share volumes/files.
Just explored and implemented and got few information and thanks to #jonas
What if I run two mysql pods on same node and the mode is ReadWriteOnce ? How my both pods will write?
We cannot schedule two pods of mysql on same node, it will result in crashloopbackoff.
What if I run two mysql pods on two different nodes and the mode is ReadWriteOnce ? How my both pods will write?
Since the mode is ReadWriteOnce, the data will be written by only one pod in one node based on order.
What if I run two mysql pods on same node and the mode is ReadWriteMany ? How my both pods will write?
Was not able to schedule two pods on single node. Resulted in crashloopbackoff
What if I run two mysql pods on two different nodes and the mode is ReadWriteMany ? How my both pods will write?
Need to use EFS CSI Driver in case of AWS or Shared volume like network attached storage. Check out this link Kubernetes PVC with ReadWriteMany on AWS
As per this official document, Kubernetes Persistent Volumes support three types of access modes.
ReadOnlyMany
ReadWriteOnce
ReadWriteMany
The given definitions of them in the document is very high-level. It would be great if someone can explain them in little more detail along with some examples of different use cases where we should use one vs other.
You should use ReadWriteX when you plan to have Pods that will need to write to the volume, and not only read data from the volume.
You should use XMany when you want the ability for Pods to access the given volume while those workloads are running on different nodes in the Kubernetes cluster. These Pods may be multiple replicas belonging to a Deployment, or may be completely different Pods. There are many cases where it's desirable to have Pods running on different nodes, for instance if you have multiple Pod replicas for a single Deployment, then having them run on different nodes can help ensure some amount of continued availability even if one of the nodes fails or is being updated.
If you don't use XMany, but you do have multiple Pods that need access to the given volume, that will force Kubernetes to schedule all those Pods to run on whatever node the volume gets mounted to first, which could overload that node if there are too many such pods, and can impact the availability of Deployments whose Pods need access to that volume as explained in the previous paragraph.
So putting all that together:
If you need to write to the volume, and you may have multiple Pods needing to write to the volume where you'd prefer the flexibility of those Pods being scheduled to different nodes, and ReadWriteMany is an option given the volume plugin for your K8s cluster, use ReadWriteMany.
If you need to write to the volume but either you don't have the requirement that multiple pods should be able to write to it, or ReadWriteMany simply isn't an available option for you, use ReadWriteOnce.
If you only need to read from the volume, and you may have multiple Pods needing to read from the volume where you'd prefer the flexibility of those Pods being scheduled to different nodes, and ReadOnlyMany is an option given the volume plugin for your K8s cluster, use ReadOnlyMany.
If you only need to read from the volume but either you don't have the requirement that multiple pods should be able to read from it, or ReadOnlyMany simply isn't an available option for you, use ReadWriteOnce. In this case, you want the volume to be read-only but the limitations of your volume plugin have forced you to choose ReadWriteOnce (there's no ReadOnlyOnce option). As a good practice, consider the containers.volumeMounts.readOnly setting to true in your Pod specs for volume mounts corresponding to volumes that are intended to be read-only.
ReadOnlyMany – the volume can be mounted read-only by many nodes
By this method, multiple pods running on multiple nodes can use a single volume and read data.
If a pod mounts a volume with ReadOnlyMany access mode, other pod can mount it and perform only read operation. Right now GCP is not supporting this method.
This means a volume can be mounted on one or many nodes of your Kubernetes cluster and you can only perform read operation.
You have one pod is running on node and you are reading stored file from volume. While on same volume you can not perform writes.
As it's ReadOnlyMany, if your pod is scheduled to another node, then also volume and data will be available to perform read operation.
ReadWriteMany – the volume can be mounted as read-write by many nodes
By this method, multiple pods running on multiple nodes can use a single volume and read/write data.
If a pod mounts a volume with ReadWriteMany access mode, other pod can also mount it.
This means the volume can be mounted on one or many node of your Kubernetes cluster and you can perform both, read and write operation.
You have one pod running on a node and you are reading & writing the stored file from the volume.
As it's ReadWriteMany if your pod schedule to another node then also the volume and data will be available there to perform read/write operation.
for this, you can use NFS (MinIO, GlusterFS) or EFS filesystem also.
ReadWriteOnce – the volume can be mounted as read-write by a single node
If a pod mounts a volume with ReadWriteOnce access mode, no other pod can mount it. In GCE (Google Compute Engine) the only allowed modes are ReadWriteOnce and ReadOnlyMany. So either one pod mounts the volume ReadWrite, or one or more pods mount the volume ReadOnlyMany.
This means the volume can be mounted on only one node of your kubernetes cluster and you can only perform read operation.
You have one pod running on node and you are reading stored file from volume. While on same volume you cannot perform writes.
As it's ReadWriteOnce if your pod is scheduled to another node then may mossible volume will be attached to the node and you can not get access of data there.
In Kubernetes you provision storage either statically(using a storage class) or dynamically (Persistent Volume). Once the storage is available to bound and claimed, you need to configure it in what way your Pods or Nodes are connecting to the storage (a persistent volume). That could be configured in below four modes.
ReadOnlyMany (ROX)
In this mode multiple pods running on different Nodes could connect to the storage and carry out read operation.
ReadWriteMany (RWX)
In this mode multiple pods running on different Nodes could connect to the storage and carry out read and write operation.
ReadWriteOnce (RWO)
In this mode multiple pods running in only one Node could connect to the storage and carry out read and write operation.
ReadWriteOncePod (RWOP)
In this mode the volume can be mounted as read-write by a single Pod. Use ReadWriteOncePod access mode if you want to ensure that only one pod across whole cluster can read that PVC or write to it. This is only supported for CSI volumes and Kubernetes version 1.22+.
Folow the documentation to get more insight.
It seems that Kubernetes supports 3 kinds of access mode for persistent volume: ReadWriteOnce, ReadOnlyMany, ReadWriteMany.
I'm really curious about the scheduler strategy for a pod which uses the ReadWriteOnce mode volume. For example, I created an RC which have pod num=2, I guess the two pods will be scheduled into the same host because they use the volume that has ReadWriteOnce mode?
I really want to know the source code of this part.
I think the upvoted answer is wrong. As per Kubernetes docs on Access Modes
The access modes are:
ReadWriteOnce -- the volume can be mounted as read-write by a single node
ReadOnlyMany -- the volume can be mounted read-only by many nodes
ReadWriteMany -- the volume can be mounted as read-write by many nodes
So AccessModes as defined today, only describe node attach (not pod mount) semantics, and doesn't enforce anything.
So to prevent two pods mount the same PVC even if they are scheduled to be run on the same node you can use pod anti-affinity. It is not the same as not to mount one volume to 2 pods scheduled on the same node. But anti-affinity can be used to ask scheduler not to run 2 pods on the same node. Therefore it prevents mounting one volume into 2 pods.
If a pod mounts a volume with ReadWriteOnce access mode, no other pod can mount it. In GCE (Google Compute Engine) the only allowed modes are ReadWriteOnce and ReadOnlyMany. So either one pod mounts the volume ReadWrite, or one or more pods mount the volume ReadOnlyMany.
The scheduler (code here) will not allow a pod to schedule if it uses a GCE volume that has already been mounted read-write.
(Documentation reference for those who didn't understand the question: persistent volume access modes)
In Kubernetes you provision storage either statically(using a storage class) or dynamically (Persistent Volume). Once the storage is available to bound and claimed, you need to configure it in what way your Pods or Nodes are connecting to the storage (a persistent volume). That could be configured in below four modes.
ReadOnlyMany (ROX)
In this mode multiple pods running on different Nodes could connect to the storage and carry out read operation.
ReadWriteMany (RWX)
In this mode multiple pods running on different Nodes could connect to the storage and carry out read and write operation.
ReadWriteOnce (RWO)
In this mode multiple pods running in only one Node could connect to the storage and carry out read and write operation.
ReadWriteOncePod (RWOP)
In this mode the volume can be mounted as read-write by a single Pod. Use ReadWriteOncePod access mode if you want to ensure that only one pod across whole cluster can read that PVC or write to it. This is only supported for CSI volumes and Kubernetes version 1.22+.
Follow the documentation to get more insight.