Cant share a PVC with multiple pods in the GCP (with the GCP-CLI)
When I apply the config with ReadWriteOnce works at once
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: <name>
namespace: <namespace>
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 50Gi
But with ReadWriteMany the status hangs on pending
Any ideas?
So it is normal that when you apply the config with ReadWriteOnce works at once, that's the rule.
ReadWriteOnce is the most common use case for Persistent Disks and works as the default access mode for most applications.
GCE persistent disk do not support ReadWriteMany !
Instead of ReadWriteMany, you can just use ReadOnlyMany.
More information you can find here: persistentdisk. But as you now result will not be the same as you want.
If you want to share volumes you could try some workarounds:
You can create services.
Your service should look after the data that is related to its area of concern and should allow access to this data to other services via an interface. Multi-service access to data is an anti-pattern akin to global variables in OOP.
If you where looking to write logs, you should have a log service which each service can call with the relevant data it needs to log. Writing directly to a shared disk means that you'd need to update every container if you change your log directory structure or add extra features.
Also try to use high-performance, fully managed file storage for applications that require a file system interface and a shared file system.
More information you can find here: access-fileshare.
Referring to the Kubernetes-Documentation, GCE does not support ReadWriteMany-Storage: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes
There are some options here:
How to share storage between Kubernetes pods?
https://cloud.google.com/filestore/docs/accessing-fileshares
Related
Kubernetes creates one PersistentVolume for each VolumeClaimTemplate definition on an statefulset. That makes each statefulset pod have its own storage that is not shared across the replicas. However, I would like to share the same volume across all the statefulset replicas.
It looks like the approach should be the following:
Create a PVC on the same namespace.
On the statefulset use Volumes to bound the PVC
Ensure that the PVC is ReadOnlyMany or ReadWriteMany
Assuming that my application is able to deal with any concurrency on the shared volume, is there any technical problem if I have one PVC to share the same volume across all statefulset replicas?
I wholeheartedly agree with the comments made by #Jonas and #David Maze:
You can do this, it should work. There is no need to use volumeClaimTemplates unless your app needs it.
Two obvious problems are that ReadWriteMany volumes are actually a little tricky to get (things like AWS EBS volumes are only ReadWriteOnce), and that many things you want to run in StatefulSets (like databases) want exclusive use of their filesystem space and use file locking to enforce this.
Answering on the question:
Is there any technical problem if I have one PVC to share the same volume across all statefulset replicas?
I'd say that this would mostly depend on the:
How the application would handle such scenario where it's having single PVC (writing concurrency).
Which storage solution are supported by your Kubernetes cluster (or could be implemented).
Subjectively speaking, I don't think there should be an issue when above points are acknowledged and aligned with the requirements and configuration that the cluster/applications allows.
From the application perspective, there is an inherent lack of the software we are talking about. Each application could behave differently and could require different tuning (look on the David Maze comment).
We do not also know anything about your infrastructure so it could be hard to point you potential issues. From the hardware perspective (Kubernetes cluster), this would inherently go into making a research on the particular storage solution that you would like to use. It could be different from cloud provider to cloud provider as well as on-premise solutions. You would need to check the requirements of your app to align it to the options you have.
Continuing on the matter of Volumes, I'd reckon the one of the important things would be accessModes.
Citing the official docs:
Access Modes
A PersistentVolume can be mounted on a host in any way supported by the resource provider. As shown in the table below, providers will have different capabilities and each PV's access modes are set to the specific modes supported by that particular volume. For example, NFS can support multiple read/write clients, but a specific NFS PV might be exported on the server as read-only. Each PV gets its own set of access modes describing that specific PV's capabilities.
The access modes are:
ReadWriteOnce -- the volume can be mounted as read-write by a single node
ReadOnlyMany -- the volume can be mounted read-only by many nodes
ReadWriteMany -- the volume can be mounted as read-write by many nodes
In the CLI, the access modes are abbreviated to:
RWO - ReadWriteOnce
ROX - ReadOnlyMany
RWX - ReadWriteMany
Kubernetes.io: Docs: Concepts: Storage: Persistent Volumes: Access modes
One of the issues you can run into is with the ReadWriteOnce when the PVC is mounted to the Node and sts-X (Pod) is scheduled onto a different Node but from the question, I'd reckon you already know about it.
However, I would like to share the same volume across all the statefulset replicas.
An example of a StatefulSet with a Volume that would be shared across all of the replicas could be following (modified example from Kubernetes documentation):
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx # has to match .spec.template.metadata.labels
serviceName: "nginx"
replicas: 3 # by default is 1
template:
metadata:
labels:
app: nginx # has to match .spec.selector.matchLabels
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
# VOLUME START
volumeMounts:
- name: example-pvc
mountPath: /usr/share/nginx/html
volumes:
- name: example-pvc
persistentVolumeClaim:
claimName: pvc-for-sts
# VOLUME END
Additional resources:
Kubernetes.io: Docs: Concepts: Workloads: Controllers: Statefulset
Kubernetes.io: Docs: Concepts: Storage: Persistent Volumes
I have got a deployment.yaml and it uses a persistentvolumeclaim like so
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mautic-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: standard
I am trying to scale my deployment horizontally using (Horizontal Pod Scheduler) but when I scale my deployment, the rest of the pods are in ContainerCreating process and this is the error I get when I describe the pod
Unable to attach or mount volumes: unmounted volume
What am I doing wrong here?
Using Deployment is great if your app can scale horizontally. However, using a Persistent Volume with a PersistentVolumeClaim can be challenging when scaling horizontally.
Persistent Volume Claim - Access Modes
A PersistentVolumeClaim can be requested for a few different Access Modes:
ReadWriteOnce (most common)
ReadOnlyMany
ReadWriteMany
Where ReadWriteOnce is the most commonly available and is typical behavior for a local disk. But to scale your app horizontally - you need a volume that is available from multiple nodes at the same time, so only ReadOnlyMany and ReadWriteMany is viable options. You need to check what what access modes are available for your storage system.
In addition, you use a regional cluster from a cloud provider, it spans over three Availability Zones and a volume typically only live in one Availability Zone, so even if you use ReadOnlyMany or ReadWriteMany access modes, it makes your volume available on multiple nodes in the same AZ, but not available in all three AZs in your cluster. You might consider using a storage class from your cloud provider that is replicated to multiple Availability Zones, but it typically costs more and is slower.
Alternatives
Since only ReadWriteOnce is commonly available, you might look for better alternatives for your app.
Object Storage
Object Storage, or Buckets, is a common way to handle file storage in the cloud instead of using filesystem volumes. With Object Storage you access you files via an API over HTTP. See e.g. AWS S3 or Google Cloud Storage.
StatefulSet
You could also consider StatefulSet where each instance of your app get its own volume. This makes your app distributed but typically not horizontally scalable. Here, your app typically needs to implement replication of data, typically using Raft and is a more advanced alterantive.
I'm just learning k3s and Helm using a raspberry pi cluster. I have added a thumbdrive to one of the workers and given it a path like /mnt/thumb and I want to store my data from Node red here (actually I have data in this directory that I want it to use). But I can't seem to change the helm chart to point to the path on that specific node to make that happen. I'm using this values.yaml. I've tried following different instructions but none of them have worked. Can someone please show me an example of how to do this? Thanks in advance
I am using the following approach for mounting a specific folder into a pod using persistent volume claims with kurlsh. It should also work with k3s on a raspberry:
Before installing your helm chart, create a new Persistent volume from the following yaml:
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-volume
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
storageClassName: local-volume
hostPath:
path: /mnt/thumb
What you're doing there is essentially creating a kubernetes volume backed by a folder on your filesystem and assigning a storage class to it ( and a size ).
Next, you will need to tell your helm chart to create a new PersistentVolumeClaim using the storageClass you've just created. To do that, fill in the persistence section of the values.yaml as follows:
persistence:
enabled: true
storageClass: "local-volume"
accessMode: ReadWriteOnce
size: 20Gi
Make sure that the size of the volume matches in both the PersistentVolume and the values.yaml.
You will need to take a closer look at the folder structure inside the persistent Volume and also make sure, that the folder's permissions are set so that your pod can actually write data into said folder, but that should be enough to get you started, without relying on fancy third party storage solutions.
I have a docker image that when created should check if the volume is empty, in case it should initialize it with some data.
This saved data must remain available for other pods with the same or different image.
What do you recommend me to do?
You have 2 options:
First option is to mount the pod into the node and save the data in the node so when new pod will create in the same node it will have an access to the same volume (persistent storage location).
Potential problem: 2 pods on the same node can create deadlock for the same resource (so you have to manage the resource).
Shared storage meaning create one storage and every pod will claim storage in the same storage.
I strongly suggest that you will take the next 55 minutes and see the webinar below:
https://www.youtube.com/watch?v=n06kKYS6LZE
I assume you create your pods using Deployment object in Kubernetes. What you want to look into is a StatefulSet, which, in opposite to deployments, retains some identity aspects for recreated pods including to some extent network and storage.
It was introduced specifically as a means to run services that need to keep their state in kube cluster (ie. running databases queues etc.)
Looking at the answers, would it not be simpler to create an NFS Persistent Volume and then allow the pods to mount the PV's?
You can use the writemany which should alleviate a deadlock.
apiVersion: v1
kind: PersistentVolume
metadata:
name: shared-volume
spec:
capacity:
storage: 1Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: ""
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /tmp
server: 172.17.0.2
Persistent Volumes
I want to setup a PVC on AWS, where I need ReadWriteMany as access mode. Unfortunately, EBS only supports ReadWriteOnce.
How could I solve this?
I have seen that there is a beta provider for AWS EFS which supports ReadWriteMany, but as said, this is still beta, and its installation looks somewhat flaky.
I could use node affinity to force all pods that rely on the EBS volume to a single node, and stay with ReadWriteOnce, but this limits scalability.
Are there any other ways of how to solve this? Basically, what I need is a way to store data in a persistent way to share it across pods that are independent of each other.
Using EFS without automatic provisioning
The EFS provisioner may be beta, but EFS itself is not. Since EFS volumes can be mounted via NFS, you can simply create a PersistentVolume with a NFS volume source manually -- assuming that automatic provisioning is not a hard requirement on your side:
apiVersion: v1
kind: PersistentVolume
metadata:
name: my-efs-volume
spec:
capacity:
storage: 100Gi # Doesn't really matter, as EFS does not enforce it anyway
volumeMode: Filesystem
accessModes:
- ReadWriteMany
mountOptions:
- hard
- nfsvers=4.1
- rsize=1048576
- wsize=1048576
- timeo=600
- retrans=2
nfs:
path: /
server: fs-XXXXXXXX.efs.eu-central-1.amazonaws.com
You can then claim this volume using a PersistentVolumeClaim and use it in a Pod (or multiple Pods) as usual.
Alternative solutions
If automatic provisioning is a hard requirement for you, there are alternative solutions you might look at: There are several distributed filesystems that you can roll out on yourcluster that offer ReadWriteMany storage on top of Kubernetes and/or AWS. For example, you might take a look at Rook (which is basically a Kubernetes operator for Ceph). It's also officially still in a pre-release phase, but I've already worked with it a bit and it runs reasonably well.
There's also the GlusterFS operator, which already seems to have a few stable releases.
You can use Amazon EFS to create PersistentVolume with ReadWriteMany access mode.
Amazon EKS Announced support for the Amazon EFS CSI Driver on Sep 19 2019, which makes it simple to configure elastic file storage for both EKS and self-managed Kubernetes clusters running on AWS using standard Kubernetes interfaces.
Applications running in Kubernetes can
use EFS file systems to share data between pods in a scale-out group,
or with other applications running within or outside of Kubernetes.
EFS can also help Kubernetes applications be highly available because
all data written to EFS is written to multiple AWS Availability zones.
If a Kubernetes pod is terminated and relaunched, the CSI driver will
reconnect the EFS file system, even if the pod is relaunched in a
different AWS Availability Zone.
You can deploy the Amazon EFS CSI Driver to an Amazon EKS cluster following the EKS-EFS-CSI user guide, basically like this:
Step 1: Deploy the Amazon EFS CSI Driver
kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"
Note: This command requires version 1.14 or greater of kubectl.
Step 2: Create an Amazon EFS file system for your Amazon EKS cluster
Step 2.1: Create a security group that allows inbound NFS traffic for your Amazon EFS mount points.
Step 2.2: Add a rule to your security group to allow inbound NFS traffic from your VPC CIDR range.
Step 2.3: Create the Amazon EFS file system configured with the security group you just created.
Now you are good to use EFS with ReadWriteMany access mode in your EKS Kubernetes project with the following sample manifest files:
1. efs-storage-class.yaml: Create the storage class
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: efs-sc
provisioner: efs.csi.aws.com
kubectl apply -f efs-storage-class.yaml
2. efs-pv.yaml: Create PersistentVolume
apiVersion: v1
kind: PersistentVolume
metadata:
name: ftp-efs-pv
spec:
storageClassName: efs-sc
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 10Gi # Doesn't really matter, as EFS does not enforce it anyway
volumeMode: Filesystem
accessModes:
- ReadWriteMany
csi:
driver: efs.csi.aws.com
volumeHandle: fs-642da695
Note: you need to replace the volumeHandle value with your Amazon EFS file system ID.
3. efs-pvc.yaml: Create PersistentVolumeClaim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ftp-pv-claim
labels:
app: ftp-storage-claim
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
storageClassName: efs-sc
That should be it. You need to refer to the aforementioned official user guide for detailed explanation, where your can also find an example app to verify your setup.
As you mention EBS volume with affinity & node selector will stop scalability however with EBS only ReadWriteOnce will work.
Sharing my experience, if you are doing many operations on the file system and frequently pushing & fetching files it might could be slow with EFS which can degrade application performance. operation rate on EFS is slow.
However, you can use GlusterFs in back it will be provisioning EBS volume. GlusterFS also support ReadWriteMany and it will be faster compared to EFS as it's block storage (SSD).