I need to make use of PVC to specify the specs of the PV and I also need to make sure it uses a custom local storage path in the PV.
I am unable to figure out how to mention the hostpath in a PVC?
This is the PVC config:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mongo-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
And this is the mongodb deployment:
spec:
replicas: 1
selector:
matchLabels:
app: mongo
template:
metadata:
labels:
app: mongo
spec:
volumes:
- name: mongo-volume
persistentVolumeClaim:
claimName: mongo-pvc
containers:
- name: mongo
image: mongo
ports:
- containerPort: 27017
volumeMounts:
- name: mongo-volume
mountPath: /data/db
How and where do I mention the hostPath to be mounted in here?
Doc says that you set hostPath when creating a PV (the step before creating PVC).
apiVersion: v1
kind: PersistentVolume
metadata:
name: task-pv-volume
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
After you create the PersistentVolumeClaim, the Kubernetes control plane looks for a PersistentVolume that satisfies the claim's requirements. If the control plane finds a suitable PersistentVolume with the same StorageClass, it binds the claim to the volume.
Please see https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/
You don't (and can't) force a specific host path in a PersistentVolumeClaim.
Typically a Kubernetes cluster will be configured with a dynamic volume provisioner and that will create the matching PersistentVolume for you. Depending on how your cluster was installed that could be an Amazon EBS volume, a Google Cloud Platform persistent disk, an iSCSI volume, or some other type of storage; as an application author you don't really control that. (You tagged this question for GKE, and the GKE documentation has a section on dynamic volume provisioning.) You don't need to specify where on the host the volume might be mounted, and there's no way to provide this detail in the PersistentVolumeClaim.
With the YAML you show, and the context of this being on GKE, I'd expect Google to automatically provision a GCE persistent disk. If the pod gets rescheduled on a different node, the persistent disk will follow the pod to the new node. You don't need to worry about what specific host directory is being used; Kubernetes will manage this for you.
In most cases you'll want to avoid hostPath storage. You don't directly control which node your pods will run on, so you're not guaranteed that the pod will actually be scheduled on the node that has the data volume. It's appropriate for something like a log collector running in a DaemonSet, where you can guarantee that there is interesting content in that path on every node, but not for your general application database storage.
Related
I have attached an EBS volume to one of the nodes in my cluster and I want that whatever pod are coming up, irrespective of the nodes they are scheduled onto, should use that EBS volume. is this possible?
My approach was to create a PV/PVC that mounts to that volume and then use that PVC in my pod, but I am not sure if it's mounting to same host that pod comes up in or a different host.
YAML for Storage Class
kind: StorageClass
metadata:
name: local-path
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate
allowVolumeExpansion: true
reclaimPolicy: Delete
PV.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-pv
labels:
type: local
spec:
capacity:
storage: 200Mi
storageClassName: local-path
claimRef:
namespace: redis
name: data-redis-0
accessModes:
- ReadWriteMany
hostPath:
path: "/mnt2/data/redis"
PVC.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-redis-0
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 200Mi
storageClassName: local-path
no when i am trying to schedule a pod the storage is also getting mounted on the same node instead
you are using local path you can not do it.
There is a different type of AccessMount ReadWriteMany, ReadWriteOnce, and ReadyWriteOnly with PVC.
A PersistentVolumeClaim (PVC) is a request for storage by a user. It
is similar to a Pod. Pods consume node resources and PVCs consume PV
resources. Pods can request specific levels of resources (CPU and
Memory). Claims can request specific size and access modes (e.g., they
can be mounted ReadWriteOnce, ReadOnlyMany or ReadWriteMany, see
AccessModes).
Read More at : https://kubernetes.io/docs/concepts/storage/persistent-volumes/
Yes you can mount the multiple PODs to a single PVC but in that case, you have to use the ReadWriteMany. Most people use the NFS or EFS for this type of use case.
EBS is ReadWriteOnce, so it won't be possible to use the EBS in your case. you have to either use NFS or EFS.
you can use GlusterFs in the back it will be provisioning EBS volume. GlusterFS support ReadWriteMany and it will be faster compared to EFS as it's block storage (SSD).
For ReadWiteMany you can also checkout : https://min.io/
Find access mode details here : https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes
I have attached an EBS volume to one of the nodes in my cluster and I want that whatever pod are coming up, irrespective of the nodes they are scheduled onto, should use that EBS volume. is this possible?
No. An EBS volume can only be attached to at most one EC2 instance, and correspondingly, one Kubernetes node. In Kubernetes terminology, it only allows the ReadWriteOnce access mode.
It looks like the volume you're trying to create is the backing store for a Redis instance. If the volume will only be attached to one pod at a time, then this isn't a problem on its own, but you need to let Kubernetes manage the volume for you. Then the cluster will know to detach the EBS volume from the node it's currently on and reattach it to the node with the new pod. Setting this up is a cluster-administration problem and not something you as a programmer can do, but it should be set up for you in environments like Amazon's EKS managed Kubernetes.
In this environment:
Don't create a StorageClass; this is cluster-level configuration.
Don't manually create a PersistentVolume; the cluster will create it for you.
You should be able to use the default storageClass: in your PersistentVolumeClaim.
You probably should use a StatefulSet to create the PersistentVolumeClaim for you.
So for example:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
spec:
volumeClaimTemplates: # automatically creates PersistentVolumeClaims
- metadata:
name: data-redis
spec:
accessModes: [ReadWriteOnce] # data won't be shared between pods
resources:
requests:
storage: 200Mi
# default storageClassName:
template:
spec:
containers:
- name: redis
volumeMounts:
- name: data-redis
mountPath: /data
I have a Kubernetes cluster setup (on-premise), that has an NFS share (my-nfs.internal.tld) mounted to /exports/backup on each node to create backups there.
Now I'm setting up my logging stack and I wanted to make the data persistent. So I figured I could start by storing the indices on the NFS.
Now I found three different ways to achieve this:
NFS-PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: logging-data
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
nfs:
server: my-nfs.internal.tld
path: /path/to/exports/backup/logging-data/
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: logging-data-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: logging-data
resources:
requests:
storage: 10Gi
apiVersion: apps/v1
kind: Deployment
...
spec:
...
template:
...
spec:
...
volumes:
- name: logging-data-volume
persistentVolumeClaim:
claimName: logging-data-pvc
This would, of course, require, that my cluster gets access to the NFS (instead of only the nodes as it is currently setup).
hostPath-PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: logging-data
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /exports/backup/logging-data/
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: logging-data-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: logging-data
resources:
requests:
storage: 10Gi
apiVersion: apps/v1
kind: Deployment
...
spec:
...
template:
...
spec:
...
volumes:
- name: logging-data-volume
persistentVolumeClaim:
claimName: logging-data-pvc
hostPath mount in deployment
As the nfs is mounted to all my nodes, I could also just use the host path directly in the deployment without pinning anything.
apiVersion: apps/v1
kind: Deployment
...
spec:
...
template:
...
spec:
...
volumes:
- name: logging-data-volume
hostPath:
path: /exports/backup/logging-data
type: DirectoryOrCreate
So my question is: Is there really any difference between these three? I'm pretty sure all three work. I tested the second and third already. I was not yet able to test the first though (in this specific setup at least). Especially the second and third solutions seem very similar to me. The second makes it easier to re-use deployment files on multiple clusters, I think, as you can use persistent volumes of different types without changing the volumes part of the deployment. But is there any difference beyond that? Performance maybe? Or is one of them deprecated and will be removed soon?
I found a tutorial mentioning, that the hostPath-PV only works on single-node clusters. But I'm sure it does also works in my case here. Maybe the comment was about: "On multi-node clusters the data changes when deployed to different nodes."
From reading to a lot of documentation and How-To's I understand, that the first one is the preferred solution. I would probably also go for it as it is the one easiest replicated to a cloud setup. But I do not really understand why this is preferred to the other two.
Thanks in advance for your input on the matter!
The NFS is indeed the preferred solution:
An nfs volume allows an existing NFS (Network File System) share to
be mounted into a Pod. Unlike emptyDir, which is erased when a Pod
is removed, the contents of an nfs volume are preserved and the
volume is merely unmounted. This means that an NFS volume can be
pre-populated with data, and that data can be shared between pods. NFS
can be mounted by multiple writers simultaneously.
So, an NFS is useful for two reasons:
Data is persistent.
It can be accessed from multiple pods at the same time and the data can be shared between pods.
See the NFS example for more details.
While the hostPath:
A hostPath volume mounts a file or directory from the host node's
filesystem into your Pod.
Pods with identical configuration (such as created from a PodTemplate)
may behave differently on different nodes due to different files on
the nodes
The files or directories created on the underlying hosts are only
writable by root. You either need to run your process as root in a
privileged Container or modify the file permissions on the host to be
able to write to a hostPath volume
hostPath is not recommended due to several reasons:
You don't directly control which node your pods will run on, so you're not guaranteed that the pod will actually be scheduled on the node that has the data volume.
You expose your cluster to security threats.
If a node goes down you need the pod to be scheduled on other node where your locally provisioned volume will not be available.
the hostPath would be good if for example you would like to use it for log collector running in a DaemonSet. Other than that, it would be better to use the NFS.
In my /mnt/ I have a number of hard drives mounted (e.g. at /mnt/hdd1/, /mnt/hdd2/). Is there any way to make a K8s Persistent Volume on /mnt that can see the content of the hard drives mounted under /mnt? When I make a Local Persistent Volume on /mnt, the K8s pods see the directories hdd1 and hdd2, but they appear as empty.
The following is what I have tested:
Undesired solution 1:
I can make a Local Persistent Volume on /mnt/hdd1 and then my K8s pod will be able to see the contents of hdd1 hard drive. But as I mentioned before, I want my pod to see all the hard drives and I don't want to make a persistent volume for each hard drive especially when I mount a new hard drive under /mnt.
Undesired solution 2:
I can mount a Local Persistent Volume on /mnt/ with the K8s option of mountPropagation: HostToContainer in the yaml file for my deployment. In this case my pod will see the content of the hard drive if I remount the hard drive. But this is not desired because if the pod restarts, I need to remount the hard drive again for the pod to see its content! (Only works when hard drive is remounted when the pod is alive)
I was able to allow the pod to see all hard drives mounted in a directory using hostpath. A PersistentVolume can be defined in hostpath "mode." My final solution was:
Most important part of the solution: Define a PersistentVolume in hostpath "mode," with a nodeAffinity to ensure it will only be mounted on the node with the hard drives:
apiVersion: v1
kind: PersistentVolume
metadata:
name: all-harddrives-pv
spec:
volumeMode: Filesystem
storageClassName: all-harddrives-storage
hostPath:
path: /mnt # Where all the hard drives are mounted
type: Directory
nodeAffinity: # Use nodeAffinity to ensure it will only be mounted on the node with harddrives.
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- MyNodeName
Define a PersistentVolumeClaim that is bound to the above PV:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: all-harddrives-pvc
spec:
storageClassName: all-harddrives-storage
Mount it on the Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
spec:
replicas: 1
selector:
matchLabels:
app: my-deployment
strategy:
type: Recreate
template:
metadata:
labels:
app: my-deployment
spec:
containers:
- name: mycontainername
image: myimage
volumeMounts:
- mountPath: /mnt
name: all-harddrives-pvc
nodeSelector:
kubernetes.io/hostname: MyNodeName
This approach, Local Persistence Volume Static Provisioner, suits better with Kubernetes way of working.
It supports metrics, storage lifecycle (eg. cleanup), node/pv affinity, extensible (eg. dynamic ephemeral storage). For example, with eks-nvme-ssd-provisioner, there can be a daemonset running to provision fast storage as local. This is ideal for workload that requires ephemeral local storage for data cache, fast compute, while no need to manually perform mount on the ec2 node before pods start.
Usage yaml examples are here, sig-storage-local-static-provisioner/examples.
I have a three node GCE cluster and a single-pod GKE deployment with three replicas. I created the PV and PVC like so:
# Create a persistent volume for web content
apiVersion: v1
kind: PersistentVolume
metadata:
name: nginx-content
labels:
type: local
spec:
capacity:
storage: 5Gi
accessModes:
- ReadOnlyMany
hostPath:
path: "/usr/share/nginx/html"
--
# Request a persistent volume for web content
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nginx-content-claim
annotations:
volume.alpha.kubernetes.io/storage-class: default
spec:
accessModes: [ReadOnlyMany]
resources:
requests:
storage: 5Gi
They are referenced in the container spec like so:
spec:
containers:
- image: launcher.gcr.io/google/nginx1
name: nginx-container
volumeMounts:
- name: nginx-content
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
volumes:
- name: nginx-content
persistentVolumeClaim:
claimName: nginx-content-claim
Even though I created the volumes as ReadOnlyMany, only one pod can mount the volume at any given time. The rest give "Error 400: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE". How can I make it so all three replicas read the same web content from the same volume?
First I'd like to point out one fundamental discrapency in your configuration. Note that when you use your PersistentVolumeClaim defined as in your example, you don't use your nginx-content PersistentVolume at all. You can easily verify it by running:
kubectl get pv
on your GKE cluster. You'll notice that apart from your manually created nginx-content PV, there is another one, which was automatically provisioned based on the PVC that you applied.
Note that in your PersistentVolumeClaim definition you're explicitely referring the default storage class which has nothing to do with your manually created PV. Actually even if you completely omit the annotation:
annotations:
volume.alpha.kubernetes.io/storage-class: default
it will work exactly the same way, namely the default storage class will be used anyway. Using the default storage class on GKE means that GCE Persistent Disk will be used as your volume provisioner. You can read more about it here:
Volume implementations such as gcePersistentDisk are configured
through StorageClass resources. GKE creates a default StorageClass for
you which uses the standard persistent disk type (ext4). The default
StorageClass is used when a PersistentVolumeClaim doesn't specify a
StorageClassName. You can replace the provided default StorageClass
with your own.
But let's move on to the solution of the problem you're facing.
Solution:
First, I'd like to emphasize you don't have to use any NFS-like filesystems to achive your goal.
If you need your PersistentVolume to be available in ReadOnlyMany mode, GCE Persistent Disk is a perfect solution that entirely meets your requirements.
It can be mounted in ro mode by many Pods at the same time and what is even more important by many Pods, scheduled on different GKE nodes. Furthermore it's really simple to configure and it works on GKE out of the box.
In case you want to use your storage in ReadWriteMany mode, I agree that something like NFS may be the only solution as GCE Persistent Disk doesn't provide such capability.
Let's take a closer look how we can configure it.
We need to start from defining our PVC. This step was actually already done by yourself but you got lost a bit in further steps. Let me explain how it works.
The following configuration is correct (as I mentioned annotations section can be omitted):
# Request a persistent volume for web content
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nginx-content-claim
spec:
accessModes: [ReadOnlyMany]
resources:
requests:
storage: 5Gi
However I'd like to add one important comment to this. You said:
Even though I created the volumes as ReadOnlyMany, only one pod can
mount the volume at any given time.
Well, actually you didn't. I know it may seem a bit tricky and somewhat surprising but this is not the way how defining accessModes really works. In fact it's a widely misunderstood concept. First of all you cannot define access modes in PVC in a sense of putting there the constraints you want. Supported access modes are inherent feature of a particular storage type. They are already defined by the storage provider.
What you actually do in PVC definition is requesting a PV that supports the particular access mode or access modes. Note that it's in a form of a list which means you may provide many different access modes that you want your PV to support.
Basically it's like saying: "Hey! Storage provider! Give me a volume that supports ReadOnlyMany mode." You're asking this way for a storage that will satisfy your requirements. Keep in mind however that you can be given more than you ask. And this is also our scenario when asking for a PV that supports ReadOnlyMany mode in GCP. It creates for us a PersistentVolume which meets our requirements we listed in accessModes section but it also supports ReadWriteOnce mode. Although we didn't ask for something that also supports ReadWriteOnce you will probably agree with me that storage which has a built-in support for those two modes fully satisfies our request for something that supports ReadOnlyMany. So basically this is the way it works.
Your PV that was automatically provisioned by GCP in response for your PVC supports those two accessModes and if you don't specify explicitely in Pod or Deployment definition that you want to mount it in read-only mode, by default it is mounted in read-write mode.
You can easily verify it by attaching to the Pod that was able to successfully mount the PersistentVolume:
kubectl exec -ti pod-name -- /bin/bash
and trying to write something on the mounted filesystem.
The error message you get:
"Error 400: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE"
concerns specifically GCE Persistent Disk that is already mounted by one GKE node in ReadWriteOnce mode and it cannot be mounted by another node on which the rest of your Pods were scheduled.
If you want it to be mounted in ReadOnlyMany mode, you need to specify it explicitely in your Deployment definition by adding readOnly: true statement in the volumes section under Pod's template specification like below:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: nginx-content
volumes:
- name: nginx-content
persistentVolumeClaim:
claimName: nginx-content-claim
readOnly: true
Keep in mind however that to be able to mount it in readOnly mode, first we need to pre-populate such volume with data. Otherwise you'll see another error message, saying that unformatted volume cannot be mounted in read only mode.
The easiest way to do it is by creating a single Pod which will serve only for copying data which was already uploaded to one of our GKE nodes to our destination PV.
Note that pre-populating PersistentVolume with data can be done in many different ways. You can mount in such Pod only your PersistentVolume that you will be using in your Deployment and get your data using curl or wget from some external location saving it directly on your destination PV. It's up to you.
In my example I'm showing how to do it using additional local volume that allows us to mount into our Pod a directory, partition or disk (in my example I use a directory /var/tmp/test located on one of my GKE nodes) available on one of our kubernetes nodes. It's much more flexible solution than hostPath as we don't have to care about scheduling such Pod to particular node, that contains the data. Specific node affinity rule is already defined in PersistentVolume and Pod is automatically scheduled on specific node.
To create it we need 3 things:
StorageClass:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
PersistentVolume definition:
apiVersion: v1
kind: PersistentVolume
metadata:
name: example-pv
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /var/tmp/test
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- <gke-node-name>
and finally PersistentVolumeClaim:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myclaim
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 10Gi
storageClassName: local-storage
Then we can create our temporary Pod which will serve only for copying data from our GKE node to our GCE Persistent Disk.
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: myfrontend
image: nginx
volumeMounts:
- mountPath: "/mnt/source"
name: mypd
- mountPath: "/mnt/destination"
name: nginx-content
volumes:
- name: mypd
persistentVolumeClaim:
claimName: myclaim
- name: nginx-content
persistentVolumeClaim:
claimName: nginx-content-claim
Paths you can see above are not really important. The task of this Pod is only to allow us to copy our data to the destination PV. Eventually our PV will be mounted in completely different path.
Once the Pod is created and both volumes are successfully mounted, we can attach to it by running:
kubectl exec -ti my-pod -- /bin/bash
Withing the Pod simply run:
cp /mnt/source/* /mnt/destination/
That's all. Now we can exit and delete our temporary Pod:
kubectl delete pod mypod
Once it is gone, we can apply our Deployment and our PersistentVolume finally can be mounted in readOnly mode by all the Pods located on various GKE nodes:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: nginx-content
volumes:
- name: nginx-content
persistentVolumeClaim:
claimName: nginx-content-claim
readOnly: true
Btw. if you are ok with the fact that your Pods will be scheduled only on one particular node, you can give up on using GCE Persistent Disk at all and switch to the above mentioned local volume. This way all your Pods will be able not only to read from it but also to write to it at the same time. The only caveat is that all those Pods will be running on a single node.
You can achieve this with a NFS like file system. On Google Cloud, Filestore is the right product for this (NFS managed). You have a tutorial here for achieving your configuration
You will need to use a shared volume claim with ReadWriteMany (RWX) type if you want to share the volume across different nodes and provide highly scalable solution. Like using NFS server.
You can find out how to deploy an NFS server here:
https://www.shebanglabs.io/run-nfs-server-on-ubuntu-20-04/
And then you can mount volumes (directories from NFS server) as follows:
https://www.shebanglabs.io/how-to-set-up-read-write-many-rwx-persistent-volumes-with-nfs-on-kubernetes/
I've used such a way to deliver shared static content between +8 k8s deployments (+200 pods) serving 1 Billion requests a month over Nginx. and it did work perfectly with that NFS setup :)
Google provides NFS like filesystem called as Google Cloud Filestore. You can mount that on multiple pods.
We have success creating the pods, services and replication controllers according to our project requirements. Now we are planning to setup persistence storage in AWS using Kubernetes. I have created the YAML file to create an EBS volume in AWS, it's working fine as expected. I am able to claim volume and successfully mount to my pod (this is for single replica only).
But when I am trying to create more the one replica, my pods are not creating successfully. When I am trying to create volumes, it's creating in only one availability zone. If my pod is created in a different zone node, since my volume is already created in different zone, due to that my pod is not creating successfully. How to create volumes in different zones for same application? How to make it successful, along with replica? How to create my persistent volumes claims?
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mongo-pvc
labels:
type: amazonEBS
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: ReplicationController
metadata:
labels:
name: mongo-pp
name: mongo-controller-pp
spec:
replicas: 2
template:
metadata:
labels:
name: mongo-pp
spec:
containers:
- image: mongo
name: mongo-pp
ports:
- name: mongo-pp
containerPort: 27017
hostPort: 27017
volumeMounts:
- mountPath: "/opt/couchbase/var"
name: mypd1
volumes:
- name: mypd1
persistentVolumeClaim:
claimName: mongo-pvc
When you are using ReadWriteOnce volumes (ones that can not be mounted to multiple pods at the same time), simple PV/PVC creation will not cut it.
Both PV and PVC are pretty "singular" in a way that if you refer in Deployment to a particular claim name, your pods will all try to get the same one claim and the same one pv bound to that claim, resulting in a race condition where only one of the pods will be the first and only allowed to mount that RWO storage.
To mitigate this, you should use not PVC directly but via volumeClaimTemplates that will create PVC dynamicaly for every new pod scaled, like below :
volumeClaimTemplates:
- metadata:
name: claimname
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
I think the problem your a facing is caused by the underlying storage mechanism, in this case EBS.
When scaling Pods behind a replication controller, each replica will attempt to mount the same persistent volume. If you look at the K8 docs in regards to EBS, you will see the following:
There are some restrictions when using an awsElasticBlockStore volume:
the nodes on which pods are running must be AWS EC2 instances those
instances need to be in the same region and availability-zone as the
EBS volume EBS only supports a single EC2 instance mounting a volume
So by default, when you scale up behind a replication controller, Kubernetes will try to spread across different nodes, this means that a second node is trying to mount this volume which is not allowed for EBS.
Basically, I see that you have two options.
Use a different volume type. nfs, Glusterfs etc
Use a StatefulSet instead of a replication controller and have each replica mount an independent volume. Would require database replication but provide high availability.