I'm aware you can use aws cli to create ebs volume and then get the Volume ID and add to PersistentVolume config as below under the volumeID.
I don't want to use aws cli to create the ebs volume, My question is, how do I use Kubernetes to create this ebs volume dynamically without using the cli ?
apiVersion: "v1"
kind: "PersistentVolume"
metadata:
name: "pv0001"
spec:
capacity:
storage: "5Gi"
accessModes:
- "ReadWriteOnce"
awsElasticBlockStore:
fsType: "ext4"
volumeID: "volume-ID"
By default this should be working on a decently provisioned cluster. Just have the storageClassName defined correctly on a matching PVC and a PV will et provisioned for it (no need to precreate PV object, just the claim)
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims
Dynamic provisioning
When none of the static PVs the administrator created matches a user’s PersistentVolumeClaim, the cluster may try to dynamically provision a volume specially for the PVC. This provisioning is based on StorageClasses: the PVC must request a class and the administrator must have created and configured that class in order for dynamic provisioning to occur. Claims that request the class "" effectively disable dynamic provisioning for themselves
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#provisioning
Follow this:
https://docs.docker.com/ee/ucp/kubernetes/storage/configure-aws-storage/
Basically, instances must have an IAM Role to create/attach/detach/delete volumes on their own.
Related
My GKE deployment consists of N pods (possibly on different nodes) and a shared volume, which is dynamically provisioned by pd.csi.storage.gke.io and is a Persistent Disk in GCP. I need to initialize this disk with data before the pods go live.
My problem is I need to set accessModes to ReadOnlyMany and be able to mount it to all pods across different nodes in read-only mode, which I assume effectively would make it impossible to mount it in write mode to the initContainer.
Is there a solution to this issue? Answer to this question suggests a good solution for a case when each pod has their own disk mounted, but I need to have one disk shared among all pods since my data is quite large.
With GKE 1.21 and later, you can enable the managed Filestore CSI driver in your clusters. You can enable the driver for new clusters
gcloud container clusters create CLUSTER_NAME \
--addons=GcpFilestoreCsiDriver ...
or update existing clusters:
gcloud container clusters update CLUSTER_NAME \
--update-addons=GcpFilestoreCsiDriver=ENABLED
Once you've done that, create a storage class (or have or platform admin do it):
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: filestore-example
provisioner: filestore.csi.storage.gke.io
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
tier: standard
network: default
After that, you can use PVCs and dynamic provisioning:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: podpvc
spec:
accessModes:
- ReadWriteMany
storageClassName: filestore-example
resources:
requests:
storage: 1Ti
...I need to have one disk shared among all pods
You can try Filestore. First your create a FileStore instance and save your data on a FileStore volume. Then you install FileStore driver on your cluster. Finally you share the data with pods that needs to read the data using a PersistentVolume referring the FileStore instance and volume above.
I am new to Kubernetes, and I struggle to understand whol idea behind Persistent Storage in Kubernetes.
So is this enough or I have to create Persistent Volume and what will happen if I deploy only these two object without creating PV?
Storage should be on local machine.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: nginx-logs
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
status: {}
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: app-web
name: app-web
spec:
selector:
matchLabels:
app: app-web
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: app-web
spec:
containers:
image: nginx:1.14.2
imagePullPolicy: Always
name: app-web
volumeMounts:
- mountPath: /var/log/nginx
name: nginx-logs
restartPolicy: Always
volumes:
- name: nginx-logs
persistentVolumeClaim:
claimName: nginx-logs
I struggle to understand whole idea behind Persistent Storage in Kubernetes
The idea is to separate the storage request that the app needs, and the physical storage - such that an app can be moved to e.g. other cloud provider that has a different storage system - but without needing any changes in the app. It also separates the responsibility for "requesting storage" and managing the underlying storage e.g. developers vs operations.
So is this enough or I have to create Persistent Volume and what will happen if I deploy only these two object without creating PV?
This depends on you environment. Most environments typically have Dynamic Volume Provisioning, e.g. the big cloud providers and now also Minikube has support for this.
When using dynamic volume provisioning, the developer only has to create a PersistentVolumeClaim - and no PersistentVolume, its instead dynamically provsioned.
A PV which matches with the PVC in terms of capacity(100Mi) and accessModes(ReadWriteOnce)is needed.If you don't have a PV you get error pod has unbound immediate PersistentVolumeClaims. So either you create the PV manually(called static provisioning) or rely on storage class and volume driver to do it automatically(called dynamic provisioning).
A PV is a kubernetes representation of volume. The actual volume still need to be provisioned.If you use dynamic volume provisioning and your cloud provider has a volume driver the provisioning is automated internally by the driver without needing to create a PV manually.
In a local non cloud system you can use local path provisioner to use dynamic provisioning. Configure a default storage class and you can avoid manual PV creation.
You can see in the doc that it mentions a Dynamic way of provisioning persistent volumes.
When none of the static PVs the administrator created match a user's
PersistentVolumeClaim, the cluster may try to dynamically provision a
volume specially for the PVC. This provisioning is based on
StorageClasses: the PVC must request a storage class and the
administrator must have created and configured that class for dynamic
provisioning to occur. Claims that request the class "" effectively
disable dynamic provisioning for themselves.
When I apply your pvc, it creates a gp2 volume because that is my default storage class.
$kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nginx-logs Bound pvc-b95a9d0c-ef46-4ff0-a034-d2dde1ac1f96 1Gi RWO gp2 6s
You can see your default storage class like this
$kubectl get storageclass
NAME PROVISIONER AGE
gp2 (default) kubernetes.io/aws-ebs 355d
You can also include a specific storage class. You can read more about that here.
Users request dynamically provisioned storage by including a storage
class in their PersistentVolumeClaim
In short, your current pvc file will use your default storage class to create a Persistent Volume and then you are mounting that volume into your pod via your Persistent Volume Claim in your Deployment.
I want to setup a pre-defined PostgreSQL cluster in a bare meta kubernetes 1.7 with local PV enable. I have three work nodes. I create local PV on each node and deploy the stateful set successfully (with some complex script to setup Postgres replication).
However I'm noticed that there's a kind of naming convention between the volumeClaimTemplates and PersistentVolumeClaim.
For example
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: postgres
volumeClaimTemplates:
- metadata:
name: pgvolume
The created pvc are pgvolume-postgres-0, pgvolume-postgres-1, pgvolume-postgres-2 .
With some tricky, I manually create PVC and bind to the target PV by selector. I test the stateful set again. It seems the stateful set is very happy to use these PVC.
I finish my test successfully but I still have this question. Can I rely on volumeClaimTemplates naming convention? Is this an undocumented feature?
Based on the statefulset API reference
volumeClaimTemplates is a list of claims that pods are allowed to reference. The StatefulSet controller is responsible for mapping network identities to claims in a way that maintains the identity of a pod. Every claim in this list must have at least one matching (by name) volumeMount in one container in the template. A claim in this list takes precedence over any volumes in the template, with the same name.
So I guess you can rely on it.
Moreover, you can define a storage class to leverage dynamic provisioning of persistent volumes, so you won't have to create them manually.
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: my-storage-class
resources:
requests:
storage: 1Gi
Please refer to Dynamic Provisioning and Storage Classes in Kubernetes for more details.
Use case:
I have a NFS directory available and I want to use it to persist data for multiple deployments & pods.
I have created a PersistentVolume:
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
nfs:
server: http://mynfs.com
path: /server/mount/point
I want multiple deployments to be able to use this PersistentVolume, so my understanding of what is needed is that I need to create multiple PersistentVolumeClaims which will all point at this PersistentVolume.
kind: PersistentVolumeClaim
apiVersion: v1
metaData:
name: nfs-pvc-1
namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 50Mi
I believe this to create a 50MB claim on the PersistentVolume. When I run kubectl get pvc, I see:
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
nfs-pvc-1 Bound nfs-pv 10Gi RWX 35s
I don't understand why I see 10Gi capacity, not 50Mi.
When I then change the PersistentVolumeClaim deployment yaml to create a PVC named nfs-pvc-2 I get this:
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
nfs-pvc-1 Bound nfs-pv 10Gi RWX 35s
nfs-pvc-2 Pending 10s
PVC2 never binds to the PV. Is this expected behaviour? Can I have multiple PVCs pointing at the same PV?
When I delete nfs-pvc-1, I see the same thing:
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
nfs-pvc-2 Pending 10s
Again, is this normal?
What is the appropriate way to use/re-use a shared NFS resource between multiple deployments / pods?
Basically you can't do what you want, as the relationship PVC <--> PV is one-on-one.
If NFS is the only storage you have available and would like multiple PV/PVC on one nfs export, use Dynamic Provisioning and a default storage class.
It's not in official K8s yet, but this one is in the incubator and I've tried it and it works well: https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client
This will enormously simplify your volume provisioning as you only need to take care of the PVC, and the PV will be created as a directory on the nfs export / server that you have defined.
From: https://docs.openshift.org/latest/install_config/storage_examples/shared_storage.html
As Baroudi Safwen mentioned, you cannot bind two pvc to the same pv, but you can use the same pvc in two different pods.
volumes:
- name: nfsvol-2
persistentVolumeClaim:
claimName: nfs-pvc-1 <-- USE THIS ONE IN BOTH PODS
A persistent volume claim is exclusively bound to a persistent volume.
You cannot bind 2 pvc to the same pv. I guess you are interested in the dynamic provisioning. I faced this issue when I was deploying statefulsets, which require dynamic provisioning for pods. So you need to deploy an NFS provisioner in your cluster, the NFS provisioner(pod) will have access to the NFS folder(hostpath), and each time a pod requests a volume, the NFS provisioner will mount it in the NFS directory on behalf of the pod. Here is the github repository to deploy it:
https://github.com/kubernetes-incubator/external-storage/tree/master/nfs/deploy/kubernetes
You have to be careful though, you must ensure the nfs provisioner always runs on the same machine where you have the NFS folder by making use of the node selector since you the volume is of type hostpath.
For my future-self and everyone else looking for the official documentation:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#binding
Once bound, PersistentVolumeClaim binds are exclusive, regardless of
how they were bound. A PVC to PV binding is a one-to-one mapping,
using a ClaimRef which is a bi-directional binding between the
PersistentVolume and the PersistentVolumeClaim.
a few points on dynamic provisioning..
using dynamic provisioning of nfs prevents you for changing any of the default nfs mount options. On my platform this uses rsize/wsize of 1M. this can cause huge problems in some applications using small files or block reading. (I've just hit this issue in a big way)
dynamic is a great option if it suits your needs. I'm now stuck with creating 250 pv/pvc pairs for my application that was being handled by dynamic due to the 1-1 relationship.
I am trying to configure Pet Set in Kubernetes 1.3. There is a NFS storage backend that works as expected with Persistent Volumes, Claims, and Pods. However, when I create a Pet Set the VolumeClaimTemplate is never bound to the Persistent Volume backed by NFS.
Below is the definition of the PV -
apiVersion: v1
kind: PersistentVolume
metadata:
name: janipv
spec:
capacity:
storage: 3Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Recycle
nfs:
path: /var/data/test
server: 172.17.4.101
When I try running the Pet Set sample available in the documentation, I see the following message.
kubelet describe pv
{persistentvolume-controller } Warning ProvisioningFailed No provisioner plugin found for the claim!
The same configuration works fine when I use normal Pod instead of a Pet Set.
We don't have an NFS dynamic provisioner, like the error indicates: http://kubernetes.io/docs/user-guide/petset/#alpha-limitations (unless you wrote one, in which case we can start debugging). The volumeclaimtemplate itself never gets bound to anything. It's a template for the creation of volume claims, i.e the controller will create pvc-0, pvc-1... for pet-0,pet-1... those claims are bound to persistent volumes through a dynamic volume provisioner (as opposed to a static volume provisioning process where a human creates the pv and attaches it to the pvc).
You can hand attach an existing pv to one of the petset claims, just like you would for a replication controller's pods, by setting the volumeName field of the pvc.