I would like to use a single mount point on a node (ie /data) and have a different sub folder for each PersistentVolumeClaim that I am going to use in my cluster.
At the moment I have multiple StorageClass and PersistentVolume for each sub folder, for example:
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: prometheus
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: prometheus
labels:
type: local
spec:
storageClassName: prometheus
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
local:
path: "/data/prometheus"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: disk
operator: In
values:
- local
As you can image having a StorageClass, a PersistentVolume for each PersistentVolumeClaim looks a bit of an overkill.
I have tried to use a single StorageClass and PersistentVolume (just pointing to /data), the usePath option (ie prometheus) with multiple PersistentVolumeClaim.
But I have noticed that if the securityContext.fsGroupChangePolicy option is enabled it will apply the user/group changes to root of the volume (ie /data) not to the subPath (ie /data/prometheus).
Is there a better solution?
Thanks
As you can image having a StorageClass, a PersistentVolume for each
PersistentVolumeClaim looks a bit of an overkill.
That's exactly how dynamic storage provisioning works. Single storage class specified in PVC used by a pod will provision single PV for that PVC. There's nothing wrong with it. I'd suggest using it if you are ok with its default volume reclaim policy of delete.
local-path-provisioner seems to be a good solution.
I'ts been 60min now and my persistent volume claim is still pending.
My storage class:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
Minikube did not supply this one, I had to add it with the yaml above. In the dashboard I can click on it and it references the persistent volume which is green/ok.
My persistent volume (green, ok):
apiVersion: v1
kind: PersistentVolume
metadata:
name: small-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /data
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- minikube
The reason I need persistent storage is that nodered store its data in /data so that whats I'm trying to do here; provide it with persistent volume to store data. And since this is locally using minikube I can take advantage of /data folder on the minikube instance that per documentation is persistent.
My persistent volume claim for my nodered app.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nodered-claim
spec:
storageClassName: local-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
If I add the deployment or not, the persistent storage claim is still yellow/pending in the dashboard. Any reason for that? What am I missing here?
Update:
kubectl describe pvc/nodered-claim:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 2m52s (x162 over 42m) persistentvolume-controller waiting for first consumer to be created before binding
Update your StorageClass to immediate:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate # <-- bind as soon as PVC is created
WaitForFirstConsumer will only bind when a Pod uses your PVC.
...If I add the deployment or not, the persistent storage claim is still yellow/pending in the dashboard.
Your deployment will also enter pending state if the PVC it needs failed to bind.
I'm trying to leverage a local volume dynamic provisioner for k8s, Rancher's one, with multiple instances, each with its own storage class so that I can provide multiple types of local volumes based on their performance (e.g. ssd, hdd ,etc).
The underlying infrastructure is not symmetric; some nodes only have ssds, some only hdds, some of them both.
I know that I can hint the scheduler to select the proper nodes by providing node affinity rules for pods.
But, is there a better way to address this problem at the level of provisioner / storage class only ? E.g., make a storage class only available for a subset of the cluster nodes.
I know that I can hint the scheduler to select the proper nodes by
providing node affinity rules for pods.
There is no need to define node affinity rules on Pod level when using local persistent volumes. Node affinity can be specified in PersistentVolume definition.
But, is there a better way to address this problem at the level of
provisioner / storage class only ? E.g., make a storage class only
available for a subset of the cluster nodes.
No, it cannot be specified on a StorageClass level. Neither you can make a StorageClass available only for a subset of nodes.
But when it comes to a provisioner, I would say yes, it should be feasible as one of the major storage provisioner tasks is creating matching PersistentVolume objects in response to PersistentVolumeClaim created by the user. You can read about it here:
Dynamic volume provisioning allows storage volumes to be created
on-demand. Without dynamic provisioning, cluster administrators have
to manually make calls to their cloud or storage provider to create
new storage volumes, and then create PersistentVolume objects to
represent them in Kubernetes. The dynamic provisioning feature
eliminates the need for cluster administrators to pre-provision
storage. Instead, it automatically provisions storage when it is
requested by users.
So looking at the whole volume provision process from the very beginning it looks as follows:
User creates only PersistenVolumeClaim object, where he specifies a StorageClass:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myclaim
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 10Gi
storageClassName: local-storage ### 👈
and it can be used in a Pod definition:
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: myfrontend
image: nginx
volumeMounts:
- mountPath: "/var/www/html"
name: mypd
volumes:
- name: mypd
persistentVolumeClaim:
claimName: myclaim ### 👈
So in practice, in a Pod definition you need only to specify the proper PVC. No need for defining any node-affinity rules here.
A Pod references a PVC, PVC then references a StorageClass, StorageClass references the provisioner that should be used:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/my-fancy-provisioner ### 👈
volumeBindingMode: WaitForFirstConsumer
So in the end it is the task of a provisioner to create matching PersistentVolume object. It can look as follows:
apiVersion: v1
kind: PersistentVolume
metadata:
name: example-pv
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /var/tmp/test
nodeAffinity: ### 👈
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- ssd-node ### 👈
So a Pod which uses myclaim PVC -> which references the local-storage StorageClass -> which selects a proper storage provisioner will be automatically scheduled on the node selected in PV definition created by this provisioner.
I am trying to setup couchbase operator 1.2 on my local system.
i followed the following steps :
Install the Couchbase Admission Controller.
Deploy the Couchbase Autonomous Operator.
Deploy the Couchbase Cluster.
Access CouchBase from UI.
But the problem with this is that as soon as the system or docker resets or the pod resets, the cluster's data is lost.
So for the same I tried to do it by adding persistent volume with local storage class as mentioned in the docs but the result was still the same. The pod still gets resets. and i am unable to find the reason for the same.
So if anyone can advise on how to do the same with persistent volume on local storage class. I have successfully created a storage class. Just having problem while getting the cluster up and keep the consistency for the same.
Here is the yamls that i used to create the storage class and pv and pv claim
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: myssd
provisioner: local
apiVersion: v1
kind: PersistentVolume
metadata:
name: couchbase-data-2
labels:
type: local
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
storageClassName: myssd
hostPath:
path: "/home/<user>/cb-storage/"
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-test-claim-2
spec:
accessModes:
- ReadWriteOnce
storageClassName: myssd
resources:
requests:
storage: 1Gi
Thanks in advance
Persistent volume using hostPath is not durable. Use a local volume. Compared to hostPath volumes, local volumes can be used in a durable and portable manner without manually scheduling Pods to nodes, as the system is aware of the volume's node constraints by looking at the node affinity on the PersistentVolume.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: couchbase-data
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /home/<User>/cb-storage/
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node1
- node2
- node3
- node4
You don't need to create a PersistentVolume manually because the storage class will do that internally.
Also you need to configure the local volume provisioner as discussed here so that dynamic provisioning using the local storage class happens.
I setup a new k8s in a single node, which is tainted. But the PersistentVolume can not be created successfully, when I am trying to create a simple PostgreSQL.
There is some detail information below.
The StorageClass is copied from the official page: https://kubernetes.io/docs/concepts/storage/storage-classes/#local
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
The StatefulSet is:
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
name: postgres
spec:
serviceName: postgres
replicas: 1
...
volumeClaimTemplates:
- metadata:
name: postgres-data
spec:
storageClassName: local-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
About the running StorageClass:
$ kubectl describe storageclasses.storage.k8s.io
Name: local-storage
IsDefaultClass: No
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"local-storage"},"provisioner":"kubernetes.io/no-provisioner","volumeBindingMode":"WaitForFirstConsumer"}
Provisioner: kubernetes.io/no-provisioner
Parameters: <none>
AllowVolumeExpansion: <unset>
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: WaitForFirstConsumer
Events: <none>
About the running PersistentVolumeClaim:
$ kubectl describe pvc
Name: postgres-data-postgres-0
Namespace: default
StorageClass: local-storage
Status: Pending
Volume:
Labels: app=postgres
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer <invalid> (x2 over <invalid>) persistentvolume-controller waiting for first consumer to be created before binding
K8s versions:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:31:33Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
The app is waiting for the Pod, while the Pod is waiting for a PersistentVolume by a PersistentVolumeClaim.
However, the PersistentVolume should be prepared by the user before using.
My previous YAMLs are lack of a PersistentVolume like this:
kind: PersistentVolume
apiVersion: v1
metadata:
name: postgres-data
labels:
type: local
spec:
storageClassName: local-storage
capacity:
storage: 1Gi
local:
path: /data/postgres
persistentVolumeReclaimPolicy: Retain
accessModes:
- ReadWriteOnce
storageClassName: local-storage
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: app
operator: In
values:
- postgres
The local path /data/postgres should be prepared before using.
Kubernetes will not create it automatically.
I just ran into this myself and was completely thrown for a loop until I realized that the StorageClass's VolumeBindingMode was set to WaitForFirstConsumer vice my intended value of Immediate. This value is immutable so you will have to:
Get the storage class yaml:
kubectl get storageclasses.storage.k8s.io gp2 -o yaml > gp2.yaml
or you can also just copy the example from the docs here (make sure the metadata names match). Here is what I have configured:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp2
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
- debug
volumeBindingMode: Immediate
And delete the old StorageClass before recreating it with the new volumeBindingMode set to Immediate.
Note: The EKS clsuter may need perms to create cloud resources like EBS or EFS. Assuming EBS you should be good with arn:aws:iam::aws:policy/AmazonEKSClusterPolicy.
After doing this you should have no problem creating and using dynamically provisioned PVs.
In my case, I had claimRef without specified namespace.
Correct syntax is:
claimRef:
namespace: default
name: my-claim
StatefulSet also prevented initialization, I had to replace it with a deployment
This was a f5g headache
For me the problem was mismatched accessModes fields in the PV and PVC. PVC was requesting RWX/ReadWriteMany while PV was offering RWO/ReadWriteOnce.
The accepted answer didn't work for me. I think it's because the app key won't be set before the the StatefulSet's Pods are deployed, preventing the PersistentVolumeClaim to match the nodeSelector (preventing the Pods to start with the error didn't find available persistent volumes to bind.). To fix this deadlock, I defined one PersistentVolume for each node (this may not be ideal but it worked):
apiVersion: v1
kind: PersistentVolume
metadata:
name: postgres-data-node1
labels:
type: local
spec:
[…]
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node1
I'm stuck in this vicious loop myself.
I'm trying to create a kubegres cluster (which relies on dynamic provisioning as per my understanding).
I'm using RKE on a local-servers-like setup.
and I have the same scheduling issue as the one initially mentioned.
noting that the accessmode of the PVC (created by kubegres) is set to nothing as per the below output.
[rke#rke-1 manifests]$ kubectl get pv,PVC
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/local-vol 20Gi RWO Delete Available local-storage 40s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/local-vol-mypostgres-1-0 Pending local-storage 6m42s
persistentvolumeclaim/postgres-db-mypostgres-1-0 Pending local-storage 6m42s
As an update, the issue in my case was that the PVC was not finding a Proper PV which was supposed to be dynamically provisioned. But for local storage classes, this feature is not yet supported therefore I had to use a third-party solution which solved my issue.
https://github.com/rancher/local-path-provisioner
This issue mainly happens with WaitForFirstConsumer when you define the nodeName in the Deployment/Pod specifications. Please make sure you don't define nodeName and hardbind the pod through it. The should be resolved once you remove nodeName.
I believe this can be a valid message that means that there are no containers started that have volumes that are bound to the persistent volume claim.
I experienced this issue on rancher desktop. It turned out the problem was caused by rancher not running properly after a macOS upgrade. The containers were not starting and would stay in a pending state.
After reseting the rancher desktop (using the UI), the containers were able to start well and the message disappeared.
waitforfirstconsumer-persistentvolumeclaim i.e. POD which requires this PVC is not scheduled. describe pods may give some more clue. In my case node was not able to schedule this POD since pod limit in node was 110 and deployment was exceeding it. Hope it helps to identify issue faster. increased the pod limit , restart kubelet in node solves it.