I created an EBS volume with 30 GiB size. Made two manifest files:
pv-ebs.yml
pvc-ebs.yml
In pv-ebs.yml:
apiVersion: v1
kind: PersistentVolume
metadata:
name: ebs
spec:
capacity:
storage: 30Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
awsElasticBlockStore:
fsType: ext4
# The EBS volume ID
volumeID: vol-111222333aaabbbccc
in pvc-ebs.yml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: prometheus-prometheus-alertmanager
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
selector:
matchLabels:
release: "stable"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: prometheus-prometheus-server
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
selector:
matchLabels:
release: "stable"
Use helm installed it: helm install --name prometheus stable/prometheus.
But on the k8s dashboard, got message:
prometheus-prometheus-alertmanager-3740839786-np7kb
No nodes are available that match all of the following predicates:: NoVolumeZoneConflict (2).
prometheus-prometheus-server-3176041168-m3w2g
PersistentVolumeClaim is not bound: "prometheus-prometheus-server" (repeated 2 times)
Is there anything wrong about my method?
Pods
Persistent Volumes
When EBS is created, it is provisioned in a particular AZ and it can not be cross-zone mounted. If you do not have nodes available in the same zone for scheduling the pod, it will not start.
Another thing is that with a properly configured kube cluster, you should not need to create PV on your own at all, just create PVC and let dynamic provisioning do it's thing.
If you installed your cluster with KOPs the PVs will be created for you automatically. Just wait a few min and refresh your screen. The errors will go away.
If you have setup your cluster in another way you want to create your volumes in AWS ec2 create-volume and then create PVs and then when helm runs it will claim those PVs.
Related
I'm trying to deploy a MongoDB replica set by using the MongoDB Community Kubernetes Operator in Minikube.
I followed the instructions on the official GitHub, so:
Install the CRD
Install the necessary roles and role-bindings
Install the Operator Deploy the Replicaset
By default, the operator will creates three pods, each of them automatically linked to a new persistent volume claim bounded to a new persistent volume also created by the operator (so far so good).
However, I would like the data to be saved in a specific volume, mounted in a specific host path. So in order I would need to create three persistent volumes, each mounted to a specific host path, and then automatically I would want to configure the replicaset so that each pod would connect to its respective persistent volume (perhaps using the matchLabels selector).
So I created three volumes by applying the following file:
apiVersion: v1
kind: PersistentVolume
metadata:
name: mongodb-pv-00
namespace: $NAMESPACE
labels:
type: local
service: mongo
spec:
storageClassName: manual
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/mongodata/00"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: mongodb-pv-01
namespace: $NAMESPACE
labels:
type: local
service: mongo
spec:
storageClassName: manual
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/mongodata/01"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: mongodb-pv-02
namespace: $NAMESPACE
labels:
type: local
service: mongo
spec:
storageClassName: manual
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/mongodata/02"
and then I set up the replica set configuration file in the following way, but it still fails to connect the pods to the volumes:
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
name: mongo-rs
namespace: $NAMESPACE
spec:
members: 3
type: ReplicaSet
version: "4.4.0"
persistent: true
podSpec:
persistence:
single:
labelSelector:
matchLabels:
type: local
service: mongo
storage: 5Gi
storageClass: manual
statefulSet:
spec:
volumeClaimTemplates:
- metadata:
name: data-volume
spec:
accessModes: [ "ReadWriteOnce", "ReadWriteMany" ]
resources:
requests:
storage: 5Gi
selector:
matchLabels:
type: local
service: mongo
storageClassName: manual
security:
authentication:
modes: ["SCRAM"]
users:
- ...
additionalMongodConfig:
storage.wiredTiger.engineConfig.journalCompressor: zlib
I can't find any documentation online, except the mongodb.com_v1_custom_volume_cr.yaml, has anyone faced this problem before? How could I make it work?
I think you could be interested into using local type of volumes. It works, like this:
First, you create a storage class for the local volumes. Something like the following:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
Since it has no-provisioner, it will be usable only if you manually create local PVs. WaitForFirstConsumer instead, will prevent attaching a PV to a PVC of a Pod which cannot be scheduled on the host on which the PV is available.
Second, you create the local PVs. Similarly to how you created them in your example, something like this:
apiVersion: v1
kind: PersistentVolume
metadata:
name: example-pv
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /path/on/the/host
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- the-node-hostname-on-which-the-storage-is-located
Notice the definition, it tells the path on the host, the capacity.. and then it explains on which node of the cluster, such PV can be used (with the nodeAffinity). It also link them to the storage class we created early.. so that if someone (a claim template) requires storage with that class, it will now find this PV.
You can create 3 PVs, on 3 different nodes.. or 3 PVs on the same node at different paths, you can organize things as you desire.
Third, you can now use the local-storage class in claim template. The claim template could be something similar to this:
volumeClaimTemplates:
- metadata:
name: the-name-of-the-pvc
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "local-storage"
resources:
requests:
storage: 5Gi
And each Pod of the StatefulSet will try to be scheduled on a node with a local-storage PV available.
Remember that with local storages or, in general, with volumes that utilize host paths.. you may want to spread the various Pods of your app on different nodes, so that the app may resist the failure of a single node on its own.
In case you want to be able to decide which Pod links to which volume, the easiest way is to create one PV at a time, then wait for the Pod to Bound with it.. before creating the next one. It's not optimal but it's the easiest way.
I am trying to setup couchbase operator 1.2 on my local system.
i followed the following steps :
Install the Couchbase Admission Controller.
Deploy the Couchbase Autonomous Operator.
Deploy the Couchbase Cluster.
Access CouchBase from UI.
But the problem with this is that as soon as the system or docker resets or the pod resets, the cluster's data is lost.
So for the same I tried to do it by adding persistent volume with local storage class as mentioned in the docs but the result was still the same. The pod still gets resets. and i am unable to find the reason for the same.
So if anyone can advise on how to do the same with persistent volume on local storage class. I have successfully created a storage class. Just having problem while getting the cluster up and keep the consistency for the same.
Here is the yamls that i used to create the storage class and pv and pv claim
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: myssd
provisioner: local
apiVersion: v1
kind: PersistentVolume
metadata:
name: couchbase-data-2
labels:
type: local
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
storageClassName: myssd
hostPath:
path: "/home/<user>/cb-storage/"
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-test-claim-2
spec:
accessModes:
- ReadWriteOnce
storageClassName: myssd
resources:
requests:
storage: 1Gi
Thanks in advance
Persistent volume using hostPath is not durable. Use a local volume. Compared to hostPath volumes, local volumes can be used in a durable and portable manner without manually scheduling Pods to nodes, as the system is aware of the volume's node constraints by looking at the node affinity on the PersistentVolume.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: couchbase-data
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /home/<User>/cb-storage/
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node1
- node2
- node3
- node4
You don't need to create a PersistentVolume manually because the storage class will do that internally.
Also you need to configure the local volume provisioner as discussed here so that dynamic provisioning using the local storage class happens.
I'm fairly new to Kubernetes and find it difficult to get it working from documentation, Kubenetes docs says that StorageClass contains the fields provisioner, parameters, and reclaimPolicy, which are used when a PersistentVolume belonging to the class needs to be dynamically provisioned however can I use StorageClass with PV(not dynamic allocation) to specify high performance disk allocation such as ssd?
without StorageClass it worked fine for me.
following is my manifest
kind: PersistentVolume
metadata:
name: gke-pv
labels:
app: test
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
gcePersistentDisk:
pdName: gce-disk
fsType: ext4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gke-pvc
labels:
app: test
spec:
accessModes:
- ReadWriteOnce
storageClassName: ssd-sc
resources:
requests:
storage: 2Gi
selector:
matchLabels:
app: test
You need storage class if the storage needs to be provisioned dynamically.
If you are provisioning persistent volumes then it is called static storage provisioning. You don't need storage class in this scenario
The problem that is going on here is that if you want to statically provision PersistentVolumes, they don't have a StorageClass. However, GKE clusters are created with a standard StorageClass which is the default, and so the PVC gets confused and tries to dynamically allocate.
The solution is to have the PVC request an empty storage class, which forces it to look at the statically provisioned PVs.
So you'd use a sequence like this to create a PV and then get it bound to a PVC:
Manually provision the ssd:
gcloud compute disks create --size=10Gi --zone=[YOUR ZONE] --type=pd-ssd already-created-ssd-disk
Then apply a PV object that uses the statically provisioned disk, like so:
apiVersion: v1
kind: PersistentVolume
metadata:
name: ssd-for-k8s-volume
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
gcePersistentDisk:
pdName: already-created-ssd-disk
fsType: ext4
Then, you can claim it with a PVC like this:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-ssd-demo
spec:
storageClassName: ""
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
You could also use labels to refine which PVs are selected, of course, for example if you have some that are SSD and others that are regular spinning metal.
Note that the idea of using a StorageClass for static provisioning isn't really the right thing, since StorageClass is tied to how you describe storage for dynamic provisioning.
I applied my PVC yaml file to my GKE cluster and checked it's state.
It says the follwing for the yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"teamcity","namespace":"default"},"spec":{"accessModes":["ReadWriteMany"],"resources":{"requests":{"storage":"3Gi"}}}}
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/gce-pd
creationTimestamp: "2019-11-05T09:45:20Z"
finalizers:
- kubernetes.io/pvc-protection
name: teamcity
namespace: default
resourceVersion: "1358093"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/teamcity
uid: fb51d295-ffb0-11e9-af7d-42010a8400aa
spec:
accessModes:
- ReadWriteMany
dataSource: null
resources:
requests:
storage: 3Gi
storageClassName: standard
volumeMode: Filesystem
status:
phase: Pending
I did not created anything like a storage or whatever needs to be done for that? Because I read it as this is provided automatically by the GKE. Any idea what I am missing?
GKE includes default support for GCP disk PV provisioning, however those implement ReadWriteOnce and ReadOnlyMany modes. I do not think GKE includes a provisioner for ReadWriteMany by default.
EDIT: While it's not set up by default (because it requires further configuration) How do I create a persistent volume claim with ReadWriteMany in GKE? shows how to use Cloud Filestore to launch a hosted NFS-compatible server and then aim a provisioner at it.
StorageClasses are the new method of specifying dynamic persistent volume claim (PVC) dependencies within Kubernetes. This avoids the need to explicitly provision one directly with the cloud provider (in my case Google Container Engine (GKE)).
Definition for the StorageClasses (GKE already has a default for standard class)
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-ssd
zone: europe-west1-b
Definition for the actual PVC
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs-server-pvc
namespace: staging
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 100Gi
storageClassName: "standard"
Here is the result of kubernetes get storageclass:
NAME TYPE
fast kubernetes.io/gce-pd
standard (default) kubernetes.io/gce-pd
Here is the result of kubernetes get pvc:
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
nfs-pvc Bound nfs 1Mi RWX 119d
nfs-server-pvc Bound pvc-905a810b-3f13-11e7-82f9-42010a840072 100Gi RWO standard 81d
I would like to continue taking snapshots of the volumes but the dynamic nature of the volume names created (in this case pvc-905a810b-3f13-11e7-82f9-42010a840072), mean i cannot continue with the following command that i had been using via cron (note the "nfs" name is now incorrect):
gcloud compute --project "XXX-XXX" disks snapshot "nfs" --zone "europe-west1-b" --snapshot-names "nfs-${DATE}"
I guess this boils down to Kubernetes allowing explicit naming through StorageClass-based PVC. The docs don't seem to allow this. Any ideas?
One approach is to manually create the PV and give it a stable name that you can use in your scripts. You can use gcloud commands to create the underlying PD disks. When you create the PV, give it a label:
apiVersion: "v1"
kind: "PersistentVolume"
metadata:
name: my-pv-0
labels:
pdName: my-pv-0
spec:
capacity:
storage: "10Gi"
accessModes:
- "ReadWriteOnce"
storageClassName: fast
gcePersistentDisk:
fsType: "ext4"
pdName: "my-pd-0"
Then attach it to the PVC using a selector:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: my-pvc-0
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: fast
selector:
matchLabels:
pdName: my-pv-0