I'm trying to specify Local SSD in a Google Cloud as a PersistedVolume. I followed the docs to set up the automated SSD provisioning, and running kubectl get pv returns a valid volume:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
local-pv-9721c951 368Gi RWO Delete Available local-scsi 1h
The problem is that I cannot get my pod to bind to it. The kubectl get pvc keeps showing this:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mapdata Pending local-scsi 7m
and kubectl get events gives me these:
LAST SEEN FIRST SEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
7m 7m 1 v3tiles.1551c0bbcb23d983 Service Normal EnsuredLoadBalancer service-controller Ensured load balancer
2m 8m 24 maptilesbackend-8645566545-x44nl.1551c0ae27d06fca Pod Warning FailedScheduling default-scheduler 0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind.
2m 8m 26 mapdata.1551c0adf908e362 PersistentVolumeClaim Normal WaitForFirstConsumer persistentvolume-controller waiting for first consumer to be created before binding
What would i need to do to bind that SSD to my pod? Here's the code I have been experimenting with:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: maptilesbackend
namespace: default
spec:
selector:
matchLabels:
app: maptilesbackend
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: maptilesbackend
spec:
containers:
- image: klokantech/openmaptiles-server
imagePullPolicy: Always
name: maptilesbackend
volumeMounts:
- mountPath: /data
name: mapdata
readOnly: true
volumes:
- name: mapdata
persistentVolumeClaim:
claimName: mapdata
readOnly: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: "local-scsi"
provisioner: "kubernetes.io/no-provisioner"
volumeBindingMode: "WaitForFirstConsumer"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mapdata
spec:
storageClassName: local-scsi
accessModes:
- ReadOnlyMany
resources:
requests:
storage: 300Gi
ReadOnlyMany doesn't make sense for local SSDs
As per the docs:
ReadOnlyMany – the volume can be mounted read-only by many nodes
You can't mount a local SSD on many nodes because it's local to one node only.
It turns out that accessMode: ReadOnlyMany does not work in this case. Not sure how to make it work... will post if I find more information.
Related
As the documentation states:
For each VolumeClaimTemplate entry defined in a StatefulSet, each Pod
receives one PersistentVolumeClaim. In the nginx example above, each
Pod receives a single PersistentVolume with a StorageClass of
my-storage-class and 1 Gib of provisioned storage. If no StorageClass
is specified, then the default StorageClass will be used. When a Pod
is (re)scheduled onto a node, its volumeMounts mount the
PersistentVolumes associated with its PersistentVolume Claims. Note
that, the PersistentVolumes associated with the Pods' PersistentVolume
Claims are not deleted when the Pods, or StatefulSet are deleted. This
must be done manually.
The part I'm interested in is this: If no StorageClassis specified, then the default StorageClass will be used
I create a StatefulSet like this:
apiVersion: apps/v1
kind: StatefulSet
metadata:
namespace: ches
name: ches
spec:
serviceName: ches
replicas: 1
selector:
matchLabels:
app: ches
template:
metadata:
labels:
app: ches
spec:
serviceAccountName: ches-serviceaccount
nodeSelector:
ches-worker: "true"
volumes:
- name: data
hostPath:
path: /data/test
containers:
- name: ches
image: [here I have the repo]
imagePullPolicy: Always
securityContext:
privileged: true
args:
- server
- --console-address
- :9011
- /data
env:
- name: MINIO_ACCESS_KEY
valueFrom:
secretKeyRef:
name: ches-keys
key: access-key
- name: MINIO_SECRET_KEY
valueFrom:
secretKeyRef:
name: ches-keys
key: secret-key
ports:
- containerPort: 9000
hostPort: 9011
resources:
limits:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: data
mountPath: /data
imagePullSecrets:
- name: edge-storage-token
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
Of course I have already created the secrets, imagePullSecrets etc and I have labeled the node as ches-worker.
When I apply the yaml file, the pod is in Pending status and kubectl describe pod ches-0 -n ches gives the following error:
Warning FailedScheduling 6s default-scheduler 0/1 nodes are
available: 1 pod has unbound immediate PersistentVolumeClaims.
preemption: 0/1 nodes are available: 1 Preemption is not helpful for
scheduling
Am I missing something here?
You need to create a PV in order to get a PVC bound. If you want the PVs automatically created from PVC claims you need a Provisioner installed in your Cluster.
First create a PV with at least the amout of space need by your PVC.
Then you can apply your deployment yaml which contains the PVC claim.
K3s when installed, also downloads a storage class which makes it as default.
Check with kubectl get storageclass:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE
ALLOWVOLUMEEXPANSION AGE local-path rancher.io/local-path Delete
WaitForFirstConsumer false 8s
K8s cluster on the other hand, does not download also a default storage class.
In order to solve the problem:
Download rancher.io/local-path storage class:
kubectl apply -f
https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
Check with kubectl get storageclass
Make this storage class (local-path) the default:
kubectl patch
storageclass local-path -p '{"metadata":
{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
RabbitMQ cluster operator does not work in Kubernetes.
I have a kubernetes cluster 1.17.17 of 3 nodes. I deployed it with a rancher.
According to this instruction I installed RabbitMQ cluster-operator:
https://www.rabbitmq.com/kubernetes/operator/quickstart-operator.html
kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"
Its ok! but..
I have created this very simple configuration for the instance according to the documentation:
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rabbitmq
namespace: test-rabbitmq
I have error:error while running "VolumeBinding" filter plugin for pod "rabbitmq-server-0": pod has unbound immediate PersistentVolumeClaims
after that i checked:
kubectl get storageclasses
and saw that there were no resources! I added the following storegeclasses:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
create pv and pvc:
kind: PersistentVolume
apiVersion: v1
metadata:
name: rabbitmq-data-sigma
labels:
type: local
namespace: test-rabbitmq
annotations:
volume.alpha.kubernetes.io/storage-class: rabbitmq-data-sigma
spec:
storageClassName: local-storage
capacity:
storage: 3Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Recycle
hostPath:
path: "/opt/rabbitmq-data-sigma"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: rabbitmq-data
namespace: test-rabbitmq
spec:
storageClassName: local-storage
accessModes:
- ReadWriteMany
resources:
requests:
storage: 3Gi
I end up getting an error in the volume - which is being generated automated:
FailedBinding no persistent volumes available for this claim and no storage class is set
please help to understand this problem!
You can configure Dynamic Volume Provisioning e.g. Dynamic NFS provisioning as describe in this article or you can manually create PersistentVolume ( it is NOT recommended approach).
I really recommend you to configure dynamic provisioning -
this will allow you to generate PersistentVolumes automatically.
Manually creating PersistentVolume
As I mentioned it isn't recommended approach but it may be useful when we want to check something quickly without configuring additional components.
First you need to create PersistentVolume:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
hostPath:
path: /mnt/rabbitmq # data will be stored in the "/mnt/rabbitmq" directory on the worker node
type: Directory
And then create the /mnt/rabbitmq directory on the node where the rabbitmq-server-0 Pod will be running. In your case you have 3 worker node so it may difficult to determine where the Pod will be running.
As a result you can see that the PersistentVolumeClaim was bound to the newly created PersistentVolume and the rabbitmq-server-0 Pod was created successfully:
# kubectl get pv,pvc -A
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pv 10Gi RWO Recycle Bound test-rabbitmq/persistence-rabbitmq-server-0 11m
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-rabbitmq persistentvolumeclaim/persistence-rabbitmq-server-0 Bound pv 10Gi RWO 11m
# kubectl get pod -n test-rabbitmq
NAME READY STATUS RESTARTS AGE
rabbitmq-server-0 1/1 Running 0 11m
I have deployed my Kubernetes cluster on GCP Compute Engines and having 3 Master Nodes and 3 Worker Nodes (It's not a GKE Cluster). Can anybody suggest me what storage options I can use for my cluster? If I create a virtual disk on GCP, can I use that disk as a persistent storage?
You can use GCE Persistent Disk Storage Class.
Here is how you create the storage class:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ssd
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-ssd
Then you do the following to create the PV & PVC and to attach to your pod.
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gce-claim
spec:
accessModes:
- ReadWriteOnce
storageClassName: ssd
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
name: webserver-pd
spec:
containers:
- image: httpd
name: webserver
volumeMounts:
- mountPath: /data
name: dynamic-volume
volumes:
- name: dynamic-volume
persistentVolumeClaim:
claimName: gce-claim
Example taken from this blog post
There are two types of provisioning Persistent Volumes: Static Provisioning and Dynamic Provisioning.
I will briefly describe each of these types.
Static Provisioning
Using this approach you need to create Disk, PersistentVolume and PersistentVolumeClaim manually.
I've create simple example for you to illustrate how it works.
First I created disk, on GCP we can use gcloud command:
$ gcloud compute disks create --size 10GB --region europe-west3-c test-disk
NAME ZONE SIZE_GB TYPE STATUS
test-disk europe-west3-c 10 pd-standard READY
Next I created PV and PVC using this manifest files:
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-test
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
gcePersistentDisk:
pdName: test-disk # This GCE PD must already exist.
fsType: ext4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: claim-test
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
After applaying this manifest files, we can check status of PV and PVC:
root#km:~# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pv-test 10Gi RWO Retain Bound default/claim-test 12m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/claim-test Bound pv-test 10Gi RWO 12m
Finally I used above claim as volume:
apiVersion: v1
kind: Pod
metadata:
name: web
spec:
containers:
- name: web
image: nginx
volumeMounts:
- mountPath: "/usr/share/nginx"
name: vol-test
volumes:
- name: vol-test
persistentVolumeClaim:
claimName: claim-test
We can inspect created Pod to check if it works as expected:
root#km:~# kubectl exec -it web -- bash
root#web:/# df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/sdb 9.8G 37M 9.8G 1% /usr/share/nginx
...
Dynamic Provisioning
In this case volume is provisioned automatically when application requires it.
First you need to create StorageClass object to define a provisioner such e.g. kubernetes.io/gce-pd.
We don't need to create PersistenVolume anymore, it's created automatically by StorageClass for us.
I've also created simple example for you to illustrate how it works.
First I created StorageClass as default storage class:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
storageclass.kubernetes.io/is-default-class: "true"
name: standard
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
fstype: ext4
And then PVC (the same as in the previous example) - but in this case PV was created automatically:
root#km:~# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-8dcd69f1-7081-45a7-8424-cc02e61a4976 10Gi RWO Delete Bound default/claim-test standard 3m10s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/claim-test Bound pvc-8dcd69f1-7081-45a7-8424-cc02e61a4976 10Gi RWO standard 3m12s
In more advanced cases it may be useful to create multiple StorageClasses with differnet persistent disks types.
I currently have a ROM, RWO persistent volume claim that I regularly use as a read only volume in a deployment that sporadically gets repopulated by some job using it as a read write volume while the deployment is scaled down to 0. However, since in-tree plugins will be deprecated in future versions of kubernetes, I'm planning to migrate this process to volumes using csi drivers.
In order to clarify my current use of this kind of volumes, I'll put a sample yaml configuration file using the basic idea:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test
spec:
storageClassName: standard
accessModes:
- ReadOnlyMany
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: batch/v1
kind: Job
metadata:
name: test
spec:
template:
spec:
containers:
- name: test
image: busybox
# Populate the volume
command:
- touch
- /foo/bar
volumeMounts:
- name: test
mountPath: /foo/
subPath: foo
volumes:
- name: test
persistentVolumeClaim:
claimName: test
restartPolicy: Never
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: test
name: test
spec:
replicas: 0
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: test
spec:
containers:
- name: test
image: busybox
command:
- sh
- '-c'
- |
# Check the volume has been populated
ls /foo/
# Prevent the pod from exiting for a while
sleep 3600
volumeMounts:
- name: test
mountPath: /foo/
subPath: foo
volumes:
- name: test
persistentVolumeClaim:
claimName: test
readOnly: true
so the job populates the the volume and later the deployment is scaled up. However, replacing the storageClassName field standard in the persistent volume claim by singlewriter-standard does not even allow the job to run.
Is this some kind of bug? Is there some workaround to this using volumes using the csi driver?
If this is a bug, I'd plan to migrate to using sci drivers later; however, if this is not a bug, how should I migrate my current workflow since in-tree plugins will eventually be deprecated?
Edit:
The version of the kubernetes server is 1.17.9-gke.1504. As for the storage classes, they are the standard and singlewriter-standard default storage classes:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
labels:
addonmanager.kubernetes.io/mode: EnsureExists
kubernetes.io/cluster-service: "true"
name: standard
parameters:
type: pd-standard
provisioner: kubernetes.io/gce-pd
reclaimPolicy: Delete
volumeBindingMode: Immediate
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
components.gke.io/component-name: pdcsi-addon
components.gke.io/component-version: 0.5.1
storageclass.kubernetes.io/is-default-class: "true"
labels:
addonmanager.kubernetes.io/mode: EnsureExists
name: singlewriter-standard
parameters:
type: pd-standard
provisioner: pd.csi.storage.gke.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
While the error is not shown in the job but in the pod itself (this is just for the singlewriter-standard storage class):
Warning FailedAttachVolume attachdetach-controller AttachVolume.Attach failed for volume "..." : CSI does not support ReadOnlyMany and ReadWriteOnce on the same PersistentVolume
The message you encountered:
Warning FailedAttachVolume attachdetach-controller AttachVolume.Attach failed for volume "..." : CSI does not support ReadOnlyMany and ReadWriteOnce on the same PersistentVolume
is not a bug. The attachdetach-controller is showing this error as it doesn't know in which accessMode it should mount the volume:
For [ReadOnlyMany, ReadWriteOnce] PV, the external attacher simply does not know if the attachment is going to be consumed as read-only(-many) or as read-write(-once)
-- Github.com: Kubernetes CSI: External attacher: Issues: 153
I encourage you to check the link above for a full explanation.
I currently have a ROM, RWO persistent volume claim that I regularly use as a read only volume in a deployment that sporadically gets repopulated by some job using it as a read write volume
You can combine the steps from below guides:
Turn on the CSI Persistent disk driver in GKE
Cloud.google.com: Kubernetes Engine: How to: Persistent volumes: Gce-pd-csi-driver
Create a PVC with pd.csi.storage.gke.io provisioner (you will need to modify YAML definitions with storageClassName: singlewriter-standard):
Cloud.google.com: Kubernetes Engine: How to: Persistent volumes: Readonlymany disks
Citing the documentation on steps to take (from ReadOnlyMany guide) that should fulfill the setup you've shown:
Before using a persistent disk in read-only mode, you must format it.
To format your persistent disk:
Create a persistent disk manually or by using dynamic provisioning.
Format the disk and populate it with data. To format the disk, you can:
Reference the disk as a ReadWriteOnce volume in a Pod. Doing this results in GKE automatically formatting the disk, and enables the Pod to pre-populate the disk with data. When the Pod starts, make sure the Pod writes data to the disk.
Manually mount the disk to a VM and format it. Write any data to the disk that you want. For details, see Persistent disk formatting.
Unmount and detach the disk:
If you referenced the disk in a Pod, delete the Pod, wait for it to terminate, and wait for the disk to automatically detach from the node.
If you mounted the disk to a VM, detach the disk using gcloud compute instances detach-disk.
Create Pods that access the volume as ReadOnlyMany as shown in the following section.
-- Cloud.google.com: Kubernetes Engine: How to: Persistent volumes: Readonlymany disks
Additional resources:
Github.com: Kubernetes: Design proposals: Storage: CSI
Kubernetes.io: Blog: Container storage interface
Kubernetes-csi.github.io: Docs: Drivers
EDIT
Following the official documentation:
Cloud.google.com: Kubernetes Engine: How to: Persistent volumes: Readonlymany disks
Please treat it as an example.
Dynamically create a PVC that will be used with ReadWriteOnce accessMode:
pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-rwo
spec:
storageClassName: singlewriter-standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 81Gi
Run a Pod with a PVC mounted to it:
pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox-pvc
spec:
containers:
- image: k8s.gcr.io/busybox
name: busybox
command:
- "sleep"
- "36000"
volumeMounts:
- mountPath: /test-mnt
name: my-volume
volumes:
- name: my-volume
persistentVolumeClaim:
claimName: pvc-rwo
Run following commands:
$ kubectl exec -it busybox-pvc -- /bin/sh
$ echo "Hello there!" > /test-mnt/hello.txt
Delete the Pod and wait for the drive to be unmounted. Please do not delete PVC as deleting it:
When you delete a claim, the corresponding PersistentVolume object and the provisioned Compute Engine persistent disk are also deleted.
-- Cloud.google.com: Kubernetes Engine: Persistent Volumes: Dynamic provisioning
Get the name (it's in VOLUME column) of the earlier created disk by running:
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-rwo Bound pvc-11111111-2222-3333-4444-555555555555 81Gi RWO singlewriter-standard 52m
Create a PV and PVC with following definition:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-rox
spec:
storageClassName: singlewriter-standard
capacity:
storage: 81Gi
accessModes:
- ReadOnlyMany
claimRef:
namespace: default
name: pvc-rox # <-- important
gcePersistentDisk:
pdName: <INSERT HERE THE DISK NAME FROM EARLIER COMMAND>
# pdName: pvc-11111111-2222-3333-4444-555555555555 <- example
fsType: ext4
readOnly: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-rox # <-- important
spec:
storageClassName: singlewriter-standard
accessModes:
- ReadOnlyMany
resources:
requests:
storage: 81Gi
You can test if your disk is in ROX accessMode when the spawned Pods were scheduled on multiple nodes and all of them have the PVC mounted:
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
replicas: 15
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
volumeMounts:
- mountPath: /test-mnt
name: volume-ro
readOnly: true
volumes:
- name: volume-ro
persistentVolumeClaim:
claimName: pvc-rox
readOnly: true
$ kubectl get deployment nginx
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 15/15 15 15 3m1s
$ kubectl exec -it nginx-6c77b8bf66-njhpm -- cat /test-mnt/hello.txt
Hello there!
I have been trying for a while to get a stateful service to start on my kubernetes cluster.
The cluster has one master and one worker. It's running on top of AWS EC2 instances running with Ubuntu 18.04.
I've tried everything that I can think of but when I create the stateful service, the pods won't get scheduled onto the nodes.
I believe that it has something to do with the PV's, but I can't figure out what.
Also, I'm having a hard time getting any diagnostics. Trying to run kubectl logs on the pod and container returns nothing.
I first tried using local hardware, i.e. a local mount, but that didn't fix the problem.
I've now created an AWS EBS volume and have created a PV that references this.
The PV binds to it correctly, but I still can't get kubernetes to schedule the pods on the worker node.
Here are the .yaml config files that I'm using.
The first one creates the storageclass called 'fast'
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
volumeBindingMode: Immediate
Here is the yaml file that creates the PV.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
volumeBindingMode: Immediate
Finally, here's the statefulset yaml file
apiVersion: apps/v1
kind: StatefulSet
metadata:
namespace: lab4a
name: apache-http
spec:
selector:
matchLabels:
app: httpd
serviceName: "httpd-service"
replicas: 3
template:
metadata:
namespace: lab4a
labels:
app: httpd
spec:
terminationGracePeriodSeconds: 10
containers:
- name: httpd
image: httpd:latest
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/local/apache2/htdocs
volumeClaimTemplates:
- metadata:
name: web-pvc
namespace: lab4a
spec:
accessModes: [ "ReadWriteMany" ]
storageClassName: "fast"
resources:
requests:
storage: 10Gi
kubectl get pv gives me:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
ebs-pv 10Gi RWX Retain Available 31m
So it stands to reason, at least as far as I can tell, that the PV is ready to go.
From my understanding, I don't need to supply a PV Claim manually as the volumeClaimTemplates section in the statefulset yaml file will do this dynamically.
kubectl get all -n lab4a gives me:
NAME READY STATUS RESTARTS AGE
pod/web-0 0/1 Pending 0 16m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx ClusterIP None <none> 80/TCP 16m
NAME READY AGE
statefulset.apps/web 0/2 16m
when I run kubectl describe pod web-0 -n lab4a I get the following:
Name: web-0
Namespace: lab4a
Priority: 0
PriorityClassName: <none>
Node: <none>
Labels: app=nginx
controller-revision-hash=web-b46f789c4
statefulset.kubernetes.io/pod-name=web-0
Annotations: <none>
Status: Pending
IP:
Controlled By: StatefulSet/web
Containers:
nginx:
Image: k8s.gcr.io/nginx-slim:0.8
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/usr/share/nginx/html from www (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mjclk (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
www:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: www-web-0
ReadOnly: false
default-token-mjclk:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mjclk
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 35s (x14 over 16m) default-scheduler 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
I have no idea what's failing, and I don't know what else to try to debug this problem. Is kubernetes failing to bind the persistent volume to the node? Or is it some other issue?
Any help appreciated.
Thanks
(1) Your Storage
AWS EBS does not provide ReadWriteMany (see table in the docs https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes).
You can
Use ReadWriteOnce instead (proposed).
Set up an in-cluster NFS that hosts PVs that allow ReadWriteMany if you do have an actual need for this.
(2) Your Taints and Tolerations
Your Pod's tolerations look okay; can you provide insight on your nodes' taints? Did you fiddle around with kubectl taint ... before on this cluster? Is this a managed cluster or did you set it up on your own on AWS machines?