Pod has unbound immediate PersistentVolumeClaims kong-ingress-controller - kubernetes

I follow this to install kong-ingress-controller in my master node. But when I deploy postgres-0 it created volume which is pending. I using my own cloud. Here is my yaml to create persistanvolume:
apiVersion: v1
kind: PersistentVolume
metadata:
name: postgre-pv-volume
namespace : kong
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/var/lib/postgresql/data"
When I run kubectl describe pod postgres-0 -n kong
Result:
Name: postgres-0
Namespace: kong
Priority: 0
Node: <none>
Labels: app=postgres
controller-revision-hash=postgres-59ccf8fcf7
statefulset.kubernetes.io/pod-name=postgres-0
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/postgres
Containers:
postgres:
Image: postgres:9.5
Port: 5432/TCP
Host Port: 0/TCP
Environment:
POSTGRES_USER: kong
POSTGRES_PASSWORD: kong
POSTGRES_DB: kong
PGDATA: /var/lib/postgresql/data/pgdata
Mounts:
/var/lib/postgresql/data from datadir (rw,path="pgdata")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-g7828 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
datadir:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: datadir-postgres-0
ReadOnly: false
default-token-g7828:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-g7828
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler pod has unbound immediate PersistentVolumeClaims
Warning FailedScheduling <unknown> default-scheduler pod has unbound immediate PersistentVolumeClaims
Please help me. Thanks

Problem may lay in bad or none configuration of StorageClass.
1. Firstly you have to ensure that you have storageclass called manual.
$ kubectl get storageclass
The name of a StorageClass object is significant, and is how users can request a particular class.
2. To create StorageClass you have to define configuration file, here is example:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: manual
provisioner: xxx
reclaimPolicy: Retain
allowVolumeExpansion: true
mountOptions:
- debug
volumeBindingMode: Immediate
Storage classes have a provisioner that determines what volume plugin is used for provisioning PVs. This field must be specified (xxx).
Take note on such definition:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
Local volumes do not currently support dynamic provisioning, however a StorageClass should still be created to delay volume binding until pod scheduling. This is specified by the WaitForFirstConsumer volume binding mode.
Delaying volume binding allows the scheduler to consider all of a pod’s scheduling constraints when choosing an appropriate PersistentVolume for a PersistentVolumeClaim.
Let me know if it helps.

Related

kubernetes.io/csi: expected valid fsGroupPolicy, received nil value or empty string

I have a problem with storage in Kubernetes. I run a Kubernetes cluster with one master and two worker nodes at Hetzner Online
EDIT - It is installed with kubeadm and runs flannel as cni provider.
Now I try to create pods with volumes for persistence (later I want to run a Kafka in the cluster). Hetzner offers a storageclass with csi provider. This always creates storage at Hetzner for a pvc and integrates this into the server. This also works so far. But when I add a PVC and a pod to debug the storage everything is created as expected but the volume can't be mounted.
Debug yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: debug-pvc
namespace: kafka
spec:
resources:
requests:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
---
kind: Pod
apiVersion: v1
metadata:
name: volume-debugger
namespace: kafka
spec:
volumes:
- name: volume-to-debug
persistentVolumeClaim:
claimName: debug-pvc
containers:
- name: debugger
image: nginx
volumeMounts:
- mountPath: "/data"
name: volume-to-debug
Description of the pod:
Name: volume-debugger
Namespace: kafka
Priority: 0
Node: jm-website-worker-02/10.98.0.4
Start Time: Tue, 09 Mar 2021 12:51:57 +0100
Labels: <none>
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
debugger:
Container ID:
Image: nginx
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/data from volume-to-debug (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-nb969 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
volume-to-debug:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: debug-pvc
ReadOnly: false
default-token-nb969:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-nb969
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 19s Successfully assigned kafka/volume-debugger to jm-website-worker-02
Normal SuccessfulAttachVolume 14s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-e5a3b3ed-f4d8-4cb7-850e-ceb31cc71eca"
Warning FailedMount 4s (x5 over 12s) kubelet, jm-website-worker-02 MountVolume.NewMounter initialization failed for volume "pvc-e5a3b3ed-f4d8-4cb7-850e-ceb31cc71eca" : kubernetes.io/csi: expected valid fsGroupPolicy, received nil value or empty string
Descrition of the pvc:
Name: debug-pvc
Namespace: kafka
StorageClass: hcloud-volumes
Status: Bound
Volume: pvc-e5a3b3ed-f4d8-4cb7-850e-ceb31cc71eca
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: csi.hetzner.cloud
volume.kubernetes.io/selected-node: jm-website-worker-02
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: volume-debugger
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 4m46s persistentvolume-controller waiting for first consumer to be created before binding
Normal ExternalProvisioning 4m46s persistentvolume-controller waiting for a volume to be created, either by external provisioner "csi.hetzner.cloud" or manually created by system administrator
Normal Provisioning 4m46s csi.hetzner.cloud_hcloud-csi-controller-0_5e46eb4e-346a-4772-ac3b-048524a37eec External provisioner is provisioning volume for claim "kafka/debug-pvc"
Normal ProvisioningSucceeded 4m42s csi.hetzner.cloud_hcloud-csi-controller-0_5e46eb4e-346a-4772-ac3b-048524a37eec Successfully provisioned volume pvc-e5a3b3ed-f4d8-4cb7-850e-ceb31cc71eca
and the description of the pv itself:
Name: pvc-e5a3b3ed-f4d8-4cb7-850e-ceb31cc71eca
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: csi.hetzner.cloud
Finalizers: [kubernetes.io/pv-protection external-attacher/csi-hetzner-cloud]
StorageClass: hcloud-volumes
Status: Bound
Claim: kafka/debug-pvc
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity:
Required Terms:
Term 0: csi.hetzner.cloud/location in [nbg1]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: csi.hetzner.cloud
FSType: ext4
VolumeHandle: 9907113
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1615290679845-8081-csi.hetzner.cloud
Events: <none>
I'm using Kubernetes version 1.20.4. And the Hetzner CSI in version master for K8s v1.20. I tried to fix this problem for a week now but I can't find anything helpful. I tried something with PodSecurity policies and security context already but it doesn't work out the right way.
Thank you for your help.
Problem solved. There was an error with updating the master node. After reinstalling the master everything worked as expected.

Cannot get Pod to bind local-storage in minikube. "node(s) didn't find available persistent volumes", "waiting for first consumer to be created"

I'm having some trouble configuring my Kubernetes deployment on minikube use local-storage. I'm trying to set up a rethinkdb instance that will mount a directory from the minikube VM to the rethinkdb Pod. My setup is the following
Storage
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: rethinkdb-pv
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /mnt/data
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- minikube
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rethinkdb-pv-claim
spec:
storageClassName: local-storage
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
So I define a storageClass of local-storage type as described online in the tutorials. I then make a PersistentVolume that asks for 10GB of storage from the /mnt/data path on the underlying host. I have made this directory on the minikube VM
$ minikube ssh
$ ls /mnt
data sda1
This PersistentVolume has the storage class of local-storage and requests it from nodes matching the nodeAffinity section of hostname in 'minikube'.
I then make a PersistentVolumeClaim that asks for the type local-storage and requests 5GB.
Everything is good here, right? Here is the output of kubectl
$ kubectl get pv,pvc,storageClass
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/rethinkdb-pv 10Gi RWO Delete Available local-storage 9m33s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/rethinkdb-pv-claim Pending local-storage 7m51s
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
storageclass.storage.k8s.io/local-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 9m33s
storageclass.storage.k8s.io/standard (default) k8s.io/minikube-hostpath Delete Immediate false 24h
RethinkDB Deployment
I now attempt to make a Deployment with a single replica of the standard RethinkDB container.
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
name: database
name: rethinkdb
spec:
progressDeadlineSeconds: 2147483647
replicas: 1
selector:
matchLabels:
service: rethinkdb
template:
metadata:
creationTimestamp: null
labels:
service: rethinkdb
spec:
containers:
- name: rethinkdb
image: rethinkdb:latest
volumeMounts:
- mountPath: /data
name: rethinkdb-data
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: rethinkdb-data
persistentVolumeClaim:
claimName: rethinkdb-pv-claim
This asks for a single replica of rethinkdb and it tries to mount the rethinkdb-pv-claim Persistent Volume Claim as the name rethinkdb-data and then attempts to mount that at /data in the container.
This is what shows, though
Name: rethinkdb-6dbf4ccdb-64gk5
Namespace: development
Priority: 0
Node: <none>
Labels: pod-template-hash=6dbf4ccdb
service=rethinkdb
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/rethinkdb-6dbf4ccdb
Containers:
rethinkdb:
Image: rethinkdb:latest
Port: <none>
Host Port: <none>
Environment: <none>
Mounts:
/data from rethinkdb-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-d5ncp (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
rethinkdb-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: rethinkdb-pv-claim
ReadOnly: false
default-token-d5ncp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-d5ncp
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 73s (x7 over 8m38s) default-scheduler 0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind.
"1 node(s) didn't find available persistent volumes to bind". I'm not sure how that is because the PVC is available.
$ kubectl describe pvc
Name: rethinkdb-pv-claim
Namespace: development
StorageClass: local-storage
Status: Pending
Volume:
Labels: <none>
Annotations: Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Mounted By: rethinkdb-6dbf4ccdb-64gk5
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 11s (x42 over 10m) persistentvolume-controller waiting for first consumer to be created before binding
I think one hint is that the field
Nodes <null> for the Pod - does that mean it isn't assigned to a node?
I think the issue is that one of mine was ReadWriteOnce and the other one was ReadWriteMany, then I had trouble getting permissions right when running minikube mount /tmp/data:/mnt/data so I just got rid of mounting it to the underlying filesystem and now it works

0/2 nodes are available: 1 node(s) didn't find available persistent volumes to bind, 1 node(s) had taints that the pod didn't tolerate

I setup a k8s in a multiple node. But the PersistentVolume can not be created successfully, when I am trying to create a simple PostgreSQL.
The StorageClass
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
The StatefulSet is:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
namespace: kong
spec:
replicas: 1
selector:
matchLabels:
app: postgres
serviceName: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- env:
...
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: my-local-pv
subPath: pgdata
terminationGracePeriodSeconds: 60
volumeClaimTemplates:
- metadata:
name: my-local-pv
spec:
storageClassName: local-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
The describe pod:
Name: postgres-0
Namespace: kong
Priority: 0
Node: <none>
Labels: app=postgres
controller-revision-hash=postgres-9c9cf868d
statefulset.kubernetes.io/pod-name=postgres-0
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/postgres
Containers:
postgres:
Image: postgres:9.5
Port: 5432/TCP
Host Port: 0/TCP
Environment:
POSTGRES_USER: kong
POSTGRES_PASSWORD: kong
POSTGRES_DB: kong
PGDATA: /var/lib/postgresql/data/pgdata
Mounts:
/var/lib/postgresql/data from postgres-data (rw,path="pgdata")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-b5mkt (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
postgres-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: postgres-data-postgres-0
ReadOnly: false
default-token-b5mkt:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-b5mkt
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler 0/2 nodes are available: 1 node(s) didn't find available persistent volumes to bind, 1 node(s) had taints that the pod didn't tolerate.
Warning FailedScheduling <unknown> default-scheduler 0/2 nodes are available: 1 node(s) didn't find available persistent volumes to bind, 1 node(s) had taints that the pod didn't tolerate.
kubectl get pv:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
my-local-pv 2Gi RWO Retain Available local-storage 3m59s
Please help me. Thanks in advance

Cannot get my stateful service to run. Pods can't get scheduled onto nodes

I have been trying for a while to get a stateful service to start on my kubernetes cluster.
The cluster has one master and one worker. It's running on top of AWS EC2 instances running with Ubuntu 18.04.
I've tried everything that I can think of but when I create the stateful service, the pods won't get scheduled onto the nodes.
I believe that it has something to do with the PV's, but I can't figure out what.
Also, I'm having a hard time getting any diagnostics. Trying to run kubectl logs on the pod and container returns nothing.
I first tried using local hardware, i.e. a local mount, but that didn't fix the problem.
I've now created an AWS EBS volume and have created a PV that references this.
The PV binds to it correctly, but I still can't get kubernetes to schedule the pods on the worker node.
Here are the .yaml config files that I'm using.
The first one creates the storageclass called 'fast'
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
volumeBindingMode: Immediate
Here is the yaml file that creates the PV.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
volumeBindingMode: Immediate
Finally, here's the statefulset yaml file
apiVersion: apps/v1
kind: StatefulSet
metadata:
namespace: lab4a
name: apache-http
spec:
selector:
matchLabels:
app: httpd
serviceName: "httpd-service"
replicas: 3
template:
metadata:
namespace: lab4a
labels:
app: httpd
spec:
terminationGracePeriodSeconds: 10
containers:
- name: httpd
image: httpd:latest
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/local/apache2/htdocs
volumeClaimTemplates:
- metadata:
name: web-pvc
namespace: lab4a
spec:
accessModes: [ "ReadWriteMany" ]
storageClassName: "fast"
resources:
requests:
storage: 10Gi
kubectl get pv gives me:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
ebs-pv 10Gi RWX Retain Available 31m
So it stands to reason, at least as far as I can tell, that the PV is ready to go.
From my understanding, I don't need to supply a PV Claim manually as the volumeClaimTemplates section in the statefulset yaml file will do this dynamically.
kubectl get all -n lab4a gives me:
NAME READY STATUS RESTARTS AGE
pod/web-0 0/1 Pending 0 16m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/nginx ClusterIP None <none> 80/TCP 16m
NAME READY AGE
statefulset.apps/web 0/2 16m
when I run kubectl describe pod web-0 -n lab4a I get the following:
Name: web-0
Namespace: lab4a
Priority: 0
PriorityClassName: <none>
Node: <none>
Labels: app=nginx
controller-revision-hash=web-b46f789c4
statefulset.kubernetes.io/pod-name=web-0
Annotations: <none>
Status: Pending
IP:
Controlled By: StatefulSet/web
Containers:
nginx:
Image: k8s.gcr.io/nginx-slim:0.8
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/usr/share/nginx/html from www (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mjclk (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
www:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: www-web-0
ReadOnly: false
default-token-mjclk:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mjclk
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 35s (x14 over 16m) default-scheduler 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
I have no idea what's failing, and I don't know what else to try to debug this problem. Is kubernetes failing to bind the persistent volume to the node? Or is it some other issue?
Any help appreciated.
Thanks
(1) Your Storage
AWS EBS does not provide ReadWriteMany (see table in the docs https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes).
You can
Use ReadWriteOnce instead (proposed).
Set up an in-cluster NFS that hosts PVs that allow ReadWriteMany if you do have an actual need for this.
(2) Your Taints and Tolerations
Your Pod's tolerations look okay; can you provide insight on your nodes' taints? Did you fiddle around with kubectl taint ... before on this cluster? Is this a managed cluster or did you set it up on your own on AWS machines?

Kubernetes pod pending when a new volume is attached (EKS)

Let me describe my scenario:
TL;DR
When I create a deployment on Kubernetes with 1 attached volume, everything works perfectly. When I create the same deployment, but with a second volume attached (total: 2 volumes), the pod gets stuck on "Pending" with errors:
pod has unbound PersistentVolumeClaims (repeated 2 times)
0/2 nodes are available: 2 node(s) had no available volume zone.
Already checked that the volumes are created in the correct availability zones.
Detailed description
I have a cluster set up using Amazon EKS, with 2 nodes. I have the following default storage class:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp2
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
And I have a mongodb deployment which needs two volumes, one mounted on /data/db folder, and the other mounted in some random directory I need. Here is an minimal yaml used to create the three components (I commented some lines on purpose):
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: my-project
creationTimestamp: null
labels:
io.kompose.service: my-project-db-claim0
name: my-project-db-claim0
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: my-project
creationTimestamp: null
labels:
io.kompose.service: my-project-db-claim1
name: my-project-db-claim1
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: my-project
name: my-project-db
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
name: my-db
spec:
containers:
- name: my-project-db-container
image: mongo
imagePullPolicy: Always
resources: {}
volumeMounts:
- mountPath: /my_dir
name: my-project-db-claim0
# - mountPath: /data/db
# name: my-project-db-claim1
ports:
- containerPort: 27017
restartPolicy: Always
volumes:
- name: my-project-db-claim0
persistentVolumeClaim:
claimName: my-project-db-claim0
# - name: my-project-db-claim1
# persistentVolumeClaim:
# claimName: my-project-db-claim1
That yaml works perfectly. The output for the volumes is:
$ kubectl describe pv
Name: pvc-307b755a-039e-11e9-b78d-0a68bcb24bc6
Labels: failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1c
Annotations: kubernetes.io/createdby: aws-ebs-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: yes
pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
Finalizers: [kubernetes.io/pv-protection]
StorageClass: gp2
Status: Bound
Claim: my-project/my-project-db-claim0
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 5Gi
Node Affinity: <none>
Message:
Source:
Type: AWSElasticBlockStore (a Persistent Disk resource in AWS)
VolumeID: aws://us-east-1c/vol-xxxxx
FSType: ext4
Partition: 0
ReadOnly: false
Events: <none>
Name: pvc-308d8979-039e-11e9-b78d-0a68bcb24bc6
Labels: failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1b
Annotations: kubernetes.io/createdby: aws-ebs-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: yes
pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
Finalizers: [kubernetes.io/pv-protection]
StorageClass: gp2
Status: Bound
Claim: my-project/my-project-db-claim1
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 10Gi
Node Affinity: <none>
Message:
Source:
Type: AWSElasticBlockStore (a Persistent Disk resource in AWS)
VolumeID: aws://us-east-1b/vol-xxxxx
FSType: ext4
Partition: 0
ReadOnly: false
Events: <none>
And the pod output:
$ kubectl describe pods
Name: my-project-db-7d48567b48-slncd
Namespace: my-project
Priority: 0
PriorityClassName: <none>
Node: ip-192-168-212-194.ec2.internal/192.168.212.194
Start Time: Wed, 19 Dec 2018 15:55:58 +0100
Labels: name=my-db
pod-template-hash=3804123604
Annotations: <none>
Status: Running
IP: 192.168.216.33
Controlled By: ReplicaSet/my-project-db-7d48567b48
Containers:
my-project-db-container:
Container ID: docker://cf8222f15e395b02805c628b6addde2d77de2245aed9406a48c7c6f4dccefd4e
Image: mongo
Image ID: docker-pullable://mongo#sha256:0823cc2000223420f88b20d5e19e6bc252fa328c30d8261070e4645b02183c6a
Port: 27017/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 19 Dec 2018 15:56:42 +0100
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/my_dir from my-project-db-claim0 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-pf9ks (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
my-project-db-claim0:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: my-project-db-claim0
ReadOnly: false
default-token-pf9ks:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-pf9ks
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 7m22s (x5 over 7m23s) default-scheduler pod has unbound PersistentVolumeClaims (repeated 2 times)
Normal Scheduled 7m21s default-scheduler Successfully assigned my-project/my-project-db-7d48567b48-slncd to ip-192-168-212-194.ec2.internal
Normal SuccessfulMountVolume 7m21s kubelet, ip-192-168-212-194.ec2.internal MountVolume.SetUp succeeded for volume "default-token-pf9ks"
Warning FailedAttachVolume 7m13s (x5 over 7m21s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-307b755a-039e-11e9-b78d-0a68bcb24bc6" : "Error attaching EBS volume \"vol-01a863d0aa7c7e342\"" to instance "i-0a7dafbbdfeabc50b" since volume is in "creating" state
Normal SuccessfulAttachVolume 7m1s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-307b755a-039e-11e9-b78d-0a68bcb24bc6"
Normal SuccessfulMountVolume 6m48s kubelet, ip-192-168-212-194.ec2.internal MountVolume.SetUp succeeded for volume "pvc-307b755a-039e-11e9-b78d-0a68bcb24bc6"
Normal Pulling 6m48s kubelet, ip-192-168-212-194.ec2.internal pulling image "mongo"
Normal Pulled 6m39s kubelet, ip-192-168-212-194.ec2.internal Successfully pulled image "mongo"
Normal Created 6m38s kubelet, ip-192-168-212-194.ec2.internal Created container
Normal Started 6m37s kubelet, ip-192-168-212-194.ec2.internal Started container
Everything is created without any problems. But if I uncomment the lines in the yaml so two volumes are attached to the db deployment, the pv output is the same as earlier, but the pod gets stuck on pending with the following output:
$ kubectl describe pods
Name: my-project-db-b8b8d8bcb-l64d7
Namespace: my-project
Priority: 0
PriorityClassName: <none>
Node: <none>
Labels: name=my-db
pod-template-hash=646484676
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/my-project-db-b8b8d8bcb
Containers:
my-project-db-container:
Image: mongo
Port: 27017/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/data/db from my-project-db-claim1 (rw)
/my_dir from my-project-db-claim0 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-pf9ks (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
my-project-db-claim0:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: my-project-db-claim0
ReadOnly: false
my-project-db-claim1:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: my-project-db-claim1
ReadOnly: false
default-token-pf9ks:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-pf9ks
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 60s (x5 over 60s) default-scheduler pod has unbound PersistentVolumeClaims (repeated 2 times)
Warning FailedScheduling 2s (x16 over 59s) default-scheduler 0/2 nodes are available: 2 node(s) had no available volume zone.
I've already read these two issues:
Dynamic volume provisioning creates EBS volume in the wrong availability zone
PersistentVolume on EBS can be created in availability zones with no nodes (Closed)
But I already checked that the volumes are created in the same zones as the cluster nodes instances. In fact, EKS creates two EBS by default in us-east-1b and us-east-1c zones and those volumes works. The volumes created by the posted yaml are on those regions too.
See this article: https://kubernetes.io/blog/2018/10/11/topology-aware-volume-provisioning-in-kubernetes/
The gist is that you want to update your storageclass to include:
volumeBindingMode: WaitForFirstConsumer
This causes the PV to not be created until the pod is scheduled. It fixed a similar problem for me.
Sounds like it's trying to create a volume in an availability zone where you don't have any volumes on. You can try restricting your StorageClass to the availability zones where you have nodes.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp2
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
allowedTopologies:
- matchLabelExpressions:
- key: failure-domain.beta.kubernetes.io/zone
values:
- us-east-1b
- us-east-1c
This is very similar to this question and this answer except that the issue described is on GCP and in this case it's AWS.
In this case, you should check the availability zone of your worker nodes (EC2 instances).
As a Example :
worker node 1 = eu-central-1b
worker node 2 = eu-central-1c
Then create the volume in including one of an availability zone which mentioned above(do not create the volume with eu-central-1a).
after you create the volume, create your PersistentVolume and PersistentVolumeClaim by attaching a newly created volume to your cluster like below.
apiVersion: v1
kind: PersistentVolume
metadata:
labels:
failure-domain.beta.kubernetes.io/region: eu-central-1
failure-domain.beta.kubernetes.io/zone: eu-central-1b
name: mongo-pv
namespace: default
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 100Gi
awsElasticBlockStore:
fsType: ext4
volumeID: aws://eu-central-1b/vol-063342ab9be5d2929
storageClassName: gp2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mongo-pvc
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: gp2
volumeName: mongo-pv