Pod does not find available persistent volume to bind - kubernetes

I want to deploy Traefik as a "Statefulset" to persist access and error logs on disk. Therefore I moved from the default "Deployment" to "Statefulset" and created a PV for each node. After defining the "volumeClaimTemplates" in the "Statefulset" the pod returns following error:
Warning FailedScheduling 98s default-scheduler 0/3 nodes are
available: 1 node(s) had untolerated taint
{node-role.kubernetes.io/control-plane: }, 3 node(s) didn't find
available persistent volumes to bind. preemption: 0/3 nodes are
available: 3 Preemption is not helpful for scheduling.
My PV + Storageclass definition:
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
{% for i in range(0,groups['k8s-workers'] | length ) %}
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: traefik-logs-{{ "%02d"|format(i+1) }}
spec:
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 50Gi
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
local:
path: "/mnt/traefik"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-worker-{{ "%02d"|format(i+1) }}
{% endfor %}
And finally the Statefulset:
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: traefik-deployment
namespace: traefik
labels:
app: traefik
spec:
replicas: 2
serviceName: traefik
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-account
terminationGracePeriodSeconds: 60
containers:
- name: traefik
image: traefik:v2.8
args:
- ...
ports:
- ...
volumeMounts:
- mountPath: "/var/log"
name: traefik-logs
volumeClaimTemplates:
- metadata:
name: traefik-logs
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
storageClassName: local-storage
The PV's are created:
root#k8s-master-01:/opt/traefik# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
elasticsearch-data-master-pv-01 120Gi RWO Retain Bound elasticsearch/elasticsearch-data-elasticsearch-es-master-0 elastic-local-master-storage 4h19m
elasticsearch-data-worker-pv-01 120Gi RWO Retain Bound elasticsearch/elasticsearch-data-elasticsearch-es-worker-0 elastic-local-worker-storage 4h19m
elasticsearch-data-worker-pv-02 120Gi RWO Retain Bound elasticsearch/elasticsearch-data-elasticsearch-es-worker-1 elastic-local-worker-storage 4h19m
traefik-logs-01 50Gi RWO Retain Available 82m
traefik-logs-02 50Gi RWO Retain Available 82m
The PVC is stuck at pending:
root#k8s-master-01:/opt/traefik# kubectl get pvc -A
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
elasticsearch elasticsearch-data-elasticsearch-es-master-0 Bound elasticsearch-data-master-pv-01 120Gi RWO elastic-local-master-storage 4h19m
elasticsearch elasticsearch-data-elasticsearch-es-worker-0 Bound elasticsearch-data-worker-pv-01 120Gi RWO elastic-local-worker-storage 4h19m
elasticsearch elasticsearch-data-elasticsearch-es-worker-1 Bound elasticsearch-data-worker-pv-02 120Gi RWO elastic-local-worker-storage 4h19m
traefik traefik-logs-traefik-deployment-0 Pending local-storage 56m
What have I misunderstood about provisioning persistent storage for a pod?

Related

1 node(s) didn't find available persistent volumes to bind error on stateful set pod creation

I'm starting out with K8s and I'm stuck at setting up mongo db in replica set mode with local persistent volume. I'm using StorageClass, PersistentVolume and PersistentVolumeClaim.
vincenzocalia#vincenzos-MacBook-Air server-node % kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
mongo-pv 1Gi RWO Retain Available mongo-storage 24m
but when inspect the pod I get
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m default-scheduler 0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
This post answer https://stackoverflow.com/a/70069138/2704032 confirmed my suspect that I might be using the wrong label..
So I had a look at the PV and I see that as I've set nodeAffinity as
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernets.io/hostname
operator: In
values:
- docker-desktop
it's looking for
Node Affinity:
Required Terms:
Term 0: kubernets.io/hostname in [docker-desktop]
I checked nodes with kubectl get nodes --show-labels
and it does have that label as the output shows
NAME STATUS ROLES AGE VERSION LABELS
docker-desktop Ready control-plane 7d9h v1.24.1 beta.kubernetes.io/arch=arm64,beta.kubernetes.io/os=linux,kubernetes.io/arch=arm64,kubernetes.io/hostname=docker-desktop,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
I tried using volumeClaimTemplate in the StatefulSet as
volumeClaimTemplates:
- metadata:
name: mongo-vctemplate
spec:
storageClassName: mongo-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
but it didn't make a difference..I also tried to specify the pic in the pv with the claimRef parameter but still the insidious error comes up at pod creation..
What else can I check or do I need to setup?
Many thanks as usual
Here are my yaml files
StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: mongo-storage
provisioner: kubernetes.io/no-provisioner
# volumeBindingMode: Immediate
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: mongo-pv
labels:
type: local
spec:
capacity:
storage: 1Gi
# persistentVolumeReclaimPolicy: Retain # prod
persistentVolumeReclaimPolicy: Delete # local tests
storageClassName: mongo-storage
# claimRef:
# name: mongo-pvc
accessModes:
- ReadWriteOnce
# volumeMode: Filesystem #default if omitted
# hostPath:
# path: /mnt/data
local:
path: /mnt/disk/vol1
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernets.io/hostname
operator: In
values:
- docker-desktop
PVC
piVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mongo-pvc
spec:
storageClassName: mongo-storage
# volumeName: mongo-pv # this will make it unbundable???
resources:
requests:
storage: 1Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo-statefulset
spec:
selector:
matchLabels:
app: mongo-pod # has to match .spec.template.metadata.labels
serviceName: mongo-clusterip-service
replicas: 1 # 3
template:
metadata:
labels:
app: mongo-pod # has to match .spec.selector.matchLabels
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongo-container
image: mongo
ports:
- containerPort: 27017
volumeMounts:
- name: mongo-pv-cont
mountPath: /data/db #/mnt/data
volumes:
- name: mongo-pv-cont
persistentVolumeClaim:
claimName: mongo-pvc
It is a typo in kubernets.io/hostname. It should be kubernetes.io/hostname in the pv definition.
similar to this one:
Error while using local persistent volumes in statefulset pod

How to Deploy an existing EBS volume to EKS PVC?

I have an existing ebs volume in AWS with data on it. I need to create a PVC in order to use it in my pods.
Following this guide: https://medium.com/pablo-perez/launching-a-pod-with-an-existing-ebs-volume-mounted-in-k8s-7b5506fa7fa3
persistentvolume.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
name: jenkins-volume
labels:
type: amazonEBS
spec:
capacity:
storage: 60Gi
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
volumeID: vol-011111111x
fsType: ext4
[$$]>kubectl describe pv
Name: jenkins-volume
Labels: type=amazonEBS
Annotations: <none>
Finalizers: [kubernetes.io/pv-protection]
StorageClass:
Status: Available
Claim:
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 60Gi
Node Affinity: <none>
Message:
Source:
Type: AWSElasticBlockStore (a Persistent Disk resource in AWS)
VolumeID: vol-011111111x
FSType: ext4
Partition: 0
ReadOnly: false
Events: <none>
persistentVolumeClaim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: jenkins-pvc-shared4
namespace: jenkins
spec:
storageClassName: gp2
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 60Gi
[$$]>kubectl describe pvc jenkins-pvc-shared4 -n jenkins
Name: jenkins-pvc-shared4
Namespace: jenkins
StorageClass: gp2
Status: Pending
Volume:
Labels: <none>
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Mounted By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 12s (x2 over 21s) persistentvolume-controller waiting for first consumer to be created before binding
[$$]>kubectl get pvc jenkins-pvc-shared4 -n jenkins
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
jenkins-pvc-shared4 Pending gp2 36s
Status is pending (waiting to the consumer to be attached) - but it should already be provisioned.
The "waiting for consumer" message suggests that your StorageClass has its volumeBindingMode set to waitForFirstConsumer.
The default value for this setting is Immediate: as soon as you register a PVC, your volume provisioner would provision a new volume.
The waitForFirstConsumer on the other hand would wait for a Pod to request usage for said PVC, before the provisioning a volume.
The messages and behavior you're seeing here sound normal. You may create a Deployment mounting that volume, to confirm provisioning works as expected.
try fsType "xfs" instead of ext4
StorageClass is empty for your PV.
According to your guide, you created storageClass "standard", so add to your PersistentVolume spec
storageClassName: standard
and also set it in persistentVolumeClaim instead of gp2
The right config should be:
[$$]>cat persistentvolume2.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-name
spec:
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
fsType: ext4
volumeID: aws://eu-west-2a/vol-123456-ID
capacity:
storage: 60Gi
persistentVolumeReclaimPolicy: Retain
storageClassName: gp2
[$$]>cat persistentVolumeClaim2.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: new-namespace
labels:
app.kubernetes.io/name: <app-name>
name: pvc-name
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 60Gi
storageClassName: gp2
volumeName: pv-name

Host Directory Not Showing Files From Kubernetes Persistent Volume Claim

I am migrating an application that consists of 1 or more tomcat servers and 1 or more wildfly servers to k8s. I have it up and working with a deployments for tomcat and wildfly with a clusterIP for each. I am now trying to combine log files from the multiple nodes into a single log directory on the host (for now single host deployment). I have created a PVC with the following yaml files:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-pv
spec:
storageClassName: local-storage
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /C/k8s/local-pv/logs
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: local-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: local-storage
resources:
requests:
storage: 2Gi
I have the tomcat deployment mapped to the voles by:
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-deployment
spec:
replicas: 2
selector:
matchLabels:
component: tomcat-deployment
template:
metadata:
labels:
component: tomcat
spec:
initContainers:
- name: volume-mount-hack
image: busybox
command: ["sh", "-c", "chmod -R 777 /usr/local/tomcat/logs"]
volumeMounts:
- name: local-persistent-storage
mountPath: /usr/local/tomcat/logs
containers:
- name: tomcat
image: tomcat-full:latest
volumeMounts:
- name: local-persistent-storage
mountPath: /usr/local/tomcat/logs
subPath: tomcat
volumes:
- name: local-persistent-storage
persistentVolumeClaim:
claimName: local-claim
I startup all the components and the system is functioning as expected. However, when I look in the host directory C/k8s/local-pv/logs/tomcat, no files are showing. I connect to the tomcat pod with docker exec and I see the log files from both tomcat servers. The files shown in the /usr/local/tomcat/logs survive a deployment restart so I know they are written somewhere. I searched my entire hard drive and the files are not anywhere.
I checked the pvc, pv, and storage class
kubectl describe pvc local-claim
Name: local-claim
Namespace: default
StorageClass: local-storage
Status: Bound
Volume: local-pv
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWX
VolumeMode: Filesystem
Mounted By: tomcat-deployment-644698fdc6-jmxz9
tomcat-deployment-644698fdc6-qvx9s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 12m (x3 over 14m) persistentvolume-controller waiting for first consumer to be created before binding
kubectl describe pv local-pv
Name: local-pv
Labels: <none>
Annotations: pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pv-protection]
StorageClass: local-storage
Status: Bound
Claim: default/local-claim
Reclaim Policy: Retain
Access Modes: RWX
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity: <none>
Message:
Source:
Type: HostPath (bare host directory volume)
Path: /C/k8s/pv/logs
HostPathType:
Events: <none>
kubectl get storageclass local-storage
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 19m
What piece am I missing? It appears that the local storage class is not bound to a volume claim and tomcat application.

Unable to attach aws ebs volume, error "instance not found"

Info:
Kubernetes Server version: 1.14
AWS Cloud Provider
EBS volume, storageclass
Details:
I have installed statefulset in our kubernetes cluster, however, it stuck it "ContainerCreating" status. Upon checking the logs, the error is "AttachVolume.Attach failed for volume pvc-xxxxxx: error finding instance ip-xxxxx : "instance not found"
It was succesfully installed around 17 days ago, but re-installing for an update caused the pod to stuck in ContainerCreating.
Manual attaching volume to the instance works. But doing it via storage class is not working and stuck in ContainerCreating status.
storageclass:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
storageclass.kubernetes.io/is-default-class: "true"
name: ssd-default
allowVolumeExpansion: true
parameters:
encrypted: "true"
type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: Immediate
pvc yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
finalizers:
- kubernetes.io/pvc-protection
labels:
app.kubernetes.io/instance: thanos-store
app.kubernetes.io/name: thanos-store
name: data-thanos-store-0
namespace: thanos
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
storageClassName: ssd-default
volumeMode: Filesystem
volumeName: pvc-xxxxxx
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 3Gi
phase: Bound
pv yaml:
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubernetes.io/createdby: aws-ebs-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: "yes"
pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
finalizers:
- kubernetes.io/pv-protection
labels:
failure-domain.beta.kubernetes.io/region: ap-xxx
failure-domain.beta.kubernetes.io/zone: ap-xxx
name: pvc-xxxx
spec:
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
fsType: ext4
volumeID: aws://xxxxx
capacity:
storage: 3Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: data-athena-thanos-store-0
namespace: thanos
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/region
operator: In
values:
- ap-xxx
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- ap-xxx
persistentVolumeReclaimPolicy: Delete
storageClassName: ssd-default
volumeMode: Filesystem
status:
phase: Bound
Describe pvc:
Name: data-athena-thanos-store-0
Namespace: athena-thanos
StorageClass: ssd-encrypted
Status: Bound
Volume: pvc-xxxx
Labels: app.kubernetes.io/instance=athena-thanos-store
app.kubernetes.io/name=athena-thanos-store
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 3Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: athena-thanos-store-0
The FailedAttachVolume error occurs when an EBS volume can’t be detached from an instance and thus cannot be attached to another. The EBS volume has to be in the available state to be attached. FailedAttachVolume is usually a symptom of an underlying failure to unmount and detach the volume.
Notice that while describing the PVC the StorageClass name is ssd-encrypted which is a mismatch with the config you showed earlier where the kind: StorageClass name is ssd-default. That's why you can mount the volume manually but not via the StorageClass. You can drop and recreate the StorageClass with a proper data.
Also, I recommend going through this article and using volumeBindingMode: WaitForFirstConsumer instead of volumeBindingMode: Immediate. This setting instructs the volume provisioner to not create a volume immediately, and instead, wait for a pod using an associated PVC to run through scheduling.

"FindExpandablePluginBySpec err:no volume plugin matched" messages in logs while volumes are working

Kubernetes version: 1.13.4 (same problem on 1.13.2).
I self-host the cluster on digitalocean.
OS: coreos 2023.4.0
I have 2 volumes on one node:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: prometheus-pv-volume
labels:
type: local
name: prometheus-pv-volume
spec:
storageClassName: local-storage
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
hostPath:
path: "/prometheus-volume"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/monitoring
operator: Exists
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: grafana-pv-volume
labels:
type: local
name: grafana-pv-volume
spec:
storageClassName: local-storage
capacity:
storage: 1Gi
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
hostPath:
path: "/grafana-volume"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/monitoring
operator: Exists
And 2 pvc's using them on a same node. Here is one:
storage:
volumeClaimTemplate:
spec:
storageClassName: local-storage
selector:
matchLabels:
name: prometheus-pv-volume
resources:
requests:
storage: 100Gi
Everything works fine.
kubectl get pv --all-namespaces output:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
grafana-pv-volume 1Gi RWO Retain Bound monitoring/grafana-storage local-storage 16m
prometheus-pv-volume 100Gi RWO Retain Bound monitoring/prometheus-k8s-db-prometheus-k8s-0 local-storage 16m
kubectl get pvc --all-namespaces output:
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
monitoring grafana-storage Bound grafana-pv-volume 1Gi RWO local-storage 10m
monitoring prometheus-k8s-db-prometheus-k8s-0 Bound prometheus-pv-volume 100Gi RWO local-storage 10m
The problem is that im getting these log messages every 2 minutes from kube-controller-manager:
W0302 17:16:07.877212 1 plugins.go:845] FindExpandablePluginBySpec(prometheus-pv-volume) -> err:no volume plugin matched
W0302 17:16:07.877164 1 plugins.go:845] FindExpandablePluginBySpec(grafana-pv-volume) -> err:no volume plugin matched
Why do they appear? How can i fix this?
Seems like this is safe to ignore message that was recently removed (Feb 20) and will not occur in future releases: https://github.com/kubernetes/kubernetes/pull/73901