How to Deploy an existing EBS volume to EKS PVC? - kubernetes

I have an existing ebs volume in AWS with data on it. I need to create a PVC in order to use it in my pods.
Following this guide: https://medium.com/pablo-perez/launching-a-pod-with-an-existing-ebs-volume-mounted-in-k8s-7b5506fa7fa3
persistentvolume.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
name: jenkins-volume
labels:
type: amazonEBS
spec:
capacity:
storage: 60Gi
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
volumeID: vol-011111111x
fsType: ext4
[$$]>kubectl describe pv
Name: jenkins-volume
Labels: type=amazonEBS
Annotations: <none>
Finalizers: [kubernetes.io/pv-protection]
StorageClass:
Status: Available
Claim:
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 60Gi
Node Affinity: <none>
Message:
Source:
Type: AWSElasticBlockStore (a Persistent Disk resource in AWS)
VolumeID: vol-011111111x
FSType: ext4
Partition: 0
ReadOnly: false
Events: <none>
persistentVolumeClaim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: jenkins-pvc-shared4
namespace: jenkins
spec:
storageClassName: gp2
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 60Gi
[$$]>kubectl describe pvc jenkins-pvc-shared4 -n jenkins
Name: jenkins-pvc-shared4
Namespace: jenkins
StorageClass: gp2
Status: Pending
Volume:
Labels: <none>
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Mounted By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 12s (x2 over 21s) persistentvolume-controller waiting for first consumer to be created before binding
[$$]>kubectl get pvc jenkins-pvc-shared4 -n jenkins
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
jenkins-pvc-shared4 Pending gp2 36s
Status is pending (waiting to the consumer to be attached) - but it should already be provisioned.

The "waiting for consumer" message suggests that your StorageClass has its volumeBindingMode set to waitForFirstConsumer.
The default value for this setting is Immediate: as soon as you register a PVC, your volume provisioner would provision a new volume.
The waitForFirstConsumer on the other hand would wait for a Pod to request usage for said PVC, before the provisioning a volume.
The messages and behavior you're seeing here sound normal. You may create a Deployment mounting that volume, to confirm provisioning works as expected.

try fsType "xfs" instead of ext4

StorageClass is empty for your PV.
According to your guide, you created storageClass "standard", so add to your PersistentVolume spec
storageClassName: standard
and also set it in persistentVolumeClaim instead of gp2

The right config should be:
[$$]>cat persistentvolume2.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-name
spec:
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
fsType: ext4
volumeID: aws://eu-west-2a/vol-123456-ID
capacity:
storage: 60Gi
persistentVolumeReclaimPolicy: Retain
storageClassName: gp2
[$$]>cat persistentVolumeClaim2.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: new-namespace
labels:
app.kubernetes.io/name: <app-name>
name: pvc-name
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 60Gi
storageClassName: gp2
volumeName: pv-name

Related

Host Directory Not Showing Files From Kubernetes Persistent Volume Claim

I am migrating an application that consists of 1 or more tomcat servers and 1 or more wildfly servers to k8s. I have it up and working with a deployments for tomcat and wildfly with a clusterIP for each. I am now trying to combine log files from the multiple nodes into a single log directory on the host (for now single host deployment). I have created a PVC with the following yaml files:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-pv
spec:
storageClassName: local-storage
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /C/k8s/local-pv/logs
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: local-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: local-storage
resources:
requests:
storage: 2Gi
I have the tomcat deployment mapped to the voles by:
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-deployment
spec:
replicas: 2
selector:
matchLabels:
component: tomcat-deployment
template:
metadata:
labels:
component: tomcat
spec:
initContainers:
- name: volume-mount-hack
image: busybox
command: ["sh", "-c", "chmod -R 777 /usr/local/tomcat/logs"]
volumeMounts:
- name: local-persistent-storage
mountPath: /usr/local/tomcat/logs
containers:
- name: tomcat
image: tomcat-full:latest
volumeMounts:
- name: local-persistent-storage
mountPath: /usr/local/tomcat/logs
subPath: tomcat
volumes:
- name: local-persistent-storage
persistentVolumeClaim:
claimName: local-claim
I startup all the components and the system is functioning as expected. However, when I look in the host directory C/k8s/local-pv/logs/tomcat, no files are showing. I connect to the tomcat pod with docker exec and I see the log files from both tomcat servers. The files shown in the /usr/local/tomcat/logs survive a deployment restart so I know they are written somewhere. I searched my entire hard drive and the files are not anywhere.
I checked the pvc, pv, and storage class
kubectl describe pvc local-claim
Name: local-claim
Namespace: default
StorageClass: local-storage
Status: Bound
Volume: local-pv
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWX
VolumeMode: Filesystem
Mounted By: tomcat-deployment-644698fdc6-jmxz9
tomcat-deployment-644698fdc6-qvx9s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 12m (x3 over 14m) persistentvolume-controller waiting for first consumer to be created before binding
kubectl describe pv local-pv
Name: local-pv
Labels: <none>
Annotations: pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pv-protection]
StorageClass: local-storage
Status: Bound
Claim: default/local-claim
Reclaim Policy: Retain
Access Modes: RWX
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity: <none>
Message:
Source:
Type: HostPath (bare host directory volume)
Path: /C/k8s/pv/logs
HostPathType:
Events: <none>
kubectl get storageclass local-storage
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 19m
What piece am I missing? It appears that the local storage class is not bound to a volume claim and tomcat application.

Unable to attach aws ebs volume, error "instance not found"

Info:
Kubernetes Server version: 1.14
AWS Cloud Provider
EBS volume, storageclass
Details:
I have installed statefulset in our kubernetes cluster, however, it stuck it "ContainerCreating" status. Upon checking the logs, the error is "AttachVolume.Attach failed for volume pvc-xxxxxx: error finding instance ip-xxxxx : "instance not found"
It was succesfully installed around 17 days ago, but re-installing for an update caused the pod to stuck in ContainerCreating.
Manual attaching volume to the instance works. But doing it via storage class is not working and stuck in ContainerCreating status.
storageclass:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
storageclass.kubernetes.io/is-default-class: "true"
name: ssd-default
allowVolumeExpansion: true
parameters:
encrypted: "true"
type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: Immediate
pvc yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
finalizers:
- kubernetes.io/pvc-protection
labels:
app.kubernetes.io/instance: thanos-store
app.kubernetes.io/name: thanos-store
name: data-thanos-store-0
namespace: thanos
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
storageClassName: ssd-default
volumeMode: Filesystem
volumeName: pvc-xxxxxx
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 3Gi
phase: Bound
pv yaml:
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubernetes.io/createdby: aws-ebs-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: "yes"
pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
finalizers:
- kubernetes.io/pv-protection
labels:
failure-domain.beta.kubernetes.io/region: ap-xxx
failure-domain.beta.kubernetes.io/zone: ap-xxx
name: pvc-xxxx
spec:
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
fsType: ext4
volumeID: aws://xxxxx
capacity:
storage: 3Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: data-athena-thanos-store-0
namespace: thanos
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/region
operator: In
values:
- ap-xxx
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- ap-xxx
persistentVolumeReclaimPolicy: Delete
storageClassName: ssd-default
volumeMode: Filesystem
status:
phase: Bound
Describe pvc:
Name: data-athena-thanos-store-0
Namespace: athena-thanos
StorageClass: ssd-encrypted
Status: Bound
Volume: pvc-xxxx
Labels: app.kubernetes.io/instance=athena-thanos-store
app.kubernetes.io/name=athena-thanos-store
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 3Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: athena-thanos-store-0
The FailedAttachVolume error occurs when an EBS volume can’t be detached from an instance and thus cannot be attached to another. The EBS volume has to be in the available state to be attached. FailedAttachVolume is usually a symptom of an underlying failure to unmount and detach the volume.
Notice that while describing the PVC the StorageClass name is ssd-encrypted which is a mismatch with the config you showed earlier where the kind: StorageClass name is ssd-default. That's why you can mount the volume manually but not via the StorageClass. You can drop and recreate the StorageClass with a proper data.
Also, I recommend going through this article and using volumeBindingMode: WaitForFirstConsumer instead of volumeBindingMode: Immediate. This setting instructs the volume provisioner to not create a volume immediately, and instead, wait for a pod using an associated PVC to run through scheduling.

kubernetes volume claim pending with specific volume name

I'm trying to create a PersistentVolumeClaim giv it a specific volumeName to use.
I use this code:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: zipkin
name: pvc-ciro
spec:
accessModes:
- ReadWriteOnce
storageClassName: nfs-provisioner
resources:
requests:
storage: 0.1Gi
volumeName: "demo"
If I remove volumeName the PVC is correctly bound otherways remain in pending status.
Why?
The volumeName is a name of the PersistentVolume you want to use.
On GKE PVC can automatically create a PV that will bound to, or you can specify the name of it using volumeName.
pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-ciro
spec:
accessModes:
- ReadWriteOnce
storageClassName: standard
resources:
requests:
storage: 0.1Gi
volumeName: demo
pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: demo
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: standard
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /tmp
server: 172.17.0.2
And the output will be:
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
demo 5Gi RWO Recycle Bound default/pvc-ciro standard 13s
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-ciro Bound demo 5Gi RWO standard 8s
You can read more details in Kubernetes documentation regarding Persistent Volumes.

Kubernetes : PVC binding status in pending

I create a PV and claimed the PV through PVC. I see that PV is created but the PVC binding status is stuck in pending.When i looked at the describe pvc output , I see that no persistent volumes available for this claim and no storage class is set. From the documentation I understand that storage class isnt mandatory . So, am unsure on what's missing in the PVC file.
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-ghost
labels:
pv: pv-ghost
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 3Gi
hostPath:
path: /ghost/data
--------------------------
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-ghost
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
selector:
matchLabels:
pv: pv-ghost
Out of describe PV and PVC
kubectl describe pv pv-ghost
Name: pv-ghost
Labels: pv=pv-ghost
Annotations: <none>
Finalizers: [kubernetes.io/pv-protection]
StorageClass:
Status: Available
Claim:
Reclaim Policy: Retain
Access Modes: RWX
Capacity: 3Gi
Node Affinity: <none>
Message:
Source:
Type: HostPath (bare host directory volume)
Path: /ghost/data
HostPathType:
Events: <none>
kubectl describe pvc pvc-ghost
Name: pvc-ghost
Namespace: default
StorageClass:
Status: Pending
Volume:
Labels: <none>
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 8m44s (x8 over 10m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
Normal FailedBinding 61s (x5 over 2m3s) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
Mounted By: <none>
You need to specify the volume source manually.
ReadWriteMany is only available for AzureFile, CephFS, Glusterfs, Quobyte, NFS, PortworxVolume.
Also Flexvolume depending on the drivers and VsphereVolume works when pods are collocated.
You can read it all in Kubernetes docs regarding Volume Mode
An example PV for aws would look like this:
apiVersion: v1
kind: PersistentVolume
metadata:
name: efs-volume
spec:
capacity:
storage: 15Gi # Doesn't really matter, as EFS does not enforce it anyway
volumeMode: Filesystem
accessModes:
- ReadWriteMany
mountOptions:
- hard
- nfsvers=4.1
- rsize=1048576
- wsize=1048576
- timeo=300
- retrans=2
nfs:
path: /
server: fs-XXX.efs.eu-central-2.amazonaws.com
In the above issue,
The Capacity specified in the persistent volume is lesser than the Persistent volume claim Capacity. Try either increasing the Capacity number in the Persistent volume to 5Gi or reducing the Capacity number in the Persistent volume claim to 3Gi.
When you are using a hostPath in the Persistent volume the accessModes should be ReadWriteOnce.
Hostpath method is currently not supported in a multi-node cluster.

Kubernetes - PVC not binding the NFS PV

I created a physical volume using NFS and PVC for the same volume. However, the PVC always creates a EBS disk storage instead of binding to the PV. Please see the log below:
> kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
mynfspv 100Gi RWX Retain Available 7s
create PVC now
> kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
mynfspvc Bound pvc-a081c470-3f23-11e7-9d30-024e42ef6b60 100Gi RWX default 4s
> kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
mynfspv 100Gi RWX Retain Available 50s
pvc-a081c470-3f23-11e7-9d30-024e42ef6b60 100Gi RWX Delete Bound default/mynfspvc default 17s
nfs-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: mynfspv
labels:
name: nfs2
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
nfs:
server: dbc56.efs.us-west-2.amazonaws.com
path: /
nfs-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mynfspvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100Gi
It looks like you have dynamic provisioning and default storageclass feature enabled, and the default class is AWS ebs. You can check your default class with following command:
$ kubectl get storageclasses
NAME TYPE
standard (default) kubernetes.io/aws-ebs
If this is correct, then I think you'll need to specify storage class to solve you problem.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: nfs-class
provisioner: kubernetes.io/fake-nfs
Add storageClassName to both of your PV
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mynfspvc
spec:
storageClassName: nfs-class
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100Gi
and PVC
apiVersion: v1
kind: PersistentVolume
metadata:
name: mynfspv
labels:
name: nfs2
spec:
storageClassName: nfs-class
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
nfs:
server: dbc56.efs.us-west-2.amazonaws.com
path: /
You can check out https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class-1 for details.
Which version of Kube is this ? The root cause is as mentioned by #ddysher. In your setup the "Default" Storage class is "EBS" as you can see in the get pv{c} outputs. According to kube version you can also make use of 'calim selector' in PVC spec. Refer # https://github.com/kubernetes/community/blob/master/contributors/design-proposals/volume-selectors.md