Disk created from GCP snapshot doesn't resize in kubernetes - kubernetes

I've created GCP's disk form a snapshot and now I'm trying to resize it using PVC in kubernetes: 100GB -> 400GB. I've applied:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: restored-resize
parameters:
type: pd-standard
provisioner: kubernetes.io/gce-pd
allowVolumeExpansion: true
reclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: restored-graphite
spec:
storageClassName: restored-resize
capacity:
storage: 400G
accessModes:
- ReadWriteOnce
gcePersistentDisk:
pdName: dev-restored-graphite
fsType: ext4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: restored-graphite
spec:
# It's necessary to specify "" as the storageClassName
# so that the default storage class won't be used, see
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class-1
storageClassName: restored-resize
volumeName: restored-graphite
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 400G
Status in PVC shows 400G:
(...)
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 400G
phase: Bound
However pod mounts previous disk value:
/dev/sdc 98.4G 72.8G 25.6G 74% /opt/graphite/storage
What am I doing wrong?

To me seems that you have setted 400G directly on the manifest, but as the manual said, you should had edited only
resources:
requests:
storage: 400G
https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/
and those, triggering the new condition: FileSystemResizePending
As of Kubernetes v1.11, those PVC will autoresize in time after some time in this status, due to that, you shouldn't even have to restart the pod bounded to the pc.
But, again on your problem: i would edit this way the manifes:
spec:
storageClassName: restored-resize
capacity:
storage: 100G
in order for the system to reload the old config and notice that the situation is not as he thinks.
or at least, that is what i would try (on another environment, not production for sure.)

Related

Cloned drive setup as one pv (or two mirroring pv's)

I'm running a k3s cluster in two different locations. I'm currently running a PV in one of these locations and I'm trying to develop a configuration that can be read as one PV but clone/mirror that drive to another location, all of this through k3s PV and PVC. Any clever ideas on how to achieve this?
My PV and PVC looks like this:
# pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-data-pv-volume
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 32Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/srv/dev-disk-by-uuid-********-****-****-****-************/kubernetes"
# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-data-pv-claim
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 12Gi

kubernetes ignoring persistentvolume

I have created a persistent volume:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-volume
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "C:/Users/xxx/Desktop/pv"
And want to make save mysql statefulset pods things on it.
So, I wrote the volumeclaimtemplate:
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
Thinking this would request the persistent storage from the only persistent volume I have. Instead, this is what happens:
StatefulSets requires you to use storage classes in order to bind the correct PVs with the correct PVCs.
The correct way to make StatefulSets mount local storage is by using local type of volumes, take a look at the procedure below.
First, you create a storage class for the local volumes. Something like the following:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
It has no-provisioner so it will not be able to automatically provision PVs, you'll need to create them manually, but that's exactly what you want for local storage.
Second, you create your local PV, something as the following:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-volume
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: "C:/Users/xxx/Desktop/pv"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- the-node-hostname-on-which-the-storage-is-located
This definition tells the local path on the node, but also forces the PV to be used on a specific node (which match the nodeSelectorTerms).
It also links this PV to the storage class created earlier. This means that now, if a StatefulSets requires a storage with that storage class, it will receive this disk (if the space required is less or equal, of course)
Third, you can now link the StatefulSet:
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "local-storage"
resources:
requests:
storage: 5Gi
When the StatefulSet Pod will need to be scheduled for the first time, the following will happen:
A PVC will be created and it will go Bound with the PV you just created
The Pod will be scheduled to run on the node on which the bounded PV is restricted to run
UPDATE:
In case you want to use hostPath storage instead of local storage (because for example you are on minikube and that is supported out of the box directly, so it's more easy) you need to change the PV declaration a bit, something like the following:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-volume
spec:
storageClassName: local-storage
accessModes:
- ReadWriteOnce
capacity:
storage: 5Gi
hostPath:
path: /data/pv0001/
Now, the /data directory and all its content is persisted on the host (so if minikube gets restarted, it's still there) but if you want to mount specific directories of your host, you need to use minikube mount, for example:
minikube mount <source directory>:<target directory>
For example, you could do:
minikube mount C:/Users/xxx/Desktop/pv:/host/my-special-pv
and then you could use /host/my-special-pv as the hostPath inside the PV declaration.
More info can be read in the docs.

How to share a cephfs volume between pods in different k8s namespaces

I'm trying to share a cephfs volumes between namespaces within k8s cluster.
I'm using ceph-csi with cephfs.
Followed https://github.com/ceph/ceph-csi/blob/devel/docs/static-pvc.md#cephfs-static-pvc to create static pv+pvc in both namespaces.
Works if I don't launch both pods on same node.
If both pods on same node, Second pod get stuck with error:
MountVolume.SetUp failed for volume "team-test-vol-pv" : rpc error: code = Internal desc = failed to bind-mount /var/lib/kubelet/plugins/k
ubernetes.io/csi/pv/team-test-vol-pv/globalmount to /var/lib/kubelet/pods/007fc605-7fa4-4dc6-890f-fc0dabe5740b/volumes/kubernetes.io~csi/team-test-vol-pv/mount: an error (exit status 32) occurred while running mount arg
s: [-o bind,_netdev /var/lib/kubelet/plugins/kubernetes.io/csi/pv/team-test-vol-pv/globalmount /var/lib/kubelet/pods/007fc605-7fa4-4dc6-890f-fc0dabe5740b/volumes/kubernetes.io~csi/team-test-vol-pv/moun
Any ideas how to resolve this or how to use a single RWX volume in different NS?
PV+PVC for team-x:
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-vol
namespace: team-x
spec:
storageClassName: ""
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
volumeMode: Filesystem
# volumeName should be same as PV name
volumeName: team-x-test-vol-pv
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: team-x-test-vol-pv
spec:
claimRef:
namespace: team-x
name: test-vol
storageClassName: ""
accessModes:
- ReadWriteMany
capacity:
storage: 1Gi
csi:
driver: cephfs.csi.ceph.com
nodeStageSecretRef:
name: csi-cephfs-secret-hd
namespace: ceph-csi
volumeAttributes:
"clusterID": "cd79ae11-1804-4c06-a97e-aeeb961b84b0"
"fsName": "cephfs"
"staticVolume": "true"
"rootPath": /volumes/team/share/8b73d3bb-282e-4c32-b13a-97459419bd5b
# volumeHandle can be anything, need not to be same
# as PV name or volume name. keeping same for brevity
volumeHandle: team-share
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
PV+PVC for team-y
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-vol
namespace: team-y
spec:
storageClassName: ""
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
volumeMode: Filesystem
# volumeName should be same as PV name
volumeName: team-y-test-vol-pv
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: team-y-test-vol-pv
spec:
claimRef:
namespace: team-y
name: test-vol
storageClassName: ""
accessModes:
- ReadWriteMany
capacity:
storage: 1Gi
csi:
driver: cephfs.csi.ceph.com
nodeStageSecretRef:
name: csi-cephfs-secret-hd
namespace: ceph-csi
volumeAttributes:
"clusterID": "cd79ae11-1804-4c06-a97e-aeeb961b84b0"
"fsName": "cephfs"
"staticVolume": "true"
"rootPath": /volumes/team-y/share/8b73d3bb-282e-4c32-b13a-97459419bd5b
# volumeHandle can be anything, need not to be same
# as PV name or volume name. keeping same for brevity
volumeHandle: team-share
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
Having volumeHandle: xyz unique for each pv done the trick. Tested deploying 3xdaemonsets in 3 different namespaces.
you might need to provide ReadWriteMany option
Reference link: https://kubernetes.io/docs/concepts/storage/persistent-volumes/

kubernetes volume claim pending with specific volume name

I'm trying to create a PersistentVolumeClaim giv it a specific volumeName to use.
I use this code:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: zipkin
name: pvc-ciro
spec:
accessModes:
- ReadWriteOnce
storageClassName: nfs-provisioner
resources:
requests:
storage: 0.1Gi
volumeName: "demo"
If I remove volumeName the PVC is correctly bound otherways remain in pending status.
Why?
The volumeName is a name of the PersistentVolume you want to use.
On GKE PVC can automatically create a PV that will bound to, or you can specify the name of it using volumeName.
pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-ciro
spec:
accessModes:
- ReadWriteOnce
storageClassName: standard
resources:
requests:
storage: 0.1Gi
volumeName: demo
pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: demo
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: standard
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /tmp
server: 172.17.0.2
And the output will be:
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
demo 5Gi RWO Recycle Bound default/pvc-ciro standard 13s
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-ciro Bound demo 5Gi RWO standard 8s
You can read more details in Kubernetes documentation regarding Persistent Volumes.

Snapshotting on google cloud/Kubernetes when using storageClass persistent volumes

StorageClasses are the new method of specifying dynamic persistent volume claim (PVC) dependencies within Kubernetes. This avoids the need to explicitly provision one directly with the cloud provider (in my case Google Container Engine (GKE)).
Definition for the StorageClasses (GKE already has a default for standard class)
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-ssd
zone: europe-west1-b
Definition for the actual PVC
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs-server-pvc
namespace: staging
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 100Gi
storageClassName: "standard"
Here is the result of kubernetes get storageclass:
NAME TYPE
fast kubernetes.io/gce-pd
standard (default) kubernetes.io/gce-pd
Here is the result of kubernetes get pvc:
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
nfs-pvc Bound nfs 1Mi RWX 119d
nfs-server-pvc Bound pvc-905a810b-3f13-11e7-82f9-42010a840072 100Gi RWO standard 81d
I would like to continue taking snapshots of the volumes but the dynamic nature of the volume names created (in this case pvc-905a810b-3f13-11e7-82f9-42010a840072), mean i cannot continue with the following command that i had been using via cron (note the "nfs" name is now incorrect):
gcloud compute --project "XXX-XXX" disks snapshot "nfs" --zone "europe-west1-b" --snapshot-names "nfs-${DATE}"
I guess this boils down to Kubernetes allowing explicit naming through StorageClass-based PVC. The docs don't seem to allow this. Any ideas?
One approach is to manually create the PV and give it a stable name that you can use in your scripts. You can use gcloud commands to create the underlying PD disks. When you create the PV, give it a label:
apiVersion: "v1"
kind: "PersistentVolume"
metadata:
name: my-pv-0
labels:
pdName: my-pv-0
spec:
capacity:
storage: "10Gi"
accessModes:
- "ReadWriteOnce"
storageClassName: fast
gcePersistentDisk:
fsType: "ext4"
pdName: "my-pd-0"
Then attach it to the PVC using a selector:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: my-pvc-0
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: fast
selector:
matchLabels:
pdName: my-pv-0