On GKE I created a statefulset containing a volumeClaimTemplates. Then all the related PersistentVolumesClaims, PersistentVolumes and Google Persistent Disks are automatically created:
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
76m
qserv-data-qserv-worker-0 Bound pvc-c5e060dc-88cb-4630-8229-c4b1fcb4f64b 3Gi RWO qserv 76m
qserv-data-qserv-worker-1 Bound pvc-5dfffc24-165c-4e2c-a1fa-fa11dd45616f 3Gi RWO qserv 76m
qserv-data-qserv-worker-2 Bound pvc-14aa9a63-fae0-4328-aaaa-17db2dee4b79 3Gi RWO qserv 76m
qserv-data-qserv-worker-3 Bound pvc-8b701396-42ab-4d15-8b68-9b03ce5a2d07 3Gi RWO qserv 76m
qserv-data-qserv-worker-4 Bound pvc-7c49e7a0-fd73-467d-b677-820d899f41ee 3Gi RWO qserv 76m
kubectl get pv
pvc-14aa9a63-fae0-4328-aaaa-17db2dee4b79 3Gi RWO Retain Bound default/qserv-data-qserv-worker-2 qserv 77m
pvc-5dfffc24-165c-4e2c-a1fa-fa11dd45616f 3Gi RWO Retain Bound default/qserv-data-qserv-worker-1 qserv 77m
pvc-7c49e7a0-fd73-467d-b677-820d899f41ee 3Gi RWO Retain Bound default/qserv-data-qserv-worker-4 qserv 77m
pvc-8b701396-42ab-4d15-8b68-9b03ce5a2d07 3Gi RWO Retain Bound default/qserv-data-qserv-worker-3 qserv 77m
pvc-c5e060dc-88cb-4630-8229-c4b1fcb4f64b 3Gi RWO Retain Bound default/qserv-data-qserv-worker-0 qserv 77m
gcloud compute disks list
NAME LOCATION LOCATION_SCOPE SIZE_GB TYPE STATUS
...
pvc-14aa9a63-fae0-4328-aaaa-17db2dee4b79 us-central1-c zone 3 pd-balanced READY
pvc-5dfffc24-165c-4e2c-a1fa-fa11dd45616f us-central1-c zone 3 pd-balanced READY
pvc-7c49e7a0-fd73-467d-b677-820d899f41ee us-central1-c zone 3 pd-balanced READY
pvc-8b701396-42ab-4d15-8b68-9b03ce5a2d07 us-central1-c zone 3 pd-balanced READY
pvc-c5e060dc-88cb-4630-8229-c4b1fcb4f64b us-central1-c zone 3 pd-balanced READY
Is there a simple way to extract PVC/PV yaml file so that I can re-create all PVs/PVCs using the same Google Disks. (This might be useful to move the data to a new GKE cluster in case I delete the current one, or to restore the data if somebody remove accidentally the PVCs/PVs)
kubectl get pv,pvc -o yaml > export.yaml
Above command does not work because there is too much technical fields set at runtime which prevent kubectl apply -f export.yaml to work. Would you know a way to remove these fields from export.yaml
As asked in the question:
Is there a simple way to extract PVC/PV yaml file so that I can re-create all PVs/PVCs using the same Google Disks.
Some scripting would be needed to extract a manifest that could be used without any hassles.
I found a StackOverflow thread about similar question (how to export manifests):
Stackoverflow.com: Questions: Kubectl export is deprecated any alternative
A side note!
I also stumbled upon kubectl neat (a plugin for kubectl) which will be referenced later in that answer.
As correctly pointed by the author of the post, kubectl neat will show the message at the time of installation:
WARNING: You installed plugin "neat" from the krew-index plugin repository.
These plugins are not audited for security by the Krew maintainers.
Run them at your own risk.
I would consider going with some backup solution as more a viable option due to the fact of data persistence and in general data protection in case of any failure.
From backup solution side you can look here:
Portwortx.com: How to migrate stateful application from one gcp region to another with portwortx kubemotion
Velero.io
PV's in GKE are in fact Google Persistent Disks. You can create a snapshot/image of a disk as a backup measure. You can also use this feature to test how your migration behaves:
Cloud.google.com: Compute: Docs: Disks: Create snapshots
Cloud.google.com: Compute: Docs: Images: Create delete deprecate private images
Please consider below example as a workaround.
I've managed to migrate an example Statefulset from one cluster to another with data stored on gce-pd.
Once more, I encourage you to check this docs about using preexisting disks in GKE to create a Statefulset:
Cloud.google.com: Kubernetes Engine: Docs: How to: Persistent Volumes: Preexisting PD
Assuming that you used manifest from official Kubernetes site:
Kubernetes.io: Docs: Concepts: Workloads: Controllers: Statefulset
You can migrate it to another cluster by:
Setting the ReclaimPolicy to Retain on each PV used. <-- IMPORTANT
Using kubectl neat to extract needed manifests
Editing previously extracted manifests
Deleting the existing workload on old cluster
Creating a workload on new cluster
Setting the ReclaimPolicy to Retain on each PV used
You will need to check if your PV ReclaimPolicy is set to Retain. This would stop gce-pd deletion after PVC and PV are deleted from the cluster. You can do it by following Kubernetes documentation:
Kubernetes.io: Docs: Tasks: Administer cluster: Change PV reclaim policy
More reference:
Cloud.google.com: Kubernetes Engine: Docs: Concepts: Persistent Volumes: Dynamic provisioning
Using kubectl neat to extract needed manifests
There are many ways to extract the manifest from Kubernetes API. I stumbled upon kubectl neat here. kubectl-neat will remove some of the fields in the manifests.
I used it in a following manner:
$ kubectl get statefulset -o yaml | kubectl neat > final-sts.yaml
$ kubectl get pvc -o yaml | kubectl neat > final-pvc.yaml
$ kubectl get pv -o yaml | kubectl neat > final-pv.yaml
Disclaimer!
This workaround will use the names of the dynamically created disks in GCP. If you were to create new disks (from snapshot for example) you would need to modify whole setup (use preexsiting disks guide referenced earlier).
Above commands would store manifests of a StatefulSet used in the Kubernetes examples.
Editing previously extracted manifests
You will need to edit this manifests to be used in newly created cluster. This part could be automated:
final-pv.yaml - delete the .claimRef in .spec
Deleting the existing workload on old cluster
You will need to release used disks so that the new cluster could use them. You will need to delete this Statefulset and accompanying PVC's and PV's. Please make sure that the PV's reclaimPolicy is set to Retain.
Creating a workload on new cluster
You will need to use previously created manifest and apply them on a new cluster:
$ kubectl apply -f final-pv.yaml
$ kubectl apply -f final-pvc.yaml
$ kubectl apply -f final-sts.yaml
As for exporting manifests you could also look (if feasible) on Kubernetes client libraries:
Kubernetes.io: Docs: Reference: Using API: Client libraries
Here is an example script which replace kubectl neat and manual edit of the manifest files (removal of .spec.claimRef field) in order to export the mapping between PVCs,PVs and Google Persistent Disks.
https://github.com/k8s-school/skateful/blob/stackoverflow/main.go
How to use it:
git clone https://github.com/k8s-school/skateful.git
git checkout stackoverflow
# Requirements: go >= 1.14.7 and a kubeconfig file
make
./skateful
it will create a pvc-pv.yaml file that can be applied to any new GKE kubernetes cluster in order to attach the existing Google Persistent Disks to new PVCs/PVs.
Related
I have a persistent volume (PV) and persistent volume claim (PVC) which got bound as well. Initially, the storage capacity was 2Gi for the PV and the requested storage from PVC was 1Gi.
I then edit the existing bound PV and increased the storage to 5Gi with the record flag as --record.
vagrant#mykubemaster:~/my-k8s$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
my-pv 2Gi RWO Retain Bound test/my-pvc 106s
vagrant#mykubemaster:~/my-k8s$ kubectl edit pv my-pv --record
persistentvolume/my-pv edited
vagrant#mykubemaster:~/my-k8s$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
my-pv 5Gi RWO Retain Bound test/my-pvc 2m37s
Now my question is if there is any way by which I can confirm that this --record flag have certainly recorded this storage change (edit PV) in history.
With deployments, it is easy to check with the kubectl rollout history <deployment name> but I'm not sure how to check this with other objects like PV.
Please assist. thanks
As mentioned in kubectl references docs:
Record current kubectl command in the resource annotation. If set to false, do not record the command. If set to true, record the command. If not set, default to updating the existing annotation value only if one already exists.
You can run kubectl get pv my-pv -o yaml and you should see that kubernetes.io/change-cause was updated with the command that you ran. In your case, it will be kubectl edit pv my-pv --record.
The rollout command that you mentioned (including rollout history) works only with the following resources:
deployments
daemonsets
statefulsets
Platform: OEL 7.7 + kube 1.15.5 + docker 19.03.1
We're building an erasure-coded object store on k8s using a containerized kubelet approach. We're having a tough time coming up with a viable disk life cycle approach. As it is now, we must provide an "extra_binds" argument to the kubelet which specifies the base mount point where our block devices are mounted. (80 SSDs per node, formatted as ext4)
That all works fine. Creating PV's and deploying apps works fine. Our problem comes when a PVC is deleted and we want to scrub the disk(s) that were used and make the disk(s) available again.
So far the only thing that works is to cordon that node, remove the extra binds from kubelet, bounce the node, reconfigure the block device, re-add the kubelet binds. Obviously this is too clunky for production. For starters, bouncing kubelet is not an option.
Once a PV gets used, something is locking this block device, even though checking lsof on the bare metal system shows non open handles. I can't unmount or create a new filesystem on the device. Merely bouncing kubelet doesn't free up the "lock".
Anyone using a containerized kubernetes control plane with an app using local disks in a similar fashion? Anyone found a viable way to work around this issue?
Our long term plan is to write an operator that manages disks but even with an operator I don't see how it can mitigate this problem.
Thanks for any help,
First look at your Finalizers:
$ kubectl describe pvc <PVC_NAME> | grep Finalizers
$ kubectl describe pv <PV_NAME> | grep Finalizers
if they are set to Finalizers: [kubernetes.io/pvc-protection] (explained here) that mean they are protected and you need to edit that, for example using:
$ kubectl patch pvc <PVC_NAME> -p '{"metadata":{"finalizers":null}}'
As for forcefully removing PersistentVolumes you can try
$ kubectl delete pv <PV_NAME> --force --grace-period=0
Also please check VolumeAttachment do still exist $ kubectl get volumeattachment as they might be blocked.
I also remember there was as issue on stack Kubernetes PV refuses to bind after delete/re-create stating that pv holds uid of pvc that was claimed by.
You can check that by displaying whole yaml of the pv:
$ kubectl get pv <PV_NAME> -o yaml and looking for:
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: packages-pvc
namespace: default
resourceVersion: "10218121"
uid: 1aede3e6-eaa1-11e9-a594-42010a9c0005
You would need to provide more information regarding your k8s cluster and pv, pvc configuration so I could go deeper into to or even test it.
I have a Kubernetes cluster with some pods deployed (DB, Frontend, Redis). A part that I can't fully grasp is what happens to the PVC after the pod is deleted.
For example, if I delete POD_A which is bound to CLAIM_A I know that CLAIM_A is not deleted automatically. If I then try to recreate the POD, it is attached back to the same PVC but the all the data is missing.
Can anyone explain what happens, I've looked at the official documentation but not making any sense at the moment.
Any help is appreciated.
PVCs have a lifetime independent of pods.
If PV still exists it may be because it has ReclaimPolicy set to Retain in which case it won't be deleted even if PVC is gone.
PersistentVolumes can have various reclaim policies, including “Retain”, “Recycle”, and “Delete”. For dynamically provisioned PersistentVolumes, the default reclaim policy is “Delete”. This means that a dynamically provisioned volume is automatically deleted when a user deletes the corresponding PersistentVolumeClaim. This automatic behavior might be inappropriate if the volume contains precious data.
Notice that the RECLAIM POLICY is Delete (default value), which is one of the two reclaim policies, the other one is Retain. (A third policy Recycle has been deprecated). In case of Delete, the PV is deleted automatically when the PVC is removed, and the data on the PVC will also be lost.
In that case, it is more appropriate to use the “Retain” policy. With the “Retain” policy, if a user deletes a PersistentVolumeClaim, the corresponding PersistentVolume is not be deleted. Instead, it is moved to the Released phase, where all of its data can be manually recovered.
This may also happens too when persistent volume is protected. You should be able to cross verify this:
Command:
$ kubectl describe pvc PVC_NAME | grep Finalizers
Output:
Finalizers: [kubernetes.io/pvc-protection]
You can fix this by setting finalizers to null using kubectl patch:
$ kubectl patch pvc PVC_NAME -p '{"metadata":{"finalizers": []}}' --type=merge
EDIT:
A PersistentVolume can be mounted on a host in any way supported by the resource provider. Each PV gets its own set of access modes describing that specific PV’s capabilities.
The access modes are:
ReadWriteOnce – the volume can be mounted as read-write by a single
node
ReadOnlyMany – the volume can be mounted read-only by many nodes
ReadWriteMany – the volume can be mounted as read-write by many nodes
In the CLI, the access modes are abbreviated to:
RWO - ReadWriteOnce
ROX - ReadOnlyMany
RWX - ReadWriteMany
So if you recreated pod and scheduler put it on different node and your PV has reclaim policy set to ReadWriteOnce it is normal that you cannot access your data.
Claims use the same conventions as volumes when requesting storage with specific access modes. My advice is to edit PV access mode to ReadWriteMany.
$ kubectl edit pv your_pv
You should be updating the access mode in PersistentVolume as shown below
accessModes:
- ReadWriteMany
I have a PV:
pvc-6b1a6054-c35e-11e9-afd7-0eeeeb629aaa 100Gi RWO Delete Bound pipeline-aws/mln13-0 performance 28h
Can I create a pvc to bind to this pv?
kubectl get pvc
doesn't show pvc mln13-0
Your pvc has bound to pv, in namespace pipeline-aws, so you can show your pvc with command:
kubectl get pvc -n pipeline-aws
In your case Persistent Volume is automatically created when it is dynamically provisioned. In following example, the PVC is defined as mln13-0, and a corresponding PV pvc-6b1a6054-c35e-11e9-afd7-0eeeeb629aaa is created and associated with PVC automatically.
Notice that the RECLAIM POLICY is Delete (default value), which is one of the two possible reclaim policies, the other one is Retain. In case of Delete, the PV is deleted automatically when the PVC is removed, and the data on the PVC will also be lost.
On the other hand, PV with Retain policy will not be deleted when the PVC is removed, and moved to Release status, so that data can be recovered by Administrators later.
With following command you can list all of PVCs in all namespaces along with the corresponding PVs:
$ kubectl get pvc --all-namespaces
Also what is interesting the PV can be accessed by any project/namespace, however once it is bound to a project, it can then only be accessed by containers from the same project/namespace. PVC is project/namespace specific, it means that if you have mulitple projects you would need to have a new PV and PVC for each project.
You can read more about binding in official K8S documentation.
Is is possible to rename a PVC? I can't seem to find an evidence it is possible.
I'm trying mitigate an "No space left of device" issue I just stumbled upon. Essentially my plan requires me to resize the volume, on which my service persists its data.
Unfortunately I'm still on Kubernetes 1.8.6 on GKE. It does not have the PersistentVolumeClaimResize admission plugin enabled:
1.9.1: config-default.sh#L254#1.9.1
1.8.6: config-default.sh#L254#1.8.6
Therefor I have to try and save the data manually. I made the following plan:
create a new, bigger volume PVC,
create a temp container with attached "victim" pvc and a new bigger pvc,
copy the data,
drop "victim" PVC,
rename new bigger pvc to take place of "victim".
The PVC in question is attached to StatefulSet, so the old and new names must match (as StatefulSet expects follows the volume naming convention).
But I don't understand how to rename persistent volume claims.
The answer of your question is NO. There is no way to change any meta name in Kubernetes.
But, there is a way to fulfill your requirement.
You want to claim your new bigger PersistentVolume by old PersistentVolumeClaim.
Lets say, old PVC named victim and new PVC named bigger. You want to claim PV created for bigger by victim PVC. Because your application is already using victim PVC.
Follow these steps to do the hack.
Step 1: Delete your old PVC victim.
Step 2: Make PV of bigger Available.
$ kubectl get pvc bigger
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
bigger Bound pvc-8badc3c2-08c5-11e8-b07a-080027b3e1a6 10Gi RWO standard 30s
Edit PV pvc-8badc3c2-08c5-11e8-b07a-080027b3e1a6 to set persistentVolumeReclaimPolicy to Retain. So that deleting PVC will not delete PV.
Now, delete PVC bigger.
$ kubectl delete pvc bigger
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-8badc3c2-08c5-11e8-b07a-080027b3e1a6 10Gi RWO Retain Released default/bigger standard 3m
See the status, PV is Released.
Now, make this PV available to be claimed by another PVC, our victim.
Edit PV again to remove claimRef
$ kubectl edit pv pvc-8badc3c2-08c5-11e8-b07a-080027b3e1a6
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-8badc3c2-08c5-11e8-b07a-080027b3e1a6 10Gi RWO Retain Available standard 6m
Now the status of PV is Available.
Step 3: Claim bigger PV by victim PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: victim
spec:
accessModes:
- ReadWriteOnce
volumeName: pvc-8badc3c2-08c5-11e8-b07a-080027b3e1a6
resources:
requests:
storage: 10Gi
Use volumeName pvc-8badc3c2-08c5-11e8-b07a-080027b3e1a6
kubectl get pvc,pv
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc/victim Bound pvc-8badc3c2-08c5-11e8-b07a-080027b3e1a6 10Gi RWO standard 9s
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv/pvc-8badc3c2-08c5-11e8-b07a-080027b3e1a6 10Gi RWO Retain Bound default/victim standard 9m
Finally: Set persistentVolumeReclaimPolicy to Delete
This is how, your PVC victim has had bigger PV.
With Kubernetes 1.11+ you can perform on-demand resizing by simply modifying the PVC's storage request (https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/).
GKE supports this (I've used it several times my self) and it's pretty straightforward and without the drama.
I cannot validate this however I am fairly certain that for GKE you can go to disks in the Google Cloud Console and find the one that the PV uses and resize it there. Once you've done that you should be able to log into the node which its attached and run resize2fs on the device. This is dirty, but fairly certain this has worked for me once in the past.
You don't have to unmount or copy to do this, which can save you if the disk is live or large.