How to manual recover a PV - kubernetes

Question

according to the official docs https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/ with the “Retain” policy a PV can be manually recovered . What does that actually mean and is there a tool how I can read the data from that "retained" PV and write it into to another PV , or does it mean you can mount that volume manual in order to gain access ?

Answer 1

The process to manually recover the volume is as below.
You can use the same PV to mount to different pod along with the data even after the PVC is deleted (PV must exist, will typically exist if the reclaim policy of storageclass is Retain)
Verify that PV is in released state. (ie no pvc has claimed it currently)
➜ ~ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-eae6acda-59c7-11e9-ab12-06151ee9837e 16Gi RWO Retain Released default/dhanvi-test-pvc gp2 52m
Edit the PV (kubectl edit pv pvc-eae6acda-59c7-11e9-ab12-06151ee9837e) and remove the spec.claimRef part. The PV claim would be unset like below.
➜ ~ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-eae6acda-59c7-11e9-ab12-06151ee9837e 16Gi RWO Retain Available gp2 57m
Then claim the PV using PVC as below.
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: dhanvi-test-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 16Gi
volumeName: "pvc-eae6acda-59c7-11e9-ab12-06151ee9837e"
Can be used in the pods as below.
volumes:
- name: dhanvi-test-volume
persistentVolumeClaim:
claimName: dhanvi-test-pvc
Update: Volume cloning might help https://kubernetes.io/blog/2019/06/21/introducing-volume-cloning-alpha-for-kubernetes/

Answer 2

There are three reclaim policies which define what happens with the persistent volume after deletion of the bound volume claim
Retain
Delete
Recycle
Delete means the persistent volume as well as the associated storage asset in the external infrastructure is deleted.
Recycle will clean up the volume rm -rf /thevolume/* and after that it will be available for new persistent volume claims.
Retain leaves persistent volume in state released which does not allow for new persistent volume claims to reclaim it. The whole reclaim process is manual. You need to delete the persistent volume yourself. You can backup the data from the storage asset and delete the data afterwards. Then you can either delete the storage asset or create a new persistent volume for this asset.
If you want to write the data to another persistent volume using Kubernetes you could use a Job to copy the data.
In that case make sure you use persistent volume access modes ROX - ReadOnlyMany or RWX - ReadWriteMany and start a Job running a container which claims the persistent volume to be backed-up using a selector and claim another destination backup volume. Then copy the data via the container.
Alternatively, you can do the backup outside Kubernetes. Your method does then depend on the type of storage asset you are using. E.g., if you are using NFS you could mount source and destination and copy the data via command line.
Both options I've framed are more or less manual backup strategy. If you aim for a more sophisticated backup strategy for production workloads you might have a look at Stash - Backup for your disks for production workloads in Kubernetes

Answer 3

Like stated in the answer by Tummala Dhanvi, the spec.claimRef section has to be tackled with. While removing the whole spec.claimRef can be useful if you have only one PV, this will prove very nasty if you have multiple PVs to be "rescued".
The first step is to ensure the PV has the Retain reclaim policy before deleting the PVC. You can edit or patch the PV to achieve that
kubectl edit pv pvc-73e9252b-67ed-4350-bed0-7f27c92ce826
find the spec.persistentVolumeReclaimPolicy key
input Retain for its value
save & exit
or, in one command kubectl patch pv pvc-73e9252b-67ed-4350-bed0-7f27c92ce826 -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"Retain\"}}"
Now you can delete the PVC(s) (either by using helm or otherwise) and the PV(s) will not be deleted.
To successfully re-mount a PV to the desired pod you have to edit the PV configuration once again, this time the spec.claimRef section. But do not delete the whole section. Instead, delete only the resourceVersion and uid keys. The resulting section would then look something like this
...
capacity:
storage: 16Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: database
namespace: staging
nodeAffinity:
...
Repeat this for all of your PVs and their status in the kubectl get pv output will be Available afterwards. By leaving the spec.claimRef.name and spec.claimRef.namespace keys intact, we ensured that a new PVC with the corresponding spec (staging/database in my case), will be bound to the exact PV it is supposed to.
Also, make sure your new claim does not specify a larger storage capacity than the PV actually has (it seems though that the new claims' capacity may be less than the existing PV's). If the new PVC claims a larger storage, a new PV will be created instead. Best to keep it the same.
To digress: If the storageClass you're using allows for volume resizing, you can resize it later; here it's explained how: https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/
My experience with this was pretty stressful. I've had 6 PVs, thakfully in Retain mode. For some reason a new deployment rollout got stuck, two pods just wouldn't want to terminate. In the end I ended up with deleting the whole deployment (using helm), restarting the cluster nodes, and then redeploying anew. This caused 6 new PVs to be created!
I found this thread, and went on to delete the spec.claimRef of all the PVs. Deleting and deploying the installation once again resulted in the PVs being reused, but they were not mounted where they supposed to, the data was not there. After a good amount of digging, I figured out that the database volume was mounted to a RabbitMQ pod, mongodb was mounted to ElasticSearch etc.
It took me about a dozen times around, to get this right. Luckily, for me the mixed-up mounting of volumes did not destroy any of the original data. The pods initializations did not clean-out the volume, just wrote their stuff there.
Hope this saves some seriuos headaches out there!

Answer 4

I found this question for slightly different reasons than the existing answers address, so I'll weigh in.
In my case I had a self-managed Microk8s cluster for dev and testing purposes with
manual, local storage (mounted path on one of the Nodes), like so:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-1
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 50Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data-1"
There was also a PVC and a Deployment using this PV. Then I nuked the namespace where these resources lived by accident.
What I wanted to achieve is to recreate the whole namespace and have it use this PersistentVolume with the same data that was still present on the server.
In my case of local, manual storage all it took was just to delete the PV and create it again. Deleting the PV does NOT delete the actual data on the server (at least with the Retain policy). Recreating the PV with a mount into a path where data already exists also works fine - the data is just used.