GKE Persistent Volume Not Saving Data - kubernetes

Question

I have created a Persistent Volume and Volume claim for an app I am working on in GKE. The claim and storage appear to be setup correctly, however, the data doesn't persist if the pod is restarted. I am able to save data initially and I can see the file in the pod, but it disappears after restarting it.
I had asked this question previously, but didn't include my .yaml files and received a sort of generic answer as a result so I decided to repost with the .yaml files hoping someone could look at them and tell me where I am going wrong. From everything I've seen, it looks like the problem is in the Persistent Volume as the claim looks exactly like everyone else's.
apiVersion: apps/v1
kind: Deployment
metadata:
name: prod-api-meta-uploads-k8s
namespace: default
resourceVersion: "4500192"
selfLink: /apis/apps/v1/namespaces/default/deployments/prod-api-meta-uploads-k8s
uid: *******
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: prod-api-meta-uploads-k8s
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
gcb-build-id: *****
gcb-trigger-id:****
creationTimestamp: null
labels:
app: prod-api-meta-uploads-k8s
app.kubernetes.io/managed-by: gcp-cloud-build-deploy
app.kubernetes.io/name: prod-api-meta-uploads-k8s
app.kubernetes.io/version: becdb864864f25d2dcde2e62a2f70501cfd09f19
spec:
containers:
- image: bitbucket.org/api-meta-uploads-k8s#sha256:7766413c0d
imagePullPolicy: IfNotPresent
name: prod-api-meta-uploads-k8s-sha256-1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /uploads/profileImages
name: uploads-volume-prod
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: uploads-volume-prod
persistentVolumeClaim:
claimName: my-disk-claim-1
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2020-09-08T21:00:40Z"
lastUpdateTime: "2020-09-10T04:54:27Z"
message: ReplicaSet "prod-api-meta-uploads-k8s-5c8f66f886" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: "2020-09-10T06:49:41Z"
lastUpdateTime: "2020-09-10T06:49:41Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 36
readyReplicas: 1
replicas: 1
updatedReplicas: 1
** Volume Claim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: "2020-09-09T16:12:51Z"
finalizers:
- kubernetes.io/pvc-protection
name: uploads-volume-prod
namespace: default
resourceVersion: "4157429"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/uploads-volume-prod
uid: f93e6134
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
storageClassName: standard
volumeMode: Filesystem
volumeName: pvc-f93e6
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 30Gi
phase: Bound
*** PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
finalizers:
- kubernetes.io/pvc-protection
name: my-disk-claim-1
namespace: default
resourceVersion: "4452471"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/my-disk-claim-1
uid: d533702b
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: fast
volumeMode: Filesystem
volumeName: pvc-d533702b
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 50Gi
phase: Bound

Answer 1

As you are using GKE you don't need to prepare PersistentVolume and PersistentVolumeClaim manually (static provisioning) in relationship 1:1, as GKE can use Dynamic Volume Provisioning.
It's good described in Persistent Volumes
When none of the static PVs the administrator created match a user's PersistentVolumeClaim, the cluster may try to dynamically provision a volume specially for the PVC.
In GKE on the beginning you have at least one storageclass named standard. It also have (default) next to name.
$ kubectl get sc
NAME PROVISIONER AGE
standard (default) kubernetes.io/gce-pd 110m
It means that if you won't specify storageClassName in your PersistentVolumeClaim, it will use storageclass which is set as default. In your YAMLs I can see that you have used storageClassName: standard. If you will check this storageclass you will se what ReclaimPolicy was set to delete. Below output:
$ kubectl describe sc standard
Name: standard
IsDefaultClass: Yes
Annotations: storageclass.kubernetes.io/is-default-class=true
Provisioner: kubernetes.io/gce-pd
Parameters: type=pd-standard
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
IsDefaultClass: means this storageclass is set as default.
ReclaimPolicy: defining ReclaimPolicy, delete in this case
As ReclaimPolicy is set to Delete:
For volume plugins that support the Delete reclaim policy, deletion removes both the PersistentVolume object from Kubernetes, as well as the associated storage asset in the external infrastructure, such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume. Volumes that were dynamically provisioned inherit the reclaim policy of their StorageClass, which defaults to Delete. The administrator should configure the StorageClass according to users' expectations; otherwise, the PV must be edited or patched after it is created.
Depends on your needs, you can use:
Recycle:
If supported by the underlying volume plugin, the Recycle reclaim policy performs a basic scrub (rm -rf /thevolume/*) on the volume and makes it available again for a new claim.
However, please keep in mind that: Warning: The Recycle reclaim policy is deprecated. Instead, the recommended approach is to use dynamic provisioning. Adding this option as I didnt see what K8s version are you using, however it's not supported on GKE.
Retain:
The Retain reclaim policy allows for manual reclamation of the resource. When the PersistentVolumeClaim is deleted, the PersistentVolume still exists and the volume is considered "released". But it is not yet available for another claim because the previous claimant's data remains on the volume.
In addition, as you are using GKE, it supports only Delete and Retain.
The StorageClass "another-storageclass" is invalid: reclaimPolicy: Unsupported value: "Recycle": supported values: "Delete", "Retain"
In addition, as you specfied revisionHistoryLimit: 10, pod after 10 restarts will be recreated, in that situation pod, pv and pvc will be deleted when ReclaimPolicy will be set as delete.
Solution
As easiest solution, you should create new StorageClass with ReclaimPolicy different from Delete and use it in your PVC.
$ kubectl get sc,pv,pvc -A
NAME PROVISIONER AGE
storageclass.storage.k8s.io/another-storageclass kubernetes.io/gce-pd 53s
storageclass.storage.k8s.io/standard (default) kubernetes.io/gce-pd 130m
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-67c35c06-3f38-4f55-98c8-6b2b41ae5313 1Gi RWO Retain Bound tst-dev/pvc-1 another-storageclass 43s
persistentvolume/pvc-be30a43f-e96c-4c9f-8863-464823899a8f 1Gi RWO Retain Bound tst-stage/pvc-2 another-storageclass 42s
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
tst-dev persistentvolumeclaim/pvc-1 Bound pvc-67c35c06-3f38-4f55-98c8-6b2b41ae5313 1Gi RWO another-storageclass 46s
tst-stage persistentvolumeclaim/pvc-2 Bound pvc-be30a43f-e96c-4c9f-8863-464823899a8f 1Gi RWO another-storageclass 45s
$ kubectl delete pvc pvc-1 -n tst-dev
persistentvolumeclaim "pvc-1" deleted
user#cloudshell:~ (project)$ kubectl delete pvc pvc-2 -n tst-stage
persistentvolumeclaim "pvc-2" deleted
user#cloudshell:~ (project)$ kubectl get sc,pv,pvc -A
NAME PROVISIONER AGE
storageclass.storage.k8s.io/another-storageclass kubernetes.io/gce-pd 7m49s
storageclass.storage.k8s.io/standard (default) kubernetes.io/gce-pd 137m
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-67c35c06-3f38-4f55-98c8-6b2b41ae5313 1Gi RWO Retain Released tst-dev/pvc-1 another-storageclass 7m38s
persistentvolume/pvc-be30a43f-e96c-4c9f-8863-464823899a8f 1Gi RWO Retain Released tst-stage/pvc-2 another-storageclass 7m37s