Failed to provision volume with StorageClass "slow": Failed to get GCE GCECloudProvider with error <nil> - kubernetes

I'm trying to install Redis cluster (StatefulSet) out of GKE and when getting pvc I've got
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 10s persistentvolume-controller Failed to provision volume with StorageClass "slow": Failed to get GCE GCECloudProvider with error <nil>
Already added "--cloud-provider=gce" on files /etc/kubernetes/manifests/kube-controller-manager.yaml and sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml. Restarted but still the same.
Can anyone help me please? What's the trick for making k8s work on GCP?
My manifest taken from here:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-cluster
labels:
app: redis-cluster
data:
fix-ip.sh: |
#!/bin/sh
CLUSTER_CONFIG="/data/nodes.conf"
if [ -f ${CLUSTER_CONFIG} ]; then
if [ -z "${POD_IP}" ]; then
echo "Unable to determine Pod IP address!"
exit 1
fi
echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"
sed -i.bak -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${CLUSTER_CONFIG}
fi
exec "$#"
redis.conf: |+
cluster-enabled yes
cluster-require-full-coverage no
cluster-node-timeout 15000
cluster-config-file /data/nodes.conf
cluster-migration-barrier 1
appendonly yes
protected-mode no
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: slow
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
replication-type: none
zone: "us-west2-a"
reclaimPolicy: Retain
---
apiVersion: v1
kind: Service
metadata:
name: redis-cluster
labels:
app: redis-cluster
spec:
ports:
- port: 6379
targetPort: 6379
name: client
- port: 16379
targetPort: 16379
name: gossip
clusterIP: None
selector:
app: redis-cluster
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-cluster
labels:
app: redis-cluster
spec:
serviceName: redis-cluster
replicas: 5
selector:
matchLabels:
app: redis-cluster
template:
metadata:
labels:
app: redis-cluster
spec:
containers:
- name: redis
image: redis:5.0-rc
ports:
- containerPort: 6379
name: client
- containerPort: 16379
name: gossip
command: ["/conf/fix-ip.sh", "redis-server", "/conf/redis.conf"]
args:
- --cluster-announce-ip
- "$(POD_IP)"
readinessProbe:
exec:
command:
- sh
- -c
- "redis-cli -h $(hostname) ping"
initialDelaySeconds: 15
timeoutSeconds: 5
livenessProbe:
exec:
command:
- sh
- -c
- "redis-cli -h $(hostname) ping"
initialDelaySeconds: 20
periodSeconds: 3
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- name: conf
mountPath: /conf
readOnly: false
- name: data
mountPath: /data
readOnly: false
volumes:
- name: conf
configMap:
name: redis-cluster
defaultMode: 0755
volumeClaimTemplates:
- metadata:
name: data
labels:
name: redis-cluster
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: slow
resources:
requests:
storage: 5Gi

Please verify your "StorageClass: slow", it seems there is an indentation problem (starting with reclaimPolicy)
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: slow
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
replication-type: none
zone: "us-west2-a"
reclaimPolicy: Retain
#
Update:
Please add --cloud-provider=gce into: kube-apiserver.yaml, kube-controller-manager.yaml, KUBELET_KUBECONFIG_ARGS. You can also enable enable-admission-plugins=DefaultStorageClass
Verify in your "VM instance details" permissiosn in "Cloud API access scopes" permissions.
Verify if your storage class pv and pvc are working properly.
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
name: slow
annotations:
storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-test
spec:
accessModes:
- ReadOnlyMany
storageClassName: slow
resources:
requests:
storage: 1Gi

Google offers two main types of persistent disk, which are provisioned automatically on kubernetes:
Standard storage (labeled pd-standard)
SSD storage (labeled pd-ssd)
By default, GKE will provision standard storage persistent disks. In fact, that’s the only storage class even available at first.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: test-storageclass
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-ssd
you can tell a persistent volume to use the new ssd storage class with the following key/value pair: storageClassName: ssd.
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: ssd-storageclass
spec:
accessModes:
- ReadWriteOnce
storageClassName: ssd
resources:
requests:
storage: 1Gi

Related

Postgresql storage on K3s

I am trying to setup a Postgresql Stateful-Set on a k3s single node (Raspberry Pi 4 8Gb) cluster which will be common between any services that end up needing postgresql. Currently, since this is just a single node, I am using the Rancher local-path storage provisioner (this will probably change as I add nodes) pointing to an external hdd set to mount, though this will probably change when I get around to adding nodes.
My pod spins up (after figuring out a small headache with mounting the data directory which was as per this issue), and I can access the postgresql instance with kubectl port-forward -n common pod/postgresql-stateful-set-0 5432:5432, and making whatever changes I need.
At this point, my problem manifests, whereby I notice that no data is persisted in my data directory. I have found this issue which is the exact issue I've encountered, but following all the examples of "fixes", from playing with the directory locations (which gave me chmod issues like encountered with the mounting data directory issues), to changing to a pv I defined.
Find below my yml file which is use kubectl apply -f postgres.yml and is based on the bitnami helm template:
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: postgresql-storage
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: postgresql-pv
namespace: common
spec:
storageClassName: postgresql-storage
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 20Gi
accessModes:
- ReadWriteMany
hostPath:
path: "/mnt/storage/k3s/common/postgresql"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
namespace: common
name: pvc-postgresql
spec:
storageClassName: postgresql-storage
accessModes:
- ReadWriteMany
resources:
requests:
storage: 20Gi
---
apiVersion: v1
kind: Secret
metadata:
name: postgresql-password
namespace: common
data:
POSTGRES_PASSWORD: <PWD>
---
apiVersion: v1
kind: ConfigMap
metadata:
name: postgresql-configmap
namespace: common
data:
POSTGRESQL_PORT_NUMBER: "5432"
PGDATA: /var/lib/postgresql/data/pgdata
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgresql-stateful-set
namespace: common
labels:
name: postgres-stateful-set
spec:
replicas: 1
serviceName: postgresql-stateful-set
updateStrategy:
rollingUpdate: {}
type: RollingUpdate
selector:
matchLabels:
app: postgresql
template:
metadata:
labels:
app: postgresql
spec:
containers:
- name: postgresql
image: docker.io/postgres:14.2-alpine
envFrom:
- configMapRef:
name: postgresql-configmap
- secretRef:
name: postgresql-password
ports:
- name: tcp-postgresql
containerPort: 5432
livenessProbe:
failureThreshold: 6
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
exec:
command:
- /bin/sh
- -c
- exec pg_isready -U "rootAdmin" -h 127.0.0.1 -p 5432 -d rootDefault
readinessProbe:
failureThreshold: 6
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
exec:
command:
- /bin/sh
- -c
- -e
- exec pg_isready -U "rootAdmin" -h 127.0.0.1 -p 5432 -d rootDefault
resources:
limits:
memory: "300Mi"
cpu: "300m"
requests:
cpu: 250m
memory: 256Mi
volumeMounts:
- name: dshm
mountPath: /dev/shm
- name: postgresql-data
mountPath: /var/lib/postgresql
volumes:
- name: dshm
emptyDir:
medium: Memory
volumeClaimTemplates:
- metadata:
name: postgresql-data
spec:
storageClassName: postgresql-storage
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "20Gi"
---
kind: Service
apiVersion: v1
metadata:
name: postgresql-svc
namespace: common
spec:
selector:
app: postgresql
type: ClusterIP
clusterIP: None
ports:
- name: tcp-postgresql
port: 5432
protocol: TCP
Couple of things I should probably mention:
I added my own Storage class for the purposes of retaining the record rather then delete on the record being removed.
using the alpine version just for the sake of smaller image size.
The base OS that k3s is hosted on is Ubuntu 20.04 LTS
K3s was setup using this ansible playbook here
I have exec-ed into the container to see what is in the directory defined by PGDATA and confirmed that data was written.

Persistent Storage - Pi K8s Cluster - NFS version transport protocol not supported

I have a Raspberry Pi Cluster consisting of 1-Master 20-Nodes:
192.168.0.92 (Master)
192.168.0.112 (Node w/ USB Drive)
I mounted a USB drive to /media/hdd & set a label - purpose=volume to it.
Using the following I was able to setup a NFS server:
apiVersion: v1
kind: Namespace
metadata:
name: storage
labels:
app: storage
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-pv
namespace: storage
spec:
capacity:
storage: 3.5Ti
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /media/hdd
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: purpose
operator: In
values:
- volume
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: local-claim
namespace: storage
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-storage
resources:
requests:
storage: 3Ti
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-server
namespace: storage
labels:
app: nfs-server
spec:
replicas: 1
selector:
matchLabels:
app: nfs-server
template:
metadata:
labels:
app: nfs-server
name: nfs-server
spec:
containers:
- name: nfs-server
image: itsthenetwork/nfs-server-alpine:11-arm
env:
- name: SHARED_DIRECTORY
value: /exports
ports:
- name: nfs
containerPort: 2049
- name: mountd
containerPort: 20048
- name: rpcbind
containerPort: 111
securityContext:
privileged: true
volumeMounts:
- mountPath: /exports
name: mypvc
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: local-claim
nodeSelector:
purpose: volume
---
kind: Service
apiVersion: v1
metadata:
name: nfs-server
namespace: storage
spec:
ports:
- name: nfs
port: 2049
- name: mountd
port: 20048
- name: rpcbind
port: 111
clusterIP: 10.96.0.11
selector:
app: nfs-server
And I was even able to make a persistent volume with this:
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-nfs-volume
labels:
directory: mysql
spec:
capacity:
storage: 200Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: slow
nfs:
path: /mysql
server: 10.244.19.5
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-nfs-claim
spec:
storageClassName: slow
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
selector:
matchLabels:
directory: mysql
But when I try to use the volume like so:
apiVersion: v1
kind: Service
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
ports:
- port: 3306
selector:
app: wordpress
tier: mysql
clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
selector:
matchLabels:
app: wordpress
tier: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-nfs-claim
I get NFS version transport protocol not supported error.
When seeing mount.nfs: requested NFS version or transport protocol is not supported error there are three main reasons:
NFS services are not running on NFS server
NFS utils not installed on the client
NFS service hung on NFS server
According tho this artice there are three solutions to resolve the problem with your error.
First one:
Login to the NFS server and check the NFS services status. If the following command
service nfs status returns an information that NFS services are stopped on the server - just start them using service nfs start. To mount NFS share on the client use the same command.
Second one:
If after trying first solution your problem isn't resolved
try installing package nfs-utils on your server.
Third one:
Open file /etc/sysconfig/nfs and try to check below parameters
# Turn off v4 protocol support
#RPCNFSDARGS="-N 4"
# Turn off v2 and v3 protocol support
#RPCNFSDARGS="-N 2 -N 3"
Removing hash from RPCNFSDARGS lines will turn off specific version support. This way clients with mentioned NFS versions won’t be able to connect to the NFS server for mounting share. If you have any of it enabled, try disabling it and mounting at the client after the NFS server service restarts.

K8s PersistentVolume shared among multiple PersistentVolumeClaims for local testing

Could someone help me please and point me what configuration should I be doing for my use-case?
I'm building a development k8s cluster and one of the steps is to generate security files (private keys) that are generated in a number of pods during deployment (let's say for a simple setup I have 6 pods that each build their own security keys). I need to have access to all these files, also they must be persistent after the pod goes down.
I'm trying to figure out now how to set up it locally for internal testing. From what I understand Local PersistentVolumes only allow 1:1 with PersistentVolumeClaims, so I would have to create a separate PersistentVolume and PersistentVolumeClaim for each pod that get's configured. I would prefer to void this and use one PersistentVolume for all.
Could someone be so nice and help me or point me to the right setup that should be used?
-- Update: 26/11/2020
So this is my setup:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hlf-nfs--server
spec:
replicas: 1
selector:
matchLabels:
app: hlf-nfs--server
template:
metadata:
labels:
app: hlf-nfs--server
spec:
containers:
- name: hlf-nfs--server
image: itsthenetwork/nfs-server-alpine:12
ports:
- containerPort: 2049
name: tcp
- containerPort: 111
name: udp
securityContext:
privileged: true
env:
- name: SHARED_DIRECTORY
value: "/opt/k8s-pods/data"
volumeMounts:
- name: pvc
mountPath: /opt/k8s-pods/data
volumes:
- name: pvc
persistentVolumeClaim:
claimName: shared-nfs-pvc
apiVersion: v1
kind: Service
metadata:
name: hlf-nfs--server
labels:
name: hlf-nfs--server
spec:
type: ClusterIP
selector:
app: hlf-nfs--server
ports:
- name: tcp-2049
port: 2049
protocol: TCP
- name: udp-111
port: 111
protocol: UDP
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shared-nfs-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: nfs
resources:
requests:
storage: 1Gi
These three are being created at once, after that, I'm reading the IP of the service and adding it to the last one:
apiVersion: v1
kind: PersistentVolume
metadata:
name: shared-nfs-pv
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
nfs:
path: /opt/k8s-pods/data
server: <<-- IP from `kubectl get svc -l name=hlf-nfs--server`
The problem I'm getting and trying to resolve is that the PVC does not get bound with the PV and the deployment keeps in READY mode.
Did I miss anything?
You can create a NFS and have the pods use NFS volume. Here is the manifest file to create such in-cluster NFS server (make sure you modify STORAGE_CLASS and the other variables below):
export NFS_NAME="nfs-share"
export NFS_SIZE="10Gi"
export NFS_IMAGE="itsthenetwork/nfs-server-alpine:12"
export STORAGE_CLASS="thin-disk"
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: ${NFS_NAME}
labels:
app.kubernetes.io/name: nfs-server
app.kubernetes.io/instance: ${NFS_NAME}
spec:
ports:
- name: tcp-2049
port: 2049
protocol: TCP
- name: udp-111
port: 111
protocol: UDP
selector:
app.kubernetes.io/name: nfs-server
app.kubernetes.io/instance: ${NFS_NAME}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app.kubernetes.io/name: nfs-server
app.kubernetes.io/instance: ${NFS_NAME}
name: ${NFS_NAME}
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: $STORAGE_CLASS
volumeMode: Filesystem
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ${NFS_NAME}
labels:
app.kubernetes.io/name: nfs-server
app.kubernetes.io/instance: ${NFS_NAME}
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: nfs-server
app.kubernetes.io/instance: ${NFS_NAME}
template:
metadata:
labels:
app.kubernetes.io/name: nfs-server
app.kubernetes.io/instance: ${NFS_NAME}
spec:
containers:
- name: nfs-server
image: ${NFS_IMAGE}
ports:
- containerPort: 2049
name: tcp
- containerPort: 111
name: udp
securityContext:
privileged: true
env:
- name: SHARED_DIRECTORY
value: /nfsshare
volumeMounts:
- name: pvc
mountPath: /nfsshare
volumes:
- name: pvc
persistentVolumeClaim:
claimName: ${NFS_NAME}
EOF
Below is an example how to point the other pods to this NFS. In particular, refer to the volumes section at the end of the YAML:
export NFS_NAME="nfs-share"
export NFS_IP=$(kubectl get --template={{.spec.clusterIP}} service/$NFS_NAME)
kubectl apply -f - <<EOF
kind: Deployment
apiVersion: apps/v1
metadata:
name: apache
labels:
app: apache
spec:
replicas: 2
selector:
matchLabels:
app: apache
template:
metadata:
labels:
app: apache
containers:
- name: apache
image: apache
ports:
- containerPort: 80
volumeMounts:
- mountPath: /var/www/html/
name: nfs-vol
subPath: html
volumes:
- name: nfs-vol
nfs:
server: $NFS_IP
path: /
EOF
It is correct that there is a 1:1 relation between a PersistentVolumeClaim and a PersistentVolume.
However, Pods running on the same Node can concurrently mount the same volume, e.g. use the same PersistentVolumeClaim.
If you use Minikube for local development, you only have one node, so you can use the same PersistentVolumeClaim. Since you want to use different files for each app, you could use a unique directory for each app in that shared volume.
So finally, I did it by using a dynamic provider.
I installed the stable/nfs-server-provisioner with helm. With proper configuration, it managed to create a pv named nfs two which my pvc's are able to bound :)
helm install stable/nfs-server-provisioner --name nfs-provisioner -f nfs_provisioner.yaml
the nfs_provisioner.yaml is as follows
persistence:
enabled: true
storageClass: "standard"
size: 20Gi
storageClass:
# Name of the storage class that will be managed by the provisioner
name: nfs
defaultClass: true

How to solve AzureFile mount error(22): Invalid argument inside container?

I've got an error on mounting AzureFile Share inside AKS(1.18.2) container (build on top of Ubuntu 18.04 with cifs-utils installed):
Warning FailedMount 0s kubelet, aks-nodepool1-37397744-vmss000001 MountVolume.SetUp failed for volume "myapplication-logs" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/5e19e1d0-0bfd-4760-a87a-00cb0f3e573a/volumes/kubernetes.io~azure-file/crawler-logs --scope -- mount -t cifs -o file_mode=0777,dir_mode=0777,vers=3.0,<masked> //myazurestorage.file.core.windows.net/crawler-logs /var/lib/kubelet/pods/5e19e1d0-0bfd-4760-a87a-00cb0f3e573a/volumes/kubernetes.io~azure-file/myapplication-logs
Output: Running scope as unit run-r403b463e326d4562a7e44dc8fe018b4b.scope.
mount error(22): Invalid argument
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Here is my yaml config:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: myapplication-logs
provisioner: kubernetes.io/azure-file
reclaimPolicy: Retain
allowVolumeExpansion: true
parameters:
skuName: Standard_LRS
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: myapplication-logs
spec:
capacity:
storage: 3Gi
accessModes:
- ReadWriteMany
storageClassName: myapplication-logs
azureFile:
secretName: azurefilesharesecretname}
shareName: myapplication-logs
readOnly: false
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myapplication-logs
spec:
accessModes:
- ReadWriteMany
storageClassName: myapplication-logs
resources:
requests:
storage: 3Gi
---
apiVersion: apps/v1
spec:
selector:
matchLabels:
app: myapplication
replicas: 1
template:
spec:
containers:
name: myapplication
readinessProbe:
httpGet:
path: /probes/ready
port: 5000
timeoutSeconds: 60
periodSeconds: 10
ports:
- containerPort: 21602
- containerPort: 5000
livenessProbe:
httpGet:
path: /probes/healthy
port: 5000
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 30
image: myappacr.azurecr.io/myapplication:1.0.391
volumeMounts:
- readOnly: true
name: secrets-volume
mountPath: /usr/bin/myapp/Secrets
- name: configuration-volume
mountPath: /usr/bin/myapp/Configuration
- name: myapplication-logs
mountPath: /usr/bin/myapp/logs
imagePullSecrets:
- name: acr-dev-regcred
volumes:
- name: secrets-volume
secret:
secretName: myapplication-secrets
- configMap:
name: myapplication-configuration
name: configuration-volume
- name: myapplication-logs
persistentVolumeClaim:
claimName: myapplication-logs
metadata:
labels:
app: myapplication
kind: Deployment
metadata:
name: myapplication-deployment
labels:
app: myapplication
StorageClass, PersistentVolume and PersistentVolumeClaim is successfully deployd without any errors\events.
Cannot find out where is the problem?
Any ideas on what is happening?
There are two ways to consume azure file share as volume from a container in AKS
Manually create and use a volume with Azure Files share. Docs here
In this case the PV need to specify mountOptions
apiVersion: v1
kind: PersistentVolume
metadata:
name: azurefile
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
storageClassName: azurefile
azureFile:
secretName: azure-secret
shareName: aksshare
readOnly: false
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=1000
- gid=1000
- mfsymlinks
- nobrl
Dynamically create and use a persistent volume with Azure Files. Docs here
In this case StorageClass need to have mountOptions
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: my-azurefile
provisioner: kubernetes.io/azure-file
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=0
- gid=0
- mfsymlinks
- cache=strict
parameters:
skuName: Standard_LRS
Now looking at your yamls It seems you are mixing the manual and dynamic mode because you are creating both PersistentVolume and a StorageClass. I suggest to follow one of the suitable approach and have mountOptions specified properly which is mandatory for both the modes.
Problem solved!
There was a problem with secret I created.
It was created with kubectl apply -f secret.json and accountstoragekey wasn't encoded in base64.
Thanx to Azure Support!

kubernetes StorageClass does not retain existing data

My Kubernetes StorageClass volume doesn't retain existing data when the pod is deleted and deployed back with my postgresql database. When I delete the pod, the new pod is created but the database is empty.
I have followed variations of the different versions of the tutorials (https://kubernetes.io/docs/concepts/storage/persistent-volumes/) but nothing seems to work.
I paste all the YAML files cause the problem might be in the combination.
storage-google.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: spingular-pvc
spec:
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 7Gi
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: standard
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
zone: us-east4-a
jhipsterpress-postgresql.yml
apiVersion: v1
kind: Secret
metadata:
name: jhipsterpress-postgresql
namespace: default
labels:
app: jhipsterpress-postgresql
type: Opaque
data:
postgres-password: NjY0NXJxd24=
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jhipsterpress-postgresql
namespace: default
spec:
replicas: 1
template:
metadata:
labels:
app: jhipsterpress-postgresql
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: spingular-pvc
containers:
- name: postgres
image: postgres:10.4
env:
- name: POSTGRES_USER
value: jhipsterpress
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: jhipsterpress-postgresql
key: postgres-password
ports:
- containerPort: 5432
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/
---
apiVersion: v1
kind: Service
metadata:
name: jhipsterpress-postgresql
namespace: default
spec:
selector:
app: jhipsterpress-postgresql
ports:
- name: postgresqlport
port: 5432
type: LoadBalancer
jhipsterpress-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: jhipsterpress
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: jhipsterpress
version: "v1"
template:
metadata:
labels:
app: jhipsterpress
version: "v1"
spec:
initContainers:
- name: init-ds
image: busybox:latest
command:
- '/bin/sh'
- '-c'
- |
while true
do
rt=$(nc -z -w 1 jhipsterpress-postgresql 5432)
if [ $? -eq 0 ]; then
echo "DB is UP"
break
fi
echo "DB is not yet reachable;sleep for 10s before retry"
sleep 10
done
containers:
- name: jhipsterpress-app
image: galore/jhipsterpress
env:
- name: SPRING_PROFILES_ACTIVE
value: prod
- name: SPRING_DATASOURCE_URL
value: jdbc:postgresql://jhipsterpress-postgresql.default.svc.cluster.local:5432/jhipsterpress
- name: SPRING_DATASOURCE_USERNAME
value: jhipsterpress
- name: SPRING_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
name: jhipsterpress-postgresql
key: postgres-password
- name: JAVA_OPTS
value: " -Xmx256m -Xms256m"
resources:
requests:
memory: "256Mi"
cpu: "500m"
limits:
memory: "512Mi"
cpu: "1"
ports:
- name: http
containerPort: 8080
readinessProbe:
httpGet:
path: /management/health
port: http
initialDelaySeconds: 20
periodSeconds: 15
failureThreshold: 6
livenessProbe:
httpGet:
path: /management/health
port: http
initialDelaySeconds: 120
jhipsterpress-service.yml
apiVersion: v1
kind: Service
metadata:
name: jhipsterpress
namespace: default
labels:
app: jhipsterpress
spec:
selector:
app: jhipsterpress
type: LoadBalancer
ports:
- name: http
port: 8080
When I included a Retain Policy I was getting this error:
#cloudshell:~ (academic-veld-230622)$ kubectl apply -f storage-google.yaml
error: error validating "storage-google.yaml": error validating data:
ValidationError(PersistentVolumeClaim.spec): unknown field "persistentVolumeReclaimPolicy" in io.k8s.api.core.v1.PersistentVolumeClaimSpec; if you choose to ignore these errors, turn validation off with --validate=false
Please, if you know of a complete example on a public image that works (in postgresql, I can make it work with Mongo), I will really appreciate it.
Thanks to all.
Note that for this to work you need to have your PVC dynamically provision a PV to satisfy its requirements, then there will be a permanent binding between the PVC and PV and every time your workload uses the PVC then it will use the same PV. Specifically indicated by this excerpt:
If a PV was dynamically provisioned for a new PVC, the loop will always bind that PV to the PVC
If in your case the Google Persistent Disk is being provisioned by the PVC, and you can verify that on GCP it's the same PV used every time, then it's probably an issue with the pod startup process where it's removing all the data. (Is there any reason why you are using /var/lib/postgresql/ vs /var/lib/postgresql?)
Also, persistentVolumeReclaimPolicy: Retain applies to a PV, not a PVC. For dynamically provisioned PVs the value is Delete. In your case, it wouldn't apply because your dynamically provisioned volume should be bound to your PVC. In other words, you are not reclaiming the volume.
Having said all that the recommended way to deploy a DB is using StatefulSets similar to this mysql example using a volumeClaimTemplate.