How to identify the storage space left in a PVC? - kubernetes

Question

I have a pod with PVC request of 10Gi and is successfully bound to PV(both definition below)
I came across the accepted answer of similar question, which suggests to run kubectl -n <namespace> exec <pod-name> df
Upon doing the same i got the following -
pavan#p1: kubectl exec mysql-deployment-95f7dd544-mmjv9 df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 51572172 5797112 43640736 12% /
tmpfs 65536 0 65536 0% /dev
tmpfs 1021680 0 1021680 0% /sys/fs/cgroup
/dev/vda1 51572172 5797112 43640736 12% /etc/hosts
shm 65536 0 65536 0% /dev/shm
tmpfs 1021680 12 1021668 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 1021680 0 1021680 0% /proc/acpi
tmpfs 1021680 0 1021680 0% /sys/firmware
pavan#p1: kubectl exec mysql-deployment-95f7dd544-mmjv9 -- df -h
Filesystem Size Used Avail Use% Mounted on
overlay 50G 5.6G 42G 12% /
tmpfs 64M 0 64M 0% /dev
tmpfs 998M 0 998M 0% /sys/fs/cgroup
/dev/vda1 50G 5.6G 42G 12% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 998M 12K 998M 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 998M 0 998M 0% /proc/acpi
tmpfs 998M 0 998M 0% /sys/firmware
I coudn't quite understand the o/p, i reqested for 10Gi and i don't see any mount with 10gi total capacity?
PV definition :
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-pv-volume
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/mysql"
PVC definition:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
Deployment definition:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql-deployment
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: mysql-pod
template:
metadata:
labels:
app: mysql-pod
spec:
containers:
- name: mysql-container
image: mysql:5.7
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
P.s: Node has a capacity of 50G
EDIT 1:
PV describe:
pavan#p1: kubectl describe pv/mysql-pv-volume
Name: mysql-pv-volume
Labels: type=local
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"type":"local"},"name":"mysql-pv-volume"},"spec":{"acc...
pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pv-protection]
StorageClass: manual
Status: Bound
Claim: default/mysql-pv-claim
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity: <none>
Message:
Source:
Type: HostPath (bare host directory volume)
Path: /mnt/mysql
HostPathType:
Events: <none>
PVC describe:
pavan#p1: kubectl describe pvc/mysql-pv-claim
Name: mysql-pv-claim
Namespace: default
StorageClass: manual
Status: Bound
Volume: mysql-pv-volume
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"mysql-pv-claim","namespace":"default"},"spec":{"acc...
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: mysql-deployment-95f7dd544-mmjv9
Events: <none>
Pod describe:
pavan#p1: kubectl describe pod/mysql-deployment-95f7dd544-mmjv9
Name: mysql-deployment-95f7dd544-mmjv9
Namespace: default
Priority: 0
Node: pay0k-k8-dev-bytq/10.130.219.196
Start Time: Mon, 09 Sep 2019 18:14:17 +0800
Labels: app=mysql-pod
pod-template-hash=95f7dd544
Annotations: <none>
Status: Running
IP: 10.244.0.123
Controlled By: ReplicaSet/mysql-deployment-95f7dd544
Containers:
mysql-container:
Container ID: docker://83f4730892fd6908ef3dfae3b9125d25cb7467d24df89323c43d3ab136376147
Image: mysql:5.7
Image ID: docker-pullable://mysql#sha256:1a121f2e7590f949b9ede7809395f209dd9910e331e8372e6682ba4bebcc020b
Port: 3306/TCP
Host Port: 0/TCP
State: Running
Started: Mon, 09 Sep 2019 18:14:19 +0800
Ready: True
Restart Count: 0
Environment:
MYSQL_DATABASE: <set to the key 'mysql-database' of config map 'mysqldb'> Optional: false
MYSQL_ROOT_PASSWORD: <set to the key 'mysql-root-password' in secret 'db-credentials'> Optional: false
MYSQL_USER: <set to the key 'mysql-user' in secret 'db-credentials'> Optional: false
MYSQL_PASSWORD: <set to the key 'mysql-password' in secret 'db-credentials'> Optional: false
Mounts:
/var/lib/mysql from mysql-persistent-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-gbpxc (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
mysql-persistent-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: mysql-pv-claim
ReadOnly: false
default-token-gbpxc:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-gbpxc
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 41m default-scheduler Successfully assigned default/mysql-deployment-95f7dd544-mmjv9 to pay0k-k8-dev-bytq
Normal Pulled 41m kubelet, pay0k-k8-dev-bytq Container image "mysql:5.7" already present on machine
Normal Created 41m kubelet, pay0k-k8-dev-bytq Created container mysql-container
Normal Started 41m kubelet, pay0k-k8-dev-bytq Started container mysql-container

Answer 1

In your PV definition your specify hostPath, so your data is stored directly on worker - this is why you see 50GB and you are skipping this additional layer provided by cloud provider which create pv directly from pvc.
Keeping data directly on a node isn't good approach because node could be removed/replaced at any time.
You should use digital ocean pvs instead of you're manually defined, if you spin new pod on digital ocean's pv output from df should show 10GB.