Deployment cannot find PVC on minikube - kubernetes

Question

I am practicing making PV and PVC with Minikube. But I encountered an error that my InfluxDB deployment couldn't find influxdb-pvc and I can't solve it.
I check the message at the top of the event, I can see that my PVC cannot be found. Therefore, I checked the status of PersistentVolumeClaim.
As far as I know, if the STATUS of influxdb-pv and influxdb-pvc is Bound, it is normally created and Deployment should be able to find influxdb-pvc. I don't know what's going on... Please help me 😢
The following is a description of Pod:
> kubectl describe pod influxdb-5b769454b8-pksss
Name: influxdb-5b769454b8-pksss
Namespace: ft-services
Priority: 0
Node: minikube/192.168.49.2
Start Time: Thu, 25 Feb 2021 01:14:25 +0900
Labels: app=influxdb
pod-template-hash=5b769454b8
Annotations: <none>
Status: Running
IP: 172.17.0.5
IPs:
IP: 172.17.0.5
Controlled By: ReplicaSet/influxdb-5b769454b8
Containers:
influxdb:
Container ID: docker://be2eec32cca22ea84f4a0034f42668c971fefe62e361f2a4d1a74d92bfbf4d78
Image: service_influxdb
Image ID: docker://sha256:50693dcc4dda172f82c0dcd5ff1db01d6d90268ad2b0bd424e616cb84da64c6b
Port: 8086/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 25 Feb 2021 01:30:40 +0900
Finished: Thu, 25 Feb 2021 01:30:40 +0900
Ready: False
Restart Count: 8
Environment Variables from:
influxdb-secret Secret Optional: false
Environment: <none>
Mounts:
/var/lib/influxdb from var-lib-influxdb (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-lfzz9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
var-lib-influxdb:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: influxdb-pvc
ReadOnly: false
default-token-lfzz9:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-lfzz9
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 20m (x2 over 20m) default-scheduler 0/1 nodes are available: 1 persistentvolumeclaim "influxdb-pvc" not found.
Normal Scheduled 20m default-scheduler Successfully assigned ft-services/influxdb-5b769454b8-pksss to minikube
Normal Pulled 19m (x5 over 20m) kubelet Container image "service_influxdb" already present on machine
Normal Created 19m (x5 over 20m) kubelet Created container influxdb
Normal Started 19m (x5 over 20m) kubelet Started container influxdb
Warning BackOff 43s (x93 over 20m) kubelet Back-off restarting failed container
The following is status information for PV and PVC:
> kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/influxdb-pv 10Gi RWO Recycle Bound ft-services/influxdb-pvc influxdb 104m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/influxdb-pvc Bound influxdb-pv 10Gi RWO influxdb 13m
I proceeded with the setting in the following order.
Create a namespace.
kubectl create namespace ft-services
kubectl config set-context --current --namespace=ft-services
Apply my config files: influxdb-deployment.yaml, influxdb-secret.yaml, influxdb-service.yaml, influxdb-volume.yaml
influxdb-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: influxdb
labels:
app: influxdb
spec:
replicas: 1
selector:
matchLabels:
app: influxdb
template:
metadata:
labels:
app: influxdb
spec:
containers:
- name: influxdb
image: service_influxdb
imagePullPolicy: Never
ports:
- containerPort: 8086
envFrom:
- secretRef:
name: influxdb-secret
volumeMounts:
- mountPath: /var/lib/influxdb
name: var-lib-influxdb
volumes:
- name: var-lib-influxdb
persistentVolumeClaim:
claimName: influxdb-pvc
influxdb-volume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: influxdb-pv
labels:
app: influxdb
spec:
storageClassName: influxdb
claimRef:
namespace: ft-services
name: influxdb-pvc
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
hostPath:
path: "/mnt/influxdb"
type: DirectoryOrCreate
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: influxdb-pvc
labels:
app: influxdb
spec:
storageClassName: influxdb
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
Build my docker image: service_influxdb
Dockerfile:
FROM alpine:3.13.1
RUN apk update && apk upgrade --ignore busybox && \
apk add \
influxdb && \
sed -i "247s/ #/ /" /etc/influxdb.conf && \
sed -i "256s/ #/ /" /etc/influxdb.conf
EXPOSE 8086
ENTRYPOINT influxd & /bin/sh
Check my minikube with dashboard
> minikube dashboard
0/1 nodes are available: 1 persistentvolumeclaim "influxdb-pvc" not found.
Back-off restarting failed container

Answer 1

I've tested your YAMLs on my Minikube cluster.
Your configuration is correct, however you missed one small detail. Container based on alpine needs to "do something" inside, otherwise container exits when its main process exits. Once container did all what was expected/configured, pod will be in Completed status.
Your pod is crashing because it starts up then immediately exits, thus Kubernetes restarts and the cycle continues. For more details please check Pod Lifecycle Documentation.
Examples
Alpine example:
$ kubectl get po alipne-test -w
NAME READY STATUS RESTARTS AGE
alipne-test 0/1 Completed 2 36s
alipne-test 0/1 CrashLoopBackOff 2 36s
alipne-test 0/1 Completed 3 54s
alipne-test 0/1 CrashLoopBackOff 3 55s
alipne-test 0/1 Completed 4 101s
alipne-test 0/1 CrashLoopBackOff 4 113s
Nginx example:
$ kubectl get po nginx
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 5m23s
Nginx is a webserver based container so it does not need additional sleep command.
Your Current Configuration
Your pod with influx is created, has nothing to do and exits.
$ kubectl get po -w
NAME READY STATUS RESTARTS AGE
influxdb-96bfd697d-wbkt7 0/1 CrashLoopBackOff 4 2m28s
influxdb-96bfd697d-wbkt7 0/1 Completed 5 3m8s
influxdb-96bfd697d-wbkt7 0/1 CrashLoopBackOff 5 3m19s
Solution
You just need add for example sleep command to keep container alive. For test I've used sleep 60 to keep container alive for 60 seconds using below configuration:
spec:
containers:
- name: influxdb
image: service_influxdb
imagePullPolicy: Never
ports:
- containerPort: 8086
envFrom:
- secretRef:
name: influxdb-secret
volumeMounts:
- mountPath: /var/lib/influxdb
name: var-lib-influxdb
command: ["/bin/sh"] # additional command
args: ["-c", "sleep 60"] # args to use sleep 60 command
And output below:
$ kubectl get po -w
NAME READY STATUS RESTARTS AGE
influxdb-65dc56f8df-9v76p 1/1 Running 0 7s
influxdb-65dc56f8df-9v76p 0/1 Completed 0 62s
influxdb-65dc56f8df-9v76p 1/1 Running 1 63s
It was running for 60 seconds, as sleep command was set to 60. As container fulfill all configured commands inside, it exit and status changed to Completed. If you will use commands to keep this container alive, you don't need to use sleep.
PV issues
As last part you mention about issue in Minikube Dashboard. I was not able to replicate it, but it might be some leftovers from your previous test.
Please let me know if you still have issue.