Kubernetes Minikube with local persistent storage - kubernetes

I am currently trying to deploy the following on Minikube. I used the configuration files to use a hostpath as a persistent storage on minikube node.
apiVersion: v1
kind: PersistentVolume
metadata:
name: "pv-volume"
spec:
capacity:
storage: "20Gi"
accessModes:
- "ReadWriteOnce"
hostPath:
path: /data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: "orientdb-pv-claim"
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "20Gi"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: orientdbservice
spec:
#replicas: 1
template:
metadata:
name: orientdbservice
labels:
run: orientdbservice
test: orientdbservice
spec:
containers:
- name: orientdbservice
image: orientdb:latest
env:
- name: ORIENTDB_ROOT_PASSWORD
value: "rootpwd"
ports:
- containerPort: 2480
name: orientdb
volumeMounts:
- name: orientdb-config
mountPath: /data/orientdb/config
- name: orientdb-databases
mountPath: /data/orientdb/databases
- name: orientdb-backup
mountPath: /data/orientdb/backup
volumes:
- name: orientdb-config
persistentVolumeClaim:
claimName: orientdb-pv-claim
- name: orientdb-databases
persistentVolumeClaim:
claimName: orientdb-pv-claim
- name: orientdb-backup
persistentVolumeClaim:
claimName: orientdb-pv-claim
---
apiVersion: v1
kind: Service
metadata:
name: orientdbservice
labels:
run: orientdbservice
spec:
type: NodePort
selector:
run: orientdbservice
ports:
- protocol: TCP
port: 2480
name: http
which results in following
#kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
pv-volume 20Gi RWO Retain Available 4h
pvc-cd14d593-78fc-11e7-a46d-1277ec3dd2b5 20Gi RWO Delete Bound default/orientdb-pv-claim standard 4h
#kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
orientdb-pv-claim Bound pvc-cd14d593-78fc-11e7-a46d-1277ec3dd2b5 20Gi RWO
#kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
orientdbservice 10.0.0.16 <nodes> 2480:30552/TCP 4h
#kubectl get pods
NAME READY STATUS RESTARTS AGE
orientdbservice-458328598-zsmw5 0/1 ContainerCreating 0 4h
#kubectl describe pod orientdbservice-458328598-zsmw5
Events:
FirstSeen LastSeen Count From SubObjectPath TypeReason Message
--------- -------- ----- ---- ------------- -------- ------ -------
4h 1m 37 kubelet, minikube Warning FailedMount Unable to mount volumes for pod "orientdbservice-458328598-zsmw5_default(392b1298-78ff-11e7-a46d-1277ec3dd2b5)": timeout expired waiting for volumes to attach/mount for pod "default"/"orientdbservice-458328598-zsmw5". list of unattached/unmounted volumes=[orientdb-databases]
4h 1m 37 kubelet, minikube Warning FailedSync Error syncing pod
I see the following error
Unable to mount volumes for pod,timeout expired waiting for volumes to attach/mount for pod
Is there something incorrect in way I am creating Persistent Volume and PersistentVolumeClaim on my node.
minikube version: v0.20.0
Appreciate all the help

Your configuration is fine.
Tested under minikube v0.24.0, minikube v0.25.0 and minikube v0.26.1 without any problem.
Take in mind that minikube is under active development, and, specially if you're under windows, is like they say experimental software.
Update to a newer version of minikube and redeploy it. This should solve the problem.
You can check for updates with the minikube update-check command which results in something like this:
$ minikube update-check
CurrentVersion: v0.25.0
LatestVersion: v0.26.1
To upgrade minikube simply type minikube delete which deletes your current minikube installation and download the new release as described.
$ minikube delete
There is a newer version of minikube available (v0.26.1). Download it here:
https://github.com/kubernetes/minikube/releases/tag/v0.26.1
To disable this notification, run the following:
minikube config set WantUpdateNotification false
Deleting local Kubernetes cluster...
Machine deleted.

For somereason the provisioner provisioner: k8s.io/minikube-hostpath in minikube doesn't work.
So:
delete default storage class kubectl delete storageclass standard
create following storage class:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: standard
provisioner: docker.io/hostpath
reclaimPolicy: Retain
Also in your volume mounts, you have one PVC bound to one PV, so instead of multiple volumes just have one volume and mount them with different subpaths, that will create three subdirectories(backup, config & databases) on your host's /data directory:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: orientdbservice
spec:
#replicas: 1
template:
metadata:
name: orientdbservice
labels:
run: orientdbservice
test: orientdbservice
spec:
containers:
- name: orientdbservice
image: orientdb:latest
env:
- name: ORIENTDB_ROOT_PASSWORD
value: "rootpwd"
ports:
- containerPort: 2480
name: orientdb
volumeMounts:
- name: orientdb
mountPath: /data/orientdb/config
subPath: config
- name: orientdb
mountPath: /data/orientdb/databases
subPath: databases
- name: orientdb
mountPath: /data/orientdb/backup
subPath: backup
volumes:
- name: orientdb
persistentVolumeClaim:
claimName: orientdb-pv-claim
- Now deploy your yaml: kubectl create -f yourorientdb.yaml

Related

0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims

As the documentation states:
For each VolumeClaimTemplate entry defined in a StatefulSet, each Pod
receives one PersistentVolumeClaim. In the nginx example above, each
Pod receives a single PersistentVolume with a StorageClass of
my-storage-class and 1 Gib of provisioned storage. If no StorageClass
is specified, then the default StorageClass will be used. When a Pod
is (re)scheduled onto a node, its volumeMounts mount the
PersistentVolumes associated with its PersistentVolume Claims. Note
that, the PersistentVolumes associated with the Pods' PersistentVolume
Claims are not deleted when the Pods, or StatefulSet are deleted. This
must be done manually.
The part I'm interested in is this: If no StorageClassis specified, then the default StorageClass will be used
I create a StatefulSet like this:
apiVersion: apps/v1
kind: StatefulSet
metadata:
namespace: ches
name: ches
spec:
serviceName: ches
replicas: 1
selector:
matchLabels:
app: ches
template:
metadata:
labels:
app: ches
spec:
serviceAccountName: ches-serviceaccount
nodeSelector:
ches-worker: "true"
volumes:
- name: data
hostPath:
path: /data/test
containers:
- name: ches
image: [here I have the repo]
imagePullPolicy: Always
securityContext:
privileged: true
args:
- server
- --console-address
- :9011
- /data
env:
- name: MINIO_ACCESS_KEY
valueFrom:
secretKeyRef:
name: ches-keys
key: access-key
- name: MINIO_SECRET_KEY
valueFrom:
secretKeyRef:
name: ches-keys
key: secret-key
ports:
- containerPort: 9000
hostPort: 9011
resources:
limits:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: data
mountPath: /data
imagePullSecrets:
- name: edge-storage-token
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
Of course I have already created the secrets, imagePullSecrets etc and I have labeled the node as ches-worker.
When I apply the yaml file, the pod is in Pending status and kubectl describe pod ches-0 -n ches gives the following error:
Warning FailedScheduling 6s default-scheduler 0/1 nodes are
available: 1 pod has unbound immediate PersistentVolumeClaims.
preemption: 0/1 nodes are available: 1 Preemption is not helpful for
scheduling
Am I missing something here?
You need to create a PV in order to get a PVC bound. If you want the PVs automatically created from PVC claims you need a Provisioner installed in your Cluster.
First create a PV with at least the amout of space need by your PVC.
Then you can apply your deployment yaml which contains the PVC claim.
K3s when installed, also downloads a storage class which makes it as default.
Check with kubectl get storageclass:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE
ALLOWVOLUMEEXPANSION AGE local-path rancher.io/local-path Delete
WaitForFirstConsumer false 8s
K8s cluster on the other hand, does not download also a default storage class.
In order to solve the problem:
Download rancher.io/local-path storage class:
kubectl apply -f
https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
Check with kubectl get storageclass
Make this storage class (local-path) the default:
kubectl patch
storageclass local-path -p '{"metadata":
{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Error while running the MongoDB as a Statefulset set in Kubernetes

I am trying to run the mongodb as a statefulset in the minikube Kubernetes cluster. I have 3 replicas but I have the following problem - which is, one replica (mongo-0) is up and running without any issue but the second replica (mongo-1) is forever in the pending state. I tried to describe the pod and I get the following output:
kubectl describe pod mongo-1 -n ng-mongo
. . .
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 17m (x70 over 6h9m) default-scheduler 0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
As per the above error, it says it cannot find the persistent volume, but there is one already.
Please find my YAML definitions for this:
apiVersion: v1
kind: Service
metadata:
name: mongodb-service
labels:
name: mongo
spec:
ports:
- port: 27017
targetPort: 27017
clusterIP: None
selector:
role: mongo
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local-storage
namespace: ng-mongo
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-pv
namespace: ng-mongo
spec:
capacity:
storage: 10Gi
# volumeMode field requires BlockVolume Alpha feature gate to be enabled.
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /tmp
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- minikube
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: local-claim
namespace: ng-mongo
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
storageClassName: local-storage
---
apiVersion: v1
kind: Service
metadata:
name: mongo
namespace: ng-mongo
labels:
name: mongo
spec:
ports:
- port: 27017
targetPort: 27017
clusterIP: None
selector:
role: mongo
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
namespace: ng-mongo
spec:
serviceName: "mongo"
replicas: 3
selector:
matchLabels:
role: mongo
template:
metadata:
labels:
role: mongo
environment: test
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongo
image: mongo
command:
- mongod
- "--bind_ip"
- "0.0.0.0"
- "--replSet"
- rs0
resources:
requests:
cpu: 0.2
memory: 200Mi
ports:
- containerPort: 27017
volumeMounts:
- name: localvolume
mountPath: /data/db
- name: mongo-sidecar
image: cvallance/mongo-k8s-sidecar
env:
- name: MONGO_SIDECAR_POD_LABELS
value: "role=mongo,environment=test"
# volumes:
# - name: localvolume
# persistentVolumeClaim:
# claimName: local-claim
volumeClaimTemplates:
- metadata:
name: localvolume
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "local-storage"
resources:
requests:
storage: 2Gi
Can someone help me find the issue here?
You are using the node affinity while creating the PV, that need to be configured correctly:
It will inform Kubernetes my disk will attach to this type of node. Due to affinity your PV is attached to one type of specific node only.
when you are deploying the deployment it's not getting scheduled on that specific node and your POD is not getting that PV or PVC.
If you are adding node affinity to PVC add it to deployment also. So both PVC and pod get scheduled on the same node.
Resolution steps:
Make sure both deployment and pvc schedule with the same node add the node affinity to deployment also so deployment schedule on the respective node.
or else
Remove the node affinity rule from PV and create a new PV and PVC and use it.
here is the place where you have mentioned the node affinity rule
Note: In your node affinity you have mentioned as minikube, verify the node by
kubectl get nodes make changes if required.

"pod has unbound immediate PersistentVolumeClaims" : Issue with Dynamic Volume Provisioning

I have created a kubernetes cluster using 2 droplets (digital ocean machines)
1 machine is set to be master and another is set to be worker
Now, I am running a project which has 2 PVCs. (Their configs are same as below)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: pvc1
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
storageClassName: my-storageclass
status: {}
I set the storage class of this PVCs to ...
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: my-storageclass
labels:
doks.digitalocean.com/managed: "true"
provisioner: dobs.csi.digitalocean.com
allowVolumeExpansion: true
parameters:
type: pd-ssd
My goal is to dynamically create PVs using the Dobs (Digital Ocean Block Storage) CSI
Currently when I run my application on kubernetes (I do that using helm), my pod gives me following error :
0/2 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 1 pod has unbound immediate PersistentVolumeClaims
I understand that master node will be having taints and therefore of no use to run my pods. the second part of error is "1 pod has unbound immediate PersistentVolumeClaims"
how do I fix that ?
Thanks in advance !
Note: I have successfully ran my project with DOKS & EKS, I am doing this exercise to understand the concepts of volume binding in depth.
-------- Deployment ------
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 1
strategy:
type: Recreate
template:
spec:
containers:
- args:
- /bin/sh
- -c
- go run server.go
image: ***.dkr.ecr.us-east-2.amazonaws.com/my-app
imagePullPolicy: Always
name: my-app
ports:
- containerPort: 9000
resources: {}
volumeMounts:
- mountPath: /app/test1
name: pvc1
- mountPath: /app/test2
name: pvc2
imagePullSecrets:
- name: my-registery-key
restartPolicy: Always
volumes:
- name: pv1
persistentVolumeClaim:
claimName: pvc1
- name: pv2
persistentVolumeClaim:
claimName: pvc2

Error: 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims

I am deploying a CouchDB cluster on Kubernetes.
It worked and I got an error when I tried to scale it.
I try scale my Statefulset and I am getting this error when I desscribe couchdb-3:
0/3 nodes are available: 3 pod has unbound immediate
PersistentVolumeClaims.
And this error when I describe hpa:
invalid metrics (1 invalid out of 1), first error is: failed to get
cpu utilization: missing request for cpu
failed to get cpu utilization: missing request for cpu
I run "kubectl get pod -o wide" and receive this result:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
couchdb-0 1/1 Running 0 101m 10.244.2.13 node2 <none> <none>
couchdb-1 1/1 Running 0 101m 10.244.2.14 node2 <none> <none>
couchdb-2 1/1 Running 0 100m 10.244.2.15 node2 <none> <none>
couchdb-3 0/1 Pending 0 15m <none> <none> <none> <none>
How can I fix it !?
Kubernetes Version: 1.22.4
Docker Version 20.10.11, build dea9396
Ubuntu 20.04
My hpa file:
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: hpa-couchdb
spec:
maxReplicas: 16
minReplicas: 6
scaleTargetRef:
apiVersion: apps/v1
kind: StatefulSet
name: couchdb
targetCPUUtilizationPercentage: 50
pv.yaml:
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: couch-vol-0
labels:
volume: couch-volume
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
nfs:
server: 192.168.1.100
path: "/var/couchnfs/couchdb-0"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: couch-vol-1
labels:
volume: couch-volume
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
nfs:
server: 192.168.1.100
path: "/var/couchnfs/couchdb-1"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: couch-vol-2
labels:
volume: couch-volume
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
nfs:
server: 192.168.1.100
path: "/var/couchnfs/couchdb-2"
I set nfs in /etc/exports: /var/couchnfs 192.168.1.0/24(rw,sync,no_subtree_check,no_root_squash)
statefulset.yaml
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: couchdb
labels:
app: couch
spec:
replicas: 3
serviceName: "couch-service"
selector:
matchLabels:
app: couch
template:
metadata:
labels:
app: couch # pod label
spec:
containers:
- name: couchdb
image: couchdb:2.3.1
imagePullPolicy: "Always"
env:
- name: NODE_NETBIOS_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NODENAME
value: $(NODE_NETBIOS_NAME).couch-service # FQDN in vm.args
- name: COUCHDB_USER
value: admin
- name: COUCHDB_PASSWORD
value: admin
- name: COUCHDB_SECRET
value: b1709267
- name: ERL_FLAGS
value: "-name couchdb#$(NODENAME)"
- name: ERL_FLAGS
value: "-setcookie b1709267" # the “password” used when nodes connect to each other.
ports:
- name: couchdb
containerPort: 5984
- name: epmd
containerPort: 4369
- containerPort: 9100
volumeMounts:
- name: couch-pvc
mountPath: /opt/couchdb/data
volumeClaimTemplates:
- metadata:
name: couch-pvc
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
selector:
matchLabels:
volume: couch-volume
You have 3 persistent volumes and 3 pods claiming each. One PV can’t be claimed by more than one pod.
Since you are using NFS as backend, you can use dynamic provisioning of persistent volumes.
https://github.com/openebs/dynamic-nfs-provisioner

How to set pvc with statefulset in kubernetes?

On GKE, I set a statefulset resource as
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
spec:
serviceName: "redis"
selector:
matchLabels:
app: redis
updateStrategy:
type: RollingUpdate
replicas: 3
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis
resources:
limits:
memory: 2Gi
ports:
- containerPort: 6379
volumeMounts:
- name: redis-data
mountPath: /usr/share/redis
volumes:
- name: redis-data
persistentVolumeClaim:
claimName: redis-data-pvc
Want to use pvc so created this one. (This step was did before the statefulset deployment)
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
When check the resource in kubernetes
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
redis-data-pvc Bound pvc-6163d1f8-fb3d-44ac-a91f-edef1452b3b9 10Gi RWO standard 132m
The default Storage Class is standard.
kubectl get storageclass
NAME PROVISIONER
standard (default) kubernetes.io/gce-pd
But when check the statafulset's deployment status. It always wrong.
# Describe its pod details
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 22s default-scheduler persistentvolumeclaim "redis-data-pvc" not found
Warning FailedScheduling 17s (x2 over 20s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
Normal Created 2s (x2 over 3s) kubelet Created container redis
Normal Started 2s (x2 over 3s) kubelet Started container redis
Warning BackOff 0s (x2 over 1s) kubelet Back-off restarting failed container
Why can't it find the redis-data-pvc name?
What you have done, should work. Make sure that the PersistentVolumeClaim and the StatefulSet is located in the same namespace.
Thats said, this is an easier solution, and that let you easier scale up to more replicas:
When using StatefulSet and PersistentVolumeClaim, use the volumeClaimTemplates: field in the StatefulSet instead.
The volumeClaimTemplates: will be used to create unique PVCs for each replica, and they have unique naming ending with e.g. -0 where the number is an ordinal used for the replicas in a StatefulSet.
So instead, use a SatefuleSet manifest like this:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
spec:
serviceName: "redis"
selector:
matchLabels:
app: redis
updateStrategy:
type: RollingUpdate
replicas: 3
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis
resources:
limits:
memory: 2Gi
ports:
- containerPort: 6379
volumeMounts:
- name: redis-data
mountPath: /usr/share/redis
volumeClaimTemplates: // this will be used to create PVC
- metadata:
name: redis-data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi