Unable to mount persistent storage on Minikube - kubernetes

Question

I am currently trying to deploy the following on Minikube.
I updated the configuration files to use a hostpath as a persistent storage on minikube node.
apiVersion: v1
kind: PersistentVolume
metadata:
name: "pv-volume"
spec:
capacity:
storage: "20Gi"
accessModes:
- "ReadWriteOnce"
hostPath:
path: /data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: "orientdb-pv-claim"
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "20Gi"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: orientdbservice
spec:
#replicas: 1
template:
metadata:
name: orientdbservice
labels:
run: orientdbservice
test: orientdbservice
spec:
containers:
- name: orientdbservice
image: orientdb:latest
env:
- name: ORIENTDB_ROOT_PASSWORD
value: "rootpwd"
ports:
- containerPort: 2480
name: orientdb
volumeMounts:
- name: orientdb-config
mountPath: /data/orientdb/config
- name: orientdb-databases
mountPath: /data/orientdb/databases
- name: orientdb-backup
mountPath: /data/orientdb/backup
volumes:
- name: orientdb-config
persistentVolumeClaim:
claimName: orientdb-pv-claim
- name: orientdb-databases
persistentVolumeClaim:
claimName: orientdb-pv-claim
- name: orientdb-backup
persistentVolumeClaim:
claimName: orientdb-pv-claim
---
apiVersion: v1
kind: Service
metadata:
name: orientdbservice
labels:
run: orientdbservice
spec:
type: NodePort
selector:
run: orientdbservice
ports:
- protocol: TCP
port: 2480
name: http
which results in the following:
#kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
pv-volume 20Gi RWO Retain Available 4h
pvc-cd14d593-78fc-11e7-a46d-1277ec3dd2b5 20Gi RWO Delete Bound default/orientdb-pv-claim standard 4h
#kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
orientdb-pv-claim Bound pvc-cd14d593-78fc-11e7-a46d-1277ec3dd2b5 20Gi RWO standard 4h
#kubectl get svc
NAME READY STATUS RESTARTS AGE
orientdbservice-458328598-zsmw5 0/1 ContainerCreating 0 3h
#kubectl describe pod orientdbservice-458328598-zsmw5
.
.
.
Events:
FirstSeen LastSeen Count From SubObjectPath TypeReason Message
--------- -------- ----- ---- ------------- -------- ------ -------
3h 41s 26 kubelet, minikube Warning FailedMount Unable to mount volumes for pod "orientdbservice-458328598-zsmw5_default(392b1298-78ff-11e7-a46d-1277ec3dd2b5)": timeout expired waiting for volumes to attach/mount for pod "default"/"orientdbservice-458328598-zsmw5". list of unattached/unmounted volumes=[orientdb-databases]
It seems that volumes are not able mount for the pod. Is there something wrong with the way I am creating a persistent volume on my node ?
Appreciate all the help

Answer 1

Few questions before I tell you what worked for me:
The directory /data on minikube machine does it have right set of permissions?
In minikube you don't need to worry about setting up volumes in other words don't worry about PersistentVolume anymore, just enable the volume provisioner addon using following command. Once you do that every PersistentVolumeClaim that tries to claim storage will get whatever it needs.
minikube addons enable default-storageclass
So here is what worked for me:
I removed the PersistentVolume
I have changed the mountPath also to match what is given in the upstream docs https://hub.docker.com/_/orientdb/
I have added separate PersistentVolumeClaim for databases and backup
I have changed the config from a PersistentVolumeClaim to configMap, so you don't need to care about how do I get the config to running cluster?, you do it using configMap. Because the config is coming from set of config Files.
Here is the config that worked for me:
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
labels:
app: orientdbservice
name: orientdb-databases
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
status: {}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
labels:
app: orientdbservice
name: orientdb-backup
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
status: {}
---
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: orientdbservice
name: orientdbservice
spec:
ports:
- name: orientdbservice-2480
port: 2480
targetPort: 0
- name: orientdbservice-2424
port: 2424
targetPort: 0
selector:
app: orientdbservice
type: NodePort
status:
loadBalancer: {}
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: orientdbservice
name: orientdbservice
spec:
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: orientdbservice
name: orientdbservice
spec:
containers:
- env:
- name: ORIENTDB_ROOT_PASSWORD
value: rootpwd
image: orientdb
name: orientdbservice
resources: {}
volumeMounts:
- mountPath: /orientdb/databases
name: orientdb-databases
- mountPath: /orientdb/backup
name: orientdb-backup
- mountPath: /orientdb/config
name: orientdb-config
volumes:
- configMap:
name: orientdb-config
name: orientdb-config
- name: orientdb-databases
persistentVolumeClaim:
claimName: orientdb-databases
- name: orientdb-backup
persistentVolumeClaim:
claimName: orientdb-backup
And for configMap I had to goto this directory for sample config examples/3-nodes-compose/var/odb3/config in github repository orientechnologies/orientdb-docker
You goto above directory or directory you have saved config in and run following command:
kubectl create configmap orientdb-config --from-file=.
If you wanna see what is being created automatically and being deployed run following:
kubectl create configmap orientdb-config --from-file=. --dry-run -o yaml
Here is the configMap I have used to do my deployment:
apiVersion: v1
data:
automatic-backup.json: |-
{
"enabled": true,
"mode": "FULL_BACKUP",
"exportOptions": "",
"delay": "4h",
"firstTime": "23:00:00",
"targetDirectory": "backup",
"targetFileName": "${DBNAME}-${DATE:yyyyMMddHHmmss}.zip",
"compressionLevel": 9,
"bufferSize": 1048576
}
backups.json: |-
{
"backups": []
}
default-distributed-db-config.json: |
{
"autoDeploy": true,
"readQuorum": 1,
"writeQuorum": "majority",
"executionMode": "undefined",
"readYourWrites": true,
"newNodeStrategy": "static",
"servers": {
"*": "master"
},
"clusters": {
"internal": {
},
"*": {
"servers": [
"<NEW_NODE>"
]
}
}
}
events.json: |-
{
"events": []
}
hazelcast.xml: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!-- ~ Copyright (c)
2008-2012, Hazel Bilisim Ltd. All Rights Reserved. ~ \n\t~ Licensed under the
Apache License, Version 2.0 (the \"License\"); ~ you may \n\tnot use this file
except in compliance with the License. ~ You may obtain \n\ta copy of the License
at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ \n\t~ Unless required by applicable
law or agreed to in writing, software ~ distributed \n\tunder the License is distributed
on an \"AS IS\" BASIS, ~ WITHOUT WARRANTIES \n\tOR CONDITIONS OF ANY KIND, either
express or implied. ~ See the License for \n\tthe specific language governing
permissions and ~ limitations under the License. -->\n\n<hazelcast\n\txsi:schemaLocation=\"http://www.hazelcast.com/schema/config
hazelcast-config-3.3.xsd\"\n\txmlns=\"http://www.hazelcast.com/schema/config\"
xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\n\t<group>\n\t\t<name>orientdb</name>\n\t\t<password>orientdb</password>\n\t</group>\n\t<network>\n\t\t<port
auto-increment=\"true\">2434</port>\n\t\t<join>\n\t\t\t<multicast enabled=\"true\">\n\t\t\t\t<multicast-group>235.1.1.1</multicast-group>\n\t\t\t\t<multicast-port>2434</multicast-port>\n\t\t\t</multicast>\n\t\t</join>\n\t</network>\n\t<executor-service>\n\t\t<pool-size>16</pool-size>\n\t</executor-service>\n</hazelcast>\n"
orientdb-client-log.properties: |
#
# /*
# * Copyright 2014 Orient Technologies LTD (info(at)orientechnologies.com)
# *
# * Licensed under the Apache License, Version 2.0 (the "License");
# * you may not use this file except in compliance with the License.
# * You may obtain a copy of the License at
# *
# * http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# *
# * For more information: http://www.orientechnologies.com
# */
#
# Specify the handlers to create in the root logger
# (all loggers are children of the root logger)
# The following creates two handlers
handlers = java.util.logging.ConsoleHandler
# Set the default logging level for the root logger
.level = ALL
com.orientechnologies.orient.server.distributed.level = FINE
com.orientechnologies.orient.core.level = WARNING
# Set the default logging level for new ConsoleHandler instances
java.util.logging.ConsoleHandler.level = WARNING
# Set the default formatter for new ConsoleHandler instances
java.util.logging.ConsoleHandler.formatter = com.orientechnologies.common.log.OLogFormatter
orientdb-server-config.xml: |
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<orient-server>
<handlers>
<handler class="com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin">
<parameters>
<parameter value="${distributed}" name="enabled"/>
<parameter value="${ORIENTDB_HOME}/config/default-distributed-db-config.json" name="configuration.db.default"/>
<parameter value="${ORIENTDB_HOME}/config/hazelcast.xml" name="configuration.hazelcast"/>
<parameter value="odb3" name="nodeName"/>
</parameters>
</handler>
<handler class="com.orientechnologies.orient.server.handler.OJMXPlugin">
<parameters>
<parameter value="false" name="enabled"/>
<parameter value="true" name="profilerManaged"/>
</parameters>
</handler>
<handler class="com.orientechnologies.orient.server.handler.OAutomaticBackup">
<parameters>
<parameter value="false" name="enabled"/>
<parameter value="${ORIENTDB_HOME}/config/automatic-backup.json" name="config"/>
</parameters>
</handler>
<handler class="com.orientechnologies.orient.server.handler.OServerSideScriptInterpreter">
<parameters>
<parameter value="true" name="enabled"/>
<parameter value="SQL" name="allowedLanguages"/>
</parameters>
</handler>
<handler class="com.orientechnologies.orient.server.plugin.livequery.OLiveQueryPlugin">
<parameters>
<parameter value="false" name="enabled"/>
</parameters>
</handler>
</handlers>
<network>
<sockets>
<socket implementation="com.orientechnologies.orient.server.network.OServerSSLSocketFactory" name="ssl">
<parameters>
<parameter value="false" name="network.ssl.clientAuth"/>
<parameter value="config/cert/orientdb.ks" name="network.ssl.keyStore"/>
<parameter value="password" name="network.ssl.keyStorePassword"/>
<parameter value="config/cert/orientdb.ks" name="network.ssl.trustStore"/>
<parameter value="password" name="network.ssl.trustStorePassword"/>
</parameters>
</socket>
<socket implementation="com.orientechnologies.orient.server.network.OServerSSLSocketFactory" name="https">
<parameters>
<parameter value="false" name="network.ssl.clientAuth"/>
<parameter value="config/cert/orientdb.ks" name="network.ssl.keyStore"/>
<parameter value="password" name="network.ssl.keyStorePassword"/>
<parameter value="config/cert/orientdb.ks" name="network.ssl.trustStore"/>
<parameter value="password" name="network.ssl.trustStorePassword"/>
</parameters>
</socket>
</sockets>
<protocols>
<protocol implementation="com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary" name="binary"/>
<protocol implementation="com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpDb" name="http"/>
</protocols>
<listeners>
<listener protocol="binary" socket="default" port-range="2424-2430" ip-address="0.0.0.0"/>
<listener protocol="http" socket="default" port-range="2480-2490" ip-address="0.0.0.0">
<commands>
<command implementation="com.orientechnologies.orient.server.network.protocol.http.command.get.OServerCommandGetStaticContent" pattern="GET|www GET|studio/ GET| GET|*.htm GET|*.html GET|*.xml GET|*.jpeg GET|*.jpg GET|*.png GET|*.gif GET|*.js GET|*.css GET|*.swf GET|*.ico GET|*.txt GET|*.otf GET|*.pjs GET|*.svg GET|*.json GET|*.woff GET|*.woff2 GET|*.ttf GET|*.svgz" stateful="false">
<parameters>
<entry value="Cache-Control: no-cache, no-store, max-age=0, must-revalidate\r\nPragma: no-cache" name="http.cache:*.htm *.html"/>
<entry value="Cache-Control: max-age=120" name="http.cache:default"/>
</parameters>
</command>
</commands>
<parameters>
<parameter value="utf-8" name="network.http.charset"/>
<parameter value="true" name="network.http.jsonResponseError"/>
<parameter value="Access-Control-Allow-Origin:*;Access-Control-Allow-Credentials: true" name="network.http.additionalResponseHeaders"/>
</parameters>
</listener>
</listeners>
</network>
<storages/>
<users>
<user resources="*" password="{PBKDF2WithHmacSHA256}8B5E4C8ABD6A68E8329BD58D1C785A467FD43809823C8192:BE5D490BB80D021387659F7EF528D14130B344D6D6A2D590:65536" name="root"/>
<user resources="connect,server.listDatabases,server.dblist" password="{PBKDF2WithHmacSHA256}268A3AFC0D2D9F25AB7ECAC621B5EA48387CF2B9996E1881:CE84E3D0715755AA24545C23CDACCE5EBA35621E68E34BF2:65536" name="guest"/>
</users>
<properties>
<entry value="1" name="db.pool.min"/>
<entry value="50" name="db.pool.max"/>
<entry value="true" name="profiler.enabled"/>
</properties>
<isAfterFirstTime>true</isAfterFirstTime>
</orient-server>
orientdb-server-log.properties: |
#
# /*
# * Copyright 2014 Orient Technologies LTD (info(at)orientechnologies.com)
# *
# * Licensed under the Apache License, Version 2.0 (the "License");
# * you may not use this file except in compliance with the License.
# * You may obtain a copy of the License at
# *
# * http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# *
# * For more information: http://www.orientechnologies.com
# */
#
# Specify the handlers to create in the root logger
# (all loggers are children of the root logger)
# The following creates two handlers
handlers = java.util.logging.ConsoleHandler, java.util.logging.FileHandler
# Set the default logging level for the root logger
.level = INFO
com.orientechnologies.level = INFO
com.orientechnologies.orient.server.distributed.level = INFO
# Set the default logging level for new ConsoleHandler instances
java.util.logging.ConsoleHandler.level = INFO
# Set the default formatter for new ConsoleHandler instances
java.util.logging.ConsoleHandler.formatter = com.orientechnologies.common.log.OAnsiLogFormatter
# Set the default logging level for new FileHandler instances
java.util.logging.FileHandler.level = INFO
# Naming style for the output file
java.util.logging.FileHandler.pattern=../log/orient-server.log
# Set the default formatter for new FileHandler instances
java.util.logging.FileHandler.formatter = com.orientechnologies.common.log.OLogFormatter
# Limiting size of output file in bytes:
java.util.logging.FileHandler.limit=10000000
# Number of output files to cycle through, by appending an
# integer to the base file name:
java.util.logging.FileHandler.count=10
kind: ConfigMap
metadata:
creationTimestamp: null
name: orientdb-config
Here is what it looks like for me:
$ kubectl get all
NAME READY STATUS RESTARTS AGE
po/orientdbservice-4064909316-pzxhl 1/1 Running 0 1h
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/orientdbservice 10.0.0.185 <nodes> 2480:31058/TCP,2424:30671/TCP 1h
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/orientdbservice 1 1 1 1 1h
NAME DESIRED CURRENT READY AGE
rs/orientdbservice-4064909316 1 1 1 1h
$ kubectl get cm
NAME DATA AGE
orientdb-config 8 1h
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
orientdb-backup Bound pvc-9c1507ea-8253-11e7-9e2b-52540058bb88 10Gi RWO standard 1h
orientdb-databases Bound pvc-9c00ca83-8253-11e7-9e2b-52540058bb88 10Gi RWO standard 1h
The config generated above might look little different, it was auto generated from the tool called kedge see the instructions of how I did it in this gist: https://gist.github.com/surajssd/6bbe43a1b2ceee01962e0a1480d8cb04