How to set chown in kubernetes deployment persistently? - mongodb

Question

I'm running a mongodb instance as a kubernetes pod in a single node cluster (bare metal ubuntu machine). The volume is configured ReadWriteOnce as the mongodb pod is accessed only by pods in one node.
apiVersion: apps/v1
kind: Deployment
metadata:
name: mongo
spec:
replicas: 1
selector:
matchLabels:
app: mongo
strategy:
type: Recreate
template:
metadata:
labels:
app: mongo
spec:
hostname: mongo
volumes:
- name: data
persistentVolumeClaim:
claimName: data
- name: restore
persistentVolumeClaim:
claimName: restore
containers:
- name: mongo
image: mongo:4.4.14
args: ["--auth"]
imagePullPolicy: IfNotPresent
ports:
- containerPort: 27017
volumeMounts:
- mountPath: /data/db
name: data
- mountPath: /restore
name: restore
But from time to time I cannot run commands like inserting documents to a non existing collection or running mongodump.
Then I do get the error MongoServerError: 1: Operation not permitted. This is caused by a chown problem: ls -ld /data/db/ is returning
drwxr-sr-x 4 nobody 4294967294 16384 Jun 28 18:19 /data/db/
I can fix the problem by running
chown mongodb:mongodb /data/db
But after some time it changes again, so the same problem happens again and I have to rerun the chown mongodb:mongodb /data/db
I tried to set
securityContext:
runAsUser: 1000
fsGroup: 2000
But then the mongodb pod is failing:
{"t":{"$date":"2022-07-03T10:09:24.379+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2022-07-03T10:09:24.383+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2022-07-03T10:09:24.384+00:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2022-07-03T10:09:24.384+00:00"},"s":"I", "c":"STORAGE", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"mongodb"}}
{"t":{"$date":"2022-07-03T10:09:24.384+00:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.14","gitVersion":"0b0843af97c3ec9d2c0995152d96d2aad725aab7","openSSLVersion":"OpenSSL 1.1.1f 31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2022-07-03T10:09:24.384+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
{"t":{"$date":"2022-07-03T10:09:24.384+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"security":{"authorization":"enabled"}}}}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"E", "c":"STORAGE", "id":20557, "ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"IllegalOperation: Attempted to create a lock file on a read-only directory: /data/db"}}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"REPL", "id":4784900, "ctx":"initandlisten","msg":"Stepping down the ReplicationCoordinator for shutdown","attr":{"waitTimeMillis":10000}}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"COMMAND", "id":4784901, "ctx":"initandlisten","msg":"Shutting down the MirrorMaestro"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"SHARDING", "id":4784902, "ctx":"initandlisten","msg":"Shutting down the WaitForMajorityService"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"NETWORK", "id":20562, "ctx":"initandlisten","msg":"Shutdown: going to close listening sockets"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"NETWORK", "id":4784905, "ctx":"initandlisten","msg":"Shutting down the global connection pool"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"STORAGE", "id":4784906, "ctx":"initandlisten","msg":"Shutting down the FlowControlTicketholder"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"-", "id":20520, "ctx":"initandlisten","msg":"Stopping further Flow Control ticket acquisitions."}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"NETWORK", "id":4784918, "ctx":"initandlisten","msg":"Shutting down the ReplicaSetMonitor"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"SHARDING", "id":4784921, "ctx":"initandlisten","msg":"Shutting down the MigrationUtilExecutor"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"CONTROL", "id":4784925, "ctx":"initandlisten","msg":"Shutting down free monitoring"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"STORAGE", "id":4784927, "ctx":"initandlisten","msg":"Shutting down the HealthLog"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"STORAGE", "id":4784929, "ctx":"initandlisten","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"-", "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"FTDC", "id":4784926, "ctx":"initandlisten","msg":"Shutting down full-time data capture"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"CONTROL", "id":20565, "ctx":"initandlisten","msg":"Now exiting"}
{"t":{"$date":"2022-07-03T10:09:24.386+00:00"},"s":"I", "c":"CONTROL", "id":23138, "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":100}}
I also tried
initContainers:
- name: mongodb-init
image: mongo:4.4.14
command: ["sh", "-c", "chown 999:999 /data/db"]
volumeMounts:
- name: data
mountPath: /data/db
But after some time I again get:
drwxr-sr-x 4 nobody 4294967294 16384 Jun 28 18:19 /data/db/
I do not understand what is causing this behaviour.

Answer 1

If you set securityContext then according to the official documentation:
the runAsUser field specifies that for any Containers in the Pod, all processes run with user ID 1000.
there is no fear of mismatching between different processes of the container.
Perhaps you could give (although not recommended) root user access to the pod by setting runAsUser to 0? Just to see if security context can fix the issue. It should.

Answer 2

It might be useful to run ls -n /data/db to see what uid has ownership of the directory when it's failing. nobody is not useful information.
You set securityContext to 1000, but in the initContainer you set ownership to 999. That won't work.
What user is the container running as? run id.
What user is the mongo process using? There's no assigned user in the Deployment.spec, so I imagine it's a mongoDB config setting. (docs say uid is 999)
Figure out which user id needs to own that directoryissue. Then a possible solution is to mount the path /data with the storage provisioner, then create the path /data/db with an initcontainer for the mongoDB deployment using the correct uid. That way the directories will have the correct ownership.