Volumemounts created as root:artifact though security context provided - kubernetes

Issue:
Deploying Artifactory as a deployment in Kubernetes. The VolumeMounts are being mounted as root:artifact and permissions of drwxr-sr-x
/var/opt/jfrog/artifactory
drwxr-sr-x 2 root artifact 4096 Jan 24 17:52 etc
/var/opt/jfrog/artifactory/etc
-rw-r--r-- 1 root artifact 1048 Jan 24 17:48 artifactory.config.import.yml
-rw-r--r-- 1 root artifact 12703 Jan 24 17:48 artifactory.system.properties
Expected:
The VolumeMount should be mounted as artifact:artifact with read and write permissions
kubernetes manifest file its incomplete due to restriction
spec:
securityContext:
runAsUser: 1030
runAsGroup: 1030
fsGroup: 1030
volumeMounts:
- name: artifactory-volume
mountPath: "/var/opt/jfrog/artifactory"
- name: bootstrap
mountPath: "/var/opt/jfrog/artifactory/etc/artifactory.config.import.yml"
subPath: bootstrap
- name: artifactory-system-properties
mountPath: "/var/opt/jfrog/artifactory/etc/artifactory.system.properties"
subPath: artifactory.system.properties
resources:
limits:
cpu: "3"
memory: 6Gi
requests:
cpu: "2"
memory: 4Gi
volumes:
- name: bootstrap
secret:
secretName: artifactory6170-artifactory
- name: artifactory-system-properties
configMap:
name: artifactory6170-artifactory-system-properties
- name: artifactory-volume
persistentVolumeClaim:
claimName: artifactory6170-artifactory
Kubernetes Version :
Server Version: version.Info{
Major: "1",
Minor: "14",
GitVersion: "v1.14.1",
GitCommit: "b7394102d6ef778017f2ca4046abbaa23b88c290",
GitTreeState: "clean",
BuildDate: "2019-04-08T17:02:58Z",
GoVersion: "go1.12.1",
Compiler: "gc",
Platform: "linux/amd64"
}
I believe the security context covers the required
runAsUser: 1030
runs the process as 1030
runAsGroup: 1030
Any files created will also be owned by user 1030 and group 1030 when runAsGroup is specified.
runs
fsGroup: 1030
the owner of any volume attached will be owner by group ID 1099.
Docker file path
Not sure why the container comes up with wrong user ownership, any help would be really appreciated.
Error:
kubectl logs artifactory6170-artifactory-756cffb9-68zjj
2020-01-26 12:28:13 [719 entrypoint-artifactory.sh] Preparing to run Artifactory in Docker
2020-01-26 12:28:13 [720 entrypoint-artifactory.sh] Running as uid=1030(artifactory) gid=1030(artifactory) groups=1030(artifactory)
2020-01-26 12:28:13 [57 entrypoint-artifactory.sh] Dockerfile for this image can found inside the container.
2020-01-26 12:28:13 [58 entrypoint-artifactory.sh] To view the Dockerfile: 'cat /docker/artifactory-pro/Dockerfile.artifactory'.
2020-01-26 12:28:13 [63 entrypoint-artifactory.sh] Checking open files and processes limits
2020-01-26 12:28:13 [66 entrypoint-artifactory.sh] Current max open files is 1048576
2020-01-26 12:28:13 [78 entrypoint-artifactory.sh] Current max open processes is unlimited
2020-01-26 12:31:13 [211 entrypoint-artifactory.sh] Testing directory /var/opt/jfrog/artifactory has read/write permissions for user 'artifactory' (id 1030)
/entrypoint-artifactory.sh: line 180: /var/opt/jfrog/artifactory/etc/test-permissions: Permission denied
2020-01-26 12:31:13 [229 entrypoint-artifactory.sh] ###########################################################
2020-01-26 12:31:13 [230 entrypoint-artifactory.sh] /var/opt/jfrog/artifactory DOES NOT have proper permissions for user 'artifactory' (id 1030)
2020-01-26 12:31:13 [231 entrypoint-artifactory.sh] Directory: /var/opt/jfrog/artifactory, permissions: 2775, owner: artifactory, group: artifactory
2020-01-26 12:31:13 [232 entrypoint-artifactory.sh] Mounted directory must have read/write permissions for user 'artifactory' (id 1030)
2020-01-26 12:31:13 [233 entrypoint-artifactory.sh] ###########################################################
2020-01-26 12:31:13 [47 entrypoint-artifactory.sh] ERROR: Directory /var/opt/jfrog/artifactory has bad permissions for user 'artifactory' (id 1030)

All i had to do was add an initContainer and mount the Configmaps to /tmp and have it moved to the necessary path /var/opt/jfrog/artifactory/etc/, instead of mounting the configmap in the volumemount /var/opt/jfrog/artifactory.
Reason: ConfigMaps are ReadOnly hence /etc was and would always be Readonly.
initContainers:
- name: "grant-permissions"
image: "busybox:1.26.2"
securityContext:
runAsUser: 0
imagePullPolicy: "IfNotPresent"
command:
- 'sh'
- '-c'
- 'mkdir /var/opt/jfrog/artifactory/etc ; cp -vf /tmp/artifactory* /var/opt/jfrog/artifactory/etc ; chown -R 1030:1030 /var/opt/jfrog/ ; rm -rfv /var/opt/jfrog/artifactory/lost+found'
volumeMounts:
- mountPath: "/var/opt/jfrog/artifactory"
name: artifactory-volume
- name: bootstrap
mountPath: "/tmp/artifactory.config.import.yml"
subPath: bootstrap
readOnly: false
- name: artifactory-system-properties
mountPath: "/tmp/artifactory.system.properties"
subPath: artifactory.system.properties
readOnly: false
then mount the volume to the main container that runs artifactory
containers:
- name: artifactory
image: "registry.eu02.dsg.arm.com/sqa/artifactory-pro:6.17.0"
volumeMounts:
- name: artifactory-volume
mountPath: "/var/opt/jfrog/artifactory"

As explained here, here, here and here you cannot change the permission of mounted directory.
As a work around you can use initContainer which runs before the actual container to change the permission to the directory:
initContainers:
- name: volume-mount
image: busybox
command: ["sh", "-c", "chown -R 1030:1030 <your_directory>"]
volumeMounts:
- name: <your volume>
mountPath: <your mountPath>

Related

How do I cofigure the volume mounts for a Postgres Kubernetes container (OSX & Minikube)

I am trying to setup a postgres pod using minikube. I have the following in my pod config in my deployment
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgredb
volumes:
- name: postgredb
persistentVolumeClaim:
claimName: postgres-pv-claim
but when I run I get
initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
It contains a dot-prefixed/invisible file, perhaps due to it being a mount point.
Using a mount point directly as the data directory is not recommended.
Create a subdirectory under the mount point.
so I change to
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgredb
subPath: db
But then I get...
2021-02-19 03:53:24.760 UTC [30] LOG: could not link file "pg_wal/xlogtemp.30" to "pg_wal/000000010000000000000001": Operation not permitted
2021-02-19 03:53:24.761 UTC [30] FATAL: could not open file "pg_wal/000000010000000000000001": No such file or directory
This is on OSX using minikube, how do I get it to work?
I also tried
env:
- name: PGDATA
value: /var/lib/postgresql/data/db_data
envFrom:
- configMapRef:
name: db-config
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgredb
subPath: db_data
But still get
2021-02-19 03:53:24.760 UTC [30] LOG: could not link file "pg_wal/xlogtemp.30" to "pg_wal/000000010000000000000001": Operation not permitted
2021-02-19 03:53:24.761 UTC [30] FATAL: could not open file "pg_wal/000000010000000000000001": No such file or directory

how to make chown command worked in nfs share folder

I am make a nfs file share and using it in kubernetes pods, but when I start pods, it give me tips :
2020-05-31 03:00:06+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.30-1debian10 started.
chown: changing ownership of '/var/lib/mysql/': Operation not permitted
I searching from internet and understand the nfs default map other root login to nfsnobody account, if the privillege not correct, this error should happen, but I follow the steps and still not solve it. This is the ways I having tried:
1 addd unsecure config no_root_squash in /etc/exports:
/mnt/data/apollodb/apollopv *(rw,sync,no_subtree_check,no_root_squash)
2 remove the PVC and PV and directly using nfs in pod like this:
volumes:
- name: apollo-mysql-persistent-storage
nfs:
server: 192.168.64.237
path: /mnt/data/apollodb/apollopv
containers:
- name: mysql
image: 'mysql:5.7'
ports:
- name: mysql
containerPort: 3306
protocol: TCP
env:
- name: MYSQL_ROOT_PASSWORD
value: gfwge4LucnXwfefewegLwAd29QqJn4
resources: {}
volumeMounts:
- name: apollo-mysql-persistent-storage
mountPath: /var/lib/mysql
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
securityContext: {}
schedulerName: default-scheduler
this tell me the problem not in pod define but in the nfs config itself.
3 give every privillege using this command
chmod 777 /mnt/data/apollodb/apollopv
4 chown to nfsnobody like this
sudo chown nfsnobody:nfsnobody -R apollodb/
sudo chown 999:999 -R apollodb
but the problem still not solved,so what should I try to make it works?
You wouldn't set this via chown, you would use fsGroup security setting instead.

Write permissions on volume mount with OpenShift

Using OpenShift 3.11, I've mounted an nfs persistent volume, but the application cannot copy into the new volume, saying:
oc logs my-project-77858bc694-6kbm6
cp: cannot create regular file '/config/dbdata/resdb.lock.db': Permission denied
...
I've tried to change the ownership of the folder by doing a chown in an InitContainers, but it tells me the operation not permitted.
initContainers:
- name: chowner
image: alpine:latest
command: ["/bin/sh", "-c"]
args:
- ls -alt /config/dbdata; chown 1001:1001 /config/dbdata;
volumeMounts:
- name: my-volume
mountPath: /config/dbdata/
oc logs my-project-77858bc694-6kbm6 -c chowner
total 12
drwxr-xr-x 3 root root 4096 Nov 7 03:06 ..
drwxr-xr-x 2 99 99 4096 Nov 7 02:26 .
chown: /config/dbdata: Operation not permitted
I expect to be able to write to the mounted volume.
You can give your Pods permission to write into a volume by using fsGroup: GROUP_ID in a Security Context. fsGroup makes your volumes writable by GROUP_ID and makes all processes inside your container part of that group.
For example:
apiVersion: v1
kind: Pod
metadata:
name: POD_NAME
spec:
securityContext:
fsGroup: GROUP_ID
...

The server must be started by the user that owns the data directory

I am trying to get some persistant storage for a docker instance of PostgreSQL running on Kubernetes. However, the pod fails with
FATAL: data directory "/var/lib/postgresql/data" has wrong ownership
HINT: The server must be started by the user that owns the data directory.
This is the NFS configuration:
% exportfs -v
/srv/nfs/postgresql/postgres-registry
kubehost*.example.com(rw,wdelay,insecure,no_root_squash,no_subtree_check,sec=sys,rw,no_root_squash,no_all_squash)
$ ls -ldn /srv/nfs/postgresql/postgres-registry
drwxrwxrwx. 3 999 999 4096 Jul 24 15:02 /srv/nfs/postgresql/postgres-registry
$ ls -ln /srv/nfs/postgresql/postgres-registry
total 4
drwx------. 2 999 999 4096 Jul 25 08:36 pgdata
The full log from the pod:
2019-07-25T07:32:50.617532000Z The files belonging to this database system will be owned by user "postgres".
2019-07-25T07:32:50.618113000Z This user must also own the server process.
2019-07-25T07:32:50.619048000Z The database cluster will be initialized with locale "en_US.utf8".
2019-07-25T07:32:50.619496000Z The default database encoding has accordingly been set to "UTF8".
2019-07-25T07:32:50.619943000Z The default text search configuration will be set to "english".
2019-07-25T07:32:50.620826000Z Data page checksums are disabled.
2019-07-25T07:32:50.621697000Z fixing permissions on existing directory /var/lib/postgresql/data ... ok
2019-07-25T07:32:50.647445000Z creating subdirectories ... ok
2019-07-25T07:32:50.765065000Z selecting default max_connections ... 20
2019-07-25T07:32:51.035710000Z selecting default shared_buffers ... 400kB
2019-07-25T07:32:51.062039000Z selecting default timezone ... Etc/UTC
2019-07-25T07:32:51.062828000Z selecting dynamic shared memory implementation ... posix
2019-07-25T07:32:51.218995000Z creating configuration files ... ok
2019-07-25T07:32:51.252788000Z 2019-07-25 07:32:51.251 UTC [79] FATAL: data directory "/var/lib/postgresql/data" has wrong ownership
2019-07-25T07:32:51.253339000Z 2019-07-25 07:32:51.251 UTC [79] HINT: The server must be started by the user that owns the data directory.
2019-07-25T07:32:51.262238000Z child process exited with exit code 1
2019-07-25T07:32:51.263194000Z initdb: removing contents of data directory "/var/lib/postgresql/data"
2019-07-25T07:32:51.380205000Z running bootstrap script ...
The deployment has the following in:
securityContext:
runAsUser: 999
supplementalGroups: [999,1000]
fsGroup: 999
What am I doing wrong?
Edit: Added storage.yaml file:
kind: PersistentVolume
apiVersion: v1
metadata:
name: postgres-registry-pv-volume
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
nfs:
server: 192.168.3.7
path: /srv/nfs/postgresql/postgres-registry
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: postgres-registry-pv-claim
labels:
app: postgres-registry
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
Edit: And the full deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: postgres-registry
spec:
replicas: 1
template:
metadata:
labels:
app: postgres-registry
spec:
securityContext:
runAsUser: 999
supplementalGroups: [999,1000]
fsGroup: 999
containers:
- name: postgres-registry
image: postgres:latest
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 5432
env:
- name: POSTGRES_DB
value: postgresdb
- name: POSTGRES_USER
value: postgres
- name: POSTGRES_PASSWORD
value: Sekret
volumeMounts:
- mountPath: /var/lib/postgresql/data
subPath: "pgdata"
name: postgredb-registry-persistent-storage
volumes:
- name: postgredb-registry-persistent-storage
persistentVolumeClaim:
claimName: postgres-registry-pv-claim
Even more debugging adding:
command: ["/bin/bash", "-c"]
args:["id -u; ls -ldn /var/lib/postgresql/data"]
Which returned:
999
drwx------. 2 99 99 4096 Jul 25 09:11 /var/lib/postgresql/data
Clearly, the UID/GID are wrong. Why?
Even with the work around suggested by Jakub Bujny, I get this:
2019-07-25T09:32:08.734807000Z The files belonging to this database system will be owned by user "postgres".
2019-07-25T09:32:08.735335000Z This user must also own the server process.
2019-07-25T09:32:08.736976000Z The database cluster will be initialized with locale "en_US.utf8".
2019-07-25T09:32:08.737416000Z The default database encoding has accordingly been set to "UTF8".
2019-07-25T09:32:08.737882000Z The default text search configuration will be set to "english".
2019-07-25T09:32:08.738754000Z Data page checksums are disabled.
2019-07-25T09:32:08.739648000Z fixing permissions on existing directory /var/lib/postgresql/data ... ok
2019-07-25T09:32:08.766606000Z creating subdirectories ... ok
2019-07-25T09:32:08.852381000Z selecting default max_connections ... 20
2019-07-25T09:32:09.119031000Z selecting default shared_buffers ... 400kB
2019-07-25T09:32:09.145069000Z selecting default timezone ... Etc/UTC
2019-07-25T09:32:09.145730000Z selecting dynamic shared memory implementation ... posix
2019-07-25T09:32:09.168161000Z creating configuration files ... ok
2019-07-25T09:32:09.200134000Z 2019-07-25 09:32:09.199 UTC [70] FATAL: data directory "/var/lib/postgresql/data" has wrong ownership
2019-07-25T09:32:09.200715000Z 2019-07-25 09:32:09.199 UTC [70] HINT: The server must be started by the user that owns the data directory.
2019-07-25T09:32:09.208849000Z child process exited with exit code 1
2019-07-25T09:32:09.209316000Z initdb: removing contents of data directory "/var/lib/postgresql/data"
2019-07-25T09:32:09.274741000Z running bootstrap script ... 999
2019-07-25T09:32:09.278124000Z drwx------. 2 99 99 4096 Jul 25 09:32 /var/lib/postgresql/data
Using your setup and ensuring the nfs mount is owned by 999:999 it worked just fine.
You're also missing an 's' in your name: postgredb-registry-persistent-storage
And with your subPath: "pgdata" do you need to change the $PGDATA? I didn't include the subpath for this.
$ sudo mount 172.29.0.218:/test/nfs ./nfs
$ sudo su -c "ls -al ./nfs" postgres
total 8
drwx------ 2 postgres postgres 4096 Jul 25 14:44 .
drwxrwxr-x 3 rei rei 4096 Jul 25 14:44 ..
$ kubectl apply -f nfspv.yaml
persistentvolume/postgres-registry-pv-volume created
persistentvolumeclaim/postgres-registry-pv-claim created
$ kubectl apply -f postgres.yaml
deployment.extensions/postgres-registry created
$ sudo su -c "ls -al ./nfs" postgres
total 124
drwx------ 19 postgres postgres 4096 Jul 25 14:46 .
drwxrwxr-x 3 rei rei 4096 Jul 25 14:44 ..
drwx------ 3 postgres postgres 4096 Jul 25 14:46 base
drwx------ 2 postgres postgres 4096 Jul 25 14:46 global
drwx------ 2 postgres postgres 4096 Jul 25 14:46 pg_commit_ts
. . .
I noticed using nfs: directly in the persistent volume took significantly longer to initialize the database, whereas using hostPath: to the mounted nfs volume behaved normally.
So after a few minutes:
$ kubectl logs postgres-registry-675869694-9fp52 | tail -n 3
2019-07-25 21:50:57.181 UTC [30] LOG: database system is ready to accept connections
done
server started
$ kubectl exec -it postgres-registry-675869694-9fp52 psql
psql (11.4 (Debian 11.4-1.pgdg90+1))
Type "help" for help.
postgres=#
Checking the uid/gid
$ kubectl exec -it postgres-registry-675869694-9fp52 bash
postgres#postgres-registry-675869694-9fp52:/$ whoami && id -u && id -g
postgres
999
999
nfspv.yaml:
kind: PersistentVolume
apiVersion: v1
metadata:
name: postgres-registry-pv-volume
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
nfs:
server: 172.29.0.218
path: /test/nfs
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: postgres-registry-pv-claim
labels:
app: postgres-registry
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
postgres.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: postgres-registry
spec:
replicas: 1
template:
metadata:
labels:
app: postgres-registry
spec:
securityContext:
runAsUser: 999
supplementalGroups: [999,1000]
fsGroup: 999
containers:
- name: postgres-registry
image: postgres:latest
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 5432
env:
- name: POSTGRES_DB
value: postgresdb
- name: POSTGRES_USER
value: postgres
- name: POSTGRES_PASSWORD
value: Sekret
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgresdb-registry-persistent-storage
volumes:
- name: postgresdb-registry-persistent-storage
persistentVolumeClaim:
claimName: postgres-registry-pv-claim
I cannot explain why those 2 IDs are different but as workaround I would try to override postgres's entrypoint with
command: ["/bin/bash", "-c"]
args: ["chown -R 999:999 /var/lib/postgresql/data && ./docker-entrypoint.sh postgres"]
This type of errors is quite common when you link a NTFS directory into your docker container. NTFS directories don't support ext3 file & directory access control. The only way to make it work is to link directory from a ext3 drive into your container.
I got a bit desperate when I played around Apache / PHP containers with linking the www folder. After I linked files reside on a ext3 filesystem the problem disappear.
I published a short Docker tutorial on youtube, may it helps to understand this problem: https://www.youtube.com/watch?v=eS9O05TTFjM

The rbd volumes config of rbd issue in kubernetes

I want to use the volumes rbd config to mount the folder on ceph images .
But it seems the container mount a host path.
I used the daemon of "https://github.com/kubernetes/kubernetes/tree/master/examples/rbd".
The pod and container start successfully.
I use the "docker exec " login the container and watch the /mnt folder.
root#test-rbd-read-01:/usr/local/tomcat# findmnt /mnt
TARGET SOURCE FSTYPE OPTIONS
/mnt /dev/vda1[/var/lib/kubelet/pods/****/volumes/kubernetes.io~rbd/rbd] xfs rw,relatime,attr2,inode64,noquota
root#test-rbd-read-01:/usr/local/tomcat# ls /mnt/
root#test-rbd-read-01:/usr/local/tomcat#
And then I watch the host path that mount on the ceph. The 1.txt had existed on ceph image.
[20:52 root#mongodb:/home] # mount |grep kubelet
/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/wujianlin-image-zlh_test type ext4 (ro,relatime,stripe=1024,data=ordered)
/dev/rbd0 on /var/lib/kubelet/pods/****/volumes/kubernetes.io~rbd/rbd type ext4 (ro,relatime,stripe=1024,data=ordered)
[20:53 root#mongodb:/home] # ll /var/lib/kubelet/pods/****/volumes/kubernetes.io~rbd/rbd
total 20K
drwx------ 2 root root 16K Mar 18 09:49 lost+found
-rw-r--r-- 1 root root 4 Mar 18 09:53 1.txt
[20:53 root#mongodb:/home] # rbd showmapped
id pool image snap device
0 wujianlin zlh_test - /dev/rbd0
It should except that the container folder /mnt is same as the host path /var/lib/kubelet/pods/ * * * */volumes/kubernetes.io~rbd/rbd, but it was not.`
And I try to write file to /mnt, it also can not see any changes in /var/lib/kubelet/pods/* * * */volumes/kubernetes.io~rbd/rbd
So is my some config wrong, or someting misunderstand ?
k8s version: Release v1.2.0
Here is my config:
apiVersion: v1
kind: Pod
metadata:
name: test-rbd-read-01
spec:
containers:
- name: tomcat-read-only-01
image: tomcat
volumeMounts:
- name: rbd
mountPath: /mnt
volumes:
- name: rbd
rbd:
monitors:
- 10.63.90.177:6789
pool: wujianlin
image: zlh_test
user: wujianlin
secretRef:
name: ceph-client-admin-keyring
keyring: /etc/ceph/ceph.client.wujianlin.keyring
fsType: ext4
readOnly: true
what did you do when you restart docker? Are you able to reproduce this issue after the docker is restarted and pod is recreated?