I have tried to deploy Kafka in k8s, so I need to persist its volume with hostpath, but when the volume configuration adds to the deployment file, this error shows in Kafka pod, and the pod state becomes Crashloopbackoff:
mkdir: cannot create directory ‘/bitnami/config’: Permission denied
I think I have to change permission so the pod can create this file.
Deployment.yml:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kafka-broker
name: kafka-broker
namespace: kafka
spec:
replicas: 1
selector:
matchLabels:
app: kafka-broker
template:
metadata:
labels:
app: kafka-broker
spec:
containers:
- env:
- name: ALLOW_PLAINTEXT_LISTENER
value: "yes"
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: zookeeper-service:2181
- name: KAFKA_LISTENERS
value: PLAINTEXT://:9092
- name: KAFKA_ADVERTISED_LISTENERS
value: PLAINTEXT://:9092
image: bitnami/kafka
imagePullPolicy: IfNotPresent
name: kafka-broker
ports:
- containerPort: 9092
volumeMounts:
- name: kafka-data
readOnly: false
mountPath: "/bitnami/kafka"
volumes:
- name: kafka-data
hostPath:
path: /data/kafka-data
I have solved the problem by changing the path (where I mount the pod data )ownership on the worker servers with this command:
sudo chown -R 1001:1001 /data/kafka-data
But I think this solution is not bestpractice.
Have you tried to set
securityContext:
runAsUser: 1001
runAsGroup: 1001
fsGroup: 1001
in your deployment config ?
Bitnami images are non-root user, and required user 1001
Same for mounted volumes (=> fsGroup)
See https://github.com/bitnami/bitnami-docker-kafka#persisting-your-data
Related
I am trying to use git-sync image as a side car in kubernetes that runs git-pull periodically and mounts cloned data to shared volume.
Everything is working fine when I configure it for sync one time. I want to run it periodically like every 10 mins. Somehow when I configure it to run periodically pod initializing is failing.
I read documentation but couldn't find proper answer. Would be nice if you help me to figure out what I am missing in my configuration.
Here is my configuration that failing.
Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx-helloworld
image: nginx
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: www-data
initContainers:
- name: git-sync
image: k8s.gcr.io/git-sync:v3.1.3
volumeMounts:
- name: www-data
mountPath: /data
env:
- name: GIT_SYNC_REPO
value: "https://github.com/musaalp/design-patterns.git" ##repo-path-you-want-to-clone
- name: GIT_SYNC_BRANCH
value: "master" ##repo-branch
- name: GIT_SYNC_ROOT
value: /data
- name: GIT_SYNC_DEST
value: "hello" ##path-where-you-want-to-clone
- name: GIT_SYNC_PERIOD
value: "10"
- name: GIT_SYNC_ONE_TIME
value: "false"
securityContext:
runAsUser: 0
volumes:
- name: www-data
emptyDir: {}
Pod
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: nginx-helloworld
name: nginx-helloworld
spec:
containers:
- image: nginx
name: nginx-helloworld
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Never
status: {}
you are using the git-sync as an initContainers, which run only during init (once in lifecycle)
A Pod can have multiple containers running apps within it, but it can also have one or more init containers, which are run before the app containers are started.
Init containers are exactly like regular containers, except:
Init containers always run to completion.
Each init container must complete successfully before the next one starts.
init-containers
So use this as a regular container
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: git-sync
image: k8s.gcr.io/git-sync:v3.1.3
volumeMounts:
- name: www-data
mountPath: /data
env:
- name: GIT_SYNC_REPO
value: "https://github.com/musaalp/design-patterns.git" ##repo-path-you-want-to-clone
- name: GIT_SYNC_BRANCH
value: "master" ##repo-branch
- name: GIT_SYNC_ROOT
value: /data
- name: GIT_SYNC_DEST
value: "hello" ##path-where-you-want-to-clone
- name: GIT_SYNC_PERIOD
value: "20"
- name: GIT_SYNC_ONE_TIME
value: "false"
securityContext:
runAsUser: 0
- name: nginx-helloworld
image: nginx
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: www-data
volumes:
- name: www-data
emptyDir: {}
I am trying to get NFS share mounted into a k8s pod but its failing with the below error
mount.nfs: rpc.statd is not running but is required for remote locking. mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
I tried to start rpcbind using CMD command in docker container that also did not work.
My deployment yaml is as below
kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-client-provisioner
spec:
selector:
matchLabels:
app: nfs-client-provisioner
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner:v4.0.0
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: nfs-provisioner
- name: NFS_SERVER
value: NFS SERVER PATH
- name: NFS_PATH
value: /filesharepath
volumes:
- name: nfs-client-root
nfs:
server: <NFS IP>
path: /filesharepath
I saw in github that there is an identical issue which says rpcbind needs to be on the base system.
https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner/issues/224
Please assist.
I'm running the theia code-editor on my EKS cluster and the image's default user is theia on which I grant read and write permissions on /home/project. However, when I mount that volume /home/project on my EFS and try to read or write on /home/project it returns permission denied I tried using initContainer but still the same problem:
apiVersion: apps/v1
kind: Deployment
metadata:
name: atouati
spec:
replicas: 1
selector:
matchLabels:
app: atouati
template:
metadata:
labels:
app: atouati
spec:
initContainers:
- name: take-data-dir-ownership
image: alpine:3
command:
- chown
- -R
- 1001:1001
- /home/project:cached
volumeMounts:
- name: project-volume
mountPath: /home/project:cached
containers:
- name: theia
image: 'xxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/theia-code-editor:latest'
ports:
- containerPort: 3000
volumeMounts:
- name: project-volume
mountPath: "/home/project:cached"
volumes:
- name: project-volume
persistentVolumeClaim:
claimName: local-storage-pvc
---
apiVersion: v1
kind: Service
metadata:
name: atouati
spec:
type: ClusterIP
selector:
app: atouati
ports:
- protocol: TCP
port: 80
targetPort: 3000
When I do ls -l on /home/project
drwxr-xr-x 2 theia theia 6 Aug 21 17:33 project
On the efs directory :
drwxr-xr-x 4 root root 6144 Aug 21 17:32
You can instead set the securityContext in your pod spec to run the Pods as uid/gid 1001.
For example
apiVersion: apps/v1
kind: Deployment
metadata:
name: atouati
spec:
replicas: 1
selector:
matchLabels:
app: atouati
template:
metadata:
labels:
app: atouati
spec:
securityContext:
runAsUser: 1001
runAsGroup: 1001
fsGroup: 1001
containers:
- name: theia
image: 'xxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/theia-code-editor:latest'
ports:
- containerPort: 3000
volumeMounts:
- name: project-volume
mountPath: "/home/project:cached"
volumes:
- name: project-volume
persistentVolumeClaim:
claimName: local-storage-pvc
Have you kubectl execd into the container to confirm that that's the uid/gid that you need to use based on the apparent ownership?
Hi I have set up an small NFS server at home using my raspberry pi.
An I want to set that as the default storage for all of my kubernetes containers.
However I keep on getting this chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted
here is my config.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: pg-ss
spec:
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:9.6
volumeMounts:
- name: pv-data
mountPath: /var/lib/postgresql/data
env:
- name: POSTGRES_USER
value: postgres
- name: POSTGRES_PASSWORD
value: postgres
ports:
- containerPort: 5432
name: postgredb
volumes:
- name: pv-data
nfs:
path: /mnt/infra-data/pg
server: 192.168.1.150
readOnly: false
I'm wondering what would be the cause of this. and how can i solve it.
Thanks,
Postgres can't start giving the error:
PANIC could not locate a valid checkpoint record
On Google, there is a lot of solution, but all of them need to connect the pod to execute some pg commands.
But, as I use K8S, my pod falls into status: CrashLoopBackOff, so I can't connect anymore to my pod.
How should I do to fix my postgres DB ?
EDIT:
I have tried to run the command:
pg_resetwal /var/lib/postgresql/data
with:
...
spec:
containers:
- args:
- pg_resetwal
- /var/lib/postgresql/data
But I get:
pg_resetwal: cannot be executed by "root"
You must run pg_resetwal as the PostgreSQL superuser.
Can go further...
EDIT2:
I tried to run a new pod with the same volumes attached, and the same postgres container, but changing the command to : pg_resetwal /var/lib/postgresql/data
I also added:
securityContext:
runAsUser: 0
Here is the yaml for deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
reloader.stakater.com/auto: "true"
labels:
app: metadata-postgres-fix
name: metadata-postgres-fix
namespace: metadata
spec:
selector:
matchLabels:
app: metadata-postgres-fix
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: metadata-postgres-fix
spec:
containers:
- args:
- pg_resetwal
- /var/lib/postgresql/data
envFrom:
- secretRef:
name: metadata-env
image: postgres:11.3
name: metadata-postgres-fix
securityContext:
runAsUser: 0
ports:
- containerPort: 5432
imagePullPolicy: Always
volumeMounts:
- mountPath: /etc/postgresql/postgresql.conf
name: metadata-postgres-data
subPath: postgres.conf
- mountPath: /docker-entrypoint-initdb.d/init.sh
name: metadata-postgres-data
subPath: init.sh
- mountPath: /var/lib/postgresql/data
name: metadata-postgres-claim
subPath: postgres
restartPolicy: Always
volumes:
- name: metadata-postgres-data
configMap:
name: cfgmap-metadata-postgres
- name: metadata-postgres-claim
persistentVolumeClaim:
claimName: metadata-postgres-claim
nodeSelector:
kops.k8s.io/instancegroup: nodes
I solved it changing
- args:
- pg_resetwal
- /var/lib/postgresql/data
with a pause to be able to get UID of postgres:
- args:
- sleep
- 1000
with
cat /etc/passwd
I could find posgres UID is 999
and finally change runAsUser: 0 with runAsUser: 999