RabbitMQ Cluster Kubernetes Operator on minikube - kubernetes

Question

I'm trying to set up RabbitMQ on Minikube using the RabbitMQ Cluster Operator:
When I try to attach a persistent volume, I get the following error:
$ kubectl logs -f rabbitmq-rabbitmq-server-0
Configuring logger redirection
20:04:40.081 [warning] Failed to write PID file "/var/lib/rabbitmq/mnesia/rabbit#rabbitmq-rabbitmq-server-0.rabbitmq-rabbitmq-headless.default.pid": permission denied
20:04:40.264 [error] Failed to create Ra data directory at '/var/lib/rabbitmq/mnesia/rabbit#rabbitmq-rabbitmq-server-0.rabbitmq-rabbitmq-headless.default/quorum/rabbit#rabbitmq-rabbitmq-server-0.rabbitmq-rabbitmq-headless.default', file system operation error: enoent
20:04:40.265 [error] Supervisor ra_sup had child ra_system_sup started with ra_system_sup:start_link() at undefined exit with reason {error,"Ra could not create its data directory. See the log for details."} in context start_error
20:04:40.266 [error] CRASH REPORT Process <0.247.0> with 0 neighbours exited with reason: {error,"Ra could not create its data directory. See the log for details."} in ra_system_sup:init/1 line 43
20:04:40.267 [error] CRASH REPORT Process <0.241.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,ra_system_sup,{error,"Ra could not create its data directory. See the log for details."}}},{ra_app,start,[normal,[]]}} in application_master:init/4 line 138
{"Kernel pid terminated",application_controller,"{application_start_failure,ra,{{shutdown,{failed_to_start_child,ra_system_sup,{error,\"Ra could not create its data directory. See the log for details.\"}}},{ra_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,ra,{{shutdown,{failed_to_start_child,ra_system_sup,{error,"Ra could not create its data directory. See the log for details."}
Crash dump is being written to: erl_crash.dump...
The issue is that RabbitMQ is not able to set up its data files in the data directory /var/lib/rabbitmq/mnesia due to a lack of permission.
My initial guess was that I needed to specify the data directory as a volumeMount, but that doesn't seem to be configurable according to the documentation.
RabbitMQ's troubleshooting documentation on persistence results in a 404.
I tried to find other resources online with the same problem but none of them were using the RabbitMQ Cluster Operator. I plan on following that route if I'm not able to find a solution but I really would like to solve this issue somehow.
Does anyone have any ideas?
The setup that I have is as follows:
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rabbitmq
spec:
replicas: 1
service:
type: NodePort
persistence:
storageClassName: local-storage
storage: 20Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rabbitmq-pvc
spec:
storageClassName: local-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: rabbitmq-pv
spec:
storageClassName: local-storage
accessModes:
- ReadWriteOnce
capacity:
storage: 20Gi
hostPath:
path: /mnt/app/rabbitmq
type: DirectoryOrCreate
To reproduce this issue on minikube:
Install the rabbitmq operator:
kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"
Apply the manifest file above
kubectl apply -f rabbitmq.yml
Running kubectl get po displays a pod named rabbitmq-rabbitmq-server-0.
Running kubectl logs -f rabbitmq-rabbitmq-server-0 to view the logs displays the above error.

Answer 1

As I alread suggested in comments, you can solve it running:
minikube ssh -- sudo chmod g+w /mnt/app/rabbitmq/
Answering to your question:
Is there a way I can add that to my manifest file rather than having to do it manually?
You can override the rabbitmq statefulset manifest fields to change last line in initContainer command script from chgrp 999 /var/lib/rabbitmq/mnesia/ to this: chown 999:999 /var/lib/rabbitmq/mnesia/.
Complete RabbitmqCluster yaml manifest looks like following:
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rabbitmq
spec:
replicas: 1
service:
type: NodePort
persistence:
storageClassName: local-storage
storage: 20Gi
override:
statefulSet:
spec:
template:
spec:
containers: []
initContainers:
- name: setup-container
command:
- sh
- -c
- cp /tmp/rabbitmq/rabbitmq.conf /etc/rabbitmq/rabbitmq.conf && chown 999:999
/etc/rabbitmq/rabbitmq.conf && echo '' >> /etc/rabbitmq/rabbitmq.conf ; cp /tmp/rabbitmq/advanced.config
/etc/rabbitmq/advanced.config && chown 999:999 /etc/rabbitmq/advanced.config
; cp /tmp/rabbitmq/rabbitmq-env.conf /etc/rabbitmq/rabbitmq-env.conf && chown
999:999 /etc/rabbitmq/rabbitmq-env.conf ; cp /tmp/erlang-cookie-secret/.erlang.cookie
/var/lib/rabbitmq/.erlang.cookie && chown 999:999 /var/lib/rabbitmq/.erlang.cookie
&& chmod 600 /var/lib/rabbitmq/.erlang.cookie ; cp /tmp/rabbitmq-plugins/enabled_plugins
/etc/rabbitmq/enabled_plugins && chown 999:999 /etc/rabbitmq/enabled_plugins
; chown 999:999 /var/lib/rabbitmq/mnesia/ # <- CHANGED THIS LINE

Answer 2

I had the same issue when deploying RabbitMQ in kubernetes inside Vagrant (not minikube though). I was using this setup.
I tried running sudo chmod g+w /mnt/app/rabbitmq/ but had no luck...
Eventually gave up and ended up running minikube inside vagrant using this box and everything worked perfectly fine out of the box! Didn't have to do anything special... Not event manually creating the persistent volume...
inside my nodes

Answer 3

I had this issue in a live version and minikube does not allow to run SSH command then.So what I did was run chmod to the my hostpath provisioner and recreated my rabbitmq-cluster
chmod 777 /tmp/hostpath-provisioner/default/*

Answer 4

I found the answer for this issue. It happen when there is few nodes in cluster.
Solution is to add securityContext: {}
https://github.com/rabbitmq/rabbitmq-website/blob/3ee8e72a7c4fe52e323ba1039eecbf3a67c554f7/site/kubernetes/operator/using-on-openshift.md#arbitrary-user-ids