Prometheus operator "Error opening query log file: permission denied" - kubernetes

Question

I'm trying to run Prometheus operator on Kubernetes, but after I trying to persist data on Rancher-RBD, there is an error:
level=info ts=2020-10-31T12:40:33.171Z caller=main.go:353 msg="Starting Prometheus" version="(version=2.22.0, branch=HEAD, revision=0a7fdd3b76960808c3a91d92267c3d815c1bc354)"
level=info ts=2020-10-31T12:40:33.171Z caller=main.go:358 build_context="(go=go1.15.3, user=root#6321101b2c50, date=20201015-12:29:59)"
level=info ts=2020-10-31T12:40:33.171Z caller=main.go:359 host_details="(Linux 4.14.35-1902.3.2.el7uek.x86_64 #2 SMP Tue Jul 30 03:59:02 GMT 2019 x86_64 prometheus-prometheus-0 (none))"
level=info ts=2020-10-31T12:40:33.171Z caller=main.go:360 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2020-10-31T12:40:33.171Z caller=main.go:361 vm_limits="(soft=unlimited, hard=unlimited)"
level=error ts=2020-10-31T12:40:33.173Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log
goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fff711299c3, 0xb, 0x14, 0x30867c0, 0xc000e6f050, 0x30867c0)
/app/promql/query_logger.go:117 +0x4cf
main.main()
/app/cmd/prometheus/main.go:388 +0x536c
this is my operator deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: prometheus-operator
app.kubernetes.io/version: v0.43.0
name: prometheus-operator
namespace: monitorings
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: prometheus-operator
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: prometheus-operator
app.kubernetes.io/version: v0.43.0
spec:
containers:
- args:
- --kubelet-service=kube-system/kubelet
- --logtostderr=true
- --prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.43.0
image: quay.io/prometheus-operator/prometheus-operator:v0.43.0
name: prometheus-operator
ports:
- containerPort: 8080
name: http
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
securityContext:
allowPrivilegeEscalation: false
nodeSelector:
beta.kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 65534
serviceAccountName: prometheus-operator
I tried to add an initContainers to change permission but the problem already exusts:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
labels:
app: strimzi
spec:
replicas: 1
serviceAccountName: prometheus-server
podMonitorSelector:
matchLabels:
app: strimzi
resources:
requests:
memory: 400Mi
enableAdminAPI: false
ruleSelector:
matchLabels:
role: alert-rules
app: strimzi
alerting:
alertmanagers:
- namespace: monitorings
name: alertmanager
port: alertmanager
additionalScrapeConfigs:
name: additional-scrape-configs
key: prometheus-additional.yaml
imagePullSecrets:
- name: nexuspullsecret
initContainers:
- name: init
image: debian:stable
command: ["chmod", "-R", "777", "/mnt"]
volumeMounts:
- name: prometheus-prometheus-db
mountPath: /mnt
subPath: prometheus
storage:
volumeClaimTemplate:
spec:
storageClassName: rancher-rbd
name: prometheus-prometheus-db
resources:
requests:
storage: 10Gi

Answer 1

I changed my initContainers like here and it works:
initContainers:
- name: "init-datapath"
image: debian:stable
command: ["chown", "-R", "65534:65534", "/data"]
command: ["/bin/chmod","-R","777","/data"]
volumeMounts:
- name: prometheus-prometheus-db
mountPath: /data
subPath: ""
storage:
volumeClaimTemplate:
spec:
storageClassName: rancher-rbd
name: prometheus-prometheus-db
resources:
requests:
storage: 10Gi