How to configure Rabbitmq Metric in Prometheus/Grafana Kubernetes - kubernetes

Question

My question i want to add rabbitmq monitoring in prometheus. I already have rabbitmq running in Kubernetes but i dont know how to add rabbitmq metric in prometheus
I have install promethues and grafana through yaml file along with pv,pvc,storage,svc,config,deploy and cluster-role
Here is the screenshot of rabbitmq showing empty in promethues
I have install Kubernetes in one vm with local storage i.e.., control-plane and node both install in one vm and everything is working fine
Here is my prometheus-config yaml file
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: monitoring
data:
prometheus.yml: |
global:
scrape_interval: 5s
evaluation_interval: 5s
rule_files:
- /etc/prometheus/prometheus.rules
alerting:
alertmanagers:
- scheme: http
static_configs:
- targets:
- "alertmanager.monitoring.svc:9093"
scrape_configs:
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
- job_name: 'kube-state-metrics'
static_configs:
- targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']
- job_name: 'kubernetes-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
- job_name: 'rabbitmq'
metrics_path: /metrics
scrape_interval: 5s
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- default
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
separator: ;
regex: rabbitmq
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: prometheus
Even after adding rabbitmq metric it is not showing in prometheus url (target)
here is my rabbitmq-svc yaml file
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: xxx-rabbitmq
spec:
selector:
matchLabels:
app: xxx-rabbitmq
serviceName: xxx-rabbitmq
replicas: 1
template:
metadata:
labels:
app: xxx-rabbitmq
spec:
terminationGracePeriodSeconds: 10
containers:
- name: xxx-rabbitmq
image: rabbitmq:3.7.3-management
ports:
- containerPort: xxx
- containerPort: xxx
- containerPort: xxx
- containerPort: xxx
volumeMounts:
- name: xxx-rabbitmq-pvc
mountPath: /var/lib/rabbitmq
subPath: rabbitmq
envFrom:
- configMapRef:
name: rabbitmq-config
volumeClaimTemplates:
- metadata:
name: xxx-rabbitmq-pvc
spec:
accessModes: ["ReadWriteOnce"]
volumeMode: Filesystem
storageClassName: xxxx-storage
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
name: xxx-rabbitmq
labels:
app: xxx-rabbitmq
spec:
type: ClusterIP
ports:
- port: xxx
name: main
- port: xxx
name: rabbitmqssl
- port: xxx
name: rabvitmqmgmt
selector:
app: xxx-rabbitmq
Please help me out how to get all information of rabbitmq in promethues/grafana

Answer 1

If you use rabbitmq of version 3.8 and above you have to use rabbitmq_prometheus plugin.
As the docs state:
The plugin exposes all RabbitMQ metrics on a dedicated TCP port, in
Prometheus text format.
To enable it run:
rabbitmq-plugins enable rabbitmq_prometheus
To validate it's working run inside rabbitmq container:
curl -s localhost:15692/metrics | head -n 3
and see if you get metrics in response.
For versions prior to 3.8, you'd have to use prometheus_rabbitmq_exporter