Prometheus Operator not scraping colocated etcd metrics - kubernetes

Question

I have a K8s cluster with colocated etcd deployed on-prem servers, using Kubespray. I don't see the etcd metrics getting scraped by Prometheus operator. Prometheus operator deployed using helm v3.5.4.
K8s version 1.22 , Helm chart prometheus-community/kube-prometheus-stack version 25.0.0 , 3 node control plane on CentOS 7.
Prometheus config shows a job for etcd - job_name: serviceMonitor/monitoring/kube-prometheus-kube-prome-kube-etcd/0 .
But there is no service for etcd in the list of Services for Prometheus.
There are no endpoints defined for etcd
Values.yml (updated with volumes ) for helm deployment
prometheus:
service:
type: NodePort
externalTrafficPolicy: Local
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: "custom"
hosts:
- prometheus.{{ cluster_domain }}.mydomain.com
paths:
- /
pathType: Prefix
tls:
- secretName:
prometheusSpec:
storageSpec:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: rook-ceph-block
resources:
requests:
storage: {{ monitoring.storage_size }}
volumeMounts:
- name: cert-vol
mountPath: "/etc/prometheus/secrets/etcd-certs"
readOnly: true
volumes:
- name: cert-vol
secret:
secretName: etcd-certs
kubeEtcd:
enabled: true
endpoints:
- 172.1.1.1
- 172.1.1.2
- 172.1.1.3
service:
port: 2379
targetPort: 2379
serviceMonitor:
scheme: https
insecureSkipVerify: true
caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
certFile: /etc/prometheus/secrets/etcd-certs/client.crt
keyFile: /etc/prometheus/secrets/etcd-certs/client.key

Answer 1

I added the endpoints to kubeEtcd section to get it to work. The updated values.yaml is like below (changed IP adresses):
prometheus:
service:
type: NodePort
externalTrafficPolicy: Local
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: "custom"
hosts:
- prometheus.{{ cluster_domain }}.mydomain.com
paths:
- /
pathType: Prefix
tls:
- secretName:
prometheusSpec:
storageSpec:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: rook-ceph-block
resources:
requests:
storage: {{ monitoring.storage_size }}
volumeMounts:
- name: cert-vol
mountPath: "/etc/prometheus/secrets/etcd-certs"
readOnly: true
volumes:
- name: cert-vol
secret:
secretName: etcd-certs
kubeEtcd:
enabled: true
endpoints:
- 172.1.1.1
- 172.1.1.2
- 172.1.1.3
service:
port: 2379
targetPort: 2379
serviceMonitor:
scheme: https
insecureSkipVerify: true
caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
certFile: /etc/prometheus/secrets/etcd-certs/client.crt
keyFile: /etc/prometheus/secrets/etcd-certs/client.key