How to monitor external service in prometheus-operator - kubernetes

Question

I am trying to monitor external service (which is exporter of cassandra metrics) in prometheus-operator. I installed prometheus-operator using helm 2.11.0. I installed it using this yaml:
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
and these commands on my kubernetes cluster:
kubectl create -f rbac-config.yml
helm init --service-account tiller --history-max 200
helm install stable/prometheus-operator --name prometheus-operator --namespace monitoring
Next, basing on article:
how monitor to an external service
I tried to do steps described in it. As suggested I created Endpoints, Service and ServiceMonitor with label for existing Prometheus. Here are my yaml files:
apiVersion: v1
kind: Endpoints
metadata:
name: cassandra-metrics80
labels:
app: cassandra-metrics80
subsets:
- addresses:
- ip: 10.150.1.80
ports:
- name: web
port: 7070
protocol: TCP
apiVersion: v1
kind: Service
metadata:
name: cassandra-metrics80
namespace: monitoring
labels:
app: cassandra-metrics80
release: prometheus-operator
spec:
externalName: 10.150.1.80
ports:
- name: web
port: 7070
protocol: TCP
targetPort: 7070
type: ExternalName
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: cassandra-metrics80
labels:
app: cassandra-metrics80
release: prometheus-operator
spec:
selector:
matchLabels:
app: cassandra-metrics80
release: prometheus-operator
namespaceSelector:
matchNames:
- monitoring
endpoints:
- port: web
interval: 10s
honorLabels: true
And in prometheus service discovery page I can see:
That this service is not active and all labels are dropped.
I did a numerous things trying to fix this, like setting targetLabels. Trying to relabel the once that are discovered, as
described here: prometheus relabeling
But unfortunately nothing works. What could be the issue or how can I investigate it better?

Answer 1

Ok, I found out that service should be in the same namespace as service monitor and endpoint, after that prometheus started to see some metrics from cassandra.

Answer 2

When using the kube-prometheus-stack helm chart, it can be done as follows
prometheus:
prometheusSpec:
additionalScrapeConfigs:
- job_name: external
metrics_path: /metrics
static_configs:
- targets:
- <IP>:<PORT>

Answer 3

To be strict only "Endpoints" and "Service" should be in the same namespace.
Additionally "Endpoints" and "Service" should have the same name as Lucas mentioned it before.
ServiceMonitor can be placed anywhere, it finds and scrapes SVC/Endpoint inside defined namespaces (namespaceSelector->matchNames) and matching all labels (selector->matchLabels):
spec:
selector:
matchLabels:
app: cassandra-metrics80
release: prometheus-operator
namespaceSelector:
matchNames:
- my-namespace
Furthermore now there is much more easier method to define additional scraping:
https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/additional-scrape-config.md
The only drawback for the second one is that it requires pod restart after the change. Configuration based on Endpoint/Service/ServiceMonitor seem to be discovered and applied automatically.