Prometheus configuration for monitoring Orleans in Kubernetes - kubernetes

Question

I'm trying to get Prometheus functioning with my Orleans silos...
I use this consumer to expose Orleans metrics for Prometheus on port 8082. With a local Prometheus instance and using the grafana.json from the same repository I see that it works.
_ = builder.AddPrometheusTelemetryConsumerWithSelfServer(port: 8082);
Following this guide to install Prometheus on Kubernetes on a different namespace that my silos are deployed.
Following instructions I added the prometheus labels to my orleans deployment yaml:
spec:
replicas: 2
selector:
matchLabels:
app: mysilo
template:
metadata:
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '8082'
labels:
app: mysilo
My job in prometheus yml:
- job_name: "orleans"
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- orleans
selectors:
- role: "pod"
label: "app=mysilo"
According to the same guide, all the pods metrics get discovered if "the pod metadata is annotated with prometheus.io/scrape and prometheus.io/port annotations.". I assume I don't need any extra installations.
With all this, and port forwarding my prometheus pod, I can see prometheus is working in http://localhost:9090/metrics but no metrics are being shown in my grafana dashboard (again, I could make it work in local machine with only one silo).
When exploring grafana I find that it seems it can't find the instances:
sum(rate(process_cpu_seconds_total{job=~"orleans", instance=~"()"}[3m])) * 100
The aim is to monitor resources my orleans silos are using (not the pods metrics themselves, but orleans metrics), but I'm missing something :(

Answer 1

Thanks to #BozoJoe's comment I could debug this.
The problem was that it was trying to scrape ports 30000 and 1111 instead of 8082 like I said before. I could see this thanks to the Prometheus dashboard at localhost:9090/targets
So I went to prometheus config file and make sure to start scrapping the correct port (also I added some restrictions to the search for name):
- job_name: "orleans"
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- orleans
selectors:
- role: "pod"
label: "app=mysilo"
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_name]
action: keep
regex: 'my-silo-name*'
- source_labels: [__address__]
action: replace
regex: ([^:]+):.*
replacement: $1:8081
target_label: __address__