wget : can't connect to remote host when connecting to kubernetes service in separate namespace - kubernetes

Question

I have setup my API for Prometheus monitoring using client instrumentation but when I check the endpoint in PromUI it is in DOWN status as below :
I checked for connectivity issues by exec into the prometheus pod and trying a wget. When I do so I get :
/prometheus $ wget metrics-clusterip.labs.svc:9216
Connecting to metrics-clusterip.labs.svc:9216 (10.XXX.XX.XXX:9216)
wget: can't connect to remote host (10.XXX.XX.XX): Connection refused
I have an existing MongoDB instance with a Prometheus exporter sidecar and its working perfectly (ie its endpoint is UP in PromUI). As a check I tried connecting to this MongoDB instance's service and it turns out the prometheus pod can indeed connect (as expected) :
wget mongodb-metrics.labs.svc:9216
Connecting to mongodb-metrics.labs.svc:9216 (10.XXX.XX.X:9216)
wget: can't open 'index.html': File exists
This is what I have also tried :
I tried both nslookup metrics-clusterip.svc.labs and nslookup metrics-clusterip.svc.labs:9216 from the prometheus pod but I got same error :
nslookup metrics-clusterip.svc.labs:9216
Server: 169.XXX.XX.XX
Address: 169.XXX.XX.XX:XX
** server can't find metrics-clusterip.svc.labs:9216: NXDOMAIN
*** Can't find metrics-clusterip.svc.labs:9216: No answer
However, when I port-forward the service I can successfully query the metrics endpoint and this shows that the metrics endpoint is up in the API container :
kubectl port-forward svc/metrics-clusterip 9216
NB:Both the API routes and the metrics endpoint are using the same port (9216)
Check if DNS pod is running
kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-XXXXXXXXX-XXXXX 1/1 Running 0 294d
coredns-XXXXXXXXX-XXXXX 1/1 Running 0 143d
This is the configuration for my deployment :
apiVersion: apps/v1
kind: Deployment
metadata:
name: reg
labels:
app: reg
namespace: labs
spec:
replicas: 1
selector:
matchLabels:
app: reg
release: loki
template:
metadata:
labels:
app: reg
release: loki
spec:
containers:
- name: reg
image: xxxxxx/sre-ops:dev-latest
imagePullPolicy: Always
ports:
- name: reg
containerPort: 9216
resources:
limits:
memory: 500Mi
requests:
cpu: 100m
memory: 128Mi
nodeSelector:
kubernetes.io/hostname: xxxxxxxxxxxx
imagePullSecrets:
- name: xxxx
---
apiVersion: v1
kind: Service
metadata:
name: metrics-clusterip
namespace: labs
labels:
app: reg
release: loki
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '9216'
prometheus.io/scrape: "true"
spec:
type: ClusterIP
selector:
app: reg
release: loki
ports:
- port: 9216
targetPort: reg
protocol: TCP
name: reg
And the ServiceMonitor :
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: reg
namespace: loki
labels:
app: reg
release: loki
spec:
selector:
matchLabels:
app: reg
release: loki
endpoints:
- port: reg
path: /metrics
interval: 15s
namespaceSelector:
matchNames:
- "labs"
I have also tried to pipe the DNS pod logs to a file but I am not sure what I should be looking to get more detail :
kubectl logs --namespace=kube-system coredns-XXXXXX-XXXX >> logs.txt
What am I missing?