Issue with monitoring custom service on prometheus in kubernetes namespace - service

Question

My goal is to monitor services with Prometheus, so I was following a guide located at:
https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md
I am relatively new to all of this, so please forgive my naiveness. I tried looking into the error, but all the answers were convoluted. I have no idea where to start on the debug process (perhaps look into the YAMLs?)
I wanted to monitor a custom Service. So, I deployed a service.yaml of the following into a custom namespace (t):
kind: Service
apiVersion: v1
metadata:
namespace: t
name: example-service-test
labels:
app: example-service-test
spec:
selector:
app: example-service-test
type: NodePort
ports:
- name: http
nodePort: 30901
port: 8080
protocol: TCP
targetPort: http
---
apiVersion: v1
kind: Pod
metadata:
name: example-service-test
namespace: t
labels:
app: example-service-test
spec:
containers:
- name: example-service-test
image: python:2.7
imagePullPolicy: IfNotPresent
command: ["/bin/bash"]
args: ["-c", "echo \"<p>This is POD1 $(hostname)</p>\" > index.html; python -m SimpleHTTPServer 8080"]
ports:
- name: http
containerPort: 8080
And deployed a service monitor into the namespace:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: example-service-test
labels:
team: frontendtest1
namespace: t
spec:
selector:
matchLabels:
app: example-service-test
endpoints:
- port: http
So far, the service monitor is detecting the service, as shown:
Prometheus Service Discovery.
However, there is an error with obtaining the metrics from the service: Prometheus Targets.
From what I know, prometheus isn't able to access the /metrics on the sample service - in that case, do I need to expose the metrics? If so, could I get a step by step guide solution to how to expose metrics? If not, what route should I take?

Answer 1

I'm afraid you could miss the key thing from the tutorial you're following on CoreOS website, about how a metrics from an app are getting to Prometheus:
First, deploy three instances of a simple example application, which
listens and exposes metrics on port 8080
Yes, your application (website) listens on port 8080, but does not expose any metrics on '/metrics' endpoint in the known to Prometheus format.
You can verify about what kind of metrics I'm talking about by hiting the endpoint from inside of Pod/Conatiner where it's hosted.
kubectl exec -it $(kubectl get po -l app=example-app -o jsonpath='{.items[0].metadata.name}') -c example-app -- curl localhost:8080/metrics
You should see similar output to this one:
# HELP codelab_api_http_requests_in_progress The current number of API HTTP requests in progress.
# TYPE codelab_api_http_requests_in_progress gauge
codelab_api_http_requests_in_progress 1
# HELP codelab_api_request_duration_seconds A histogram of the API HTTP request durations in seconds.
# TYPE codelab_api_request_duration_seconds histogram
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0001"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00015000000000000001"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00022500000000000002"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0003375"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00050625"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.000759375"} 0
Please read more here on ways of exposing metrics.