I'm running a 3 node k8s cluster where I have a prometheus instance that scrape my services metrics using object like serviceMonitor. In addition to my cluster services metrics, I would like to get metrics from an ambari-server. I know that it is possible to import ambari metrics as a datasource in grafana but I haven't seen anything about Prometheus. My goal would be to have those metrics in Prometheus to set alerts with AlertManager. I also saw that it is possible to write a go program that would expose metrics in the Prometheus format, but I preferred to ask to the community before trying stuffs.. :-)
If anyone had this issue, i would be happy if you cna help !
You can, Prometheus allows to scrape metrics exposed by external services: obviously they need to respect Prometheus format but I saw that it's possible.
You have to create static endpoints and related service:
apiVersion: v1
kind: Endpoints
metadata:
name: ambari-metrics
labels:
app: ambari
subsets:
- addresses:
- ip: "ambari-external-ip"
ports:
- name: metrics
port: 9100
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: ambari-metrics-svc
namespace: your-monitoring-namespace
labels:
app: ambari
spec:
type: ExternalName
externalName: ambari-external-ip
clusterIP: ""
ports:
- name: metrics
port: 9100
protocol: TCP
targetPort: 9100
And finally, the ServiceMonitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: ambari-metrics-sm
labels:
app: ambari
prometheus: kube-prometheus
spec:
selector:
matchLabels:
app: ambari
namespaceSelector:
matchNames:
your-monitoring-namespace
endpoints:
- port: metrics
interval: 30s
Obviously, YMMV and can find a detailed guide on this article
Related
I'm trying to make Prometheus find metrics from RabbitMQ (and a few other services, but the logic is the same).
My current configuration is:
# RabbitMQ Service
# This live in the `default` namespace
kind: Service
apiVersion: v1
metadata:
name: rabbit-metrics-service
labels:
name: rabbit-metrics-service
spec:
ports:
- protocol: TCP
port: 15692
targetPort: 15692
selector:
# This selects the deployment and it works
app: rabbitmq
I then created a ServiceMonitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
# The name of the service monitor
name: rabbit-monitor
# The namespace it will be in
namespace: kube-prometheus-stack
labels:
# How to find this service monitor
# The name I should use in `serviceMonitorSelector`
name: rabbit-monitor
spec:
endpoints:
- interval: 5s
port: metrics
path: /metrics
# The namespace of origin service
namespaceSelector:
matchNames:
- default
selector:
matchLabels:
# Where the monitor will attach to
name: rabbit-metrics-service
kube-prometheus-stack has the following values.yml
# values.yml
prometheusSpec:
serviceMonitorSelector:
matchLabels:
name:
- rabbit-monitor
So from what I understand is: in the metadata/label section I define a labelKey/labelValue pair, and then reference this pair on the selector/matchLabels. I then add a custom serviceMonitorSelector that will match N labels. If it finds the labels, Prometheus should discover the ServiceMonitor, and hence, the metrics endpoint, and start scraping. But I guess there's something wrong with this logic. I tried a few other variations of this as well, but no success.
Any ideas on what I might be doing wrong?
Documentation usually uses the same name everywhere, so I can quite understand where exactly should that name come from, since I tend to prefer to add -service, -deployment suffixes to the resources to be able to easily identify them later. I already add RabbitMQ prometheus plugin, and the endpoint seems to be working fine.
I am following the documentation to create services operator. I am not sure why I cannot access the Prometheus services.
My apps.yml:
kind: Service
apiVersion: v1
metadata:
name: sms-config-service
labels:
app: sms-config-service
spec:
type: NodePort
selector:
app: sms-config-service
ports:
- port: 8080
targetPort: 8080
name: http
My ServiceMonitor yml:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app: servicemonitor-sms-services
name: servicemonitor-sms-config-services
namespace: metrics
spec:
selector:
matchLabels:
app: sms-config-service
endpoints:
- port: http
Prometheus yml:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
spec:
serviceAccountName: prometheus
serviceMonitorSelector:
matchLabels:
app: servicemonitor-sms-services
resources:
requests:
memory: 800Mi
enableAdminAPI: true
Prometheus config yml:
apiVersion: v1
kind: Service
metadata:
name: prometheus
spec:
type: NodePort
ports:
- name: web
nodePort: 30900
port: 9090
protocol: TCP
targetPort: web
selector:
prometheus: prometheus
When I access the url below, browser shows "unable to connect". I am not sure where I did wrong? Should I set up a deployment for the Prometheus?
$ minikube service prometheus --url
http://192.168.64.3:30900
Update:
I have the prometheus pod running on NodePort 32676.
Should I change the Prometheus config yml to fix the issue?
I find the issue is I don't have serviceAccountName created.
I have a service running in a k8s cluster, which I want to monitor using Prometheus Operator. The service has a /metrics endpoint, which returns simple data like:
myapp_first_queue_length 12
myapp_first_queue_processing 2
myapp_first_queue_pending 10
myapp_second_queue_length 4
myapp_second_queue_processing 4
myapp_second_queue_pending 0
The API runs in multiple pods, behind a basic Service object:
apiVersion: v1
kind: Service
metadata:
name: myapp-api
labels:
app: myapp-api
spec:
ports:
- port: 80
name: myapp-api
targetPort: 80
selector:
app: myapp-api
I've installed Prometheus using kube-prometheus, and added a ServiceMonitor object:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: myapp-api
labels:
app: myapp-api
spec:
selector:
matchLabels:
app: myapp-api
endpoints:
- port: myapp-api
path: /api/metrics
interval: 10s
Prometheus discovers all the pods running instances of the API, and I can query those metrics from the Prometheus graph. So far so good.
The issue is, those metrics are aggregate - each API instance/pod doesn't have its own queue, so there's no reason to collect those values from every instance. In fact it seems to invite confusion - if Prometheus collects the same value from 10 pods, it looks like the total value is 10x what it really is, unless you know to apply something like avg.
Is there a way to either tell Prometheus "this value is already aggregate and should always be presented as such" or better yet, tell Prometheus to just scrape the values once via the internal load balancer for that service, rather than hitting each pod?
edit
The actual API is just a simple Deployment object:
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-api
labels:
app: myapp-api
spec:
replicas: 2
selector:
matchLabels:
app: myapp-api
template:
metadata:
labels:
app: myapp-api
spec:
imagePullSecrets:
- name: mysecret
containers:
- name: myapp-api
image: myregistry/myapp:2.0
ports:
- containerPort: 80
volumeMounts:
- name: config
mountPath: "app/config.yaml"
subPath: config.yaml
volumes:
- name: config
configMap:
name: myapp-api-config
In your case to avoid metrics aggregation you can use, as already mentioned in your post, avg() operator to or PodMonitor instead of ServiceMonitor.
The PodMonitor custom resource definition (CRD) allows to
declaratively define how a dynamic set of pods should be monitored.
Which pods are selected to be monitored with the desired configuration
is defined using label selections.
This way it will scrape the metrics from the specified pod only.
Prometheus Operator developers are kindly working (as of Jan 2023) on a generic ScrapeConfig CRD that is designed to solve exactly the use case you describe: https://github.com/prometheus-operator/prometheus-operator/issues/2787
In the meantime, you can use the "additional scrape config" facility of Prometheus Operator to setup a custom scrape target.
The idea is that the configured scrape target will be hit only once per scrape period and the service DNS will load-balance the request to one of the N pods behind the service, thus avoiding duplicate metrics.
In particular, you can override the kube-prometheus-stack Helm values as follows:
prometheus:
prometheusSpec:
additionalScrapeConfigs:
- job_name: 'myapp-api-aggregates':
metrics_path: '/api/metrics'
scheme: 'http'
static_configs:
- targets: ['myapp-api:80']
My goal is to monitor services with Prometheus, so I was following a guide located at:
https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md
I am relatively new to all of this, so please forgive my naiveness. I tried looking into the error, but all the answers were convoluted. I have no idea where to start on the debug process (perhaps look into the YAMLs?)
I wanted to monitor a custom Service. So, I deployed a service.yaml of the following into a custom namespace (t):
kind: Service
apiVersion: v1
metadata:
namespace: t
name: example-service-test
labels:
app: example-service-test
spec:
selector:
app: example-service-test
type: NodePort
ports:
- name: http
nodePort: 30901
port: 8080
protocol: TCP
targetPort: http
---
apiVersion: v1
kind: Pod
metadata:
name: example-service-test
namespace: t
labels:
app: example-service-test
spec:
containers:
- name: example-service-test
image: python:2.7
imagePullPolicy: IfNotPresent
command: ["/bin/bash"]
args: ["-c", "echo \"<p>This is POD1 $(hostname)</p>\" > index.html; python -m SimpleHTTPServer 8080"]
ports:
- name: http
containerPort: 8080
And deployed a service monitor into the namespace:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: example-service-test
labels:
team: frontendtest1
namespace: t
spec:
selector:
matchLabels:
app: example-service-test
endpoints:
- port: http
So far, the service monitor is detecting the service, as shown:
Prometheus Service Discovery.
However, there is an error with obtaining the metrics from the service: Prometheus Targets.
From what I know, prometheus isn't able to access the /metrics on the sample service - in that case, do I need to expose the metrics? If so, could I get a step by step guide solution to how to expose metrics? If not, what route should I take?
I'm afraid you could miss the key thing from the tutorial you're following on CoreOS website, about how a metrics from an app are getting to Prometheus:
First, deploy three instances of a simple example application, which
listens and exposes metrics on port 8080
Yes, your application (website) listens on port 8080, but does not expose any metrics on '/metrics' endpoint in the known to Prometheus format.
You can verify about what kind of metrics I'm talking about by hiting the endpoint from inside of Pod/Conatiner where it's hosted.
kubectl exec -it $(kubectl get po -l app=example-app -o jsonpath='{.items[0].metadata.name}') -c example-app -- curl localhost:8080/metrics
You should see similar output to this one:
# HELP codelab_api_http_requests_in_progress The current number of API HTTP requests in progress.
# TYPE codelab_api_http_requests_in_progress gauge
codelab_api_http_requests_in_progress 1
# HELP codelab_api_request_duration_seconds A histogram of the API HTTP request durations in seconds.
# TYPE codelab_api_request_duration_seconds histogram
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0001"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00015000000000000001"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00022500000000000002"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0003375"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00050625"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.000759375"} 0
Please read more here on ways of exposing metrics.
Currently, I have working K8s API pods in a K8s service that connects to a K8s Redis service, with K8s pods of it's own. The problem is, I am using NodePort meaning BOTH are exposed to the public. I only want the API accessable to the public. The issue is that if I make the Redis service not public, the API can't see it. Is there a way to connect two Services without exposing one to the public?
This is my API service yaml:
apiVersion: v1
kind: Service
metadata:
name: app-api-svc
spec:
selector:
app: app-api
tier: api
ports:
- protocol: TCP
port: 5000
nodePort: 30400
type: NodePort
And this is my Redis service yaml:
apiVersion: v1
kind: Service
metadata:
name: app-api-redis-svc
spec:
selector:
app: app-api-redis
tier: celery_broker
ports:
- protocol: TCP
port: 6379
nodePort: 30537
type: NodePort
First, configure the Redis service as a ClusterIP service. It will be private, visible only for other services. This is could be done removing the line with the option type.
apiVersion: v1
kind: Service
metadata:
name: app-api-redis-svc
spec:
selector:
app: app-api-redis
tier: celery_broker
ports:
- protocol: TCP
port: 6379
targetPort: [the port exposed by the Redis pod]
Finally, when you configure the API to reach Redis, the address should be app-api-redis-svc:6379
And that's all. I have a lot of services communicating each other in this way. If this doesn't work for you, let me know in the comments.
I'm going to try to take the best from all answers and my own research and make a short guide that I hope you will find helpful:
1. Test connectivity
Connect to a different pod, eg ruby pod:
kubectl exec -it some-pod-name -- /bin/sh
Verify it can ping to the service in question:
ping redis
Can it connect to the port? (I found telnet did not work for this)
nc -zv redis 6379
2. Verify your service selectors are correct
If your service config looks like this:
kind: Service
apiVersion: v1
metadata:
name: redis
labels:
app: redis
role: master
tier: backend
spec:
ports:
- port: 6379
targetPort: 6379
selector:
app: redis
role: master
tier: backend
verify those selectors are also set on your pods?
get pods --selector=app=redis,role=master,tier=backend
Confirm that your service is tied to your pods by running:
$> describe service redis
Name: redis
Namespace: default
Labels: app=redis
role=master
tier=backend
Annotations: <none>
Selector: app=redis,role=master,tier=backend
Type: ClusterIP
IP: 10.47.250.121
Port: <unset> 6379/TCP
Endpoints: 10.44.0.16:6379
Session Affinity: None
Events: <none>
check the Endpoints: field and confirm it's not blank
More info can be found at:
https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#my-service-is-missing-endpoints
I'm not sure about redis, but I have a similar application. I have a Java web application running as a pod that is exposed to the outside world through a nodePort. I have a mongodb container running as a pod.
In the webapp deployment specifications, I map it to the mongodb service through its name by passing the service name as parameter, I have pasted the specification below. You can modify accordingly.There should be a similar mapping parameter in Redis also where you would have to use the service name which is "mongoservice" in my case.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: empappdepl
labels:
name: empapp
spec:
replicas: 1
template:
metadata:
labels:
name: empapp
spec:
containers:
- resources:
limits:
cpu: 0.2
image: registryip:5000/employee:1
imagePullPolicy: IfNotPresent
name: wsemp
ports:
- containerPort: 8080
name: wsemp
command: ["java","-Dspring.data.mongodb.uri=mongodb://mongoservice/microservices", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
imagePullSecrets:
- name: myregistrykey
---
apiVersion: v1
kind: Service
metadata:
labels:
name: empwhatever
name: empservice
spec:
ports:
- port: 8080
targetPort: 8080
protocol: TCP
name: http
nodePort: 30062
type: NodePort
selector:
name: empapp
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: mongodbdepl
labels:
name: mongodb
spec:
replicas: 1
template:
metadata:
labels:
name: mongodb
spec:
containers:
- resources:
limits:
cpu: 0.3
image: mongo
imagePullPolicy: IfNotPresent
name: mongodb
ports:
- containerPort: 27017
---
apiVersion: v1
kind: Service
metadata:
labels:
name: mongowhatever
name: mongoservice
spec:
ports:
- port: 27017
targetPort: 27017
protocol: TCP
selector:
name: mongodb
Note that the mongodb service doesnt need to be exposed as a NodePort.
Kubernetes enables inter service communication by allowing services communicate with other services using their service name.
In your scenario, redis service should be accessible from other services on
http://app-api-redis-svc.default:6379. Here default is the namespace under which your service is running.
This internally routes your requests to your redis pod running on the target container port
Checkout this link for different modes of service discovery options provided by kubernetes
Hope it helps