I have deployed kube-state-metrics into kube-system namespace and in the same cluster we are having prometheus-operator running I've written the below service monitor file for sending metrics to prometheus but it is not working. Please find the files below.
Servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: kube-state-metrics
labels:
app.kubernetes.io/name: kube-state-metrics
namespace: kube-system
spec:
selector:
matchLabels:
prometheus-scrape: "true"
endpoints:
- port: metrics
path: /metrics
targetPort: 8080
honorLabels: true
scheme: https
tlsConfig:
insecureSkipVerify: true
Prometheus-deploy.yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
annotations:
argocd.argoproj.io/sync-wave: "1"
name: prometheus
labels:
name: prometheus
spec:
serviceAccountName: prometheus
serviceMonitorSelector: {}
serviceMonitorNamespaceSelector:
matchLabels:
prometheus-scrape: "true"
podMonitorSelector: {}
podMonitorNamespaceSelector:
matchLabels:
prometheus-scrape: "true"
resources:
requests:
memory: 400Mi
enableAdminAPI: false
additionalScrapeConfigs:
name: additional-scrape-configs
key: prometheus-additional.yaml
Can any one please help me out regarding this issue.
Thanks.
ServiceMonitor's selector>matchLabels should match with "Service"'s labels. Check if your service has correct label.
Related
I installed kube-prometheus-stack via helm:
kubectl create namespace monitoring && \
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && \
helm repo update && \
helm install -n monitoring prometheus-stack prometheus-community/kube-prometheus-stack
Then proceeded to deploy a fastapi application which has a metrics endpoint that Prometheus is supposed to scrape. /metrics endpoint works fine as seen below:
from starlette_prometheus import metrics, PrometheusMiddleware
from fastapi import FastAPI, Request
app = FastAPI()
# Add Prometheus metrics as middleware
app.add_middleware(PrometheusMiddleware)
app.add_route("/metrics", metrics)
I expected both the target and service discovery to work but only service discovery appears to be working
Prometheus Operator version:
2.36.1
Kubernetes version information:
Client Version: v1.24.2
Kubernetes cluster kind:
Minikube v1.25.2-1
Here are the manifests for the deployed application:
Namespace and deployment
apiVersion: v1
kind: Namespace
metadata:
name: code-detector-demo
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: code-detector
namespace: code-detector-demo
labels:
release: prometheus-stack
spec:
replicas: 1
selector:
matchLabels:
app: code-detector
release: prometheus-stack
template:
metadata:
labels:
app: code-detector
release: prometheus-stack
spec:
containers:
- name: code-detector
image: <MY-IMAGE:TAG>
resources:
limits:
memory: 512Mi
cpu: 1000m
ports:
- containerPort: 8000
Service
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/scrape: "true"
labels:
release: prometheus-stack
name: code-detector-service
namespace: code-detector-demo
spec:
selector:
app: code-detector
release: prometheus-stack
ports:
- name: code-detector-port
port: 8000
targetPort: 8000
ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: code-detector-servicemonitor
# same namespace that Prometheus is running in
namespace: monitoring
labels:
app: code-detector
release: prometheus-stack
spec:
selector:
matchLabels:
app: code-detector
release: prometheus-stack
endpoints:
- path: /metrics
port: code-detector-port
interval: 15s
namespaceSelector:
matchNames:
- code-detector-demo # namespace where the app is running
Here's the labels in service discovery:
Targets pane don't show any picked up scraping job:
What could I be doing wrong?
You have in your servicemonitor:
spec:
selector:
matchLabels:
app: code-detector
release: prometheus-stack
which basicaly means "get all endpoints from services with labels app=code-detector and release=prometheus-stack".
Your service don't have the label app=code-detector.
I am trying to follow this instruction to monitoring my prometheus
https://logiq.ai/scraping-nginx-ingress-controller-metrics-using-helm-prometheus/
anyhow, I got a problem when trying to apply this file configuration
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app: kubernetes-ingress
name: service-monitor
namespace: nginx-ingress
spec:
endpoints:
- interval: 30s
path: /metrics
port: prometheus
namespaceSelector:
matchNames:
- logiq
selector:
matchLabels:
app: kubernetes-ingress
this is the error
error: error when retrieving current configuration of:
Resource: "monitoring.coreos.com/v1, Resource=servicemonitors", GroupVersionKind: "monitoring.coreos.com/v1, Kind=ServiceMonitor"
Name: "", Namespace: "default"
from server for: "servicemonitor.yaml": resource name may not be empty
I thought it was about the CRD, but my monitoring.coreos.com has installed.
thank you in advance
this is my prometheus-kube clusterrole
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
annotations:
meta.helm.sh/release-name: prometheus
meta.helm.sh/release-namespace: ingress
creationTimestamp: "2022-01-17T03:09:49Z"
generation: 1
labels:
app: kube-prometheus-stack-prometheus
app.kubernetes.io/managed-by: Helm
chart: kube-prometheus-stack-10.1.3
heritage: Helm
release: prometheus
name: prometheus-kube-prometheus-prometheus
namespace: ingress
resourceVersion: "2311107"
uid: 48a57afb-2d9a-4f9f-9885-33ca66c59b16
spec:
alerting:
alertmanagers:
- apiVersion: v2
name: prometheus-kube-prometheus-alertmanager
namespace: ingress
pathPrefix: /
port: web
baseImage: quay.io/prometheus/prometheus
enableAdminAPI: false
externalUrl: http://prometheus-kube-prometheus-prometheus.ingress:9090
listenLocal: false
logFormat: logfmt
logLevel: info
paused: false
podMonitorNamespaceSelector: {}
podMonitorSelector:
matchLabels:
release: prometheus
portName: web
probeNamespaceSelector: {}
probeSelector:
matchLabels:
release: prometheus
replicas: 1
retention: 10d
routePrefix: /
ruleNamespaceSelector: {}
ruleSelector:
matchLabels:
app: kube-prometheus-stack
release: prometheus
securityContext:
fsGroup: 2000
runAsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: prometheus-kube-prometheus-prometheus
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
matchLabels:
release: prometheus
version: v2.21.0
For k8s resources, metadata.name is a required field. You must provide the metadata.name in resource YAML before applying it.
In case of metadata.namespace, if you don't provide it, it defaults to default namespace.
I think you have some unwanted leading spaces before name and namespace fields.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: service-monitor
namespace: nginx-ingress
labels:
app: kubernetes-ingress
spec:
endpoints:
- interval: 30s
path: /metrics
port: prometheus
namespaceSelector:
matchNames:
- logiq
selector:
matchLabels:
app: kubernetes-ingress
Update:
In your Prometheus CR, you have serviceMonitorSelector set.
spec:
serviceMonitorSelector:
matchLabels:
release: prometheus
Add these labels to your serviceMonitor CR.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: service-monitor
namespace: nginx-ingress
labels:
app: kubernetes-ingress
release: prometheus
Or, you can also update serviceMonitorSelector from the Prometheus CR side.
I configure all of the following configurations but the request_per_second does not appear when I type the command
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
In the node.js that should be monitored I installed prom-client, I tested the /metrics and it's working very well and the metric "resquest_count" is the object it returns
Here are the important parts of that node code
(...)
const counter = new client.Counter({
name: 'request_count',
help: 'The total number of processed requests'
});
(...)
router.get('/metrics', async (req, res) => {
res.set('Content-Type', client.register.contentType)
res.end(await client.register.metrics())
})
This is my service monitor configuration
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: un1qnx-validation-service-monitor-node
namespace: default
labels:
app: node-request-persistence
release: prometheus
spec:
selector:
matchLabels:
app: node-request-persistence
endpoints:
- interval: 5s
path: /metrics
port: "80"
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
namespaceSelector:
matchNames:
- un1qnx-aks-development
This the node-request-persistence configuration
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: node-request-persistence
namespace: un1qnx-aks-development
name: node-request-persistence
spec:
selector:
matchLabels:
app: node-request-persistence
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: /metrics
prometheus.io/port: "80"
labels:
app: node-request-persistence
spec:
containers:
- name: node-request-persistence
image: node-request-persistence
imagePullPolicy: Always # IfNotPresent
resources:
requests:
memory: "200Mi" # Gi
cpu: "100m"
limits:
memory: "400Mi"
cpu: "500m"
ports:
- name: node-port
containerPort: 80
This is the prometheus adapter
prometheus:
url: http://prometheus-server.default.svc.cluster.local
port: 9090
rules:
custom:
- seriesQuery: 'request_count{namespace!="", pod!=""}'
resources:
overrides:
namespace: {resource: "namespace"}
pod: {resource: "pod"}
name:
as: "request_per_second"
metricsQuery: "round(avg(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>))"
This is the hpa
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: un1qnx-validation-service-hpa-angle
namespace: un1qnx-aks-development
spec:
minReplicas: 1
maxReplicas: 10
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: un1qnx-validation-service-angle
metrics:
- type: Pods
pods:
metric:
name: request_per_second
target:
type: AverageValue
averageValue: "5"
The command
kubectl get hpa -n un1qnx-aks-development
results in "unknown/5"
Also, the command
kubectl get --raw "http://prometheus-server.default.svc.cluster.local:9090/api/v1/series"
Results in
Error from server (NotFound): the server could not find the requested resource
I think it should return some value about the collected metrics... I think that the problem is from the service monitor, but I am new to this
As you noticed I am trying to scale a deployment based on another deployment pods, don't know if there is a problem there
I appreciate an answer, because this is for my thesis
kubernetes - version 1.19.9
Prometheus - chart prometheus-14.2.1 app version 2.26.0
Prometheus Adapter - chart 2.14.2 app version 0.8.4
And all where installed using helm
After some time I found the problems and I changed the following
Changed the port on the prometheus adapter, the time on the query and the names of the resource override. But to know the names of the resources override you need to port forward to the prometheus server and check the labels on the targets page of the app that you are monitoring.
prometheus:
url: http://prometheus-server.default.svc.cluster.local
port: 80
rules:
custom:
- seriesQuery: 'request_count{kubernetes_namespace!="", kubernetes_pod_name!=""}'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
name:
matches: "request_count"
as: "request_count"
metricsQuery: "round(avg(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>))"
I also added annotations on the deployment yaml
spec:
selector:
matchLabels:
app: node-request-persistence
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: /metrics
prometheus.io/port: "80"
labels:
I am following the documentation to create services operator. I am not sure why I cannot access the Prometheus services.
My apps.yml:
kind: Service
apiVersion: v1
metadata:
name: sms-config-service
labels:
app: sms-config-service
spec:
type: NodePort
selector:
app: sms-config-service
ports:
- port: 8080
targetPort: 8080
name: http
My ServiceMonitor yml:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app: servicemonitor-sms-services
name: servicemonitor-sms-config-services
namespace: metrics
spec:
selector:
matchLabels:
app: sms-config-service
endpoints:
- port: http
Prometheus yml:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
spec:
serviceAccountName: prometheus
serviceMonitorSelector:
matchLabels:
app: servicemonitor-sms-services
resources:
requests:
memory: 800Mi
enableAdminAPI: true
Prometheus config yml:
apiVersion: v1
kind: Service
metadata:
name: prometheus
spec:
type: NodePort
ports:
- name: web
nodePort: 30900
port: 9090
protocol: TCP
targetPort: web
selector:
prometheus: prometheus
When I access the url below, browser shows "unable to connect". I am not sure where I did wrong? Should I set up a deployment for the Prometheus?
$ minikube service prometheus --url
http://192.168.64.3:30900
Update:
I have the prometheus pod running on NodePort 32676.
Should I change the Prometheus config yml to fix the issue?
I find the issue is I don't have serviceAccountName created.
I have a Redis cluster and Redis-exporter in two separate deployments in the same namespace of a Kubernetes cluster. I am using Prometheus operator to monitor the cluster, but I can not find a way to set up the exporter and the operator. I have set up a service targeting the Redis exporter(check below) and a ServiceMonitor(also below). If I port forward to the Redis exporter service I can see the metrics. Also, the Redis exporter logs do not show issues.
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: foo
name: redis-exporter
labels:
app: redis-exporter
spec:
replicas: 1
selector:
matchLabels:
app: redis-exporter
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9121"
labels:
app: redis-exporter
spec:
containers:
- name: redis-exporter
image: oliver006/redis_exporter:latest
resources:
requests:
cpu: 100m
memory: 100Mi
env:
- name: REDIS_ADDR
value: redis-cluster.foo.svc:6379
ports:
- containerPort: 9121
My service and servicemonitor
kind: Service
metadata:
name: redis-external-exporter
namespace: foo
labels:
app: redis
k8s-app: redis-ext
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: "9121"
spec:
ports:
- name: redis-ext
port: 9121
protocol: TCP
targetPort: 9121
selector:
app: redis-exporter
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: redis-external-exporter
namespace: bi-infra
labels:
app: redis-external-exporter
k8s-app: redis-monitor
spec:
jobLabel: app
selector:
matchLabels:
app: redis-ext
namespaceSelector:
matchNames:
- foo
endpoints:
- port: redis-ext
interval: 30s
honorLabels: true
If I switch to a sidecar Redis exporter next to the Redis cluster all is working properly. Has anyone faced such issue?
I was missing spec.endpoints.path on the ServiceMonitor
Here is an example manifest from adding new scraping targets and troubleshooting tutorial.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: monitoring-pili
namespace: monitoring
labels:
app: pili-service-monitor
spec:
selector:
matchLabels:
# Target app service
app: pili
endpoints:
- interval: 15s
path: /metrics <---
port: uwsgi
namespaceSelector:
matchNames:
- default