prometheus do not pull data from treafik service in kubernetes cluster - kubernetes

Question

I am using prometheus(quay.azk8s.cn/prometheus/prometheus:v2.15.2) to monitor traefik 2.1.6 in kubernetes monitoring namespace,now I am make traefik expose metics and I could using curl command to get config from http://traefik-ip:8080/metrics,but prometheus do not pull data.I already added annotation to treafik service yaml in kubernetes kube-system namespace,this is the prometheus service config:
{
"kind": "StatefulSet",
"apiVersion": "apps/v1beta2",
"metadata": {
"name": "prometheus-k8s",
"namespace": "monitoring",
"selfLink": "/apis/apps/v1beta2/namespaces/monitoring/statefulsets/prometheus-k8s",
"uid": "4190d704-aa3b-40da-ab99-bac3cb10f186",
"resourceVersion": "18281285",
"generation": 7,
"creationTimestamp": "2020-03-04T16:31:01Z",
"labels": {
"prometheus": "k8s"
},
"annotations": {
"prometheus-operator-input-hash": "4895445337133709592"
},
"ownerReferences": [
{
"apiVersion": "monitoring.coreos.com/v1",
"kind": "Prometheus",
"name": "k8s",
"uid": "ddf7e48d-f982-4881-9312-0d50466870a9",
"controller": true,
"blockOwnerDeletion": true
}
]
},
"spec": {
"replicas": 2,
"selector": {
"matchLabels": {
"app": "prometheus",
"prometheus": "k8s"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "prometheus",
"prometheus": "k8s"
}
},
"spec": {
"volumes": [
{
"name": "config",
"secret": {
"secretName": "prometheus-k8s",
"defaultMode": 420
}
},
{
"name": "tls-assets",
"secret": {
"secretName": "prometheus-k8s-tls-assets",
"defaultMode": 420
}
},
{
"name": "config-out",
"emptyDir": {}
},
{
"name": "prometheus-k8s-rulefiles-0",
"configMap": {
"name": "prometheus-k8s-rulefiles-0",
"defaultMode": 420
}
},
{
"name": "prometheus-k8s-db",
"emptyDir": {}
}
],
"containers": [
{
"name": "prometheus",
"image": "quay.azk8s.cn/prometheus/prometheus:v2.15.2",
"args": [
"--web.console.templates=/etc/prometheus/consoles",
"--web.console.libraries=/etc/prometheus/console_libraries",
"--config.file=/etc/prometheus/config_out/prometheus.env.yaml",
"--storage.tsdb.path=/prometheus",
"--storage.tsdb.retention.time=24h",
"--web.enable-lifecycle",
"--storage.tsdb.no-lockfile",
"--web.route-prefix=/"
],
"ports": [
{
"name": "web",
"containerPort": 9090,
"protocol": "TCP"
}
],
"resources": {
"requests": {
"memory": "400Mi"
}
},
"volumeMounts": [
{
"name": "config-out",
"readOnly": true,
"mountPath": "/etc/prometheus/config_out"
},
{
"name": "tls-assets",
"readOnly": true,
"mountPath": "/etc/prometheus/certs"
},
{
"name": "prometheus-k8s-db",
"mountPath": "/prometheus"
},
{
"name": "prometheus-k8s-rulefiles-0",
"mountPath": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0"
}
],
"livenessProbe": {
"httpGet": {
"path": "/-/healthy",
"port": "web",
"scheme": "HTTP"
},
"timeoutSeconds": 3,
"periodSeconds": 5,
"successThreshold": 1,
"failureThreshold": 6
},
"readinessProbe": {
"httpGet": {
"path": "/-/ready",
"port": "web",
"scheme": "HTTP"
},
"timeoutSeconds": 3,
"periodSeconds": 5,
"successThreshold": 1,
"failureThreshold": 120
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "FallbackToLogsOnError",
"imagePullPolicy": "IfNotPresent"
},
{
"name": "prometheus-config-reloader",
"image": "quay.azk8s.cn/coreos/prometheus-config-reloader:v0.37.0",
"command": [
"/bin/prometheus-config-reloader"
],
"args": [
"--log-format=logfmt",
"--reload-url=http://localhost:9090/-/reload",
"--config-file=/etc/prometheus/config/prometheus.yaml.gz",
"--config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml"
],
"env": [
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.name"
}
}
}
],
"resources": {
"limits": {
"cpu": "100m",
"memory": "25Mi"
},
"requests": {
"cpu": "100m",
"memory": "25Mi"
}
},
"volumeMounts": [
{
"name": "config",
"mountPath": "/etc/prometheus/config"
},
{
"name": "config-out",
"mountPath": "/etc/prometheus/config_out"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "FallbackToLogsOnError",
"imagePullPolicy": "IfNotPresent"
},
{
"name": "rules-configmap-reloader",
"image": "jimmidyson/configmap-reload:v0.3.0",
"args": [
"--webhook-url=http://localhost:9090/-/reload",
"--volume-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0"
],
"resources": {
"limits": {
"cpu": "100m",
"memory": "25Mi"
},
"requests": {
"cpu": "100m",
"memory": "25Mi"
}
},
"volumeMounts": [
{
"name": "prometheus-k8s-rulefiles-0",
"mountPath": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "FallbackToLogsOnError",
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 600,
"dnsPolicy": "ClusterFirst",
"nodeSelector": {
"kubernetes.io/os": "linux"
},
"serviceAccountName": "prometheus-k8s",
"serviceAccount": "prometheus-k8s",
"securityContext": {
"runAsUser": 1000,
"runAsNonRoot": true,
"fsGroup": 2000
},
"schedulerName": "default-scheduler"
}
},
"serviceName": "prometheus-operated",
"podManagementPolicy": "Parallel",
"updateStrategy": {
"type": "RollingUpdate"
},
"revisionHistoryLimit": 10
},
"status": {
"observedGeneration": 7,
"replicas": 2,
"readyReplicas": 2,
"currentReplicas": 2,
"updatedReplicas": 2,
"currentRevision": "prometheus-k8s-6f76f69569",
"updateRevision": "prometheus-k8s-6f76f69569",
"collisionCount": 0
}
}
this is the config:
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "traefik",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/services/traefik",
"uid": "b2695279-2467-4480-aab5-a720a43951c1",
"resourceVersion": "18280221",
"creationTimestamp": "2020-01-29T10:26:34Z",
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{\"prometheus.io/port\":\"8080\",\"prometheus.io/scrape\":\"true\"},\"name\":\"traefik\",\"namespace\":\"kube-system\"},\"spec\":{\"ports\":[{\"name\":\"web\",\"port\":80},{\"name\":\"websecure\",\"port\":443},{\"name\":\"metrics\",\"port\":8080}],\"selector\":{\"app\":\"traefik\"}}}\n",
"prometheus.io/port": "8080",
"prometheus.io/scrape": "true"
}
},
"spec": {
"ports": [
{
"name": "web",
"protocol": "TCP",
"port": 80,
"targetPort": 80
},
{
"name": "websecure",
"protocol": "TCP",
"port": 443,
"targetPort": 443
},
{
"name": "metrics",
"protocol": "TCP",
"port": 8080,
"targetPort": 8080
}
],
"selector": {
"app": "traefik"
},
"clusterIP": "10.254.169.66",
"type": "ClusterIP",
"sessionAffinity": "None"
},
"status": {
"loadBalancer": {}
}
}
I read some docs give tips that I should config the pull task in kubernetes(v1.15.2) config map like this:
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: kube-ops
data:
prometheus.yml: |
global:
scrape_interval: 30s
scrape_timeout: 30s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'traefik'
static_configs:
- targets: ['traefik-ingress-service.kube-system.svc.cluster.local:8080']
and I add the config in my prometheus yaml.Am I missing something? I do this steps:
expose traefik metrics url(success)
add annotation to my traefik service(success)
but the metrics data not collected,I am stucking in this problem for 2 days,what should I do to make it work? This is my prometheus's service discovery dashboard:
But when I query data from prometheus I found nothing.
http_requests_total{job="traefik"}

Answer 1

Pay attention new version(v2.1.6) of treafik's request query to check pull data is:
traefik_entrypoint_requests_total{job="traefik"}
you can see the prometheus pulled data successful.