I need Prometheus to scrape several mongodb exporters one after another in order to compute a valid replication lag. However, the targets are scraped with a difference of several dozen seconds between them, which makes replication lag impossible to compute.
The job yaml is below:
- job_name: mongo-storage
honor_timestamps: true
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
static_configs:
- targets:
- mongo-1a-exporter.monitor:9216
- mongo-2a-exporter.monitor:9216
- mongo-3a-exporter.monitor:9216
- mongos-exporter.monitor:9216
- mongo-1b-exporter.monitor:9216
- mongo-2b-exporter.monitor:9216
- mongo-3b-exporter.monitor:9216
labels:
cluster: mongo-storage
This isn't possible, Prometheus makes no guarantees about the phase of scrapes or rule evaluations. Nor is this something you should depend upon, as it'd be very fragile.
I'd aim for knowing the lag within a scrape interval, rather than trying to get it perfect. You generally care if replication is completely broken, rather than if it's slightly delayed. A heartbeat job could also help.
This isn't possible with Prometheus... normally.
However it might be possible to exploit the prometheus/pushgateway to achieve what you want. My thinking is that you write a script/tool to scrape the mongo exporters in a synchronised way, threads/forks/whatever, and then push those metrics into a prometheus/pushgateway instance.
Then configure prometheus to scrape the prometheus/pushgateway instead of the mongo exporters, and since all the metrics are in the one endpoint they will hopefully always be in sync and avoid any lag regarding being up to date.
Hope this helps.
Related
I am getting metrics exposed by kube-state-metrics by querying Prometheus-server but the issue is I am getting duplicate metrics with difference only in the job field. . I am doing query such as :
curl 'http://10.101.202.25:80/api/v1/query?query=kube_pod_status_phase'| jq
The only difference is coming the job field. Metrics coming when querying Prometheus-Server
All pods running in the cluster: https://imgur.com/PKIc3ug
Any help is appreciated.
Thank You
prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
# - "first.rules"
# - "second.rules"
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
You are running (or at least ingesting) two copies of kube-state-metrics. Probably one you installed and configured yourself and another from something like kube-prometheus-stack?
I was able to get what I wanted eventually. What I did was to remove the scraping config of prometheus-kube-state-metrics from the value.yml and defining that in the config file i.e. prometheus.yml. For now it's working fine. Thank You #SYN and #coderanger for the help.
Is there a way to check history of different kubernetes resource type? It can be an additional plugin.
Use case:
For example currently we have a statefulset on a 5 node cluster:
name: X
replica:3
resources:
memory:
limit: 2Gi
request: 1Gi
currently replica 1 is on node_1, replica 2 is on node_2, replica 3 is on node_3.
I am curious about the state of the resources for any given time.
Let's say I want to check that one month ago what were the resource limits. How many replica we had and on which node were those allocated.
To directly answer your question - you can't do that using oob functionality.
You need any existing monitoring solution for Kubernetes that is capable of exposing metrics you need. What comes to my mind:
kube-state-metrics server + promethues
kube-state-metrics server + metricbeat
For instance kube-state-metrics server exposes such metric like Pod's container resource limit (kube_pod_container_resource_limits), and metricbeat digest that metrics and helps to visualize it.
Collecting Kubernetes state metrics and events
A single instance is deployed to collect Kubernetes metrics. It is
integrated with the kube-state-metrics API to monitor state changes of
objects managed by Kubernetes. This is the section of the config that
defines state_metrics collection.
$HOME/k8s-o11y-workshop/Metricbeat/Metricbeat.yml:
kubernetes.yml: |-
- module: kubernetes
metricsets:
- state_node
- state_deployment
- state_replicaset
- state_pod
- state_container
# Uncomment this to get k8s events:
- event
period: 10s
host: ${NODE_NAME}
hosts: ["kube-state-metrics:8080"]
I have 3 kubernetes clusters (prod, test, monitoring). Iam new to prometheus so i have tested it by installing it in my test environment with the helm chart:
# https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack
But if i want to have metrics from the prod and test clusters, i have to repeat the same installation of the helm and each "kube-prometheus-stack" would be standalone in its own cluster. It is not ideal at all. Iam trying to find a way to have a single prometheus/grafana which would federate/agregate the metrics from each cluster's prometheus server.
I found this link, saying about prometheus federation:
https://prometheus.io/docs/prometheus/latest/federation/
If install the helm chart "kube-prometheus-stack" and get rid of grafana on the 2 other cluster, how can i make the 3rd "kube-prometheus-stack", on the 3rd cluster, scrapes metrics from the 2 other ones?
thanks
You have to modify configuration for prometheus federate so it can scrape metrics from other clusters as described in documentation:
scrape_configs:
- job_name: 'federate'
scrape_interval: 15s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job="prometheus"}'
- '{__name__=~"job:.*"}'
static_configs:
- targets:
- 'source-prometheus-1:9090'
- 'source-prometheus-2:9090'
- 'source-prometheus-3:9090'
params field checks for jobs to scrape metrics from. In this particular example
It will scrape any series with the label job="prometheus" or a metric name starting
with job: from the Prometheus servers at
source-prometheus-{1,2,3}:9090
You can check following articles to give you more insight of prometheus federation:
Monitoring Kubernetes with Prometheus - outside the cluster!
Prometheus federation in Kubernetes
Monitoring multiple federated clusters with Prometheus - the secure way
Monitoring a Multi-Cluster Environment Using Prometheus Federation and Grafana
You have few options here:
Option 1:
You can achieve this buy having vmagent or grafana-agent in prod and test clusters and configure remote write on them to your monitoring cluster.
But in this case you will need to install kube-state-metrics and node-exporter separately into prod and test cluster.
Also it's important to add extra label for a cluster name (or any unique identifier) before sending metrics to remote write, to make sure that recording rules from "kube-prometheus-stack" are working correctly
diagram
Option 2:
You can install victoria-metrics-k8s-stack chart. It has similar functionality as kube-prometheus-stack - also installs bunch of components recording rules and dashboards.
With this case you install victoria-metrics-k8s-stack in every cluster, but with different values.
For monitoring cluster you can use default values, with
grafana:
sidecar:
dashboards:
multicluster: true
and proper configured ingress for vmsingle
For prod and test cluster you need to disable bunch of components
defaultRules:
create: false
vmsingle:
enabled: false
alertmanager:
enabled: false
vmalert:
enabled: false
vmagent:
spec:
remoteWrite:
- url: "<vmsingle-ingress>/api/v1/write"
externalLabels:
cluster: <cluster-name>
grafana:
enabled: false
defaultDashboardsEnabled: false
in this case chart will deploy vmagent, kube-state-metrics, node-exporter and scrape configurations for vmagent.
diagram
You could try looking at Wavefront. It's a commercial tool now but you can get a 30 day trial free - also, it understands promQL. So essentially, you could use the same prometheus rules and config across all clusters, and then use wavefront to just connect to all of those prom instances.
Another option may be Thanos, but I've never used it personally.
We have several clusters where our applications are running. We would like to set up a Central Monitoring cluster which can scrape metrics from rest of cluster using Prometheus Federation.
So to do that, I need to install prometheus server in each of cluster and install prometheus server via federation in central cluster.I will install Grafana as well in central cluster to visualise the metrics that we gather from rest of prometheus server.
So the question is;
Where should I setup the Alert Manager? Only for Central Cluster or each cluster has to be also alert manager?
What is the best practice alerting while using Federation?
I though ı can use ingress controller to expose each prometheus server? What is the best practice to provide communication between prometheus server and federation in k8s?
Based on this blog
Where should I setup the Alert Manager? Only for Central Cluster or each cluster has to be also alert manager?
What is the best practice alerting while using Federation?
The answer here would be to do that on each cluster.
If the data you need to do alerting is moved from one Prometheus to another then you've added an additional point of failure. This is particularly risky when WAN links such as the internet are involved. As far as is possible, you should try and push alerting as deep down the federation hierarchy as possible. For example an alert about a target being down should be setup on the Prometheus scraping that target, not a global Prometheus which could be several steps removed.
I though ı can use ingress controller to expose each prometheus server? What is the best practice to provide communication between prometheus server and federation in k8s?
I think that depends on use case, in each doc I checked they just use targets in scrape_configs.static_configs in the prometheus.yml
like here
scrape_configs:
- job_name: 'federate'
scrape_interval: 15s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job="prometheus"}'
- '{__name__=~"job:.*"}'
static_configs:
- targets:
- 'source-prometheus-1:9090'
- 'source-prometheus-2:9090'
- 'source-prometheus-3:9090'
OR
like here
prometheus.yml:
rule_files:
- /etc/config/rules
- /etc/config/alerts
scrape_configs:
- job_name: 'federate'
scrape_interval: 15s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job="prometheus"}'
- '{__name__=~"job:.*"}'
static_configs:
- targets:
- 'prometheus-server:80'
Additionally, worth to check how they did this in this tutorial, where they used helm to build central monitoring cluster with two prometheus servers on two clusters.
I want to monitor disk usages of persistent volumes in the cluster. I am using CoreOS Kube Prometheus. A dashboard is trying to query with a metric called kubelet_volume_stats_capacity_bytes which is not available anymore with Kubernetes versions starting from v1.12.
I am using Kubernetes version v1.13.4 and hostpath-provisioner to provision volumes based on persistent volume claim. I want to access current disk usage metrics for each persistent volume.
kube_persistentvolumeclaim_resource_requests_storage_bytes is available but it shows only the persistent claim request in bytes
container_fs_usage_bytes is not fully covers my problem.
Per-PVC disk space usage in percentage can be determined with the following query:
100 * sum(kubelet_volume_stats_used_bytes) by (persistentvolumeclaim)
/
sum(kubelet_volume_stats_capacity_bytes) by (persistentvolumeclaim)
The kubelet_volume_stats_used_bytes metric shows per-PVC disk space usage in bytes.
The kubelet_volume_stats_capacity_bytes metric shows per-PVC disk size in bytes.
Yes, in newest version of Kubernetes you cannot monitor metric such as kubelet_volume_stats_capacity_bytes, but there are some workarounds. Unfortunately this is a bit fragmented in Kubernetes today. PVCs may have capacity and usage metrics, depending on the volume provider, and it seems that any CSI based volume doesn't have these at all. We can do this on a best effort basis butit is simple to quickly hit cases where these metrics are not available.
First, just simply write your own script which will be every time values of metric like container_fs_usage_bytes are gathered will be count difference between capacity before measurement and container usage in bytes (metric will container_fs_usage_bytes be helpful).
Prometheus is quite popular solution but to monitor capacity especially disk usage you can use Heapster, now he is about to "retire", but just for this special case you can use it, but you will have to implement script too. Take look on repository:
heapster-memory
"res.Containers = append(res.Containers,
metrics.ContainerMetrics{Name: c.Name, Usage: usage})"
I hope it helps.
I have a job like the following in my prom config:
- job_name: 'kubernetes-nodes'
kubernetes_sd_configs:
- role: node
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
With this job in place I see the following metrics available in Prometheus:
kubelet_volume_stats_available_bytes
kubelet_volume_stats_capacity_bytes
kubelet_volume_stats_inodes
kubelet_volume_stats_inodes_free
kubelet_volume_stats_inodes_used
kubelet_volume_stats_used_bytes
More here: https://github.com/google/cadvisor/issues/1702
I have a prometheus exporter for monitoring pvc usage and provides the mapping between pod and pvc. If you are interested you can try it.
https://github.com/kais271/pvc-exporter
Prometheus metrics:
pvc_usage
pvc_mapping