I enabled monitoring for my project in Rancher UI, and it installed succesfully. But when I click "Go to grafana" at my workload (such as nginx), it moves to Grafana dashboard, but Grafana show nothing: 0 CPU, 0 memory, 0 networking ,...
Why doesn't it have data ?
And how I can know consumed quota of my resource (workload, service, pod)?
Please see my screenshots:
Many thanks
Prometheus is necessary component for using Grafana.
You can check this blogpost:
Kubernetes Monitoring with Prometheus, Grafana and Rancher
Prometheus is an open-source application for monitoring systems and generating alerts. ... Prometheus will scrape targets at designated intervals and store the information in a time-series database.
Grafana is also open source and runs as a web application. ... Grafana makes it easy to create graphs and assemble those graphs into dashboards.
Check, if check if Prometheus is turned on and whether it is configured correctly:
Configuring Project Monitoring
From the Global view, navigate to the project that you want to configure project monitoring.
Select Tools > Monitoring in the navigation bar.
Select Enable to show the Prometheus configuration options. Enter in your desired configuration options.
Click Save.
Also check these settings:
Prometheus Configuration
Enable Persistent Storage for Prometheus Whether or not to configure storage for Prometheus so that metrics can be retained even if the Prometheus pod fails.
Enable Persistent Storage for Grafana Whether or not to configure storage for Grafana so that the Grafana dashboards and configuration can be retained even if the Grafana pod fails.
Related
I have to build a monitoring solution using Prometheus and Graphana for a service which is built using React(front end)+ Node js + Db2(containerised) . I have no idea where to start,can someone suggest me the resources where to learn?Thank you.
First of all, you need to install Prometheus and Grafana in your Kubernetes cluster following the instructions given for each:
Prometheus: https://prometheus.io/docs/prometheus/latest/installation/
Grafana: https://grafana.com/docs/grafana/latest/installation/
Next, you need to understand that Prometheus is a pull-based metrics collection system. It retrieves metrics from configured targets (endpoints) at given intervals and displays the results.
You can setup the working monitoring system by implementing the below steps:
Instrument your application code for Prometheus to be able to scrape metric from -
For this, you need to add instrumentation to the code via one of the supported Prometheus client libraries.
Configure Prometheus to scrape the metrics exposed by the service - Prometheus supports a K8s custom resource named ServiceMonitor introduced by the Prometheus Operator that can be used to configure Prometheus to scrape the metric defined in step 1.
Observe the scraped metrics - Next, you can observe the defined metric in either the Prometheus UI or Grafana UI by configuring Grafana support for Prometheus.
I have a cluster with 7 nodes and a lot of services, nodes, etc in the Google Cloud Platform. I'm trying to get some metrics with StackDriver Legacy, so in the Google Cloud Console -> StackDriver -> Metrics Explorer I have all the set of anthos metrics listed but when I try to create a chart based on that metrics it doesn't show the data, actually the only response that I get in the panel is no data is available for the selected time frame even changing the time frame and stuffs.
Is right to think that with anthos metrics I can retrieve information about my cronjobs, pods, services like failed initializations, jobs failures ? And if so, I can do it with StackDriver Legacy or I need to Update to StackDriver kubernetes Engine Monitoring ?
Anthos solution, includes what’s called GKE-on prem. I’d take a look at the instructions to use logging and monitoring on GKE-on prem. Stackdriver monitors GKE On-Prem clusters in a similar way as cloud-based GKE clusters.
However, there’s a note where they say that currently, Stackdriver only collects cluster logs and system component metrics. The full Kubernetes Monitoring experience will be available in a future release.
You can also check that you’ve met all the configuration requirements.
I have two Kubernetes clusters representing dev and staging environments.
Separately, I am also deploying a custom DevOps dashboard which will be used to monitor these two clusters. On this dashboard I will need to show information such as:
RAM/HD Space/CPU usage of each deployed Pod in each environment
Pod health (as in if it has too many container restarts etc)
Pod uptime
All these stats have to be at a cluster level and also per namespace, preferably. As in, if I query a for a particular namespace, I have to get all the resource usages of that namespace.
So the webservice layer of my dashboard will send a service request to the master node of my respective cluster in order to fetch this information.
Another thing I need is to implement real time notifications in my DevOps dashboard. Every time a container fails, I need to catch that event and notify relevant personnel.
I have been reading around and two things that pop up a lot are Prometheus and Metric Server. Do I need both or will one do? I set up Prometheus on a local cluster but I can't find any endpoints it exposes which could be called by my dashboard service. I'm also trying to set up Prometheus AlertManager but so far it hasn't worked as expected. Trying to fix it now. Just wanted to check if these technologies have the capabilities to meet my requirements.
Thanks!
I don't know why you are considering your own custom monitoring system. Prometheus operator provides all the functionality that you mentioned.
You will end up only with your own grafana dashboard with all required information.
If you need custom notification you can set it up in Alertmanager creating correct prometheusrules.monitoring.coreos.com, you can find a lot of preconfigured prometheusrules in kubernetes-mixin
.
Using labels and namespaces in Alertmanager you can setup a correct route to notify person responsible for a given deployment.
Do I need both or will one do?, yes, you need both - Prometheus collects and aggregates metric when Metrick server exposes metrics from your cluster node for your Prometheus to scrape it.
If you have problems with Prometheus, Alertmanger and so on consider using helm chart as entrypoint.
Prometheus + Grafana are a pretty standard setup.
Installing kube-prometheus or prometheus-operator via helm will give you
Grafana, Alertmanager, node-exporter and kube-state-metrics by default and all be setup for kubernetes metrics.
Configure alertmanager to do something with the alerts. SMTP is usually the first thing setup but I would recommend some sort of event manager if this is a service people need to rely on.
Although a dashboard isn't part of your requirements, this will inform how you can connect into prometheus as a data source. There is docco on adding prometheus data source for grafana.
There are a number of prebuilt charts available to add to Grafana. There are some charts to visualise alertmanager too.
Your external service won't be querying the metrics directly with prometheus, in will be querying the collected data in prometheus stored inside your cluster. To access the API externally you will need to setup an external path to the prometheus service. This can be configured via an ingress controller in the helm deployment:
prometheus.ingress.enabled: true
You can do the same for the alertmanager API and grafana if needed.
alertmanager.ingress.enabled: true
grafana.ingress.enabled: true
You could use Grafana outside the cluster as your dashboard via the same prometheus ingress if it proves useful.
I have used this documentation in order to deploy Prometheus with Grafana on the cluster.
The problem arises whenever we restart our Prometheus and Grafana with some changed configuration all our dashboards and visualizations are gone.
Is there a workaround where we can persist the dashboards and visualizations?
You need to define volumes, which will be used in the Grafana/Prometheus containers to store data persistently.
Doc: https://docs.mesosphere.com/1.7/administration/storage/mount-disk-resources/
would like to see k8 Service level metrics in Grafana from underlying prometheus server.
For instance:
1) If i have 3 application pods exposed through a service i would like to see service level metrics for CPU,memory & network I/O pressure ,Total # of requests,# of requests failed
2)Also if i have group of pods(replicas) related to an application which doesn"t have Service on top of them would like to see the aggregated metrics of the pods related to that application in a single view on grafana
What would be the prometheus queries to achieve the same
Service level metrics for CPU, memory & network I/O pressure
If you have Prometheus installed on your Kubernetes cluster, all those statistics are being already collected by Prometheus. There are many good articles about how to install and how to use Kubernetes+Prometheus, try to check that one, as an example.
Here is an example of a request to fetch container memory usage:
container_memory_usage_bytes{image="CONTAINER:VERSION"}
Total # of requests,# of requests failed
Those are service-level metrics, and for collecting them, you need to use Prometheus Exporter created especially for your service. Check the list with exporters, find one which you need for your service and follow its instruction.
If you cannot find an Exporter for your application, you can write it yourself, here is an official documentation about it.
application which doesn"t have Service on top of them would like to see the aggregated metrics of the pods related to that application in a single view on grafana
It is possible to combine any graphics in a single view in Grafana using Dashboards and Panels. Check an official documentation, all that topics pretty detailed and easy to understand.
Aggregation can be done by Prometheus itself by aggregation operations.
All metrics from Kubernetes has labels, so you can group by them:
sum(http_requests_total) by (application, group), where application and group is labels.
Also, here is an official Prometheus instruction about how to add Prometheus to Grafana as a Datasourse.