I have a REST API and I would like to check the health of specific endpoints. Can Grafana be used to monitor the health of specific endpoints? Any plugins that can be used for that? I know that it can be integrated with Zabbix, are there any other ways to do it?
Any help would be great.
You can probably look at Prometheus blackbox exporter and use that to monitor the health of your endpoint.
https://github.com/prometheus/blackbox_exporter
Prometheus can then be added as a data source in Grafana and then Yes you have to first create a dashboard/panel if you want to use Grafana alerting capability.
That really depends what to monitor the health of specific endpoints means for you. Usually it is:
metric collecting: No, that is not a native task for Grafana. There was special Worldping plugin for Grafana https://grafana.com/grafana/plugins/raintank-worldping-app, but that is deprecated. Usually your monitoring tool (Zabbix, Prometheus, Dynatrace, ...) is doing this.
metric visualization: Yes, this is the best task for Grafana. It can visualize metrics from supported time series databases/apps
alerting: Yes - but only in the graph panel, so there is overhead - you need to manage dashboards/panels for every single metric just to have alerting. Again, monitoring tools have usually better design for this task.
=> Use monitoring tools for monitoring and use Grafana just for the graphs.
Related
Kubernetes. Prometheus. A test application that can send metrics or a simple phrase to an HTTP request.
The end goal is to see graphs in Grafana. I want to make sure that everything works up to the current setup moment.
Now I want to see the metrics by URL, to see that the ServiceMonitor is working correctly, that they reach the Graphana. But so far I haven't been able to find a good enough troubleshooting guide.
I assume it's okay not to show everyone your metrics. But I want to make sure that this is normal behavior and which way to google in case I want to allow it.
This is a completely intended behavior when trying to use Grafana to visualize data from Managed Service for Prometheus. Metrics will be available via http request when trying to browse Grafana, as during the deployment we port-forward the frontend service to our local machine. As it is not a good practice to leave our metrics available to everyone for some security reasons, Grafana port-forwards the frontend service to our local machine.
Google Cloud APIs all require authentication using OAuth2. However, Grafana doesn't support OAuth2 authentication for Prometheus data sources. To use Grafana with Managed Service for Prometheus, you must use the Prometheus UI as an authentication proxy.
You can refer to these documentations to know more about Grafana & How-to-Deploy-Grafana.
I would like to monitor all ELK service running in our kubernetes clusters to be sure, that is still running properly.
I am able to monitor Kibana portal via URL. ElasticSearch via Prometheus and his metrics (ES have some interested metrics to be sure, that ES is working well).
But exist something similar for Filebeat, Logstash, ... ? Have these daemons some exposed metrics for Prometheus, which is possible to watching and analizing it states?
Thank you very much for all hints.
There is an exporter for ElasticSearch found here: https://github.com/prometheus-community/elasticsearch_exporter and an exporter for Kibana found here: https://github.com/pjhampton/kibana-prometheus-exporter These will enable your Prometheus to scrape the endpoints and collect metrics.
We are also working on a new profiler inside of OpenSearch which will provide much more detailed metrics and fix a lot of bugs. That will also natively provide an exporter for Prometheus to scrape : https://github.com/opensearch-project/OpenSearch/issues/539 you can follow along here, this is in active development if you are looking for an open-source alternative to ElasticSearch and Kibana.
Yes, both the beats and logstash have metrics endpoint for monitoring.
These monitoring endpoints are built to be consumed using metricbeat, but since they return a json you can use other tools to monitor it.
For logstash the metrics endpoint is enabled by default, listening on localhost at port 9600, and from the documentation you have these two endpoints:
node
node_stats
For the beats family you need to enable it as if you would consume the metrics using metricbeat, this documentation explains how to do that.
Then you will have two endpoints:
stats
state
So you would just need to use those endpoints to collect the metrics.
We are trying to figure out which monitoring options will be suitable for our environment.
We have two clusters in GCP and we installed Istio (with Helm) in both of them. We are also using Workload Identity and Stackdriver Monitoring.
Now, we would like to create dashboards (or charts) for kubernetes objects (such as, deployments, containers, cronjobs, services, etc.) and want to set alerts on them. So can anyone suggest free monitoring options to achieve these all? We don't want to go with any third party paid software.
Thank you in advance.
If you are using GCP GKE then default stack driver logging & monitoring is best option.
It's free if you are using GCP service and using stack driver monitoring you can monitoring and creat respective dashboards as per need.
For alerts, you can use the GCP Uptime check option available in monitoring itself which sends the email. For call alerts, you may have to use some custom or third applications.
You can read more at : https://cloud.google.com/monitoring/docs
Uptime checks : https://cloud.google.com/monitoring/uptime-checks
I have a mule application which mostly does HTTP requests, which is logging as plain text. I want to push these logs as metrics to Prometheus. Since this is a legacy application it would take a substantial amount of time to change code and push metrics directly into Prometheus storage.
Idea is to show Prometheus metrics in Grafana Dashboard.
Is there any intermediate tool that converts plain text to metrics?
Anything that helps with this requirement.
FYI- We have Nagios and Splunk which is doing this task as of now, we are looking to move our solution to Prometheus and Grafana
In situations like these you can use tools like https://github.com/fstab/grok_exporter to convert logs into metrics.
I am trying to figure out how to best collect metrics from a set of spring boot based services running within a Kubernetes cluster. Looking at the various docs, it seems that the choice for internal monitoring is between Actuator or Spectator with metrics being pushed to an external collection store such as Redis or StatsD or pulled, in the case of Prometheus.
Since the number of instances of a given service is going to vary, I dont see how Prometheus can be configured to poll those running services since it will lack knowledge of them. I am also building around a Eureka service registry so not sure if that is polled first in this configuration.
Any real world insight into this kind of approach would be welcome.
You should use the Prometheus java client (https://www.robustperception.io/instrumenting-java-with-prometheus/) for instrumenting. Approaches like redis and statsd are to be avoided, as they mean hitting the network on every single event - greatly limiting what you can monitor.
Use file_sd service discovery in Prometheus to provide it with a list of targets from Eureka (https://www.robustperception.io/using-json-file-service-discovery-with-prometheus/), though if you're using Kubernetes like your tag hints Prometheus has a direct integration there.