I'm trying to configure a Grafana dashboard for multiple Prometheus datasources. These two datasources are present in two different servers.
Prometheus scrape the metrics from three jobs, prometheus itself, node-exporter for host metrics, advisor for docker containers metrics.
This is what prometheus.yaml looks like:
global:
scrape_interval: 5s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node-exporter'
static_configs:
- targets: ['node-exporter:9100']
- job_name: 'cadvisor'
static_configs:
- targets: ['cadvisor:8080']
The same yaml configuration is being used in both the servers.
I created two datasources in Grafana UI and their respective names are Prometheus, and Prometheus-Certify. The first datasource is the default one. I'm using Grafana v7.1.0.
This is what my dashboard looks like for the datasource Prometheus:
There are other panels as well, but showing these panels would be enough to explain the issue.
If I click on the Host dropdown option, then I can see the hosts cadvisor:8080, localhost:9090, and node-exporter:9100.
Requirement:
What I'm trying to achieve is to configure this dashboard for both the data sources. I can see all the options in the datasource dropdown menu default, Prometheus, Prometheus-Certify. I want the dashboard to show me the data from a particular source when I select that particular datasource. And this is where I'm getting stuck.
I did my research on this and found that we can show data from two different sources at the same time in a panel. I saw this option in the Grafana documentation and also this thread which actually gets me closer to what I'm trying to achieve.
So I followed this approach and configured --Mixed-- datasources for a panel.
It worked flawlessly and performing the same operation on different panels resulted in this:
Though it worked but, this is not what I was trying to do. The problems here are:
I'll have to do this for every panel in order to see data from both the sources.
It shows data from both sources at the same time. I want to see data from the selected datasource only, not both.
If I add another datasource in the future, I'll have to perform all the operations all over again.
I saw [templates and variables][6] being suggested in many places. I gave it a shot but it didn't do anything.
If changing variables is the way to go then please provide suggestions specific to this issue only as I've already gone through that section and couldn't find anything specific to this.
I hope I was able to explain my problem to you people. I just want to see the data from the selected datasource without creating a new dashboard. Your help would be very much appreciated. Thank you.
I realized that the dashboard I imported creates a variable DS_PROMETHEUS, and its type is Datasource.
This variable holds the value of the selected datasource. I was unable to see the data from the selected datasource because the datasource was kinda hardcoded in the panels' queries.
As you can see in the image above the preselected datasource is Prometheus. Changing this value to $DS_PROMETHEUS did the job.
So it all boiled down to how do I make this variable as the datasource for every query in every panel? The question now was how to run a query/command to modify settings of multiple panels in one go.
After doing some research I found this thread. It has been categorized as a feature request. Person/People have already developed a tool to handle these kinds of scenarios. As of now, I haven't used this tool but I'll update my answer once I use it.
The simple and the old-school way was to replace prometheus with $DS_PROMETHEUS everywhere in the dashboard's JSON. So I did that and Voila! it worked.
For me disabling a default Prometheus datasource did the trick. Without a default data source you can select the Prometheus datasource in the query.
Related
I want to add a new annotation query to every dashboard. Is there a way to do this without just manually doing it 1 by 1 for each individual dashboard?
The example we want to add is to include the GitHub Releases for our GitHub repo to each graph so we can see the effect of releases on metrics. We're using the GitHub plugin for this.
Is there a way to do this en-masse?
The team at Grafana said there currently is not a way to do this automatically or via the UI.
Their recommendation would be to write a script which uses the Grafana API to add the annotations:
https://grafana.com/docs/grafana/latest/developers/http_api/annotations/
I've not attempted this, but it might be beneficial to others.
We did it manually instead.
I have a test Grafana setup pointing to my AWS CloudWatch service and CloudWatch logs. I can setup a graph pointing to a specific log group or groups, but what I want is to be able to dynamically set the logs groups, e.g., aws/containerinsights/$cluster/application. But I can't see how to do that or whether its even possible. The log group selection appears to be only from a dropdown and can't be edited. Is this possible? If it's not, I don't see how you can create a dashboard that can be used across clusters.
Usually, what looks like a dropdown (for example also dashboard variable) in the Grafana also takes user input - you can write/paste own text there. So write desired string with dashboard variable manually and then use offered Create: ... option:
We are using grafana to visualize our metrics and very glad to use it.
For one dashboard I need to create custom variable to show host name. For example, we have the first variable: host with query collectd.localhost.ping.ping-* and regexp /ping-(.*)/. It returns a list of hosts that we ping. Now I want to add showing title depends on selected host. I think need to add another custom variable title like map host1:Title1;host2:Title2 and in the panel's title write something like Proxy ${title:json}.${host} in result get Proxy TitleN.
How to do it? We use 7.2.1 grafana version.
Thank you!
I'm not sure if this is a great answer, but given it's been nearly two years without any answer, I'll proffer it. Having just commented how I was surprised no one had in fact answered, I kept poking around. Eventually I found that if added a variable interactively, and then I clicked dashboard settings followed by JSON Model I got to a description that included new data in the templating field. I pasted that into my original JSON description of the dashboard, and restarted Grafana --- the change persisted.
I would like to get a table in my grafana dashboard that lists all annotations shown on the dashboard (gathered via dashboard configuration and (if possible) also the ones contained in panels.
Any idea? I cannot find a detailed documentation on that...
Please see picture,
I didn't add Grafana, Grafana metrics, Graphite Carbon metrics... I don't know where these dashboards are defined but I need to remove them
You can go to the Manage Dashboards page, select the ones you don't want and delete them.