Example
I am getting series for a metric with a label called "store" which contains the "storeId". Now I would like to add another label "storeName" and fill it with external information (API-call/CSV/other Prometheus "metric"/merging from mysql/...). Is this possible? If so, which ways can I take?
Goal
is to show the resulting metric in Grafana graph-panels so that it contains names rather than IDs.
Related
I'm trying to add a derived field that consist of multiple parts. The help text in Grafana suggests that it should be possible to access multiple capture groups, but all I can find is the ${__value.raw} variable. Are there other undocumented variables to access other capture groups?
Is it possible to have 2 different sources i.e influxdb and prometheus in a single dashboard in grafana.
Yes, each panel or variable on a dashboard can use different data source.
Variables can use different data source and you can use it on panel no matter what data source uses. It allows you to mix information to have a better dashboard with more details.
Also, you have three especial data sources:
Grafana built-in data source that generates random walk data.
Mixed query multiple data sources in the same panel.
Dashboard use a result set from another panel in the same dashboard.
More info about data sources here
Yes, you can select the desired data sources in several points of the dashboard configuration (like panels and variables) and you can use different values for each one.
I've got metrics that looks like this
kafka_lag{client_id="dcp-0",partition="53"} 1977005
kafka_lag{client_id="dcp-10",partition="53"} 2345234
When I visualize in grafana I get two different lines, however I would like to drop client_id and display only kafka_lag{partition="53"} values.
How can I drop a tag from Prometheus output?
If you have two values at the same time (in your case, the lag for different client) and you want to view only one, you need to combine them in a way that make sense.
By example:
max(kafka_lag) by(partition) if you want to view the maximum lag
avg(kafka_lag) by(partition) if you want to have a sense of overall lag
Any of the aggregation operators can be used to extract information.
I'm using Grafana and Prometheus to monitor our server. We have a lot of database procedures like "select_users" or "insert_task". In order to monitor how many pending database procedure calls are there in the server, we add data points for every procedure call in Prometheus dynamically. Now we have data points like "pending_select_users", "pending_insert_task" in Prometheus.
However, since there are so many database procedures(and the number will increase during developing), it's not very practical for us to add metrics in Grafana for each data point manually. Is there a way we can add metrics dynamically in Grafana? Since all the data point have a common name prefix("pending_"), can we add metrics in Grafana with wildcard? Or is there a better way to do this?
Since Grafana uses JSON as the underlying dashboard DSL, you could dynamically create dashboards, every time you add a new metric, and import it (via API) into Grafana.
I'd add an automation on top of your Prometheus targets, scrape the metrics, and if new metrics (with the required prefix) are found without a matching dashboard, the automation would create it and import it into Grafana.
Grafana API: http://docs.grafana.org/http_api/ (specifically for Dashbboards).
The solution described by #Eitan is definitely feasible. The same goes for using a library like grafonnet to generate dashboards dynamically.
But the simplest approach in my opinion would be to create a variable in Grafana that contains all the label values you are interested in. Something like
label_values(metric_name{label_name=~"prefix*"}, label_name)
should work for that. And then use the repeating panels / rows feature of Grafana to repeat a set of panels for every value in the variable. Though this could get out of hand if you have dozens / hundreds of distinct values.
https://grafana.com/docs/grafana/latest/variables/repeat-panels-or-rows/
https://grafana.com/blog/2020/06/09/learn-grafana-how-to-automatically-repeat-rows-and-panels-in-dynamic-dashboards/
If you want to generate just a single dashboard from your Proimetheus metrics sample, you can use this service:
http://eljah.tatar/micrometer2grafana/
I have Grafana with Bosun connected as OpenTSDB source. Problem is Grafana interprets data in different way than Bosun. To be precise, when I set same query in Bosun and in Grafana, resulting graphs differ. When I turn on gauge downsample, graphs are same. So I guess there is implicit gauging of some sort in Grafana. I would be grateful for some hint how to disable that gauging.
Bosun:
Grafana:
The os.net.bytes metric includes metadata to indicate that it is a rate. When you use the default "auto" in Bosun's graph page it will convert the raw counter data into a rate calculation. Grafana's OpenTSDB data source does not have an auto mode, so things always default to a gauge unless you check the Rate box at the bottom of the metric.
In your example you should just need to check the rate box to get the graphs to match. You can also use the Counter option and provide a max or reset value if you need to deal with counter overflows
You can also use the Bosun data source if you want to use a Bosun query instead of accessing OpenTSDB directly. In this example we combine two queries to generate a Singlestat panel (displays last value and a line graph in the background)
The __ny-nexus01/02 part comes from using tsdbrelay to denormalize the metric and address high tag cardinality issues.