In Prometheus exists some custom metrics from DB
In Graphana I maked Graph Dashboard with datasource from Prometheus
count(custom_metrics_project1<1)
If condition custom_metrics_project1<1 not found any metrics Graphana displays Points not found.
How change condition to be displyed 0?
You can select how to display NULL values in the edit menu for a given graph.
Please follow these steps:
click on your graph
click on edit to bring up the edit menu for the graph
switch to the tag Display
choose one of the options from the Null Value dropdown
The option you want is: null as zero.
Just use count(custom_metrics_project1<1) or on() vector(0). This will substitute all the gaps with zeroes. See docs for or operator and docs for vector() function.
Note that the q or on() vector(0) trick works only when q returns a single time series. If q returns multiple time series, then Prometheus doesn't provide an easy way to fill gaps in every time series with zeroes :(
If you still need to fill gaps in multiple time series, then you can use default operator in VictoriaMetrics. For example, the following query would fill all the gaps for all the time series returned from q with zeroes:
q default 0
VictoriaMetrics also provides interpolate, keep_last_value and keep_next_value functions, which can be used for filling gaps in more sophisticated ways.
Related
I'm using Grafana v8.3.4 (a551d74b11)
I've a dashboard with many variables (filters). One of those is for exemple TransactionType.
I would like my charts to filter on click.
For example with a pie chart using TransactionType, I want to filter the TransactionType variable ,when I click in one of the element of the pie chart, by this element.
The closest feature I found is to use data link.
So far I tried in the pie chart the two data link there after.
The syntax is OK but the effect is not doing what i’m expecting.
a)
/d/${__dashboard.uid}/${__dashboard}?var-TransactionType=${__series.name}
For this one the data link apply the filter on TransactionType but clear all the other filters variables, even clear the All, so the dashboard is in error.
b)
/d/${__dashboard.uid}/${__dashboard}?${__all_variables}&var-TransactionType=${__series.name}
this one apply all filters first then add the TransactionType filter but, it keep the previous All value, so the filter become All + value, that don’t work, the dashboard is in error too.
==> Is there a way to keep all the existing variable filter unchanged and replace the value for one variable filter ?
move var-TransactionType=${__series.name} before ${__all_variables}
I created a SQL query that counts the number of servers running test jobs on a specific Jenkins server at a specified time. I'm trying to chart it on Grafana but for some reason, it's displaying the value as NaN.
The data source is a MySQL server. I'm running Grafana version 8.1.5. I went on the server (phpMyAdmin) to check the results of the query and I can see numbers.
When we look at the grafana chart/panel, the bars on the chart look like it matches the values, but the chart shows NaN instead of the value.
What setting do I need to change so that it can print the numeric value on the chart instead of displaying NaN?
EDIT: Looks like I didn't include enough info to my question.
Here's the query that's used, although, I don't know how feasible it is to include the tables with data to demo the issue:
SELECT COUNT(DISTINCT(tb_name)) as val, jenkins FROM (
SELECT
hw.collected_date AS mytime,
hw.jenkins AS jenkins,
hw.name AS tb_name
FROM hw_report as hw LEFT OUTER JOIN jenkins_owner jo
ON hw.jenkins = jo.jenkins
WHERE jo.org IN ('Enterprise Readiness')) as y
WHERE mytime = '2021-12-03 00:00:00'
GROUP BY jenkins
HAVING count(distinct(tb_name)) > '0'
Here are screenshots of my panel. The "Show values" setting is set to "Always", which II assume should show the values.
One thing I noticed is when I hover over the bar, it shows
COUNT(DISTINCT(name_cnt))
instead of a numeric value. Not sure if this is indicative of anything. I checked other charts in the dashboard that someone created and their bars either have a numeric value or it's just a column name (like name_cnt)
Sorry I'm a noob and didn't know what to configure, look up, or mention in my question. I accidentally came across the answer.
So it seems like I need to select something under the "Transform" tab for the panel. Calculation needs to be "Total" instead of "Count" and I need to hide the fields specified in my SQL query and show the "Total"
I use grafana to plot timeseries data. In a timeseries plot i want to add a constant line which comes from a monitoring level. The value of that level is dynamic (from a postgres database) the timeseries come from a ifluxdb Datasource.
The monitoring level have no timestamp. The result should look like this:
I have searched quite a while how to do this, but not found a good explanation.
You can now add thresholds (bottom of the edit screen).
Each threshold can be represented as a solid line and/or region with individual color
Another way to do it in a dirty way is to create panel with a mixed data source.
Create your variable in grafana - it can be a query, constant or custom. Just remember to keep it a single floating point.
Add your original query and add the prometheus data source to query your variable.
${net_ordered_storage}
You will have to play a little bit with the number of data points displayed (query options>max data points) and minimum step of data point in prometheus query to make grafana connect dots.
Green horizontal line from variable
To draw a line like that you have to "fake" a timeseries. (thresholds don't work since they can not be dynamic as far as I know)
First thing to keep in mind is that grafana needs timestamp to plot it, for this reason the global variables ${__to} and ${__from} come in handy.
Then, to draw a line, grafana needs at least two points. ([t0, t1][y0, y1])
So this is the sql (postgre) query that lead to the desired result:
SELECT
${__from} AS time,
level_1,
FROM my_table where display_name = '${my_grafana_var:raw}'
union all
SELECT
${__to} AS time,
level_1,
FROM my_table where display_name = '${my_grafana_var:raw}';
It is possible to add dynamic thresholds using the Config from query option.
Add a new query below your standard metric query. This will likely be called B.
Here you query the static reference value.
Eg SELECT expected_number_of_widgets FROM baseline
Open Transformation tab
Find Configuration from Query results
Choose Config query = B
At the dropdown for expected_number_of_widgets, choose Use as: Threshold1.
In the right panel, under Thresholds, make sure Show Tresholds is enabled and remove the default threshold of 80.
For more details, see https://grafana.com/docs/grafana/latest/panels-visualizations/query-transform-data/transform-data/#config-from-query-results
I have multiple Prometheus instances providing the same metric, such as:
my_metric{app="foo", state="active", instance="server-1"} 20
my_metric{app="foo", state="inactive", instance="server-1"} 30
my_metric{app="foo", state="active", instance="server-2"} 20
my_metric{app="foo", state="inactive", instance="server-2"} 30
Now I want to display this metric in a Grafana singlestat widget. When I use the following query...
sum(my_metric{app="foo", state="active"})
...it, of course, sums up all values and returns 40. So I tell Prometheus to sum it by instance...
sum(my_metric{app="foo", state="active"}) by (instance)
...which results in a "Multiple Series Error" in Grafana. Is there a way to tell Prometheus/Grafana to only use the first of the results?
I don't know of a distinct, but I think this would work too:
topk(1, my_metric{app="foo", state="active"} by (instance))
Check out the second to last example in here:
https://prometheus.io/docs/prometheus/latest/querying/examples/
One way I just found is to additionally do an average over all values:
avg(sum(my_metric{app="foo", state="active"}) by(instance))
If you need to return an arbitrary time series out of multiple matching time series, then this can be done with topk() or bottomk() functions. For example, the following query returns a single time series with the maximum value out of multiple time series which match my_metric{app="foo", state="active"}:
topk(1, my_metric{app="foo", state="active"})
You need to set instant query option in Grafana when using topk(). Otherwise topk(1, ...) may return multiple time series when it is used for building a graph with range query. This is because topk(1, ...) selects a single time series with the max value individually per each point on the graph. Different points on the graph may have different time series with the max value. There is a workaround, which allows returning a single series out of many series on a graph in alternative Prometheus-like systems such as VictoriaMetrics. It provides topk_* and bottomk_* functions for this purpose. See, for example, topk_last or topk_avg.
Note that topk() has no common grounds with DISTINCT from SQL. If you need to select distinct label values with PromQL, then you need to use count(...) by (label). It will return unique label values for the given label alongside the number of unique time series per each label value. For example, count(my_metric) by (app) will return unique app label names for time series with my_metric name. This is roughly equivalent to the following SQL with DISTINCT clause:
SELECT DISTINCT app FROM my_metric
See count() docs for details.
I have two exporters for feeding data into prometheus - the node exporter and the elasticsearch exporter. I'm trying to combine sources from both exporters into one query, but unfortunately get "No data points" in the graph.
Each of the series successfully shows data:
elasticsearch_jvm_memory_max_bytes{cluster="$cluster", name=~"$node"}
node_memory_MemTotal{name=~"$node"}
This is the result when I try to subtract the two series from one another:
node_memory_MemTotal{name=~"$node"} - elasticsearch_jvm_memory_max_bytes{cluster="$cluster", name=~"$node"}
What am I missing here?
Thanks.
The subtraction you are trying here is more complex than it reads in the beginning. On both sides of the - operator are queries that can result in one or more time series. So the operation requested works as follows: Execute the query on the left hand side and get a result of one or more time series. A time series means a unique combination of a metric and all its labels and their values. Then a second query for your right hand side is executed which also results in one or more time series. Now to calculate the results, only those combinations with matching label combinations are used.
For your example this means that the metrics from node_exporter and from the elasticsearch_exporter have different label names (or even only different values for the labels). When there are no combinations that exist on both sides, you will see the empty result. For details on how operators are applied, please see the prometheus docs.
To solve your problem, you could do the following:
Check the metrics of both left and right side on their own
Evaluate if there are additional labels that could be ignored
See if there is a good label to match on (e.g. instance / node / hostname)
Use the ignoring(a,b,c) on the required side(s) to drop superfluous dimensions, e.g. the job
Try the following query:
node_memory_MemTotal{name=~"$node"}
- on(name)
sum(elasticsearch_jvm_memory_max_bytes{cluster="$cluster", name=~"$node"}) by (name)
It works in the following way:
It selects all the time series matching the node_memory_MemTotal{name=~"$node"} time series selector.
It selects all the time series matching the elasticsearch_jvm_memory_max_bytes{cluster="$cluster", name=~"$node"} selector.
It groups time series found at step 2 by name label value and sums time series in each group with sum() aggregate function. The end result of the sum(...) by (name) is per-name sums.
It finds pairs of time series with identical name label value from the step 1 and step 3 and calculates the difference between the first and the second time series in each pair. The on(name) modifier is used for limiting the set of labels, which are used for finding time series pairs with matching labels. See more details about this process here.