I was trying to display the metrics for 64 nodes on my k8s clsuter. Then I found out that whenever I select more than 60 nodes in the variable dropdown
Grafana throws query error that looks like this:
The exception message is not particularly helpful, could somebody provide me more insights? Thanks!
I've had a similar problem after selecting too many variables. As long as the rest of your monitor is able to pull the info successfully from prometheus, you can disable the annotation query. Go to the dashboard and remove the annotations under settings.
Related
I'm trying to create a Grafana dashboard that will reflect my AWS RDS cluster metrics.
For the simplicity I've chose CloudWatch as a datasource, It works well for showing the 'direct' metrics from the RDS cluster.
Problem is that we've switched to use RDS Proxy due the high number of connections we are required to support.
Now, I'm adjusting my dashboard to reflect few metrics that are lacking, most important is number of actual connections, which in AWS CloudWatch console presented by this query:
SELECT AVG(DatabaseConnections)
FROM SCHEMA("AWS/RDS", ProxyName,Target,TargetGroup)
WHERE Target = 'db:my-db-1'
AND ProxyName = 'my-db-rds-proxy'
AND TargetGroup = 'default'
Problem is that I can't find it anywhere in the CloudWatch Grafana query editor:
The only metric with "connections" is the standard DatabaseConnections which represents the 'direct' connections to the RDS cluster and not the connections to the RDS Proxy.
Any ideas?
That UI editor is generated from hardcoded list of metrics, which may not contain all metrics and dimensions (especially if they have been added recently), so in that case UI doesn't generate them in the selectbox.
But that is not a problem, because that selectbox is not a standard selectbox. It is an input, where you can write your own metric and dimension name. Just click there, write what you need and Hit enter to add (the same is applicable for:
Pro tip: don't use UI query builder (that's for beginners), but switch to Code and write your queries directly (anyway UI builder builds that query under the hood):
It would be nice if you create a Grafana PR - add these metrics and dimensions which are missing in the UI builder to metrics.go.
So for who ever will ever get here you should use ClientConnections and use the ProxyName as the dimension (which I didn't set initially
I was using old Grafana version (7.3.5) which didn't have it built in.
I am new into microstrategy an trying to find out the specific metrics(different metrics for different vendors) used in a project and trying to pull the total number of metrics in every group/vendor. How can we do this using the command manager scripts like : LIST ALL METRICS IN FOLDER FOR PROJECT.
Using this command I get multiple like 50k + metrics and my system freezes out. So need help with correct script or some solution.
Try search support in rest API. You can add metric as the query parameter and use page by to query object list.
https://www2.microstrategy.com/producthelp/Current/RESTSDK/Content/topics/REST_API/REST_API_Workflow_Searching.htm
I am newbie to grafana and prometheus. I setup prometheus, grafana, alertmanager, nodeexporter and cadvisor using the docker-compose.yml from this post https://github.com/vegasbrianc/prometheus
And imported grafana dashboard #893 from https://grafana.com/dashboards/893
But the dashboard is not working as I can see N/A in some panels. For example below are the queries used by the panels and I couldn't figure out how to get the values for the template variable in the query. I looked at http://node-exporter:9100/metrics and do not see a value for variable '$server'
Query1: time() - node_boot_time{instance=~"$server:.*"}
Query2:min((node_filesystem_size_bytes{fstype=~"xfs|ext4",instance=~"$server:.*"} - node_filesystem_free_bytes{fstype=~"xfs|ext4",instance=~"$server:.*"} )/ node_filesystem_size_bytes{fstype=~"xfs|ext4",instance=~"$server:.*"})
What should I configure for node-exporter and prometheus to evaluate the template variable $server in the queries?
$server is a Grafana template variable. These usually show up as dropdowns at the top of the Grafana dashboard.
label_values is a Prometheus-specific Grafana function that is applied to a Prometheus query. Your particular example, label_values(node_boot_time, instance) will return all values of the instance label for all node_boot_time metrics collected by Prometheus (i.e. all node exporter targets monitored by Prometheus).
I have no experience with the particular dashboard you are using (or node exporter, for that matter), but usually the cause for some panels displaying "N/A" or no values while other panels work just fine is that the underlying metric names might have changed. You can click on the header of the problematic panel in Grafana, select Edit, then click on the Metrics tab to try different metric names. For "inspiration", check the /metrics endpoint of your node exporter. If you don't know how to get to it, on the Prometheus web interface navigate to Status > Targets and click on the URL of your node exporter.
An old question, but it still didn't work for me.
The reason is that the label_values(...) works fine obtaining all the instance names that have a node_boot_time metric.
The problem is in the regex that follows the expression (next line). In my case it was something tricky resembling "/([^:].*):/". My instance names start with "i-" and contain no colon, so nothing was being selected. I just used a ProductCode to figure out the right instances instead.
I am trying to put a dropdown for each API end point which will show the QPS and Latency of http requests (RED metrics).
I used Grafana's templating and used the following prometheus query.
label_values(http_duration_milliseconds_count, api_path)
But the problem here is sort order. It shows some longtail api requests like /admin/phpMyAdmin all.
I want to do only the top 10 endpoints by count to be shown in this drop down. How do I achieve this?
Attached an image for reference on my first dashboard.
We can use query_result to achieve this.
https://grafana.com/docs/grafana/latest/datasources/prometheus/template-variables/#use-query-variables
query_result(topk(10, sort_desc(sum(http_tt_ms_count) by (api_path))))
http_tt_ms_count - is my metric timeseries of Prometheus with time taken.
api_path - is my label name
This query_result will give three-tuple value like this.
{api_path="/search/query"} 25704195 1507641522000
used the Regex field in query path to get only the api names.
*api_path="(.*)".*
This looks like a long way but
label_values((topk(10, sort_desc(sum(http_tt_ms_count) by (api_path)))), api_path)
is not working in Grafana which made me to go into this path.
I am using Grafana for my application, where I have metrics being exposed from my data source on demand, and I want to monitor such on-demand metrics in Grafana in a user-friendly graph. For example, until an exception has been hit by my application, the data source does NOT expose the metric named 'Exception'. However, I want to create a graph before hand where I should be able to specify the metric 'Exception' and it should log it in the graph whenever my data source exposes the 'Exception' metric.
When I try to create a graph on Grafana using the web GUI, I'm unable to see these 'on-demand metrics' since they've not yet been exposed by my data source. However, I should be able to configure the graph such that in case these metrics are exposed then show them. If I go ahead and type out the non-exposed metric name in the metrics field, I get an error "Timeseries data request error".
Does Grafana provide a method to do this? If so, what am I missing?
It depends on what data source you are using (Graphite, InfluxDB, OpenTSDB?).
For graphite you can enter raw query mode (pen button). To specify what ever query you want, it does not need to exist. Same is true InfluxDB, you find the raw query mode in the hamburger menu drop down to the right of eacy query.
You can also use wildcards in a graphite query (or regex in InfluxDB) to create generic graphs that will add series to the graph as they come in.