Kubernetes ingress nginx grafana dashboard prometheus data was empty - kubernetes

I just completely setup prometheus and grafana dashboard using this tutorial https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/.
I try to query something in prometheus and it was successfully plotting the graph. But when I access my Grafana dashboard with connecting to prometheus data, it returns empty charts like the below pic.
Do I miss something in the step?


Probably, you didn't create datasource in Grafana before the dashboard import. It is not specified in the manual, but dashboard will not work correctly without it.
How to create Data Source in Grafana:
Open Configuration(gear) -> Data Sources
Press "Add data source"
Select Prometheus
Specify Prometheus server URL: (e.g: http://10.22.0.3:32150/)
Press "Save & Test"
See the confirmation about passed test.
Now, select existing Data Source from the drop-down list during the import of the Nginx Ingress Dashboard from JSON. URL to the dashboard didn't work for me, so I've just copypasted the whole JSON content except two first lines with comments.
For existing dashboard you can change the data source:
Open Dashboards -> Manage
Click on "Nginx Ingress Controller"
Open its settings (gear picture on the top)
Select "JSON Model"
Update all lines with "datasource": "old_data_source_name", to the desired name
Press "Save changes"
Press green "Save" button on the left, under "Settings"
Alternatively, you can edit every element on the dashboard and select the desired data source from the drop-down list. Not very convenient way, so I'd prefer to import dashboard again.


Check that the annotation is applied to the metrics service object: prometheus.io/scrape: "true".
I was having the same problem as you and this solved it for me. I am using helm to deploy my nginx-ingress. I have the following settings that made this work:
.controller.metrics.enabled: true
.controller.metrics.serviceMonitor.enabled: true
.controller.metrics.service.annotations.prometheus.io/scrape: true

Related

Should I remove comments in the .ini files in Grafana?

I am trying to work on Grafana as code. And I have installed Grafana on my windows 10 machine. Now I wnat to configure Grafana in the Configuration files. This page ( Grafana Configuration)
says that I have to uncomment all the ini. files. But somethings are not necessary. Should I still uncomment every line? thanks for your answer

You don't need to uncomment. There are default values usually. For example Grafana port 3000.
You need to uncomment only if you want to change those default values. For example if you want to use different Grafana port, e. g. 5000.

Krakend grafana dashoboard shows no data

I am using krakend-ce 1.4 and influx 1.X
I have configured grafana dashboard and hoping to see dashboard panels for all the layers.
There are 4 layers as per dashboard.
Router
Proxy
Backend
System
I see router panels data is getting charted as expected. But for other panels I see empty charts. "No Data to show"
my configuration for krakend-metrics and influx modules is as follows:
"github_com/devopsfaith/krakend-metrics": {
"collection_time": "30s",
"proxy_disabled": false,
"router_disabled": false,
"backend_disabled": false,
"endpoint_disabled": false,
"listen_address": "127.0.0.1:8090"
},
"github_com/letgoapp/krakend-influx":{
"address":"http://influxdb-service:80",
"ttl":"25s",
"buffer_size":0,
"db": "krakend",
"username": "admin",
"password": "adminadmin"
}
I also added opencensus as follows:
"github_com/devopsfaith/krakend-opencensus": {
"sample_rate": 100,
"reporting_period": 1,
"enabled_layers": {
"backend": true,
"router": true,
"pipe": true
},
"influxdb": {
"address": "http://influxdb-service:80",
"db": "krakend",
"timeout": "1s",
"username": "admin",
"password": "adminadmin"
}
}
I thought may be my data is not ending up in influxDB, so I went in and checked what does it have. show measurements give me following output, and all measurements does have some data.
I am using grafana dashboard ID 5722. which is specified in docs.
how can I change my setup so that panels for proxy, backend and system shows charts?
__________________________
UPDATE
I upgrade krakend to version 2.1
Removed opencensus metrics exporter
Now using dashboard 15029 per krakend 2.1.2 documentation.
I still do not see other layer charts getting populated.
PS: I have checked what metrics are getting exposed on http://krakend-host:8090/__stats I see layer.backend and layer.pipe metrics.
__________________________
UPDATE 2
I was also checking for other available dashboards which can work. I stumbled upon this one https://github.com/letgoapp/krakend-influx/blob/master/examples/grafana-dashboard.json
I see 2 more panels showing up. but not all of them.

kale
If you want to use the Grafana dashboards published by KrakenD they work with only with the native InfluxDB exporter (github_com/letgoapp/krakend-influx) and NOT with the OpenCensus one.
You should delete the github_com/devopsfaith/krakend-opencensus configuration block, otherwise, you have KrakenD reporting the same metrics in different measurements twice.
The influxdb port is usually http://influxdb-service:8086, but you show port 80 in the settings, make sure these are not old data. If you have changed the port it's fine.
Another thing you have to check is if the InfluxDB client is started during KrakenD startup (there is a line in the logs telling about it), old versions had a random bug that prevented the metrics from being sent.
Finally, the maintained dashboard is now 15029 for the InfluxDB v1. Try to use that one.
There have been several improvements in metrics and in KrakenD in general system, I would avoid staying with the old 1.4 being the transition as simple as it is (but this is another topic)
Hope this helps

After hours of debugging, I finally found the solution. Key here to note is If your configurations are correct, you should see data showing up in influxDB. First, make sure that you see data in influxDB. In my case, this was correct.
Like I mentioned in the second update to the question, I see some more panels getting populated when I used the dashboard from https://github.com/letgoapp/krakend-influx/blob/master/examples/grafana-dashboard.json
This was a huge lead. Later I went on to compare the specific JSON blocks within the working dashboard and the one which was missing some charts. I realized all of those panels had "datasource": null. And the ones which were working had the "datasource": "InfluxDB",
I updated the JSON definition of dashboard and viola !! All the panels started showing the charts.
PS: If you see any datasource as null, or which does not correspond to your influx DB datasource, you should update it to use influx db data source which is defined in the datasources section on the admin panel.

Grafana Geo Map Dashboard - Geo IP Configuration

I have installed Grafana in my AKS cluster and now I want to create a Dashboard which shows the Geo location of the users connecting. So I found the below dashboard in Grafana website.
https://grafana.com/grafana/dashboards/12559
But some of the panels shows "No Data" in this dashboard. For instance, Requests per Country doesn't show any data. The query used by this panel is
sum by (geoip_country_code) (count_over_time({$label_name=~"$label_value", job=~"$job", instance=~"$instance"} | json | geoip_country_code != "" | __error__="" [$__interval]))
I searched regarding this, and as far as I know, we need "Geo Map" plugin installed in our Grafana Dashboard. But I see the plugin installed in Grafana.
So in this case, do I need to install any kind of agent or something to gather metrics regarding location? I also see in the grafana dashboard saying below.
I can't understand where I should put this configuration. Is these parameters should be placed in the Config Map or else the NGINX Ingress Controller Annotation section?

How to configure Jaeger data source in Grafana properly?

I would like to configure Jaeger data source in Grafana. I have Loki, Jaeger, Grafana installed in Kubernetes cluster. All services are up and running. Then, I navigate to Grafana to set up a new data source for Jaeger. Specify Jaeger url (http://jaeger-tracing-query.monitoring.svc.cluster.local:16687), click on [Save & test] button and the 'Data source connected, but no services received. Verify that Jaeger is configured properly.' error message is shown. If I navigate to Jaeger UI, I can clearly see 2 services.
Could you please guide me on what is probably missing in the configuration?

16687 is admin port. Use port 80 to configure the data source in grafana.
it should be like jaeger-tracing-query.monitoring.svc.cluster.local:80

I was scratching my head over this issue.
All the documents say you need to connet to {pod}.{namespace}:{port}
I noticed if you go to the Jaeger UI it's querying for the services on this endpoint
http://localhost:16686/jaeger/api/services
So I went back to Grafana and set the URL to http://jaeger-query:16686/jaeger

Grafana custom data source

help me with this fairly new to Grafana
Things accomplished so far:
1. Managed to get a JSON output from CSV (basically looking to plot transaction availability based on success/ fail criteria)
2. Wanted a Prometheus source imported
Can you help me how I proceed in either building a Graphite dashboard (or) Prometheus? Much appreciated. thanks

In order to use Prometheus datasource, you need to follow below steps
Start Prometheus server. By default, it listens at 9090 port.
login to Grafana console, go to option - Add datasource
select Prometheus as type of datasource from the drop down.
Insert the Prometheus URL (You got this in step 1)
Access should be "direct" if you aren't using any proxy to establish connection.
Hit save & test.