Grafana HTTP Error Bad Gateway and Templating init failed errors - kubernetes

Question

Use helm installed Prometheus and Grafana on minikube at local.
$ helm install stable/prometheus
$ helm install stable/grafana
Prometheus server, alertmanager grafana can run after set port-forward:
$ export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 9090
$ export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 9093
$ export POD_NAME=$(kubectl get pods --namespace default -l "app=excited-crocodile-grafana,component=grafana" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 3000
Add Data Source from grafana, got HTTP Error Bad Gateway error:
Import dashboard 315 from:
https://grafana.com/dashboards/315
Then check Kubernetes cluster monitoring (via Prometheus), got Templating init failed error:
Why?

Answer 1

In the HTTP settings of Grafana you set Access to Proxy, which means that Grafana wants to access Prometheus. Since Kubernetes uses an overlay network, it is a different IP.
There are two ways of solving this:
Set Access to Direct, so the browser directly connects to Prometheus.
Use the Kubernetes-internal IP or domain name. I don't know about the Prometheus Helm-chart, but assuming there is a Service named prometheus, something like http://prometheus:9090 should work.

Answer 2

I turned off the firewall on appliance, post that adding http://prometheus:9090 on URL did not throw bad gateway error.

Answer 3

I was never able to find a "proper" fix, but I found a workaround:
apiVersion: v1
kind: Service
metadata:
labels:
prometheus: k8s
name: prometheus-k8s
namespace: monitoring
spec:
selector:
app: prometheus
prometheus: k8s
sessionAffinity: ClientIP
clusterIP: None
By setting the clusterIP to None, the service changes to "Headless" mode, which means that requests are sent directly to a random one of the pods in that service/cluster. More info here: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services
There's probably a better solution, but this is the only one I've found that actually works for me, with kube-prometheus. (I've tried docker-desktop, k3d, and kind, and all of them have the same issue, so I doubt it's the emulator's fault; and I stripped my config down to basically just kube-prometheus, so it's hard to understand where the problem lies, but oh well.)