Accessing the Grafana API through k8s proxy - kubernetes

Question

I am running Grafana v6.2.4 in kubernetes, using basic auth. I want to use the k8s proxy for testing (i.e. kubectl proxy --port=8080). I have changed the GF_SERVER_ROOT_URL environment variable to:
{
"name": "GF_SERVER_ROOT_URL",
"value": "http://localhost:8080/api/v1/namespaces/my-namespace/services/grafana-prom:80/proxy/"
}
This allows me to log in and use Grafana through my browser at http://localhost:8080/api/v1/namespaces/my-namespace/services/grafana-prom:80/proxy/.
However, I want to use it via the API. If I send a request to http://localhost:8080/api/v1/namespaces/my-namespace/services/grafana-prom:80/proxy/api/dashboards/db I get back
{
"message": "Unauthorized"
}
However, if I set up a kubernetes port forward and send the identical request to http://localhost:30099/api/dashboards/db, then it succeeds.
Is there a different environment variable aside from GF_SERVER_ROOT_URL that I should be changing so that the API server root goes through the k8s proxy, i.e. http://localhost:8080/api/v1/namespaces/my-namespace/services/grafana-prom:80/proxy/api/dashboards/db? I have looked here but couldn't find it.
Otherwise what is the correct way to access the API through the k8s proxy?
I should add that I am specifically trying to use kubetctl proxy as an alternative to kubectl port-forward so I'm hoping to find an alternative to the suggestion here https://stackoverflow.com/a/45189081/1011724

Answer 1

I tried to replicate this in minikube and i might have found the reason for your requests through the API server proxy (using kubectl proxy) not getting authorized correctly.
Running the following curl-command:
curl -H "Authorization: Bearer <TOKEN>" http://localhost:8080/api/v1/namespaces/my-namespace/services/grafana-prom:80/proxy/api/dashboards/home
and using tcpdump to capture the requests in the Pod with tcpdump -vvvs 0 -l -A -i any yielded the following result(s):
GET /api/dashboards/home HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.58.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 127.0.0.1, 192.168.99.1
X-Forwarded-Uri: /api/v1/namespaces/my-namespace/services/grafana-prom:80/proxy/api/dashboards/home
This GET request doesn't have the Authorization header (resulting in a 401 Unauthorized) so basically the API server seem to strip this HTTP header as it passes the request down to the Pod.
If i instead use kubectl port-forward -n my-namespace svc/grafana-prom 8080:80 the GET request now looks like this:
GET /api/dashboards/home HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.58.0
Accept: */*
Authorization: Bearer <TOKEN>
When writing this answer i found this issue in the k/k repo #38775, to qoute one of the comments:
this is working as expected. "proxying" through the apiserver will not get you standard proxy behavior (preserving Authorization headers end-to-end), because the API is not being used as a standard proxy
This basically means that kubectl proxy will not work when trying to authorize through it, it's not a "regular" reverse proxy and probably for a good reason won't preserve Authorization headers.
Note that i tested both token and basic authorization using curl although token based auth is used above.
Hopefully this clear things up a bit!