I recently upgraded my Grafana to v7.0.3 and started the image-rendering service as a separate pod in my k8 cluster.
I have specified both GF_RENDERING_SERVER_URL and GF_RENDERING_CALLBACK_URL
My Grafana is configured to use the active directory (AuthN). Only authenticated users can see dashboards.
Now the problem is when my Image rendering service calls for Grafana chart I think as it is behind AD; it fails to get it (there was http 401 as well)
Can someone suggests what am I missing/how can I pass authentication details?
t=60&timezone=Europe%2FLondon&url=http%3A%2F%2Fmobile-grafana.mobile-grafana.svc.cluster.local%3A3000%2Fd-solo%2F000000017%2Fjenkins-performance-and-health-overview%3ForgId%3D1%26refresh%3D1m%26from%3D1591535203773%26to%3D1591546003773%26var-node%3Djenkins-stg.k8s.mobile.sbx.zone%26panelId%3D4%26width%3D1000%26height%3D500%26tz%3DEurope%252FLondon%26render%3D1&width=1000" t=2020-06-07T16:06:45+0000 lvl=eror msg="Remote rendering request failed" logger=rendering renderer=http error="403 Forbidden"
t=2020-06-07T16:06:45+0000 lvl=eror msg="Rendering failed." logger=context userId=2 orgId=1 uname="Pankaj Sainic" error="Remote rendering request failed. 403: 403 Forbidden" ```
If you are using a proxy, you have to add this "NO_PROXY" property to make it works !
NO_PROXY:0.0.0.0,127.0.0.1,renderer,grafana
renderer and grafana are here the service name declared in my docker-compose file
Related
I'm recently started a kubernetes cluster. I'm be able to access it in my browser. I was trying to visualize my cluster resources (pods,...). I'm using prometheus for that and it gives me the error 404 not found. Can aynone help?
enter image description here
enter image description here
I want the status up and be able to access it.
I am working on creating a new request in AWS API Gateway. I am having issues with a 404 not found on the URL request.
The request (had to create fake one for the question):
GET https://hello.stackoverflow.com/services/misc/myroute/v1/swagger.json
I created a route in API Gateway ANY /services/misc/myroute/{proxy+}
I attached the route to a Load Balancer Listener integration
I set up the listener rule in the Load Balancer:
IF Path is /services/misc* Then Forward to Target
IF Requests otherwise not routed Then Forward to Default
Created logs for this system in the AWS API Gateway: Monitor -> Logging -> Set Log Destination
Set variables for the log format using the $context variables, Context Variables
Ex Log:
{ "requestId":"QWRHQKWFHWAFZ=",
"routeKey":"ANY /services/misc/myroute/{proxy+}",
"path":"/services/misc/myroute/v1/swagger.json",
"domain":"hello.stackoverflow.com",
"domain_prefix":"hello",
"httpMethod":"GET", "status":"404","protocol":"HTTP/1.1", "endpoint":-" }
One final check I have done to make sure its completing its "route" was see the requests in the monitoring and seeing the 4xx come from this ALB listener.
I can send the request via localhost and get a response with the json body
GET https://localhost:8080/v1/swagger.json --> Status 200 OK with body filled
In my quest to solve the issue, it has lead me to many older (2019) stack overflow questions that seem to be outdated with the AWS Console, same with the AWS documentation. See links below...
AWS API Gateway Method request path parameter not working
AWS API Gateway 404 page not found error when invoking endpoint url
AWS API Gateway Method request path parameter not working
With this being my first project in the AWS cloud space, I am not sure where else to turn. My guess would be the authentication headers from the API Gateway are being lost, but not sure where I can see this loss happening.
From my understanding of how the AWS Request Flow goes, I created this diagram:
After installing microk8s and then enabling kubeflow I'm given the username, password and link to Kubeflow dashboard. Then I access the dashboard as expected and all is well. BUT after restarting my machine and executing microk8s start I can no longer get to the kubeflow dashboard.
All the pods start fine and then I go to access the dashboard and get:
Access to 10.64.140.44.nip.io was denied
You don't have authorisation to view this page.
HTTP ERROR 403
Looking at the kubernetes logs for the pod/container oidc-gatekeeper-xxxxx / oidc-gatekeeper I have:
level=error msg="Failed to exchange authorization code with token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Invalid client credentials.\"}" ip=10.1.252.88 request="/authservice/oidc/callback?code=ipcb55gymqsy5pcgjn7eaenad&state=MTYyMjYzNjE4OHxFd3dBRURoMVZtSm9Wak4yUXpWQlYxZ3pPVWs9fPTKezGok06ig6bjtYvWt9sqhzaCpO_xhSMeTUFDL81j"
And for pod/container dex-auth-5d9bf87db9-rjtm8 / dex-auth:
level=info msg="invalid client_secret on token request for client: authservice-oidc"
Only by removing microk8s altogether and reinstalling everytime I restart my machine can I get this working again which is obviously not workable.
Any help would be greatly appreciated.
I've managed to resolve the issue but I'm not 100% sure which action resolved it.
I tried using Firefox rather than Chrome and noticed some documentation used IP http://10.64.140.43.nip.io/ rather than http://10.64.140.44.nip.io/.
Having been refused access as above for http://10.64.140.44.nip.io/ I found http://10.64.140.43.nip.io/ took me straight into the dashboard.
I restarted my machine to see if it was just the IP (note: checking "microk8s kubectl get services -n kubeflow" specified 10.64.150.44 as the external IP), but this time http://10.64.140.44.nip.io/ just gave me the dex log in screen and after logging in took me to the dashboard without issue.
Perhaps I just did something wrong somewhere, I'm not sure and can't check now it works as it should. Apologise if you get here with the issue and this doesn't help.
I had a similar error. Solution for me was to enable dns, istio, and storage first. Wait until the pods were running, and then enable Kubeflow. Then make sure to port-forward using the istio-system namespace with the istio-ingressgateway pod. Kubeflow also makes a istio-ingressgateway pod, but connecting to that yielded the error. Per Kubeflow guide
I have installed the Kubernetes dashboard on the cloud server but I can not access the dashboard publicly.
While checking namespace, it shows running.
How I can fix such an issue or can anybody share me how to expose the Kubernetes dashboard publicly?
while accessing publicly it throws an error message can anybody help me on this case?
"forbidden: User \"system:anonymous\" cannot get path \"/\"
I have KeyCloak Gateway running successfully locally providing Google OIDC authentication for the Kubernetes dashboard. However using the same settings results in an error when the app is deployed as a pod in the cluster itself.
The error I see when the Gateway is running in a K8S pod is:
unable to exchange code for access token {"error": "invalid_request: Credentials in post body and basic Authorization header do not match"}
I'm calling the gateway with the following options:
--enable-logging=true
--enable-self-signed-tls=true
--listen=:443
--upstream-url=https://mydashboard
--discovery-url=https://accounts.google.com
--client-id=<client id goes here>
--client-secret=<secret goes here>
--resources=uri=/*
With these settings applied to a container in a pod I can browse to the Gateway, am redirected to Google to log in, and then am redirected back to the Gateway where the error above is generated.
What could account for the difference between running the application locally and running it in a pod that would generate the above error?
This turned out to be a copy/paste fail in the end, with the client secret being incorrect. The error message wasn't much help here, but at least it was a simple fix.