I have KeyCloak Gateway running successfully locally providing Google OIDC authentication for the Kubernetes dashboard. However using the same settings results in an error when the app is deployed as a pod in the cluster itself.
The error I see when the Gateway is running in a K8S pod is:
unable to exchange code for access token {"error": "invalid_request: Credentials in post body and basic Authorization header do not match"}
I'm calling the gateway with the following options:
--enable-logging=true
--enable-self-signed-tls=true
--listen=:443
--upstream-url=https://mydashboard
--discovery-url=https://accounts.google.com
--client-id=<client id goes here>
--client-secret=<secret goes here>
--resources=uri=/*
With these settings applied to a container in a pod I can browse to the Gateway, am redirected to Google to log in, and then am redirected back to the Gateway where the error above is generated.
What could account for the difference between running the application locally and running it in a pod that would generate the above error?
This turned out to be a copy/paste fail in the end, with the client secret being incorrect. The error message wasn't much help here, but at least it was a simple fix.
Related
After installing microk8s and then enabling kubeflow I'm given the username, password and link to Kubeflow dashboard. Then I access the dashboard as expected and all is well. BUT after restarting my machine and executing microk8s start I can no longer get to the kubeflow dashboard.
All the pods start fine and then I go to access the dashboard and get:
Access to 10.64.140.44.nip.io was denied
You don't have authorisation to view this page.
HTTP ERROR 403
Looking at the kubernetes logs for the pod/container oidc-gatekeeper-xxxxx / oidc-gatekeeper I have:
level=error msg="Failed to exchange authorization code with token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Invalid client credentials.\"}" ip=10.1.252.88 request="/authservice/oidc/callback?code=ipcb55gymqsy5pcgjn7eaenad&state=MTYyMjYzNjE4OHxFd3dBRURoMVZtSm9Wak4yUXpWQlYxZ3pPVWs9fPTKezGok06ig6bjtYvWt9sqhzaCpO_xhSMeTUFDL81j"
And for pod/container dex-auth-5d9bf87db9-rjtm8 / dex-auth:
level=info msg="invalid client_secret on token request for client: authservice-oidc"
Only by removing microk8s altogether and reinstalling everytime I restart my machine can I get this working again which is obviously not workable.
Any help would be greatly appreciated.
I've managed to resolve the issue but I'm not 100% sure which action resolved it.
I tried using Firefox rather than Chrome and noticed some documentation used IP http://10.64.140.43.nip.io/ rather than http://10.64.140.44.nip.io/.
Having been refused access as above for http://10.64.140.44.nip.io/ I found http://10.64.140.43.nip.io/ took me straight into the dashboard.
I restarted my machine to see if it was just the IP (note: checking "microk8s kubectl get services -n kubeflow" specified 10.64.150.44 as the external IP), but this time http://10.64.140.44.nip.io/ just gave me the dex log in screen and after logging in took me to the dashboard without issue.
Perhaps I just did something wrong somewhere, I'm not sure and can't check now it works as it should. Apologise if you get here with the issue and this doesn't help.
I had a similar error. Solution for me was to enable dns, istio, and storage first. Wait until the pods were running, and then enable Kubeflow. Then make sure to port-forward using the istio-system namespace with the istio-ingressgateway pod. Kubeflow also makes a istio-ingressgateway pod, but connecting to that yielded the error. Per Kubeflow guide
I am using Vault with postgres storage backend along with kv secret engine. I am uisng kubernetes auth method to get the vault token. I followed the below documentation to setup the vault with kubernetes
https://learn.hashicorp.com/tutorials/vault/kubernetes-minikube?in=vault/kubernetes
When I start the webapplication for the first time and try to retrieve the tokens it is working but when I delete the webapp deployment and try to deploy webapp again and try to retrieve the vault token again with the api
v1/auth/kubernetes/login
I get the following error
error: 400 Bad Request: [{"errors":["missing client token"]}
But the request has the jwt token of service account. Please see the below image
Due to this error Pod keeps restarting and all of a sudden after some time vault honours the request and returns the vault token.
This looks strange any reason for such behavior?
UPDATE:
This issue does not happen for consul backend
I recently upgraded my Grafana to v7.0.3 and started the image-rendering service as a separate pod in my k8 cluster.
I have specified both GF_RENDERING_SERVER_URL and GF_RENDERING_CALLBACK_URL
My Grafana is configured to use the active directory (AuthN). Only authenticated users can see dashboards.
Now the problem is when my Image rendering service calls for Grafana chart I think as it is behind AD; it fails to get it (there was http 401 as well)
Can someone suggests what am I missing/how can I pass authentication details?
t=60&timezone=Europe%2FLondon&url=http%3A%2F%2Fmobile-grafana.mobile-grafana.svc.cluster.local%3A3000%2Fd-solo%2F000000017%2Fjenkins-performance-and-health-overview%3ForgId%3D1%26refresh%3D1m%26from%3D1591535203773%26to%3D1591546003773%26var-node%3Djenkins-stg.k8s.mobile.sbx.zone%26panelId%3D4%26width%3D1000%26height%3D500%26tz%3DEurope%252FLondon%26render%3D1&width=1000" t=2020-06-07T16:06:45+0000 lvl=eror msg="Remote rendering request failed" logger=rendering renderer=http error="403 Forbidden"
t=2020-06-07T16:06:45+0000 lvl=eror msg="Rendering failed." logger=context userId=2 orgId=1 uname="Pankaj Sainic" error="Remote rendering request failed. 403: 403 Forbidden" ```
If you are using a proxy, you have to add this "NO_PROXY" property to make it works !
NO_PROXY:0.0.0.0,127.0.0.1,renderer,grafana
renderer and grafana are here the service name declared in my docker-compose file
I am running Kubeflow pipeline(docker approach) and cluster uses the endpoint to navigate to the dashboard. The Clusters is created followed by the instructions mentioned in this link Deploy Kubeflow. Everything is successfully created and the cluster generated the endpoints and its working perfectly.
Endpoint link would be something like this https://appname.endpoints.projectname.cloud.goog.
Every workload of the pipeline is working fine except the last one. In the last workload, I am trying to submit a job to the cloud-ml engine. But it logs shows that the application has no access to the project. Here is the full image of the log.
ERROR:
(gcloud.ml-engine.versions.create) PERMISSION_DENIED: Request had
insufficient authentication scopes.
ERROR:
(gcloud.ml-engine.jobs.submit.prediction) User
[clustername#project_name.iam.gserviceaccount.com]
does not have permission to access project [project_name]
(or it may not exist): Request had insufficient authentication scopes.
From the logs, it's clear that this service account doesn't have access to the project itself. However, I tried to give access for Cloud ML Service to this service account but still, it's throwing the same error.
Any other ways to give Cloud ML service credentials to this application.
Check two things:
1) GCP IAM: if clustername-user#projectname.iam.gserviceaccount.com has ML Engine Admin permission.
2) Your pipeline DSL: if the cloud-ml engine step calls apply(gcp.use_gcp_secret('user-gcp-sa')), e.g. https://github.com/kubeflow/pipelines/blob/ea07b33b8e7173a05138d9dbbd7e1ce20c959db3/samples/tfx/taxi-cab-classification-pipeline.py#L67
I'm building a CloudFoundry application in Bluemix using the API Management service. For this I'm following this tutorial http://www.ibm.com/developerworks/cloud/library/cl-bluemix-api-mgmt-app/index.html.
I can successfully create an API Plan for a custom REST API application (running on a Liberty server on Bluemix as well) and it is published.
I can also create a service in my Bluemix dashboard using the new Custom API, which I take as the API plan was successfully deployed on Bluemix.
Whenever I try to bind this new service I get the following message:
BXNUI2055E: Unable to connect to Cloud Foundry because of the
following exception: "Read timed out." If the problem persists, see
the Troubleshooting topics in the IBM Bluemix Documentation to check
service status, review troubleshooting information, or for information
about getting help.
From time to time I also get this message
The service broker returned an invalid response for the request to
https://apimasv1-stage.stage1.mybluemix.net/d118dceb-edbf-4a7f-9bab-d44371b0c9f9/privateservices/v2/service_instances/1a60830c-0796-4105-afb4-e3477424acf9/service_bindings/ebb853dc-ec88-4987-b8f2-e9acd38d1741.
Status Code: 502 Bad Gateway, Body: 502 Bad Gateway: Registered
endpoint failed to handle the request.
Also, I can open the API portal and see the services listed in there. However, whenever I try to test the service, I get the following error
A security error has occurred. If using a self-signed certificate on
your gateway, you will need to accept it in your browser, which you
can do by clicking the following link.
https://api.wawona.apim.ibmcloud.com/victorshmx1ibmcom-dev/sb/LibraryREST/rs/authors/1
Also, below in the response I get this message:
NetworkError: Failed to execute 'send' on 'XMLHttpRequest': Failed to
load
'https://api.wawona.apim.ibmcloud.com/victorshmx1ibmcom-dev/sb/LibraryREST/rs/authors/1'.
I must clarify this service (the liberty app) doesn't have any security constraint to access the REST service nor I added some kind of security in the API Management portal.
Another thing to clarify is that I can bind other services, but not this one.
Does anyone know how I can fix those problems? Is there a known issue with IBM API Management service?
this seems to be an error with the service API Management instance you are trying to bind to.
You could open a ticket to support team following the link you can find here:
https://developer.ibm.com/bluemix/support/
Click on 'Contact IBM' and open a 'Support ticket'