Grafana pod crashloopbackoff after updating domain and port - kubernetes

Im integrating keycloak OAuth login to Grafana in Openshift.
Keycloak Image Version - quay.io/keycloak/keycloak:15.0.2
Grafana Image Version - grafana/grafana:7.1.5
Kubernetes Version - v1.21
Openshift Version - 4.8
The keyclaok is exposed at Route: http://keycloak-keycloak.router.default.svc.cluster.local.167.254.203.104.nip.io
The Grafana is exposed at Route: https://grafana.router.default.svc.cluster.local.167.254.203.104.nip.io
The keycloak is created with Realm - devops and client - grafana and these values are added to Grafana deployment as Environmental variable as follows
GF_AUTH_GENERIC_OAUTH_NAME=OAuth
GF_AUTH_GENERIC_OAUTH_ENABLED=true
GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=33341c00-daf2-4170-a66f-c2c7c23ad151
GF_AUTH_GENERIC_OAUTH_AUTH_URL=http://keycloak-keycloak.router.default.svc.cluster.local.167.254.203.104.nip.io/auth/realms/devops/protocol/openid-connect/auth
GF_AUTH_GENERIC_OAUTH_TOKEN_URL=http://keycloak-keycloak.router.default.svc.cluster.local.167.254.203.104.nip.io/auth/realms/devops/protocol/openid-connect/token
GF_AUTH_GENERIC_OAUTH_API_URL=http://keycloak-keycloak.router.default.svc.cluster.local.167.254.203.104.nip.io/auth/realms/devops/protocol/openid-connect/userinfo
GF_AUTH_GENERIC_OAUTH_TLS_SKIP_VERIFY_INSECURE=true
With this when I browse the Grafana route and click on Sign in with OAuth I get error in screen - Inavalid Prameter Redirect URI. In the keycloak logs i see error - error=invalid_redirect_uri, redirect_uri=http://localhost:3000/login/generic_oauth.
Its taking localhost:3000 as Redirect URI, but i have specified the right redirect URI in client section of keycloak web i.e, : https://grafana.router.default.svc.cluster.local.167.254.203.104.nip.io/*.
While debugging this issue came across a similar issue : https://github.com/cetic/helm-fadi/issues/39.
Hence I have added these two environmental variables to Grafana:
GF_SERVER_DOMAIN=grafana.router.default.svc.cluster.local.167.254.203.104.nip.io
GF_SERVER_HTTP_PORT=80
This has resulted in pod crashloopbackoff with following error in logs:
t=2021-12-31T11:29:26+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:80 protocol=https subUrl=//grafana.router.default.svc.cluster.local.167.254.203.104.nip.io:80 socket=
t=2021-12-31T11:32:53+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated"
t=2021-12-31T11:32:53+0000 lvl=info msg="Stopped Stream Manager"
Please help me to proceed.

It is in the Grafana documentation:
You may have to set the root_url option of [server] for the callback URL to be correct.
So remove GF_SERVER_DOMAIN,GF_SERVER_HTTP_PORT and configure GF_SERVER_ROOT_URL properly (I guess correct value for your setup is https://grafana.router.default.svc.cluster.local.167.254.203.104.nip.io)
Grafana will be able to generate correct redirect URL with this setup.

Related

Keycloak admin console not working internal it gives "Loading the admin console"

I am having keycloak installed and working in Prod and we are currently migrating to Kubernetes(EKS) so I used Bitnami chart and used the same config as in Prod but admin console not working internally I tried version 19.0.0 and 20.0.0
here is the keycloak config
KC_HTTPS_KEY_STORE_FILE: **
KC_PROXY: edge
KC_DB: postgres
KC_DB_URL_HOST: **
KC_DB_USERNAME: ***
KC_HOSTNAME: public_url
KEYCLOAK_ADMIN: admin
KEYCLOAK_HOSTNAME: public_url
PROXY_ADDRESS_FORWARDING: true
KK_TO_RMQ_URL: **
KK_TO_RMQ_USERNAME: **
KK_TO_RMQ_PASSWORD: **
KEYCLOAK_IDENTITY_URL: **
KC_HOSTNAME_STRICT: true
KC_HOSTNAME_ADMIN: internal_url
in the Docker file I give start --proxy edge
here is the error that comes in the browser
Timeout when waiting for 3rd party check iframe message.
Error: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
I am trying to get keycloak admin console to work internally but it keeps loading forever without opening the admin dashboard
I also tried the fix on the thread Keycloak admin console loading indefinitely
but it didn't help
I am able to fix the issue, and here is the solution explained:
I removed the admin console internal connection configuration to dig more
I was using Bitnami chart for keycloak in the Bitnami chart there is a config like that
containerSecurityContext:
enabled: true
runAsUser: 1001
runAsNonRoot: false
as keycloak needs to create tmp folder for caching the css and js files it was not able to create it
KC-SERVICES0075: Failed to get theme request: java.lang.RuntimeException: Temporary directory /opt/keycloak/bin/../data/tmp does not exist and it was not possible to create it
so I disabled this feature in bitnami
containerSecurityContext:
enabled: false
set the user in the image as keycloak or any other privileged user
and that's it, now working as a charm.

Grafana on ECS cluster return not allowed origin

Created ECS cluster let's call it tools.
tools cluster have 2 services:
sso-proxy
grafana - opensource
Going through sso-proxy to get grafana.
when trying to enter credentials to grafana i got the following error:
When incognito mode the login working for the first time.
This is the ECS logs:
Please someone can help.
Grafana must receive proper Host header in the request header. I guess your "sso-proxy" doesn't do that. You didn't provide reproducible example (why not, when you want to help), so it is only a guess.
Lazy and insecure workaround will be Grafana downgrade to version 8.3.4-, where CSRF fix for CVE-2022-21703 is not included.

How to configure Jaeger data source in Grafana properly?

I would like to configure Jaeger data source in Grafana. I have Loki, Jaeger, Grafana installed in Kubernetes cluster. All services are up and running. Then, I navigate to Grafana to set up a new data source for Jaeger. Specify Jaeger url (http://jaeger-tracing-query.monitoring.svc.cluster.local:16687), click on [Save & test] button and the 'Data source connected, but no services received. Verify that Jaeger is configured properly.' error message is shown. If I navigate to Jaeger UI, I can clearly see 2 services.
Could you please guide me on what is probably missing in the configuration?
16687 is admin port. Use port 80 to configure the data source in grafana.
it should be like jaeger-tracing-query.monitoring.svc.cluster.local:80
I was scratching my head over this issue.
All the documents say you need to connet to {pod}.{namespace}:{port}
I noticed if you go to the Jaeger UI it's querying for the services on this endpoint
http://localhost:16686/jaeger/api/services
So I went back to Grafana and set the URL to http://jaeger-query:16686/jaeger

websphere application server behind ingress redirects to dns:port

I am trying to run websphere behind ingress.
I have successfully configured the ingress for it and I am able to access the console.
The ip on which I am able to access the console is
https://mydomain/ibm/console/logon.jsp
Note: I have a domain , for which I created a A record.
When I login into the console ,
I am redirected to
https://mydomain:9043/ibm/console/
This page does not exist.
If I explicitly run the URL https://dns/ibm/console/login.do?action=secure. My application works fine.
Can someone tell me where is the configuration needed so that it is not redirected to dns:port?
I have created all the config following the URL:
Problem configuring websphere application server behind ingress
The request is http://hostname/ibm/console/logon.jsp and it returns http://hostname:9080/ibm/console/logon.jsp
This is causing confusion and thus we are facing an issue.
The port which our was app uses is controlled by two properties:
trusthostheaderport = true
com.ibm.ws.webcontainer.extractHostHeaderPort = true
These can be created in the WAS Admin console under:
Servers > Server Types > WebSphere application servers > [server_name] > Web Container Settings > Web container > Additional Properties > Custom Properties
Setting both of these properties with a value of "true" should force WAS to use the front-end port (from the Host header) instead of the Webcontainer port.
Note: You need to restart the service after the change ( restart in case of docker containers)
Please find the documentation here.

Grafana datasources connection using proxy

How Grafana can be configured to connect to the data sources (for example Graphite) via proxy?
Grafana -> Proxy -> Graphite
When adding Graphite data source in Grafana there is an option to choose access with proxy (Proxy=Grafana backend will proxy the request) but there is no place in the configureation.ini to specify the IP of the proxy to be used for backend data transfer.
In the ini file there is location to specify "authentication proxy" for accessing the Grafana UI but can this be used for proxy between Grafana and Graphite?
[auth.proxy]
enabled = true
header_name = X-WEBAUTH-USER
header_property = username
auto_sign_up = true
Where the proxy details between Grafana and its data sources can be configured?
Thanks for the reply of AussieDan
I have added the environment variables :
set HTTP_PROXY="http://proxy_IP:proxy_port"
set HTTPS_PROXY="https://proxy_IP:proxy_port"
And started the Grafana in same command prompt .
When trying to save the Graphite data source in Grafana UI, it says in the log "proxy error: Service Unavailable". Telnet to the proxy on same port works fine so the port is open.
` 2016/12/13 15:55:05 http: proxy error: Service Unavailable ←[31mEROR←[0m[12-13|15:55:05] Request Completed ←[31mlogger←[0m=context ←[31muserId←[0m=1 ←[31morgId←[0m=1 ←[31muname←[0m=admin ←[31mmethod←[0m=GET ←[31mpath←[0m=/metrics/find ←[31mstatus←[0m=500 ←[31mremote_addr←[0m=IP ←[31mtime_ns←[0m=15ns ←[31msize←[0m=0 `
Is there anything additional that needs to be configured for proxy communication?
Grafana defines a new user named grafana with its own environment, as stated here under Start the server (init.d service) and Environment file sections.
The environment file on ubuntu is located in /etc/default/grafana-server
Paste your proxy environment variable definitions in this file as so:
https_proxy=http://<ip>:<port>/
http_proxy=http://<ip>:<port>/
no_proxy=localhost,127.0.0.0/8,::1,<other ip/networks to ignore>
Since the grafana backend is written in go, you can use the HTTP_PROXY="http://proxyIp:proxyPort" environment variable to specify a proxy for it to use for all outgoing requests.