Kong OIDC: discovery url timeout - keycloak

Question

I am building the kong + kong-oidc + keycloak stack for my application. All of them are deployed into minikube cluster.
What I want to achieve is to perform Open ID Connect using Google Identity Provider.
I was referring to this blog post to set the things up https://www.jerney.io/secure-apis-kong-keycloak-1/.
The only difference is that instead of docker-compose, I used minikube to build the stacks up and the keycloak version is image: jboss/keycloak:10.0.1.
Below are the steps I have taken:
Setup Keycloak exposed as NodePort
Create keycloak realm + client
Setup Kong
Install kong-oidc
Define ingress gateway
Configure Kong ingress using OIDC configuration + keycloak client's details from step 2
When I setup it up using Ip address, everything works well. I managed browser my application, and it was redirected to Google Sign in page.
Next, when I put a bind my Ip address with a domain name + SSL encryption. The kong-oidc unable to reach my keycloak discovery URL.
below are the error I got from my browser. Due to the privacy, I have change all domain to example.com
I browsed https://example.com/data
It waited for around 1 min, then the results shows this:
accessing discovery url (https://keycloak.example.com/auth/realms/example/.well-known/openid-configuration) failed: timeout
I have a domain says example.com, and a subdomain keycloak.example.com.
Once I have setup Keycloak behind the Kong proxy, I have removed the NodePort settings.
Below are my ingress definitions:
Ingress for example.com
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gateway-api
annotations:
kubernetes.io/tls-acme: "true"
cert-manager.io/cluster-issuer: letsencrypt-prod
konghq.com/plugins: oidc
spec:
tls:
- secretName: gateway-api
hosts:
- example.com
rules:
- host: example.com
http:
paths:
- path: /data
pathType: Prefix
backend:
serviceName: data-service
servicePort: 8080
Ingress for keycloak.example.com
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gateway-keycloak
annotations:
konghq.com/plugins: transform-function
spec:
rules:
- host: keycloak.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
serviceName: admin-keycloak
servicePort: 8080
The transform-function was used to override the default port forward to ensure the Keycloak admin console able to access through keycloak.example.com using https
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: transform-function
disabled: false
config:
functions:
- ngx.var.upstream_x_forwarded_port=443
plugin: post-function
My OIDC plugin definition:
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: oidc
disabled: false
config:
client_id: <client_id>
client_secret: <client_secret>
ssl_verify: "true"
discovery: https://keycloak.example.com/auth/realms/example/.well-known/openid-configuration
introspection_endpoint: http://admin-keycloak:8080/auth/realms/example/protocol/openid-connect/token/introspect
plugin: oidc
Also, I have set the keycloak environment with PROXY_ADDRESS_FORWARDING: "true"
The certificate also installed correctly, I can browser keycloak admin console using https.
Below are the logs from kong pod.
Logs from proxy container
2020/05/11 16:12:16 [error] 22#0: *87800 [lua] openidc.lua:483: openidc_discover(): accessing discovery url (https://keycloak.example.com/auth/realms/example/.well-known/openid-configuration) failed: timeout, client: 172.17.0.1, server: kong, request: "GET /data HTTP/2.0", host: "example.com"
172.17.0.1 - - [11/May/2020:16:12:16 +0000] "GET /data HTTP/2.0" 500 130 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36"
No error messages from ingress-controller container.
I have run out of ideas on how to troubleshoot this error. Can anyone help on this?