Keycloak on Kubernetes high availability cluster (with ldap as user federation) using codecentrics Helm Charts - kubernetes

Question

We wanted to set up a high available Keycloak cluster on Kubernetes (with ldap as a user federation). We decided to use codecentrics helm charts since we were trying them out for a single Keycloak instance setup and that worked well. For the cluster we ran into a few issues while trying to set up everything correctly and didn't find the best sources in the wide internet. Therefore I decided to write a short summary what our main issues where and how we got through them.
Solutions to our problems where described on this website (amongst others), but things are described kind of very briefly and felt partly incomplete.
Issues we faced where:
Choosing the correct jgroups.discoveryProtocol
Adding the correct discoveryProperties
Parts that need to be overridden in your own values.yaml
Bonus issues (we already faced with the single instance setup):
Setting up an truststore to connect ldap as a user federation via ladps
Adding a custom theme for keycloak
I will try and update this if things change due to codecentrics updating their helm charts.
Thanks to codecentrics for providing the helm charts by the way!

Answer 1

Disclaimer:
This is the way we set it up - I hope this is helpful, but I do not take responsibility for configuration errors and resulting security flaws. Also we went through many different sources on the internet, I am sorry that I can't give credits to all of them, but it has been a few days since than an I can't get them together anymore...
CODECENTRIC CHART VERSION < 9.0.0
The main issues:
1. Choosing the correct jgroups.discoveryProtocol:
I will not explain things here but for us the correct protocol to use was org.jgroups.protocols.JDBC_PING. Find out more about the protocols (and general cluster setup) here.
discoveryProtocol: org.jgroups.protocols.JDBC_PING
With JDBC_PING jgroups will manage instance discovery. Therefore and for caching user sessions the database provided for keycloak will be enhanced with extra tables, e.g. JGROUPSPING.
2. Setting up the discoveryProperties:
This needs to be set to
discoveryProperties: >
"datasource_jndi_name=java:jboss/datasources/KeycloakDS"
to avoid an error like:
java.lang.IllegalStateException: java.lang.IllegalArgumentException:
Either the 4 configuration properties starting with 'connection_' or
the datasource_jndi_name must be set
3. Other parts that need to be set (as mostly described in the readme of codecentrics github and in the comments of the values.yaml in github as well):
setting the clusterDomain according to your cluster
setting the number of replicas greater than 1 to enable clustering
setting the service.type: We went with ClusterIP but it also can work with other setups like LoadBalancer depending on your setup
optional but recommended: Setting either maxUnavailable or minAvailable to always have sufficient pods available according to your needs.
setting up our Ingress (which looks pretty much standard):
ingress:
enabled: true
path: /
annotations: {
kubernetes.io/ingress.class: nginx
}
hosts:
- your.host.org
Bonus issues:
1. The truststore:
To have Keycloak communicate with ldap via ldaps we had to set up a truststore with the certificate of our ldap in it:
Receive the certificate from ldap and save it somewhere:
openssl s_client -connect your.ldap.domain.org < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /path/to/ldap.cert
Create a new keystore:
keytool -genkey -alias replserver \
-keyalg RSA -keystore /path/to/keystore.jks \
-dname "CN=AddCommonName, OU=AddOrganizationalUnit, O=AddOrganisation, L=AddLocality, S=AddStateOrProvinceName, C=AddCountryName" \
-storepass use_the_same_password \
-keypass use_the_same_password \
-deststoretype pkcs12
Add the downloaded certificate to the keystore:
keytool -import -alias ldaps -file /path/to/ldap.cert -storetype JKS -keystore path/to/keystore.jks
Type in the required password: use_the_same_password.
Trust the certificate by typing 'yes'.
Provide the keystore in a configmap:
kubectl create configmap cert-keystore --from-file=path/to/keystore.jks
Enhance your values.yaml for the truststore:
Add and mount the config map:
extraVolumes: |
- name: cert-keystore
configMap:
name: cert-keystore
extraVolumeMounts: |
- name: cert-keystore
mountPath: "/keystore/"
readOnly: true
Tell java tu use it:
javaToolOptions: >-
-[maybe some other settings of yours]
-Djavax.net.ssl.trustStore=/keystore/keystore.jks
-Djavax.net.ssl.trustStorePassword=<<keystore_password>>
Since we didn't want to upload the keystore password to git we added a step to our pipeline where it gets sed into the values.yaml, replacing the <<keystore_password>>.
2. Adding a custom theme:
Mainly we are providing a docker container with our custom theme in it:
extraInitContainers: |
- name: theme-provider
image: docker_repo_url/themeContainer:version
imagePullPolicy: IfNotPresent
command:
- sh
args:
- -c
- |
echo "Copying theme..."
cp -R /custom-theme/* /theme
volumeMounts:
- name: theme
mountPath: /theme
Add and mount the theme:
extraVolumes: |
- name: theme
emptyDir: {}
extraVolumeMounts: |
- name: theme
mountPath: /opt/jboss/keycloak/themes/custom-theme
You now should be able to choose the custom theme in the Keycloak admin UI via Realm Settings -> Themes.
CODECENTRIC CHART VERSION 9.0.0 to 9.3.2 (and maybe higher)
1. Clustering
We are still going with JDBC_PING since we had problems with DNS_PING as described in the Codecentric Repo readme:
extraEnv: |
## KEYCLOAK CONFIG
- name: PROXY_ADDRESS_FORWARDING
value: "true"
### CLUSTERING
- name: JGROUPS_DISCOVERY_PROTOCOL
value: org.jgroups.protocols.JDBC_PING
- name: JGROUPS_DISCOVERY_PROPERTIES
value: 'datasource_jndi_name=java:jboss/datasources/KeycloakDS'
- name: CACHE_OWNERS_COUNT
value: "2"
- name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
value: "2"
With the service set up as ClusterIP:
service:
annotations: {}
labels: {}
type: ClusterIP
loadBalancerIP: ""
httpPort: 80
httpNodePort: null
httpsPort: 8443
httpsNodePort: null
httpManagementPort: 9990
httpManagementNodePort: null
extraPorts: []
2. 502 Error Ingress Problem
We encountered a 502 error with Codecentrics chart 9.x.x for which fixing took a while to figure out. A solution for this is also described here, where we took our inspiration but for us the following ingress setup was enough:
ingress:
enabled: true
servicePort: http
# Ingress annotations
annotations: {
kubernetes.io/ingress.class: nginx,
nginx.ingress.kubernetes.io/proxy-buffer-size: 128k,
}
CODECENTRIC CHART VERSION 9.5.0 (and maybe higher)
Updating to 9.5.0 needs to be tested. Especially if desired to go with KUBE_PING and maybe even Autoscaling.
I will update after testing if something changed significantly.