K3s Allow Unauthenticated Access to OIDC Endpoints - kubernetes

Question

Goal
I want to allow unauthenticated access to the following OIDC endpoints in my K3s cluster (from other pods inside the cluster mostly but access from outside is also acceptable):
https://kubernetes.default.svc/.well-known/openid-configuration
https://kubernetes.default.svc/openid/v1/jwks
Problem
By default, Kubernetes requires an authorization token for accessing those endpoints and despite my efforts to enable unauthenticated access, I cannot seem to get the unauthenticated access to work.
What I have tried
According to the Kubernetes documentation on Service account issuer discovery, one must create a ClusterRoleBinding that maps the ClusterRole system:service-account-issuer-discovery to the Group system:unauthenticated.
I also found this helpful example for a different use-case but they're exposing the OIDC endpoints anonymously just like I want to do:
OIDC issuer discovery for Kubernetes service accounts
Based on both of those, I created my ClusterRoleBinding with:
kubectl create clusterrolebinding service-account-issuer-discovery-unauthenticated --clusterrole=system:service-account-issuer-discovery --group=system:unauthenticated
This results in the following spec:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: "2022-11-28T15:24:41Z"
name: service-account-issuer-discovery-unauthenticated
resourceVersion: "92377634"
uid: 75402324-a8cf-412f-923e-a7a87ed082c2
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:service-account-issuer-discovery
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:unauthenticated
I also confirmed RBAC is enabled by running this which showed an entry:
kubectl api-versions | grep 'rbac.authorization.k8s.io/v1'
Unfortunately, despite creating that ClusterRoleBinding, it still seems to be requiring an authorization token to access the OIDC endpoints. For example, when I call from outside the cluster:
curl -vk https://my-cluster-hostname:6443/.well-known/openid-configuration
Output:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}
Accessing from the Ingress controller inside the pod (by execing into it), I get the same error.
curl -vk https://kubernetes.default.svc/.well-known/openid-configuration
I am using K3s for this cluster... Is there something special about K3s that I have not accounted for? Do I need to do something to get this ClusterRoleBinding to take effect? Something else I missed?

Answer 1

The issue was that the --anonymous-auth api server setting is set to false by default.
I was able to adjust this with my already-installed K3s server nodes by editing the systemd unit for the k3s service. Step-by-step guide:
Edit the systemd unit with:
sudo vim /etc/systemd/system/k3s.service
You'll see the K3s unit:
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target
[Install]
WantedBy=multi-user.target
[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
server \
'--cluster-init' \
'--disable=traefik' \
'--write-kubeconfig-mode' \
'644' \
Add the last two lines to the ExecStart:
ExecStart=/usr/local/bin/k3s \
server \
'--cluster-init' \
'--disable=traefik' \
'--write-kubeconfig-mode' \
'644' \
'--kube-apiserver-arg' \
'--anonymous-auth=true' \
Reload the systemd unit:
sudo systemctl daemon-reload
Finally, restart the k3s service:
sudo systemctl restart k3s
References:
Control Plane Execution and Arguments - You can see the default for K3s there
API discovery roles - also mentions this flag at the top