Cannot connect to Kubernetes Dashboard as non-admin user with kubectl proxy - kubernetes

I want to allow non-admin users to use the Kubernetes Dashboard to view the K8 objects in their namespaces. As cluster-admin, I have no issues connecting the the Kubernetes Dashboard using kubectl proxy. When I first attempted to access it with an application service account with read-only access to their entire namespace, I received the error below:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "services \"https:kubernetes-dashboard:\" is forbidden: User \"system:serviceaccount:ops-jenkins-lab:k8-dashboard-ops-jenkins-lab\" cannot get resource \"services/proxy\" in API group \"\" in the namespace \"kubernetes-dashboard\"",
"reason": "Forbidden",
"details": {
"name": "https:kubernetes-dashboard:",
"kind": "services"
},
"code": 403
}
I added additional RBAC roles to allow the application service account access to services and services/proxy in the kubernetes-dashboard namespace. Now I get the following error:
Forbidden (403): Http failure response for http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/api/v1/login: 403 Forbidden
If I create an ingress for the dashboard I can connect without out issue to the Kubernetes Dashboard using the same application service account and have access to view all the kubernetes objects within the namespace (once I switch from default to the correct namespace). I'd actually prefer to use the ingress but for some reason once I connect to the Kubernetes Dashboard via a browser it hijacks the ingress for all my other applications. No matter which ingress I try to connect to it automatically redirects me to the Kubernetes Dashboard. I have to clear all browser data to connect to other applications.
RBAC clusterrole and rolebinding:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
name: k8-dashboard
rules:
- apiGroups:
- extensions
- apps
resources:
- '*'
verbs:
- get
- list
- watch
- apiGroups:
- batch
resources:
- jobs
- cronjobs
verbs:
- get
- list
- watch
- apiGroups:
- autoscaling
resources:
- horizontalpodautoscalers
verbs:
- get
- list
- watch
- apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- get
- list
- watch
- apiGroups:
- '*'
resources:
- persistentvolumeclaims
verbs:
- get
- list
- watch
- apiGroups:
- '*'
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- '*'
resources:
- events
verbs:
- get
- list
- watch
- apiGroups:
- '*'
resources:
- configmaps
verbs:
- get
- list
- watch
- apiGroups:
- apps
resources:
- '*'
verbs:
- patch
- apiGroups:
- apps
resources:
- deployments/scale
verbs:
- update
- apiGroups:
- ""
resources:
- pods/attach
- pods/exec
- pods/log
- pods/status
- pods/delete
verbs:
- '*'
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- delete
- apiGroups:
- ""
resources:
- secrets
verbs:
- create
- get
- delete
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
annotations:
labels:
subjectName: k8-dashboard-sa
name: k8-dashboard-ops-jenkins-lab
namespace: ops-jenkins-lab
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: k8-dashboard
subjects:
- kind: ServiceAccount
name: k8-dashboard-ops-jenkins-lab
namespace: ops-jenkins-lab
So this leaves me with needing to connect to the Kubernetes Dashboard using kubectl proxy. I'm certain there's additional RBAC required when using kubectl proxy as a non-admin user; however, I have yet to figure it out. Any suggests?

Your ClusterRole is associated with RoleBinding, and in documentation you can read:
A RoleBinding can also reference a ClusterRole to grant the
permissions defined in that ClusterRole to resources inside the
RoleBinding’s namespace
This means that even though you are using ClusterRole, the permisions are limited to one namespace, which is ops-jenkins-lab in your case.
And a long as the dashboard you are trying to access is in kubernetes-dashboard namespace you won't be able to do it beacause your RoleBinding is in wrong namespace.
To allow k8-dashboard-ops-jenkins-lab serviceAccount to access resources in different namespace you should either create ClusterRoleBinding (clusterrolebindings are not namespaced) or (better option) RoleBinding in namespace you want to access (in your case that would be kubernetes-dashboard namespace).
Let me know if something needs more clarification.

Related

eks:cloud-controller-manager cluster role in EKS/K8s

I noticed that a new cluster role - "eks:cloud-controller-manager" appeared in our EKS cluster. we never created it.I tried to find origin/creation of this cluster role but not able to find it.
any idea what does "eks:cloud-controller-manager" cluster role does in EKS cluster?
$ kubectl get clusterrole eks:cloud-controller-manager -o yaml
kind: ClusterRole
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"eks:cloud-controller-manager"},"rules":[{"apiGroups":[""],"resources":["events"],"verbs":["create","patch","update"]},{"apiGroups":[""],"resources":["nodes"],"verbs":["*"]},{"apiGroups":[""],"resources":["nodes/status"],"verbs":["patch"]},{"apiGroups":[""],"resources":["services"],"verbs":["list","patch","update","watch"]},{"apiGroups":[""],"resources":["services/status"],"verbs":["list","patch","update","watch"]},{"apiGroups":[""],"resources":["serviceaccounts"],"verbs":["create","get"]},{"apiGroups":[""],"resources":["persistentvolumes"],"verbs":["get","list","update","watch"]},{"apiGroups":[""],"resources":["endpoints"],"verbs":["create","get","list","watch","update"]},{"apiGroups":["coordination.k8s.io"],"resources":["leases"],"verbs":["create","get","list","watch","update"]},{"apiGroups":[""],"resources":["serviceaccounts/token"],"verbs":["create"]}]}
creationTimestamp: "2022-08-02T00:25:52Z"
name: eks:cloud-controller-manager
resourceVersion: "762242250"
uid: 34e568bb-20b5-4c33-8a7b-fcd081ae0a28
rules:
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- update
- apiGroups:
- ""
resources:
- nodes
verbs:
- '*'
- apiGroups:
- ""
resources:
- serviceaccounts/token
verbs:
- create```
I tried to find this object in our Gitops repo but do not find it.
This role is created by AWS when you provision the cluster. This role is for the AWS cloud-controller-manager to integrate AWS services (eg. CLB/NLB, EBS) with Kubernetes. You will also find other roles like eks:fargate-manager to integrate with Fargate.

Kubernetes Role should grant access to all resources but it ignores some resources

The role namespace-limited should have full access to all resources (of the specified API groups) inside of a namespace. My Role manifest looks like this:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: namespace-limited
namespace: restricted-xample
rules:
- apiGroups:
- core
- apps
- batch
- networking.k8s.io
resources: ["*"] # asterisk to grant access to all resources of the specified api groups
verbs: ["*"]
I associated the Role to a ServiceAccount using a RoleBinding but unfortunately this ServiceAccount has no access to Pod, Service, Secret, ConfigMap and Endpoint Resources. These resources are all part of the core API group. All the other common Workloads work though. Why is that?
The core group, also referred to as the legacy group, is at the REST path /api/v1 and uses apiVersion: v1
You need to use "" for core API group.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: restricted-xample
name: namespace-limited
rules:
- apiGroups: ["", "apps", "batch", "networking.k8s.io"] # "" indicates the core API group
resources: ["*"]
verbs: ["*"]
To test the permission of the service account use below commands
kubectl auth can-i get pods --as=system:serviceaccount:restricted-xample:default -n restricted-xample
kubectl auth can-i get secrets --as=system:serviceaccount:restricted-xample:default -n restricted-xample
kubectl auth can-i get configmaps --as=system:serviceaccount:restricted-xample:default -n restricted-xample
kubectl auth can-i get endpoints --as=system:serviceaccount:restricted-xample:default -n restricted-xample
Just figured out, that it works when I omit the core keyword, like in this example. Following Role manifest works:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: namespace-limited
namespace: restricted-xample
rules:
- apiGroups: ["", "apps", "batch", "networking.k8s.io"]
resources: ["*"]
verbs: ["*"]
But why it does not work if I specify the core API group is a mystery to me.

kubernetes api servers showing down in prometheus (kube-state-metrics) --"forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",

I'm new to k8s, prometheus. I'm trying to collect the metrics of each pods with prometheus but unable to so because of the error:
API ERROR.
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
"reason": "Forbidden",
"details": {
},
"code": 403
}
system:anonymous means that an unauthenticated user is trying to get a resource from your cluster, which is forbidden. You will need to create a service account, then give that service account some permissions through RBAC, then make that service account to get the metrics. All that is documented.
As a workaround, you can do this:
kubectl create clusterrolebinding prometheus-admin --clusterrole cluster-admin --user system:anonymous
Now, note that this is a terrible idea, unless you are playing with kubernetes. With this permission you are giving any unauthenticated user total permissions into your cluster.
Create the following manifests:
ServiceAccount.yaml:
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/name: kube-state-metrics
name: kube-state-metrics
namespace: grafana
ClusterRole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/name: kube-state-metrics
name: kube-state-metrics
rules:
- apiGroups:
- ""
resources:
- configmaps
- secrets
- nodes
- pods
- services
- resourcequotas
- replicationcontrollers
- limitranges
- persistentvolumeclaims
- persistentvolumes
- namespaces
- endpoints
verbs:
- list
- watch
- apiGroups:
- extensions
resources:
- daemonsets
- deployments
- replicasets
- ingresses
verbs:
- list
- watch
- apiGroups:
- apps
resources:
- statefulsets
- daemonsets
- deployments
- replicasets
verbs:
- list
- watch
- apiGroups:
- batch
resources:
- cronjobs
- jobs
verbs:
- list
- watch
- apiGroups:
- autoscaling
resources:
- horizontalpodautoscalers
verbs:
- list
- watch
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
- apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- list
- watch
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests
verbs:
- list
- watch
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- list
- watch
- nonResourceURLs:
- "/metrics"
verbs:
- get
ClusterRoleBinding.yaml:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/name: kube-state-metrics
name: kube-state-metrics
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-state-metrics
subjects:
- kind: ServiceAccount
name: kube-state-metrics
namespace: grafana
And inform your Kube-State-Metrics deployment to use the new ServiceAccount with the following addition to your Template Spec: serviceAccountName: kube-state-metrics.

Kubernetes service account with cluster role

I have created a service account with cluster role, is it possible to deploy pods across different namespaces with this service account through APIs?
Below is the template from which the role creation and binding is done:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: api-access
rules:
-
apiGroups:
- ""
- apps
- autoscaling
- batch
- extensions
- policy
- rbac.authorization.k8s.io
resources:
- componentstatuses
- configmaps
- daemonsets
- deployments
- events
- endpoints
- horizontalpodautoscalers
- ingress
- jobs
- limitranges
- namespaces
- nodes
- pods
- persistentvolumes
- persistentvolumeclaims
- resourcequotas
- replicasets
- replicationcontrollers
- serviceaccounts
- services
verbs: ["*"]
- nonResourceURLs: ["*"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: api-access
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: api-access
subjects:
- kind: ServiceAccount
name: api-service-account
namespace: default
Kubernetes Service Accounts are not namespace objects, so answer of "can i use service account between namespaces?" is yes.
For second part: I don't know what you mean with API's but if it is kubernetes-apiserver then yes, you can use service account with kubectl make sure you are executing as service account. You can impersonate user for this and reference: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation
If you mean you built new API for deployment or using external deployer then you should deploy that with this service account as described here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
Yes, your service account will be able to create and act on resources in any namespace because a you've granted it these permissions at the cluster scope using a ClusterRoleBinding.

kubernetes RBAC role verbs to exec to pod

I my 1.9 cluster created this deployment role for the dev user. Deployment works as expected. Now I want to give exec and logs access to developer. What role I need to add for exec to the pod?
kind: Role
name: deployment-manager
rules:
- apiGroups: ["", "extensions", "apps"]
resources: ["deployments", "replicasets", "pods"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
Error message:
kubectl exec nginx -it -- sh
Error from server (Forbidden): pods "nginx" is forbidden: User "dev" cannot create pods/exec in the namespace "dev"
Thanks
SR
The RBAC docs say that
Most resources are represented by a string representation of their name, such as “pods”, just as it appears in the URL for the relevant API endpoint. However, some Kubernetes APIs involve a “subresource”, such as the logs for a pod. [...] To represent this in an RBAC role, use a slash to delimit the resource and subresource.
To allow a subject to read both pods and pod logs, and be able to exec into the pod, you would write:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: pod-and-pod-logs-reader
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
Some client libraries may do an http GET to negotiate a websocket first, which would require the "get" verb. kubectl sends an http POST instead, that's why it requires the "create" verb in that case.