I am using Keycloak as my identity provider for kubernetes. I am using kubelogin to get the token. The token seems to work but I am getting the below error. I think there is some issue in the ClusterRoleBinding which is not allowing it to work.
Whats the error
Error from server (Forbidden): pods is forbidden: User "test" cannot list resource "pods" in API group "" in the namespace "default"
Additional Information
Api Manifest
- --oidc-issuer-url=https://test1.example.com/auth/realms/kubernetes
- --oidc-username-claim=preferred_username
- --oidc-username-prefix=-
- --oidc-groups-claim=groups
- --oidc-client-id=kubernetes
- --oidc-ca-file=/etc/ssl/certs/ca.crt
Cluster role and cluster role binding
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cluster-admin
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: admin-rolebinding
subjects:
- kind: User
name: //test1.example.com.com/auth/realms/kubernetes#23fd6g03-e03e-450e-8b5d-07b19007c443
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
Is there anything I am missing to get this to work?
After digging a lot I could find the issue. Rather than adding the keycloak url for the user, we have to use the user name itself. Here is the example yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cluster-admin
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: admin-rolebinding
subjects:
- kind: User
name: test
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
Related
I followed all the instructions from here : https://console.cloud.google.com/kubernetes/clusters/details/us-central1-c/myapp/details?project=plenary-axon-332219&pli=1
So far I can log in successfully, but I cannot list any pods.
I tried checking different formats for the cluster role binding but still no difference
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: people-who-view-secrets
subjects:
- kind: User
name: Issuer_URI#email
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: secret-viewer
apiGroup: rbac.authorization.k8s.io
Has anyone seen this?
You need to add resources to manipulate
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: secret-viewer
rules:
- apiGroups: [""]
# The resource type for which access is granted
resources: ["secrets", "pods"] #here or namespaces , nodes
# The permissions granted by the ClusterRole
verbs: ["get", "watch", "list"]
I want to allow a ServiceAccount in namespace A to access a resource in namespace B.
To achieve this I connect the ServiceAccount to a ClusterRole via a ClusterRoleBinding.
The documentation says I can "use a ClusterRole to [1.] define permissions on namespaced resources and be granted within individual namespace(s)"
But looking through the K8s documentation I can't find a way how to create a ClusterRole with namespaced resources. How can I achieve this?
...how to create a ClusterRole with namespaced resources...
Read further down a bit:
A ClusterRole can be used to grant the same permissions as a Role.
Because ClusterRoles are cluster-scoped. You can also use them to
grant access to:
...
namespaced resources (like Pods), across all namespaces
ClusterRole won't help you to restraint access to a single namespaced object. You can however use RoleBinding to reference a ClusterRole and restraint access to the object in the namespace of the RoleBinding.
I believe you need to create clusterrole not role.
example:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: role-grantor
rules:
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["rolebindings"]
verbs: ["create"]
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["clusterroles"]
verbs: ["bind"]
# omit resourceNames to allow binding any ClusterRole
resourceNames: ["admin","edit","view"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: role-grantor-binding
namespace: user-1-namespace
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: role-grantor
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: user-1
above example is from this link.
I find both other answers a little confusing, hopefully this is clearer.
You did the right thing in creating a ClusterRole, but you want to bind it using a namespaced RoleBinding, not a ClusterRoleBinding.
Example using your examples. Notice how the RoleBinding is in the B namespace, giving A's ServiceAccount the permissions defined in the ClusterRole, but limited to the B namespace.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: what-a-is-allowed-to-do-in-b
rules:
- apiGroups: [""]
resources: ["pods", "deployments"] # etc
verbs: ["get", "list", "create"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-app
namespace: namespace-a
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: what-a-is-allowed-to-do-in-b
namespace: namespace-b
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: what-a-is-allowed-to-do-in-b
subjects:
- kind: ServiceAccount
name: my-app
namespace: namespace-a
Notes:
You have to use the ClusterRole because you can't get outside your own namespace without one. By using a RoleBinding, which is namespaced, you can then limit the access to the scope of the namespace of that RoleBinding.
I am seeing this error for user jenkins when deploying.
Error: pods is forbidden: User "system:serviceaccount:ci:jenkins" cannot list pods in the namespace "kube-system"
I have created a definition for service account
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
I have created a ClusterRoleBinding
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: jenkins
namespace: kube-system
Any advice?
Without a namespace (in the ServiceAccount creation) you will automatically create it in the default namespace. The same with a Role (which is always a one-namespace resource).
What you need to do is to create a ClusterRole with the correct permissions (basically just change your role into a ClusterRole) and then set the correct namespace either on the ServiceAccount resource or in the ClusterRole binding.
You can also skip creating the Role and RoleBinding, as the ClusterRole and ClusterRoleBinding will override it either way.
--
With that said. It's always good practice to create a specific ServiceAccount and RoleBinding per namespace when it comes to deploys, so that you don't accidently create an admin account which is used in a remote CI tool like... Jenkins ;)
So I have namespaces
ns1, ns2, ns3, and ns4.
I have a service account sa1 in ns1. I am deploying pods to ns2, ns4 that use sa1. when I look at the logs it tells me that the sa1 in ns2 can't be found.
error:
Error creating: pods "web-test-2-795f5fd489-" is forbidden: error looking up service account ns2/sa: serviceaccount "sa" not found
Is there a way to make service accounts cluster wide? Or, can I create multiple service accounts with the same secret? in different namespaces?
you can use that
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kubernetes-enforce
rules:
- apiGroups: ["apps"]
resources: ["deployments","pods","daemonsets"]
verbs: ["get", "list", "watch", "patch"]
- apiGroups: ["*"]
resources: ["namespaces"]
verbs: ["get", "list", "watch"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kubernetes-enforce
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubernetes-enforce-logging
namespace: cattle-logging
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-enforce
subjects:
- kind: ServiceAccount
name: kubernetes-enforce
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubernetes-enforce-prome
namespace: cattle-prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-enforce
subjects:
- kind: ServiceAccount
name: kubernetes-enforce
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubernetes-enforce-system
namespace: cattle-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-enforce
subjects:
- kind: ServiceAccount
name: kubernetes-enforce
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubernetes-enforce-default
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-enforce
subjects:
- kind: ServiceAccount
name: kubernetes-enforce
namespace: kube-system
No there is no way to create a cluster wide service account as service account is a namespace scoped resources. This follows the principle of least privilege.
You can create a service account with same name(for example default) into all the necessary namespaces where you are deploying pod pretty easily by applying the service account yaml targeting those namespaces.
Then you can deploy the pod using yaml. This way you don't need to change anything in the pod because the service account name is same although it will have different secret and that should not matter as long as you have defined RBAC via role and rolebinding to all the service accounts across those namespaces.
While service accounts can not be cluster scoped you can have clusterrole and clusterrolebinding which are cluster scoped.
If your namespaces for example are in values.yaml (that is they are somehow dynamic), you could do:
apiVersion: v1
kind: List
items:
{{- range $namespace := .Values.namespaces }}
- kind: ServiceAccount
apiVersion: v1
metadata:
name: <YourAccountName>
namespace: {{ $namespace }}
{{- end }}
where in values.yaml you would have:
namespaces:
- namespace-a
- namespace-b
- default
# define a clusterrole.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: supercr
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
# define a serviceaccount
apiVersion: v1
kind: ServiceAccount
metadata:
name: supersa
namespace: namespace-1
---
# bind serviceaccount to clusterrole
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: supercrb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: supercr
subjects:
- kind: ServiceAccount
name: supersa
namespace: namespace-1
Please note serviceaccount is namespaced. You can't create a cluster-wide serviceaccount. However you can bind a serviceaccount to a clusterrole with permissions to all api resources.
I have a RBAC enabled Kubernetes cluster in GCP
There are one namespace for Tiller and multiple for Services
Right now, I can assign reader role for a specific service account given it's full name
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
name: tiller-reader
namespace: tiller
rules:
- apiGroups: [""]
resources: ["pods"]
verbs:
- "get"
- "watch"
- "list"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tiller-reader-role-binding
namespace: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: tiller-reader
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: system:serviceaccount:my-namespace-1:my-namespace-1-service-account
The Service namespaces and accounts are created dynamically. How do I automatically give all services accounts access to the Tiller namespace, for example: to get pods?
to grant a role to all service accounts you must use the Group system:serviceaccounts.
you can try the following config :
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
name: tiller-reader
namespace: tiller
rules:
- apiGroups: [""]
resources: ["pods"]
verbs:
- "get"
- "watch"
- "list"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tiller-reader-role-binding
namespace: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tiller-reader
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:serviceaccounts