I want to make a very simple example to learn how to use RBAC authorization in kubernetes. Therefore I use the example from the docs:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: dev
name: dev-readpods-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: dev-tester-rolebinding
namespace: dev
subjects:
- kind: User
name: Tester
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: dev-readpods-role
apiGroup: rbac.authorization.k8s.io
The role and the rolebinding are created.
When I log in with Tester and try
kubectl get pods -n dev
I get
Error from server (Forbidden): pods is forbidden: User "<url>:<port>/oidc/endpoint/OP#Tester" cannot list pods in the namespace "dev"
I read here (RBAC Error in Kubernetes) that the api-server have to be started with --authorization-mode=…,RBAC. How can I check this? I read somewhere else that if I run
kubectl api-versions | findstr rbac
and find entries RBAC should be activated. Is that true?
What am I doing wrong? Is there a good way to troubleshoot?
Thanks!
P.S. I'm running kubernetes inside IBM Cloud Private.
In ICP, it looks encouraging to use Teams (ICP's own term, I think). Try starting with it. But you need an LDAP server outside of ICP.
https://www.ibm.com/support/knowledgecenter/en/SSBS6K_2.1.0.3/user_management/admin.html
You would need to determine the invocation of the apiserver to see what --authorization-mode flag was passed to it. Normally this is contained in a systemd unit file or pod manifest. I'm not sure how IBM Cloud launches the apiserver
Related
I'm trying to get access to my kubernetes cluster in my self hosted gitlab instance as it is described in the docs.
deploy:
stage: deployment
script:
- kubectl create secret docker-registry gitlab-registry --docker-server="$CI_REGISTRY" --docker-username="$CI_DEPLOY_USER" --docker-password="$CI_DEPLOY_PASSWORD" --docker-email="$GITLAB_USER_EMAIL" -o yaml --dry-run=client | kubectl apply -f -
But I do get the error
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=secrets", GroupVersionKind: "/v1, Kind=Secret"
Name: "gitlab-registry", Namespace: "gitlab"
from server for: "STDIN": secrets "gitlab-registry" is forbidden: User "system:serviceaccount:gitlab:default" cannot get resource "secrets" in API group "" in the namespace "gitlab"
I do not understand the error. Why do I get a forbidden error?
Update
The kubernetes cluster is integrated in gitlab at instance level.
But running kubectl config view in the CI pipeline gives me
apiVersion: v1
clusters: null
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null
Update2
Thanks to AndD, the secret can be created with this role / service account:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
namespace: gitlab
name: gitlab-deploy
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: use-secrets
namespace: gitlab
subjects:
- kind: ServiceAccount
name: default
namespace: gitlab
roleRef:
kind: ClusterRole
name: gitlab-deploy
apiGroup: rbac.authorization.k8s.io
But running a simple apply for this namespace.yaml file
apiVersion: v1
kind: Namespace
metadata:
name: myns
gives me a similar error:
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=namespaces", GroupVersionKind: "/v1, Kind=Namespace"
Name: "myns", Namespace: ""
from server for: "namespace.yaml": namespaces "myns" is forbidden: User "system:serviceaccount:gitlab:default" cannot get resource "namespaces" in API group "" in the namespace "myns"
I used ClusterBinding to get this working even for a different namespace. What am I doing wrong?
Kubernetes makes use of a Role-based access control (RBAC) to prevent Pods and Users from being able to interact with resources in the cluster, unless they are not authorized.
From the error, you can see that Gitlab is trying to use the secrets resource and also that it is using as ServiceAccount the default service account in its namespace.
This means that Gitlab is not configured to use a particular ServiceAccount, which means it makes use of the default one (there's a default service account in each namespace of the cluster)
You can attach role auth and permissions to service accounts by using Role / ClusterRole and RoleBinding / ClusterRoleBinding.
Roles or ClusterRoles describe permissions. For example, a Role could be:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: gitlab
name: secret-user
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
and this states that "whoever has this role, can do whatever (all the verbs) with secrets but only in the namespace gitlab"
If you want to give generic permissions in all namespaces, you can use a ClusterRole instead, which is very similar.
Once the Role is created, you then can attach it to a User, a Group or a ServiceAccount, for example:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: use-secrets
namespace: gitlab
subjects:
subjects:
- kind: ServiceAccount
name: default
namespace: gitlab
roleRef:
# "roleRef" specifies the binding to a Role / ClusterRole
kind: Role # this must be Role or ClusterRole
name: secret-user # this must match the name of the Role or ClusterRole you wish to bind to
apiGroup: rbac.authorization.k8s.io
and this bind the role previously created to the ServiceAccount called default in the namespace gitlab.
Then, all Pods running in the namespace gitlab and using the default service account, will be able to use secrets (use the verbs listed in the Role) but only in the namespace specified by the Role.
As you can see, this aspect of Kubernetes is pretty complex and powerful, so have a look at the docs because they explain things really well and are also full of examples:
Service Accounts - https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
RBAC - https://kubernetes.io/docs/reference/access-authn-authz/rbac/
A list of RBAC resources - How to refer to all subresources in a Role definition?
UPDATE
You are doing nothing wrong. It's just that you are trying to use the resource namespace but Gitlab has no Bind that gives access to that type of resource. With your ClusterRole you just gave it access to secrets, but nothing more.
Consider giving the ClusterRole more permissions, changing it to list all resources that you need to access:
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["secrets", "namespaces", "pods"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
For example this will give access to secrets, namespaces and Pods.
As an alternative, you can bind Gitlab's service account to cluster-admin to directly give it access to everything.
kubectl create clusterrolebinding gitlab-is-now-cluster-admin \
--clusterrole=cluster-admin \
--serviceaccount=gitlab:default
Before doing so tho, consider the following:
Fine-grained role bindings provide greater security, but require more
effort to administrate. Broader grants can give unnecessary (and
potentially escalating) API access to ServiceAccounts, but are easier
to administrate.
So, it is way more secure to first decide which resources can be used by Gitlab and then create a Role / ClusterRole giving access to only those resources (and with the verbs that you need)
I have a Kubernetes cluster where my application is deployed.
there are some other users they should only be able to copy files into and from a pod. Using kubectl cp command.
This user context should not allow the user to do any other operations on the cluster other than kubectl cp.
kubectl cp internally uses exec. There is no way to provide permission to only copy but you can provide only exec permission.
Create a role with permission to pods/exec
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: pod-exec
rules:
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
Create a Rolebinding to assign the above role to a user.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pod-exec-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pod-exec
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: user
Rather than use kubectl cp, instead run a sidecar container with an sftp or rsync server. That will give you better control at all levels.
You can use opa and admission controller which only permit to run api manifest has a specific label like "cp" or "username" etc. and also benefits from gatekeeper
https://www.youtube.com/watch?v=ZJgaGJm9NJE&t=3040s
I have a problem in my Kubernetes cluster, that suddendly appeared two weeks ago. The ClusterRoles I create are not visible when RBAC for a given ServiceAccount are resolved. Here is a minimal set to reproduce the problem.
Create relevant ClusterRole, ClusterRoleBinding and a ServiceAccount in the default namespace to have the rights to see Endpoints with this SA.
# test.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: test-sa
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: test-cr
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: test-crb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: test-cr
subjects:
- kind: ServiceAccount
name: test-sa
namespace: default
$ kubectl apply -f test.yaml
serviceaccount/test-sa created
clusterrole.rbac.authorization.k8s.io/test-cr created
clusterrolebinding.rbac.authorization.k8s.io/test-crb created
All objects, in particular the ClusterRole, are visible if requested directly.
$ kubectl get serviceaccount test-sa
NAME SECRETS AGE
test-sa 1 57s
$ kubectl get clusterrolebinding test-crb
NAME AGE
test-crb 115s
$ kubectl get clusterrole test-cr
NAME AGE
test-cr 2m19s
However, when I try to resolve the effective rights for this ServiceAccount, here the error I get back:
$ kubectl auth can-i get endpoints --as=system:serviceaccount:default:test-sa
no - RBAC: clusterrole.rbac.authorization.k8s.io "test-cr" not found
The RBAC rules created before the breakage are working properly. For instance, here for the ServiceAccount of my etcd-operator that I deployed with Helm several months ago:
$ kubectl auth can-i get endpoints --as=system:serviceaccount:etcd:etcd-etcd-operator-etcd-operator
yes
The version of Kubernetes in this cluster is the 1.17.0-0.
I am also seeing very slow deployements lately of new Pods, that can take up to 5 mins to start to be deployed after they have been created by a StatefulSet or a Deployment, if this can help.
Do you have any insight of what is going on, or even what I could do about it? Please note that my Kubernetes cluster is managed, so I do not have any control on the underlying system, I just have the cluster-admin privileges as a customer. But it would greatly help anyway if I could give any direction to the administrators.
Thanks in advance!
Thanks a lot for your answers!
It turned out that we will certainly never have the final world about what happen. The cluster provider just restarted the kube-apiserver, and this fixed the issue.
I suppose that something went wrong like caching or other transient failures, that can not be defined as a reproductible error.
To give a little more data for a future reader, the error occured on a Kubernetes cluster managed by OVH, and their specificity is to run the control plane itself as pods deployed in a master Kubernetes cluster on their side.
I'm trying to lock down a namespace in kubernetes using RBAC so I followed this tutorial.
I'm working on a baremetal cluster (no minikube, no cloud provider) and installed kubernetes using Ansible.
I created the folowing namespace :
apiVersion: v1
kind: Namespace
metadata:
name: lockdown
Service account :
apiVersion: v1
kind: ServiceAccount
metadata:
name: sa-lockdown
namespace: lockdown
Role :
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: lockdown
rules:
- apiGroups: [""] # "" indicates the core API group
resources: [""]
verbs: [""]
RoleBinding :
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rb-lockdown
subjects:
- kind: ServiceAccount
name: sa-lockdown
roleRef:
kind: Role
name: lockdown
apiGroup: rbac.authorization.k8s.io
And finally I tested the authorization using the next command
kubectl auth can-i get pods --namespace lockdown --as system:serviceaccount:lockdown:sa-lockdown
This SHOULD be returning "No" but I got "Yes" :-(
What am I doing wrong ?
Thx
A couple possibilities:
are you running the "can-i" check against the secured port or unsecured port (add --v=6 to see). Requests made against the unsecured (non-https) port are always authorized.
RBAC is additive, so if there is an existing clusterrolebinding or rolebinding granting "get pods" permissions to that service account (or one of the groups system:serviceaccounts:lockdown, system:serviceaccounts, or system:authenticated), then that service account will have that permission. You cannot "ungrant" permissions by binding more restrictive roles
I finally found what was the problem.
The role and rolebinding must be created inside the targeted namespace.
I changed the following role and rolebinding types by specifying the namespace inside the yaml directly.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: lockdown
namespace: lockdown
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- watch
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rb-lockdown
namespace: lockdown
subjects:
- kind: ServiceAccount
name: sa-lockdown
roleRef:
kind: Role
name: lockdown
apiGroup: rbac.authorization.k8s.io
In this example I gave permission to the user sa-lockdown to get, watch and list the pods in the namespace lockdown.
Now if I ask to get the pods : kubectl auth can-i get pods --namespace lockdown --as system:serviceaccount:lockdown:sa-lockdown it will return yes.
On the contrary if ask to get the deployments : kubectl auth can-i get deployments --namespace lockdown --as system:serviceaccount:lockdown:sa-lockdown it will return no.
You can also leave the files like they were in the question and simply create them using kubectl create -f <file> -n lockdown.
According to https://docs.gitlab.com/runner/executors/kubernetes.html#connecting-to-the-kubernetes-api, I can connect to another K8s cluster from my current Gitlab runner, but the thing is that they do not provide any information of what do you do or where do you need to specify these options.
EDIT: Finally managed to specify these options in the config.toml, but now any time my Gitlab-runner gets a job it fails as it follows:
ERROR: Job failed (system failure): User "system:serviceaccount:test-djg:default" cannot create secrets in project "gitlab" job=17304 project=3128 runner=c36ccf98
Any idea?
You get this error because executor pod doesn't have necessary permissions to run when RBAC is enabled in cluster. You should create a Role and RoleBinding and maybe a ServiceAccount based on your setup.
There is a pending merge request in gitlab-runner project that documents necessary permission and some other info about RBAC.
But for now if you want a quick setup it's possible to allow full access to namespace with following snippet. Note that you're gonna need to modifications to match you setup.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: gitlab
name: gitlab-admin
rules:
- apiGroups: [""] # The API group "" indicates the core API Group.
resources: ["*"]
verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: gitlab-admin
namespace: gitlab
subjects:
- kind: ServiceAccount # May be "User", "Group" or "ServiceAccount"
name: default
namespace: gitlab
roleRef:
kind: Role
name: gitlab-admin
apiGroup: rbac.authorization.k8s.io