RBAC role to manage single pod with dynamic name - kubernetes

I need to grant access to one deployment and all pods of this deployment using RBAC.
I've managed to configure Role and RoleBinding for the deploymet, and it's working fine:
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: <my-namespace>
name: <deployment>-manager-role
rules:
- apiGroups: ["", "extensions", "apps"]
resources: ["deployments"]
resourceNames: ["<deployment>"]
verbs: ["get", "list", "watch", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: <deployment>-manager-binding
namespace: <my-namespace>
subjects:
- kind: User
name: <username>
apiGroup: ""
roleRef:
kind: Role
name: <deployment>-manager-role
apiGroup: ""
Using this role user can access, update and patch the deployment. This deployment creates pods with dynamic names (like <deployment>-5594cbfcf4-v4xx8). I tried to allow this user to access these pods (get, list, watch, read logs, exec, delete) using deployment name and using deployment name + wildcard char *:
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: <my-namespace>
name: <deployment>-pods-manager-role
rules:
- apiGroups: ["", "extensions", "apps"]
resources: ["pods"]
resourceNames: ["<deployment>*"]
verbs: ["get", "list", "watch", "update", "patch", "exec", "delete"]
I also updated the role binding. But when I try to get the pod:
kubectl --context=<username>-ctx -n <namespace> get pods <deployment>-5594cbfcf4-v4xx8
I'm getting error:
Error from server (Forbidden): pods "<deployment>-5594cbfcf4-v4xx8" is forbidden: User "<username>" cannot get resource "pods" in API group "" in the namespace "<namespace>"
If I add <deployment>-5594cbfcf4-v4xx8 to the list of resourceNames, user can access this pod.
Is it possible to grant access to the specific pods based on deployment name?

In Kubernetes, pods are considered as an ephemeral "cattle", they come and go. You shouldn't try to manage RBAC per pod.
In your use case, there is unfortunately no way to grant a role over a set of pods matching a certain name, because the resourceNames field doesn't support patterns like prefixes/suffixes. Don't get confused: a single asterisk character ('*') has a special meaning that means "all", but it's not a pattern. So, 'my-app-* in resourceNames will not work. There were tickets opened for this feature, but it wasn't implemented:
https://github.com/kubernetes/kubernetes/issues/56582
There was also a request to be able to manage RBAC over labels, but that feature isn't implemented neither:
https://github.com/kubernetes/kubernetes/issues/44703
Therefore, you probably need to change your model to grant roles to users to manage all pods in a certain namespace. Your deployment should be the only "source of pods" in that namespace. That way, you will not need to specify any resource names.

Related

Kubernetes error: Cannot create resource "jobs" in API group "batch", I am using kubectl

I am trying to deploy a job that downloads a file to a PVC that I have successfully created. The method of deploying the job is by applying a yaml with the kubectl at command line (on windows).
kubectl apply -f file-download.yaml
I get this error:
Error from server (Forbidden): error when creating "file-download.yaml": jobs.batch is forbidden: User "token-{my user token}" cannot create resource "jobs" in API group "batch" in the namespace "{my context}"
I read that I must create a role and then bind that role to either the user or the job. So I did:
role.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: mycontext
name: example-role
rules:
- apiGroups: ["batch", "extensions"]
resources: ["jobs"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
binding.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: example-rolebinding
namespace: mycontext
subjects:
- kind: User
name: token-mytoken
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: example-role
apiGroup: rbac.authorization.k8s.io
What I am looking to find out if I have done this correctly, and also how do you 'invoke' these files such that the new rules apply to what I am trying to do? Is it sufficient to leave these two files in the working directory, of is there a kubectl command to apply them, for example (Or maybe a spot in the file-download.yaml that I have to reference these files?

Forbidden error to connect to kubernetes cluster in gitlab CI

I'm trying to get access to my kubernetes cluster in my self hosted gitlab instance as it is described in the docs.
deploy:
stage: deployment
script:
- kubectl create secret docker-registry gitlab-registry --docker-server="$CI_REGISTRY" --docker-username="$CI_DEPLOY_USER" --docker-password="$CI_DEPLOY_PASSWORD" --docker-email="$GITLAB_USER_EMAIL" -o yaml --dry-run=client | kubectl apply -f -
But I do get the error
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=secrets", GroupVersionKind: "/v1, Kind=Secret"
Name: "gitlab-registry", Namespace: "gitlab"
from server for: "STDIN": secrets "gitlab-registry" is forbidden: User "system:serviceaccount:gitlab:default" cannot get resource "secrets" in API group "" in the namespace "gitlab"
I do not understand the error. Why do I get a forbidden error?
Update
The kubernetes cluster is integrated in gitlab at instance level.
But running kubectl config view in the CI pipeline gives me
apiVersion: v1
clusters: null
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null
Update2
Thanks to AndD, the secret can be created with this role / service account:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
namespace: gitlab
name: gitlab-deploy
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: use-secrets
namespace: gitlab
subjects:
- kind: ServiceAccount
name: default
namespace: gitlab
roleRef:
kind: ClusterRole
name: gitlab-deploy
apiGroup: rbac.authorization.k8s.io
But running a simple apply for this namespace.yaml file
apiVersion: v1
kind: Namespace
metadata:
name: myns
gives me a similar error:
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=namespaces", GroupVersionKind: "/v1, Kind=Namespace"
Name: "myns", Namespace: ""
from server for: "namespace.yaml": namespaces "myns" is forbidden: User "system:serviceaccount:gitlab:default" cannot get resource "namespaces" in API group "" in the namespace "myns"
I used ClusterBinding to get this working even for a different namespace. What am I doing wrong?
Kubernetes makes use of a Role-based access control (RBAC) to prevent Pods and Users from being able to interact with resources in the cluster, unless they are not authorized.
From the error, you can see that Gitlab is trying to use the secrets resource and also that it is using as ServiceAccount the default service account in its namespace.
This means that Gitlab is not configured to use a particular ServiceAccount, which means it makes use of the default one (there's a default service account in each namespace of the cluster)
You can attach role auth and permissions to service accounts by using Role / ClusterRole and RoleBinding / ClusterRoleBinding.
Roles or ClusterRoles describe permissions. For example, a Role could be:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: gitlab
name: secret-user
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
and this states that "whoever has this role, can do whatever (all the verbs) with secrets but only in the namespace gitlab"
If you want to give generic permissions in all namespaces, you can use a ClusterRole instead, which is very similar.
Once the Role is created, you then can attach it to a User, a Group or a ServiceAccount, for example:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: use-secrets
namespace: gitlab
subjects:
subjects:
- kind: ServiceAccount
name: default
namespace: gitlab
roleRef:
# "roleRef" specifies the binding to a Role / ClusterRole
kind: Role # this must be Role or ClusterRole
name: secret-user # this must match the name of the Role or ClusterRole you wish to bind to
apiGroup: rbac.authorization.k8s.io
and this bind the role previously created to the ServiceAccount called default in the namespace gitlab.
Then, all Pods running in the namespace gitlab and using the default service account, will be able to use secrets (use the verbs listed in the Role) but only in the namespace specified by the Role.
As you can see, this aspect of Kubernetes is pretty complex and powerful, so have a look at the docs because they explain things really well and are also full of examples:
Service Accounts - https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
RBAC - https://kubernetes.io/docs/reference/access-authn-authz/rbac/
A list of RBAC resources - How to refer to all subresources in a Role definition?
UPDATE
You are doing nothing wrong. It's just that you are trying to use the resource namespace but Gitlab has no Bind that gives access to that type of resource. With your ClusterRole you just gave it access to secrets, but nothing more.
Consider giving the ClusterRole more permissions, changing it to list all resources that you need to access:
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["secrets", "namespaces", "pods"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
For example this will give access to secrets, namespaces and Pods.
As an alternative, you can bind Gitlab's service account to cluster-admin to directly give it access to everything.
kubectl create clusterrolebinding gitlab-is-now-cluster-admin \
--clusterrole=cluster-admin \
--serviceaccount=gitlab:default
Before doing so tho, consider the following:
Fine-grained role bindings provide greater security, but require more
effort to administrate. Broader grants can give unnecessary (and
potentially escalating) API access to ServiceAccounts, but are easier
to administrate.
So, it is way more secure to first decide which resources can be used by Gitlab and then create a Role / ClusterRole giving access to only those resources (and with the verbs that you need)

How to set up permissions for Kubernetes resources?

First off, I'm aware of the Kubernetes RBAC method. My question is: is there a way to create Kubernetes resources that can only be read and/or written by a specific Role (or a ClusterRole)?
For example, let's say I have a Kubernetes Secret. I want this Secret to be bound to a specific ClusterRole, then only a ServiceAccount bound to this specific ClusterRole could read it. Is there a way to set up something like that?
Edit: it looks like what I want here is not possible. Kubernetes RBAC was designed to GRANT access to certain resources. I wanted to DENY access based on a specific group (or set of rules).
You can use the RBAC for managing the Role-based access in K8s
For example, let's say I have a Kubernetes Secret. I want this Secret
to be bound to a specific ClusterRole, so only a ServiceAccount bound
to this specific ClusterRole could read it. Is there a way to set up
something like that?
No, you can not use the ClusterRole for granular level access, however, you can create some Role to restrict secret.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: secret-read-role
rules:
- apiGroups: ["*"]
resources: ["secret"]
verbs: ["get", "watch", "list"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: secret-read-sa
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: secret-read-rolebinding
namespace: default
subjects:
- kind: ServiceAccount
name: secret-read-sa
apiGroup: ""
roleRef:
kind: Role
name: secret-read-role
apiGroup: ""
Checkout about the resourceNames you can also give a name or pattern in name so this way it might be helpful to attach a specific secret to Role.
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
resourceNames: ["userA-*"]
If you planning to Go with RBAC you can use the RBAC manager for better management : https://github.com/FairwindsOps/rbac-manager
Extra :
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: data-engineering
name: umbrella:data-engineering-app
rules:
– apiGroups: [“”]
resources: [“configmaps”]
resourceNames: [“data-engineering-app-configmap”] <<<<<<<<<
verbs: [“get”]
—
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: umbrella:data-engineering-app
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: umbrella:data-engineering-app
subjects:
– kind: ServiceAccount
name: data-engineering-app
namespace: data-engineering
You can also refer to resources by name for certain requests through the resourceNames list. When specified, requests can be restricted to individual instances of a resource. Here is an example that restricts its subject to only get or update a ConfigMap named my-configmap
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: configmap-updater
rules:
- apiGroups: [""]
#
# at the HTTP level, the name of the resource for accessing ConfigMap
# objects is "configmaps"
resources: ["configmaps"]
resourceNames: ["my-configmap"]
verbs: ["update", "get"]
https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-resources
Good Example : https://thenewstack.io/three-realistic-approaches-to-kubernetes-rbac/
It is not possible to restrict access on a resource per resource basis.
The RBAC framework works by allowing specified Roles to perform certain actions (get, update, delete etc.) over certain resources (pods, secrets etc) in a certain namespace.
Clusterroles are used to grant access across all namespaces or to non namespaced resources like nodes.
To achieve what you are looking for you need to isolate your Kubernetes secret in a namespace where you only allow your specific role to read secrets.

kubernetes resource level RBAC with object filtering

i'm doing some resource level RBAC for k8s custom objects and finding it difficult to get filter resources using native k8s calls
cluster is my custom CRD and user john has access to only one crd instance not all instances of CRD using k8s native RBAC
➜ k get clusters
NAME AGE
aws-gluohfhcwo 3d2h
azure-cikivygyxd 3d1h
➜ k get clusters --as=john
Error from server (Forbidden): clusters.operator.biqmind.com is forbidden: User "ranbir" cannot list resource "clusters" in API group "operator.biqmind.com" in the namespace "biqmind"
➜ k get clusters --as=john aws-gluohfhcwo
NAME AGE
aws-gluohfhcwo 3d2h
i have explicitly specify object name to get the list of objects to which user is authenticated. any suggestions on how this can be solved?
full rbac is posted here
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: biqmind
name: cluster-admin-aws-gluohfhcwo
rules:
- apiGroups: ["operator.biqmind.com"]
resources: ["clusters"]
resourceNames: ["aws-gluohfhcwo"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cluster-admin-aws-gluohfhcwo-binding
namespace: biqmind
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cluster-admin-aws-gluohfhcwo
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: ranbir
User "ranbir" cannot list resource "clusters" in API group "operator.biqmind.com" in the namespace "biqmind"
You must add RBAC permissions with the verb list for the specified user in the specified namespace, to let that user list "clusters".
When doing
kubectl get clusters --as=john aws-gluohfhcwo
you use the RBAC verb get, but to list without specifying a specific name, the user also need permission to list.
Example of giving list permission, without resourceName::
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: biqmind
name: cluster-admin-aws-gluohfhcwo
rules:
- apiGroups: ["operator.biqmind.com"]
resources: ["clusters"]
resourceNames: ["aws-gluohfhcwo"]
verbs: ["get", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["operator.biqmind.com"]
resources: ["clusters"]
verbs: ["get", "list"]
Enforcing RBAC on the user side is easy in concept. You can create RoleBindings for individual users, but this is not the recommended path as there’s a high risk of operator insanity.
The better approach for sane RBAC is to create that your users map to; how this mapping is done is dependent on your cluster’s authenticator (e.g. the aws-iam-authenticator for EKS uses mapRoles to map a role ARN to a set of groups).
Groups and the APIs they have access to are ultimately determined based on an organization’s needs, but a generic reader (for new engineers just getting the hang of things), writer (for your engineers), and admin (for you) role is a good start. (Hey, it’s better than admin for everyone.)
Here is example of configuration file:
# An example reader ClusterRole – ClusterRole so you’re not worried about namespaces at this time. Remember, we’re talking generic reader/writer/admin roles.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: reader
rules:
– apiGroups: [“*”]
resources:
– deployments
– configmaps
– pods
– secrets
– services
verbs:
– get
– list
– watch
---
# An example reader ClusterRoleBinding that gives read permissions to
# the engineering and operations groups
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: reader-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: reader
subjects:
- kind: Group
name: umbrella:engineering
- kind: Group
name: umbrella:operations
---
# An example writer ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: writer
rules:
- apiGroups: [“*”]
resources:
- deployments
- configmaps
- pods
- secrets
- services
verbs:
- create
- delete
- patch
- update
---
# An example writer ClusterRoleBinding that gives write permissions to
# the operations group
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: reader-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: reader
subjects:
- kind: Group
name: umbrella:operations
Here is exact explanation: rbac-article.
Take notice that default Roles and Role Bindings
API servers create a set of default ClusterRole and ClusterRoleBinding objects. Many of these are system: prefixed, which indicates that the resource is “owned” by the infrastructure. Modifications to these resources can result in non-functional clusters. One example is the system:node ClusterRole. This role defines permissions for kubelets. If the role is modified, it can prevent kubelets from working.
All of the default cluster roles and rolebindings are labeled with kubernetes.io/bootstrapping=rbac-defaults.
Remember about auto-reconciliation
At each start-up, the API server updates default cluster roles with any missing permissions, and updates default cluster role bindings with any missing subjects. This allows the cluster to repair accidental modifications, and to keep roles and rolebindings up-to-date as permissions and subjects change in new releases.
To opt out of this reconciliation, set the rbac.authorization.kubernetes.io/autoupdate annotation on a default cluster role or rolebinding to false. Be aware that missing default permissions and subjects can result in non-functional clusters.
Auto-reconciliation is enabled in Kubernetes version 1.6+ when the RBAC authorizer is active.
Useful article: understanding-rbac.

How can I allow port-forwarding for a specific deployment in Kubernetes?

I am trying to allow some users in my org to forward ports to our production namespace in Kubernetes. However, I don't want them to be able to forward ports to all services. I want to restrict access to only certain services. Is this possible?
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: allow-port-forward-for-deployment-a
rules:
- apiGroups: [""]
resources: ["pods/portforward"]
verbs: ["get", "list", "create"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: allow-port-forward-for-deployment-a
namespace: production
subjects:
- kind: User
name: "xyz#org.com"
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: allow-port-forward-for-deployment-a
apiGroup: rbac.authorization.k8s.io
The above set up allows all services, but I don't want that.
I believe you can't. According to the docs
Resources can also be referred to by name for certain requests through
the resourceNames list. When specified, requests can be restricted to
individual instances of a resource. To restrict a subject to only
“get” and “update” a single configmap, you would write:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: configmap-updater
rules:
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["my-configmap"]
verbs: ["update", "get"]
Note that create requests
cannot be restricted by resourceName, as the object name is not known
at authorization time. The other exception is deletecollection.
Since you want to give the user permissions to create the forward ports, I don't think you can.
These rules worked for me
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: port-forward
rules:
- apiGroups: [""]
resources: ["pods/portforward"]
verbs: ["get", "create"]
- apiGroups: [""]
resources: ["pods", "services"]
verbs: ["get", "list"]
Assuming users already have access to your kubernetes cluster and relevant namespace. They can simply port-forward local port to a pod (resource) port.
How can you do this?
kubectl port-forward <POD_NAME> <LOCAL_PORT>:<POD_PORT>
See Documentation
Quoting from the document - kubectl port-forward allows using resource name, such as a pod name, to select a matching pod to port forward to since Kubernetes v1.10.
Refer this article if you wish, this nicely explains when you would need RBAC vs kubectl port-forward
RBAC could have been useful only when, you wanted person or a group of people only to port-forward for any services in a relevant namespace in your kubernetes cluster.
Workaround A: StatefulSets and resourceNames
It is possible to restrict port forwarding to a pod with a specific name. resourceNames refer to resources, not subresources:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: allow-port-forward-for-deployment-a
rules:
- apiGroups: [""]
resources: ["pods/portforward"]
resourceNames: ["my-app"]
verbs: ["create"]
A StatefulSet generates predictable pod names, but is different from a ReplicaSet and might not fit your use case.
Workaround B: Jump pod and NetworkPolicy
Sketch:
A StatefulSet that runs kubectl port-forward services/my-service inside the cluster (JUMP).
A NetworkPolicy which restricts traffic from pods belonging to JUMP to the target service
RBAC which restricts creation of subresource portforward to the pods of JUMP up to a predefined maximum number of replicas resourceNames: ["jump-0", "jump-1", ..., "jump-N"].