I am seeing this error for user jenkins when deploying.
Error: pods is forbidden: User "system:serviceaccount:ci:jenkins" cannot list pods in the namespace "kube-system"
I have created a definition for service account
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
I have created a ClusterRoleBinding
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: jenkins
namespace: kube-system
Any advice?
Without a namespace (in the ServiceAccount creation) you will automatically create it in the default namespace. The same with a Role (which is always a one-namespace resource).
What you need to do is to create a ClusterRole with the correct permissions (basically just change your role into a ClusterRole) and then set the correct namespace either on the ServiceAccount resource or in the ClusterRole binding.
You can also skip creating the Role and RoleBinding, as the ClusterRole and ClusterRoleBinding will override it either way.
--
With that said. It's always good practice to create a specific ServiceAccount and RoleBinding per namespace when it comes to deploys, so that you don't accidently create an admin account which is used in a remote CI tool like... Jenkins ;)
Related
Im running a pipeline that creates a kubernetes namespace but when I run it I get:
Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:gitlab-runner:default" cannot create resource "namespaces" in API group "" at the cluster scope
I created a ClusterRole and a ClusterRoleBinding to allow the service user default in the gitlab-runner namespace to create namespaces with:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: modify-namespace
rules:
- apiGroups: [""]
resources:
- namespace
verbs:
- create
and:
ind: ClusterRoleBinding
metadata:
name: modify-namespace-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: modify-namespace
subjects:
- kind: ServiceAccount
name: default
namespace: gitlab-runner
But that gives me the same error.
What am I doing wrong?
[""] in clusterrole manifest it should be just "".
because [""] will be array where apiGroups expects a string.
under resources it should be namespaces not namespace because :
kubectl api-resources | grep 'namespace\|NAME'
NAME SHORTNAMES APIVERSION NAMESPACED KIND
namespaces ns v1 false Namespace
so clusterrole manifest should be as following :
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: modify-namespace
rules:
- apiGroups: ""
resources:
- namespaces
verbs:
- create
I had this issue below:
Namespaces is forbidden: User "system:serviceaccount:openshift-operators:minio-operator" cannot create resource "namespaces" in API group "" at the cluster scope
Got solved with below yaml:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-role-cesar-3
rules:
- apiGroups: [""]
resources:
- namespaces
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: role-binding-cesar-3
namespace: openshift-operators
subjects:
- kind: ServiceAccount
name: minio-operator
namespace: openshift-operators
roleRef:
kind: ClusterRole
name: cluster-role-cesar-3
apiGroup: rbac.authorization.k8s.io
in order to make a high availability test in kubernetes cluster, i use a tool such as chaoskube or kube-monkey , which kills random pods in namespaces to create a "chaos" and to see how the system and applications will react.
by default these tools need a cluster role, in order to let its service account to list/kill pods for all namespaces in cluster.
in my situation i want to install this tool and make the test just in one namespace (namespace x)
is there any way to restrict the permissions of the service account just to give it the permissions to list/kill pods from (namespace x) and the whole cluster ?
i already tried to create a role & rolebinding in (namespace x) but still have the same RBAC error, as the service account expects to have the cluster permissions :
"pods is forbidden: User \"system:serviceaccount:x:chaoskube-sa\" cannot list resource \"pods\" in API group \"\ at the cluster scope"
update: role & rolebinding
this is the default permissions for its service account:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: chaoskube-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: chaoskube-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: chaoskube-role
subjects:
- kind: ServiceAccount
name: chaoskube-sa
namespace: x
with these configration it works fine.
now with restricted permissions for a specific namespace :
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: chaoskube-role
namespace: x
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: chaoskube-rolebinding
namespace: x
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: chaoskube-role
subjects:
- kind: ServiceAccount
name: chaoskube-sa
namespace: x
it can not list the pods , and i receive the RBAC error.
I have a service account, in the default namespace, with a clusterRoleBinding to a clusterRole that can observe jobs.
I wish to use this service account in any namespace rather than have to define a new service account in each new namespace. The service account is used by an init container to check a job has completed before allowing deployment to continue.
Not sure what extra info I need to provide but will do so on request.
You can simply reference a ServiceAccount from another namespace in the RoleBinding:
For example, below is sample use to refer the service account in one namespace to another for just reading the pods.
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: pod-reader
namespace: ns2
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pod-reader-from-ns1
namespace: ns2
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pod-reader
subjects:
- kind: ServiceAccount
name: ns1-service-account
namespace: ns1
So like you said I created a another pod which is of kind:job and included the script.sh.
In the script.sh file, I run "kubectl exec" to the main pod to run few commands
The script gets executed, but I get the error "cannot create resource "pods/exec in API group"
So I created a clusterrole with resources: ["pods/exec"] and bind it to the default service account using ClusterRoleBinding
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: service-account-role-binding
namespace: default
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: ClusterRole
name: pod-reader
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: default
In the pod which is of kind:job, I include the service account like shown below
restartPolicy: Never
serviceAccountName: default
but I still get the same error. What am I doing wrong here ?
Error from server (Forbidden): pods "mongo-0" is forbidden: User "system:serviceaccount:default:default" cannot create resource "pods/exec" in API group "" in the namespace "default"
If this is something that needs to be regularly run for maintenance look into Kubernetes daemon set object.
I have couple of namespaces - assume NS1 and NS2. I have serviceaccounts created in those - sa1 in NS1 and sa2 in NS2. I have created roles and rolebindings for sa1 to do stuff within NS1 and sa2 within NS2.
What I want is give sa1 certain access within NS2 (say only Pod Reader role).
I am wondering if that's possible or not?
You can simply reference a ServiceAccount from another namespace in the RoleBinding:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: pod-reader
namespace: ns2
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pod-reader-from-ns1
namespace: ns2
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pod-reader
subjects:
- kind: ServiceAccount
name: ns1-service-account
namespace: ns1