I'm trying to execute K8S kubectl cmds from inside the container(name: autodeploy).
I have configured ClusterRole, ServiceAccount and ClusterRoleBinding. But getting Forbidden error while performing Describe and Scale actions on K8S Deployments.
Error from server (Forbidden): deployments.apps "test-deployment" is
forbidden: User "system:node:ip-xx-xx-xx-xx.ec2.internal" cannot get
resource "deployments" in API group "apps" in the namespace "abc"
autodeploy container also in same namespace abc
ClusterRole:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: autodeploy
rules:
- apiGroups: ["*"]
resources: ["deployments", "deployments/scale", "pods"]
verbs: ["get", "list", "update"]
ServiceAccount:
apiVersion: v1
kind: ServiceAccount
metadata:
name: autodeploy
namespace: abc
ClusterRoleBinding:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: autodeploy
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: autodeploy
subjects:
- kind: ServiceAccount
name: autodeploy
namespace: abc
Related
Im running a pipeline that creates a kubernetes namespace but when I run it I get:
Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:gitlab-runner:default" cannot create resource "namespaces" in API group "" at the cluster scope
I created a ClusterRole and a ClusterRoleBinding to allow the service user default in the gitlab-runner namespace to create namespaces with:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: modify-namespace
rules:
- apiGroups: [""]
resources:
- namespace
verbs:
- create
and:
ind: ClusterRoleBinding
metadata:
name: modify-namespace-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: modify-namespace
subjects:
- kind: ServiceAccount
name: default
namespace: gitlab-runner
But that gives me the same error.
What am I doing wrong?
[""] in clusterrole manifest it should be just "".
because [""] will be array where apiGroups expects a string.
under resources it should be namespaces not namespace because :
kubectl api-resources | grep 'namespace\|NAME'
NAME SHORTNAMES APIVERSION NAMESPACED KIND
namespaces ns v1 false Namespace
so clusterrole manifest should be as following :
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: modify-namespace
rules:
- apiGroups: ""
resources:
- namespaces
verbs:
- create
I had this issue below:
Namespaces is forbidden: User "system:serviceaccount:openshift-operators:minio-operator" cannot create resource "namespaces" in API group "" at the cluster scope
Got solved with below yaml:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-role-cesar-3
rules:
- apiGroups: [""]
resources:
- namespaces
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: role-binding-cesar-3
namespace: openshift-operators
subjects:
- kind: ServiceAccount
name: minio-operator
namespace: openshift-operators
roleRef:
kind: ClusterRole
name: cluster-role-cesar-3
apiGroup: rbac.authorization.k8s.io
minikube start
--extra-config=apiserver.enable-admission-plugins=PodSecurityPolicy
--addons=pod-security-policy
we have a default namespace in which the nginx service account does not have the rights to launch the nginx container
when creating a pod, use the command
kubectl run nginx --image=nginx -n default --as system:serviceaccount:default:nginx-sa
as a result, we get an error
Error: container has runAsNonRoot and image will run as root (pod: "nginx_default(49e939b0-d238-4e04-a122-43f4cfabea22)", container: nginx)
as I understand it, it is necessary to write a psp policy that will allow the nginx-sa service account to run under, but I do not understand how to write it correctly for a specific service account
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-sa
namespace: default
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nginx-sa-role
namespace: default
rules:
- apiGroups: ["extensions", "apps",""]
resources: [ "deployments","pods" ]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nginx-sa-role-binding
namespace: default
subjects:
- kind: ServiceAccount
name: nginx-sa
namespace: default
roleRef:
kind: Role
name: nginx-sa-role
apiGroup: rbac.authorization.k8s.io
...but I do not understand how to write it correctly for a specific service account
After you get your special psp ready for your nginx, you can grant your nginx-sa to use the special psp like this:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: role-to-use-special-psp
rules:
- apiGroups:
- policy
resourceNames:
- special-psp-for-nginx
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: bind-to-role
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: role-to-use-special-psp
subjects:
- kind: ServiceAccount
name: nginx-sa
namespace: default
In my scenario user has access to four namespaces only, he will switch between namespaces using contexts below. How can I give him access to CRD's along with his exiting access to four namespaces.
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* dev-crd-ns-user dev dev-crd-ns-user dev-crd-ns
dev-mon-fe-ns-user dev dev-mon-fe-ns-user dev-mon-fe-ns
dev-strimzi-operator-ns dev dev-strimzi-operator-ns-user dev-strimzi-operator-ns
dev-titan-ns-1 dev dev-titan-ns-1-user dev-titan-ns-1
hifi#101common:/root$ kubectl get secret
NAME TYPE DATA AGE
default-token-mh7xq kubernetes.io/service-account-token 3 8d
dev-crd-ns-user-token-zd6xt kubernetes.io/service-account-token 3 8d
exfo#cmme101common:/root$ kubectl get crd
Error from server (Forbidden): customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:dev-crd-ns:dev-crd-ns-user" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
Tried below two options. Option 2 is the recommendation but didn't work with either one.
Error from server (Forbidden): customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:dev-crd-ns:dev-crd-ns-user" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the **cluster scope**
Option 1: Adding CRD to existing role
role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
annotations:
name: dev-ns-user-full-access
namespace: dev-crd-ns
rules:
- apiGroups:
- ""
- extensions
- apps
- networking.k8s.io
- apiextensions.k8s.io
resources:
- '*'
- customresourcedefinitions
verbs:
- '*'
- apiGroups:
- batch
resources:
- jobs
- cronjobs
verbs:
- '*'
role binding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
annotations:
name: dev-crd-ns-user-view
namespace: dev-crd-ns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: dev-crd-ns-user-full-access
subjects:
- kind: ServiceAccount
name: dev-crd-ns-user
namespace: dev-crd-ns
Option 2 : Adding CRD as a new role to "dev-crd-ns" namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: dev-crd-ns
name: crd-admin
rules:
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: crd-admin
namespace: dev-crd-ns
subjects:
- kind: ServiceAccount
name: dev-crd-ns-user
namespace: dev-crd-ns
roleRef:
kind: Role
name: crd-admin
apiGroup: rbac.authorization.k8s.io
You need to create Role and RoleBinding for each service account like dev-crd-ns-user.
For dev-crd-ns-user:
Update the existing Role or create a new one:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: dev-crd-ns
name: crd-admin
rules:
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
$ kubectl apply -f crd-admin-role.yaml
Update the existing RoleBinding with this new Role or create a new one:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: crd-admin
namespace: dev-crd-ns
subjects:
- kind: ServiceAccount
name: dev-crd-ns-user
namespace: dev-crd-ns
roleRef:
kind: Role
name: crd-admin
apiGroup: rbac.authorization.k8s.io
$ kubectl apply -f crd-admin-role-binding.yaml
Now, the SA dev-crd-ns-user will have all the access to customresourcedefinitions.
Follow similar steps for the rest of the service accounts.
So like you said I created a another pod which is of kind:job and included the script.sh.
In the script.sh file, I run "kubectl exec" to the main pod to run few commands
The script gets executed, but I get the error "cannot create resource "pods/exec in API group"
So I created a clusterrole with resources: ["pods/exec"] and bind it to the default service account using ClusterRoleBinding
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: service-account-role-binding
namespace: default
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: ClusterRole
name: pod-reader
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: default
In the pod which is of kind:job, I include the service account like shown below
restartPolicy: Never
serviceAccountName: default
but I still get the same error. What am I doing wrong here ?
Error from server (Forbidden): pods "mongo-0" is forbidden: User "system:serviceaccount:default:default" cannot create resource "pods/exec" in API group "" in the namespace "default"
If this is something that needs to be regularly run for maintenance look into Kubernetes daemon set object.
I've tried creating a cluster role that only has access to view pods, however, for some reason that account can still see everything; secrets, deployments, nodes etc. I also enabled skip-login, and it seems like by default anonymous users don't have any restrictions either.
Service account:
apiVersion: v1
kind: ServiceAccount
metadata:
name: sa-example
namespace: default
Cluster Role:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cr-example
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"]
Cluster Role Binding:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: crb-example
roleRef:
apiGroup: rbac.authorization.k8s.io
name: cr-example
kind: ClusterRole
subjects:
- kind: ServiceAccount
name: sa-example
namespace: default
Context:
K8s version: 1.17.3
Dashboard version: v2.0.0-rc5
Cluster type: bare metal
authorization-mode=Node,RBAC
How did You check if it works or no?
I made a reproduction of your issue with below yamls
apiVersion: v1
kind: ServiceAccount
metadata:
name: sa-example
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cr-example
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: crb-example
roleRef:
apiGroup: rbac.authorization.k8s.io
name: cr-example
kind: ClusterRole
subjects:
- kind: ServiceAccount
name: sa-example
namespace: default
And I used kubectl auth can-i to verify if it works
kubectl auth can-i get pods --as=system:serviceaccount:default:sa-example
yes
kubectl auth can-i get deployment --as=system:serviceaccount:default:sa-example
no
kubectl auth can-i get secrets --as=system:serviceaccount:default:sa-example
no
kubectl auth can-i get nodes --as=system:serviceaccount:default:sa-example
no
And it seems like everything works just fine
The only thing which if different in my yaml is
kind: ClusterRole
metadata:
name: cr-example instead of cr-<role>
So it actually match ClusterRoleBinding
I hope it help you with your issues. Let me know if you have any more questions.