After creating a new GKE cluster, creating a cluster role failed with the following error:
Error from server (Forbidden): error when creating "./role.yaml":
clusterroles.rbac.authorization.k8s.io "secret-reader" is forbidden:
attempt to grant extra privileges: [PolicyRule{Resources:["secrets"],
APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["secrets"],
APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["secrets"],
APIGroups:[""], Verbs:["list"]}] user=&{XXX#gmail.com
[system:authenticated] map[authenticator:[GKE]]} ownerrules= .
[PolicyRule{Resources:["selfsubjectaccessreviews"
"selfsubjectrulesreviews"], APIGroups:["authorization.k8s.io"], Verbs:
["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis"
"/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json"
"/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}]
ruleResolutionErrors=[]
My account has the following permissions in IAM:
Kubernetes Engine Admin
Kubernetes Engine Cluster Admin
Owner
This is my role.yaml (from the Kubernetes docs):
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: secret-reader
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "watch", "list"]
According to the RBAC docs of GCloud, I need to
create a RoleBinding that gives your Google identity a cluster-admin role before attempting to create additional Role or ClusterRole permissions.
So I tried this:
export GCP_USER=$(gcloud config get-value account | head -n 1)
kubectl create clusterrolebinding cluster-admin-binding
--clusterrole=cluster-admin --user=$GCP_USER
which succeeded, but I still get the same error when creating the cluster role.
Any ideas what I might be doing wrong?
According to Google Container Engine docs you must first create a RoleBinding that grants you all of the permissions included in the role you want to create.
Get current google identity
$ gcloud info | grep Account
Account: [myname#example.org]
Grant cluster-admin to your current identity
$ kubectl create clusterrolebinding myname-cluster-admin-binding --clusterrole=cluster-admin --user=myname#example.org
Clusterrolebinding "myname-cluster-admin-binding" created
Now you can create your ClusterRole without any problem.
I found the answer in CoreOS FAQ / Troubleshooting check it out for more information.
#S.Heutmaker`s comment led me to the solution.
For me, the solution was to create the cluster-admin-binding with the correct casing on the email address. Check the casing in error message or google cloud console IAM
$ kubectl create clusterrolebinding myname-cluster-admin-binding --clusterrole=cluster-admin --user=MyName#example.org
That's the correct solution. Is the GCP_USER obtained the same as the XXX#gmail.com username in the role creation error message?
If you got the casing right, try to add both googlemail domain variants (i.e. #gmail.com and #googlemail.com). For me gcloud info | grep Account returned <name>#googlemail.com but I had to create a clusterrole binding with <name>#gmail.com for the command to work.
Related
I deleted my cluster-admin role via kubectl using:
kubectl delete clusterrole cluster-admin
Not sure what I expected, but now I don't have access to the cluster from my account. Any attempt to get or change resources using kubectl returns a 403, Forbidden.
Is there anything I can do to revert this change without blowing away the cluster and creating a new one? I have a managed cluster on Digital Ocean.
Not sure what I expected, but now I don't have access to the cluster from my account.
If none of the kubectl commands actually work, unfortunately you will not be able to create a new cluster role. The problem is that you won't be able to do anything without an admin role. You can try creating the cluster-admin role directly through the API (not using kubectl), but if that doesn't help you have to recreate the cluster.
Try applying this YAML to creaste the new Cluster role
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
apply the YAML file changes
kubectl apply -f <filename>.yaml
I would like to read state of K8s using µK8s, but I don't want to have rights to modify anything. How to achieve this?
The following will give me full access:
microk8s.kubectl Insufficient permissions to access MicroK8s. You can either try again with sudo or add the user digital to the 'microk8s' group:
sudo usermod -a -G microk8s digital sudo chown -f -R digital ~/.kube
The new group will be available on the user's next login.
on Unix/Linux we can just set appropriate file/directory access
permission - just rx, decrease shell limits (like max memory/open
file descriptors), decrease process priority (nice -19). We are
looking for similar solution for K8S
This kind of solutions in Kubernetes are handled via RBAC (Role-based access control). RBAC prevents unauthorized users from viewing or modifying the cluster state. Because the API server exposes a REST interface, users perform actions by sending HTTP requests to the server. Users authenticate themselves by including credentials in the request (an authentication token, username and password, or a client certificate).
As for REST clients you get GET, POST, PUT,DELETE etc. These are send to specific URL paths that represents specific REST API resources (Pods, Services, Deployments and so).
RBAC auth is configured with two groups:
Roles and ClusterRoles - this specify which actions/verbs can be performed
RoleBinding and ClusterRoleBindings - this bind the above roles to a user, group or service account.
As you might already find out the ClusterRole is the one your might be looking for. This will allow to restrict specific user or group against the cluster.
In the example below we are creating ClusterRole that can only list pods. The namespace is omitted since ClusterRoles are not namepsaced.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: pod-viewer
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list"]
This permission has to be bound then via ClusterRoleBinding :
apiVersion: rbac.authorization.k8s.io/v1
# This cluster role binding allows anyone in the "manager" group to list pods in any namespace.
kind: ClusterRoleBinding
metadata:
name: list-pods-global
subjects:
- kind: Group
name: manager # Name is case sensitive
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: pod-viewer
apiGroup: rbac.authorization.k8s.io
Because you don't have the enough permissions on your own you have to reach out to appropriate person who manage those to create user for you that has the ClusterRole: View. View role should be predefined already in cluster ( kubectl get clusterrole view)
If you wish to read more Kubernetes docs explains well its whole concept of authorization.
Is possible to gain k8s cluster access with serviceaccount token?
My script does not have access to a kubeconfig file, however, it does have access to the service account token at /var/run/secrets/kubernetes.io/serviceaccount/token.
Here are the steps I tried but it is not working.
kubectl config set-credentials sa-user --token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
kubectl config set-context sa-context --user=sa-user
but when the script ran "kubectl get rolebindings" I get the following error:
Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:test:default" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "test"
Is possible to gain k8s cluster access with serviceaccount token?
Certainly, that's the point of a ServiceAccount token. The question you appear to be asking is "why does my default ServiceAccount not have all the privileges I want", which is a different problem. One will benefit from reading the fine manual on the topic
If you want the default SA in the test NS to have privileges to read things in its NS, you must create a Role scoped to that NS and then declare the relationship explicitly. SAs do not automatically have those privileges
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: test
name: test-default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: whatever-role-you-want
subjects:
- kind: ServiceAccount
name: default
namespace: test
but when the script ran "kubectl get pods" I get the following error: Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:test:default" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "test"
Presumably you mean you can kubectl get rolebindings, because I would not expect running kubectl get pods to emit that error
Yes, it is possible. For instance, if you login K8S dashboard via token it does use the same way.
Follow these steps;
Create a service account
$ kubectl -n <your-namespace-optional> create serviceaccount <service-account-name>
A role binding grants the permissions defined in a role to a user or set of users. You can use a predefined role or you can create your own. Check this link for more info. https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-example
$ kubectl create clusterrolebinding <binding-name> --clusterrole=cluster-admin --serviceaccount=<namespace>:<service-account-name>
Get the token name
$ TOKENNAME=`kubectl -n <namespace> get serviceaccount/<service-account-name> -o jsonpath='{.secrets[0].name}'`
Finally, get the token and set the credentials
$ kubectl -n <namespace> get secret $TOKENNAME -o jsonpath='{.data.token}'| base64 --decode
$ kubectl config set-credentials <service-account-name> --token=<output from previous command>
$ kubectl config set-context --current --user=<service-account-name>
If you follow these steps carefully your problem will be solved.
I am getting this error while creating custom role for kubernetes namespace.
RBAC Error : Error from server (Forbidden): error when creating
"role-qa.yaml": roles.rbac.authorization.k8s.io "qa-role" is forbidden
I have tried this on many version of kubernetes.
I have checked many solutions provided by various users about assigning "cluster-admin" role to user and did the same but still it didn't resolve this issue.
Below are the details required.
namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: qa
---
apiVersion: v1
kind: Namespace
metadata:
name: prod
I have double check the user used for kubernetes access as below.
gcloud config list
output :
[core]
account = xyz#gmail.com
disable_usage_reporting = True
project = sonar-198615
gcloud auth list
output:
Credentialed Accounts
ACTIVE ACCOUNT
* xyz#gmail.com
To set the active account, run:
$ gcloud config set account `ACCOUNT`
I have also checked if the cluster role is assigned to the user or not and user is having cluster admin role.
Below is the full error which i am getting.
Error from server (Forbidden): error when creating "role-qa.yaml":
roles.rbac.authorization.k8s.io "qa-role" is forbidden: attempt to
grant extra privileges: [PolicyRule{APIGroups:[""],
Resources:["pods"], Verbs:["get"]} PolicyRule{APIGroups:[""],
Resources:["pods"], Verbs:["watch"]} PolicyRule{APIGroups:[""],
Resources:["pods"], Verbs:["list"]}] user=&{xyz#gmail.com
[system:authenticated]
map[user-assertion.cloud.google.com:[AF1jyJCtSZd2sdmeNfdVbJyylD/nTw8h9aQznfgDOI8n4n7MlK9ncU0r+UXrVCgySWVv4wJHg85db75ekmhV67qyxwVP7tv0KzEKtEz7agxSXSu+qZsxBjoKHIQpjlhrT9mc7cRAvB/OxTxvi8xexAC7fvf563Ttwoejx11F6Bs3qXElIhDDtTKT0O8S0eWIFcHoWMrs+nIdvcsbXaQLHL4+E2+Uufjrp3f+8nyC]]}
ownerrules=[PolicyRule{APIGroups:["authorization.k8s.io"],
Resources:["selfsubjectaccessreviews" "selfsubjectrulesreviews"],
Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis"
"/apis/*" "/healthz" "/openapi" "/openapi/*" "/swagger-2.0.0.pb-v1"
"/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version" "/version/"],
Verbs:["get"]}] ruleResolutionErrors=[]
Kindly let me know if anyone needs any further information on this.
I have my Kubernetes cluster running in GKE I want to run an application outside the cluster and talk to the Kubernetes API.
By using password retrieved from running:
gcloud container clusters get-credentials cluster-2 --log-http
I am able to access the API with Basic authentication.
But I want to create multiple Kubernetes service accounts and configure them with required authorization and use appropriately.
So, I created service accounts and obtained the tokens using:
kubectl create serviceaccount sauser1
kubectl get serviceaccounts sauser1 -o yaml
kubectl get secret sauser1-token-<random-string-as-retrieved-from-previous-command> -o yaml
If I try to access the Kubernetes API with the obtained token using Bearer authentication then I get a 401 HTTP error. I thought that some permissions may have to be set for the service account, so based on the documentation here, I created below YAML file:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: pod-reader
subjects:
- kind: ServiceAccount
name: sauser1
namespace: default
roleRef:
kind: ClusterRole
name: pod-reader
apiGroup: rbac.authorization.k8s.io
and tried to apply it using the below command:
kubectl apply -f default-sa-rolebinding.yaml
I got the following error:
clusterrolebinding "pod-reader" created
Error from server (Forbidden): error when creating "default-sa-rolebinding.yaml"
: clusterroles.rbac.authorization.k8s.io "pod-reader" is forbidden: attempt to g
rant extra privileges: [PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["g
et"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule
{Resources:["pods"], APIGroups:[""], Verbs:["list"]}] user=&{xyz#gmail.
com [system:authenticated] map[authenticator:[GKE]]} ownerrules=[PolicyRule{Res
ources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:[
"create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healt
hz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/versi
on"], Verbs:["get"]}] ruleResolutionErrors=[]
I dont know how to proceed from here. Is my approach correct or am I missing something here?
UPDATE: As per the post referred by #JanosLenart in the comments, modified the kubectl command and the above error is not observed. But accessing the API, still gives 401 error. The curl command that I am using is:
curl -k -1 -H "Authorization: Bearer <token>" https://<ip-address>/api/v1/namespaces/default/pods -v
#Janos pointed out the potential problem, however I think you will need an actual Cloud IAM Service Account as well, because you said:
I want to run an application outside the cluster [...]
If you're authenticating to GKE from outside, I believe you can only use the Google IAM identities. (I might be wrong, if so, please let me know.)
In this case, what you need to do:
Create an IAM service account and download a json key file of it.
set GOOGLE_APPLICATION_CREDENTIALS to that file.
either:
use RBAC like in your question to give permissions to the email address of the IAM Service Account
use IAM Roles to give the IAM Service Account on the GKE API (e.g. Container Developer role is usually sufficient)
Use kubectl command against the cluster (make sure you have a .kube/config file with the cluster's IP/CA cert beforehand) with the environment variable above, it should work.
YMMV
I managed to get it work without USING an actual Cloud IAM Service Account
First, I decided to use an shell inside GKE's k8s cluster by running
kubectl run curl-random --image=radial/busyboxplus:curl -i --tty --rm
Second, I made sure I decoded my token by copying the token and then running through
pbpaste | base64 -D
Third, I created the rolebinding for the serviceaccount, NOT the username.
kubectl create clusterrolebinding shaoserverless-cluster-admin-binding --clusterrole=cluster-admin --serviceaccount=default:shaoserverless
The third step was particularly tricky but I got the inspiration since the error message used to be
Unknown user \"system:serviceaccount:default:shaoserverless\"",
Lastly, then this works
curl -k -1 -H "Authorization: Bearer <token>" https://<ip-address>/api/v1/namespaces/default/pods -v