I can't get a rolebinding right in order to get node status from an app which runs in a pod on GKE.
I am able to create a pod from there but not get node status.
He is the role I am creating:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: node-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["nodes"]
verbs: ["get", "watch", "list"]
This is the error I get when I do a getNodeStatus:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "nodes \"gke-cluster-1-default-pool-36c26e1e-2lkn\" is forbidden: User \"system:serviceaccount:default:sa-poc\" cannot get nodes/status at the cluster scope: Unknown user \"system:serviceaccount:default:sa-poc\"",
"reason": "Forbidden",
"details": {
"name": "gke-cluster-1-default-pool-36c26e1e-2lkn",
"kind": "nodes"
},
"code": 403
}
I tried with some minor variations but did not succeed.
Kubernetes version on GKE is 1.8.4-gke.1
Any help appreciated, thanks!
Subresource permissions are represented as <resource>/<subresource>, so in the role, you would specify resources: ["nodes","nodes/status"]
Related
I have deploy a k8s cluster with kubeadm, I want to get controller manager's metrics with following command:
curl -k https://localhost:10257/metrics
but got the following error:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
"reason": "Forbidden",
"details": {
},
"code": 403
}
So my question is, how to get k8s controller manager's metrics?
This is a forbidden error due to permission issues which need to be authenticated with a valid user. For this,You need to create a service account, then give that service account access permissions to the metrics Path through RBAC, then this will make that service account to get the metrics.
As per this Role and Cluster Binding doc, you need to allow metrics path(replace with /healthz) as below and give a try.
Allow GET and POST requests to the non-resource endpoint /healthz and all subpaths (must be in a ClusterRole bound with a ClusterRoleBinding to be effective):
rules:
- nonResourceURLs: ["/healthz", "/healthz/*"] # '*' in a nonResourceURL is a suffix glob match
verbs: ["get", "post"]
I am trying to deploy applications on GKE using REST APIs. However, the GKE documentation is all mixed up and unclear as to how to enable the Kubernetes REST API access.
Does anyone here have a clear idea about how to create a Deployment on Kubernetes cluster on Google Cloud?
If yes, I would love to know the detailed steps for enabling the same. Currently, this is what I get.
https://xx.xx.xx.xx/apis/apps/v1/namespaces/default/deployments/nginx-1 GET call gives below JSON output despite valid authorization token
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "deployments.apps \"nginx-1\" is forbidden: User \"system:serviceaccount:default:default\" cannot get resource \"deployments\" in API group \"apps\" in the namespace \"default\"",
"reason": "Forbidden",
"details": {
"name": "nginx-1",
"group": "apps",
"kind": "deployments"
},
"code": 403
}
Administration APIs however seems to be enabled:
Following the instructions at this link and executing the below commands:
# Check all possible clusters, as your .KUBECONFIG may have multiple contexts:
kubectl config view -o jsonpath='{"Cluster name\tServer\n"}{range .clusters[*]}{.name}{"\t"}{.cluster.server}{"\n"}{end}'
# Select name of cluster you want to interact with from above output:
export CLUSTER_NAME="some_server_name"
# Point to the API server referring the cluster name
APISERVER=$(kubectl config view -o jsonpath="{.clusters[?(#.name==\"$CLUSTER_NAME\")].cluster.server}")
# Gets the token value
TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(#.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}"|base64 --decode)
# Explore the API with TOKEN
curl -X GET $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure
gives the desired output.
The service account default in default namespace does not have RBAC to perform get verb on deployment resource in default namespace.
Use below role and rolebinding to provide the necessary permission to the service account.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: deployment-reader
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-deployment
namespace: default
subjects:
# You can specify more than one "subject"
- kind: ServiceAccount
name: default # "name" is case sensitive
namespace: default
roleRef:
# "roleRef" specifies the binding to a Role / ClusterRole
kind: Role #this must be Role or ClusterRole
name: deployment-reader # this must match the name of the Role or ClusterRole you wish to bind to
apiGroup: rbac.authorization.k8s.io
To verify the permission
kubectl auth can-i get deployments --as=system:serviceaccount:default:default -n default
yes
When accessing the k8s api endpoint (FQDN:6443), a successful retrieval will return a JSON object containing REST endpoint paths. I have a user who is granted cluster-admin privileges on the cluster who is able to successfully interact with that endpoint.
I've created a certificate for another user and granted them a subset of privileges in my cluster. The error I'm attempting to correct: They cannot access FQDN:6443, but instead get a 403 with a message that "User cannot get path /". I get the same behavior whether I specify as FQDN:6443/ or FQDN:6443 (no trailing slash). I've examined the privileges granted to cluster-admin role users, but have not recognized the gap.
Other behavior: They CAN access FQDN:6443/api, which I have not otherwise explicitly granted them, as well as the various endpoints I have explicitly granted. I believe they the api endpoint via the system:discovery role granted to the system:authenticated group. Also, if I attempt to interact with the cluster without a certificate, I correctly am identified as an anonymous user. If I interact with the cluster with a certificate whose user name does not match my rolebindings, I get the expected behaviors for all but the FQDN:6443 endpoint.
I had a similar similar issue. I was trying to curl the base url: https://api_server_ip:6443 with the correct certificates.
I got this error:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"kubernetes\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {
},
"code": 403
}
It appears the system:discovery doesn't grant access to the base url: https://api_server_ip:6443/. The system:discovery roles only gives access to the following paths:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:discovery
rules:
- nonResourceURLs:
- /api
- /api/*
- /apis
- /apis/*
- /healthz
- /openapi
- /openapi/*
- /swagger-2.0.0.pb-v1
- /swagger.json
- /swaggerapi
- /swaggerapi/*
- /version
- /version/
verbs:
- get
No access to / granted. So I created the following ClusterRole which I called discover_base_url. It grants access to the / path:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: discover_base_url
rules:
- nonResourceURLs:
- /
verbs:
- get
Then I created a ClusterRoleBinding binding the forbidden user "kubernetes", (it could be any user) to the above cluster role. The following is the yaml for the ClusterRoleBinding (replace "kubernetes" with your user):
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: discover-base-url
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: discover_base_url
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubernetes
After creating these two resources, the curl request works:
curl --cacert ca.pem --cert kubernetes.pem --key kubernetes-key.pem https://api_server_ip:6443
{
"paths": [
"/api",
"/api/v1",
"/apis",
"/apis/",
"/apis/admissionregistration.k8s.io",
"/apis/admissionregistration.k8s.io/v1beta1",
"/apis/apiextensions.k8s.io",
"/apis/apiextensions.k8s.io/v1beta1",
"/apis/apiregistration.k8s.io",
"/apis/apiregistration.k8s.io/v1",
"/apis/apiregistration.k8s.io/v1beta1",
"/apis/apps",
"/apis/apps/v1",
"/apis/apps/v1beta1",
"/apis/apps/v1beta2",
"/apis/authentication.k8s.io",
"/apis/authentication.k8s.io/v1",
"/apis/authentication.k8s.io/v1beta1",
"/apis/authorization.k8s.io",
"/apis/authorization.k8s.io/v1",
"/apis/authorization.k8s.io/v1beta1",
"/apis/autoscaling",
"/apis/autoscaling/v1",
"/apis/autoscaling/v2beta1",
"/apis/autoscaling/v2beta2",
"/apis/batch",
"/apis/batch/v1",
"/apis/batch/v1beta1",
"/apis/certificates.k8s.io",
"/apis/certificates.k8s.io/v1beta1",
"/apis/coordination.k8s.io",
"/apis/coordination.k8s.io/v1beta1",
"/apis/events.k8s.io",
"/apis/events.k8s.io/v1beta1",
"/apis/extensions",
"/apis/extensions/v1beta1",
"/apis/networking.k8s.io",
"/apis/networking.k8s.io/v1",
"/apis/policy",
"/apis/policy/v1beta1",
"/apis/rbac.authorization.k8s.io",
"/apis/rbac.authorization.k8s.io/v1",
"/apis/rbac.authorization.k8s.io/v1beta1",
"/apis/scheduling.k8s.io",
"/apis/scheduling.k8s.io/v1beta1",
"/apis/storage.k8s.io",
"/apis/storage.k8s.io/v1",
"/apis/storage.k8s.io/v1beta1",
"/healthz",
"/healthz/autoregister-completion",
"/healthz/etcd",
"/healthz/log",
"/healthz/ping",
"/healthz/poststarthook/apiservice-openapi-controller",
"/healthz/poststarthook/apiservice-registration-controller",
"/healthz/poststarthook/apiservice-status-available-controller",
"/healthz/poststarthook/bootstrap-controller",
"/healthz/poststarthook/ca-registration",
"/healthz/poststarthook/generic-apiserver-start-informers",
"/healthz/poststarthook/kube-apiserver-autoregistration",
"/healthz/poststarthook/rbac/bootstrap-roles",
"/healthz/poststarthook/scheduling/bootstrap-system-priority-classes",
"/healthz/poststarthook/start-apiextensions-controllers",
"/healthz/poststarthook/start-apiextensions-informers",
"/healthz/poststarthook/start-kube-aggregator-informers",
"/healthz/poststarthook/start-kube-apiserver-admission-initializer",
"/healthz/poststarthook/start-kube-apiserver-informers",
"/logs",
"/metrics",
"/openapi/v2",
"/swagger-2.0.0.json",
"/swagger-2.0.0.pb-v1",
"/swagger-2.0.0.pb-v1.gz",
"/swagger-ui/",
"/swagger.json",
"/swaggerapi",
"/version"
]
}
Not being able to create a ClusterRole using K8s REST APIs. I receive "forbidden: attempt to grant extra privileges" error. Even though can create same ClusterRole using 'kubectl apply'. Using same user. Running in GCP. Version: '1.11.6-gke.3'.
Here are my steps:
1. IAM Config
IAM User: berlioz-robot#xxx.iam.gserviceaccount.com
Role: Kubernetes Engine Admin
2. Make the user admin
apply using kubectl:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: berlioz:robot-cluster-admin-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: berlioz-robot#xxx.iam.gserviceaccount.com
3. Generate Login Token
Header:
{
"alg": "RS256",
"typ": "JWT",
"kid": "xxxxxxxxxxxxxxxxxxxxxxxxxx"
}
Payload:
{
"iss": "berlioz-robot#xxx.iam.gserviceaccount.com",
"sub": "berlioz-robot#xxx.iam.gserviceaccount.com",
"aud": "https://www.googleapis.com/oauth2/v4/token",
"scope": "https://www.googleapis.com/auth/cloud-platform",
"iat": 1548743213,
"exp": 1548746813
}
4. Login
URL: https://www.googleapis.com/oauth2/v4/token
Method: POST
Body: {
'grant_type': 'urn:ietf:params:oauth:grant-type:jwt-bearer',
'assertion': here-goes-the-signed-token
}
Result:
{
"access_token": "ya29.xxxxxxxxxxxxxxxx",
"expires_in": 3600,
"token_type": "Bearer"
}
5. Create new ClusterRole with REST APIs
URL: https://CLUSTER-IP-ADDRESS/apis/rbac.authorization.k8s.io/v1/clusterroles
Method: POST
Headers: {
Authorization: "Bearer ya29.xxxxxxxxxxxxxxxx",
Content-Type: "application/json"
}
Body: {
"metadata": {
"name": "berlioz:controller-cluster-role"
},
"rules": [
{
"verbs": [
"get",
"list",
"watch"
],
"apiGroups": [
""
],
"resources": [
"nodes"
]
}
],
"kind": "ClusterRole",
"apiVersion": "rbac.authorization.k8s.io/v1"
}
RESULT:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "clusterroles.rbac.authorization.k8s.io \"berlioz:controller-cluster-role-test\" is forbidden: attempt to grant extra privileges: [{[get] [] [nodes] [] []} {[list] [] [nodes] [] []} {[watch] [] [nodes] [] []}] user=&{110887992956644566571 [system:authenticated] map[user-assertion.cloud.google.com:[xxxxx]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]",
"reason": "Forbidden",
"details": {
"name": "berlioz:controller-cluster-role-test",
"group": "rbac.authorization.k8s.io",
"kind": "clusterroles"
},
"code": 403
}
Interestingly, if I set list of rules empty, things go through. As mentioned above, same cluster role is getting created using kubectl successfully.
According to Google cloud RBAC documentation:
In GKE, Cloud IAM and Kubernetes RBAC are integrated to authorize users to perform actions if they have sufficient permissions according to either tool. This is an important part of bootstrapping a GKE cluster, since by default GCP users do not have any Kubernetes RBAC RoleBindings.
Once users or GCP service accounts are authenticated, they must also be authorized to perform any action on a GKE cluster.
In GKE clusters using GKE v1.11.x and older, there is a limitation that Cloud IAM cannot grant the ability to create a Kubernetes RBAC Role or ClusterRole. However, the Kubernetes Engine Admin Cloud IAM role does grant users with the ability to create a Kubernetes RBAC RoleBinding or ClusterRoleBinding for any user, including themselves, which can be used to bind GCP users to predefined RBAC Roles.
In particular, the cluster-admin predefined RBAC role grants users full permissions in the cluster. Therefore, to bootstrap a user to allow them to create RBAC Roles and ClusterRoles, issue the following command, replacing [USER_ACCOUNT] with the target user's GCP login email address.
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin \
--user [USER_ACCOUNT]
Note: The [USER_ACCOUNT] is case sensitive. To avoid errors, enter the target user's email address in lowercase.
Alternatively you can use the following yaml:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: cluster-admin-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: username#google-account-domain.com
After creating such ClusterRoleBinding you'll be able to create ClusterRole.
I created a cluster role "try-usr"
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: try-usr
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- get
- list
- watch
While accessing the Web UI(Dashboard), it's throwing an error as follows:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "services \"https:kubernetes-dashboard:\" is forbidden: User \"xyz\" cannot get services/proxy in the namespace \"kube-system\"",
"reason": "Forbidden",
"details": {
"name": "https:kubernetes-dashboard:",
"kind": "services"
},
"code": 403
}
Depending on the kubernetes version, the dashboard will require different permissions according to the docs
v1.7
create and watch permissions for secrets in kube-system namespace required to - create and watch for changes of kubernetes-dashboard-key-holder secret.
get, update and delete permissions for secrets named kubernetes-dashboard-key-holder and kubernetes-dashboard-certs in kube-system namespace.
proxy permission to heapster service in kube-system namespace required to allow getting metrics from heapster.
v1.8
create permission for secrets in kube-system namespace required to create kubernetes-dashboard-key-holder secret.
get, update and delete permissions for secrets named kubernetes-dashboard-key-holder and kubernetes-dashboard-certs in kube-system namespace.
get and update permissions for config map named kubernetes-dashboard-settings in kube-system namespace.
proxy permission to heapster service in kube-system namespace required to allow getting metrics from heapster.