k3s: permissions necessary to access metrics server? - kubernetes

I'd like to grant a service account the ability to access the metrics exposed by the metrics-server service (https://metrics-server.kube-system/metrics). If I create a serviceaccount...
apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-reader
namespace: prometheus
...and then grant it cluster-admin privileges...
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metrics-reader-crb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: metrics-reader
namespace: prometheus
...it works! I can use the account token to access the metrics server:
curl -k --header "Authorization: Bearer $token" https://metrics-server.kube-system/metrics
But I don't want to require cluster-admin access just to read
metrics. I tried to use the view cluster role instead of
cluster-admin, but that fails.
Is there an existing role that would grant the appropriate access?
If not, what are the specific permissions necessary to grant read-only
access to the metrics-server /metrics endpoint?

Interesting question. I've found some info for you, however i'm not sure that 100% helpful. It needs more research and reproduce.
check RBAC Deny when requesting metrics. Smth like below?
apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-reader
namespace: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: view-metrics
rules:
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: view-metrics
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: view-metrics
subjects:
- kind: ServiceAccount
name: metrics-reader
namespace: prometheus
It seems, there is a aggregated-metrics-reader clusterrole (or there was)
Aggregated ClusterRoles are documented in:
https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles.
The purpose of the system:aggregated-metrics-reader ClusterRole, is to
aggregate the rules, that grant permission to get the pod and node
metrics, to the view, edit and admin roles.
however I wasnt able to find any reference to aggregated-metrics-reader clusterrole in current version of that doc.
You can find huge example of using this clusterrole in Metrics server unable to scrape
IN addition check This adds the aggregated-metrics-reader ClusterRole which was missing github PR:
What this PR does / why we need it: This adds the
aggregated-metrics-reader ClusterRole which was missing, and seems to
be required for k8s 1.8+ per the metrics-server documentation and
default deploy manfiests
Unfortunately link in that PR direct to nowhere. I start thinking this obsolete info for 1.8 clusters.. Will update answer in case find anything more relevant

Related

Kubernetes service account to access all the namespaces

I am trying to access all the namespaces and pods from my another pod. So, I have created clusterrole, clusterrolebinding and service account. I am able access the only customer namespace resources. But I need to access all the namespace resources. Is it possible?
apiVersion: v1
kind: ServiceAccount
metadata:
name: spinupcontainers
namespace: customer
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: spinupcontainers
namespace: customer
rules:
- apiGroups: [""]
resources: ["pods", "pods/exec"]
verbs: ["get", "list", "delete", "patch", "create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: spinupcontainers
namespace: customer
subjects:
- kind: ServiceAccount
name: spinupcontainers
roleRef:
kind: ClusterRole
name: spinupcontainers
apiGroup: rbac.authorization.k8s.io
Could anyone help to resolve this problem?
Thanks in advance
It seems in your YAML example you are using a RoleBinding as opposed to a ClusterRoleBinding. A RoleBinding only grants those permissions inside of a namespace. See also the Kubernetes Documentation on this topic:
A RoleBinding grants permissions within a specific namespace whereas a
ClusterRoleBinding grants that access cluster-wide.
Most important thing is that you have to connect your service account to your cluster role with proper cluster role binding. Because binding types decide that scope of service account abilities. Under these circumstances, you have to describe cluster role binding as shown below;
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: spinupcontainers
subjects:
- kind: ServiceAccount
name: spinupcontainers
namespace: customer
roleRef:
kind: ClusterRole
name: spinupcontainers
apiGroup: "rbac.authorization.k8s.io"
If you want to test this within the pod you would describe respective service account for pod like below:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: busybox
name: busybox
spec:
containers:
- args:
- sleep
- "4800"
image: busybox:1.28
name: busybox
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Never
serviceAccountName: default
status: {}
And then finally you need to ssh to pod and can execute proper curl command with using service account token. Do not forget that you can find the token file in pod by defined service account to pod yaml before (in /var/run/secrets/kubernetes.io/serviceaccount). After that you have to execute API call to use kubernetes API server service (ıf you used kubeadm to create the cluster. It has been already defined in default namespace as named kubernetes). In the below, you can find proper apı call to get default namespace secrets
curl -k -H "Authorization: Bearer $TOKEN" https://<kubernetes-apı-fqdn>/api/v1/namespaces/default/secrets

Kubernetes cluster role with permissions to watch events

I'm trying to create a cluster role with permissions to watch events, but it seems that I'm missing something.
I'm using the following:
apiVersion: v1
kind: ServiceAccount
metadata:
name: watch-events
namespace: test
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: watch-events-cluster
rules:
- apiGroups:
- ""
resources:
- events
verbs:
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: watch-events-cluster
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: watch-events-cluster
subjects:
- kind: ServiceAccount
name: watch-events
namespace: test
No mater what I try with kubectl auth can-i watch events --as watch-events I always get a no.
Am I missing something?
The RBAC is correct and will give cluster wide permission to watch events across all namespaces but the kubectl command is incorrect.The command should be
kubectl auth can-i watch events --as=system:serviceaccount:test:watch-events
If you are making api calls against the swagger api for Kubernetes, you need to specify the Events api group properly with the suffix .k8s.io
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#-strong-api-groups-strong-
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: my-custom-role
namespace: default
rules:
- apiGroups:
- ''
- events.k8s.io
resources:
- events
verbs:
- '*'
---
https://kubernetes.io/docs/reference/access-authn-authz/rbac/#service-account-permissions
Default RBAC policies grant scoped permissions to control-plane components, nodes, and controllers, but grant no permissions to service accounts outside the kube-system namespace (beyond discovery permissions given to all authenticated users).

ClusterRoleBinding requires namespace

I have the following:
apiVersion: v1
kind: ServiceAccount
metadata:
name: SomeServiceAccount
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: SomeClusterRole
rules:
- apiGroups:
- "myapi.com"
resources:
- 'myapi-resources'
verbs:
- '*'
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: SomeClusterRoleBinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: SomeClusterRole
subjects:
- kind: ServiceAccount
name: SomeServiceAccount
But it throws:
The ClusterRoleBinding "SomeClusterRoleBinding" is invalid: subjects[0].namespace: Required value
I thought the whole point of "Cluster"RoleBinding is that it's not limited to a single namespace. Anyone can explain this?
Kubernetes version 1.13.12
Kubectl version v1.16.2
Thanks.
You are not required set a namespace while creating a ServiceAccount, the case here is that you are required to specify the namespace of your Service account when you refer to it while creating a ClusterRoleBinding to select it.
ServiceAccounts are namespace scoped subjects, so when you refer to
them, you have to specify the namespace of the service account you
want to bind. Source
In your case you can just use default namespace while creating your ClusterRoleBinding for example.
By doing this you are not tieing your ClusterRoleBinding to any namespace, as you can see in this example.
$ kubectl get clusterrolebinding.rbac.authorization.k8s.io/tiller -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"tiller"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"cluster-admin"},"subjects":[{"kind":"ServiceAccount","name":"tiller","namespace":"kube-system"}]}
creationTimestamp: "2019-11-18T13:47:59Z"
name: tiller
resourceVersion: "66715"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/tiller
uid: 085ed826-0a0a-11ea-a665-42010a8000f7
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
A kubernetes service account is scoped to a namespace. If you don't specify the namespace while creating the service account, your service account gets created in the 'default' namespace.
This allows you to create service accounts with the same name in different namespaces. i.e. All the namespaces have a service account named "default" when the namespace gets created.
To create service account in a namespace:
kubectl create serviceaccount my-sa -n my-namespace
The namespaces you have to put in the subject here refers to "where the service account is" and not "which namespaces this cluster role binding binds the resource access to".
The cluster-wide aspect of a ClusterRole is that the resources in the rules are cluster-wide. For example, you could use a ClusterRole to give a subject get access to all Pods in all namespaces. With a Role, you could only give a subject get access to Pods in specific namespaces.
The cluster-wide aspect of a ClusterRoleBinding does not apply in any way to the subjects of the binding. In your example, you cannot create a binding for all service accounts with a particular name in all namespaces.
Indeed, you're right, the ClusterRoleBinding should not be tied to a namespace, but
I think the Subject may be tied or not (that's what the error says). You can debug this by checking the Kubernetes API spec for your specific version.
For example here, on the subject's namespace, it says: If the object kind is non-namespace, such as "User" or "Group", and this value is not empty the Authorizer should report an error.

RBAC: roles with multiple namespaces

Trying to write my first set of RBAC roles. So trying to figure out the best way to have 2 roles for multiple namespaced components.
Admin-role (RW for 3 namespaces say default, ns1 & ns2)
user-role (Read-only for 3 namespaces say default, ns1 & ns2)
Was thinking will need a service account with 2 clusterRoles for admin/user
apiVersion: rbac.authorization.k8s.io/v1
kind: ServiceAccount
metadata:
name: sa
namespace: default
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: admin-master
rules:
- apiGroups:
- batch
resources:
- pods
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: user-master
rules:
- apiGroups:
- batch
resources:
- pods
verbs:
- get
- list
- watch
Then make use of roleBindings:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: admin-rw
namespace: ns1
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: admin-master
subjects:
- kind: ServiceAccount
name: sa
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: user-readonly
namespace: ns1
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: user-master
subjects:
- kind: ServiceAccount
name: sa
namespace: default
But not sure how the best way to bind roles admin-rw/user-readonly with namespace 2 (ns2)?
Roles are scoped, either bound to an specific namespace or cluster-wide. For namespace-scoped roles, you can just simply deploy the same role in multiple namespaces.
The idea behind this is to have partitioned permissions in the cluster, although it implies more administrative effort but is a safer practice.
Additionally, in your definition, you're trying to bind permissions to specific namespaces, however, you're using ClusterRole which is a cluster-scoped resource. You might want to change that to Role if you want namespace-scoped permissions.
You might find this CNCF article useful on this matter.
This answer https://stackoverflow.com/a/57729174/2660452 is wrong.
ClusterRole and Role defines which resources you can operated. if your role need to manage resources in multiple namespaces, you need to use ClusterRole
and RoleBinding defines which namespace your account will be granted.
here is the example from official document: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-example
A RoleBinding can also reference a ClusterRole to grant the permissions defined in that ClusterRole to resources inside the RoleBinding's namespace. This kind of reference lets you define a set of common roles across your cluster, then reuse them within multiple namespaces.
For instance, even though the following RoleBinding refers to a ClusterRole, "dave" (the subject, case sensitive) will only be able to read Secrets in the "development" namespace, because the RoleBinding's namespace (in its metadata) is "development".
apiVersion: rbac.authorization.k8s.io/v1
# This role binding allows "dave" to read secrets in the "development" namespace.
# You need to already have a ClusterRole named "secret-reader".
kind: RoleBinding
metadata:
name: read-secrets
#
# The namespace of the RoleBinding determines where the permissions are granted.
# This only grants permissions within the "development" namespace.
namespace: development
subjects:
- kind: User
name: dave # Name is case sensitive
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: secret-reader
apiGroup: rbac.authorization.k8s.io

Openshift Admin Token

I am trying to create a script that records project resources every 15 minutes. How do I authenticate with Openshift API? Is there a token I can use that has read access on all namespaces? How do I create a service account that has access over all namespaces?
You'll need to create a ClusterRole that has read access to the resources and use ClusterRoleBinding to associate the ServiceAccount to that ClusterRole. Rough example, not tested but it should work:
# creates the service account "ns-reader"
apiVersion: v1
kind: ServiceAccount
metadata:
name: ns-reader
namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
# "namespace" omitted since ClusterRoles are not namespaced
name: global-reader
rules:
- apiGroups: [""]
# add other rescources you wish to read
resources: ["pods", "secrets"]
verbs: ["get", "watch", "list"]
---
# This cluster role binding allows service account "ns-reader" to read pods in all available namespace
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-ns
subjects:
- kind: ServiceAccount
name: ns-reader
namespace: default
roleRef:
kind: ClusterRole
name: global-reader
apiGroup: rbac.authorization.k8s.io
When the ServiceAccount is setup, a number of secrets are created automatically associated with it. A couple of these secrets hold a token which can then be used when using the REST API directly or using oc. Use oc describe on the ServiceAccount to see the names of the Secret for the tokens. Then use oc describe on one of the Secrets to see the token.