So i have 3 name spaces when i deployed prometheus on kubernetes i see the error in the logs. it is unable to monitor all the name spaces.
Error :
\"system:serviceaccount:development:default\" cannot list endpoints at the cluster scope"
level=error ts=2018-06-28T21:22:07.390161824Z caller=main.go:216 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:268: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:devops:default\" cannot list endpoints at the cluster scope"
You'd better use a service account to access the kubernetes, and give the sa special privilidge that the prometheus needed. like the following:
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: kube-system
Presumes that you deploy prometheus in the kube-system namespace. Also you need specify the sa like ' serviceAccount: prometheus' in your prometheus deployment file .
Related
Here my first ServiceAccount, ClusterRole, And ClusterRoleBinding
---
# Create namespace
apiVersion: v1
kind: Namespace
metadata:
name: devops-tools
---
# Create Service Account
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: devops-tools
name: bino
---
# Set Secrets for SA
# k8s >= 1.24 need to manualy created
# https://stackoverflow.com/a/72258300
apiVersion: v1
kind: Secret
metadata:
name: bino-token
namespace: devops-tools
annotations:
kubernetes.io/service-account.name: bino
type: kubernetes.io/service-account-token
---
# Create Cluster Role
# Beware !!! This is Cluster wide FULL RIGHTS
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: devops-tools-role
namespace: devops-tools
rules:
- apiGroups:
- ""
- apps
- autoscaling
- batch
- extensions
- policy
- networking.k8s.io
- rbac.authorization.k8s.io
resources:
- pods
- componentstatuses
- configmaps
- daemonsets
- deployments
- events
- endpoints
- horizontalpodautoscalers
- ingress
- jobs
- limitranges
- namespaces
- nodes
- pods
- persistentvolumes
- persistentvolumeclaims
- resourcequotas
- replicasets
- replicationcontrollers
- serviceaccounts
- services
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
# Bind the SA to Cluster Role
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: devops-tools-role-binding
subjects:
- namespace: devops-tools
kind: ServiceAccount
name: bino
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: devops-tools-role
---
It work when I use to create NameSpace, Deployment, and Service.
But it fail (complain about 'have no right') when I try to create kind: Ingress.
Then I try to add
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: devops-tools-role-binding-admin
subjects:
- namespace: devops-tools
kind: ServiceAccount
name: bino
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
and now 'bino' can do all things.
My question is: Is there any docs on what 'apiGroups' and 'resources' need to be assigned so one service account can do some-things (not all-things)?
Sincerely
-bino-
You can run this command to determine the apiGroup of a resource:
kubectl api-resources
You will see something like:
NAME SHORTNAMES APIVERSION NAMESPACED KIND
ingresses ing networking.k8s.io/v1 true Ingress
So you would need to add this to the rules of your ClusterRole:
- apiGroups:
- "networking.k8s.io/v1"
resources:
- "ingresses"
verbs:
- "get"
in order to make a high availability test in kubernetes cluster, i use a tool such as chaoskube or kube-monkey , which kills random pods in namespaces to create a "chaos" and to see how the system and applications will react.
by default these tools need a cluster role, in order to let its service account to list/kill pods for all namespaces in cluster.
in my situation i want to install this tool and make the test just in one namespace (namespace x)
is there any way to restrict the permissions of the service account just to give it the permissions to list/kill pods from (namespace x) and the whole cluster ?
i already tried to create a role & rolebinding in (namespace x) but still have the same RBAC error, as the service account expects to have the cluster permissions :
"pods is forbidden: User \"system:serviceaccount:x:chaoskube-sa\" cannot list resource \"pods\" in API group \"\ at the cluster scope"
update: role & rolebinding
this is the default permissions for its service account:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: chaoskube-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: chaoskube-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: chaoskube-role
subjects:
- kind: ServiceAccount
name: chaoskube-sa
namespace: x
with these configration it works fine.
now with restricted permissions for a specific namespace :
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: chaoskube-role
namespace: x
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: chaoskube-rolebinding
namespace: x
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: chaoskube-role
subjects:
- kind: ServiceAccount
name: chaoskube-sa
namespace: x
it can not list the pods , and i receive the RBAC error.
I'm trying to grant the default service account in my namespace the ability to read ingress resources. I want to be able to read all ingress resources for the cluster, would that necessitate a ClusterRole? This is the role and binding I've been trying.
The kubectl command kubectl auth can-i list ingress -n my-namespace --as=system:serviceaccount:my-namespace:default also returns "no"
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: my-namespace
name: my-ingress-reader
rules:
- apiGroups: ["", "networking.k8s.io", "networking", "extensions"] # "" indicates the core API group
resources: ["ingress"]
verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: my-ingress-reader
namespace: my-namespace
subjects:
- kind: ServiceAccount
name: default
namespace: my-namespace
roleRef:
kind: Role
name: my-ingress-reader
apiGroup: rbac.authorization.k8s.io
your Role rules is using incorrect api-resources that is resources: ["ingress"], it must be resources: ["ingresses"]
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: my-namespace
name: my-ingress-reader
rules:
- apiGroups: ["", "networking.k8s.io", "networking", "extensions"] # "" indicates the core API group
resources: ["ingresses"]
verbs: ["get", "watch", "list"]
to check the correct api-resources, you can use below command
root#controlplane:~# kubectl api-resources | grep -i ingress
ingresses ing extensions/v1beta1 true Ingress
ingressclasses networking.k8s.io/v1 false IngressClass
ingresses ing networking.k8s.io/v1 true Ingress
I have access to only one namespace inside the cluster and that too is restricted.
kind: Role
kind: ClusterRole
kind: RoleBinding
kind: ClusterRoleBinding
are forbidden to me. So im not able to create kubernetes dashboard as per the recommended yaml.
How to get around this?
It's not possible to achieve it unless you ask someone with enough rights to create the objects you can't for you.
Here is a sample manifest used to apply the dashboard to a cluster. As you can see you have to be able to manage Role, ClusterRole, RoleBinding and ClusterRoleBinding to apply it.
So it's impossible to create it with the rights you have as they are essential in this case.
Here is the part affected by lack of your rights:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
rules:
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster", "dashboard-metrics-scraper"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
verbs: ["get"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
rules:
# Allow Metrics Scraper to get metrics from the Metrics server
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
```
I'm trying to create a read only user. I want the user to be able to list nodes and pods and view the dashboard. I got the certs created and can connect but I'm getting the following error.
$ kubectl --context minikube-ro get pods --all-namespaces
Error from server (Forbidden): pods is forbidden: User "erst-operation" cannot list pods at the cluster scope
My cluster role...
$ cat helm/namespace-core/templates/pod-reader-cluster-role.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: '*'
name: pod-reader
rules:
- apiGroups: ["extensions", "apps"]
resources: ["pods"]
verbs: ["get", "list", "watch"]
My cluster role binding...
$ cat helm/namespace-core/templates/pod-reader-role-binding.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-pods
namespace: default
subjects:
- kind: User
name: erst-operation
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: pod-reader
apiGroup: rbac.authorization.k8s.io
I'm aware the above shouldn't grant permissions to see the dashboard but how do I get it to just list the pods?
You cluster role should contain Core group as resource pods are in Core group.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: '*'
name: pod-reader
rules:
- apiGroups: ["extensions", "apps", ""]
resources: ["pods"]
verbs: ["get", "list", "watch"]