Forbidden resource in API group at the cluster scope - kubernetes

Question

I am unable to identify what the exact issue with the permissions with my setup as shown below. I've looked into all the similar QAs but still unable to solve the issue. The aim is to deploy Prometheus and let it scrape /metrics endpoints that my other applications in the cluster expose fine.
Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"services\" in API group \"\" at the cluster scope"
...
...
The command below returns no to all services, nodes, pods etc.
kubectl auth can-i get services --as=system:serviceaccount:default:default -n default
Minikube
$ minikube start --vm-driver=virtualbox --extra-config=apiserver.Authorization.Mode=RBAC
😄 minikube v1.14.2 on Darwin 11.2
✨ Using the virtualbox driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🔄 Restarting existing virtualbox VM for "minikube" ...
🐳 Preparing Kubernetes v1.19.2 on Docker 19.03.12 ...
▪ apiserver.Authorization.Mode=RBAC
🔎 Verifying Kubernetes components...
🌟 Enabled addons: storage-provisioner, default-storageclass, dashboard
🏄 Done! kubectl is now configured to use "minikube" by default
Roles
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: monitoring-cluster-role
rules:
- apiGroups: [""]
resources: ["nodes", "services", "pods", "endpoints"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get"]
- apiGroups: ["extensions"]
resources: ["deployments"]
verbs: ["get", "list", "watch"]
apiVersion: v1
kind: ServiceAccount
metadata:
name: monitoring-service-account
namespace: default
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: monitoring-cluster-role-binding
roleRef:
kind: ClusterRole
name: monitoring-cluster-role
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: monitoring-service-account
namespace: default
Prometheus
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config-map
namespace: default
data:
prometheus.yml: |
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-deployment
namespace: default
labels:
app: prometheus
spec:
replicas: 1
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
containers:
- name: prometheus
image: prom/prometheus:latest
ports:
- name: http
protocol: TCP
containerPort: 9090
volumeMounts:
- name: config
mountPath: /etc/prometheus/
- name: storage
mountPath: /prometheus/
volumes:
- name: config
configMap:
name: prometheus-config-map
- name: storage
emptyDir: {}
apiVersion: v1
kind: Service
metadata:
name: prometheus-service
namespace: default
spec:
type: NodePort
selector:
app: prometheus
ports:
- name: http
protocol: TCP
port: 80
targetPort: 9090

Answer 1

User "system:serviceaccount:default:default" cannot list resource "endpoints" in API group "" at the cluster scope"
User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope"
User "system:serviceaccount:default:default" cannot list resource "services" in API group "" at the cluster scope"
Something running with ServiceAccount default in namespace default is doing things it does not have permissions for.
apiVersion: v1
kind: ServiceAccount
metadata:
name: monitoring-service-account
Here you create a specific ServiceAccount. You also give it some Cluster-wide permissions.
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-deployment
namespace: default
You run Prometheus in namespace default but do not specify a specific ServiceAccount, so it will run with ServiceAccount default.
I think your problem is that you are supposed to set the ServiceAccount that you create in the Deployment-manifest for Prometheus.