Two Kubernetes clusters act differently for RBAC - kubernetes

Question

I have created an application that needs to have access to list, create, update and delete different Kubernetes resources and I created a clusterrole for it as below. Everything works fine on my local K8s cluster that runs on Microk8s but when I deployed it on a bare-metal cluster with the same version of K8s I am getting errors that I don't have proper access.
How this is possible (both should act the same) and is there a way to find these errors in advance?
My clusterrole:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: {{ .Release.Namespace }}-cluster-manager-role
rules:
- apiGroups: ["","apps","core", "autoscaling"] # --> I was getting error that I cannot create HPA but after I added "autoscaling" to the apigroup now I can create HPA
resources: ["*", "namespaces"]
verbs: ["get", "watch", "list", "patch", "create", "delete", "update"]
# ================
# Current clusterrole on microk8s (which allows me to do all the things)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2021-05-31T12:05:58Z"
name: default-cluster-manager-role
resourceVersion: "937643"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/default-cluster-manager-role
uid: 16fb63d6-1261-48a9-bc7f-5c8fffb72c9d
rules:
- apiGroups:
- ""
- apps
- core
resources:
- '*'
- namespaces
verbs:
- get
- watch
- list
- patch
- create
- delete
- update
Kubernetes version:
# Microk8s
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15", GitCommit:"2adc8d7091e89b6e3ca8d048140618ec89b39369", GitTreeState:"clean", BuildDate:"2020-09-02T11:31:21Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
# Bare-metal
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15", GitCommit:"2adc8d7091e89b6e3ca8d048140618ec89b39369", GitTreeState:"clean", BuildDate:"2020-09-02T11:31:21Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Some of the errors that I got:
time="2021-06-22T08:45:31Z" level=error msg="Getting list of PVCs for namespace wws-test failed." func=src/k8s.CreateClusterRole file="/src/k8s/k8s.go:1304"
time="2021-06-22T08:45:31Z" level=error msg="clusterroles.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:wws:wws-cluster-manager-sa\" cannot create resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope" func=src/k8s.CreateClusterRole file="/src/k8s/k8s.go:1305"
time="2021-06-22T08:45:31Z" level=error msg="Getting list of PVCs for namespace wws-test failed." func=src/k8s.CreateClusterRoleBinding file="/src/k8s/k8s.go:1232"
time="2021-06-22T08:45:31Z" level=error msg="clusterrolebindings.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:wws:wws-cluster-manager-sa\" cannot create resource \"clusterrolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope" func=src/k8s.CreateClusterRoleBinding file="/src/k8s/k8s.go:1233"
time="2021-06-22T08:45:32Z" level=error msg="Getting list of PVCs for namespace wws-test failed." func=src/k8s.CreateRole file="/src/k8s/k8s.go:1448"
time="2021-06-22T08:45:32Z" level=error msg="roles.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:wws:wws-cluster-manager-sa\" cannot create resource \"roles\" in API group \"rbac.authorization.k8s.io\" in the namespace \"wws-test\"" func=src/k8s.CreateRole file="/src/k8s/k8s.go:1449"

Answer 1

You should have a look at the ClusterRoleBindings (k get ClusterRoleBinding -o wide) that are applied to the ServiceAccount: system:serviceaccount:wws:wws-cluster-manager-sa)
I guess that on Minikube your user can do anything on your local cluster.
But, the real cluster would not allow you to create new ClusterRoles/CluterRoleBindings with the default user.

Answer 2

I don't know why this happened but I solved the issue by using * for all three fields of apiGroups, resources and verbs:
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
I am aware that this is not a clean and perfect solution especially if you want more control on the role and the resources or the verbs that the role should have access to but since no one (even I posted this on Kubernetes repo github as an issue) knows why this happened and I don't have time to dig deeper into this I accept my own answer.