When running (on GCP):
$ helm upgrade \
--values ./values.yaml \
--install \
--namespace "weaviate" \
"weaviate" \
weaviate.tgz
It returns;
UPGRADE FAILED
Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "configmaps" in API group "" in the namespace "ku
be-system"
Error: UPGRADE FAILED: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "configmaps" in API group "" in t
he namespace "kube-system"
UPDATE: based on solution
$ vim rbac-config.yaml
Add to the file:
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
Run:
$ kubectl create -f rbac-config.yaml
$ helm init --service-account tiller --upgrade
Note: based on Helm v2.
tl;dr: Setup Helm with the appropriate authorization settings for your cluster, see https://v2.helm.sh/docs/using_helm/#role-based-access-control
Long Answer
Your experience is not specific to the Weaviate Helm chart, rather it looks like Helm is not setup according to the cluster authorization settings. Other Helm commands should fail with the same or a similar error.
The following error
Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "configmaps" in API group "" in the namespace "ku
be-system"
means that the default service account in the kube-system namespace is lacking permissions. I assume you have installed Helm/Tiller in the kube-system namespace as this is the default if no other arguments are specified on helm init. Since you haven't created a specific Service Account for Tiller to use it defaults to the default service account.
Since you are mentioning that you are running on GCP, I assume this means you are using GKE. GKE by default has RBAC Authorization enabled. In an RBAC setting no one has any rights by default, all rights need to be explicitly granted.
The helm docs list several options on how to make Helm/Tiller work in an RBAC-enabled setting. If the cluster has the sole purpose of running Weaviate you can choose the simplest option: Service Account with cluster-admin role. The process described there essentially creates a dedicated service account for Tiller, and adds the required ClusterRoleBinding to the existing cluster-admin ClusterRole. Note that this effectively makes Helm/Tiller an admin of the entire cluster.
If you are running a multi-tenant cluster and/or want to limit Tillers permissions to a specific namespace, you need to choose one of the alternatives.
Related
I installed my jenkins using this guide:
https://www.jenkins.io/doc/book/installing/kubernetes/#install-jenkins-with-helm-v3
And also I created the service account according to the article:
kubectl apply -f jenkins-sa.yaml
My pipelines is in it github repo:
https://github.com/joedayz/node-k8s-cicd/blob/main/Jenkinsfile
But When I execute my pipeline I got it error:
helm upgrade --install --wait --set 'image.tag=22' node-app-chart ./k8s/node-app-chart
Release "node-app-chart" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists.
Unable to continue with install: could not get information about the
resource ServiceAccount "node-app-chart" in namespace "jenkins":
serviceaccounts "node-app-chart" is forbidden: User
"system:serviceaccount:jenkins:jenkins" cannot get resource
"serviceaccounts" in API group "" in the namespace "jenkins"
script returned exit code 1
I am using minikube, helm v3, docker hub, github. The repo is public.
Any idea?
Thanks in advance.
"could not get information" ... "permission denied". Your error message suggests your jenkins ServiceAccount does not have privileges getting resources within its own namespace.
And given the name of the ServiceAccount jenkins tries to get is some "node-app-something", we should probably assume that beyond getting objects, Jenkins would eventually need to create them as well.
Anyway, you would want to create a RoleBinding, in your jenkins namespace:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: jenkins-admin
namespace: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: admin
subjects:
- namespace: webapps
kind: ServiceAccount
name: jenkins
Using the admin ClusterRole is probably not the best pick. Still, if you are managing ServiceAccounts for your app, your pipeline may eventually create RoleBindings as well granting that ServiceAccount with privileges over your cluster. If you need to create RBAC configurations, admin would work for sure.
If you do not want Jenkins to manage privileges delegations within its own namespace, you may instead use the edit ClusterRole.
If you do not want Jenkins to create objects within its own namespace, you could go with the view ClusterRole.
Note that Jenkins may need edit privileges, assuming you did setup Jenkins Kubernetes cluster integration (and dynamic agents provisioning, running your Pipelines)
Going with admin: maybe you should consider deploying your applications into a separate namespace: don't grant Jenkins with admin privileges over its own namespace, rather grant those privileges over whichever namespace should host your jenkins-managed applications.
I could do that works!!
In values.yamls
serviceAccount: # Specifies whether a service account should be
created create: false
No create service account. Because it's trying to create a service account with the name of the chart.
In jenkinsfile change the command to:
sh "helm upgrade --install --namespace jenkins ${HELM_APP_NAME} --set
image.tag=${BUILD_NUMBER} ./${HELM_CHART_DIRECTORY}"
Thanks for your time.
Introduction :
I am trying to deploy a RabbitMq Helm Chart to GKE, with my Gitlab CI/CD pipeline. The command I use to install my chart is:
helm upgrade --install rabbitmq --create-namespace --namespace kubi-app-main -f envs/main/rabbitmq/rabbitmq.yaml bitnami/rabbitmq
envs/rabbitmq/rabbitmq.yaml:
auth:
username: user
password: password
# The used vhost is default-vhost
extraConfiguration: |-
default_vhost = default-vhost
default_permissions.configure = .*
default_permissions.read = .*
default_permissions.write = .*
The Gitlab job first connect to GKE cluster with gcloud:
echo "$SERVICE_ACCOUNT_KEY" > key.json
gcloud auth activate-service-account --key-file=key.json
gcloud config set project project-kubi-app
gcloud container clusters get-credentials cluster-1 --zone europe-west9-a --project project-kubi-app
The Issue:
But the helm upgrade fails:
Error: roles.rbac.authorization.k8s.io is forbidden: User "kubiapp-cluster-sa#project-kubi-app.iam.gserviceaccount.com" cannot create resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kubi-app-main": requires one of ["container.roles.create"] permission(s).
Checking the roles of the user (service account) on the project
gcloud projects get-iam-policy project-kubi-app --flatten="bindings[].members" --format='table(bindings.role)' --filter="bindings.members:kubiapp-cluster-sa#project-kubi-app.iam.gserviceaccount.com"
This will return ROLE roles/editor, meaning that my service account has an editor role on the project.
From what I understand, the service account kubiapp-cluster-sa#project-kubi-app.iam.gserviceaccount.com has the editor role on the project project-kubi-app.
BUT the service account that I am using can't create a role in the namespace kubi-app-main.
I don't understand the use of this role, but it's origin is from the RabbitMq Helm Chart.
From the RabbitMq Helm Chart:
...
# Source: rabbitmq/templates/rolebinding.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rabbitmq-endpoint-reader
namespace: "kubi-app-main"
labels:
app.kubernetes.io/name: rabbitmq
helm.sh/chart: rabbitmq-10.1.8
app.kubernetes.io/instance: rabbitmq
app.kubernetes.io/managed-by: Helm
subjects:
- kind: ServiceAccount
name: rabbitmq
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rabbitmq-endpoint-reader
...
---
EDIT:
I have changed my service account role to Owner and now it works, but I would like to know the role required to create other roles.
roles/editor allows you to create/update/delete resources for most/many services, but does not include the permission to perform any of those operations on roles in general. roles/owner, on the other hand, does as it essentially makes you an admin of (almost every) resource.
For GKE, the usual role required to create/modify/update roles within the cluster is roles/container.clusterAdmin. Check out GKE roles.
Is possible to gain k8s cluster access with serviceaccount token?
My script does not have access to a kubeconfig file, however, it does have access to the service account token at /var/run/secrets/kubernetes.io/serviceaccount/token.
Here are the steps I tried but it is not working.
kubectl config set-credentials sa-user --token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
kubectl config set-context sa-context --user=sa-user
but when the script ran "kubectl get rolebindings" I get the following error:
Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:test:default" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "test"
Is possible to gain k8s cluster access with serviceaccount token?
Certainly, that's the point of a ServiceAccount token. The question you appear to be asking is "why does my default ServiceAccount not have all the privileges I want", which is a different problem. One will benefit from reading the fine manual on the topic
If you want the default SA in the test NS to have privileges to read things in its NS, you must create a Role scoped to that NS and then declare the relationship explicitly. SAs do not automatically have those privileges
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: test
name: test-default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: whatever-role-you-want
subjects:
- kind: ServiceAccount
name: default
namespace: test
but when the script ran "kubectl get pods" I get the following error: Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:test:default" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "test"
Presumably you mean you can kubectl get rolebindings, because I would not expect running kubectl get pods to emit that error
Yes, it is possible. For instance, if you login K8S dashboard via token it does use the same way.
Follow these steps;
Create a service account
$ kubectl -n <your-namespace-optional> create serviceaccount <service-account-name>
A role binding grants the permissions defined in a role to a user or set of users. You can use a predefined role or you can create your own. Check this link for more info. https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-example
$ kubectl create clusterrolebinding <binding-name> --clusterrole=cluster-admin --serviceaccount=<namespace>:<service-account-name>
Get the token name
$ TOKENNAME=`kubectl -n <namespace> get serviceaccount/<service-account-name> -o jsonpath='{.secrets[0].name}'`
Finally, get the token and set the credentials
$ kubectl -n <namespace> get secret $TOKENNAME -o jsonpath='{.data.token}'| base64 --decode
$ kubectl config set-credentials <service-account-name> --token=<output from previous command>
$ kubectl config set-context --current --user=<service-account-name>
If you follow these steps carefully your problem will be solved.
I am following this helm + secure - guide:
https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html#helm
I deployed the cluster with this command: $ helm install my-release --values my-values.yaml cockroachdb/cockroachdb --namespace=thesis-crdb
This is how it looks: $ helm list --namespace=thesis-crdb
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
my-release thesis-crdb 1 2021-01-31 17:38:52.8102378 +0100 CET deployed cockroachdb-5.0.4 20.2.4
Here is how it looks using: $ kubectl get all --namespace=thesis-crdb
NAME READY STATUS RESTARTS AGE
pod/my-release-cockroachdb-0 1/1 Running 0 7m35s
pod/my-release-cockroachdb-1 1/1 Running 0 7m35s
pod/my-release-cockroachdb-2 1/1 Running 0 7m35s
pod/my-release-cockroachdb-init-fhzdn 0/1 Completed 0 7m35s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/my-release-cockroachdb ClusterIP None <none> 26257/TCP,8080/TCP 7m35s
service/my-release-cockroachdb-public ClusterIP 10.xx.xx.x <none> 26257/TCP,8080/TCP 7m35s
NAME READY AGE
statefulset.apps/my-release-cockroachdb 3/3 7m35s
NAME COMPLETIONS DURATION AGE
job.batch/my-release-cockroachdb-init 1/1 43s 7m36s
In the my-values.yaml-file I only changed the tls from false to true:
tls:
enabled: true
So far so good, but from here on the guide isn't really working for me anymore. I try as they say with getting the csr: kubectl get csr --namespace=thesis-crdb
No resources found
Ok, perhaps not needed. I carry on to deploy the client-secure
I download the file: https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/client-secure.yaml
And changes the serviceAccountName: cockroachdb to serviceAccountName: my-release-cockroachdb.
I try to deploy it with $ kubectl create -f client-secure.yaml --namespace=thesis-crdb but it throws this error:
Error from server (Forbidden): error when creating "client-secure.yaml": pods "cockroachdb-client-secure" is forbidden: error looking up service account thesis-crdb/my-release-cockroachdb: serviceaccount "my-release-cockroachdb" not found
Anyone got an idea how to solve this? I'm fairly sure it's something with the namespace that is messing it up.
I have tried to put the namespace in the metadata-section
metadata:
namespace: thesis-crdb
And then try to deploy it with: kubectl create -f client-secure.yaml but to no avail:
Error from server (Forbidden): error when creating "client-secure.yaml": pods "cockroachdb-client-secure" is forbidden: error looking up service account thesis-crdb/my-release-cockroachdb: serviceaccount "my-release-cockroachdb" not found
You mention in question that you have changed serviceAccountName in YAML.
And changes the serviceAccountName: cockroachdb to serviceAccountName: my-release-cockroachdb.
So Root Cause of your issue is related with ServiceAccount misconfiguration.
Background
In your cluster you have something called ServiceAccount.
When you (a human) access the cluster (for example, using kubectl), you are authenticated by the apiserver as a particular User Account (currently this is usually admin, unless your cluster administrator has customized your cluster). Processes in containers inside pods can also contact the apiserver. When they do, they are authenticated as a particular Service Account (for example, default).
To ServiceAccount you also should configure RBAC which grants you permissions to create resources.
Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization.
RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure policies through the Kubernetes API.
If you don't have proper RBAC permissions you will not be able to create resources.
In Kubernetes you can find Role and ClusterRole. Role sets permissions within a particular namespace and ClusterRole sets permissions in whole cluster.
Besides that, you also need to bind roles using RoleBinding and ClusterRoleBinding.
In addition, if you would use Cloud environment, you would also need special rights in project. Your guide provides instructions to do it here.
Root cause
I've checked cockroachdb chart and it creates ServiceAccount, Role, ClusterRole, RoleBinding and ClusterRoleBinding for cockroachdb and prometheus. There is no configuration for my-release-cockroachdb.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cockroachdb
...
verbs:
- create
- get
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cockroachdb
labels:
app: cockroachdb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cockroachdb
...
In client-secure.yaml you change serviceAccountName to my-release-cockroachdb and Kubernetes cannot find that ServiceAccount as it was not created by cluster administrator or cockroachdb chart.
To list ServiceAccounts in default namespace you can use command $ kubectl get ServiceAccount, however if you would check all ServiceAccounts in cluster you should add -A to your command - $ kubectl get ServiceAccount -A.
Solution
Option 1 is to use existing ServiceAccount with proper permissions like SA created by cockroachdb chart which is cockroachdb, not my-release-cockroachdb.
Option 2 is to create ServiceAccount, Role/ClusterRole and RoleBinding/ClusterRoleBinding for my-release-cockroachdb.
I deployed HA K8s Cluster with 3 masters & 2 worker Nodes. I access my K8s Dashboard through kubectl client(local), kubectl proxy. My K8s Dashboard is accessed through tokens by some RBAC users, where they have limited access on namespaces & Cluster admin users. I want to give anonymous access to all my users for viewing the deployment logs i.e., to Kibana Dashboard(Add-on). Can anyone help me regarding this?
Below, I specified the required artifacts that are running on my cluster with their versions:
K8s version: 1.8.0
kibana: 5.6.4
elasticsearch-logging : 5.6.4
You can try creating a ClusterRoleBinding for some specific users. In my case, I am using LDAP authentication for accessing the Kubernetes API. I have assigned admin privileges to some users and readonly access to some specific users. Refer to the ClusterRoleBinding yaml below:
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: oidc-readonly-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:aggregate-to-view
subjects:
- kind: User
name: https://dex.domain.com/dex#user1#domain.com
I am using dex tool for the LDAP authentication. You can try giving the RBAC username directly.