I'm trying to install Helm as described here.
But when I'm doing kubectl create serviceaccount -n kube-system tiller
I get a message saying Error from server (AlreadyExists): serviceaccounts "tiller" already exists. But I can't see it when I visit https://console.cloud.google.com/iam-admin/iam nor https://console.cloud.google.com/iam-admin/serviceaccounts. How can that be? I had just made sure I was working the correct cluster:
cloud container clusters get-credentials my-cluster
Fetching cluster endpoint and auth data.
kubeconfig entry generated for my-cluster.
You are confusing Google service accounts and Kubernetes service accounts.
kubectl get serviceaccount -n kube-system tiller -o yaml
https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/
Related
I'm trying to understand why on one cluster an operation is permitted but on the other i'm getting the following
Exception encountered setting up namespace watch from Kubernetes API v1 endpoint https://10.100.0.1:443/api: namespaces is forbidden: User \"system:serviceaccount:kube-system:default\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope ({\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"namespaces is forbidden: User \\\"system:serviceaccount:kube-system:default\\\" cannot list resource \\\"namespaces\\\" in API group \\\"\\\" at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"namespaces\"},\"code\":403}\n)"
I'm managing two Kubernetes clusters -
clusterA booted with Kops version v1.14.8
clusterB booted on AWS EKS version v1.14.9-eks-f459c0
So i've tried using the kubectl auth command to try figuring out and I do see that on one i'm allowed however on the second i'm not as you can see:
kubectl config use-context clusterA
Switched to context "clusterA".
kubectl auth can-i list pods --as=system:serviceaccount:kube-system:default -n kube-system
yes
kubectl config use-context clusterB
Switched to context "clusterB".
kubectl auth can-i list pods --as=system:serviceaccount:kube-system:default -n kube-system
no
Is there a way to understand what are these two decisions based on yes/no?
Thanks for helping out!
The decision yes/no is based on whether there is a clusterrole and a clusterrolebinding or rolebinding which permits the default serviceaccount in kube-system namespace to perform verb list on resource namespace.
The trick in case of namespace resource is that there needs to be a clusterrole instead of role because namespace is a cluster scoped resource.
You check what are the clusterrole,role, clusterrolebinding,rolebinding exists in a kubernetes cluster using below command
kubectl get clusterrole,clusterrolebinding
kubectl get role,rolebinding -n namespacename
For more details refer Kubernetes RBAC here
I have deployed jupyterhub on my GKE cluster using helm. However, when I run helm list --all (or helm list --failed etc) I see no output.
I can confirm that tiller is running in my cluster:
$ helm version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
And I can see the tiller pod:
$ kubectl get pods -n kube-system | grep tiller
tiller-deploy-778f674bf5-5jksm 1/1 Running 0 132d
I can also see that my deployment of jupyterhub is running using kubectl get pods -n jhub.
How I can I determine why the output of helm list is empty?
I had same issue where helm list was showing empty output.
In case if anyone lands on this page looking for solution, please check below.
source (similar) : https://github.com/helm/helm/issues/7146
In short : Issue here we need to use namespaces when we do listing. or --all-namespace would help as well.
helm list --all-namespaces
I have a strong feeling you are missing some permissions. This is a GKE cluster. So RBAC is enabled.
The standard practice is to first create a dedicated Service account in the appropriate namespace. For example sake, lets say kube-system
kubectl create serviceaccount tiller --namespace kube-system
Then you need to give appropriate permissions to this service account.
FOR TESTING / NON-SECURE !!!
Lets allow this service account to run with super user privileges i.e. run as cluster-admin
kubectl create clusterrolebinding tiller-admin --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
FOR PRODUCTION / SECURE
Create a Role that gives the minimum privileges for tiller to run and associate with the tiller service account using a RoleBinding.
Then go ahead and initialize tiller with the associated serviceAccount.
helm init --service-account tiller
I am trying to access my Kubernetes cluster on google cloud with the service account, but I am not able to make this works. I have a running system with some pods and ingress. I want to be able to update images of deployments.
I would like to use something like this (remotely):
kubectl config set-cluster cluster --server="<IP>" --insecure-skip-tls-verify=true
kubectl config set-credentials foo --token="<TOKEN>"
kubectl config set-context my-context --cluster=cluster --user=foo --namespace=default
kubectl config use-context cluster
kubectl set image deployment/my-deployment boo=eu.gcr.io/project-123456/image:v1
So I created the service account and then get the secret token:
kubectl create serviceaccount foo
kubectl get secret foo-token-gqvgn -o yaml
But, when I try to update the image in any deployment, I receive:
error: You must be logged in to the server (Unauthorized)
IP address for API I use the address, which is shown in GKE administration as cluster endpoint IP.
Any suggestions? Thanks.
I have tried to recreate your problem.
Steps I have followed
kubectl create serviceaccount foo
kubectl get secret foo-token-* -o yaml
Then, I have tried to do what you have done
What I have used as token is base64 decoded Token.
Then I tried this:
$ kubectl get pods
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:foo" cannot list pods in the namespace "default": Unknown user "system:serviceaccount:default:foo"
This gave me error as expected. Because, I need to grant permission to this ServiceAccount.
How can I grant permission to this ServiceAccount? I need to create ClusterRole & ClusterRoleBinding with necessary permission.
Read more to learn more role-based-access-control
I can do another thing
$ kubectl config set-credentials foo --username="admin" --password="$PASSWORD"
This will grant you admin authorization.
You need to provide cluster credential.
Username: admin
Password: -----
You will get this info in GKE -> Kubernetes Engine -> {cluster} -> Show credential
I'm running Kubernetes 1.6.2 with RBAC enabled. I've created a user kube-admin that has the following Cluster Role binding
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: k8s-admin
subjects:
- kind: User
name: kube-admin
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
When I attempt to kubectl exec into a running pod I get the following error.
kubectl -n kube-system exec -it kubernetes-dashboard-2396447444-1t9jk -- /bin/bash
error: unable to upgrade connection: Forbidden (user=system:anonymous, verb=create, resource=nodes, subresource=proxy)
My guess is I'm missing a ClusterRoleBinding ref, which role am I missing?
The connection between kubectl and the api is fine, and is being authorized correctly.
To satisfy an exec request, the apiserver contacts the kubelet running the pod, and that connection is what is being forbidden.
Your kubelet is configured to authenticate/authorize requests, and the apiserver is not providing authentication information recognized by the kubelet.
The way the apiserver authenticates to the kubelet is with a client certificate and key, configured with the --kubelet-client-certificate=... --kubelet-client-key=... flags provided to the API server.
See https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#overview for more information.
I had this exact same error happening, but for me the problem was due to my setup with kops. I wanted to share my result here because it may help someone in the future.
With a bug that exists in kops v1.19.1, the config for kubelet anonymousAuth must be explicitly set to false
I was using kops version 1.19, and upgrading my cluster from k8s v1.11 to v1.19. After the upgrade is when I started seeing this error when trying to run kubectl port-forward, kubectl logs, kubectl exec, helm list, etc. The issue was a combination of a bug in kops and not having the anonymousAuth configuration set, or having it set to true. With this bug in kops, the config for kubelet anonymousAuth must be set explicitly to false.
To Fix
Edit the cluster
$ kops edit cluster
Add the config under spec.kubelet.anonymousAuth, i.e.
spec:
kubelet:
anonymousAuth: false
Update the cluster
$ kops update cluster --yes
$ kops rolling-update cluster --yes
Related
PR that fixes the immediate issue
PR that fixes a related issue
Related kops docs
I'm trying to deploy an application in a GKE 1.6.2 cluster running ContainerOS but the instructions on the website / k8s are not accurate anymore.
The error that I'm getting is:
Error from server (Forbidden): User "circleci#gophers-slack-bot.iam.gserviceaccount.com"
cannot get deployments.extensions in the namespace "gopher-slack-bot".:
"No policy matched.\nRequired \"container.deployments.get\" permission."
(get deployments.extensions gopher-slack-bot)
The repository for the application is available here available here.
Thank you.
I had a few breaking changes in the past with using the gcloud tool to authenticate kubectl to a cluster, so I ended up figuring out how to auth kubectl to a specific namespace independent of GKE. Here's what works for me:
On CircleCI:
setup_kubectl() {
echo "$KUBE_CA_PEM" | base64 --decode > kube_ca.pem
kubectl config set-cluster default-cluster --server=$KUBE_URL --certificate-authority="$(pwd)/kube_ca.pem"
kubectl config set-credentials default-admin --token=$KUBE_TOKEN
kubectl config set-context default-system --cluster=default-cluster --user=default-admin --namespace default
kubectl config use-context default-system
}
And here's how I get each of those env vars from kubectl.
kubectl get serviceaccounts $namespace -o json
The service account will contain the name of it's secret. In my case, with the default namespace, it's
"secrets": [
{
"name": "default-token-655ls"
}
]
Using the name, I get the contents of the secret
kubectl get secrets $secret_name -o json
The secret will contain ca.crt and token fields, which match the $KUBE_CA_PEM and $KUBE_TOKEN in the shell script above.
Finally, use kubectl cluster-info to get the $KUBE_URL value.
Once you run setup_kubectl on CI, your kubectl utility will be authenticated to the namespace you're deploying to.
In Kubernetes 1.6 and GKE, we introduce role based cess control. The authors of your took need to give the service account the ability to get deployments (along with probably quite a few others) to its account creation.
https://kubernetes.io/docs/admin/authorization/rbac/