I changed something and deployed on a new cluster then I got this error even though I didn't change anything in the code. Has anybody seen it before?
from server for:
"/builds/dropcunt/nettmoster.com/deployment/webapp.yml": ingresses.extensions "nettmoster.comn-273414" is forbidden: User "system:serviceaccount:gitlab-managed-apps:default" cannot get resource "ingresses" in API group "extensions" in the namespace "nettmoster-com-9777808"
59 Error from server (Forbidden): error when retrieving current configuration of:
As suggested I runkubectl auth can-i --list --as=system:serviceaccount:gitlab-managed-apps:default
It returns:
This is a RBAC problem. The service account system:serviceaccount:gitlab-managed-apps:default does not have permission to get ingress resource in the new cluster.
You can compare what permission a service account have by running below command in both the cluster
kubectl auth can-i --list --as=system:serviceaccount:gitlab-managed-apps:default
Run below commands to get permission via RBAC
kubectl create role ingress-reader --verb=get,list,watch,update --resource=ingress
kubectl create rolebinding ingress-reader-role --role=ingress-reader --serviceaccount=gitlab-managed-apps:default
Related
I was having issues with kubeadm init, and so i ran kubeadm reset and then kubeadm init and the problem at hand went away, but now I have another problem and that is that when I run kubectl get all, I get the following response:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h6m
Error from server (Forbidden): replicationcontrollers is forbidden: User "system:node:abc-server.localdomain" cannot list resource "replicationcontrollers" in API group "" in the namespace "default"
Error from server (Forbidden): daemonsets.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "daemonsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): deployments.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "deployments" in API group "apps" in the namespace "default"
Error from server (Forbidden): replicasets.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "replicasets" in API group "apps" in the namespace "default"
Error from server (Forbidden): statefulsets.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "statefulsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:node:abc-server.localdomain" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default"
Error from server (Forbidden): jobs.batch is forbidden: User "system:node:abc-server.localdomain" cannot list resource "jobs" in API group "batch" in the namespace "default"
Error from server (Forbidden): cronjobs.batch is forbidden: User "system:node:abc-server.localdomain" cannot list resource "cronjobs" in API group "batch" in the namespace "default"
I've exhausted my googling abilities with my limited kubernetes vocabulary, so hoping someone here could help me with the following:
what's happening?! (is this a RBAC Authorization issue?)
how can i resolve this? as this is a dev environment that will definitely require some clean up, I don't mind a quick and dirty way just so i can continue with the task at hand (which is to just get things up and running again)
As #Software Engineer mentioned in his comment there is a github issue with a fix for that:
User neolit123 on github posted this solution:
getting a permission error during pod network setup, means you are trying to kubectl apply manifest files using a kubeconfig file which does not have the correct permissions.
make sure that your /etc/kubernetes/admin.conf is generated by kubeadm and contains kubernetes-admin as the user.
root#master:~# kubectl auth can-i create deploy
which kubeconfig is this command using?
try
root#master:~# KUBECONFIG=/etc/kubernetes/admin.conf kubectl auth can-i create deploy
I wanted to check the release notes, but there is no much information, or I don't know interpret it. Does anyone have any information about what are the changes, or what am I doing wrong?
AFAIK, there is no such change that breaks this between 1.14.4 and .3.
Answers to the best of my ability without reproducing the error:
Yes, this looks like a RBAC issue. You ran:
kubectl get all
As you might know, kubectl handles and locates the Kubernetes API server (also known as kube-apiserver in the cluster). Because the API server returns 403 (forbidden), we can proceed knowing that this is an authorization issue.
That is, we can request resources from the kube-apiserver (authentication) but we don't have the privileges to: User "system:node:abc-server.localdomain" does not have the permission to perform a HTTP GET request on such and so objects. Moreover, it's likely that this user doesn't have permission to perform HTTP requests (CRUD -- Create Read Update Delete) for any objects in the Kubernetes cluster.
In RBAC terms, there exists a Role that defines permissions and a RoleBinding that couples the user with that Role to get the permissions. As an example, a cluster has a Role that is equal to Unix/Linux root in terms of permissions, which is bound to the kube-apiserver. Obviously, we're not gonna use that role but generally, we can use those permissions to create roles that have CRUD permissions.
So when you bootstrapped the control plane, system:node:abc-server.localdomain should have been bound to a Role that gives permissions to perform a Read request on the objects in the default namespace. But I'm guessing that didn't happen.
I'm just gonna say and it would be most easy to restart the process by wiping the hosts and recreating the cluster. If you run into errors once more, please document all the steps you took so anyone can reproduce the problem and apply more thorough troubleshooting.
I am getting an error message after running some kubectl commands (GCP command line - gcloud). I have a K8S cluster created in GKE.
Example:
kubectl describe node
gke_k8s_cluster_name
Error from server (Forbidden): leases.coordination.k8s.io "gke_k8s_cluster_name" is forbidden: User "MY_SERVICE_ACCOUNT" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": Required "container.leases.get" permission.
The point is that "container.leases.get" permission is not listed in IAM (as custom permissions or regular role).
How could I grant that permission to the service account in GCP ?
thanks,
Jose
You may need to grant additional permissions to yourself on GCP IAM and GKE sides, for example:
PROJECT_ID=$(gcloud config get-value core/project)
USER_ID=$(gcloud config get-value core/account)
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=user:${USER_ID} --role=roles/container.admin
kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user ${USER_ID}
See also GCP IAM & GKE RBAC integration.
I'm trying to follow this istio tutorial, while I use google cloud.
After I ran the istio.yaml, I got the following error:
Error from server (Forbidden): error when creating "istio.yaml": clusterroles.rbac.authorization.k8s.io "istio-sidecar-injector-default" is forbidden: attempt to grant extra privileges
After looking online, I saw a solution on github community:
For the benefit of others who might run into this issue, I worked around the problem by running:
$ gcloud config set container/use_client_certificate True
$ export CLOUDSDK_CONTAINER_USE_CLIENT_CERTIFICATE=True
Before running $ gcloud container clusters get-credentials ...
After I tried this solution, I got forbidden error for almost every command, i.e.:
$ kubectl get pods
Error from server (Forbidden): pods is forbidden: User "client" cannot list pods in the namespace "default"
$ kubectl get namespaces
Error from server (Forbidden): namespaces is forbidden: User "client" cannot list namespaces at the cluster scope
I tried use the solution on this stack overflow question, so I ran:
$gcloud config unset container/use_client_certificate
$gcloud container clusters get-credentials my-cluster
But I still got the forbidden error. Any idea?
I'm trying to enable RBAC on my cluster and iadded those following line to the kube-apiserver.yml :
- --authorization-mode=RBAC
- --runtime-config=rbac.authorization.k8s.io/v1beta1
- --authorization-rbac-super-user=admin
and i did systemctl restart kubelet ;
the apiserver starts successfully but i'm not able to run kubectl command and i got this error :
kubectl get po
Error from server (Forbidden): pods is forbidden: User "kubectl" cannot list pods in the namespace "default"
where am I going wrong or i should create some roles to the kubectl user ? if so how that possible
Error from server (Forbidden): pods is forbidden: User "kubectl" cannot list pods in the namespace "default"
You are using user kubectl to access cluster by kubectl utility, but you set --authorization-rbac-super-user=admin, which means your super-user is admin.
To fix the issue, launch kube-apiserver with superuser "kubectl" instead of "admin."
Just update the value of the option: --authorization-rbac-super-user=kubectl.
Old question but for google searchers, you can use the insecure port:
If your API server runs with the insecure port enabled (--insecure-port), you can also make API calls via that port, which does not enforce authentication or authorization.
Source: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#privilege-escalation-prevention-and-bootstrapping
So add --insecure-port=8080 to your kube-apiserver options and then restart it.
Then run:
kubectl create clusterrolebinding kubectl-cluster-admin-binding --clusterrole=cluster-admin --user=kubectl
Then turn the insecure-port off.
I am trying to access my Kubernetes cluster on google cloud with the service account, but I am not able to make this works. I have a running system with some pods and ingress. I want to be able to update images of deployments.
I would like to use something like this (remotely):
kubectl config set-cluster cluster --server="<IP>" --insecure-skip-tls-verify=true
kubectl config set-credentials foo --token="<TOKEN>"
kubectl config set-context my-context --cluster=cluster --user=foo --namespace=default
kubectl config use-context cluster
kubectl set image deployment/my-deployment boo=eu.gcr.io/project-123456/image:v1
So I created the service account and then get the secret token:
kubectl create serviceaccount foo
kubectl get secret foo-token-gqvgn -o yaml
But, when I try to update the image in any deployment, I receive:
error: You must be logged in to the server (Unauthorized)
IP address for API I use the address, which is shown in GKE administration as cluster endpoint IP.
Any suggestions? Thanks.
I have tried to recreate your problem.
Steps I have followed
kubectl create serviceaccount foo
kubectl get secret foo-token-* -o yaml
Then, I have tried to do what you have done
What I have used as token is base64 decoded Token.
Then I tried this:
$ kubectl get pods
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:foo" cannot list pods in the namespace "default": Unknown user "system:serviceaccount:default:foo"
This gave me error as expected. Because, I need to grant permission to this ServiceAccount.
How can I grant permission to this ServiceAccount? I need to create ClusterRole & ClusterRoleBinding with necessary permission.
Read more to learn more role-based-access-control
I can do another thing
$ kubectl config set-credentials foo --username="admin" --password="$PASSWORD"
This will grant you admin authorization.
You need to provide cluster credential.
Username: admin
Password: -----
You will get this info in GKE -> Kubernetes Engine -> {cluster} -> Show credential