After kubernetes upgrades we faced with problem that we can't do anything with kubernetes.
At first we were getting You must be logged in to the server (Unauthorized) , but I found solution
kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > /etc/kubernetes/kubelet.conf
cp /etc/kubernetes/kubelet.conf ~/.kube/config
cp /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf
systemctl restart kubelet
so now we can list nodes, all pods, but can't create namespace, clusterroles, check apiserver and so on.
[root#foo.example ~]# kubectl get roles,rolebindings,clusterroles
Error from server (Forbidden): roles.rbac.authorization.k8s.io is forbidden: User "system:node:foo.example" cannot list resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "default"
Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:node:foo.example" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default"
Error from server (Forbidden): clusterroles.rbac.authorization.k8s.io is forbidden: User "system:node:foo.example" cannot list resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope
Tried to renew certificate, but still getting same errors. I can't reinstall kubernetes cluster since it is in production now. We are running on premises kubernetes
Kubernetes version: v1.17.9
Related
I have installed the Upbound CLI locally and since then I (I think) am receiving Error from server (Forbidden) error message when I try to execute different commands using kubectl:
kubectl get secrets
Error from server (Forbidden): secrets is forbidden: User "upbound-cloud-impersonator" cannot list resource "secrets" in API group "" in the namespace "default"
kubectl get all
Error from server (Forbidden): replicationcontrollers is forbidden: User "upbound-cloud-impersonator" cannot list resource "replicationcontrollers" in API group "" in the namespace "default"
Error from server (Forbidden): services is forbidden: User "upbound-cloud-impersonator" cannot list resource "services" in API group "" in the namespace "default"
Error from server (Forbidden): daemonsets.apps is forbidden: User "upbound-cloud-impersonator" cannot list resource "daemonsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): deployments.apps is forbidden: User "upbound-cloud-impersonator" cannot list resource "deployments" in API group "apps" in the namespace "default"
Error from server (Forbidden): replicasets.apps is forbidden: User "upbound-cloud-impersonator" cannot list resource "replicasets" in API group "apps" in the namespace "default"
Error from server (Forbidden): statefulsets.apps is forbidden: User "upbound-cloud-impersonator" cannot list resource "statefulsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "upbound-cloud-impersonator" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default"
Error from server (Forbidden): cronjobs.batch is forbidden: User "upbound-cloud-impersonator" cannot list resource "cronjobs" in API group "batch" in the namespace "default"
Error from server (Forbidden): jobs.batch is forbidden: User "upbound-cloud-impersonator" cannot list resource "jobs" in API group "batch" in the namespace "default"
It seems that the user has been changed to "upbound-cloud-impersonator", but I am not sure why and how to switch it back to what it was before.
If it helps, these are the namespaces:
crossplane-system Active 2d21h
default Active 2d21h
kube-node-lease Active 2d21h
kube-public Active 2d21h
kube-system Active 2d21h
upbound-system Active 2d21h
velero Active 2d21h
And the list of users kubectl config view -o jsonpath='{.users[*].name}':
minikube upbound-3f93ea79-ba0e-4fdc-ae69-f2c562279579
The user upbound-cloud-impersonator is not having enough RBAC permissions to get the secrets. The alternative solution is to set setup enough RBAC permission.
You can validate this using the below command:
kubectl auth can-i get secret --as upbound-cloud-impersonator
Or just run(as your context is already changed):
kubectl auth can-i get secret
Your current context is switched to something else. The new context is using upbound-cloud-impersonator. you may run the following command to list your current context
kubectl config current-context
To switch to another context, you may first run the following command to list your context list. Notice the * sign that signifies the current context. Note that the context name used here are examples and may differ for your cluster.
kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
default default default
* upbound-cloud-impersonator#default default upbound-cloud-impersonator
To switch to the other context, will change the current context to the provided one. Use appropriate context to switch.
kubectl config use-context default
Notice the * sign:
kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* default default default
upbound-cloud-impersonator#default default upbound-cloud-impersonator
Validate the authorization:
kubectl auth can-i get secret
yes
I was having issues with kubeadm init, and so i ran kubeadm reset and then kubeadm init and the problem at hand went away, but now I have another problem and that is that when I run kubectl get all, I get the following response:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h6m
Error from server (Forbidden): replicationcontrollers is forbidden: User "system:node:abc-server.localdomain" cannot list resource "replicationcontrollers" in API group "" in the namespace "default"
Error from server (Forbidden): daemonsets.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "daemonsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): deployments.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "deployments" in API group "apps" in the namespace "default"
Error from server (Forbidden): replicasets.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "replicasets" in API group "apps" in the namespace "default"
Error from server (Forbidden): statefulsets.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "statefulsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:node:abc-server.localdomain" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default"
Error from server (Forbidden): jobs.batch is forbidden: User "system:node:abc-server.localdomain" cannot list resource "jobs" in API group "batch" in the namespace "default"
Error from server (Forbidden): cronjobs.batch is forbidden: User "system:node:abc-server.localdomain" cannot list resource "cronjobs" in API group "batch" in the namespace "default"
I've exhausted my googling abilities with my limited kubernetes vocabulary, so hoping someone here could help me with the following:
what's happening?! (is this a RBAC Authorization issue?)
how can i resolve this? as this is a dev environment that will definitely require some clean up, I don't mind a quick and dirty way just so i can continue with the task at hand (which is to just get things up and running again)
As #Software Engineer mentioned in his comment there is a github issue with a fix for that:
User neolit123 on github posted this solution:
getting a permission error during pod network setup, means you are trying to kubectl apply manifest files using a kubeconfig file which does not have the correct permissions.
make sure that your /etc/kubernetes/admin.conf is generated by kubeadm and contains kubernetes-admin as the user.
root#master:~# kubectl auth can-i create deploy
which kubeconfig is this command using?
try
root#master:~# KUBECONFIG=/etc/kubernetes/admin.conf kubectl auth can-i create deploy
I wanted to check the release notes, but there is no much information, or I don't know interpret it. Does anyone have any information about what are the changes, or what am I doing wrong?
AFAIK, there is no such change that breaks this between 1.14.4 and .3.
Answers to the best of my ability without reproducing the error:
Yes, this looks like a RBAC issue. You ran:
kubectl get all
As you might know, kubectl handles and locates the Kubernetes API server (also known as kube-apiserver in the cluster). Because the API server returns 403 (forbidden), we can proceed knowing that this is an authorization issue.
That is, we can request resources from the kube-apiserver (authentication) but we don't have the privileges to: User "system:node:abc-server.localdomain" does not have the permission to perform a HTTP GET request on such and so objects. Moreover, it's likely that this user doesn't have permission to perform HTTP requests (CRUD -- Create Read Update Delete) for any objects in the Kubernetes cluster.
In RBAC terms, there exists a Role that defines permissions and a RoleBinding that couples the user with that Role to get the permissions. As an example, a cluster has a Role that is equal to Unix/Linux root in terms of permissions, which is bound to the kube-apiserver. Obviously, we're not gonna use that role but generally, we can use those permissions to create roles that have CRUD permissions.
So when you bootstrapped the control plane, system:node:abc-server.localdomain should have been bound to a Role that gives permissions to perform a Read request on the objects in the default namespace. But I'm guessing that didn't happen.
I'm just gonna say and it would be most easy to restart the process by wiping the hosts and recreating the cluster. If you run into errors once more, please document all the steps you took so anyone can reproduce the problem and apply more thorough troubleshooting.
I changed something and deployed on a new cluster then I got this error even though I didn't change anything in the code. Has anybody seen it before?
from server for:
"/builds/dropcunt/nettmoster.com/deployment/webapp.yml": ingresses.extensions "nettmoster.comn-273414" is forbidden: User "system:serviceaccount:gitlab-managed-apps:default" cannot get resource "ingresses" in API group "extensions" in the namespace "nettmoster-com-9777808"
59 Error from server (Forbidden): error when retrieving current configuration of:
As suggested I runkubectl auth can-i --list --as=system:serviceaccount:gitlab-managed-apps:default
It returns:
This is a RBAC problem. The service account system:serviceaccount:gitlab-managed-apps:default does not have permission to get ingress resource in the new cluster.
You can compare what permission a service account have by running below command in both the cluster
kubectl auth can-i --list --as=system:serviceaccount:gitlab-managed-apps:default
Run below commands to get permission via RBAC
kubectl create role ingress-reader --verb=get,list,watch,update --resource=ingress
kubectl create rolebinding ingress-reader-role --role=ingress-reader --serviceaccount=gitlab-managed-apps:default
I'm trying to follow this istio tutorial, while I use google cloud.
After I ran the istio.yaml, I got the following error:
Error from server (Forbidden): error when creating "istio.yaml": clusterroles.rbac.authorization.k8s.io "istio-sidecar-injector-default" is forbidden: attempt to grant extra privileges
After looking online, I saw a solution on github community:
For the benefit of others who might run into this issue, I worked around the problem by running:
$ gcloud config set container/use_client_certificate True
$ export CLOUDSDK_CONTAINER_USE_CLIENT_CERTIFICATE=True
Before running $ gcloud container clusters get-credentials ...
After I tried this solution, I got forbidden error for almost every command, i.e.:
$ kubectl get pods
Error from server (Forbidden): pods is forbidden: User "client" cannot list pods in the namespace "default"
$ kubectl get namespaces
Error from server (Forbidden): namespaces is forbidden: User "client" cannot list namespaces at the cluster scope
I tried use the solution on this stack overflow question, so I ran:
$gcloud config unset container/use_client_certificate
$gcloud container clusters get-credentials my-cluster
But I still got the forbidden error. Any idea?
I'm trying to enable RBAC on my cluster and iadded those following line to the kube-apiserver.yml :
- --authorization-mode=RBAC
- --runtime-config=rbac.authorization.k8s.io/v1beta1
- --authorization-rbac-super-user=admin
and i did systemctl restart kubelet ;
the apiserver starts successfully but i'm not able to run kubectl command and i got this error :
kubectl get po
Error from server (Forbidden): pods is forbidden: User "kubectl" cannot list pods in the namespace "default"
where am I going wrong or i should create some roles to the kubectl user ? if so how that possible
Error from server (Forbidden): pods is forbidden: User "kubectl" cannot list pods in the namespace "default"
You are using user kubectl to access cluster by kubectl utility, but you set --authorization-rbac-super-user=admin, which means your super-user is admin.
To fix the issue, launch kube-apiserver with superuser "kubectl" instead of "admin."
Just update the value of the option: --authorization-rbac-super-user=kubectl.
Old question but for google searchers, you can use the insecure port:
If your API server runs with the insecure port enabled (--insecure-port), you can also make API calls via that port, which does not enforce authentication or authorization.
Source: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#privilege-escalation-prevention-and-bootstrapping
So add --insecure-port=8080 to your kube-apiserver options and then restart it.
Then run:
kubectl create clusterrolebinding kubectl-cluster-admin-binding --clusterrole=cluster-admin --user=kubectl
Then turn the insecure-port off.