Issues after running kubeadm reset - kubernetes

I was having issues with kubeadm init, and so i ran kubeadm reset and then kubeadm init and the problem at hand went away, but now I have another problem and that is that when I run kubectl get all, I get the following response:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h6m
Error from server (Forbidden): replicationcontrollers is forbidden: User "system:node:abc-server.localdomain" cannot list resource "replicationcontrollers" in API group "" in the namespace "default"
Error from server (Forbidden): daemonsets.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "daemonsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): deployments.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "deployments" in API group "apps" in the namespace "default"
Error from server (Forbidden): replicasets.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "replicasets" in API group "apps" in the namespace "default"
Error from server (Forbidden): statefulsets.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "statefulsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:node:abc-server.localdomain" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default"
Error from server (Forbidden): jobs.batch is forbidden: User "system:node:abc-server.localdomain" cannot list resource "jobs" in API group "batch" in the namespace "default"
Error from server (Forbidden): cronjobs.batch is forbidden: User "system:node:abc-server.localdomain" cannot list resource "cronjobs" in API group "batch" in the namespace "default"
I've exhausted my googling abilities with my limited kubernetes vocabulary, so hoping someone here could help me with the following:
what's happening?! (is this a RBAC Authorization issue?)
how can i resolve this? as this is a dev environment that will definitely require some clean up, I don't mind a quick and dirty way just so i can continue with the task at hand (which is to just get things up and running again)

As #Software Engineer mentioned in his comment there is a github issue with a fix for that:
User neolit123 on github posted this solution:
getting a permission error during pod network setup, means you are trying to kubectl apply manifest files using a kubeconfig file which does not have the correct permissions.
make sure that your /etc/kubernetes/admin.conf is generated by kubeadm and contains kubernetes-admin as the user.
root#master:~# kubectl auth can-i create deploy
which kubeconfig is this command using?
try
root#master:~# KUBECONFIG=/etc/kubernetes/admin.conf kubectl auth can-i create deploy
I wanted to check the release notes, but there is no much information, or I don't know interpret it. Does anyone have any information about what are the changes, or what am I doing wrong?
AFAIK, there is no such change that breaks this between 1.14.4 and .3.

Answers to the best of my ability without reproducing the error:
Yes, this looks like a RBAC issue. You ran:
kubectl get all
As you might know, kubectl handles and locates the Kubernetes API server (also known as kube-apiserver in the cluster). Because the API server returns 403 (forbidden), we can proceed knowing that this is an authorization issue.
That is, we can request resources from the kube-apiserver (authentication) but we don't have the privileges to: User "system:node:abc-server.localdomain" does not have the permission to perform a HTTP GET request on such and so objects. Moreover, it's likely that this user doesn't have permission to perform HTTP requests (CRUD -- Create Read Update Delete) for any objects in the Kubernetes cluster.
In RBAC terms, there exists a Role that defines permissions and a RoleBinding that couples the user with that Role to get the permissions. As an example, a cluster has a Role that is equal to Unix/Linux root in terms of permissions, which is bound to the kube-apiserver. Obviously, we're not gonna use that role but generally, we can use those permissions to create roles that have CRUD permissions.
So when you bootstrapped the control plane, system:node:abc-server.localdomain should have been bound to a Role that gives permissions to perform a Read request on the objects in the default namespace. But I'm guessing that didn't happen.
I'm just gonna say and it would be most easy to restart the process by wiping the hosts and recreating the cluster. If you run into errors once more, please document all the steps you took so anyone can reproduce the problem and apply more thorough troubleshooting.

Related

Error from server (Forbidden): namespaces is forbidden: User "system:anonymous" cannot create resource "namespaces" in API group

I want to create a Kubernetes service account:
kubernetes#kubernetes1:~$ kubectl create namespace devops-tools
Please enter Username: kubernetes
Please enter Password: Error from server (Forbidden): namespaces is forbidden: User "system:anonymous" cannot create resource "namespaces" in API group "" at the cluster scope
kubernetes#kubernetes1:~$
But I get this error. Do you know how I can fix this issue?
Kubernetes indicates that it has identified you as system:anonymous (unkown)
Please see this part of the Kubernetes documentation that discusses the different authentication methods https://kubernetes.io/docs/reference/access-authn-authz/authentication/
For username and password authentication to work, you should look at the Static Password File section.

kubectl get secrets: Error from server (Forbidden)

I have installed the Upbound CLI locally and since then I (I think) am receiving Error from server (Forbidden) error message when I try to execute different commands using kubectl:
kubectl get secrets
Error from server (Forbidden): secrets is forbidden: User "upbound-cloud-impersonator" cannot list resource "secrets" in API group "" in the namespace "default"
kubectl get all
Error from server (Forbidden): replicationcontrollers is forbidden: User "upbound-cloud-impersonator" cannot list resource "replicationcontrollers" in API group "" in the namespace "default"
Error from server (Forbidden): services is forbidden: User "upbound-cloud-impersonator" cannot list resource "services" in API group "" in the namespace "default"
Error from server (Forbidden): daemonsets.apps is forbidden: User "upbound-cloud-impersonator" cannot list resource "daemonsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): deployments.apps is forbidden: User "upbound-cloud-impersonator" cannot list resource "deployments" in API group "apps" in the namespace "default"
Error from server (Forbidden): replicasets.apps is forbidden: User "upbound-cloud-impersonator" cannot list resource "replicasets" in API group "apps" in the namespace "default"
Error from server (Forbidden): statefulsets.apps is forbidden: User "upbound-cloud-impersonator" cannot list resource "statefulsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "upbound-cloud-impersonator" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default"
Error from server (Forbidden): cronjobs.batch is forbidden: User "upbound-cloud-impersonator" cannot list resource "cronjobs" in API group "batch" in the namespace "default"
Error from server (Forbidden): jobs.batch is forbidden: User "upbound-cloud-impersonator" cannot list resource "jobs" in API group "batch" in the namespace "default"
It seems that the user has been changed to "upbound-cloud-impersonator", but I am not sure why and how to switch it back to what it was before.
If it helps, these are the namespaces:
crossplane-system Active 2d21h
default Active 2d21h
kube-node-lease Active 2d21h
kube-public Active 2d21h
kube-system Active 2d21h
upbound-system Active 2d21h
velero Active 2d21h
And the list of users kubectl config view -o jsonpath='{.users[*].name}':
minikube upbound-3f93ea79-ba0e-4fdc-ae69-f2c562279579
The user upbound-cloud-impersonator is not having enough RBAC permissions to get the secrets. The alternative solution is to set setup enough RBAC permission.
You can validate this using the below command:
kubectl auth can-i get secret --as upbound-cloud-impersonator
Or just run(as your context is already changed):
kubectl auth can-i get secret
Your current context is switched to something else. The new context is using upbound-cloud-impersonator. you may run the following command to list your current context
kubectl config current-context
To switch to another context, you may first run the following command to list your context list. Notice the * sign that signifies the current context. Note that the context name used here are examples and may differ for your cluster.
kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
default default default
* upbound-cloud-impersonator#default default upbound-cloud-impersonator
To switch to the other context, will change the current context to the provided one. Use appropriate context to switch.
kubectl config use-context default
Notice the * sign:
kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* default default default
upbound-cloud-impersonator#default default upbound-cloud-impersonator
Validate the authorization:
kubectl auth can-i get secret
yes

Since Fluentd Daemonset has access to the pod log locations why it needs to contact API server?

I have deployed Fluentd as DaemonSet on my cluster using fluentd-daemonset-elasticsearch.
In the logs I can see this error
start_pod_watch: Exception encountered setting up pod watch from Kubernetes API v1 endpoint https://<ip>:443/api: pods is forbidden: User "system:serviceaccount:development:default" cannot list resource "pods" in API group "" at the cluster scope ({"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \\"system:serviceaccount:development:default\\" cannot list resource \\"pods\\" in API group \\"\\" at the cluster scope","reason":"Forbidden","details":{"kind":"pods"},"code":403} (Fluent::ConfigError)
If you check the volumeMounts you can see that Fluentd has access to all the logs. So, my question is if it has access why it tries to contact APIServer? and how to resolve this issue?

Problem deploying K8s with gitlab runner get an error

I changed something and deployed on a new cluster then I got this error even though I didn't change anything in the code. Has anybody seen it before?
from server for:
"/builds/dropcunt/nettmoster.com/deployment/webapp.yml": ingresses.extensions "nettmoster.comn-273414" is forbidden: User "system:serviceaccount:gitlab-managed-apps:default" cannot get resource "ingresses" in API group "extensions" in the namespace "nettmoster-com-9777808"
59 Error from server (Forbidden): error when retrieving current configuration of:
As suggested I runkubectl auth can-i --list --as=system:serviceaccount:gitlab-managed-apps:default
It returns:
This is a RBAC problem. The service account system:serviceaccount:gitlab-managed-apps:default does not have permission to get ingress resource in the new cluster.
You can compare what permission a service account have by running below command in both the cluster
kubectl auth can-i --list --as=system:serviceaccount:gitlab-managed-apps:default
Run below commands to get permission via RBAC
kubectl create role ingress-reader --verb=get,list,watch,update --resource=ingress
kubectl create rolebinding ingress-reader-role --role=ingress-reader --serviceaccount=gitlab-managed-apps:default

Traefik installation error on GCS: Unknown user "system:serviceaccount:default:default"

I've been racking my head on how to set up traefik with GKE. I'm currently getting what I think is an RBAC related error...
github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86:
Failed to list *v1beta1.Ingress:
ingresses.extensions is forbidden: User system:serviceaccount:default:default"
cannot list ingresses.extensions at the cluster scope:
Unknown user "system:serviceaccount:default:default"`
Any idea on how to get around this error?
That is an error the GKE IAM authorizer returns when a user is not authorized to do something. In GKE, the RBAC authorizer is then consulted. The fact that you are seeing that error message means both the GKE and RBAC authorizers forbid that user. Make sure you are deploying traefik using the expected service account and have set up an RBAC policy that gives that service account required permissions