Kubernetes-dashboard pod is crashing again and again - kubernetes

I have installed and configured Kubernetes on my ubuntu machine, followed this Document
After deploying the Kubernetes-dashboard, container keep crashing
kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
Started the Proxy using:
kubectl proxy --address='0.0.0.0' --accept-hosts='.*' --port=8001
Pod status:
kubectl get pods -o wide --all-namespaces
....
....
kube-system kubernetes-dashboard-64576d84bd-z6pff 0/1 CrashLoopBackOff 26 2h 192.168.162.87 kb-node <none>
Kubernetes system log:
root#KB-master:~# kubectl -n kube-system logs kubernetes-dashboard-64576d84bd-z6pff --follow
2018/09/11 09:27:03 Starting overwatch
2018/09/11 09:27:03 Using apiserver-host location: http://192.168.33.30:8001
2018/09/11 09:27:03 Skipping in-cluster config
2018/09/11 09:27:03 Using random key for csrf signing
2018/09/11 09:27:03 No request provided. Skipping authorization
2018/09/11 09:27:33 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service account's configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get http://192.168.33.30:8001/version: dial tcp 192.168.33.30:8001: i/o timeout
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ
Getting the msg when I'm trying to hit below link on the browser
URL:http://192.168.33.30:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login
Error: 'dial tcp 192.168.162.87:8443: connect: connection refused'
Trying to reach: 'https://192.168.162.87:8443/'
Anyone can help me with this.

http://192.168.33.30:8001 is not a legitimate API server URL. All communications with the API server use TLS internally (https:// URL scheme). These communications are verified using the API server CA certificate and are authenticated by mean of tokens signed by the same CA.
What you see is the result of a misconfiguration. At first sight it seems like you mixed pod, service and host networks.
Make sure you understand the difference between Host network, Pod network and Service network. These 3 networks can not overlap. For example --pod-network-cidr=192.168.0.0/16 must not include the IP address of your host, change it to 10.0.0.0/16 or something smaller if necessary.
After you have a clear overview of the network topology, run the setup again and everything will be configured correctly, including the Kubernetes CA.

Related

Kubernetes - Failed to Apply a yaml from a raw url, Unable to connect to the server: dial tcp: lookup raw.githubusercontent.com on: server misbehaving

Im Anddiy and im working with a kubernetes cluster deployed by Rancher.
Its important to say that all my machines don't have direct access to the internet, im using a proxy to use the internet for downloads or something, so i've setted RKE2 with this proxy during the installation steps.
Here i have a machine with an RKE2 that build up my Rancher, and from the Rancer U.I i've created my Kubernetes Cluster, here it tis:
[15:19] root#vmrmmstnodehom01 [~]:# kubectl get nodes
NAME STATUS ROLES AGE VERSION
vmrmmstnodehom01 Ready controlplane 5d16h v1.24.8
vmrmwrknodehom01 Ready controlplane,etcd,worker 5d20h v1.24.8
vmrmwrknodehom02 Ready worker 5d19h v1.24.8
vmrmwrknodehom03 Ready worker 5d19h v1.24.8
vmrmwrknodehom04 Ready worker 5d19h v1.24.8
My cluster is a clean cluster, no applications installed on it at moment.
I've tried to install the longhorn application by using this command ( got this on longhorn documentation):
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.4.0/deploy/longhorn.yaml
But when i tried that, this error message are displayed to me:
Unable to connect to the server: dial tcp: lookup raw.githubusercontent.com on 10.129.251.125:53: server misbehaving
I've tried to check if it is my proxy that don't connect to this url or something, but my machine connected succesfully to this url, i've tried that using the CURL -V and the longhorn url to test that.
I don't know if the kubernetes api has imported the proxy configs of my rke2/rancher, so i don't know if i need to set the proxy manually internal or something, really don't know what is happening here.

How to send http requests to the kubelet api server

I have a kubernetes cluster on EKS, in which I would like to send requests to the kubelet server (not the kube-apiserver, but the kubelet, which runs on each of the worker nodes).
My understanding is that the kubelet runs an http server on port 10250, so I opened the firewall (security group) in one of the worker nodes for that port so I can reach it from my IP. Example of a request:
curl -k https://public-ip-of-worker-node:10250/metrics/probes
but I get a 401 in response. I guess this is expected, as I am not authenticating in any way.
So, how can I authenticate to the kubelet server? I can communicate without problem with the kube-apiserver using kubectl, so I do have enough permissions from the IAM side.
From the docs start the kubelet with the --authentication-token-webhook and the --kubeconfig flags.
Then you can create a service-account and define role and rolebinding on the service account and use the service accounts bearer token with the curl command to call the kubelet API.

CRD probe failing

I am installing service catalog which uses CRD and have created the same. Now I am running my controller deployment file and the image running in it runs a CRD list command to verify CRD are in place. This use to work fine previously but now CRD Probe is failing with error:
1226 07:45:01.539118 1 round_trippers.go:438] GET https://169.72.128.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions?labelSelector=svcat%3Dtrue in 30000 milliseconds
I1226 07:45:01.539158 1 round_trippers.go:444] Response Headers:
Error: while waiting for ready Service Catalog CRDs: failed to list CustomResourceDefinition: Get https://169.72.128.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions?labelSelector=svcat%3Dtrue: dial tcp 169.72.128.1:443: i/o timeout
I have followed same steps as previously but could not debug now.
Inside the controller code it is trying to make following call:
list, err := r.client.ApiextensionsV1beta1().CustomResourceDefinitions().List(v1.ListOptions{LabelSelector: labels.SelectorFromSet(labels.Set{"svcat": "true"}).String()})
Which is failing.
Update 1 : Installation works fine in default namespace but fails in specific namespace.
Environment Info: On Prem k8s cluster, latest k8s, 2 node cluster.
It's not a port issue.Service accounts use 443 port to connect to Kubernetes API Server. Check the if there is any network policy blocking the communication between your namespace and Kube-System namespace.

How do we debug networking issues within istio pods?

I am working on setting up istio in my kubernetes cluster.
I downloaded istio-1.4.2 and installed demo profile and did manual sidecar injection.
But when I check sidecar pod logs, I am getting the below error.
2019-12-26T08:54:17.694727Z error k8s.io/client-go#v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:98: Failed to list *v1beta1.MutatingWebhookConfiguration: Get https://10.96.0.1:443/apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations?fieldSelector=metadata.name%3Distio-sidecar-injector&limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
It seems to be the networking issue, but could you please let me know what it is trying to do exactly?
Is there a way to get more logs than just 'connection refused'?
How do we verify networking issues between istio pods. It seems I cannot run 'wget', 'curl', 'tcpdump', 'netstat' etc within istio sidecar pod to debug further.
All the pods in kube-system namespace are working fine.
Check what port your API Server is serving https traffic(controlled by this flag --secure-port int Default: 6443). It may be 6443 instead of 443.
Check what is the value of server in your kubeconfig and are you able to connect to your kubernetes via kubectl using that kubeconfig.
Another thing to check is whether you have network policy attached to the namespace which blocks egress traffic.
And you could use an ephemeral container to debug issue with the sidecar
https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/
https://github.com/aylei/kubectl-debug

Kubernetes dashboard cannot be started

I create dashboard after I installed kubernetes with kubeadm.
kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml
Wait a while, the pod is crashed like:
kubectl get pods --all-namespaces
kubernetes-dashboard-3203831700-wq0v4 0/1 CrashLoopBackOff 3 3m
And I checked the pod log:
kubectl logs -f kubernetes-dashboard-3203831700-wq0v4 -n kube-system Using HTTP port: 9090
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
But I tried it mannually, the url works:
# curl https://10.96.0.1:443/version
curl: (35) Peer reports incompatible or unsupported protocol version.
Have anybody encountered this issue before? or help me?
I execute the following command:
rm -rf ~/.kube
Now it works. still a bit strange :-(