This was taken from Github (issue #24407) to Stackoverflow.
Even with the commit from Friday, May 6th 2016 (commit c11229f) to cluster/vsphere, this error
Error: 'dial tcp 172.17.0.2:9090: no route to host'
Trying to reach: 'http://172.17.0.2:9090/'
remains.
I tried on a fresh install of VMware vSphere ESXi 6.0.0; installed k8s with the standard KUBERNETES_PROVIDER=vsphere cluster/kube-up.sh and the script finished with positive results, this time with "kubernetes-dashboard" enabled from the start:
Cluster validation succeeded
Done, listing cluster services:
Kubernetes master is running at https://192.168.1.36
KubeDNS is running at https://192.168.1.36/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://192.168.1.36/api/v1/proxy /namespaces/kube-system/services/kubernetes-dashboard
Yet still unable to connect to the dashboard from my Mac with the infamous "no route to host"...
Am I mistakenly under the impression that a k8s installation should work out of the box on VMware vSphere?
Or is e.g. the lack of an external IP a probable cause in this? (if so I need to find out how to enable one - am under the impression kube-proxy is taking care of stuff)
$ kubectl get svc --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 10.244.240.240 <none> 53/UDP,53/TCP 2h
kubernetes-dashboard 10.244.240.121 <none> 80/TCP 2h
Related
I'm setting up an on-premise kubernetes cluster with kubeadm.
Here is the Kubernestes version
clientVersion:
buildDate: "2022-10-12T10:57:26Z"
compiler: gc
gitCommit: 434bfd82814af038ad94d62ebe59b133fcb50506
gitTreeState: clean
gitVersion: v1.25.3
goVersion: go1.19.2
major: "1"
minor: "25"
platform: linux/amd64
kustomizeVersion: v4.5.7
serverVersion:
buildDate: "2022-10-12T10:49:09Z"
compiler: gc
gitCommit: 434bfd82814af038ad94d62ebe59b133fcb50506
gitTreeState: clean
gitVersion: v1.25.3
goVersion: go1.19.2
major: "1"
minor: "25"
platform: linux/amd64
I have installed metallb version 0.13.7
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml
Everything is running
$ kubectl get all -n metallb-system
NAME READY STATUS RESTARTS AGE
pod/controller-84d6d4db45-l2r55 1/1 Running 0 35s
pod/speaker-48qn4 1/1 Running 0 35s
pod/speaker-ds8hh 1/1 Running 0 35s
pod/speaker-pfbcp 1/1 Running 0 35s
pod/speaker-st7n2 1/1 Running 0 35s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/webhook-service ClusterIP 10.104.14.119 <none> 443/TCP 35s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/speaker 4 4 4 4 4 kubernetes.io/os=linux 35s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/controller 1/1 1 1 35s
NAME DESIRED CURRENT READY AGE
replicaset.apps/controller-84d6d4db45 1 1 1 35s
But when i try to apply an IPaddressPool CRD i get an error
kubectl apply -f ipaddresspool.yaml
ipaddresspool.yaml file content
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: first-pool
namespace: metallb-system
spec:
addresses:
- 192.168.2.100-192.168.2.199
The error is a fail to call the validation webhook no route to host
Error from server (InternalError): error when creating "ipaddresspool.yaml": Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": dial tcp 10.104.14.119:443: connect: no route to host
Here is the same error with line brakes
Error from server (InternalError):
error when creating "ipaddresspool.yaml":
Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io":
failed to call webhook:
Post "https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s":
dial tcp 10.104.14.119:443: connect: no route to host
The IP -address is correct
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
webhook-service ClusterIP 10.104.14.119 <none> 443/TCP 18m
I have also tried installing metallb v 0.13.7 using helm but with the same result
Does someone know why the webhook cannot be called?
EDIT
As an answer to Thomas question, here is the description for webhook-service. NOTE that this is from another cluster with the same problem because I deleted the last cluster so the IP is not the same as last time
$ kubectl describe svc webhook-service -n metallb-system
Name: webhook-service
Namespace: metallb-system
Labels: <none>
Annotations: <none>
Selector: component=controller
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.105.157.72
IPs: 10.105.157.72
Port: <unset> 443/TCP
TargetPort: 9443/TCP
Endpoints: 172.17.0.3:9443
Session Affinity: None
Events: <none>
Once understood the issue is fairly simple.
The metallb setup described above works as it is supposed to.
However, the Kubernetes setup does not. Most likely due to bad network configuration.
Understanding the error
The key to understanding what is going on is the following error:
Error from server (InternalError): error when creating "ipaddresspool.yaml": Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": dial tcp 10.104.14.119:443: connect: no route to host
Part of the applied metallb manifest is going to deploy a so-called ValidatingWebhookConfiguration.
In the case of metallb this validating webhook will force the kube-apiserver to:
send metallb-related objects like IPAddressPool to the webhook whenever someone creates or updates such an object
wait for the webhook to perform some checks on the object (e.g. validate that CIDRs and IPs are valid and not something like 481.9.141.12.27)
and finally receive an answer from the webhook whether or not that object satisfies metallb's requirements and is allowed to be created (persisted to etcd)
The error above pretty clearly suggests that the first out of the three outlined steps is failing.
Debugging
To fix this error one has to debug the current setup, particularly the connection from the kube-apiserver to webhook-service.metallb-system.svc:443.
There is a wide range of possible network misconfigurations that could lead to the error. However, with the information available to us it is most likely going to be an error with the configured CNI.
Knowing that here is some help and a bit of guidance regarding the further debugging process:
Since the kube-apiserver is hardened by default it won't be possible to execute a shell into it.
For that reason one should deploy a debug application with the same network configuration as the kube-apiserver onto one of the control-plane nodes.
This can be achieved by executing the following command:
kubectl debug -n kube-system node/<control-plane-node> -it --image=nicolaka/netshoot
Using common tools one can now reproduce the error inside the interactive shell. The following command is expected to fail (in a similar fashion to the kube-apiserver):
curl -m 10 -k https://<webhook-service-ip>:443/
Given above error message it should fail due to bad routing on the node.
To check the routing table execute the following command:
routel
Does someone know why the webhook cannot be called?
The output should show multiple CIDR ranges configured one of which is supposed to include the IP queried earlier.
Most likely the CIDR range in question will either be missing or a bad gateway configured which leads to the no route to host error.
It is the CNIs job to update routing tables on all nodes and ensure that nodes can reach these addresses so adding or editing new Kubernetes related entries to the routing table manually is not recommended.
Further debugging is dependent on the exact setup.
Depending on the setup and CNI of choice kube-proxy may or may not be involved in the issue as well.
However, inspecting the CNI configuration and logs is a good next step.
Some bonus information
Some CNIs require the user to pay more attention to certain features and configuration as there can be issues involved otherwise.
Here are some popular CNIs that fall into this category:
Calico (see here)
Weave (see here)
Kube-Router (see here)
Here is the output of minikube dashbaord
ubuntu#ip-172-31-5-166:~$ minikube dashboard
* Enabling dashboard ...
* Verifying dashboard health ...
* Launching proxy ...
* Verifying proxy health ...
* Opening http://127.0.0.1:45493/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser...
- http://127.0.0.1:45493/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/
I have enabled port 45493 at Security Group Level and also on Linux VM. However,, when I'm trying to access the Kube dashboard, I don't have luck
wget http://13.211.44.210:45493/
--2020-04-16 05:50:52-- http://13.211.44.210:45493/
Connecting to 13.211.44.210:45493... failed: Connection refused.
However, when I do the below, it works and produces index.html file with status code 200
wget http://127.0.0.1:45493/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/
--2020-04-16 05:52:55-- http://127.0.0.1:45493/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/
Connecting to 127.0.0.1:45493... connected.
HTTP request sent, awaiting response... 200 OK
Steps to reproduce at high level is as below:
EC2 Ubuntu of size t2.large
Install minikube, minikube start --driver=docker
Perform deployment as like kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
kubectl get pods -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
dashboard-metrics-scraper-84bfdf55ff-xx8pl 1/1 Running 0 26m
kubernetes-dashboard-bc446cc64-7nl68 1/1 Running 0 26m
5.kubectl get svc -n kubernetes-dashboard
TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 10.102.85.110 <none> 8000/TCP 40m
kubernetes-dashboard ClusterIP 10.99.75.241 <none> 80/TCP 40m
My question is why I'm unable to access the internet?
This is by design, minikube is a development tool for local environments.
You can deploy an ingress or loadbalancer service to expose the dashboard, if you really know what you are doing.
I newbie question related with k8s. I've just installed a k3d cluster.
I've deployed an this helm chart:
$ helm install stable/docker-registry
It's been installed and pod is running correctly.
Nevertheless, I don't quite figure out how to get access to this just deployed service.
According to documentation, it's listening on 5000 port, and is using a ClusterIP. A service is also deployed.
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 42h
docker-registry-1580212517 ClusterIP 10.43.80.185 <none> 5000/TCP 19m
EDIT
I've been able to say to chard creates an ingress:
$ kubectl get ingresses.networking.k8s.io -n default
NAME HOSTS ADDRESS PORTS AGE
docker-registry-1580214408 chart-example.local 172.20.0.4 80 10m
Nevertheless, I'm still without being able tp push images to registry:
$ docker push 172.20.0.4/feedly:v1
The push refers to repository [172.20.0.4/feedly]
Get https://172.20.0.4/v2/: x509: certificate has expired or is not yet valid
Since the service type is ClusterIP, you can't access the service from host system. You can run below command to access the service from your host system.
kubectl port-forward --address 0.0.0.0 svc/docker-registry-1580212517 5000:5000 &
curl <host IP/name>:5000
I have a very simple springboot service deployed on minikube in windows 10.
C:\Software\Kubernetes>kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
myspringbootserver 1/1 1 1 68m
C:\Software\Kubernetes>kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 49d
myspringbootserver NodePort 10.110.179.207 <none> 9080:30451/TCP 6m50s
C:\Software\Kubernetes>minikube service myspringbootserver --url
http://192.168.99.101:30451
But when I try to hit the service from my chrome browser with url
http://192.168.99.101:30451/MySpringBootServer/heartbeat
getting connection refused exception.Not sure what is going wrong.Could anyone help to resolve it please?
enter image description here
Can you curl or wget using the IP address of the pod?
For example kubectl exec -it podname -- curl http://podip:9080/MySpringBootServer/heartbeat
if not, ensure the path is correct
if yes, make sure the pod exists as an endpoint of the service
kubectl get endpoints myspringbootserver
there is a good debugging document regarding services here:
https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/#debugging-services
I am beginner to kubernetes. I am trying to install minikube wanted to run my application in kubernetes. I am using ubuntu 16.04
I have followed the installation instructions provided here
https://kubernetes.io/docs/setup/learning-environment/minikube/#using-minikube-with-an-http-proxy
Issue1:
After installing kubectl, virtualbox and minikube I have run the command
minikube start --vm-driver=virtualbox
It is failing with following error
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0912 17:39:12.486830 17689 start.go:305] Error restarting
cluster: restarting kube-proxy: waiting for kube-proxy to be
up for configmap update: timed out waiting for the condition
But when I checked the virtualbox I see the minikube VM running and when I run the kubectl
kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.10
I see the deployments
kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
hello-minikube 1 1 1 1 27m
I exposed the hello-minikube deployment as service
kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-minikube LoadBalancer 10.102.236.236 <pending> 8080:31825/TCP 15m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19h
I got the url for the service
minikube service hello-minikube --url
http://192.168.99.100:31825
When I try to curl the url I am getting the following error
curl http://192.168.99.100:31825
curl: (7) Failed to connect to 192.168.99.100 port 31825: Connection refused
1)If minikube cluster got failed while starting, how did the kubectl able to connect to minikube to do deployments and services?
2) If cluster is fine, then why am i getting connection refused ?
I was looking at this proxy(https://kubernetes.io/docs/setup/learning-environment/minikube/#starting-a-cluster) what is my_proxy in this ?
Is this minikube ip and some port ?
I have tried this
Error restarting cluster: restarting kube-proxy: waiting for kube-proxy to be up for configmap update: timed out waiting for the condition
but do not understand how #3(set proxy) in solution will be done. Can some one help me getting instructions for proxy ?
Adding the command output which was asked in the comments
kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
etcd-minikube 1/1 Running 0 4m
kube-addon-manager-minikube 1/1 Running 0 5m
kube-apiserver-minikube 1/1 Running 0 4m
kube-controller-manager-minikube 1/1 Running 0 6m
kube-dns-86f4d74b45-sdj6p 3/3 Running 0 5m
kube-proxy-7ndvl 1/1 Running 0 5m
kube-scheduler-minikube 1/1 Running 0 5m
kubernetes-dashboard-5498ccf677-4x7sr 1/1 Running 0 5m
storage-provisioner 1/1 Running 0 5m
I deleted minikube and removed all files under ~/.minikube and
reinstalled minikube. Now it is working fine. I did not get the output
before but I have attached it after it is working to the question. Can
you tell me what does the output of this command tells ?
It will be very difficult or even impossible to tell what was exactly wrong with your Minikube Kubernetes cluster when it is already removed and set up again.
Basically there were a few things that you could do to properly troubleshoot or debug your issue.
Adding the command output which was asked in the comments
The output you posted is actually only part of the task that #Eduardo Baitello asked you to do. kubectl get po -n kube-system command simply shows you a list of Pods in kube-system namespace. In other words this is the list of system pods forming your Kubernetes cluster and, as you can imagine, proper functioning of each of these components is crucial. As you can see in your output the STATUS of your kube-proxy pod is Running:
kube-proxy-7ndvl 1/1 Running 0 5m
You were also asked in #Eduardo's question to check its logs. You can do it by issuing:
kubectl logs kube-proxy-7ndvl
It could tell you what was wrong with this particular pod at the time when the problem occured. Additionally in such case you may use describe command to see other pod details (sometimes looking at pod events may be very helpful to figure out what's going on with it):
kubectl describe pod kube-proxy-7ndvl
The suggestion to check this particular Pod status and logs was most probably motivated by this fragment of the error messages shown during your Minikube startup process:
E0912 17:39:12.486830 17689 start.go:305] Error restarting
cluster: restarting kube-proxy: waiting for kube-proxy to be
up for configmap update: timed out waiting for the condition
As you can see this message clearly suggests that there is in short "something wrong" with kube-proxy so it made a lot of sense to check it first.
There is one more thing you may have not noticed:
kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-minikube LoadBalancer 10.102.236.236 <pending> 8080:31825/TCP 15m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19h
Your hello-minikube service was not completely ready. In EXTERNAL-IP column you can see that its state was pending. As you can use describe command to describe Pods you can do so to get details of the service. Simple:
describe service hello-minikube
could tell you quite a lot in such case.
1)If minikube cluster got failed while starting, how did the kubectl
able to connect to minikube to do deployments and services? 2) If
cluster is fine, then why am i getting connection refused ?
Remember that Kubernetes Cluster is not a monolith structure and consists of many parts that depend on one another. The fact that kubectl worked and you could create deployment doesn't mean that the whole cluster was working fine and as you can see in the error message it was suggesting that one of its components, namely kube-proxy, could actually not function properly.
Going back to the beginning of your question...
I have followed the installation instructions provided here
https://kubernetes.io/docs/setup/learning-environment/minikube/#using-minikube-with-an-http-proxy
Issue1: After installing kubectl, virtualbox and minikube I have run
the command
minikube start --vm-driver=virtualbox
as far as I understood you don't use the http proxy so you didn't follow instructions from this particular fragment of the docs that you posted, did you ?
I have the impression that you mix 2 concepts. kube-proxy which is a Kubernetes cluster component and which is deployed as pod in kube-system space and http proxy server mentioned in this fragment of documentation.
I was looking at this
proxy(https://kubernetes.io/docs/setup/learning-environment/minikube/#starting-a-cluster)
what is my_proxy in this ?
If you don't know what is your http proxy address, most probably you simply don't use it and if you don't use it to connect to the Internet from your computer, it doesn't apply to your case in any way.
Otherwise you need to set it up for your Minikube by providing additional flags when you start it as follows:
minikube start --docker-env http_proxy=http://$YOURPROXY:PORT \
--docker-env https_proxy=https://$YOURPROXY:PORT
If you were able to start your Minikube and now it works properly only using the command:
minikube start --vm-driver=virtualbox
your issue was caused by something else and you don't need to provide the above mentioned flags to tell your Minikube what is your http proxy server that you're using.
As far as I understand currently everything is up and running and you can access the url returned by the command minikube service hello-minikube --url without any problem, right ? You can also run the command kubectl get service hello-minikube and check if its output differs from what you posted before. As you didn't attach any yaml definition files it's difficult to tell if it was nothing wrong with your service definition. Also note that Load Balancer is a service type designed to work with external load balancers provided by cloud providers and minikube uses NodePort instead of it.