minikube ip is not reachable - kubernetes

I have created one service called fleetman-webapp:
apiVersion: v1
kind: Service
metadata:
name: fleetman-webapp
spec:
selector:
app: webapp
ports:
- name: http
port: 80
nodePort: 30080
type: NodePort
also, a pod named webapp:
apiVersion: v1
kind: Pod
metadata:
name: webapp
labels:
app: webapp
spec:
containers:
- name: webapp
image: richardchesterwood/k8s-fleetman-webapp-angular:release0
I have checked the minikube ip:
192.168.99.102
But when I type in the browser 192.168.99.102:30080, the webapp is not reachable:
Please note that I use Ubuntu latest version. I have verified furthermore if proxies and firewalls are active:
cat /etc/environment:
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
iptables -L:
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
I have also disabled ufw in Ubuntu, but no success, the url 192.168.99.102:30080 .
Would you help me please ? thanks in advance for your answer.

Even though, you are exposing port 30080 via NodePort in minikube, minikube will still not expose it because it will use its own external port to listen to this service. Minikube tunnels the service to expose to the outer world. To find out that exposed port:
minikube service $SERVICE_NAME
so, in your case
minikube service fleetman-webapp

There are a lot of different hypervisors which can work with minikube. Choosing one will be highly dependent on variables like operating system. Some of them are:
Virtualbox
Hyper-V
VMware Fusion
KVM2
Hyperkit
"Docker (--vm-driver=none)" (see the quotes)
There is official documentation talking about it: Kubernetes.io: Minikube: Specifying the vm driver
Choosing Hypervisor will affect how the minikube will behave.
Focusing on:
Docker: --vm-driver=none
Virtualbox: --vm-driver=virtualbox
Docker
Official documentation sums it up:
Minikube also supports a --vm-driver=none option that runs the Kubernetes components on the host and not in a VM. Using this driver requires Docker and a Linux environment but not a hypervisor.
-- Kubernetes.io: Install minikube: Install a hypervisor
The output of command$ sudo minikube ip will show IP address of a host machine.
Service object type of NodePort will be available with IP_ADDRESS_OF_HOST:NODEPORT_PORT.
Following with command: $ kubectl get nodes -o wide:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
K8S Ready master 95s v1.17.3 192.168.0.114 <none> Ubuntu 18.04.4 LTS 5.3.0-28-generic docker://19.3.8
Please take a specific look on:
INTERNAL-IP
192.168.0.114
It's the same IP address as a host it's working on. You can (for example) curl pods without any restrictions. Please consider reading the article in included citing:
Caution: The none VM driver can result in security and data loss issues. Before using --vm-driver=none, consult this documentation for more information.
You can check what was exposed with command:
$ sudo netstat -tulpn
Virtualbox
Creating a minikube instance with --vm-driver=virtualbox will create a virtual machine with Virtualbox as host.
Virtual machine created with this kind of --vm-driver will have 2 network interfaces provided below:
NAT
Host-only adapter
What is important is that your minikube instance will be accessible by Host-only adapter.
Host-only networking. This can be used to create a network containing the host and a set of virtual machines, without the need for the host's physical network interface. Instead, a virtual network interface, similar to a loopback interface, is created on the host, providing connectivity among virtual machines and the host.
-- Virtualbox.org: Virtual networking
For example:
minikube host-only adapter will have an address: 192.168.99.103
Your host-only adapter will have an address: 192.168.99.1
They must be different!
If you are having issues with connecting to this adapter please check:
If minikube's host-only adapter address is responding to ping when minikube start completed successfully.
Your host-only adapter is present in your network configuration by issuing either:
ip a
ifconfig
Your host-only adapter address is in range of your minikube instance (subnet)
From my experience reboot/recreation of this adapter worked all the time if something wasn't right.
The output of command$ sudo minikube ip will show IP address of a Host-only adapter.
Following with command: $ kubectl get nodes -o wide:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
m01 Ready master 29m v1.17.3 192.168.99.103 <none> Buildroot 2019.02.9 4.19.94 docker://19.3.6
Please take a specific look once more on INTERNAL-IP and ip address associated with it.
Service object type of NodePort will be available with:
IP_ADDRESS_OF_HOST_ONLY_ADAPTER:NODEPORT_PORT.
I recreated your Deployment and Service attached to it and it worked in both --vm-driver=none and --vm-driver=virtualbox cases.
Please let me know if you have any questions in this topic.

I have had the same issue and have been trying to solve that for the last 2 days I have tried to install ingress addon:
minikube addons enable ingress
and also tried to run :
minikube tunnel
looked for a way to allow the host machine to access the container IP address but apparently couldn't find a way to do that.
If you run minikube on docker:
minikube start --driver=docker
you won't be able to access the minikube IP from your host machine since the minikube container's IP address would by accessible through DockerDesktopVM but not from your host machine.
You could run minikube on another driver such as VirtualBox or Hyperv, and that might help.
minikube start --driver=hyperv
minikube start --driver=virtualbox
Read more about the minikube drivers
In fact, that's really annoying if you don't have enough resources on your computer to run both the Docker desktop VM and the minikube VM at the same time and will eventually slow down your computer.
To solve that docker-desktop UI for Mac and Windows provides an easier alternative compared to minikube, which you could simply activate the Kubernetes feature on your docker-desktop UI:
once it is setup you can right click on the docker desktop icon > Kubernetes
To verify now that your deployement/service works properly:
kubectl apply -f /file.yaml

For this specific (and really great) course about Kubernetes on Udemy from Richard Chesterwood the following solution should work out of the box on Windows: just start the minikube with hyper-v driver, then it will automatically map all the ports you are expecting onto your host machine, like detailed explained here by Dawid Kruk. Therefore all you need to start minikube "correctly" is the following command:
minikube start --driver=hyperv
Be careful by specifying exact amount of memory you give to this minikube instance. In my experiences hyper-v is a bit sensitive about how much memory you will give to it, what could result in errors:
minikube start --driver=hyperv --memory=8192
...
Not enough memory in the system to start the virtual machine minikube.
Could not initialize memory: Not enough memory resources are available to complete this operation. (0x8007000E).
'minikube' failed to start. (Virtual machine ID D4BC7B61-4E4D-4079-94DE-...)
Not enough memory in the system to start the virtual machine minikube with ram size 8192 megabytes. (Virtual machine ID ...)
Therefore just use the unspecific command given above and hyper-v will figure out, how much memory it really needs on it's own.

If you are running minikube in a Windows, then minikube must run as an Administrator command prompt window.

Related

Unable to reach the application using Ingress in Minikube (macOS)

I am trying to follow this tutorial to setup Minikube on my MacBook.
Minikube with ingress
I have also referred Stack overflow question and Stack over flow question 2 both of these are not working for me.
When I run Minikube tunnel it says to enter the password and then get stuck after entering my password.
sidharth#Sidharths-MacBook-Air helm % minikube tunnel
✅ Tunnel successfully started
📌 NOTE: Please do not close this terminal as this process must stay alive for the tunnel to be accessible ...
❗ The service/ingress example-ingress requires privileged ports to be exposed: [80 443]
🔑 sudo permission will be asked for it.
❗ The service/ingress minimal-ingress requires privileged ports to be exposed [80 443]
🏃 Starting tunnel for service example-ingress.
🔑 sudo permission will be asked for it.
🏃 Starting tunnel for service minimal-ingress.
Password:
I am getting the below response when I run kubectl ge ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
example-ingress nginx hello-world.info localhost 80 34m
This is an issue specifically with the docker driver, and it's only an output issue. If you use a VM driver (like hyperkit for macOS), you'll get the expected output in the documentation.
This stems from the fact that we need to do two discrete things to tunnel for a container driver (since it needs to route to 127.0.0.1) and for a VM driver.
We can potentially look into fixing this so that the output for both versions are similar, but the tunnel itself is working fine.
Refer this github link for more information.
On the Mac's with M1 chips, I can use minikube with podman (can brew install both). The "minikube tunnel" will ask for password and then appear to hang, because it's essentially establishing a tunnel similar to "kubectl port-forward" with that process between 127.0.0.1:443 and your ingress, and that tunnel will disappear once you ctrl-c out of the "minikube tunnel" process.
Note, this is different from the experience/behavior you'll get if you're using minikube with virtualbox on an Intel-based Mac, which will return the list of mapped ports and always has the ingress reachable from on the virtualbox VM.

Access Minikube Services from public IP [duplicate]

I know minikube should be used for local only, but i'd like to create a test environment for my applications.
In order to do that, I wish to expose my applications running inside the minikube cluster to external access (from any device on public internet - like a 4G smartphone).
note : I run minikube with --driver=docker
kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
web8080 NodePort 10.99.39.162 <none> 8080:31613/TCP 3d1h
minikube ip
192.168.49.2
One way to do it is as follows :
firewall-cmd --add-port=8081/tcp
kubectl port-forward --address 0.0.0.0 services/web8080 8081:8080
then I can access it using :
curl localhost:8081 (directly from the machine running the cluster inside a VM)
curl 192.168.x.xx:8081 (from my Mac in same network - this is the private ip of the machine running the cluster inside a VM)
curl 84.xxx.xxx.xxx:8081 (from a phone connected in 4G - this is the public ip exposed by my router)
I don't want to use this solution because kubectl port-forward is weak and need to be run every time the port-forwarding is no longer active.
How can I achieve this ?
(EDITED) - USING LOADBALANCER
when using LoadBalancer type and minikube tunnel, I can expose the service only inside the machine running the cluster.
kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-service LoadBalancer 10.111.61.218 10.111.61.218 8080:31831/TCP 3d3h
curl 10.111.61.218:8080 (inside the machine running the cluster) is working
but curl 192.168.x.xx:8080 (from my Mac on same LAN) is not working
Thanks
Minikube as a development tool for a single node Kubernetes cluster provides inherent isolation layer between Kubernetes and the external devices (being specific the inbound traffic to your cluster from LAN/WAN).
Different --drivers are allowing for flexibility when it comes to the place where your Kubernetes cluster will be spawned and how it will behave network wise.
A side note (workaround)!
As your minikube already resides in a VM and uses --driver=docker you could try to use --driver=none (you will be able to curl VM_IP:NodePort from the LAN). It will spawn your Kubernetes cluster directly on the VM.
Consider checking it's documentation as there are some certain limitations/disadvantages:
Minikube.sigs.k8s.io: Docs: Drivers: None
As this setup is already basing on the VM (with unknown hypervisor) and the cluster is intended to be exposed outside of your LAN, I suggest you going with the production-ready setup. This will inherently eliminate the connectivity issues you are facing. Kubernetes cluster will be provisioned directly on a VM and not in the Docker container.
Explaining the --driver=docker used: It will spawn a container on a host system with Kubernetes inside of it. Inside of this container, Docker will be used once again to spawn the necessary Pods to run the Kubernetes cluster.
As for the tools to provision your Kubernetes cluster you will need to chose the option that suits your needs the most. Some of them are the following:
Kubeadm
Kubespray
MicroK8S
After you created your Kubernetes cluster on a VM you could forward the traffic from your router directly to your VM.
Additional resources that you might find useful:
Stackoverflow.com: Questions Expose Kubernetes cluster to the Internet (Virtualbox with minikube)
curl $(minikube ip):$NODE_PORT : Now we can test that the app is exposed outside of the cluster using curl, the IP of the Node and the externally exposed port.
For you : curl 192.168.49.2:31613
Use nginx reverse-proxy
https://www.zepworks.com/posts/access-minikube-remotely-kvm/
install nginx, then in nginx.conf add this
stream {
server {
listen 8081;
proxy_pass 192.168.49.2:8080;
}
}
restart nginx
One way that I use to get around the fact that the process of kubectl port-forward stops after a while is to create a detach session using tmux following this. With that, I haven't had any problems with the exact same Minikube cluster configuration that you have.

How to expose Minikube cluster to internet

I know minikube should be used for local only, but i'd like to create a test environment for my applications.
In order to do that, I wish to expose my applications running inside the minikube cluster to external access (from any device on public internet - like a 4G smartphone).
note : I run minikube with --driver=docker
kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
web8080 NodePort 10.99.39.162 <none> 8080:31613/TCP 3d1h
minikube ip
192.168.49.2
One way to do it is as follows :
firewall-cmd --add-port=8081/tcp
kubectl port-forward --address 0.0.0.0 services/web8080 8081:8080
then I can access it using :
curl localhost:8081 (directly from the machine running the cluster inside a VM)
curl 192.168.x.xx:8081 (from my Mac in same network - this is the private ip of the machine running the cluster inside a VM)
curl 84.xxx.xxx.xxx:8081 (from a phone connected in 4G - this is the public ip exposed by my router)
I don't want to use this solution because kubectl port-forward is weak and need to be run every time the port-forwarding is no longer active.
How can I achieve this ?
(EDITED) - USING LOADBALANCER
when using LoadBalancer type and minikube tunnel, I can expose the service only inside the machine running the cluster.
kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-service LoadBalancer 10.111.61.218 10.111.61.218 8080:31831/TCP 3d3h
curl 10.111.61.218:8080 (inside the machine running the cluster) is working
but curl 192.168.x.xx:8080 (from my Mac on same LAN) is not working
Thanks
Minikube as a development tool for a single node Kubernetes cluster provides inherent isolation layer between Kubernetes and the external devices (being specific the inbound traffic to your cluster from LAN/WAN).
Different --drivers are allowing for flexibility when it comes to the place where your Kubernetes cluster will be spawned and how it will behave network wise.
A side note (workaround)!
As your minikube already resides in a VM and uses --driver=docker you could try to use --driver=none (you will be able to curl VM_IP:NodePort from the LAN). It will spawn your Kubernetes cluster directly on the VM.
Consider checking it's documentation as there are some certain limitations/disadvantages:
Minikube.sigs.k8s.io: Docs: Drivers: None
As this setup is already basing on the VM (with unknown hypervisor) and the cluster is intended to be exposed outside of your LAN, I suggest you going with the production-ready setup. This will inherently eliminate the connectivity issues you are facing. Kubernetes cluster will be provisioned directly on a VM and not in the Docker container.
Explaining the --driver=docker used: It will spawn a container on a host system with Kubernetes inside of it. Inside of this container, Docker will be used once again to spawn the necessary Pods to run the Kubernetes cluster.
As for the tools to provision your Kubernetes cluster you will need to chose the option that suits your needs the most. Some of them are the following:
Kubeadm
Kubespray
MicroK8S
After you created your Kubernetes cluster on a VM you could forward the traffic from your router directly to your VM.
Additional resources that you might find useful:
Stackoverflow.com: Questions Expose Kubernetes cluster to the Internet (Virtualbox with minikube)
curl $(minikube ip):$NODE_PORT : Now we can test that the app is exposed outside of the cluster using curl, the IP of the Node and the externally exposed port.
For you : curl 192.168.49.2:31613
Use nginx reverse-proxy
https://www.zepworks.com/posts/access-minikube-remotely-kvm/
install nginx, then in nginx.conf add this
stream {
server {
listen 8081;
proxy_pass 192.168.49.2:8080;
}
}
restart nginx
One way that I use to get around the fact that the process of kubectl port-forward stops after a while is to create a detach session using tmux following this. With that, I haven't had any problems with the exact same Minikube cluster configuration that you have.

Exposing bare metal kubernetes cluster to internet

I am trying to setup own single-node kubernetes cluster on bare metal dedicated server. I am not that experienced in dev-ops but I need some service to be deployed for my own project. I already did a cluster setup with juju and conjure-up kubernetes over LXD. I have running cluster pretty fine.
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
conjure-canonical-kubern-3b3 conjure-up-localhost-db9 localhost/localhost 2.4.3 unsupported 23:49:09Z
App Version Status Scale Charm Store Rev OS Notes
easyrsa 3.0.1 active 1 easyrsa jujucharms 195 ubuntu
etcd 3.2.10 active 3 etcd jujucharms 338 ubuntu
flannel 0.10.0 active 2 flannel jujucharms 351 ubuntu
kubeapi-load-balancer 1.14.0 active 1 kubeapi-load-balancer jujucharms 525 ubuntu exposed
kubernetes-master 1.13.1 active 1 kubernetes-master jujucharms 542 ubuntu
kubernetes-worker 1.13.1 active 1 kubernetes-worker jujucharms 398 ubuntu exposed
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 10.213.117.66 Certificate Authority connected.
etcd/0* active idle 1 10.213.117.171 2379/tcp Healthy with 3 known peers
etcd/1 active idle 2 10.213.117.10 2379/tcp Healthy with 3 known peers
etcd/2 active idle 3 10.213.117.238 2379/tcp Healthy with 3 known peers
kubeapi-load-balancer/0* active idle 4 10.213.117.123 443/tcp Loadbalancer ready.
kubernetes-master/0* active idle 5 10.213.117.172 6443/tcp Kubernetes master running.
flannel/1* active idle 10.213.117.172 Flannel subnet 10.1.83.1/24
kubernetes-worker/0* active idle 7 10.213.117.136 80/tcp,443/tcp Kubernetes worker running.
flannel/4 active idle 10.213.117.136 Flannel subnet 10.1.27.1/24
Entity Meter status Message
model amber user verification pending
Machine State DNS Inst id Series AZ Message
0 started 10.213.117.66 juju-b03445-0 bionic Running
1 started 10.213.117.171 juju-b03445-1 bionic Running
2 started 10.213.117.10 juju-b03445-2 bionic Running
3 started 10.213.117.238 juju-b03445-3 bionic Running
4 started 10.213.117.123 juju-b03445-4 bionic Running
5 started 10.213.117.172 juju-b03445-5 bionic Running
7 started 10.213.117.136 juju-b03445-7 bionic Running
I also deployed Hello world application to output some hello on port 8080 inside the pod and nginx-ingress for it to re-route the traffic to this service on specified host.
NAME READY STATUS RESTARTS AGE
pod/hello-world-696b6b59bd-fznwr 1/1 Running 1 176m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/example-service NodePort 10.152.183.53 <none> 8080:30450/TCP 176m
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 10h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello-world 1/1 1 1 176m
NAME DESIRED CURRENT READY AGE
replicaset.apps/hello-world-696b6b59bd 1 1 1 176m
When I do curl localhost as expected I have connection refused, which looks still fine as it's not exposed to cluster. when I curl the kubernetes-worker/0 with public address 10.213.117.136 on port 30450 (which I get from kubectl get all)
$ curl 10.213.117.136:30450
Hello Kubernetes!
Everything works like a charm (which is obvious). When I do
curl -H "Host: testhost.com" 10.213.117.136
Hello Kubernetes!
It works again like charm! That means ingress controller is successfully routing port 80 based on host rule to correct services. At this point I am 100% sure that cluster works as it should.
Now I am trying to access this service over the internet externally. When I load <server_ip> obviously nothing loads as it's living inside own lxd subnet. Therefore I was thinking forward port 80 from server eth0 to this IP. So I added this rule to iptables
sudo iptables -t nat -A PREROUTING -p tcp -j DNAT --to-destination 10.213.117.136 (For the sake of example let's route everything not only port 80). Now when I open on my computer http://<server_ip> it loads!
So the real question is how to do that on production? Should I setup this forwarding rule in iptables? Is that normal approach or hacky solution and there is something "standard" which I am missing? The thing is to add this rule with static worker node will make the cluster completely static. IP eventually change, I can remove/add units to workers and it will stop working. I was thinking about writing script which will obtain this IP address from juju like this:
$ juju status kubernetes-worker/0 --format=json | jq '.machines["7"]."dns-name"'
"10.213.117.136"
and add it to IP-tables, which is more okay-ish solution than hardcoded IP but still I feel it's a tricky and there must be a better way.
As last idea I get to run HAProxy outside of the cluster, directly on the machine and just do forwarding of traffic to all available workers. This might eventually also work. But still I don't know the answer what is the correct solution and what is usually used in this case. Thank you!
So the real question is how to do that on production?
The normal way to do this in a production system is to use a Service.
The simplest case is when you just want your application to be accessible from outside on your node(s). In that case you can use a Type NodePort Service. This would create the iptables rules necessary to forward the traffic from the host IP address to the pod(s) providing the service.
If you have a single node (which is not recommended in production!), you're ready at this point.
If you have multiple nodes in your Kubernetes cluster, all of them would be configured by Kubernetes to provide access to the service (your clients could use any of them to access the service). Though, you'd have to solve the problem of how the clients would know which nodes are available to be contacted...
There are several ways to handle this:
use a protocol understood by the client to publish the currently available IP addresses (for example DNS),
use a floating (failover, virtual, HA) IP address managed by some software on your Kubernetes nodes (for example pacemaker/corosync), and direct the clients to this address,
use an external load-balancer, configured separately, to forward traffic to some of the operating nodes,
use an external load-balancer, configured automatically by Kubernetes using a cloud provider integration script (by using a Type LoadBalancer Service), to forward traffic to some of the operating nodes.

How does DNS resolution work on Kubernetes with multiple networks?

I have a 4 node Kubernetes cluster, 1 x controller and 3 x workers. The following shows how they are configured with the versions.
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-ctrl-1 Ready master 1h v1.11.2 192.168.191.100 <none> Ubuntu 18.04.1 LTS 4.15.0-1021-aws docker://18.6.1
turtle-host-01 Ready <none> 1h v1.11.2 192.168.191.53 <none> Ubuntu 18.04.1 LTS 4.15.0-29-generic docker://18.6.1
turtle-host-02 Ready <none> 1h v1.11.2 192.168.191.2 <none> Ubuntu 18.04.1 LTS 4.15.0-34-generic docker://18.6.1
turtle-host-03 Ready <none> 1h v1.11.2 192.168.191.3 <none> Ubuntu 18.04.1 LTS 4.15.0-33-generic docker://18.6.1
Each of the nodes has two network interfaces, for arguments sake eth0 and eth1. eth1 is the network that I want to the cluster to work on. I setup the controller using kubeadm init and passed --api-advertise-address 192.168.191.100. The worker nodes where then joined using this address.
Finally on each node I modified the kubelet service to have --node-ip set so that the layout looks as above.
The cluster appears to be working correctly and I can create pods, deployments etc. However the issue I have is that none of the pods are able to use the kube-dns service for DNS resolution.
This is not a problem with resolution, rather that the machines cannot connect to the DNS service to perform the resolution. For example if I run a busybox container and access it to perform nslookup i get the following:
/ # nslookup www.google.co.uk
nslookup: read: Connection refused
nslookup: write to '10.96.0.10': Connection refused
I have a feeling that this is down to not using the default network and because of that I suspect some Iptables rules are not correct, that being said these are just guesses.
I have tried both the Flannel overlay and now Weave net. The pod CIDR range is 10.32.0.0/16 and the service CIDR is as default.
I have noticed that with Kubernetes 1.11 there are now pods called coredns rather than one kube-dns.
I hope that this is a good place to ask this question. I am sure I am missing something small but vital so if anyone has any ideas that would be most welcome.
Update #1:
I should have said that the nodes are not all in the same place. I have a VPN running between them all and this is the network I want things to communicate over. It is an idea I had to try and have distributed nodes.
Update #2:
I saw another answer on SO (DNS in Kubernetes not working) that suggested kubelet needed to have --cluster-dns and --cluster-domain set. This is indeed the case on my DEV K8s cluster that I have running at home (on one network).
However it is not the case on this cluster and I suspect this is down to a later version. I did add the two settings to all nodes in the cluster, but it did not make things work.
Update #3
The topology of the cluster is as follows.
1 x Controller is in AWS
1 x Worker is in Azure
2 x Worker are physical machines in a colo Data Centre
All machines are connected to each other using ZeroTier VPN on the 192.168.191.0/24 network.
I have not configured any special routing. I agree that this is probably where the issue is, but I am not 100% sure what this routing should be.
WRT to kube-dns and nginx, I have not tainted my controller so nginx is not on the master, not is busybox. nginx and busybox are on workers 1 and 2 respectively.
I have used netcat to test connection to kube-dns and I get the following:
/ # nc -vv 10.96.0.10 53
nc: 10.96.0.10 (10.96.0.10:53): Connection refused
sent 0, rcvd 0
/ # nc -uvv 10.96.0.10 53
10.96.0.10 (10.96.0.10:53) open
The UDP connection does not complete.
I modified my setup so that I could run containers on the controller, so kube-dns, nginx and busybox are all on the controller, and I am able to connect and resolve DNS queries against 10.96.0.10.
So all this does point to routing or IPTables IMHO, I just need to work out what that should be.
Update #4
In response to comments I can confirm the following ping test results.
Master -> Azure Worker (Internet) : SUCCESS : Traceroute SUCCESS
Master -> Azure Worker (VPN) : SUCCESS : Traceroute SUCCESS
Azure Worker -> Master (Internet) : SUCCESS : Traceroute FAIL (too many hops)
Azure Worker -> Master (VPN) : SUCCESS : Traceroute SUCCESS
Master -> Colo Worker 1 (Internet) : SUCCESS : Traceroute SUCCESS
Master -> Colo Worker 1 (VPN) : SUCCESS : Traceroute SUCCESS
Colo Worker 1 -> Master (Internet) : SUCCESS : Traceroute FAIL (too many hops)
Colo Worker 1 -> Master (VPN) : SUCCESS : Traceroute SUCCESS
Update 5
After running the tests above, it got me thinking about routing and I wondered if it was as simple as providing a route to the controller over the VPN for the service CIDR range (10.96.0.0/12).
So on a host, not included in the cluster, I added a route thus:
route add -net 10.96.0.0/12 gw 192.168.191.100
And I could then resolve DNS using the kube-dns server address:
nslookup www.google.co.uk 10.96.0.10
SO I then added a route, as above, to one of the worker nodes and tried the same. But it is blocked and I do not get a response.
Given that I can resolve DNS over the VPN with the appropriate route from a non-kubernetes machine, I can only think that there is an IPTables rule that needs updating or adding.
I think this is almost there, just one last bit to fix.
I realise this is wrong as it it the kube-proxy should do the DNS resolution on each host. I am leaving it here for information.
Following the instruction at this page, try to run this:
apiVersion: v1
kind: Pod
metadata:
namespace: default
name: dns-example
spec:
containers:
- name: test
image: nginx
dnsPolicy: "None"
dnsConfig:
nameservers:
- 1.2.3.4
searches:
- ns1.svc.cluster.local
- my.dns.search.suffix
options:
- name: ndots
value: "2"
- name: edns0
and see if a manual configuration works or you have some networking DNS problem.
Sounds like you are running on AWS. I suspect that your AWS security group is not allowing DNS traffic to go through. You can try allowing all traffic to the Security Group(s) where all your master and nodes are, to see if that's the problem.
You can also check that all your masters and nodes are allowing routing:
cat /proc/sys/net/ipv4/ip_forward
If not
echo 1 > /proc/sys/net/ipv4/ip_forward
Hope it helps.