I have two separate clusters (Application and DB) in the same namespace. Statefulset for DB cluster and Deployment for Application cluster. For internal communication I have configured a Headless Service. When I ping from a pod in application cluster to the service it works (Works the other way round too - DB pod to service works). But sometimes, for example if I continuously execute ping command for like 3 times, the third time it gives an error - "ping: : Temporary failure in name resolution". Why is this happening?
As far as I know this is usually a name resolution error and shows that your DNS server cannot resolve the domain names into their respective IP addresses. This can present a grave challenge as you will not be able to update, upgrade, or even install any software packages on your Linux system. Here I have listed few reasons
1.Forgot configuring or Wrongly Configured resolv.conf File
The /etc/resolv.conf file is the resolver configuration file in Linux systems. It contains the DNS entries that help your Linux system to resolve domain names into IP addresses.
If this file is not present or is there but you are still having the name resolution error, create one and append the Google public DNS server as nameserver 8.8.8.8
Save the changes and restart the systemd-resolved service as shown.
$ sudo systemctl restart systemd-resolved.service
It’s also prudent to check the status of the resolver and ensure that it is active and running as expected:
$ sudo systemctl status systemd-resolved.service
2. Due to Firewall Restrictions
By some chance if the first solution did not work for you, firewall restrictions could be preventing you from successfully performing DNS queries. Check your firewall and confirm if port 53 (used for DNS – Domain Name Resolution ) and port 43 are open. If the ports are blocked, open them as follows:
For UFW firewall (Ubuntu / Debian and Mint)
To open ports 53 & 43 on the UFW firewall run the commands below:
$ sudo ufw allow 43/tcp
$ sudo ufw reload```
For firewalld (RHEL / CentOS / Fedora)
For Redhat based systems such as CentOS, invoke the commands below:
```$ sudo firewall-cmd --add-port=53/tcp --permanent
$ sudo firewall-cmd --add-port=43/tcp --permanent
$ sudo firewall-cmd --reload
I hope that you now have an idea about the ‘temporary failure in name resolution‘ error. I also found a similar git issue hope that helps
https://github.com/kubernetes/kubernetes/issues/6667
Related
I'm currently using minikube and I'm trying to access my application by utilizing the minikube tunnel since the service type is LoadBalancer.
I'm able to obtain an external IP when I execute the minikube tunnel, however, when I try to check it on the browser it doesn't work. I've also tried Postman and curl, they both don't work.
To add to this, if I shell into the pod I can use curl and it does work. Furthermore, I executed kubectl port-forward and I was able to access my application through localhost.
Does anyone have any idea as to why I'm not being able to access my application even though everything seems to be running correctly?
Your service is probably bound to localhost. Minikube starts the cluster in a VM or docker (depending on the driver you are using) that is bound to an external IP, $(minikube ip).
When you are running a minikube tunnel you're tunneling from minikube cluster external IP to the internal IP of the load balancer, the LB service in Kubernete the External IP goes from "Pending" to an actual internal IP and something like this should work:
curl -H 'Host: localhost' -v $(minikube ip)
However, it doesn't in the browser, since in the above command you are sending the request to the minikube's IP, not localhost. What I do for this to work is a ssh tunnel like this one:
ssh -i $(minikube ssh-key) docker#$(minikube ip) -L 8008:localhost:80
This maps the LB listener in port 80, in minikube's cluster, to 8008 in localhost. The external IP of the service remains pending but it works since the Kube controller can still find it. If you want to map port 80 then you will need to add sudo.
If the version of ssh on your system (the one in your path) is less than 8.0, 'minikube tunnel' will silently fail to instantiate the ssh tunnel for some port forwards. (e.g. privileged ports)
Open a command prompt as administrator, and type 'where.exe ssh'. Navigate to that location in windows explorer, and right-click on 'ssh.exe'. Choose Properties->Details to see the version.
If this is less than version 8.0 you must upgrade that to at least version 8.0 to prevent this silent failure of ssh by 'minikube tunnel'.
After upgrading, ssh, ensure that the newer version is the one that will be executed by using the 'where.exe' command again. If there are two on your system, then reorder the paths in your path environment variable. Restart your shell (or better) reboot the system so that all processes environments pick up the path changes.
Then try 'minikube tunnel' again. When it is working, you should see an ssh instance in the task manager for each tunnel that minikube creates.
In my case minikube service <serviceName> solved this issue.
For further details look here in minikube docs.
I have enabled all required ports. When i enable the firewalld service then the core-dns doesnt resolve any domain-name with command $ kubectl exec -ti busybox -- nslookup kubernetes.default
This seems to be a know case, which you can find on GitHub Fresh deploy with CoreDNS not resolving any dns lookup #1056.
There seems to be few solutions which would mean different problems.
One being:
sudo systemctl stop firewalld
sudo systemctl stop firewalld
Please remember this is not recommended.
Another solution might be:
Adding iptables -p FORWARD ACCEPT.
Also check if core dns daemon controller has enough resources, as this might be causing restarts.
You need to provide more details regarding your cluster so we can pinpoint the issue.
This problem may originate due to forwarding packets between interfaces. There are two options:
First, for sessions, I also recommend this for testing:
$ vim /proc/sys/net/ipv4/ip_forward
# set to 1
For a more permanent solution:
$ vim /etc/sysctl.conf
# ADD net.ipv4.ip_forward=1
$ sudo /sbin/sysctl -p
I recently installed kubernetes on VMware and also configured few pods , while configuring those pods , it automatically used IP of the VMware and configured. I was able to access the application during that time but then recently i rebooted VM and machine which hosts the VM, during this - IP of the VM got changed i guess and now - I am getting below error when using command kubectl get pod -n <namspaceName>:
userX#ubuntu:~$ kubectl get pod -n NameSpaceX
Unable to connect to the server: dial tcp 192.168.214.136:6443: connect: no route to host
userX#ubuntu:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Unable to connect to the server: dial tcp 192.168.214.136:6443: connect: no route to host
kubectl cluster-info as well as other related commands gives same output.
in VMware workstation settings, we are using network adapter which is sharing host IP address setting. We are not sure if it has any impact.
We also tried to add below entry in /etc/hosts , it is not working.
127.0.0.1 localhost \n
192.168.214.136 localhost \n
127.0.1.1 ubuntu
I expect to run the pods back again to access the application.Instead of reinstalling all pods which is time consuming - we are looking for quick workaround so that pods will get back to running state.
If you use minikube sometimes all you need is just to restart minikube.
Run:
minikube start
I encountered the same issue - the problem was that the master node didn't expose port 6443 outside.
Below are the steps I took to fix it.
1 ) Check IP of api-server.
This can be verified via the .kube/config file (under server field) or with: kubectl describe pod/kube-apiserver-<master-node-name> -n kube-system.
2 ) Run curl https://<kube-apiserver-IP>:6443 and see if port 6443 is open.
3 ) If port 6443 you should get something related to the certificate like:
curl: (60) SSL certificate problem: unable to get local issuer certificate
4 ) If port 6443 is not open:
4.A ) SSH into master node.
4.B ) Run sudo firewall-cmd --add-port=6443/tcp --permanent (I'm assuming firewalld is installed).
4.C ) Run sudo firewall-cmd --reload.
4.D ) Run sudo firewall-cmd --list-all and you should see port 6443 is updated:
public
target: default
icmp-block-inversion: no
interfaces:
sources:
services: dhcpv6-client ssh
ports: 6443/tcp <---- Here
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
The common practice is to copy config file to the home directory
sudo cp /etc/kubernetes/admin.conf ~/.kube/config && sudo chown $(id -u):$(id -g) $HOME/.kube/config
Also, make sure that api-server address is valid.
server: https://<master-node-ip>:6443
If not, you can manually edit it using any text editor.
You need to export the admin.conf file as kubeconfig before running the kubectl commands. You may put this as your env variable
export kubeconfig=<path>/admin.conf
after this you should be able to run the kubectl command. I am hoping that your setup of K8S cluster is proper.
Last night I had the exact same error installing Kubernetes using this puppet module: https://forge.puppet.com/puppetlabs/kubernetes
Turns out that it is an incorrect iptables setting in the master that blocks all non-local requests towards the api.
The way I solved it (bruteforce solution) is by
completely remove alle installed k8s related software (also all config files, etcd data, docker images, mounted tmpfs filesystems, ...)
wipe the iptables completely https://serverfault.com/questions/200635/best-way-to-clear-all-iptables-rules
reinstall
This is what solved the problem in my case.
There is probably a much nicer and cleaner way to do this (i.e. simply change the iptables rules to allow access).
if you getting the below error then you also check once the token validity.
Unable to connect to the server: dial tcp 192.168.93.10:6443: connect: no route to host
Check your token validity by using the command kubeadm token list if your token is expired then you have to reset the cluster using kubeadm reset and than initialize again using command kubeadm init --token-ttl 0.
Then again check the status of the token using kubeadm token list. Note here the TTL value will be <forever> and Expire value will be <never>.
example:-
[root#master1 ~]# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
nh48tb.d79ysdsaj8bchms9 <forever> <never> authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token
Ubuntu 22.04 LTS Screenshot
Select docker-desktop and run again your command, e.g kubectl apply -f <myimage.yaml>
Run minikube start command
The reason behind that is your minikube cluster with driver docker stopped
when you shutdown the system
To all those who are trying to learn and experiment kubernetes using Ubuntu on Oracle VM:
IP address is assigned to Guest OS/VM based on the network adapter selection. Based on your network adapter selection, you need to configure the settings in Oracle VM network section or your router settings.
See the link for most common Oracle VM network adapter.
https://www.nakivo.com/blog/virtualbox-network-setting-guide/
I was using bridge adapter which put VM and host OS in parallel. So the my router was randomly assigning IP to my VM after every restart and my cluster stopped working and getting the same exact error message posted in the question.
> k get pods -A
> Unable to connect to the server: dial tcp 192.168.214.136:6443: connect: no route to host
> systemctl status kubelet
> ........
> ........ "Error getting node" err="node \"node\" not found"
Cluster started working again after reserving static IP address to my VM in router settings.(if you are using NAT adapter, you should configure it in VM network settings)
When you are reserving IP address to your VM, make sure to assign the same old IP address which was used for configuring kubelet.
I want to completely purge Minikube so that I could start over as if I installed it for the first time to avoid some configuration clashes. Mailnly to have initial IP 192.168.99.100, unfortunatelly it increases on next minikube start to 192.168.99.101, etc. I've run to delete Minikube:
minikube delete
rm -rf ~/.minikube
rm -rf ~/.kube
I'm running minikube version: v0.31.0 on Ubuntu 18.04 with driver VirtualBox 5.2.18
I found this problem on the Mac with VirtualBox as well. I tried removing the Host Network Manager, but it did not work for me. However, I did find another solution.
After issuing minikube delete, I removed the following file:
/Users/{username}/Library/VirtualBox/HostInterfaceNetworking-vboxnet0-Dhcpd.leases
Starting minikube again reset the address to .100.
File contents:
<?xml version="1.0"?>
<Leases version="1.0">
<Lease mac="08:00:27:66:6a:19" id="01080027666a19" network="0.0.0.0" state="expired">
<Address value="192.168.99.102"/>
<Time issued="1555277299" expiration="1200"/>
</Lease>
<Lease mac="08:00:27:08:03:a3" id="010800270803a3" network="0.0.0.0" state="expired">
<Address value="192.168.99.101"/>
<Time issued="1555276993" expiration="1200"/>
</Lease>
<Lease mac="08:00:27:32:ed:f8" id="0108002732edf8" network="0.0.0.0" state="expired">
<Address value="192.168.99.100"/>
<Time issued="1555276537" expiration="1200"/>
</Lease>
</Leases>
I've recently run into this issue on mpb; tracking down issues with minikube helm and tiller on VirtualBox v6.0.10
Cleanest solution I've found works as expected
#!/bin/sh
function minikube_reset_vbox_dhcp_leases() {
# # Reset Virtualbox DHCP Lease Info
echo "Resetting Virtualbox DHCP Lease Info..."
kill -9 $(ps aux |grep -i "vboxsvc\|vboxnetdhcp" | awk '{print $2}') 2>/dev/null
if [[ -f ~/Library/VirtualBox/HostInterfaceNetworking-vboxnet0-Dhcpd.leases ]] ; then
rm ~/Library/VirtualBox/HostInterfaceNetworking-vboxnet0-Dhcpd.leases
fi
}
minikube_reset_vbox_dhcp_leases
Credit: issues/951
Minikube is used on different platforms, so it might be helpful to add information related to most popular of them.
Minikube is not responsible for assigning IP address to its VM.
If you are starting minikube on Windows or MacOS new VM is created. That VM gets the first available IP address from the pool of hypervisor DHCP service. In brief, DHCP service reserves this IP for the VM for some period of time, usually from 24 hours to 7 days. If in this period the client doesn't refresh DHCP lease and this IP is not available on the network, the IP is considered free and can be offered to another client.
VirtualBox has only basic settings for its DHCP service, you are not allowed to configure lease time or static ip binding. So, you may try to change ip configuration of minikube VM network interface, after VM is created using minikube ssh. Or you can play with VM MAC address right after creation because DHCP offers IP address based on host MAC address.
HyperV uses existed DHCP on local network for shared networks or manually configured DHCP server for internal networks. If you have access to DHCP administration console you can delete the old minikube VM IP binding before starting a new VM using minikube start.
For Linux you can choose two options, you can use virtualbox hypervisor and create VM like it works on Windows or MAC, so DHCP will work like I've mentioned previously, or you can use -vm-driver=none argument and setup Kubernetes cluster inside host environment without VM. In this case your host machine becomes a Kubernetes master node with the same IP configuration.
If you use virtualbox as the vm-driver, you can use this Python script on Linux/Mac to reset the ip to 192.168.99.100:
./minikube_reset
#!/usr/bin/env python3
import subprocess as sp
from sys import platform
import os
if __name__ == "__main__":
print("Resetting Virtualbox DHCP...")
procs = sp.run("ps aux", shell=True, stdout=sp.PIPE)\
.stdout.decode("utf8").lower().split('\n')
pids = [
p.split()[1] for p in procs if 'vboxsvc' in p or 'vboxnetdhcp' in p
]
for pid in pids:
sp.run(['kill', '-9', pid])
cfg_dir = ".config" if platform != 'darwin' else 'Library'
file = f"~/{cfg_dir}/VirtualBox/HostInterfaceNetworking-vboxnet0-Dhcpd.leases"
try:
os.remove(os.path.expanduser(file))
except OSError as e:
pass
If you make the script an executable via chmod +x minikube_reset and put it into your path, you can run:
minikube stop # Stop your running minikube instance.
minikube_reset # Reset the ip.
minikube start # Start new minikube instance with 192.168.99.100.
Your minikube instance should always start with 192.168.99.100 after minikube_reset.
For Ubuntu 18.0.4 you can try
rm -r /home/username/.config/VirtualBox/HostInterfaceNetworking-vboxnet0-Dhcpd.leases
I just got a new router and internet connection.
My phone and my second computer have no problem to connect to internet (wifi or cable). My first computer, where I am running Limux Mint 18.1 Serena, can connect with cable or wifi but I am unable to browser.
I can ping 8.8.8.8 but not google.com.
I do not understand much of DNS but I tried to follow some online tutorials.
I tried to modify the /etc/resolconf directory by adding a resolv.conf file with a new servername but didn't work (even after restarting network-manager). I tried as well to modify the /etc/resolv.conf.d/tail file (as read in other tutorials) by adding a new servername but still not working.
Any suggestion?
Your DNS resolving doesn't work.
Check your firewall for an open port UDP/53 to 8.8.8.8
# iptables -L -n -v
Check with your ISP for DNS server provided. They are probably blocking UDP/53 somewhere on the way out to force you to use internal DNS server.
If you are on DHCP, renew the lease.
Depends on distro (most probably networkmanager, ifdown/ifup combination may work as well). This happens quite often with resolvconfd and dnsmasq. You may need to restart the NetworkManager then
Again - command depends on distro, usually with systemd
# systemctl restart NetworkManager
If you are on static IP and have noone to ask or check their config, you may sniff (tcpdump -nnvv -i <interface> udp port 53).
The problem occurs due to trouble with DNS setting
If you are using linux follow:
Clear this file content (root permission required)
nano /etc/resolv.conf
Add the folling text:
nameserver 1.1.1.1
nameserver 8.8.8.8
Save and Reboot
If using Windows Follow:
Control Panel\Network and Internet\Network and Sharing Center
Select current active connection (wifi/broadband)
Properties > Networking > ipv4 > custom dns > 1.1.1.1 & 8.8.8.8
check Validate setting upon exit > save