I have a 4 node Kubernetes cluster, 1 x controller and 3 x workers. The following shows how they are configured with the versions.
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-ctrl-1 Ready master 1h v1.11.2 192.168.191.100 <none> Ubuntu 18.04.1 LTS 4.15.0-1021-aws docker://18.6.1
turtle-host-01 Ready <none> 1h v1.11.2 192.168.191.53 <none> Ubuntu 18.04.1 LTS 4.15.0-29-generic docker://18.6.1
turtle-host-02 Ready <none> 1h v1.11.2 192.168.191.2 <none> Ubuntu 18.04.1 LTS 4.15.0-34-generic docker://18.6.1
turtle-host-03 Ready <none> 1h v1.11.2 192.168.191.3 <none> Ubuntu 18.04.1 LTS 4.15.0-33-generic docker://18.6.1
Each of the nodes has two network interfaces, for arguments sake eth0 and eth1. eth1 is the network that I want to the cluster to work on. I setup the controller using kubeadm init and passed --api-advertise-address 192.168.191.100. The worker nodes where then joined using this address.
Finally on each node I modified the kubelet service to have --node-ip set so that the layout looks as above.
The cluster appears to be working correctly and I can create pods, deployments etc. However the issue I have is that none of the pods are able to use the kube-dns service for DNS resolution.
This is not a problem with resolution, rather that the machines cannot connect to the DNS service to perform the resolution. For example if I run a busybox container and access it to perform nslookup i get the following:
/ # nslookup www.google.co.uk
nslookup: read: Connection refused
nslookup: write to '10.96.0.10': Connection refused
I have a feeling that this is down to not using the default network and because of that I suspect some Iptables rules are not correct, that being said these are just guesses.
I have tried both the Flannel overlay and now Weave net. The pod CIDR range is 10.32.0.0/16 and the service CIDR is as default.
I have noticed that with Kubernetes 1.11 there are now pods called coredns rather than one kube-dns.
I hope that this is a good place to ask this question. I am sure I am missing something small but vital so if anyone has any ideas that would be most welcome.
Update #1:
I should have said that the nodes are not all in the same place. I have a VPN running between them all and this is the network I want things to communicate over. It is an idea I had to try and have distributed nodes.
Update #2:
I saw another answer on SO (DNS in Kubernetes not working) that suggested kubelet needed to have --cluster-dns and --cluster-domain set. This is indeed the case on my DEV K8s cluster that I have running at home (on one network).
However it is not the case on this cluster and I suspect this is down to a later version. I did add the two settings to all nodes in the cluster, but it did not make things work.
Update #3
The topology of the cluster is as follows.
1 x Controller is in AWS
1 x Worker is in Azure
2 x Worker are physical machines in a colo Data Centre
All machines are connected to each other using ZeroTier VPN on the 192.168.191.0/24 network.
I have not configured any special routing. I agree that this is probably where the issue is, but I am not 100% sure what this routing should be.
WRT to kube-dns and nginx, I have not tainted my controller so nginx is not on the master, not is busybox. nginx and busybox are on workers 1 and 2 respectively.
I have used netcat to test connection to kube-dns and I get the following:
/ # nc -vv 10.96.0.10 53
nc: 10.96.0.10 (10.96.0.10:53): Connection refused
sent 0, rcvd 0
/ # nc -uvv 10.96.0.10 53
10.96.0.10 (10.96.0.10:53) open
The UDP connection does not complete.
I modified my setup so that I could run containers on the controller, so kube-dns, nginx and busybox are all on the controller, and I am able to connect and resolve DNS queries against 10.96.0.10.
So all this does point to routing or IPTables IMHO, I just need to work out what that should be.
Update #4
In response to comments I can confirm the following ping test results.
Master -> Azure Worker (Internet) : SUCCESS : Traceroute SUCCESS
Master -> Azure Worker (VPN) : SUCCESS : Traceroute SUCCESS
Azure Worker -> Master (Internet) : SUCCESS : Traceroute FAIL (too many hops)
Azure Worker -> Master (VPN) : SUCCESS : Traceroute SUCCESS
Master -> Colo Worker 1 (Internet) : SUCCESS : Traceroute SUCCESS
Master -> Colo Worker 1 (VPN) : SUCCESS : Traceroute SUCCESS
Colo Worker 1 -> Master (Internet) : SUCCESS : Traceroute FAIL (too many hops)
Colo Worker 1 -> Master (VPN) : SUCCESS : Traceroute SUCCESS
Update 5
After running the tests above, it got me thinking about routing and I wondered if it was as simple as providing a route to the controller over the VPN for the service CIDR range (10.96.0.0/12).
So on a host, not included in the cluster, I added a route thus:
route add -net 10.96.0.0/12 gw 192.168.191.100
And I could then resolve DNS using the kube-dns server address:
nslookup www.google.co.uk 10.96.0.10
SO I then added a route, as above, to one of the worker nodes and tried the same. But it is blocked and I do not get a response.
Given that I can resolve DNS over the VPN with the appropriate route from a non-kubernetes machine, I can only think that there is an IPTables rule that needs updating or adding.
I think this is almost there, just one last bit to fix.
I realise this is wrong as it it the kube-proxy should do the DNS resolution on each host. I am leaving it here for information.
Following the instruction at this page, try to run this:
apiVersion: v1
kind: Pod
metadata:
namespace: default
name: dns-example
spec:
containers:
- name: test
image: nginx
dnsPolicy: "None"
dnsConfig:
nameservers:
- 1.2.3.4
searches:
- ns1.svc.cluster.local
- my.dns.search.suffix
options:
- name: ndots
value: "2"
- name: edns0
and see if a manual configuration works or you have some networking DNS problem.
Sounds like you are running on AWS. I suspect that your AWS security group is not allowing DNS traffic to go through. You can try allowing all traffic to the Security Group(s) where all your master and nodes are, to see if that's the problem.
You can also check that all your masters and nodes are allowing routing:
cat /proc/sys/net/ipv4/ip_forward
If not
echo 1 > /proc/sys/net/ipv4/ip_forward
Hope it helps.
Related
I dig everywhere to see why we don't have DNS resolution for static pods and couldn't find a right answer. Most basic stuff and couldn't find appealing answer.
Like you create a static pod, exec into it, do a "nslookup pod-name" or like "nslookup 10-44-0-7.default.pod.cluster.local", I know the second one is for Deployment and DaemonSet which creates A record, why not for static pods because they are ephemeral, in that way Deployment also is. Please tell me if it is possible and how we enable it.
My testing for the failed queries, all are static pods created with "kubectl run busybox-2 --image=busybox --command sleep 1d"
Used this as syntax:
In general a pod has the following DNS resolution: pod-ip-address.my-namespace.pod.cluster-domain.example.
vagrant#kubemaster:~$ kubectl get pods -n default -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox-1 1/1 Running 0 135m 10.36.0.6 kubenode02 <none> <none>
busybox-2 1/1 Running 0 134m 10.44.0.7 kubenode01 <none> <none>
busybox-sleep 1/1 Running 19 (24h ago) 23d 10.44.0.3 kubenode01 <none> <none>
/ # cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local Dlink
options ndots:5
/ # nslookup 10-44-0-7.default.pod.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10:53
Name: 10-44-0-7.default.pod.cluster.local
Address: 10.44.0.7
*** Can't find 10-44-0-7.default.pod.cluster.local: No answer
/ # nslookup 10-44-0-6.default.pod.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10:53
*** Can't find 10-44-0-6.default.pod.cluster.local: No answer
Appreciate the help.
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:
Katacoda
Play with Kubernetes
Your cluster must be running the CoreDNS add-on. Migrating to CoreDNS explains how to use kubeadm to migrate from kube-dns.
Your Kubernetes server must be at or later than version v1.12. To check the version, enter kubectl version.
DNS Lookup & Reverse lookups does not work for pods/podIPs (By Design!).
Why ?
I also had the similar question , After spending a lot of time exploring following are reasons that convinced me :
Pods & and its IPs are ephemeral. even static pods when they get restarted(recreated) they might end up getting a different IP Address.
It will be huge overload on coredns/any dns server to keep a track of ever changing POD IP Addresses.
Due to above reasons It is recommended to access the POD through a service because service will have a constant IP irrespective of how the endpoint IPS have changed. for services DNS lookups & reverse loookups work fine.
I am trying kubernetes and seem to have hit bit of a hurdle. The problem is that from within my pod I can't curl local hostnames such as wrkr1 or wrkr2 (machine hostnames on my network) but can successfully resolve hostnames such as google.com or stackoverflow.com.
My cluster is a basic setup with one master and 2 worker nodes.
What works from within the pod:
curl to google.com from pod -- works
curl to another service(kubernetes) from pod -- works
curl to another machine on same LAN via its IP address such as 192.168.x.x -- works
curl to another machine on same LAN via its hostname such as wrkr1 -- does not work
What works from the node hosting pod:
curl to google.com --works
curl to another machine on same LAN via
its IP address such as 192.168.x.x -- works
curl to another machine
on same LAN via its hostname such as wrkr1 -- works.
Note: the pod cidr is completely different from the IP range used in
LAN
the node contains a hosts file with entry corresponding to wrkr1's IP address (although I've checked node is able to resolve hostname without it also but I read somewhere that a pod inherits its nodes DNS resolution so I've kept the entry)
Kubernetes Version: 1.19.14
Ubuntu Version: 18.04 LTS
Need help as to whether this is normal behavior and what can be done if I want pod to be able to resolve hostnames on local LAN as well?
What happens
Need help as to whether this is normal behavior
This is normal behaviour, because there's no DNS server in your network where virtual machines are hosted and kubernetes has its own DNS server inside the cluster, it simply doesn't know about what happens on your host, especially in /etc/hosts because pods simply don't have access to this file.
I read somewhere that a pod inherits its nodes DNS resolution so I've
kept the entry
This is a point where tricky thing happens. There are four available DNS policies which are applied per pod. We will take a look at two of them which are usually used:
"Default": The Pod inherits the name resolution configuration from the node that the pods run on. See related discussion for more details.
"ClusterFirst": Any DNS query that does not match the configured cluster domain suffix, such as "www.kubernetes.io", is forwarded to the upstream nameserver inherited from the node. Cluster administrators may have extra stub-domain and upstream DNS servers configured
The trickiest ever part is this (from the same link above):
Note: "Default" is not the default DNS policy. If dnsPolicy is not
explicitly specified, then "ClusterFirst" is used.
That means that all pods that do not have DNS policy set will be run with ClusterFirst and they won't be able to see /etc/resolv.conf on the host. I tried changing this to Default and indeed, it can resolve everything host can, however internal resolving stops working, so it's not an option.
For example coredns deployment is run with Default dnsPolicy which allows coredns to resolve hosts.
How this can be resolved
1. Add local domain to coreDNS
This will require to add A records per host. Here's a part from edited coredns configmap:
This should be within .:53 { block
file /etc/coredns/local.record local
This part is right after block above ends (SOA information was taken from the example, it doesn't make any difference here):
local.record: |
local. IN SOA sns.dns.icann.org. noc.dns.icann.org. 2015082541 7200 3600 1209600 3600
wrkr1. IN A 172.10.10.10
wrkr2. IN A 172.11.11.11
Then coreDNS deployment should be added to include this file:
$ kubectl edit deploy coredns -n kube-system
volumes:
- configMap:
defaultMode: 420
items:
- key: Corefile
path: Corefile
- key: local.record # 1st line to add
path: local.record # 2nd line to add
name: coredns
And restart coreDNS deployment:
$ kubectl rollout restart deploy coredns -n kube-system
Just in case check if coredns pods are running and ready:
$ kubectl get pods -A | grep coredns
kube-system coredns-6ddbbfd76-mk2wv 1/1 Running 0 4h46m
kube-system coredns-6ddbbfd76-ngrmq 1/1 Running 0 4h46m
If everything's done correctly, now newly created pods will be able to resolve hosts by their names. Please find an example in coredns documentation
2. Set up DNS server in the network
While avahi looks similar to DNS server, it does not act like a DNS server. It's not possible to setup requests forwarding from coredns to avahi, while it's possible to proper DNS server in the network and this way have everything will be resolved.
3. Deploy avahi to kubernetes cluster
There's a ready image with avahi here. If it's deployed into the cluster with dnsPolicy set to ClusterFirstWithHostNet and most importantly hostNetwork: true it will be able to use host adapter to discover all available hosts within the network.
Useful links:
Pods DNS policy
Custom DNS entries for kubernetes
I have Kubernetes cluster on Ubuntu 16.04 servers, deployed using Kubespray.
Kube-dns pod is restarting continuously on master node. It has restarted 3454 times.
Can anyone let me know how to troubleshoot and solve this issue?
Starting logs of kube-dns:
# 1, # 2
k8s-cluster.yml
#1 #2
SkyDNS by default forwards the nameservers to the ones listed in the /etc/resolv.conf. Since SkyDNS runs inside the kube-dns pod as a cluster addon, it inherits the configuration of /etc/resolv.conf from its host as described in the kube-dns documentations.
From your error, it looks like your host's /etc/resolv.conf is configured to use 10.233.100.1 as its nameserver and that becomes the forwarding server in your SkyDNS config. Looks like 10.233.100.1 is not routable from your Kubernetes cluster and this is the reason why you are getting the error:
skydns: failure to forward request "read udp 10.233.100.1:40155->ourdnsserverIP:53: i/o timeout"
The solution would be to change the flag --nameservers in the SkyDNS configuration. I said change because currently you have it set to "" and you might change it to nameservers=8.8.8.8:53,8.8.4.4:53.
I have a kubernetes master and node setup in two centos VMs on my Win 10.
I used flannel for CNI and deployed ambassador as an API gateway.
As the ambassador routes did not work, I analysed further to understand that the DNS (ip-10.96.0.10) is not accessible from busybox pod which means that none of the service names can be accessed. Could I get any suggestion please.
1. You should use newest version of Flannel.
Flannel does not setup service IPs but kube-proxy does, you should look at kube-proxy on your nodes and ensure they are not reporting errors.
I'd suggest taking a look at https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#tabs-pod-install-4 and ensure you have met the requirements stated there.
Similar issue but with Calico plugin you can find here: https://github.com/projectcalico/calico/issues/1798
2. Check if you have open port 8285, flannel uses UDP port 8285 for sending encapsulated IP packets. Make sure to enable this traffic to pass between the hosts.
3. Ambassador includes an integrated diagnostics service to help with troubleshooting, this may be useful for you. By default, this is not exposed to the Internet. To view it, we'll need to get the name of one of the Ambassador pods:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
ambassador-3655608000-43x86 1/1 Running 0 2m
ambassador-3655608000-w63zf 1/1 Running 0 2m
Forwarding local port 8877 to one of the pods:
kubectl port-forward ambassador-3655608000-43x86 8877
will then let us view the diagnostics at http://localhost:8877/ambassador/v0/diag/.
First spot should solve your problem, if not, try remainings.
I hope this helps.
I am trying to setup own single-node kubernetes cluster on bare metal dedicated server. I am not that experienced in dev-ops but I need some service to be deployed for my own project. I already did a cluster setup with juju and conjure-up kubernetes over LXD. I have running cluster pretty fine.
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
conjure-canonical-kubern-3b3 conjure-up-localhost-db9 localhost/localhost 2.4.3 unsupported 23:49:09Z
App Version Status Scale Charm Store Rev OS Notes
easyrsa 3.0.1 active 1 easyrsa jujucharms 195 ubuntu
etcd 3.2.10 active 3 etcd jujucharms 338 ubuntu
flannel 0.10.0 active 2 flannel jujucharms 351 ubuntu
kubeapi-load-balancer 1.14.0 active 1 kubeapi-load-balancer jujucharms 525 ubuntu exposed
kubernetes-master 1.13.1 active 1 kubernetes-master jujucharms 542 ubuntu
kubernetes-worker 1.13.1 active 1 kubernetes-worker jujucharms 398 ubuntu exposed
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 10.213.117.66 Certificate Authority connected.
etcd/0* active idle 1 10.213.117.171 2379/tcp Healthy with 3 known peers
etcd/1 active idle 2 10.213.117.10 2379/tcp Healthy with 3 known peers
etcd/2 active idle 3 10.213.117.238 2379/tcp Healthy with 3 known peers
kubeapi-load-balancer/0* active idle 4 10.213.117.123 443/tcp Loadbalancer ready.
kubernetes-master/0* active idle 5 10.213.117.172 6443/tcp Kubernetes master running.
flannel/1* active idle 10.213.117.172 Flannel subnet 10.1.83.1/24
kubernetes-worker/0* active idle 7 10.213.117.136 80/tcp,443/tcp Kubernetes worker running.
flannel/4 active idle 10.213.117.136 Flannel subnet 10.1.27.1/24
Entity Meter status Message
model amber user verification pending
Machine State DNS Inst id Series AZ Message
0 started 10.213.117.66 juju-b03445-0 bionic Running
1 started 10.213.117.171 juju-b03445-1 bionic Running
2 started 10.213.117.10 juju-b03445-2 bionic Running
3 started 10.213.117.238 juju-b03445-3 bionic Running
4 started 10.213.117.123 juju-b03445-4 bionic Running
5 started 10.213.117.172 juju-b03445-5 bionic Running
7 started 10.213.117.136 juju-b03445-7 bionic Running
I also deployed Hello world application to output some hello on port 8080 inside the pod and nginx-ingress for it to re-route the traffic to this service on specified host.
NAME READY STATUS RESTARTS AGE
pod/hello-world-696b6b59bd-fznwr 1/1 Running 1 176m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/example-service NodePort 10.152.183.53 <none> 8080:30450/TCP 176m
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 10h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello-world 1/1 1 1 176m
NAME DESIRED CURRENT READY AGE
replicaset.apps/hello-world-696b6b59bd 1 1 1 176m
When I do curl localhost as expected I have connection refused, which looks still fine as it's not exposed to cluster. when I curl the kubernetes-worker/0 with public address 10.213.117.136 on port 30450 (which I get from kubectl get all)
$ curl 10.213.117.136:30450
Hello Kubernetes!
Everything works like a charm (which is obvious). When I do
curl -H "Host: testhost.com" 10.213.117.136
Hello Kubernetes!
It works again like charm! That means ingress controller is successfully routing port 80 based on host rule to correct services. At this point I am 100% sure that cluster works as it should.
Now I am trying to access this service over the internet externally. When I load <server_ip> obviously nothing loads as it's living inside own lxd subnet. Therefore I was thinking forward port 80 from server eth0 to this IP. So I added this rule to iptables
sudo iptables -t nat -A PREROUTING -p tcp -j DNAT --to-destination 10.213.117.136 (For the sake of example let's route everything not only port 80). Now when I open on my computer http://<server_ip> it loads!
So the real question is how to do that on production? Should I setup this forwarding rule in iptables? Is that normal approach or hacky solution and there is something "standard" which I am missing? The thing is to add this rule with static worker node will make the cluster completely static. IP eventually change, I can remove/add units to workers and it will stop working. I was thinking about writing script which will obtain this IP address from juju like this:
$ juju status kubernetes-worker/0 --format=json | jq '.machines["7"]."dns-name"'
"10.213.117.136"
and add it to IP-tables, which is more okay-ish solution than hardcoded IP but still I feel it's a tricky and there must be a better way.
As last idea I get to run HAProxy outside of the cluster, directly on the machine and just do forwarding of traffic to all available workers. This might eventually also work. But still I don't know the answer what is the correct solution and what is usually used in this case. Thank you!
So the real question is how to do that on production?
The normal way to do this in a production system is to use a Service.
The simplest case is when you just want your application to be accessible from outside on your node(s). In that case you can use a Type NodePort Service. This would create the iptables rules necessary to forward the traffic from the host IP address to the pod(s) providing the service.
If you have a single node (which is not recommended in production!), you're ready at this point.
If you have multiple nodes in your Kubernetes cluster, all of them would be configured by Kubernetes to provide access to the service (your clients could use any of them to access the service). Though, you'd have to solve the problem of how the clients would know which nodes are available to be contacted...
There are several ways to handle this:
use a protocol understood by the client to publish the currently available IP addresses (for example DNS),
use a floating (failover, virtual, HA) IP address managed by some software on your Kubernetes nodes (for example pacemaker/corosync), and direct the clients to this address,
use an external load-balancer, configured separately, to forward traffic to some of the operating nodes,
use an external load-balancer, configured automatically by Kubernetes using a cloud provider integration script (by using a Type LoadBalancer Service), to forward traffic to some of the operating nodes.