Kubernetes coredns is not reachable from the pod - kubernetes

Question

I have a pod deployed named 'sample_pod' in rancher cluster having a container named 'sample_container'. The sample pod has a service named 'test'. Inside the sample_container, if I try to resolve the cluster domain names using 'host' or 'dig' or 'nslookup' command, I am always getting connection refused; no servers could be reached.
I have coredns pods running inside my cluster
user#abc$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-7fbff695b4-f7vxc 1/1 Running 0 21h
canal-928m6 2/2 Running 0 21h
canal-d7vjr 2/2 Running 0 20h
coredns-6f85d5fb88-9txmx 1/1 Running 0 21h
coredns-autoscaler-79599b9dc6-ndgfj 1/1 Running 0 21h
kube-multus-ds-769n6 1/1 Running 0 20h
metrics-server-8449844bf-jz66w 1/1 Running 0 21h
rke-coredns-addon-deploy-job-dlvlh 0/1 Completed 0 21h
rke-ingress-controller-deploy-job-jcj6w 0/1 Completed 0 21h
rke-metrics-addon-deploy-job-wnhbq 0/1 Completed 0 21h
rke-network-plugin-deploy-job-wzqfb 0/1 Completed 0 21h
whereabouts-p6vcc 1/1 Running 0 20h
I am not touching the default Corefile of coredns
Corefile:
.:53 {
log
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . "/etc/resolv.conf"
cache 30
loop
reload
loadbalance
}
/etc/hosts file of sample_container:
[root#sample_container]# cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.42.1.18 sample_pod
# Entries added by HostAliases.
127.0.0.1 localhost
10.94.66.8 netboot.com
/etc/resolv.conf of sample_container:
[root#sample_container]# cat /etc/resolv.conf
nameserver 10.43.0.10
search default.svc.cluster.local svc.cluster.local cluster.local openstacklocal
options ndots:5
Host or dig command I used to resolve following domains and got the error:
[root#sample_container]# ping 10.43.0.10
PING 10.43.0.10 (10.43.0.10) 56(84) bytes of data.
^C
--- 10.43.0.10 ping statistics ---
99 packets transmitted, 0 received, 100% packet loss, time 98003ms
[root#sample_container]# host kube-dns.kube-system
;; connection timed out; no servers could be reached
[root#sample_container]# host localhost
;; connection timed out; no servers could be reached
I tried to resolve test service in the default namespace (where sample_container, sample_pod resides in same namespace)
[root#sample_container]# host test
;; connection timed out; no servers could be reached
dig or nslookup command also returns same
[root#sample_container]# nslookup localhost
;; connection timed out; no servers could be reached
[root#sample_container]# dig localhost
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.8 <<>> localhost
;; global options: +cmd
;; connection timed out; no servers could be reached
Additional information on pod ip and service ip:
root#user$ kubectl get all -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/sample_pod 1/1 Running 0 177m 10.42.1.18 dsc-worker-node <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/test ClusterIP 10.43.19.85 <none> 80/TCP,443/TCP 177m role=test
Note: I deployed this pod such a way that some containers will access the baremetal machine to serve its purpose. And I need to achieve forwarding certain domain names to that baremetal server which will reply for that dns query. Also I am aware of forward plugin which does this job. But without touching the Corefile, I am unable to reach coredns for cluster domain names itself.
Could someone help me to solve this issue? It would be really helpful for me. Thanks in advance!!!

Answer 1

I solved this issue after changing the route. By default, the dns queries are sent to kubernetes nameserver via private interface instead of sending via default gateway (public interface). After changing the route to make dns queries to be sent via default gateway, it was solved.