Kubernetes.default nslookup not able to resolve from different namespaces - kubernetes

Question

I'm facing a problem resolving kubernetes.default.svc.cluster.local from outside default namespace
I'm running two busybox:1.30 pods on each namespace and the name successfully resolves from the default namespace only
[admin#devsvr3 ~]$ kubectl exec -n default -ti busybox -- nslookup kubernetes
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
[admin#devsvr3 ~]$ kubectl exec -n namespace-dev -ti busybox -- nslookup kubernetes
Server: 10.96.0.10
Address: 10.96.0.10:53
** server can't find kubernetes.namespace-dev.svc.cluster.local: NXDOMAIN
*** Can't find kubernetes.svc.cluster.local: No answer
*** Can't find kubernetes.cluster.local: No answer
*** Can't find kubernetes.namespace-dev.svc.cluster.local: No answer
*** Can't find kubernetes.svc.cluster.local: No answer
*** Can't find kubernetes.cluster.local: No answer
[admin#devsvr3 ~]$
I'm running CentOS 7 kubernetes cluster on an air-gaped environment and using weave net CNI add-on and this is my CoreDNS config
apiVersion: v1
data:
Corefile: |
.:53 {
log
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
proxy . /etc/resolv.conf
cache 30
reload
loadbalance
}
kind: ConfigMap
metadata:
creationTimestamp: "2019-01-28T10:59:25Z"
name: coredns
namespace: kube-system
resourceVersion: "1177652"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: c6b5ddae-22eb-11e9-8689-0017a4770068

Answer 1

Following your steps indeed I approached the same issue. But if you create the pod using this yaml it works correctly. Changing the busybox image seems to end up with your described error. Will try to find out why. But for now this is the solution.
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: namespace-dev
spec:
containers:
- name: busybox
image: busybox:1.28
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always
and then:
kubectl exec -ti -n=namespace-dev busybox -- nslookup kubernetes.default
it works as intended and explained here.
/ # nslookup kubernetes
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
nslookup: can't resolve 'kubernetes'
/ # nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local