DNS lookup not working properly in Kubernetes cluster - kubernetes

Question

I spin up a cluster with minikube then apply this dummy deployment/service
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
run: nginx-label
replicas: 2
template:
metadata:
labels:
run: nginx-label
spec:
containers:
- name: nginx-container
image: nginx
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
labels:
run: nginx-label
spec:
ports:
- port: 1234
targetPort: 80
protocol: TCP
selector:
run: nginx-label
Then I create a dummy curl pod to test internal network with the following
kubectl run curl --image=radial/busyboxplus:curl -i --tty
Inside that curl instance, I'm able to access the nginx with $NGINX_SERVICE_SERVICE_HOST:$NGINX_SERVICE_SERVICE_PORT or nginx-service.default:1234, but not nginx-service:1234, even though those pods belong to the same namespace.
ubuntu:~$ kubectl get pods --namespace=default
NAME READY STATUS RESTARTS AGE
curl-69c656fd45-d7w8t 1/1 Running 1 29m
nginx-deployment-58595d65fc-9ln25 1/1 Running 0 29m
nginx-deployment-58595d65fc-znkqp 1/1 Running 0 29m
Any idea what could cause this? Following is the nslookup result
[ root#curl-69c656fd45-d7w8t:/ ]$ nslookup nginx-service
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: nginx-service
Address 1: 23.202.231.169 a23-202-231-169.deploy.static.akamaitechnologies.com
Address 2: 23.217.138.110 a23-217-138-110.deploy.static.akamaitechnologies.com
[ root#curl-69c656fd45-d7w8t:/ ]$ nslookup nginx-service.default
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: nginx-service.default
Address 1: 10.103.69.73 nginx-service.default.svc.cluster.local
[ root#curl-69c656fd45-d7w8t:/ ]$
Update: here's the content of /etc/resolv.conf
[ root#curl-69c656fd45-d7w8t:/ ]$ cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local attlocal.net
options ndots:5
[ root#curl-69c656fd45-d7w8t:/ ]$

Answer 1

Answering on your question in comment.
Check following lines below the field name in nslookup file, for host nginx-service they are :
Name: nginx-service
Address 1: 23.202.231.169 a23-202-231-169.deploy.static.akamaitechnologies.com
Address 2: 23.217.138.110 a23-217-138-110.deploy.static.akamaitechnologies.com
for nginx-service.default:
Name: nginx-service.default
Address 1: 10.103.69.73 nginx-service.default.svc.cluster.local
The flow is:
Executing command nslookup
Checking entrypoints(line below host with adress and hostname)
Comparing entrypoints and its domains to domain listed in /etc/resolv.conf file if it doesn't match this mean that we cannot reach specific host.
Your nginx-service"hit" a23-202-231-169.deploy.static.akamaitechnologies.com NOTnginx-service.default.svc.cluster.local.
nginx-service.default "hit" nginxservice.default.svc.cluster.local as you said that is why curl is working.
10.96.0.10 is the address of our system's Domain Name Server named kube-dns.kube-system.svc.cluster.local. This is the server your system is configured to use to translate domain names into IP addresses.
Speaking about the domain deploy.static.akamaitechnologies.com
Akamai is a Content Delivery Network used by Symantec (and many other companies). Kubernetes is a cloud computing service used by a large number of companies as well for internet content. These services are essential for providing content when you visit certain websites, and for delivering things like product updates. It is normal to see connections to these sites when you visit certain other websites or have certain products, like Norton, installed. They essentially provide the servers needed to propagate large amounts of data to various regions of the world quickly while balancing internet traffic so that individual server locations are not overloaded..
Hmm, sorry, not really...My question is that, based on the search
directive in /etc/resolv.conf, if nginx-service.default is resolved
to nginx-service.default.svc.cluster.local, so should nginx-service.
Did I miss anything?
Answer:
Keep in mind that:
Look firstly at nslookup
And then find match in /etc/resolv.conf
Note: In case you will do these steps vice versa, it won't work.
Reasons what may be the problem of wrong resolution of domain in nslookup are many - see: dns-debugging . Try to execute command dig nginx-service. Then interpret the output from this file to find the real problem. Because it is obvious that you cannot curl nginx-service (I have explained it above) but why nslookup shows different records for nameservers it is completely different question.
More information you can find here: nslookup, akamai.