Kube-dns service discovery cannot discover port number of service - kubernetes

Question

I am using DNS based service discovery to discover services in K8s cluster.
From this link it is very clear that to discover a service named my-service we can do name lookup "my-service.my-ns" and the pod should be able to find the services.
However in case of port discovery for the service the solution is to use
is "_http._tcp.my-service.my-ns"
where
_http refers to the port named http in my-service.
But even after using _http._tcp.my-service it doesn't resolve port number. Below are the details.
my-service which needs to be discovered
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: my-service
ports:
- name: http
protocol: TCP
port: 5000
targetPort: 5000
client-service yaml snippet trying to discover my-service and its port.
spec:
containers:
- name: client-service
image: client-service
imagePullPolicy: Always
ports:
- containerPort: 7799
resources:
limits:
cpu: "100m"
memory: "500Mi"
env:
- name: HOST
value: my-service
- name: PORT
value: _http._tcp.my-service
Now when I make a request it fails and logs following request which is clearly incorrect as it doesn't discover port number.
http://my-service:_http._tcp.my-service
I am not sure what wrong I am doing here, but I am following the same instructions mentioned in document.
Can somebody suggest what is wrong here and how we can discover port using DNS based service discovery? Is my understanding wrong here that it will return the literal value of port?
Cluster details
K8s cluster version is 1.11.5-gke.5
and Kube-dns is running
Additional details trying to discover service from busybox and its not able to discover port value 5000
kubectl exec busybox -- nslookup my-service
Server: 10.51.240.10
Address: 10.51.240.10:53
Name: my-service.default.svc.cluster.local
Address: 10.51.253.236
*** Can't find my-service.svc.cluster.local: No answer
*** Can't find my-service.cluster.local: No answer
*** Can't find my-service.us-east4-a.c.gdic-infinity-dev.internal: No answer
*** Can't find my-service.c.gdic-infinity-dev.internal: No answer
*** Can't find my-service.google.internal: No answer
*** Can't find my-service.default.svc.cluster.local: No answer
*** Can't find my-service.svc.cluster.local: No answer
*** Can't find my-service.cluster.local: No answer
*** Can't find my-service.us-east4-a.c.gdic-infinity-dev.internal: No answer
*** Can't find my-service.c.gdic-infinity-dev.internal: No answer
*** Can't find my-service.google.internal: No answer
kubectl exec busybox -- nslookup _http._tcp.my-service
Server: 10.51.240.10
Address: 10.51.240.10:53
** server can't find _http._tcp.my-service: NXDOMAIN
*** Can't find _http._tcp.my-service: No answer

Answer 1

Since Services come with their own (Kubernetes-internal) IP addresses, the easy answer here is to not pick arbitrary ports for Services. Change to port: 80 in your Service definition, and clients will be able to reach it using the default HTTP port. When you set the environment variable, set
- name: PORT
value: "80"
DNS supports several different record types; for example, an A record translates a host name to its IPv4 address, and AAAA to an IPv6 address. The Kubernetes Service documentation you cite notes (emphasis mine)
you can do a DNS SRV query ... to discover the port number for "http".
While SRV records seems like they solve both halves of this problem (they provide a port and a host name for a service) in practice they seem to get fairly little use. The linked Wikipedia page has a list of services that use it, but "connect to the thing this SRV record points at" isn't an option in mainstream TCP clients that I know of.
You should be able to verify this with a command like (running this debugging image)
kubectl run debug --rm -it --image giantswarm/tiny-tools sh
# dig -t srv _http._tcp.my-service
(But notice the -t srv argument; it is not the default record type.)
Most things that expect a PORT environment variable or similar expect a number, or if not, a name they can find in an /etc/services file. The syntax you're trying to use here and trying to provide a DNS SRV name instead probably just won't work, unless you know the specific software supports it.