service via ingress not reachable, only via nodeport:ip - kubernetes

Good afternoon,
i'd like to ask.
im a "little" bit upset regarding ingress and its traffic flow
i created test nginx deployment with service and ingress. ( in titaniun cloud )
i have no direct connect via browser so im using tunneling to get access via browser abd sock5 proxy in firefox.
deployment:
k describe deployments.apps dpl-nginx
Name: dpl-nginx
Namespace: xxx
CreationTimestamp: Thu, 09 Jun 2022 07:20:48 +0000
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 1
field.cattle.io/publicEndpoints:
[{"port":32506,"protocol":"TCP","serviceName":"xxx:xxx-svc","allNodes":true},{"addresses":["172.xx.xx.117","172.xx.xx.131","172.xx.x...
Selector: app=xxx-nginx
Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=xxx-nginx
Containers:
nginx:
Image: nginx
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/usr/share/nginx/html/ from nginx-index-file (rw)
Volumes:
nginx-index-file:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: index-html-configmap
Optional: false
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: xxx-dpl-nginx-6ff8bcd665 (2/2 replicas created)
Events: <none>
service:
Name: xxx-svc
Namespace: xxx
Labels: <none>
Annotations: field.cattle.io/publicEndpoints: [{"port":32506,"protocol":"TCP","serviceName":"xxx:xxx-svc","allNodes":true}]
Selector: app=xxx-nginx
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.95.33
IPs: 10.43.95.33
Port: http-internal 888/TCP
TargetPort: 80/TCP
NodePort: http-internal 32506/TCP
Endpoints: 10.42.0.178:80,10.42.0.179:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
ingress:
Name: test-ingress
Namespace: xxx
Address: 172.xx.xx.117,172.xx.xx.131,172.xx.xx.132
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
test.xxx.io
/ xxx-svc:888 (10.42.0.178:80,10.42.0.179:80)
Annotations: field.cattle.io/publicEndpoints:
[{"addresses":["172.xx.xx.117","172.xx.xx.131","172.xx.xx.132"],"port":80,"protocol":"HTTP","serviceName":"xxx:xxx-svc","ingressName...
nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-redirect: false
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 9m34s (x37 over 3d21h) nginx-ingress-controller Scheduled for sync
when i try curl/wget to host / nodeIP ,direcly from cluster , both option works, i can get my custom index
wget test.xxx.io --no-proxy --no-check-certificate
--2022-06-13 10:35:12-- http://test.xxx.io/
Resolving test.xxx.io (test.xxx.io)... 172.xx.xx.132, 172.xx.xx.131, 172.xx.xx.117
Connecting to test.xxx.io (test.xxx.io)|172.xx.xx.132|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 197 [text/html]
Saving to: ‘index.html.1’
index.html.1 100%[===========================================================================================>] 197 --.-KB/s in 0s
curl:
curl test.xxx.io --noproxy '*' -I
HTTP/1.1 200 OK
Date: Mon, 13 Jun 2022 10:36:31 GMT
Content-Type: text/html
Content-Length: 197
Connection: keep-alive
Last-Modified: Thu, 09 Jun 2022 07:20:49 GMT
ETag: "62a19f51-c5"
Accept-Ranges: bytes
nslookup
nslookup,dig,ping from cluster is working as well:
nslookup test.xxx.io
Server: 127.0.0.53
Address: 127.0.0.53#53
Name: test.xxx.io
Address: 172.xx.xx.131
Name: test.xxx.io
Address: 172.xx.xx.132
Name: test.xxx.io
Address: 172.xx.xx.117
dig
dig test.xxx.io +noall +answer
test.xxx.io. 22 IN A 172.xx.xx.117
test.xxx.io. 22 IN A 172.xx.xx.132
test.xxx.io. 22 IN A 172.xx.xx.131
ping
ping test.xxx.io
PING test.xxx.io (172.xx.xx.132) 56(84) bytes of data.
64 bytes from xx-k3s-1 (172.xx.xx.132): icmp_seq=1 ttl=64 time=0.038 ms
64 bytes from xx-k3s-1 (172.xx.xx.132): icmp_seq=2 ttl=64 time=0.042 ms
also from ingress nginx pod curl works fine...
in firefox via nodeIP:port, i can get index, but via host its not possible
seems that ingress forwarding traffic to the pod, but is this issue only something to do with browser ?
Thanks for any advice

so for clarification,
as I'm using tunneling to reach ingress from local pc via browser with SOCKS5 proxy.
ssh xxxx#100.xx.xx.xx -D 1090
solution is trivial, add
172.xx.xx.117 test.xxx.io
into /etc/hosts on jump server.

Related

Kubernetes on SuSE: NodePort Service Problems

for days I'm trying to track down a weird behaviour concerning NodePort Services when running Kubernetes on openSUSE Leap 15.3.
For testing purposes on my own server I installed 3 VMs with openSUSE 15.3. With this article: How to install kubernetes in Suse Linux enterprize server 15 virtual machines? I set up this Kubernetes Cluster:
kubix01:~ # k get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubix01 Ready control-plane,master 25h v1.22.2 192.168.42.51 <none> openSUSE Leap 15.3 5.3.18-59.27-default docker://20.10.6-ce
kubix02 Ready <none> 25h v1.22.2 192.168.42.52 <none> openSUSE Leap 15.3 5.3.18-59.27-default docker://20.10.6-ce
kubix03 Ready <none> 25h v1.22.2 192.168.42.53 <none> openSUSE Leap 15.3 5.3.18-59.27-default docker://20.10.6-ce
For testing things out I made a new 3 Replica Deployment for a traefik/whoami Image with this yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: whoami
labels:
app: whoami
spec:
replicas: 3
selector:
matchLabels:
app: whoami
template:
metadata:
labels:
app: whoami
spec:
containers:
- name: whoami
image: traefik/whoami
ports:
- containerPort: 80
This results in three Pods spread over the 2 worker nodes as expected:
kubix01:~/k8s/whoami # k get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
whoami-8557b59f65-2qkvq 1/1 Running 2 (24h ago) 25h 10.244.2.7 kubix03 <none> <none>
whoami-8557b59f65-4wnmd 1/1 Running 2 (24h ago) 25h 10.244.1.6 kubix02 <none> <none>
whoami-8557b59f65-xhx5x 1/1 Running 2 (24h ago) 25h 10.244.1.7 kubix02 <none> <none>
After that I created a NodePort Service for making things available to the world with this yaml:
apiVersion: v1
kind: Service
metadata:
name: whoami
spec:
type: NodePort
selector:
app: whoami
ports:
- protocol: TCP
port: 8080
targetPort: 80
nodePort: 30080
This is the result:
kubix01:~/k8s/whoami # k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 25h
whoami NodePort 10.105.214.86 <none> 8080:30080/TCP 25h
kubix01:~/k8s/whoami # k describe svc whoami
Name: whoami
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=whoami
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.105.214.86
IPs: 10.105.214.86
Port: <unset> 8080/TCP
TargetPort: 80/TCP
NodePort: <unset> 30080/TCP
Endpoints: 10.244.1.6:80,10.244.1.7:80,10.244.2.7:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
So everything looks fine and I tested things out with curl:
curl on one Cluster Node to PodIP:PodPort
kubix01:~/k8s/whoami # curl 10.244.1.6
Hostname: whoami-8557b59f65-4wnmd
IP: 127.0.0.1
IP: 10.244.1.6
RemoteAddr: 10.244.0.0:50380
GET / HTTP/1.1
Host: 10.244.1.6
User-Agent: curl/7.66.0
Accept: */*
kubix01:~/k8s/whoami # curl 10.244.1.7
Hostname: whoami-8557b59f65-xhx5x
IP: 127.0.0.1
IP: 10.244.1.7
RemoteAddr: 10.244.0.0:36062
GET / HTTP/1.1
Host: 10.244.1.7
User-Agent: curl/7.66.0
Accept: */*
kubix01:~/k8s/whoami # curl 10.244.2.7
Hostname: whoami-8557b59f65-2qkvq
IP: 127.0.0.1
IP: 10.244.2.7
RemoteAddr: 10.244.0.0:43924
GET / HTTP/1.1
Host: 10.244.2.7
User-Agent: curl/7.66.0
Accept: */*
==> Everything works as expected
curl on Cluster Node to services ClusterIP:ClusterPort:
kubix01:~/k8s/whoami # curl 10.105.214.86:8080
Hostname: whoami-8557b59f65-xhx5x
IP: 127.0.0.1
IP: 10.244.1.7
RemoteAddr: 10.244.0.0:1106
GET / HTTP/1.1
Host: 10.105.214.86:8080
User-Agent: curl/7.66.0
Accept: */*
kubix01:~/k8s/whoami # curl 10.105.214.86:8080
Hostname: whoami-8557b59f65-4wnmd
IP: 127.0.0.1
IP: 10.244.1.6
RemoteAddr: 10.244.0.0:9707
GET / HTTP/1.1
Host: 10.105.214.86:8080
User-Agent: curl/7.66.0
Accept: */*
kubix01:~/k8s/whoami # curl 10.105.214.86:8080
Hostname: whoami-8557b59f65-2qkvq
IP: 127.0.0.1
IP: 10.244.2.7
RemoteAddr: 10.244.0.0:25577
GET / HTTP/1.1
Host: 10.105.214.86:8080
User-Agent: curl/7.66.0
Accept: */*
==> Everything fine, Traffic is LoadBalanced to the different pods.
curl on Cluster Node to NodeIP:NodePort
kubix01:~/k8s/whoami # curl 192.168.42.51:30080
Hostname: whoami-8557b59f65-2qkvq
IP: 127.0.0.1
IP: 10.244.2.7
RemoteAddr: 10.244.0.0:5463
GET / HTTP/1.1
Host: 192.168.42.51:30080
User-Agent: curl/7.66.0
Accept: */*
kubix01:~/k8s/whoami # curl 192.168.42.52:30080
^C [NoAnswer]
kubix01:~/k8s/whoami # curl 192.168.42.53:30080
^C [NoAnswer]
==> NodePort Service is only working at the same Node, no answer from the other nodes
curl from another Network Host to NodeIP:NodePort
user#otherhost:~$ curl 192.168.42.51:30080
^C [NoAnswer]
user#otherhost:~$ curl 192.168.42.52:30080
^C [NoAnswer]
user#otherhost:~$ curl 192.168.42.53:30080
^C [NoAnswer]
==> Service is not reachable from the outside at all, no answer on all nodes
Has anybody an idea what is going wrong here?
Thx in advance
T0mcat
PS:
Additionally here a little image for clearing things a bit more. Red curved arrows are the non - working connections, gree curved arrows the working ones:
FINALLY found the solution:
Weird behaviour of SUSE concerning ip_forward sysctl setting. During OS Installation the Option "IPv4 forwarding" was activated in the installer. After OS was installed, additionally added "net.ipv4.ip_forward = 1" and
"net.ipv4.conf.all.forwarding = 1" in /etc/sysctl.conf.
With "sysctl -l|grep ip_forward" checked, both options where activated but this obviously wasn't the truth.
Found the file "/etc/sysctl.d/70-yast.conf"... here both options where set to 0, which obviously was the truth. But manually setting these options to 1 in this file still weren't enough...
Using SUSEs YAST tool and entering "System/Network Settings" -> Routing one can see the Option "Enable IPv4 Forwarding", which was checked (remember: in 70-yast.conf these options where set to 0). After playing a bit with the options in YAST, so that the 70-yast.conf gets rewritten and both options where set to 1 in 70-yast.conf, NodePort services are working as expected now.
Conclusion:
IMHO this seems to be a Bug in SUSE... Activating "IPv4 Forwarding" during Installation leads to a 70-yast.conf with disabled forwarding options. And as a plus, "sysctl -l" displays wrong settings, so as YAST does...
The solution is to play around in YASTs network Settings, so that 70-yast.conf gets rewritten with activated ip_forwarding options.

Not able to access statefulset pod via headless service using fqdn

I have a k8 setup that looks like this
ingress -> headless service (k8 service with clusterIp: none) -> statefulsets ( 2pods)
Fqdn looks like this:
nslookup my-service
Server: 100.4.0.10
Address: 100.4.0.10#53
Name: my-service.my-namespace.svc.cluster.local
Address: 100.2.2.8
Name: my-service.my-namespace.svc.cluster.local
Address: 100.1.4.2
I am trying to reach one of the pod directly via the service using the following fqdn but not able to do so.
curl -I my-pod-0.my-service.my-namespace.svc.cluster.local:8222
curl: (6) Could not resolve host: my-pod-0.my-service.my-namespace.svc.cluster.local
If I try to hit the service directly then it works correctly (as a loadbalancer)
curl -I my-service.my-namespace.svc.cluster.local:8222
HTTP/1.1 200 OK
Date: Sat, 31 Jul 2021 21:24:42 GMT
Content-Length: 656
If I try to hit the pod directly using it's cluster ip, it also works fine
curl -I 100.2.2.8:8222
HTTP/1.1 200 OK
Date: Sat, 31 Jul 2021 21:29:22 GMT
Content-Length: 656
Content-Type: text/html; charset=utf-8
But my use case requires me to be able to hit the statefulset pod using fqdn i.e my-pod-0.my-service.my-namespace.svc.cluster.local . What am I missing here?
example statefulset called foo with image nginx:
k get statefulsets.apps
NAME READY AGE
foo 3/3 8m55s
This stateful set created following pods(foo-0,foo-1,foo-2):
k get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 1 3h47m 10.1.198.71 ps-master <none> <none>
foo-0 1/1 Running 0 12m 10.1.198.121 ps-master <none> <none>
foo-1 1/1 Running 0 12m 10.1.198.77 ps-master <none> <none>
foo-2 1/1 Running 0 12m 10.1.198.111 ps-master <none> <none>
Now create a headless service(clusterIP is none) as follow:(make sure to use correct selector same as your statefulset)
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: foo
spec:
type: ClusterIP
clusterIP: None
ports:
- port: 80
name: web
selector:
app: foo
Now, do nslookup to see the dns resolution working for the service.(Optional step)
k exec -it busybox -- nslookup nginx.default.svc.cluster.local
Server: 10.152.183.10
Address 1: 10.152.183.10 kube-dns.kube-system.svc.cluster.local
Name: nginx.default.svc.cluster.local
Address 1: 10.1.198.77 foo-1.nginx.default.svc.cluster.local
Address 2: 10.1.198.111 foo-2.nginx.default.svc.cluster.local
Address 3: 10.1.198.121 foo-0.nginx.default.svc.cluster.local
Now validate that, individual resolution per-pod is working:
k exec -it busybox -- nslookup foo-1.nginx.default.svc.cluster.local
Server: 10.152.183.10
Address 1: 10.152.183.10 kube-dns.kube-system.svc.cluster.local
Name: foo-1.nginx.default.svc.cluster.local
Address 1: 10.1.198.77 foo-1.nginx.default.svc.cluster.local
More info: Here
Note: In this case OP had incorrect mapping of headless service and the statefulset, this can be verified with below command:
k get statefulsets.apps foo -o jsonpath="{.spec.serviceName}{'\n'}"
nignx
Ensure that, the mapping.
Original answer didn't clarify how OP fixed the issue, the problem was in serviceName property under statefulset implementation.

Kubernetes DNS Troubleshooting

I am trying to troubleshoot a DNS issue in our K8 cluster v1.19. There are 3 nodes (1 controller, 2 workers) all running vanilla Ubuntu 20.04 using Calico network with Metallb for inbound load balancing. This is all hosted on premise and has full access to the internet. There is also a proxy server (Traefik) in front of it that is handling the SSL to the K8 cluster and other services in the infrastructure.
This issue happened when I upgraded the helm chart for the pod that was/is connecting to the redis pod, but otherwise had been happy to run for the past 36 days.
In the log of one of the pods it is showing an error that it cannot determine where the redis pod(s) is/are:
2020-11-09 00:00:00 [1] [verbose]: [Cache] Attempting connection to redis.
2020-11-09 00:00:00 [1] [verbose]: [Cache] Successfully connected to redis.
2020-11-09 00:00:00 [1] [verbose]: [PubSub] Attempting connection to redis.
2020-11-09 00:00:00 [1] [verbose]: [PubSub] Successfully connected to redis.
2020-11-09 00:00:00 [1] [warn]: Secret key is weak. Please consider lengthening it for better security.
2020-11-09 00:00:00 [1] [verbose]: [Database] Connecting to database...
2020-11-09 00:00:00 [1] [info]: [Database] Successfully connected .
2020-11-09 00:00:00 [1] [verbose]: [Database] Ran 0 migration(s).
2020-11-09 00:00:00 [1] [verbose]: Sending request for public key.
Error: getaddrinfo EAI_AGAIN oct-2020-redis-master
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:67:26) {
errno: -3001,
code: 'EAI_AGAIN',
syscall: 'getaddrinfo',
hostname: 'oct-2020-redis-master'
}
[ioredis] Unhandled error event: Error: getaddrinfo EAI_AGAIN oct-2020-redis-master
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:67:26)
Error: connect ETIMEDOUT
at Socket.<anonymous> (/app/node_modules/ioredis/built/redis/index.js:307:37)
at Object.onceWrapper (events.js:421:28)
at Socket.emit (events.js:315:20)
at Socket.EventEmitter.emit (domain.js:486:12)
at Socket._onTimeout (net.js:483:8)
at listOnTimeout (internal/timers.js:554:17)
at processTimers (internal/timers.js:497:7) {
errorno: 'ETIMEDOUT',
code: 'ETIMEDOUT',
syscall: 'connect'
}
I have gone through the steps outlined in https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
ubuntu#k8-01:~$ kubectl exec -i -t dnsutils -- nslookup kubernetes.default
;; connection timed out; no servers could be reached
command terminated with exit code 1
ubuntu#k8-01:~$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-f9fd979d6-lfm5t 1/1 Running 17 37d
coredns-f9fd979d6-sw2qp 1/1 Running 18 37d
ubuntu#k8-01:~$ kubectl logs --namespace=kube-system -l k8s-app=kube-dns
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[INFO] Reloading
[INFO] plugin/health: Going into lameduck mode for 5s
[INFO] plugin/reload: Running configuration MD5 = 3d3f6363f05ccd60e0f885f0eca6c5ff
[INFO] Reloading complete
[INFO] 10.244.210.238:34288 - 28733 "A IN oct-2020-redis-master.default.svc.cluster.local. udp 75 false 512" NOERROR qr,aa,rd 148 0.001300712s
[INFO] 10.244.210.238:44532 - 12032 "A IN oct-2020-redis-master.default.svc.cluster.local. udp 75 false 512" NOERROR qr,aa,rd 148 0.001279312s
[INFO] 10.244.210.235:44595 - 65094 "A IN oct-2020-redis-master.default.svc.cluster.local. udp 75 false 512" NOERROR qr,aa,rd 148 0.000163001s
[INFO] 10.244.210.235:55945 - 20758 "A IN oct-2020-redis-master.default.svc.cluster.local. udp 75 false 512" NOERROR qr,aa,rd 148 0.000141202s
ubuntu#k8-01:~$ kubectl get services --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default oct-2020-api ClusterIP 10.107.89.213 <none> 80/TCP 37d
default oct-2020-nginx-ingress-controller LoadBalancer 10.110.235.175 192.168.2.150 80:30194/TCP,443:31514/TCP 37d
default oct-2020-nginx-ingress-default-backend ClusterIP 10.98.147.246 <none> 80/TCP 37d
default oct-2020-redis-headless ClusterIP None <none> 6379/TCP 37d
default oct-2020-redis-master ClusterIP 10.109.58.236 <none> 6379/TCP 37d
default oct-2020-webclient ClusterIP 10.111.204.251 <none> 80/TCP 37d
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 37d
kube-system coredns NodePort 10.101.104.114 <none> 53:31245/UDP 15h
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 37d
When I enter the pod:
/app # grep "nameserver" /etc/resolv.conf
nameserver 10.96.0.10
/app # nslookup
BusyBox v1.31.1 () multi-call binary.
Usage: nslookup [-type=QUERY_TYPE] [-debug] HOST [DNS_SERVER]
Query DNS about HOST
QUERY_TYPE: soa,ns,a,aaaa,cname,mx,txt,ptr,any
/app # ping 10.96.0.10
PING 10.96.0.10 (10.96.0.10): 56 data bytes
^C
--- 10.96.0.10 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
/app # nslookup oct-20-redis-master
;; connection timed out; no servers could be reached
Any ideas on troubleshooting would be greatly appreciated.
To answer my own question, I deleted the DNS pods and then it worked again. The command was the following:
kubectl delete pod coredns-f9fd979d6-sw2qp --namespace=kube-system
This doesn't get to the underlying problem of why this is happening, or why K8 isn't detecting that something is wrong with those pods and recreating them. I am going to keep digging into this and put some more instrumenting on the DNS pods to see what it actually is that is causing this problem.
If anyone has any ideas on instrumenting to hook up or look at specifically, that would be appreciated.
This is how we test dns
Create below deployment
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
labels:
app: nginx
spec:
serviceName: "nginx"
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: k8s.gcr.io/nginx-slim:0.8
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumes:
- name: www
emptyDir:
Run the below tests
master $ kubectl get po
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 1m
web-1 1/1 Running 0 1m
master $ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 35m
nginx ClusterIP None <none> 80/TCP 2m
master $ kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm
If you don't see a command prompt, try pressing enter.
/ # nslookup nginx
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: nginx
Address 1: 10.40.0.1 web-0.nginx.default.svc.cluster.local
Address 2: 10.40.0.2 web-1.nginx.default.svc.cluster.local
/ #
/ # nslookup web-0.nginx
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: web-0.nginx
Address 1: 10.40.0.1 web-0.nginx.default.svc.cluster.local
/ # nslookup web-0.nginx.default.svc.cluster.local
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: web-0.nginx.default.svc.cluster.local
Address 1: 10.40.0.1 web-0.nginx.default.svc.cluster.local

certificate is valid for ingress.local, not gitlab.mydomain

I'm new to Kubernetes.
I have installed a freshly new Kubernetes Cluster by using RKE (rancher tool for creating k8 clusters).
I added the gitlab chart (https://charts.gitlab.io/) and launch it.
Being on several issues with PersistentStorage, etc that I managed to resolve.
But I'm now stuck on one last issue: the pod for gitlab-runner is failing with the following logs:
ERROR: Registering runner... failed runner=Mk5hMxa5 status=couldn't execute POST against https://gitlab.mydomain.com/api/v4/runners: Post https://gitlab.mydomain.com/api/v4/runners: x509: certificate is valid for ingress.local, not gitlab.mydomain.com
PANIC: Failed to register this runner. Perhaps you are having network problems
Description of the certificate using kubectl describe certificate gitlab-gitlab-tls -n gitlab:
Name: gitlab-gitlab-tls
Namespace: gitlab
Labels: app=unicorn
chart=unicorn-2.4.6
heritage=Tiller
io.cattle.field/appId=gitlab
release=gitlab
Annotations: <none>
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2019-11-13T13:49:10Z
Generation: 3
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: gitlab-unicorn
UID: 5640645f-550b-4073-bdf0-df8b089b0c94
Resource Version: 6824
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/gitlab/certificates/gitlab-gitlab-tls
UID: 30ac32bd-c7f3-4f9b-9e3b-966b6090e1a9
Spec:
Acme:
Config:
Domains:
gitlab.mydomain.com
http01:
Ingress Class: gitlab-nginx
Dns Names:
gitlab.mydomain.com
Issuer Ref:
Kind: Issuer
Name: gitlab-issuer
Secret Name: gitlab-gitlab-tls
Status:
Conditions:
Last Transition Time: 2019-11-13T13:49:10Z
Message: Certificate issuance in progress. Temporary certificate issued.
Reason: TemporaryCertificate
Status: False
Type: Ready
Events: <none>
Description of the issuer using kubectl describe issuer gitlab-issuer -n gitlab:
Name: gitlab-issuer
Namespace: gitlab
Labels: app=certmanager-issuer
chart=certmanager-issuer-0.1.0
heritage=Tiller
release=gitlab
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Issuer","metadata":{"annotations":{},"creationTimestamp":"2019-11-13T13:49:10Z","gener...
API Version: certmanager.k8s.io/v1alpha1
Kind: Issuer
Metadata:
Creation Timestamp: 2019-11-13T13:49:10Z
Generation: 4
Resource Version: 24537
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/gitlab/issuers/gitlab-issuer
UID: b9971d7a-5220-47ca-a7f9-607aa3f9be4f
Spec:
Acme:
Email: mh#mydomain.com
http01:
Private Key Secret Ref:
Name: gitlab-acme-key
Server: https://acme-v02.api.letsencrypt.org/directory
Status:
Acme:
Last Registered Email: mh#mydomain.com
Uri: https://acme-v02.api.letsencrypt.org/acme/acct/71695690
Conditions:
Last Transition Time: 2019-11-13T13:49:12Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
Description of the challenge using kubectl describe challenges.certmanager.k8s.io -n gitlab gitlab-gitlab-tls-3386074437-0:
Name: gitlab-gitlab-tls-3386074437-0
Namespace: gitlab
Labels: acme.cert-manager.io/order-name=gitlab-gitlab-tls-3386074437
Annotations: <none>
API Version: certmanager.k8s.io/v1alpha1
Kind: Challenge
Metadata:
Creation Timestamp: 2019-11-13T13:49:15Z
Finalizers:
finalizer.acme.cert-manager.io
Generation: 4
Owner References:
API Version: certmanager.k8s.io/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: Order
Name: gitlab-gitlab-tls-3386074437
UID: 1f01771e-2e38-491f-9b2d-ab5f4fda60e2
Resource Version: 6915
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/gitlab/challenges/gitlab-gitlab-tls-3386074437-0
UID: 4c115a6f-a76f-4859-a5db-6acd9c039d71
Spec:
Authz URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/1220588820
Config:
http01:
Ingress Class: gitlab-nginx
Dns Name: gitlab.mydomain.com
Issuer Ref:
Kind: Issuer
Name: gitlab-issuer
Key: lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8.lPWns02SmS3zXwFzHdma_RyhwwlzWLRDkdlugFXDlZY
Token: lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8
Type: http-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/1220588820/AwsnPw
Wildcard: false
Status:
Presented: true
Processing: true
Reason: Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
State: pending
Events: <none>
Logs found in cert-manager pod:
I1113 14:20:21.857235 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "level"=0 "msg"="found one existing HTTP01 solver pod" "dnsName"="gitlab.mydomain.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ttkmj" "related_resource_namespace"="gitlab" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01"
I1113 14:20:21.857458 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "level"=0 "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="gitlab.mydomain.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-sdlw7" "related_resource_namespace"="gitlab" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01"
I1113 14:20:21.857592 1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "level"=0 "msg"="found one existing HTTP01 solver ingress" "dnsName"="gitlab.mydomain.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-7jzwk" "related_resource_namespace"="gitlab" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01"
E1113 14:20:21.864785 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="gitlab.mydomain.com" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01"
The DNS gitlab.mydomain.com is set to point to the IP of my LoadBalancer where NGINX is running.
If I go to https://gitlab.mydomain.com in the browser:
The browser is saying the connexion is not secure
The result is "default backend - 404".
Edits
Description of the ingress-controller by using kubectl describe svc gitlab-nginx-ingress-controller -n gitlab:
Name: gitlab-nginx-ingress-controller
Namespace: gitlab
Labels: app=nginx-ingress
chart=nginx-ingress-0.30.0-1
component=controller
heritage=Tiller
io.cattle.field/appId=gitlab
release=gitlab
Annotations: field.cattle.io/ipAddresses: null
field.cattle.io/targetDnsRecordIds: null
field.cattle.io/targetWorkloadIds: null
Selector: <none>
Type: ExternalName
IP:
External Name: gitlab.mydomain.com
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 31487/TCP
Endpoints: 10.42.0.7:80,10.42.1.9:80,10.42.2.12:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31560/TCP
Endpoints: 10.42.0.7:443,10.42.1.9:443,10.42.2.12:443
Port: gitlab-shell 22/TCP
TargetPort: gitlab-shell/TCP
NodePort: gitlab-shell 30539/TCP
Endpoints: 10.42.0.7:22,10.42.1.9:22,10.42.2.12:22
Session Affinity: None
Events: <none>
Running kubectl get ingress -n gitlab gives me a bunch of ingress:
NAME HOSTS ADDRESS PORTS AGE
cm-acme-http-solver-5rjg4 minio.mydomain.com gitlab.mydomain.com 80 4d23h
cm-acme-http-solver-7jzwk gitlab.mydomain.com gitlab.mydomain.com 80 4d23h
cm-acme-http-solver-tzs25 registry.mydomain.com gitlab.mydomain.com 80 4d23h
gitlab-minio minio.mydomain.com gitlab.mydomain.com 80, 443 4d23h
gitlab-registry registry.mydomain.com gitlab.mydomain.com 80, 443 4d23h
gitlab-unicorn gitlab.mydomain.com gitlab.mydomain.com 80, 443 4d23h
Description of the gitlab-unicorn by using kubectl describe ingress gitlab-unicron -n gitlab
Name: gitlab-unicorn
Namespace: gitlab
Address: gitlab.mydomain.com
Default backend: default-http-backend:80 (<none>)
TLS:
gitlab-gitlab-tls terminates gitlab.mydomain.com
Rules:
Host Path Backends
---- ---- --------
gitlab.mydomain.com
/ gitlab-unicorn:8181 (10.42.0.9:8181,10.42.1.8:8181)
/admin/sidekiq gitlab-unicorn:8080 (10.42.0.9:8080,10.42.1.8:8080)
Annotations:
certmanager.k8s.io/issuer: gitlab-issuer
field.cattle.io/publicEndpoints: [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"gitlab:gitlab-unicorn","ingressName":"gitlab:gitlab-unicorn","hostname":"gitlab.mydomain.com","path":"/","allNodes":false},{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"gitlab:gitlab-unicorn","ingressName":"gitlab:gitlab-unicorn","hostname":"gitlab.mydomain.com","path":"/admin/sidekiq","allNodes":false}]
kubernetes.io/ingress.class: gitlab-nginx
kubernetes.io/ingress.provider: nginx
nginx.ingress.kubernetes.io/proxy-body-size: 512m
nginx.ingress.kubernetes.io/proxy-connect-timeout: 15
nginx.ingress.kubernetes.io/proxy-read-timeout: 600
Events: <none>
Description of cm-acme-http-solver-7jzwk by using kubectl describe ingress cm-acme-http-solver-7jzwk -n gitlab:
Name: cm-acme-http-solver-7jzwk
Namespace: gitlab
Address: gitlab.mydomain.com
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
gitlab.mydomain.com
/.well-known/acme-challenge/lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8 cm-acme-http-solver-sdlw7:8089 (10.42.2.19:8089)
Annotations:
field.cattle.io/publicEndpoints: [{"addresses":[""],"port":80,"protocol":"HTTP","serviceName":"gitlab:cm-acme-http-solver-sdlw7","ingressName":"gitlab:cm-acme-http-solver-7jzwk","hostname":"gitlab.mydomain.com","path":"/.well-known/acme-challenge/lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8","allNodes":false}]
kubernetes.io/ingress.class: gitlab-nginx
nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
Events: <none>
Ports open on my LoadBalancer and on every nodes of my cluster (I know I should close somes but I will first manage to make my gitlab setup working):
80/tcp ALLOW Anywhere
443/tcp ALLOW Anywhere
22/tcp ALLOW Anywhere
2376/tcp ALLOW Anywhere
2379/tcp ALLOW Anywhere
2380/tcp ALLOW Anywhere
6443/tcp ALLOW Anywhere
6783/tcp ALLOW Anywhere
6783:6784/udp ALLOW Anywhere
8472/udp ALLOW Anywhere
4789/udp ALLOW Anywhere
9099/tcp ALLOW Anywhere
10250/tcp ALLOW Anywhere
10254/tcp ALLOW Anywhere
30000:32767/tcp ALLOW Anywhere
30000:32767/udp ALLOW Anywhere
80/tcp (v6) ALLOW Anywhere (v6)
443/tcp (v6) ALLOW Anywhere (v6)
22/tcp (v6) ALLOW Anywhere (v6)
2376/tcp (v6) ALLOW Anywhere (v6)
2379/tcp (v6) ALLOW Anywhere (v6)
2380/tcp (v6) ALLOW Anywhere (v6)
6443/tcp (v6) ALLOW Anywhere (v6)
6783/tcp (v6) ALLOW Anywhere (v6)
6783:6784/udp (v6) ALLOW Anywhere (v6)
8472/udp (v6) ALLOW Anywhere (v6)
4789/udp (v6) ALLOW Anywhere (v6)
9099/tcp (v6) ALLOW Anywhere (v6)
10250/tcp (v6) ALLOW Anywhere (v6)
10254/tcp (v6) ALLOW Anywhere (v6)
30000:32767/tcp (v6) ALLOW Anywhere (v6)
30000:32767/udp (v6) ALLOW Anywhere (v6)
kubectl get pods -n gitlab
cm-acme-http-solver-4d8s5 1/1 Running 0 5d
cm-acme-http-solver-ttkmj 1/1 Running 0 5d
cm-acme-http-solver-ws7kv 1/1 Running 0 5d
gitlab-certmanager-57bc6fb4fd-6rfds 1/1 Running 0 5d
gitlab-gitaly-0 1/1 Running 0 5d
gitlab-gitlab-exporter-57b99467d4-knbgk 1/1 Running 0 5d
gitlab-gitlab-runner-64b74bcd59-mxwvm 0/1 CrashLoopBackOff 10 55m
gitlab-gitlab-shell-cff8b68f7-zng2c 1/1 Running 0 5d
gitlab-gitlab-shell-cff8b68f7-zqvfr 1/1 Running 0 5d
gitlab-issuer.1-lqs7c 0/1 Completed 0 5d
gitlab-migrations.1-c4njn 0/1 Completed 0 5d
gitlab-minio-75567fcbb6-jjxhw 1/1 Running 6 5d
gitlab-minio-create-buckets.1-6zljh 0/1 Completed 0 5d
gitlab-nginx-ingress-controller-698fbc4c64-4wt97 1/1 Running 0 5d
gitlab-nginx-ingress-controller-698fbc4c64-5kv2h 1/1 Running 0 5d
gitlab-nginx-ingress-controller-698fbc4c64-jxljq 1/1 Running 0 5d
gitlab-nginx-ingress-default-backend-6cd54c5f86-2jrkd 1/1 Running 0 5d
gitlab-nginx-ingress-default-backend-6cd54c5f86-cxlmx 1/1 Running 0 5d
gitlab-postgresql-66d8d9574b-hbx78 2/2 Running 0 5d
gitlab-prometheus-server-6fb685b9c7-c8bqj 2/2 Running 0 5d
gitlab-redis-7668c4d476-tcln5 2/2 Running 0 5d
gitlab-registry-7bb984c765-7ww6j 1/1 Running 0 5d
gitlab-registry-7bb984c765-t5jjq 1/1 Running 0 5d
gitlab-sidekiq-all-in-1-8fd95bf7b-hfnjz 1/1 Running 0 5d
gitlab-task-runner-5cd7bf5bb9-gnv8p 1/1 Running 0 5d
gitlab-unicorn-864bd864f5-47zxg 2/2 Running 0 5d
gitlab-unicorn-864bd864f5-gjms2 2/2 Running 0 5d
Their are 3 acme-http-solver:
One for registry.mydomain.com
One for minio.mydomain.com
One for gitlab.mydomain.com
The logs for the one pointing to gitlab.mydomain.com:
I1113 13:49:21.207782 1 solver.go:39] cert-manager/acmesolver "level"=0 "msg"="starting listener" "expected_domain"="gitlab.mydomain.com" "expected_key"="lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8.lPWns02SmS3zXwFzHdma_RyhwwlzWLRDkdlugFXDlZY" "expected_token"="lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8" "listen_port"=8089
Results of kubectl get svc -n gitlab:
cm-acme-http-solver-48b2j NodePort 10.43.58.52 <none> 8089:30090/TCP 5d23h
cm-acme-http-solver-h42mk NodePort 10.43.23.141 <none> 8089:30415/TCP 5d23h
cm-acme-http-solver-sdlw7 NodePort 10.43.86.27 <none> 8089:32309/TCP 5d23h
gitlab-gitaly ClusterIP None <none> 8075/TCP,9236/TCP 5d23h
gitlab-gitlab-exporter ClusterIP 10.43.187.247 <none> 9168/TCP 5d23h
gitlab-gitlab-shell ClusterIP 10.43.246.124 <none> 22/TCP 5d23h
gitlab-minio-svc ClusterIP 10.43.117.249 <none> 9000/TCP 5d23h
gitlab-nginx-ingress-controller ExternalName <none> gitlab.mydomain.com 80:31487/TCP,443:31560/TCP,22:30539/TCP 5d23h
gitlab-nginx-ingress-controller-metrics ClusterIP 10.43.152.252 <none> 9913/TCP 5d23h
gitlab-nginx-ingress-controller-stats ClusterIP 10.43.173.191 <none> 18080/TCP 5d23h
gitlab-nginx-ingress-default-backend ClusterIP 10.43.116.121 <none> 80/TCP 5d23h
gitlab-postgresql ClusterIP 10.43.97.139 <none> 5432/TCP 5d23h
gitlab-prometheus-server ClusterIP 10.43.67.220 <none> 80/TCP 5d23h
gitlab-redis ClusterIP 10.43.36.138 <none> 6379/TCP,9121/TCP 5d23h
gitlab-registry ClusterIP 10.43.54.244 <none> 5000/TCP 5d23h
gitlab-unicorn ClusterIP 10.43.76.61 <none> 8080/TCP,8181/TCP 5d23h
Logs of the pod gitlab-nginx-ingress-controller-698fbc4c64-jxljq (others nginx-ingress-controller gives same logs): https://textuploader.com/1o9we
Any hint on what could be wrong in my configuration ?
Fell free to ask for more information on my setup.
Many thanks.
Well the issue is, Gitlab requires a valid SSL certificate for the domain in question. Which you do not seem to have according to the output of:
E1113 14:20:21.864785 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="gitlab.mydomain.com" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01"
Status:
Presented: true
Processing: true
Reason: Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
State: pending
The http-01 challenge is where it will try to do a web request to your domain, and it should return a 200 HTTP response. When you said yourself that https://gitlab.mydomain.com gives you a 404 response (hence it will fail to issue a valid certificate). To further diagnose this, check the output of the ingress responsible for the domain, and follow it down the "chain" until you identify where the 404 is being responded by.
The http01 challenge relies on port 80 (http) to be exposed to be able to answer the challenge. The option controller.service.enableHttp configures http, and is enabled by default (see here. But even if you've not touched this config, there might be an upstream component (i.e. a firewall) that blocks traffic on port 80.
Could you check if your ingress Service is listening on port 80, and reachable from the internet? You can try to go to your public IP on port 80 via a browser to check if you get a response from the ingress controller (or a backend).

Kubernetes ExternalName service not visible in DNS

I'm trying to expose a single database instance as a service in two Kubernetes namespaces. Kubernetes version 1.11.3 running on Ubuntu 16.04.1. The database service is visible and working in the default namespace. I created an ExternalName service in a non-default namespace referencing the fully qualified domain name in the default namespace as follows:
kind: Service
apiVersion: v1
metadata:
name: ws-mysql
namespace: wittlesouth
spec:
type: ExternalName
externalName: mysql.default.svc.cluster.local
ports:
- port: 3306
The service is running:
eric$ kubectl describe service ws-mysql --namespace=wittlesouth
Name: ws-mysql
Namespace: wittlesouth
Labels: <none>
Annotations: <none>
Selector: <none>
Type: ExternalName
IP:
External Name: mysql.default.svc.cluster.local
Port: <unset> 3306/TCP
TargetPort: 3306/TCP
Endpoints: <none>
Session Affinity: None
Events: <none>
If I check whether the service can be found by name from a pod running in the wittlesouth namespace, this service name does not resolve, but other services in that namespace (i.e. Jira) do:
root#rs-ws-diags-8mgqq:/# nslookup mysql.default.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: mysql.default.svc.cluster.local
Address: 10.99.120.208
root#rs-ws-diags-8mgqq:/# nslookup ws-mysql.wittlesouth
Server: 10.96.0.10
Address: 10.96.0.10#53
*** Can't find ws-mysql.wittlesouth: No answer
root#rs-ws-diags-8mgqq:/# nslookup ws-mysql
Server: 10.96.0.10
Address: 10.96.0.10#53
*** Can't find ws-mysql: No answer
root#rs-ws-diags-8mgqq:/# nslookup ws-mysql.wittlesouth
Server: 10.96.0.10
Address: 10.96.0.10#53
*** Can't find ws-mysql.wittlesouth: No answer
root#rs-ws-diags-8mgqq:/# nslookup ws-mysql.wittlesouth.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
*** Can't find ws-mysql.wittlesouth.svc.cluster.local: No answer
root#rs-ws-diags-8mgqq:/# nslookup ws-mysql.wittlesouth
Server: 10.96.0.10
Address: 10.96.0.10#53
*** Can't find ws-mysql.wittlesouth: No answer
root#rs-ws-diags-8mgqq:/# nslookup jira.wittlesouth
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: jira.wittlesouth.svc.cluster.local
Address: 10.105.30.239
Any thoughts on what might be the issue here? For the moment I've worked around it by updating applications that need to use the database to reference the fully qualified domain name of the service running in the default namespace, but I'd prefer to avoid that. My intent eventually is to have the namespaces have separate database instances, and would like to deploy apps configured to work that way now in advance of actually standing up the second instance.
This doesn't work for me with Kubernetes 1.11.2 with coredns and calico. It works only if you reference the external service directly in whichever namespace it runs:
$ kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
mysql-0 2/2 Running 0 17m
mysql-1 2/2 Running 0 16m
$ kubectl get pods -n wittlesouth
NAME READY STATUS RESTARTS AGE
ricos-dummy-pod 1/1 Running 0 14s
kubectl exec -it ricos-dummy-pod -n wittlesouth bash
root#ricos-dummy-pod:/# ping mysql.default.svc.cluster.local
PING mysql.default.svc.cluster.local (192.168.1.40): 56 data bytes
64 bytes from 192.168.1.40: icmp_seq=0 ttl=62 time=0.578 ms
64 bytes from 192.168.1.40: icmp_seq=1 ttl=62 time=0.632 ms
64 bytes from 192.168.1.40: icmp_seq=2 ttl=62 time=0.628 ms
^C--- mysql.default.svc.cluster.local ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.578/0.613/0.632/0.025 ms
root#ricos-dummy-pod:/# ping ws-mysql
ping: unknown host
root#ricos-dummy-pod:/# exit
$ kubectl get svc mysql
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mysql ClusterIP None <none> 3306/TCP 45d
$ kubectl describe svc mysql
Name: mysql
Namespace: default
Labels: app=mysql
Annotations: <none>
Selector: app=mysql
Type: ClusterIP
IP: None
Port: mysql 3306/TCP
TargetPort: 3306/TCP
Endpoints: 192.168.1.40:3306,192.168.2.25:3306
Session Affinity: None
Events: <none>
The ExternalName service feature is only supported using kube-dns as per the docs and Kubernetes 1.11.x defaults to coredns. You might want to try changing from coredns to kube-dns or possibly changing the configs for your coredns deployment. I expect this to available at some point using coredns.