Cannot ping containers in the same pod in Kubernetes(minikube) - kubernetes

Question

On my local I run a mysql container and then ping it from another container on the same network:
$ docker run -d tutum/mysql
$ docker run -it plumsempy/plum bash
# ping MYSQL_CONTAINER_ID
PING 67e35427d638 (198.105.244.24): 56 data bytes
64 bytes from 198.105.244.24: icmp_seq=0 ttl=37 time=0.243 ms
...
That is good. Then, using Kubernetes(minikube) locally, I deploy tutum/mysql using the following YAML:
...
- name: mysql
image: tutum/mysql
...
There is nothing else for the mysql container. Then I deploy it, ssh into the minikube pod, spin up a random container and try pinging the mysql container inside the pod this time:
$ kubectl create -f k8s-deployment.yml
$ minikube ssh
$ docker ps
$ docker run -it plumsempy/plum bash
# ping MYSQL_CONTAINER_ID_INSIDE_MINIKUBE
PING mysql (198.105.244.24): 56 data bytes
^C--- mysql ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss
# traceroute MYSQL_CONTAINER_ID_INSIDE_MINIKUBE
traceroute to aa7f7ed7af01 (198.105.244.24), 30 hops max, 60 byte packets
1 172.17.0.1 (172.17.0.1) 0.031 ms 0.009 ms 0.007 ms
2 10.0.2.2 (10.0.2.2) 0.156 ms 0.086 ms 0.050 ms
3 * * *
4 * * *
5 dtr02gldlca-tge-0-2-0-1.gldl.ca.charter.com (96.34.102.201) 16.153 ms 16.107 ms 16.077 ms
6 crr01lnbhca-bue-200.lnbh.ca.charter.com (96.34.98.188) 18.753 ms 18.011 ms 30.642 ms
7 crr01mtpkca-bue-201.mtpk.ca.charter.com (96.34.96.63) 30.779 ms 30.523 ms 30.428 ms
8 bbr01mtpkca-bue-2.mtpk.ca.charter.com (96.34.2.24) 24.089 ms 23.900 ms 23.814 ms
9 bbr01ashbva-tge-0-1-0-1.ashb.va.charter.com (96.34.3.139) 26.061 ms 25.949 ms 36.002 ms
10 10ge9-10.core1.lax1.he.net (65.19.189.177) 34.027 ms 34.436 ms 33.857 ms
11 100ge12-1.core1.ash1.he.net (184.105.80.201) 107.873 ms 107.750 ms 104.078 ms
12 100ge3-1.core1.nyc4.he.net (184.105.223.166) 100.554 ms 100.478 ms 100.393 ms
13 xerocole-inc.10gigabitethernet12-4.core1.nyc4.he.net (216.66.41.242) 109.184 ms 111.122 ms 111.018 ms
14 * * *
15 * * *
...(til it ends)
the plumsempy/plum can be any container since they are both on the same network and same pod, the pinging should go through. The question is Why can I not reach mysql on minikube and how could I fix that?

Answer 1

From k8s multi-container pod docs:
Pods share fate, and share some resources, such as storage volumes and IP addresses.
Hence the mysql container is reachable from the plum container at the IP address 127.0.0.1.
Also, since mysql runs on port 3306 by default, you probably want telnet 127.0.0.1 3306 to check if it's reachable (ping uses ICMP which doesn't have the concept of ports).

Answer 2

I guess the container ID just don't work with Kubernetes. You can also see, that the container ID resolved to the public IP 198.105.244.24, which looks wrong.
You have multiple ways to contact this pod:
get the pod IP via kubectl describe -f k8s-deployment.yml
create a service for that pod and do one of these (assuming the service name is mysql):
use environment variables like ping ${MYSQL_SERVICE_HOST}
use DNS like ping mysql.default.svc.cluster.local