Gitlab CI cannot resolve service hostname on kubernetes runner - kubernetes

Question

I have a GitLab CI pipeline configured to run on Kubernetes runner. Everything worked great until I tried to add services (https://docs.gitlab.com/ee/ci/services/mysql.html) for test job. The service hostname (eg.: mysql) cannot be resolved on kubernetes, resulting into the following error dial tcp: lookup mysql on 10.96.0.10:53: no such host. However, it works on docker runner, but that's just not what I want. Is there any way to
The job definition from .gitlab-ci.yml:
test:
stage: test
variables:
MYSQL_ROOT_PASSWORD: --top-secret--
MYSQL_DATABASE: --top-secret--
MYSQL_USER: --top-secret--
MYSQL_PASSWORD: --top-secret--
services:
- mysql:latest
- nats:latest
script:
- ping -c 2 mysql
- go test -cover -coverprofile=coverage.prof.tmp ./...
Edit:
Logs from runner-jd6sxcl7-project-430-concurrent-0g5bm8 pod show that the services started. There are 4 containers total inside the pod: build,helper,svc-0 (mysql), svc-1 (nats)
svc-0 logs show the mysql service started successfully:
2019-12-09 21:52:07+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2019-12-09 21:52:07+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2019-12-09 21:52:08+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2019-12-09 21:52:08+00:00 [Note] [Entrypoint]: Initializing database files
2019-12-09T21:52:08.226747Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2019-12-09T21:52:08.233097Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.18) initializing of server in progress as process 46
svc-1 logs show the nats service started successfully as well:
[1] 2019/12/09 21:52:12.876121 [INF] Starting nats-server version 2.1.2
[1] 2019/12/09 21:52:12.876193 [INF] Git commit [679beda]
[1] 2019/12/09 21:52:12.876393 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2019/12/09 21:52:12.876522 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2019/12/09 21:52:12.876548 [INF] Server id is NCPAQNFKKWPI67DZHSWN5EWOCQSRACFG2FXNGTLMW2NNRBAMLSDY4IYQ
[1] 2019/12/09 21:52:12.876552 [INF] Server is ready
[1] 2019/12/09 21:52:12.876881 [INF] Listening for route connections on 0.0.0.0:6222

Answer 1

This was a known issue with older versions of GitLab Runner (< 12.8), or when running Kubernetes executor in older versions of Kubernetes (< 1.7).
From the Kubernetes executor documentation of GitLab Runner:
Since GitLab Runner 12.8 and Kubernetes 1.7, the services are accessible via their DNS names. If you are using an older version you will have to use localhost.
(emphasis mine)
It's important to keep in mind that the other restrictions and implications of the Kubernetes executor. From the same document:
Note that when services and containers are running in the same Kubernetes
pod, they are all sharing the same localhost address.
So, even if you're able to use the service specific hostname to talk to your service, it's all really localhost (127.0.0.1) underneath.
As such, keep in mind the other important restriction from the same document:
You cannot use several services using the same port
(thanks to #user3154003 for the link to the GitLab Runner issue in a currently deleted answer that pointed me in the right direction for this answer.)

Answer 2

From the documentation link you provided, I see that the mysql hostname should be accessible.
How services are linked to the job
To better understand how the container linking works, read Linking
containers together.
To summarize, if you add mysql as service to your application, the
image will then be used to create a container that is linked to the
job container.
The service container for MySQL will be accessible under the hostname
mysql. So, in order to access your database service you have to
connect to the host named mysql instead of a socket or localhost. Read
more in accessing the services.
Also from the doc
If you don’t specify a service alias, when the job is run, service will be started and you will have access to it from your build container
So can you check if there are errors when the mysql service started.
I provided this as answer since I could not fit this in comment.