Kafka & Zookeeper in Gitlab CI - apache-kafka

Question

I'm trying to run a simple test if my application is running properly without any issues. My issue is that faust needs a connection to kafka on initialization - so I'm trying to run kafka with zookeeper as services but I'm not able to connect them properly.
Error:
2021-12-16T13:53:51.385341793Z [2021-12-16 13:53:51,385] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
2021-12-16T13:53:51.391012666Z [2021-12-16 13:53:51,390] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn)
2021-12-16T13:53:51.395158219Z [2021-12-16 13:53:51,395] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
2021-12-16T13:53:51.399485772Z [2021-12-16 13:53:51,397] ERROR Unable to resolve address: zookeeper:2181 (org.apache.zookeeper.client.StaticHostProvider)
2021-12-16T13:53:51.399499707Z java.net.UnknownHostException: zookeeper: Name or service not known
2021-12-16T13:53:51.399503169Z at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
2021-12-16T13:53:51.399506400Z at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
2021-12-16T13:53:51.399509510Z at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519)
2021-12-16T13:53:51.399512353Z at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
2021-12-16T13:53:51.399531020Z at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
2021-12-16T13:53:51.399534098Z at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
2021-12-16T13:53:51.399537044Z at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
2021-12-16T13:53:51.399540881Z at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2021-12-16T13:53:51.399544771Z at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2021-12-16T13:53:51.399548877Z at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2021-12-16T13:53:51.399553025Z at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1207)
2021-12-16T13:53:51.406655054Z [2021-12-16 13:53:51,406] WARN Session 0x0 for sever zookeeper:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
2021-12-16T13:53:51.406696302Z java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
2021-12-16T13:53:51.406703099Z at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:78)
2021-12-16T13:53:51.406707676Z at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41)
2021-12-16T13:53:51.406711700Z at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1161)
2021-12-16T13:53:51.406715631Z at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1210)
2021-12-16T13:53:52.508636206Z [2021-12-16 13:53:52,508] ERROR Unable to resolve address: zookeeper:2181 (org.apache.zookeeper.client.StaticHostProvider)
2021-12-16T13:53:52.508665462Z java.net.UnknownHostException: zookeeper
.gitlab-ci.yml:
.zoo_service: &zoo_service
name: zookeeper:latest
alias: zookeeper
.kafka_service: &kafka_service
name: bitnami/kafka:latest
alias: kafka
faust:
variables:
ALLOW_ANONYMOUS_LOGIN: "yes"
KAFKA_BROKER_ID: 1
KAFKA_CFG_LISTENERS: "PLAINTEXT://:9092"
KAFKA_CFG_ADVERTISED_LISTENERS: "PLAINTEXT://127.0.0.1:9092"
KAFKA_CFG_ZOOKEEPER_CONNECT: "zookeeper:2181"
ALLOW_PLAINTEXT_LISTENER: "yes"
stage: test
<<: *python_image
services:
- *zoo_service
- *kafka_service
before_script:
- *setup_venv_script
script:
- faust -A runner worker -l info & sleep 15; kill -HUP $!
<<: *load_env
except:
- schedules
I was hoping I'm doing it the right way - sadly there is not many resources I can read about this issue. I understand the issue is between kafka and zookeeper, but I'm not sure how to fix it (Thought this is the correct way). Can even 2 services communicate to each other in CI?
Thanks!

Answer 1

Glancing over the GitLab CI docs about connecting to different services, it mentions a feature flag to allow cross-service communication, so try
faust:
variables:
FF_NETWORK_PER_BUILD: 1
...
services:
...
Also, for Kafka communication, it need to advertise its alias rather than localhost, so change
KAFKA_CFG_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092"