Kafka cluster on muliple nodes with docker-compose - apache-kafka

Question

I'm trying to set up Kafka cluster on 3 nodes, with docker-compose.
node:
version: '3.1'
services:
zookeeper:
image: zookeeper:3.7.0
container_name: zookeeper
hostname: zookeeper
ports:
- 2181:2181
- 2888:2888
- 3888:3888
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=node2:2888:3888;2181 server.3=node3:2888:3888;2181
kafka:
image: wurstmeister/kafka:2.13-2.7.0
container_name: kafka
hostname: kafka
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: node1:2181,node2:2181,node3:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://node1:9095
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9095
KAFKA_CREATE_TOPICS: "test:4:1"
ports:
- 9095:9095
node:
version: '3.1'
services:
zookeeper:
image: zookeeper:3.7.0
container_name: zookeeper
hostname: zookeeper
ports:
- 2181:2181
- 2888:2888
- 3888:3888
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=node1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=node3:2888:3888;2181
kafka:
image: wurstmeister/kafka:2.13-2.7.0
container_name: kafka
hostname: kafka
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 2
KAFKA_ZOOKEEPER_CONNECT: node1:2181,node2:2181,node3:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://node2:9095
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9095
#KAFKA_CREATE_TOPICS: "test:4:1"
ports:
- 9095:9095
node:
version: '3.1'
services:
zookeeper:
image: zookeeper:3.7.0
container_name: zookeeper
hostname: zookeeper
ports:
- 2181:2181
- 2888:2888
- 3888:3888
environment:
ZOO_MY_ID: 3
ZOO_SERVERS: server.1=node1:2888:3888;2181 server.2=node2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181
kafka:
image: wurstmeister/kafka:2.13-2.7.0
container_name: kafka
hostname: kafka
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 3
KAFKA_ZOOKEEPER_CONNECT: node1:2181,node2:2181,node3:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://node3:9095
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9095
#KAFKA_CREATE_TOPICS: "test:4:1"
ports:
- 9095:9095
When I run it (on respective nodes):
docker-compose -f docker-compose-node1.yml -d
docker-compose -f docker-compose-node2.yml -d
docker-compose -f docker-compose-node3.yml -d
If I check logs, there are no errors. On contrary I can see something like:
INFO Registered broker 1 at path /brokers/ids/1 ...
INFO Registered broker 2 at path /brokers/ids/2 ...
INFO Registered broker 3 at path /brokers/ids/3 ...
Two things that are strange, first on the first node where I try to create a topic, logs stop at
creating topics: test:4:1
Created topic test.
If I set up a single node (localhost) everything sets up just fine. The second thing is that node2 is always elected as controller.
This is what I get with kafkacat:
kafkacat -b node1:9095,node2:9095,node3:9095 -L
Metadata for all topics (from broker 2: node2:9095/2):
3 brokers:
broker 2 at node2:9095 (controller)
broker 3 at node3:9095
broker 1 at node1:9095
1 topics:
topic "test" with 4 partitions:
partition 0, leader 2, replicas: 2, isrs: 2
partition 1, leader 3, replicas: 3, isrs: 3
partition 2, leader 1, replicas: 1, isrs: 1
partition 3, leader 2, replicas: 2, isrs: 2
But if I run:
docker exec -it kafka kafka-console-consumer.sh --bootstrap-server node1:9095,node2:9095,node3:9095 --topic test --from-beginning
I get loads of warnings:
[2021-04-12 14:31:06,848] WARN [Consumer clientId=consumer-console-consumer-95791-1, groupId=console-consumer-95791] Received unknown topic or partition error in ListOffset request for partition test-2 (org.apache.kafka.clients.consumer.internals.Fetcher)
and
[2021-04-12 14:31:06,848] WARN [Consumer clientId=consumer-console-consumer-95791-1, groupId=console-consumer-95791] Received unknown topic or partition error in ListOffset request for partition test-1 (org.apache.kafka.clients.consumer.internals.Fetcher)
Like there's no partition 1 and 2.
If I try to produce a message:
docker exec -it kafka-tn kafka-console-producer.sh --bootstrap-server tn00:9095,tn01:9095,tn02:9095 --topic test
I get loads of errors:
[2021-04-12 14:40:33,273] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 41 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
Am I doing something crazy here or it's just some minor misconfiguration?