Debezium error when connecting to kafka multi-broker in docker swarm - apache-kafka

Question

when I set up my Swarm with this stack: Kafka(multi-broker), zookeeper, debezium. Kafka and zookeeper are working, can create topic, consumer and producer, but debezium show error: org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties. I'm not modifying anything, just default config as docker-stack below:
version: '3.6'
services:
zoo:
image: wurstmeister/zookeeper
ports:
- '2181:2181'
volumes:
- zoo-data:/tmp/zookeeper
deploy:
replicas: 1
placement:
constraints:
- node.labels.type==zoo
kafka:
image: wurstmeister/kafka:latest
ports:
- target: 9094
published: 9094
protocol: tcp
mode: host
environment:
HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2"
KAFKA_ZOOKEEPER_CONNECT: zoo:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
#KAFKA_CREATE_TOPICS: "Topic1:1:2,Topic2:1:1:compact"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- kafka-data:/tmp/kafka-logs
deploy:
mode: global
placement:
constraints:
- node.labels.name==kafka
depends_on:
- zoo
debezium:
image: debezium/connect:0.8
hostname: connect
ports:
- '8083:8083'
environment:
BOOTSTRAP_SERVERS: kafka:9094
GROUP_ID: 1
CONFIG_STORAGE_TOPIC: my_connect_configs
OFFSET_STORAGE_TOPIC: my_connect_offsets
deploy:
placement:
constraints:
- node.labels.type==dbz
depends_on:
- kafka
volumes:
kafka-data:
zoo-data:
When I check docker services log debezium, it show error
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | 2018-08-03 04:33:27,034 ERROR || Stopping due to error [org.apache.kafka.connect.cli.ConnectDistributed]
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:258)
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | ... 2 more
shippo_kafka_debezium.1.5l1yhz27r6p2#kafka1 | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
Can anyone show me how to fix this error, I'm new with this stack so, during a few days research, I can't figure out it. Thank you so much !

Answer 1

You can change this line HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2" to HOSTNAME_COMMAND: "docker info | grep 'Node Address:' | cut -d' ' -f 4"
or just you can use this docker-compose file
version: '3.2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
deploy:
mode: global
volumes:
- /shared_data/zoo1/data:/data
- /shared_data/zoo1/datalog:/datalog
environment:
ZOO_MY_ID: 1
ZOO_PORT: 2181
ZOO_SERVERS: server.1=zookeeper:2888:3888
kafka:
image: wurstmeister/kafka:latest
ports:
- target: 9094
published: 9094
protocol: tcp
mode: host
deploy:
mode: global
environment:
HOSTNAME_COMMAND: "docker info | grep 'Node Address:' | cut -d' ' -f 4"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /shared_data/kafka:/var/lib/kafka/data
depends_on:
- zookeeper