Create Kafka-Connect cluster with Docker Compose to be used by ksqlDB - apache-kafka

Question

What I essentially try to do is to have multiple Kafka Connect instances with Docker Compose. I want ksqlDB to use this cluster. For now, they all run on a single machine, but eventually I want to deploy this to a multi-node environment. My problem is that ksqlDB apparently can't find the Kafka Connect cluster. There is the KSQL_KSQL_CONNECT_URL, which stands for the URL of a single Kafka Connect instance. Not providing this variable results in the default value, which is localhost:8083.
I found this docker-compose file, which I think does what I want to do: ksqlDB and multiple Kafka Connect instances. Unfortunately, it didn't help me that much, since it uses an old version of KSQL Server. Here is my docker-compose file:
---
version: '3'
services:
ksqldb-server-connect-test:
image: confluentinc/ksqldb-server:0.15.0
hostname: ksqldb-server-connect-test
container_name: ksqldb-server-connect-test
#ports:
# - "8088:8088"
network_mode: "host"
environment:
KSQL_KSQL_SERVICE_ID: "default_"
KSQL_LISTENERS: http://0.0.0.0:8088
KSQL_BOOTSTRAP_SERVERS: localhost:9092
KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
KSQL_KSQL_SCHEMA_REGISTRY_URL: http://localhost:8081
#KSQL_KSQL_CONNECT_URL: http://localhost:8083
ksqldb-cli-connect-test:
image: confluentinc/ksqldb-cli:0.15.0
container_name: ksqldb-cli-connect-test
network_mode: "host"
depends_on:
- ksqldb-server-connect-test
entrypoint: /bin/sh
tty: true
schema-registry-connect-test:
image: confluentinc/cp-schema-registry:6.0.1
container_name: schema-registry-connect-test
network_mode: "host"
#ports:
# - "8081:8081"
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: localhost:9092
restart: always
kafka-connect-1:
image: confluentinc/cp-kafka-connect-base:6.0.1
container_name: kafka-connect-1
network_mode: "host"
environment:
CONNECT_BOOTSTRAP_SERVERS: "localhost:9092"
CONNECT_REST_PORT: 8082
CONNECT_GROUP_ID: kafka-connect-test
CONNECT_CONFIG_STORAGE_TOPIC: _connect-configs-test
CONNECT_OFFSET_STORAGE_TOPIC: _connect-offsets-test
CONNECT_STATUS_STORAGE_TOPIC: _connect-status-test
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: 'http://localhost:8081'
CONNECT_REST_ADVERTISED_HOST_NAME: "localhost"
CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n"
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_OFFSET_STORAGE_PARTITIONS: "25"
CONNECT_STATUS_STORAGE_PARTITIONS: "5"
CONNECT_PLUGIN_PATH: /usr/share/java,/usr/share/confluent-hub-components,/data/connect-jars
volumes:
- $PWD/data/connect-jars/:/usr/share/java/kafka-connect-jdbc/jars/
- $PWD/jmx:/usr/app/
kafka-connect-2:
image: confluentinc/cp-kafka-connect-base:6.0.1
container_name: kafka-connect-2
network_mode: "host"
environment:
CONNECT_BOOTSTRAP_SERVERS: "localhost:9092"
CONNECT_REST_PORT: 8084
CONNECT_GROUP_ID: kafka-connect-test
CONNECT_CONFIG_STORAGE_TOPIC: _connect-configs-test
CONNECT_OFFSET_STORAGE_TOPIC: _connect-offsets-test
CONNECT_STATUS_STORAGE_TOPIC: _connect-status-test
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: 'http://localhost:8081'
CONNECT_REST_ADVERTISED_HOST_NAME: "localhost"
CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n"
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_OFFSET_STORAGE_PARTITIONS: "25"
CONNECT_STATUS_STORAGE_PARTITIONS: "5"
CONNECT_PLUGIN_PATH: /usr/share/java,/usr/share/confluent-hub-components,/data/connect-jars
volumes:
- $PWD/data/connect-jars/:/usr/share/java/kafka-connect-jdbc/jars/
- $PWD/jmx:/usr/app/
Note that I use network_mode: "host" because the Kafka cluster itself does not run in a Docker container, so this eases the communication to Kafka in my case.
Does anybody have an idea or a solution on how to get ksqlDB connected to a Kafka Connect cluster using only docker-compose?

Answer 1

what I need to achieve is fault tolerance.
OK, so what you need is >1 Kafka Connect worker, within a single Kafka Connect group. This is what you've got with your configuration of the same storage topics and group.id 👍
So the question is how to get ksqlDB to connect to a cluster of Kafka Connect workers. Since Kafka Connect uses Kafka itself to hold configuration, it doesn't matter which worker it connects to. ksql.connect.url (and thus KSQL_KSQL_CONNECT_URL environment variable in docker) is the correct way to do this, but it's not clear from the docs if you can specify multiple values.
If you can't then I'm guessing you'd need to stick a stateless load balancer in front of the workers and point ksqlDB at that.
Also, the hostname is going to be the name of the container (kafka-connect-1 / kafka-connect-2), not localhost.