kafka-connect consumer fails with bootstrap broker disconnected - apache-kafka

Question

I have deployed kafka-connect cluster and have also did the right configuration for all connect properties but still my configuration fails to consume messages and fails with error.
bootstrap broker backbone.redpanda.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
my kafka-connect deployment looks as below:
apiVersion: apps/v1
kind: Deployment
metadata:
name: kafka-connect
namespace: s3-connector
spec:
replicas: 1
selector:
matchLabels:
app: kafka-connect
template:
metadata:
labels:
app: kafka-connect
spec:
containers:
- name: kafka-connect
image: confluentinc/cp-kafka-connect:6.1.9
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8083
securityContext:
runAsUser: 0
env:
- name: CONNECT_BOOTSTRAP_SERVERS
value: "backbone.redpanda.svc.cluster.local:9092"
- name: CONNECT_KAFKA_HEAP_OPTS
value: "-Xms256M -Xmx2G"
- name: CONNECT_OFFSETS_TOPIC_REPLICATION_FACTOR
value: "1"
- name: CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR
value: "1"
- name: CONNECT_STATUS_STORAGE_REPLICATION_FACTOR
value: "1"
# - name: CONNECT_LOG4J_ROOT_LOGLEVEL
# value: "DEBUG"
- name: CONNECT_REST_PORT
value: "8083"
- name: CONNECT_GROUP_ID
value: "kafka-connect-worker"
- name: CONNECT_CONFIG_STORAGE_TOPIC
value: "kafka-connect-configs"
- name: CONNECT_OFFSET_STORAGE_TOPIC
value: "kafka-connect-offsets"
- name: CONNECT_STATUS_STORAGE_TOPIC
value: "kafka-connect-statuses"
- name: CONNECT_REST_ADVERTISED_HOST_NAME
value: "kafka-connect:8083"
- name: CONNECT_KEY_CONVERTER
value: "org.apache.kafka.connect.json.JsonConverter"
- name: CONNECT_VALUE_CONVERTER
value: "org.apache.kafka.connect.json.JsonConverter"
- name: CONNECT_PLUGIN_PATH
value: "/usr/share/java,/usr/share/confluent-hub-components,/data/connect-jars"
- name: AWS_ACCESS_KEY_ID
value: "---"
- name: AWS_SECRET_ACCESS_KEY
value: "---"
- name: CONNECT_LOG4J_LOGGERS
value: "org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR"
- name: CONNECT_SECURITY_PROTOCOL
value: SSL
- name: CONNECT_SSL_ENABLED_PROTOCOLS
value: "TLSv1.2"
- name: CONNECT_SSL_TRUSTSTORE_LOCATION
value: "/var/private/ssl/truststore.jks"
- name: CONNECT_SSL_TRUSTSTORE_PASSWORD
value: "backbone"
volumeMounts:
- name: connect-config-props
mountPath: /home/appuser
- name: h2-bundle
mountPath: /etc/tls-bundle
- mountPath: /var/private/ssl
name: tlscert
command:
- bash
- -c
- |
echo "Installing Connector"
confluent-hub install --no-prompt confluentinc/kafka-connect-s3:10.4.0
confluent-hub install --no-prompt mdrogalis/voluble:0.1.0
#
echo "Launching Kafka Connect worker"
/etc/confluent/docker/run &
#
sleep infinity
volumes:
- name: connect-config-props
configMap:
name: connect-config
- name: h2-bundle
configMap:
name: h2-bundle
- name: truststore-thing
configMap:
name: client-cert
- name: tlscert
secret:
defaultMode: 420
items:
- key: truststore.jks
path: truststore.jks
secretName: redpanda-client-cert
Properties config map:
apiVersion: v1
kind: ConfigMap
metadata:
name: connect-config
namespace: s3-connector
data:
connect.properties: |-
bootstrap.servers=backbone.redpanda.svc.cluster.local:9092
connector.class=io.confluent.connect.s3.S3SinkConnector
tasks.max=1
topics=demo
s3.bucket.name=kafka-connect-s3-demo-devops
s3.region=eu-west-1
flush.size=3
storage.class=io.confluent.connect.s3.storage.S3Storage
format.class=io.confluent.connect.s3.format.json.JsonFormat
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
schema.compatibility=NONE
security.protocol=SSL
ssl.enabled.protocols=TLSv1.2
ssl.truststore.location=/var/private/ssl/truststore.jks
ssl.truststore.password=backbone
consumer.bootstrap.servers=backbone.redpanda.svc.cluster.local:9092
consumer.security.protocol=SSL
consumer.ssl.truststore.location=/var/private/ssl/truststore.jks
consumer.ssl.truststore.password=backbone
If I explicitely define the properties, it works but doesn't takes from the mounted properties.
Works fine:
kafka-console-consumer --bootstrap-server backbone.redpanda.svc.cluster.local:9092 --topic demo --from-beginning --consumer.config client.properties
Expected to work:
kafka-console-consumer --bootstrap-server backbone.redpanda.svc.cluster.local:9092 --topic demo --from-beginning
Same issue while connecting to s3 sink connector too:
curl -i -X PUT -H "Accept:application/json" \
-H "Content-Type:application/json" http://localhost:8083/connectors/confluentinc-kafka-connect-s3/config \
-d '
{
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
"key.converter":"org.apache.kafka.connect.storage.StringConverter",
"tasks.max": "1",
"topics": "demo",
"s3.region": "eu-west-1",
"s3.bucket.name": "kafka-connect-s3-demo-devops",
"flush.size": "1",
"storage.class": "io.confluent.connect.s3.storage.S3Storage",
"format.class": "io.confluent.connect.s3.format.json.JsonFormat",
"partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
}
'

Answer 1

connect.properties doesn't do anything.
CONNECT_BOOTSTRAP_SERVERS is the only thing that is affecting your error, and if backbone Service does not exist in the redpanda namespace, or that namespace does not have a NetworkPolicy to allow for connections from pods in the s3-connector namespace, then it will be unable to connect.