Kafka source connector for postgres - Day0 load - postgresql

Question

I am looking for a Kafka source connector for Day0 load from Postgres to Kafka.
Came across Debezium postgres connector.
Docker image,
debezium/connect:1.4
docker run -it --rm --name postgres-connect -p 8083:8083 -e BOOTSTRAP_SERVERS=host1:8080 -e GROUP_ID=test-1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses debezium/connect:1.4
How to pass the postgres host details and kafka sasl config?
Any help would be appreciated.

Answer 1

1. SASL configuration
1.1. In common case you need to add the following properties to your connect-distributed.properties:
sasl.mechanism=PLAIN
security.protocol=SASL_PLAINTEXT
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="connect" \
password="connect-secret";
producer.sasl.mechanism=PLAIN
producer.security.protocol=SASL_PLAINTEXT
producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="connect" \
password="connect-secret";
consumer.sasl.mechanism=PLAIN
consumer.security.protocol=SASL_PLAINTEXT
consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="connect" \
password="connect-secret";
Source: StackOverflow answer "ACL configuration in Kafka connect is not working"
Reference: Kafka Connect Security docs
1.2. For debezium/connect docker image you can try to pass SASL config directly via environment variables (using these transformation steps):
docker run -it --rm --name postgres-connect -p 8083:8083 \
-e BOOTSTRAP_SERVERS=host1:8080 -e GROUP_ID=test-1 \
-e CONFIG_STORAGE_TOPIC=my_connect_configs \
-e OFFSET_STORAGE_TOPIC=my_connect_offsets \
-e STATUS_STORAGE_TOPIC=my_connect_statuses \
-e CONNECT_SASL_MECHANISM=PLAIN \
-e CONNECT_SECURITY_PROTOCOL=SASL_PLAINTEXT \
-e CONNECT_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="connect" password="connect-secret"; \
-e CONNECT_PRODUCER_SASL_MECHANISM=PLAIN \
-e CONNECT_PRODUCER_SECURITY_PROTOCOL=SASL_PLAINTEXT \
-e CONNECT_PRODUCER_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="connect" password="connect-secret"; \
-e CONNECT_CONSUMER_SASL_MECHANISM=PLAIN \
-e CONNECT_CONSUMER_SECURITY_PROTOCOL=SASL_PLAINTEXT \
-e CONNECT_CONSUMER_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="connect" password="connect-secret"; \
debezium/connect:1.4
2. PostgreSQL host configaration
Host details should be passed via Kafka Connect REST API using connector config:
curl -i -X PUT -H "Content-Type:application/json" \
http://localhost:8083/connectors/debezium_postgres_source/config \
-d '{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "source-db",
"database.port": "5432",
"database.user": "postgresusersource",
"database.password": "postgrespw",
"database.dbname" : "sourcedb",
"database.server.name": "dbserver1"
}'