Kafka to hdfs3 sink Missing required configuration "confluent.topic.bootstrap.servers" which has no default value - apache-kafka

Question

Status
My HDFS was installed via ambari, HDP.
I'm Currently trying to load kafka topics into HDFS sink. Kafka and HDFS was installed in the same machine x.x.x.x.
I didn't change much stuff from the default settings, except some port that according to my needs.
Here is how i execute kafka:
/usr/hdp/3.1.4.0-315/kafka/bin/connect-standalone.sh /etc/kafka/connect-standalone.properties /etc/kafka-connect-hdfs/quickstart-hdfs.properties
Inside connect-standalone.properties
bootstrap.servers=x.x.x.x:6667
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
inside quickstart-hdfs.properties
name=hdfs-sink
#connector.class=io.confluent.connect.hdfs.HdfsSinkConnector
connector.class=io.confluent.connect.hdfs3.Hdfs3SinkConnector
tasks.max=1
topics=test12
hdfs.url=hdfs://x.x.x.x:9000
flush.size=3
Here are the results i get when excute it:
[2020-06-23 03:26:00,918] INFO Started o.e.j.s.ServletContextHandler#71d9cb05{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:855)
[2020-06-23 03:26:00,928] INFO Started http_8083#329a1243{HTTP/1.1,[http/1.1]}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:292)
[2020-06-23 03:26:00,928] INFO Started #10495ms (org.eclipse.jetty.server.Server:410)
[2020-06-23 03:26:00,928] INFO Advertised URI: http://x.x.x.x:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:267)
[2020-06-23 03:26:00,928] INFO REST server listening at http://x.x.x.x:8083/, advertising URL http://x.x.x.x:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:217)
[2020-06-23 03:26:00,928] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:55)
[2020-06-23 03:26:00,959] ERROR Failed to create job for quickstart-hdfs.properties (org.apache.kafka.connect.cli.ConnectStandalone:102)
[2020-06-23 03:26:00,960] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:113)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 1 error(s):
Missing required configuration "confluent.topic.bootstrap.servers" which has no default value.
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:79)
at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:66)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:110)
Caused by: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 1 error(s):
Missing required configuration "confluent.topic.bootstrap.servers" which has no default value.
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
at org.apache.kafka.connect.runtime.AbstractHerder.maybeAddConfigErrors(AbstractHerder.java:415)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:189)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:107)
[2020-06-23 03:26:00,961] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:65)
[2020-06-23 03:26:00,961] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:223)
[2020-06-23 03:26:00,964] INFO Stopped http_8083#329a1243{HTTP/1.1,[http/1.1]}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:341)
[2020-06-23 03:26:00,965] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:167)
[2020-06-23 03:26:00,972] INFO Stopped o.e.j.s.ServletContextHandler#71d9cb05{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:1045)
[2020-06-23 03:26:00,974] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:241)
[2020-06-23 03:26:00,974] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:95)
[2020-06-23 03:26:00,974] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:184)
[2020-06-23 03:26:00,974] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:67)
[2020-06-23 03:26:00,975] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:205)
[2020-06-23 03:26:00,975] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:112)
[2020-06-23 03:26:00,975] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:70)
I'm really new in kafka and hdfs envinronment. Any suggestion and help will be appreciated so much. Thank you
edit:
i've add my connect-standalone.properties into
bootstrap.servers=x.x.x.x:6667
confluent.license=
confluent.topic.bootstrap.server=x.x.x.x:6667
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
nothing changes it still showing the same log error
EDIT SOLVING (Thanks to Robin)
quickstart-hdfs.properties
name=hdfs-sink
connector.class=io.confluent.connect.hdfs3.Hdfs3SinkConnector
tasks.max=1
topics=test12
hdfs.url=hdfs://x.x.x.x:8020
flush.size=3
confluent.license=
confluent.topic.bootstrap.servers=x.x.x.x:6667
connect-standalone.properties
bootstrap.servers=x.x.x.x:6667
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/usr/local/share/java,/usr/local/share/kafka/plugins,/opt/connectors,
plugin.path=/usr/share/java,/usr/share/confluent-hub-components

Answer 1

Here's the error:
Missing required configuration "confluent.topic.bootstrap.servers" which has no default value.
The problem is that you've taken the config for the HDFS Sink connector, and changed the connector for a different one (HDFS 3 Sink), and this one has different configuration requirements.
You can follow the quickstart for the HDFS 3 sink connector, or fix your existing configuration by adding
confluent.topic.bootstrap.servers=10.64.2.236:6667
confluent.topic.replication.factor=1
Note: in your example you missed the s from confluent.topic.bootstrap.servers which is why it didn't work