Kafka HDFS Sink Connector Protobuf not being written - apache-kafka

Question

I am trying to use the Kafka HDFS 3 sink connector to write protobuf binary files to the HDFS. However, the connector keeps writing avro files.
I have setup my sink connector with the following config
{
"name": "hdfs3-connector-test",
"config": {
"connector.class": "io.confluent.connect.hdfs3.Hdfs3SinkConnector",
"tasks.max": "1",
"topics": "testproto",
"hdfs.url": "hdfs://10.8.0.1:9000",
"flush.size": "3",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "io.confluent.connect.protobuf.ProtobufConverter",
"value.converter.schema.registry.url":"http://10.8.0.1:8081",
"confluent.topic.bootstrap.servers": "10.8.0.1:9092",
"confluent.topic.replication.factor": "1",
"key.converter.schemas.enable": "true",
"value.converter.schemas.enable": "true"
}
}
as you can see I am using the ProtobufConverter for the value converter, and the plugin is installed. (Is the ProtobufConverter converting to Avro file format?).
I also registered my schema and sent data to the topic using the following Java file:
package app;
import java.util.Properties;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.clients.producer.KafkaProducer;
import test.Test.*;
public class App
{
public static void main( String[] args )
{
try {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "10.8.0.1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer.class.getName());
props.put("schema.registry.url", "http://10.8.0.1:8081");
KafkaProducer<String, MyMsg> producer = new KafkaProducer<String, MyMsg>(props);
String topic = "testproto";
String key = "testkey";
MyMsg m = MyMsg.newBuilder().setF1("testing").build();
ProducerRecord<String, MyMsg> record = new ProducerRecord<String, MyMsg>(topic, key, m);
producer.send(record).get();
producer.close();
} catch (Exception e) {
System.out.println(e.toString());
}
}
}
here is my proto file
syntax = "proto3";
package test;
message MyMsg {
string f1 = 1;
}
So my question is, is this correct? Can I only write Avro files to the HDFS using this connector? Or is my configuration incorrect and I should be expecting protobuf files in the HDFS?

Answer 1

You need to set format.class config,
format.class is
The format class to use when writing data to the store. Format classes implement the io.confluent.connect.storage.format.Format interface.
Type: class
Default: io.confluent.connect.hdfs3.avro.AvroFormat
Importance: high
These classes are available by default:
io.confluent.connect.hdfs3.avro.AvroFormat
io.confluent.connect.hdfs3.json.JsonFormat
io.confluent.connect.hdfs3.parquet.ParquetFormat
io.confluent.connect.hdfs3.string.StringFormat
https://docs.confluent.io/kafka-connect-hdfs3-sink/current/configuration_options.html#hdfs3-config-options