S3SinkConnector: Issue while consuming compressed data from Kafka - apache-kafka

Question

I have data in Kafka Topic which is Avro serialised and compressed using zstd codec. To transfer this data to S3 I have created a S3SinkConnector with below config -
{
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
"s3.region": "ap-south-1",
"topics.dir": "0/test_debezium_sept_12_mon_5/public/test_table",
"flush.size": "10000",
"tasks.max": "1",
"s3.part.size": "67108864",
"timezone": "Asia/Calcutta",
"rotate.interval.ms": "60000",
"locale": "en_GB",
"format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"s3.bucket.name": "zeta-aws-aps1-metis-0-s3-pvt",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"partition.duration.ms": "86400000",
"schema.compatibility": "NONE",
"topics": "cdc_test_debezium_sept_12_mon_5.public.test_table",
"parquet.codec": "gzip",
"connect.meta.data": "true",
"value.converter.schema.registry.url": {{url}},
"partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
"name": "cdc_test_debezium_sept_12_mon_5.public.test_table_cdc_zeta-aws-aps1-metis-0-s3-pvt_ap-south-1_sink",
"storage.class": "io.confluent.connect.s3.storage.S3Storage",
"path.format": "'date'=YYYY-MM-dd",
"rotate.schedule.interval.ms": "180000",
"timestamp.extractor": "RecordField",
"key.converter.schema.registry.url": "{{url}}",
"timestamp.field": "cdc_source_ts_ms"
}
The above S3SinkConnector fails with following error
Caused by: org.apache.kafka.connect.errors.DataException: Failed to deserialize data for topic cdc_test_debezium_sept_12_mon_5.public.test_table to Avro: \n\tat io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:118)\n\tat org.apache.kafka.connect.storage.Converter.toConnectData(Converter.java:87)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$0(WorkerSinkTask.java:492)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:146)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:180)\n\t... 13 more\nCaused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id 1106\nCaused by: java.io.EOFException
NOTE: If I disable compression on producer/kafka side then S3 connector works fine. Issue is only while enabling compression on Kafka side.