ClassNotFound exception running Kafka Connect S3 Source Connector - apache-kafka

Question

I am evaluating Confluent Kafka S2 Source Connector and stuck with the issues with following stacktrace:
[2020-12-22 15:27:41,636] ERROR WorkerConnector{id=s3-source-connector} Error while starting connector (org.apache.kafka.connect.runtime.WorkerConnect
or)
org.apache.kafka.connect.errors.ConnectException: Failed to get list of folders from S3 bucket - kafka-connect for key path - topics/ and delimiter - /
at io.confluent.connect.s3.source.S3Storage.listFolders(S3Storage.java:286)
at io.confluent.connect.s3.source.S3Storage.getPartitions(S3Storage.java:98)
at io.confluent.connect.storage.partitioner.TimeBasedPartitioner.getPartitions(TimeBasedPartitioner.java:50)
at io.confluent.connect.cloud.storage.source.StorageSourceConnector.doStart(StorageSourceConnector.java:77)
at io.confluent.connect.cloud.storage.source.StorageSourceConnector.start(StorageSourceConnector.java:69)
at org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:111)
at org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:136)
at org.apache.kafka.connect.runtime.WorkerConnector.transitionTo(WorkerConnector.java:196)
at org.apache.kafka.connect.runtime.Worker.startConnector(Worker.java:242)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startConnector(DistributedHerder.java:908)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1300(DistributedHerder.java:110)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder$15.call(DistributedHerder.java:924)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder$15.call(DistributedHerder.java:920)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.SdkClientException: Couldn't initialize a SAX driver to create an XMLReader
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.<init>(XmlResponsesSaxParser.java:123)
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsV2Unmarshaller.unmarshall(Unmarshallers.java:127)
at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsV2Unmarshaller.unmarshall(Unmarshallers.java:117)
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1714)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1434)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1356)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5052)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4992)
at com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:938)
at io.confluent.connect.s3.source.S3Storage.listFolders(S3Storage.java:283)
... 16 more
Caused by: org.xml.sax.SAXException: SAX2 driver class org.apache.xerces.parsers.SAXParser not found
java.lang.ClassNotFoundException: org.apache.xerces.parsers.SAXParser
at org.xml.sax.helpers.XMLReaderFactory.loadClass(XMLReaderFactory.java:230)
at org.xml.sax.helpers.XMLReaderFactory.createXMLReader(XMLReaderFactory.java:191)
at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.<init>(XmlResponsesSaxParser.java:120)
... 37 more
Caused by: java.lang.ClassNotFoundException: org.apache.xerces.parsers.SAXParser
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.xml.sax.helpers.NewInstance.newInstance(NewInstance.java:82)
at org.xml.sax.helpers.XMLReaderFactory.loadClass(XMLReaderFactory.java:228)
... 39 more
Connector config:
{
"name": "source-connector",
"config": {
"connector.class":"io.confluent.connect.s3.source.S3SourceConnector",
"s3.bucket.name":"bucket-test",
"s3.region":"us-west-2",
"tasks.max":"1",
"topics":"migration-topic",
"topics.dir":"topics/events",
"format.class":"io.confluent.connect.s3.format.json.JsonFormat",
"behavior.on.error": "log",
"partitioner.class":"io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
"path.format":"'date'=YYYY-MM-dd/'hour'=HH",
"key.converter":"com.pandadoc.kafka.connect.msgpack.converter.MessagePackConverter",
"key.converter.schemas.enable":"false",
"value.converter":"com.pandadoc.kafka.connect.msgpack.converter.MessagePackConverter",
"value.converter.schemas.enable":"false",
"errors.tolerance": "all",
"errors.deadletterqueue.topic.name": "kafka-connect-dead-letter-queue",
"errors.deadletterqueue.context.headers.enable": true,
"confluent.license":"",
"confluent.topic.bootstrap.servers":"localhost:9092",
"confluent.topic.replication.factor":"3"
}
}
Versions:
[2020-12-22 15:27:41,640] INFO Kafka version: 2.2.2-cp3 (org.apache.kafka.common.utils.AppInfoParser)
[2020-12-22 15:27:41,640] INFO Kafka commitId: 602b2e2e105b4d34 (org.apache.kafka.common.utils.AppInfoParser)
It could be a JDK bug: https://bugs.openjdk.java.net/browse/JDK-8015099.
It has been fixed in JDK 9+.
Confluent docker image confluentinc/cp-kafka-connect:5.2.4 uses JDK8:
openjdk version "1.8.0_172"
OpenJDK Runtime Environment (Zulu 8.30.0.1-linux64) (build 1.8.0_172-b01)
OpenJDK 64-Bit Server VM (Zulu 8.30.0.1-linux64) (build 25.172-b01, mixed mode)
Any other ideas on what could be wrong?

Answer 1

I've sorted the issue out 😅
It turned out the JDK bug that caused the kind of behavior.
There is an interoperability table for Kafka Connect version and Kafka here hence there are two options:
Tweak docker Kafka Connect image by installing JDK9+
Bump up Kafka Connect to 6.x (if Kafka version allows) that uses JDK11.