Use Kafka Connect with jcustenborder / kafka-connect-twitter - apache-kafka

Question

I'm trying to use Kafka Connect with kafka-connect-twitter from jcustenborder in Github to introduce Twitter tweets into Kafka. The instructions say:
mvn clean package
export CLASSPATH="$(find target/ -type f -name '*.jar'| grep '\-package' | tr '\n' ':')"
$CONFLUENT_HOME/bin/connect-standalone connect/connect-avro-docker.properties config/TwitterSourceConnector.properties
The export CLASSPATH line in fact does not work and returns nothing when run. The connect avro docker properties file seems to want to use the jars available in target/kafka-connect-target/usr/share/kafka-connect after running mvn clean package in the kafka-connect-twitter repository.
When I run
connect-standalone.sh connect-avro-docker.properties TwitterSourceConnector.properties in the directory where these two .properties are present, since connect-standalone.sh is in the path, I get the error:
2021-11-12 18:22:05,267] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectStandalone:126)
org.apache.kafka.common.config.ConfigException: Invalid value io.confluent.connect.avro.AvroConverter for configuration key.converter: Class io.confluent.connect.avro.AvroConverter could not be found.
at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:744)
at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:490)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:483)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:129)
at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:452)
at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.<init>(StandaloneConfig.java:42)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:80)
It's not finding the jar where the AvroConverter is.
I'm using Kafka 2.13-2.8.0 and the 0.3.34 jcustenborder kafka-connect-twitter.
I see nowhere possible jars where the AvroConverter might be, in the Kafka distribution. Does it include Kafka Connect?
Note that I'm using an install of Kafka in an iMac, I'm not using Docker for running Kafka.
EDIT:
instead of using the avro properties file, I'm using the connect-standalone.properties. Although the log says it has loaded the guava jar:
INFO Loading plugin from: /Users/paupaches/dev/books/kafkabeginnerscourse/kafka-connect/connectors/kafka-connectors-twitter/guava-30.1.1-jre.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:246)
[2021-11-13 10:50:22,992] INFO Registered loader: PluginClassLoader{pluginLocation=file:/Users/paupaches/dev/books/kafkabeginnerscourse/kafka-connect/connectors/kafka-connectors-twitter/guava-30.1.1-jre.jar} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:269)
I get the error
ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectStandalone:126)
java.lang.NoClassDefFoundError: com/google/common/collect/Multimap
I am using openjdk 17.

Answer 1

It is not necessary to export the CLASSPATH, using the "connectors" path in the plugin.path property in the file connect-standalone.properties is enough.
The "connectors" directory contains the kafka-connect-twitter directory which contains all the jars generated when running "mvn clean package" in the connector working copy.
In the end I used openjdk 8, although 17 is also installed in my Mac.