I am new to Kafka and I want to see if I can sync MongoDb data with another system using Kafka.
My set up:
I am running AWS MSK Cluster and I have created an EC2 instance with Kafka client manually.
I have added MongoDB Kafka Connect Plugin to /usr/local/share/kafka/plugins.
I am running Kafka connect and can see that it loads the plugin
./bin/connect-standalone.sh ./config/connect-standalone.properties /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/etc/MongoSourceConnector.properties
[2020-10-17 13:57:22,304] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:264)
[2020-10-17 13:57:22,305] INFO Added plugin 'com.mongodb.kafka.connect.MongoSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
[2020-10-17 13:57:22,305] INFO Added plugin 'com.mongodb.kafka.connect.MongoSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:193)
Unpacked plugin has this structure
Archive: mongodb-kafka-connect-mongodb-1.3.0.zip
creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/
creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/etc/
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/etc/MongoSourceConnector.properties
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/etc/MongoSinkConnector.properties
creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/doc/
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/doc/README.md
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/doc/LICENSE.txt
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/manifest.json
creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/lib/
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/lib/mongo-kafka-1.3.0-all.jar
creating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/assets/
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/assets/mongodb-leaf.png
inflating: /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/assets/mongodb-logo.png
This plugin is from confluent page, I have also tried downloading it from Maven page.
The problem is when I run Kafka Connect it fails because plugin is missing a Java dependency.
[2020-10-17 13:57:24,898] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:121)
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/apache/avro/Schema
at com.mongodb.kafka.connect.source.MongoSourceConfig.createConfigDef(MongoSourceConfig.java:591)
at com.mongodb.kafka.connect.source.MongoSourceConfig.<clinit>(MongoSourceConfig.java:293)
at com.mongodb.kafka.connect.MongoSourceConnector.config(MongoSourceConnector.java:91)
at org.apache.kafka.connect.connector.Connector.validate(Connector.java:129)
at com.mongodb.kafka.connect.MongoSourceConnector.validate(MongoSourceConnector.java:51)
at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:313)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:192)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:115)
Caused by: java.lang.NoClassDefFoundError: org/apache/avro/Schema
... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.avro.Schema
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 8 more
My impression was that plugin should look for dependencies in the jar file /usr/local/share/kafka/plugins/mongodb-kafka-connect-mongodb-1.3.0/lib/mongo-kafka-1.3.0-all.jar not in the Java SDK.
What am I missing in this set up?
A quick look at this should tell you if the error is correct...
jar -tf mongo-kafka-1.3.0-all.jar | grep avro
If that JAR doesn't bundle Avro itself, then MSK very likely doesn't include Avro like Confluent Platform does (which I assume Mongo bundled their connector primarily for). At least, Avro is not a dependency of Apache Kafka, so that would explain that error.
You will need to download the Avro JAR and place it on your Kafka Connect Classpath (or at least in that lib folder)
I faced same issue, when running on my local. I downloaded the jar(mongo-kafka-connect-1.6.0-confluent.jar) from confluent platform which does not provide uber jar anymore.
So I searched for uber jar and found below site from where, I could download uber jar(select all in Download dropdown) and that resolved the issue.
https://search.maven.org/search?q=a:mongo-kafka-connect
Related
I'm able to stream data out of a database into a Kafka topic using debezium and Kafka connect. I can do so successfully using org.apache.kafka.connect.json.JsonConverter or org.apache.kafka.connect.storage.StringConverter.
However, if I try to output using
value.converter=io.confluent.connect.protobuf.ProtobufConverter
value.converter.schema.registry.url=http://my-schema-registry
I get
java.lang.NoClassDefFoundError: io/confluent/kafka/schemaregistry/client/SchemaRegistryClient
java.lang.ClassNotFoundException: io.confluent.kafka.schemaregistry.client.SchemaRegistryClient
Alternatively, if I use
value.converter=com.blueapron.connect.protobuf.ProtobufConverter
value.converter.protoClassName=myClassName
I get
org.apache.kafka.connect.errors.ConnectException: Proto class myClassName not found in the classpath
Any ideas? I've tried putting the kafka-schema-registry-client jar and a jar containing myClassName in the plugin directory, but neither get picked up as plugins.
I was using a Kafka installation downloaded from Apache. I switched to the complete Confluent Platform installation from Confluent and it worked.
While trying to read a Kafka topic in a InfoSpfhere Job, I got the error
Kafka_Customer: java.lang.NoClassDefFoundError: org.apache.kafka.clients.consumer.ConsumerRebalanceListener
at com.ibm.is.cc.kafka.runtime.KafkaProcessor.validateConfiguration (KafkaProcessor.java: 145)
at com.ibm.is.cc.javastage.connector.CC_JavaAdapter.initializeProcessor (CC_JavaAdapter.java: 1008)
at com.ibm.is.cc.javastage.connector.CC_JavaAdapter.getSerializedData (CC_JavaAdapter.java: 705)
Kafka_Customer: Java runtime exception occurred: java.lang.NoClassDefFoundError: org.apache.kafka.clients.consumer.ConsumerRebalanceListener (com.ibm.is.cc.kafka.runtime.KafkaProcessor::validateConfiguration, file KafkaProcessor.java, line 145)
I should add the jar file, which is missing, but where and how can I see which version is nedeed?. I could'n find anything after a lot of googling.
kafka-clients JAR versions are backwards compatible down to 0.10.2, however I would assume that the Kafka processors should have this, so you may want to reach out to IBM support
I am not able to get kafka-connect-jdbc
working with Kafka 0.10.1 on HDInsight Cluster. Here are the steps I went thru so far:
I cloned the repo and ran mvn install and after struggling with the dependencies (not SNAPSHOTS), I got the jar.
moved it to ./libs and I was able to see io.confluent.connect.jdbc.JdbcSourceConnector & io.confluent.connect.jdbc.JdbcSinkConnector when hitting GET /connector-plugins.
I created a source connector to Azure SQLServer & a topic.
I am getting the following error once, the source connector is created:
[2018-02-14 15:46:16,960] ERROR Error while starting connector azure-source-connector-test (org.apache.kafka.connect.runtime.WorkerConnector:108)
java.lang.NoSuchFieldError: SYSTEM
at io.confluent.connect.jdbc.source.JdbcSourceConnectorConfig.(JdbcSourceConnectorConfig.java:184)
at io.confluent.connect.jdbc.JdbcSourceConnector.start(JdbcSourceConnector.java:69)
at org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:100)
at org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:125)
at org.apache.kafka.connect.runtime.WorkerConnector.transitionTo(WorkerConnector.java:182)
at org.apache.kafka.connect.runtime.Worker.startConnector(Worker.java:165)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startConnector(DistributedHerder.java:773)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startWork(DistributedHerder.java:747)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.handleRebalanceCompleted(DistributedHerder.java:708)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:204)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:174)
at java.lang.Thread.run(Thread.java:748)
I tried different versions/branches of kafka-connect-jdbc but all trials end with the same error.
This question
has same issue, so I tried to force the build to use connect-api-0.10.1.2.6.2.3-1.jar via systemPath in pom.xml but if the build goes thru, it still has the same issue.
Any idea?
The instructions for running kafka from the quick start page are not working for me.
http://kafka.apache.org/07/quickstart.html
Kafka builds fine
05:55:01/kafka-0.8.1-src:58 $sbt package
[info] Set current project to kafka-0-8-1-src (in build file:/shared/kafka-0.8.1-src/)
[info] Packaging /shared/kafka-0.8.1-src/target/scala-2.10/kafka-0-8-1-src_2.10-0.1-SNAPSHOT.jar ...
[info] Done packaging.
[success] Total time: 0 s, completed Apr 17, 2014 5:55:07 AM
But does not run fine..
05:55:07/kafka-0.8.1-src:59 $bin/zookeeper-server-start.sh config/zookeeper.properties
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/zookeeper/server/quorum/QuorumPeerMain
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.server.quorum.QuorumPeerMain
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Similar errors occur for kafka-server-start.sh and all the other scripts inside bin
You downloaded kafka-0.8.1-src.tgz from download page. The instructions on quickstart link are meant for Binary download . Download one from the Binary downloads section of http://kafka.apache.org/downloads.html page. Now try .It should work. Or if you want to build from the src.tgz package you downloaded, then run ./gradlew jar. It will download all the required dependencies.
I was facing same problem On Windows 10, What i have done is :
Don't download/install Zookeeper separately, download only kafka_2.12-1.1.0 (or higher version)
Create temp folder , ( like this E:\DevApplications\kafka\temp)
Open zookeeper.properties ( i have it # E:\DevApplications\kafka\kafka_2.12-1.1.0\config)
Update dataDir (for me : dataDir=E:/DevApplications/kafka/temp) Note the forward slash
Open CMD and run zookeeper-server-start.bat with zookeeper.properties as second parameter like
.\zookeeper-server-start.bat ..\..\config\zookeeper.properties
Once zookeeper is started start kafka server by typing
.\kafka-server-start.bat ..\..\config\server.properties
Hope this helps.
To add to the Chandra Kant solutions, if you have proxy connection in your network then please use the below command
./gradlew -Dhttp.proxyHost=<PROXY-HOST> -Dhttp.proxyPort=<PROXY-PORT> jar
Thanks #Chandra kant it helped me a lot
You can also reach this exception if you try starting Kafka 0.9.0.0 running a java version less than java 1.7. Set your $JAVA_HOME to 1.7 or above and make sure JAVA_HOME/bin is on your classpath.
Regarding Apache-Kafka messaging queue.
I have downloaded Apache Kafka from the Kafka download page. I've extracted it to /opt/apache/installed/kafka-0.7.0-incubating-src.
The quickstart page says you need to start zookeeper and then start Kafka by running: >bin/kafka-server-start.sh config/server.properties
I'm using a separate Zookeeper server, so i edited config/server.properties to point to that Zookeeper instance.
When i run Kafka, as instructed in the quickstart page, I get the following error:
Exception in thread "main" java.lang.NoClassDefFoundError: kafka/Kafka
Caused by: java.lang.ClassNotFoundException: kafka.Kafka
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class: kafka.Kafka. Program will exit.
I used telnet to make sure the Zookeeper instance is accessible from the machine that Kafka runs on. Everything is OK.
Why am i getting this error?
You must first build Kafka by running the following commands:
> ./sbt update
> ./sbt package
Only then will Kafka be ready for use.
You should know that
./sbt update
./sbt package
will produce Kafka binaries for Scala 2.8.0 by default. If you need it for a different version, you need to do
./sbt "++2.9.2 update"
./sbt "++2.9.2 package"
replacing 2.9.2 with the desired version number. This will make the appropriate binaries. In general, when you switch versions, you should run
./sbt clean
to clean up the binaries from previous versions.
Actually, in addition, you might also need to perform this command
./sbt "++2.9.2 assembly-package-dependency"
This command resolves all the dependencies for running Kafka, and creates a jar that contains just these. Then the start scripts would add this to the class path and you should have all your desired classes.
It seems that without the SCALA_VERSION environment variable, the executable doesn't know how to load the libraries necessary. Try the following from the Kafka installation directory:
SCALA_VERSION=2.9.3 bin/kafka-server-start.sh config/server.properties
See http://kafka.apache.org/documentation.html#quickstart.
Just to add to the previous answer, if you're running IntelliJ, and want to run Kafka inside IntelliJ and/or step through it, make sure to run
> ./sbt idea
I spent easily half a day trying to create the IntelliJ project from scratch, and it turns out that single command was all I needed to get it working. Also, make sure you have the Scala plugin for IntelliJ installed.
You can also use the binary downloads provided by Apache.
For example download kafka version - 0.9.0.1 from this link.
For other version download from link2 and download the binary versions instead. These are already built version. Not need to build again using Scala.
Use are using the source download instead.
You've downloaded the source version. Download the binary package of Kafka and proceed with your testing.
You can find following two options on Kafka downloads page
https://kafka.apache.org/downloads.html
Source download:
Binary downloads
You have downloaded "kafka-0.7.0-incubating-src" it' source code
Download the binary package of Kafka
Scala 2.10 - kafka_2.10-0.10.1.1.tgz (asc, md5)