I would like to understand the development life-cycle for Kafka connect modules.
Following http://kafka.apache.org/090/documentation.html#connect, I would like to write a custom Kafka connect module using the "connect api" but I dont know where to start. Are there any minimal examples of how to do this? Project setup etc?
Incidentally I built this one https://github.com/confluentinc/kafka-connect-jdbc and tried to run it (on google cloud) but i find errors - clearly a missing dependency but I dont know what to add. Of course it might be that this is only supposed to run on confluent platform. If it could run elsewhere then great. But if it cant, I would like to know how to build one from scratch hence my question.
java.lang.NoClassDefFoundError: org/apache/kafka/common/config/ConfigDef$Recommender
at io.confluent.connect.jdbc.JdbcSourceConnector.start(JdbcSourceConnector.java:66)
at org.apache.kafka.connect.runtime.Worker.addConnector(Worker.java:186)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.startConnector(StandaloneHerder.java:197)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:145)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:85)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.config.ConfigDef$Recommender
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 5 more
The most common cause of this kind of errors is the configuration of your CLASSPATH. For Kafka Connect to find your classes on runtime you need to include them on your classpath. The following text is taken directly from the documentation of Kafka connect :
All that is required to install a new plugin is to place it in the CLASSPATH of the Kafka Connect process. All the scripts for running Kafka Connect will use the CLASSPATH environment variable if it is set when they are invoked, making it easy to run with additional connector plugins:
And how to do it:
export CLASSPATH=/path/to/my/connectors/*
bin/connect-standalone standalone.properties new-custom-connector.properties
I have also written a how to guide for Kafka Connect that you might find helpful.
Update the kafka-clients.jar to kafka-clients-0.10.0.0.jar. The old version doesn't include this class: org/apache/kafka/common/config/ConfigDef$Recommender.class
You can download it here:
http://central.maven.org/maven2/org/apache/kafka/kafka-clients/0.10.0.0/kafka-clients-0.10.0.0.jar
Would have added this as a comment, but SO indicated that I had insufficient points. Anyway, the point of this answer is to demonstrate that the JDBC Connector can run without installing the entire Confluent package and schema registry.
I was able to get Confluent's JDBC connector running with without installing the Confluent platform (specifically, the schema registry). There are four Confluent libraries that you need in the classpath when you run the connector (common-config, common-metrics, common-utils and kafka-connect). For more detailed instructions, please see https://prefrontaldump.wordpress.com/2016/05/02/using-confluents-jdbc-connector-without-installing-the-entire-platform/
Related
I am trying to use kafka connect in a docker container with a custom connector (PROGRESS_DATADIRECT_JDBC_OE_ALL.jar) to connect to an openedge database.
I have put the JAR file in the plugin path (usr/share/java) but it won't load as a connector.
COPY Openedge/PROGRESS_DATADIRECT_JDBC_OE_ALL.jar /usr/share/java/progress
I can load another (standard) connector by putting it in the plugin path. This works
COPY confluentinc-kafka-connect-jdbc-10.3.2 /usr/share/java/confluentinc-kafka-connect-jdbc-10.3.2
A little lost on how to move forward and I'm very new to kafka. My main sources of information are
openedge to kafka streaming and How to use Kafka connect
#OneCricketeer had the solution. As a retro for me and hopefully helpful to someone else, here are my steps to make this work.
Copy the JDBC Connector to CONNECT_PLUGIN_PATH and install with confluent hub install:
COPY confluentinc-kafka-connect-jdbc-10.3.2.zip /usr/share/java
RUN confluent-hub install --no-prompt /usr/share/java/confluentinc-kafka-connect-jdbc-10.3.2.zip
Copy the driver (I ended up using openedge.jar) to the path where other jars are located (like sqllite) according to #OneCricketeer suggestion.
COPY Openedge/openedge.jar /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib
Verify with this by enabling DEBUG as suggested by this page
Finally add a .properties file to create the connector. In my case based on the one in “openedge to kafka streaming” link above
JDBC Drivers are not Connect plugins, nor are they connectors themselves.
You'd need to set the JVM CLASSPATH environment variable for detecting JDBC Drivers, as with any Java process.
The instructions on the linked site suggest you should copy the JDBC Drivers into the directory for the existing Confluent JDBC connector. While you could use a Docker COPY command, the better way would be to use confluent-hub install
I'm trying to install MQ source & sink connectors for our confluent cloud. I've done this for on-prem apache kafka but doing the same of cloud seems to be different. Following the confluent documents says I need to have a platform installed on my local, which I did, and then to run a confluent-hub install which does install the connector on my local and then use the json for distributed instance.
My problem is when I run the json, it says the class for mq was not found, I tried to point the CLASSPATH to the dir where the jars are but still get the same error. How do I run this successfully?
--
ERROR Uncaught exception in REST call to /connectors (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:61)
org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.confluent.connect.ibm.mq.IbmMQSourceConnector,
Also want to understand how installing connector on local would apply to my cloud cluster? Not sure what I'm missing!
Confluent Cloud doesn't support custom connector installation, last I checked. They need to explicitly support and offer it.
I assume you're reading some documentation that indicates you need to run your own Connect cluster (not necessarily locally), where you have full control over the connectors that are installed
I am trying to deploy hbase sink connector for kafka (https://github.com/mravi/kafka-connect-hbase). So I downloaded and configured Hbase and Confluent Platform as per step 1 & 2.
Then it says,
Copy hbase-sink.jar and hbase-sink.properties from the project build location to $CONFLUENT_HOME/share/java/kafka-connect-hbase
But I don't see hbase-sink.jar and hbase-sink.properties anywhere in the Hbase and confluent directory location. Any help where I can get them.
But I don't see hbase-sink.jar and hbase-sink.properties
Sounds like you've not cloned that repo and ran mvn clean package, then opened up the target directory
As the other answer says, that project seems abandoned.
Try looking at this one too https://docs.lenses.io/connectors/sink/hbase.html
Have you seen the Hbase connector from Confluent kafka-connect-hbase? The one which you are using seems to be abandoned (no commits for the last 4 years).
kafka-connect-hbase documentation
I have kafka conenct running in Marathon container. If I want to update the connector plugin (jar) I have to upload the new one and then restart the Connect task.
Is it possible to do that without restarting/downtime?
The updated jar for the connector plugin needs to be added to the classpath and then the classloader for the worker needs to pick it up. The best way to do this currently is to take an outage as described here.
Depending on your connector, you might be able to do rolling upgrades, but the generic answer is that if you need to upgrade the connector plugin, you currently have to take an outage.
I am getting following error on starting hazelcast server using server.sh in all the versions 3.1.7, 3.2.6, 3.3.3
Error while creating AWSJoiner!
java.lang.ClassNotFoundException: com.hazelcast.cluster.TcpIpJoinerOverAWS
Multicast and tcp-ip are working fine
Hazelcast-all and all other jars are included in the lib.
Did you include 'hazelcast-cloud' jar. This is needed to use AWS discovery.