I have installed Kafka in Windows.
Also unpacked connector files jar.
Where to specify path to these files and how to configure the connector and launch it?
In connect properties file, there is plugin.path, which should be set to the absolute path of the parent directory of all connectors folders.
https://docs.confluent.io/current/connect/managing/community.html
Related
I have created a jar containing class files used for SASL callbacks and is present in my application directory.
Is there a way to link this jar with Kafka start up (using kafka-server-start.sh script) without having to explicitly copy this jar under Kafka's libs directory?
Environment: RHEL
You can export the CLASSPATH variable prior to running kafka-server-start, or you can simply symlink the file
We are planning to create our own repo of connector (sink or source plugins) for Apache Kafka like one here
We tried to search for the documentation or help on how to create a plugin jar for Kafka.
There is no mention of developing a plugin in the official documentation from apache Kafka.
Any help or pointer will be helpful, can share it back with the open community once developed.
Here is guide on How to Build a Connector
As Well here is a Connector Developer Guide
Developing a connector only requires implementing two interfaces, the Connector and Task
Refer to the example source code for full examples for simple example
Once you’ve developed and tested your connector, you must package it so that it can be easily installed into Kafka Connect installations. The two techniques described here both work with Kafka Connect’s plugin path mechanism.
If you plan to package your connector and distribute it for others to use, you are obligated to properly license and copyright your own code and to adhere to the licensing and copyrights of all libraries your code uses and that you include in your distribution.
Creating an Archive
The most common approach to packaging a connector is to create a tarball or ZIP archive. The archive should contain a single directory whose name will be unique relative to other connector implementations, and will therefore often include the connector’s name and version. All of the JAR files and other resource files needed by the connector, including third party libraries, should be placed within that top-level directory. Note, however, that the archive should never include the Kafka Connect API or runtime libraries.
To install the connector, a user simply unpacks the archive into the desired location. Having the name of the archive’s top-level directory be unique makes it easier to unpack the archive without overwriting existing files. It also makes it easy to place this directory on Installing Connect Plugins or for older Kafka Connect installations to add the JARs to the CLASSPATH.
Creating an Uber JAR
An alternative approach is to create an uber JAR that contains all of the connector’s JAR files and other resource files. No directory internal structure is necessary.
To install, a user simply places the connector’s uber JAR into one of the directories listed in Installing Connect Plugins.
I am using Kafka connect to create a MQTT Kafka connection.I put all the kafka MQTT connector specific jar downloaded from confluent site to "/data" folder. And accordingly update the "connect-standalone.properties" file to reflect the plugin path i.e
plugin.path=/opt/kafka_2.11-2.1.1/libs,/data
When I run the Kafka Connect
./connect-standalone.sh ../config/connect-standalone.properties ../config/connect-mqtt-source.properties
I get following error :
[2019-07-18 10:26:05,823] INFO Loading plugin from:
/data/kafka-connect-mqtt-1.2.1.jar
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-07-18 10:26:05,829] ERROR Stopping due to error
(org.apache.kafka.connect.cli.ConnectStandalone:128)
java.lang.NoClassDefFoundError:
com/github/jcustenborder/kafka/connect/utils/VersionUtil
at io.confluent.connect.mqtt.MqttSourceConnector.version(MqttSourceConnector.java:29)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:344)
screenshot as below :
Please note that "connect-utils-0.3.140.jar" is present in "/data" folder as highlighted by red underlines.
Now If I make a soft link screenshot below or copy all the jars from "/data" folder and update the plugin path to :
plugin.path=/opt/kafka_2.11-2.1.1/libs
Kafka connect works perfectly fine.
Any help why it does not work in the first scenario i.e kafka connector specific jars in different folders
From Kafka Connect user guide on Confluent page:
...
Kafka Connect isolates each plugin from one another so that libraries in one plugin are not affected by the libraries in any other plugins. This is very important when mixing and matching connectors from multiple providers.
A Kafka Connect plugin is:
an uber JAR containing all of the classfiles for the plugin and its third-party dependencies in a single JAR file; or
a directory on the file system that contains the JAR files for the plugin and its third-party dependencies.
In your case you have to put plugin jars in one folder, ex /data/pluginName not directly in /data/
More details can be found here: Installing Plugins
I have downloaded plugin files from https://www.confluent.io/connector/kafka-connect-cdc-microsoft-sql/,
It has three folders lib, etc, doc, manifest.json. etc has connect-avro-docker.properties, mssqlsource.properties, repro.properties. I can add CONNECT_PLUGIN_PATH to lib, but what about these config files?
In https://docs.confluent.io/current/connect/userguide.html page they did not give clear instructions on where to copy these files.
How to copy or configure kafka connect plugin files for confluent?
Any direction is appreciated.
I installed only kafka-connect, schema-registry, kafka-broker, zookeeper. I didn't find a way to install hub on windows.
The instructions are here
You need to point plugin.path line in the Connect properties to the absoulte path of the extracted lib folder - the parent folder of where the JARs are located.
You can also use confluent-hub tool, which should set this all up for you.
I am working on RAD(WebSphere Portal Server 6.1v) and i am getting java build path error for ServiceManager and BOFactory jar files.
Kindly help.
I believe this is part of WebSphere Process Server. These jar files are located in the <INSTALL_ROOT>/plugins directory
BOFactory is located in com.ibm.soacore.runtime_6.1.0.jar
ServiceManager is located in com.ibm.soacore.sca_6.1.0.jar
You need to have the server or the client jars installed on the machine you are developing on.