Failed to find any class that implements Connector and which name matches io.debezium.connector.mysql.MySqlConnector, available connectors are - apache-kafka

Question

Setting up Data Streaming Pipeline using Kafka, MySQL and Debezium. I'm this version of Kafka - 3.4.0, MySQL - 8, Debezium - 2.2.1, Java - 11. AIM : I want to capture all CDC from MySQL and stream the data to Kafka Topics.
If have followed these steps:
Downloaded Kafka - 3.4.0 and extracted and kept in this path : /home/divum/kafka-all/kafka_2.13-3.4.0
Downloaded Debezium - 2.2.1 and extracted it. This is the path for it path : /usr/local/share/kafka/plugins/debezium-connector-mysql
Added this line plugin.path=/usr/local/share/kafka/plugins/debezium-connector-mysql in file connect-distributed.properties which is present inside this path : /home/divum/kafka-all/kafka_2.13-3.4.0/config.
And I have MySql and Debezium Configuration like the below. Enabled binary logs as well. The file name is connector-config.json, which is present in this path /home/divum/kafka-all/kafka_2.13-3.4.0.
{ "name": "etl-connector",
"config": {
"connector.class":"io.debezium.connector.mysql.MySqlConnector",
"tasks.max": 1,
"database.hostname": "localhost",
"database.port": 3306,
"database.user": "root",
"database.password": "iphone21",
"topic.prefix": "stream",
"database.server.id": 4816,
"database.include.list": "etl",
"table.include.list": ".*",
"column.include.list": ".*",
"include.schema.changes": false,
"database.server.name": "mysql",
"tombstones.on.delete": false,
"database.history.kafka.bootstrap.servers": "localhost:9092",
"database.history.kafka.topic": "history",
"plugin.path":"/usr/local/share/kafka/plugins/debezium-connector-mysql"
}
}
I'm using this command for starting the broker, zookeeper, and connector.
Zookeeper - bin/zookeeper-server-start.sh config/zookeeper.properties
Kafka Broker - bin/kafka-server-start.sh config/server.properties
Connector - bin/connect-distributed.sh ./config/connect-distributed.properties
Checking the connector-plugins with curl --location --request GET 'http://localhost:8083/connector-plugins'. Getting the below output.
[
{
"class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector",
"type": "source",
"version": "3.4.0"
},
{
"class": "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector",
"type": "source",
"version": "3.4.0"
},
{
"class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"type": "source",
"version": "3.4.0"
}
]
Trying to start the connector with this command ** curl -X POST -H "Content-Type: application/json" --data #connector-config.json http://localhost:8083/connectors** and getting below error. It's saying like this io.debezium.connector.mysql.MySqlConnector Class is not available.
{"error_code":500,"message":"Failed to find any class that implements Connector and which name matches io.debezium.connector.mysql.MySqlConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='3.4.0', encodedVersion=3.4.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='3.4.0', encodedVersion=3.4.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='3.4.0', encodedVersion=3.4.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='3.4.0', encodedVersion=3.4.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='3.4.0', encodedVersion=3.4.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='3.4.0', encodedVersion=3.4.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='3.4.0', encodedVersion=3.4.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='3.4.0', encodedVersion=3.4.0, type=source, typeName='source', location='classpath'}"}
Command Used : **bin/connect-distributed.sh ./config/connect-distributed.properties **. Getting this logs.
Plugin scan logs:
I'm trying with setup for past few days not able to do it. What was the cause of this error? Any idea how to solve it?
Is there any open source alternative is there for Debezium?

Answer 1

The issue is that the debezium connector is not being loaded. When you run a GET on /connector-plugins, you should see a debezium entry.
After Step 2, try to move the debezium artifact (which is probably a jar), into the Kafka plugin directory, instead of changing the plugin.path as in step 3. Check the following article from Confluent.
When you get to step 6, you need to register the connector by making a POST request at http://localhost:8083/connectors with the following payload:
{
"name": "<some-unique-name>",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"database.hostname": "<>",
"database.port": "<>",
"database.user": "<>",
"database.password": "<>",
"database.server.name": "<>",
"database.server.id": <>,
"database.include.list": "<>",
"table.include.list": "<>",
"database.history.kafka.bootstrap.servers": "<kafka-url>",
"database.history.kafka.topic": "<result-topic-name>",
"topics": "<result-topic-name>"
}
}
Follow the documentation for a proper setup.

Answer 2

I have found the solution after experimenting multiple configurations. Posting the solution finally worked.
Use Java - 11
Just changing the connector.class from this io.debezium.connector.mysql.MySqlConnector to io.confluent.connect.jdbc.JdbcSourceConnector.
{
"name": "etl-connector",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"tasks.max": 1,
"database.hostname": "localhost",
"database.port": 3306,
"database.user": "root",
"database.password": "iphone21",
"topic.prefix": "stream",
"database.server.id": 4816,
"database.include.list": "etl",
"table.include.list": ".*",
"column.include.list": ".*",
"include.schema.changes": false,
"database.server.name": "mysql",
"tombstones.on.delete": false,
"database.history.kafka.bootstrap.servers": "localhost:9092",
"database.history.kafka.topic": "history",
"plugin.path": "/usr/local/share/kafka/plugins/debezium-connector-mysql"
}
}
The issue is with my plugin path, just add the path like this:
nano ./config/connect-distributed.properties
plugin.path=<path/of-your-debezium/*>
Eg : plugin.path=/usr/local/share/kafka/plugins/debezium-connector-mysql/*
Once plugin path is added, please verify it after running the below command.
bin/connect-distributed.sh ./config/connect-distributed.properties
Go through the INFO logs, it should show the path which you have used for the configuration.
jvm.classpath=<path/of-your-debezium/*>
After Step - 5 if you able to start the connectors with curl -X POST -H "Content-Type: application/json" --data #connector-config.json http://localhost:8083/connectors then step -6,7 are not required.
If same issue then add CLASSPATH in your .bashrc
nano ~/.bashrc
export CLASSPATH=<path/of-your-debezium/*>
Eg : export CLASSPATH=/usr/local/share/kafka/plugins/debezium-connector-mysql/*
source ~/.bashrc
Once everything is added just kill the existing terminal, stop the kafka connector and start it again or restart it in the new terminal. Because in existing won't reload the new configuration. Then everything will work fine.