I am trying to to use Kafka Connect to HBase and there are no Confluent supported connectors available for HBase, though there are some community connectors available. We are not really ready to take risk in production with out support to the connectors: Is there any other work around for HBase connectivity from Kafka Connect? Can we use Kafka JDBC connector for Kafka Connect?
Related
I'm trying to use AWS DocumentDB as a sink for storing data received from Kafka and was wondering if the MongoDB Kafka connector works with DocumentDB as its documentation mentions that it is compatible with MongoDB drivers.
https://www.mongodb.com/docs/kafka-connector/current/
https://aws.amazon.com/documentdb/
If not this connector what is the alternate way other than building a custom kafka connect?
You can use MongoDB Kafka connector with DocumentDB for source as well Sink.
Kafka Connector worker(with Mongodb Kafka connector) can be run in distributed mode using containers as well as EC2 hosts.
You can refer blog here which has step by step details
https://aws.amazon.com/blogs/database/stream-data-with-amazon-documentdb-and-amazon-msk-using-a-kafka-connector/
I know Vertica has vkconfig to import data from Kafka. However, it seems like Confluent also has a connector that does the same. As their web site states: "Vertica Sink Connector for Confluent Platform - You can use the Kafka Connect Vertica Sink connector to export data from Apache Kafka topics to Vertica. The Vertica Sink connector periodically polls records from Kafka and adds them to a Vertica table."
Are the two connectors aim at doing the same tasks? If not, what are the differences?
At a high level, the difference in software is that the Vertica one is done on a periodic schedule
Kafka Connect is more real-time, depending on how you've configured it.
Of course, there also comes the details around installation, support, and licensing.
You should expect more Vertica-specific features from the Vertica importer than from Confluent, which may only focus on the bare minimum requirements to take Kafka records to DB events, and who knows is adding enhacements to that connector.
Worked and done a comparison of both Vertica and confluent Vertica sink connectors. Vertica inbuilt connector does not have the capability to handle Kafka tombstone messages(logged an enhancement request with Vertica ) and is slow with avro. Worked with confluent in fixing at least 15 bugs and enhancements and they improved and released latest version of vertica sink connector supporting most of the features. Working on licensing with confluent is the difficult part as they might not provide license to their connector pack unless the entire eco system to manage Kafka is bought
I am using Kafka MSK in AWS. So we don't have native kafka connect with all required connectors like on confluent.
Actually I work with kakfa mongo connector and I want to find a way to push the kafka mongo connector jar to an on an instance of kafka MSK cluster.
The path to which the jar will be pushed is the plugins.path as defined in the properties of the used connector.
ANy way to make it please ?
MSK doesn't give you a hosted Kafka Connect worker. You'd need to provision and run this yourself, e.g. on EC2. This work would then connect to your Kafka cluster (MSK in this case)
To be clear: MSK is only the hosted Kafka brokers (and Zookeeper). It does not include Kafka Connect, which is what you need in order to run connectors.
Can we use Confluent Kafka JDBC connect in Apache Kafka Cluster ?
I am trying to use Confluent JDBC connect to install and configure it on apache kafka cluster for source properties connecting to mssql database.
Yes.
Kafka Connect is part of Apache Kafka. The JDBC source and sink connector from Confluent is just a plugin for the Kafka Connect framework.
You can find out more here:
https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector
http://rmoff.dev/ksldn19-kafka-connect
From the following issue at CrateDB GitHub page it seems it is not possible, i.e., the Kafka protocol is not supported by CrateDB.
https://github.com/crate/crate/issues/7459
Is there another way to load data from Kafka into CrateDB?
Usually you'd use Kafka Connect for integrating Kafka to target (and source) systems, using the appropriate connector for the destination technology.
I can't find a Kafka Connect connector for CrateDB, but there is a JDBC sink connector for Kafka Connect, and a JDBC driver for CrateDB, so this may be worth a try.
You can read more about Kafka Connect here, and see it in action in this blog series:
https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/
https://www.confluent.io/blog/blogthe-simplest-useful-kafka-connect-data-pipeline-in-the-world-or-thereabouts-part-2/
https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-3/
Disclaimer: I work for Confluent, and I wrote the above blog posts.