From the following issue at CrateDB GitHub page it seems it is not possible, i.e., the Kafka protocol is not supported by CrateDB.
https://github.com/crate/crate/issues/7459
Is there another way to load data from Kafka into CrateDB?
Usually you'd use Kafka Connect for integrating Kafka to target (and source) systems, using the appropriate connector for the destination technology.
I can't find a Kafka Connect connector for CrateDB, but there is a JDBC sink connector for Kafka Connect, and a JDBC driver for CrateDB, so this may be worth a try.
You can read more about Kafka Connect here, and see it in action in this blog series:
https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/
https://www.confluent.io/blog/blogthe-simplest-useful-kafka-connect-data-pipeline-in-the-world-or-thereabouts-part-2/
https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-3/
Disclaimer: I work for Confluent, and I wrote the above blog posts.
Related
I installed Neo4j and I can access the server. I can make nodes though cypher.
Now I want to use it for data streams. But I'm not sure how to do so. I just started Neo4j and I'm struggling with installing 'Stream Plugin'.
Any help is highly appreciated.
You should copy the jar files for the Neo4j streams plugin directly into your /plugins folder and configure the connections to Kafka and Zookeeper as well as other Neo4j property values at the neo4j.conf file as described here. For example:
kafka.zookeeper.connect=zookeeper-host:2181
kafka.bootstrap.servers=kafka-host:9092
Alternatively, if you are looking only for a sink connection from Kafka (i.e. moving records from Kafka topics to into Neo4j), you can also use Kafka Connect with the the supported Kafka Connect Neo4j Sink. More at https://www.confluent.io/hub/neo4j/kafka-connect-neo4j
I know Vertica has vkconfig to import data from Kafka. However, it seems like Confluent also has a connector that does the same. As their web site states: "Vertica Sink Connector for Confluent Platform - You can use the Kafka Connect Vertica Sink connector to export data from Apache Kafka topics to Vertica. The Vertica Sink connector periodically polls records from Kafka and adds them to a Vertica table."
Are the two connectors aim at doing the same tasks? If not, what are the differences?
At a high level, the difference in software is that the Vertica one is done on a periodic schedule
Kafka Connect is more real-time, depending on how you've configured it.
Of course, there also comes the details around installation, support, and licensing.
You should expect more Vertica-specific features from the Vertica importer than from Confluent, which may only focus on the bare minimum requirements to take Kafka records to DB events, and who knows is adding enhacements to that connector.
Worked and done a comparison of both Vertica and confluent Vertica sink connectors. Vertica inbuilt connector does not have the capability to handle Kafka tombstone messages(logged an enhancement request with Vertica ) and is slow with avro. Worked with confluent in fixing at least 15 bugs and enhancements and they improved and released latest version of vertica sink connector supporting most of the features. Working on licensing with confluent is the difficult part as they might not provide license to their connector pack unless the entire eco system to manage Kafka is bought
Reading the documentation of this connector there isn't a mention about this characteristic.
So, does this connector guarantee that it won't produce duplicated records under broker crashes or whatever could happen?
Do we have to configure something to get indempotence the same way we would do with any other Kafka Producer (enable.idempotence: true)?
Kafka Connect JDBC source connector, is not idempotent at the moment. Here's the relevant KIP-318 and JIRA ticket.
I am trying to find the api that connects kinesis to kafka.
The API listed on the kafka connecter website is dead.
see https://github.com/jcustenborder/kafka-connect-kinesis
Any suggested replacement?
This connector will be included in the Confluent Enterprise connector program, and will be available for download soon from https://confluent.io
I want to have all of the changes of a couchdb database in kafka at application run time as they arrive. Is there any reliable existing tool for that?
You may try to use Kafka Connect tool. Also, Confluent Platform provides long list of different connectors for Kafka Connect.
I'm not a CouchDB user, but you may choose one of applicable source connectors here or create your own Kafka CouchDB source connector.