I'm using the Azure HDInsight's managed Apache Kafka solution since unfortunately there's no managed Confluent Kafka solution on Azure. Is it possible to run the Confluent Schema Registry and connect it to the HDInsight Apache Kafka cluster's brokers?
I'm hoping to install just the Schema Registry on a single VM, then using this line in the schema-registry.properties file, point it to the HDInsight cluster's list of brokers:
kafkastore.bootstrap.servers=PLAINTEXT://localhost:9092
Will this work? Or do the brokers need to be Confluent installations and not Apache?
The Apache Kafka brokers in Confluent Platform are Apache Kafka. So yes you can self-host Schema Registry and connect it to Apache Kafka from another distribution.
Related
Is it possible to use AWS MSK with confluent schema registry with docker instance? I dont need any Kafka connector or sink connector. Before publish want to use AVRO with confluent schema registry and deserialize the same during consumption. What are all the properties i need to set on confluent schema-registry docker? When i try to run i get this error java.lang.RuntimeException: No endpoints found for security protocol [PLAINTEXT]. Endpoints found in ZK. Any pointers are greatly appreciated.
My pipeline is:
Kerberized Kafka --> Logstash (hosted on a different server) --> Splunk.
Can I replace the Logstash component with Kafka Connect?
Could you point me to a resource/guide where I can use kerberized Kafka as a source for my Kafka connect (which is hosted separately)?
From the documentation, what I understood is that if Kafka Connect is hosted on the same cluster as that of Kafka, that's quite possible. But I don't have that option right now, as our Kafka cluster is multi-tenant and hence not approved for additional processes on the cluster.
Kerberos keytabs aren't commonly machine/JVM specific, so yes, Kafka Connect should be able to be configured very similarly to Logstash since both are JVM processes using native Kafka protocol.
You shouldn't run Connect on the brokers anyway
If you can't add Kafka Connect to an existing Kafka cluster, you will have to spin up a separate Kafka Connect (Cluster or standalone).
I've written about it here: enter link description here
I am using Kafka MSK in AWS. So we don't have native kafka connect with all required connectors like on confluent.
Actually I work with kakfa mongo connector and I want to find a way to push the kafka mongo connector jar to an on an instance of kafka MSK cluster.
The path to which the jar will be pushed is the plugins.path as defined in the properties of the used connector.
ANy way to make it please ?
MSK doesn't give you a hosted Kafka Connect worker. You'd need to provision and run this yourself, e.g. on EC2. This work would then connect to your Kafka cluster (MSK in this case)
To be clear: MSK is only the hosted Kafka brokers (and Zookeeper). It does not include Kafka Connect, which is what you need in order to run connectors.
Can we use Confluent Kafka JDBC connect in Apache Kafka Cluster ?
I am trying to use Confluent JDBC connect to install and configure it on apache kafka cluster for source properties connecting to mssql database.
Yes.
Kafka Connect is part of Apache Kafka. The JDBC source and sink connector from Confluent is just a plugin for the Kafka Connect framework.
You can find out more here:
https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector
http://rmoff.dev/ksldn19-kafka-connect
I am trying to upgrade from the apache kafka to the confluent kafka
As the storage of the temp folder is quite limited I have changed the log.dirs of server.properties to a custom folder
log.dirs=<custom location>
Then try to start kafka server via the Confluent CLI (version 4.0) using below command :
bin/confluent start kafka
However when I check the kafka data folder, the data still persitted under the temp folder instead of the customzied one.
I have tried to start kafka server directly which is not using the Confluent CLI
bin/kafka-server-start etc/kafka/server.properties
then seen the config has been picked up properly
is this a bug with confluent CLI or it is supposed to be
I am trying to upgrade from the apache kafka to the confluent kafka
There is no such thing as "confluent kafka".
You can refer to the Apache or Confluent Upgrade documentation steps for switching Kafka versions, but at the end of the day, both are Apache Kafka.
On a related note: You don't need Kafka from the Confluent site to run other parts of the Confluent Platform.
The confluent command, though, will read it's own embedded config files for running on localhost only, and is not intended to integrate with external brokers / zookeepers.
Therefore, kafka-server-start is the production way to run Apache Kafka
Confluent CLI is meant to be used during development with Confluent Platform. Therefore, it currently gathers all the data and logs under a common location in order for a developer to be able to easily inspect (with confluent log or manually) and delete (with confluent destroy or manually) such data.
You are able to change this common location by setting
export CONFLUENT_CURRENT=<top-level-logs-and-data-directory>
and get which location is used any time with:
confluent current
The rest of the properties are used as set in the various .properties files for each service.