I'm new in Confluent world, and I know how to start kafka, zookeepers from confluent, but it's not that what I need.
I have already 3 kafka nodes and 2 zookeepers installed by Ambari. Afterwards I downloaded 3.0.0 version of Confluent and now I want to connect Confluent with already running Kafka and zookeeper. I don't want to instance new kafka server or zookeeper server which confluent is giving.
Does anyone has an idea how to accomplish that, what to actually run from Confluent and what to change.
By now I was only chaning files in ./etc/kafka or ./etc/zookeeper which are in Confluent dir. Thank you!
clarify some basics about Confluent and how manage communication between Confluent and Kafka
First things first, there is no single application called "Confluent" that can be started all on its own.
There's is nothing to configure for Kafka or Zookeeper. The Confluent Platform doesn't add anything on top of the existing Apache Kafka you have (presumably, via Hortonworks or Cloudera).
In fact, those companies do add patches to Kafka that would be slightly different than the base Apache versions you would get from Confluent.
That being said, if you read through each of extra services that Confluent provides, you'll notice either a Zookeeper or a Bootstrap server configuration option. Fill out those fields, start the respective services, and you're good to go.
what to actually run from Confluent
Look in the bin directory, you can find all the start scripts. From the comments, looks like you're trying to use Connect Distributed (which should already be installed by any recent Kafka installation, it's not Confluent specific), and Schema Registry. You'll have to be more specific about the errors that you get, but the config files are all in the etc path.
Unless you're using KSQL, REST Proxy or Control Center, there's not much to run because, as mentioned, Kafka Connect is included with the base Apache Kafka project and Hortonworks is maintaining their own Schema Registry project
2 zookeepers installed by Ambari
This is a highly non-recommended setup. Please install an odd number of Zookeepers. 3 or 5, preferably
Related
i have setup Aerospike with all the configuration required for Kakfa and m using confluent local cluster for Kafka and have installed https://www.confluent.io/hub/aerospike/kafka-connect-aerospike-source and have started confluent cluster but still connector is not starting
also i found that there is no jar in share folder of confluent is it still under development ?
It works and is fully generally available but requires extra licensing on the Aerospike side. I would not expect it to work with Community Edition.
we are using apache kafka 2.7 version, Before that we used confluent-platform version of kafka where we can access control center to view the statistics like incoming/outgoing data rate,disk space like those metrices.
Now we are using apache kafka and need to view the same kind of statistics in control center
is it possible..?
if possible can anyone tell how to perform these?
You'd need to download Confluent Platform, still, then find and copy all relevant JAR files for the Confluent interceptors and metric reporters into the Kafka libs directory in each broker (and other clients like Connect), then reconfigure the relevant properties and restart each system. You might also be able to grab the jar(s) directly from Maven
Note that Apache Kafka brokers will not detect any broker configuration that starts with confluent. prefix, so you'll also need to manually create some topics like _confluent-metrics
https://docs.confluent.io/platform/current/kafka/metrics-reporter.html
I have a Kafka cluster that I work with which is managed by my team and runs on Kubernetes. We want to install the Kafka connect via helm into our cluster to work with our Kafka. This Kafka we are running is NOT the confluent platform Kafka. Is there a good way to do this? I was wondering if this would work cp-helm-charts. Will using the confluentinc Kafka connect container be compatible with my Kafka cluster that is on non-confluent platform?
Kafka Connect has never been labelled as a Confluent Platform exclusive product.
The Framework is entirely Apache 2.0 Licensed and Open Source.
Similarly, "Confluent Platform Kafka" is just Apache Kafka
Which is the better way to install kafka, zookeeper and schema registry? I already have installed kafka and zookeeper from their official site, but I noticed that on the confluent schema registry package it comes with kafka and zookeeper too, so will I need download the package and only use schema registry? or there is another link where I can download schema registry without the kafka and zookeper part?, thanks in advance
I recommend to install it all from the same place. Confluent offers the Confluent Platform community edition that includes:
Zookeeper
Kafka
Schema Registry
Kafka Connect
REST Proxy
KSQL Server
That way you will avoid any version incompatibilities. I followed the instructions from Confluent to get the entire suite installed and running.
Together with the command line tool you will be able to start and stop all necessary services with only one command:
confluent local start
They also offer Docker installation which might be of interest to you as well.
I have an exisiting Kafka Cluster. I want to install the Kafka REST Proxy:
https://github.com/confluentinc/kafka-rest
If I install confluent does that come with Kafka? I am afraid if I still it on my master Kafka node confluent will override all my settings and mess up my Kafka cluster.
How do you install Kafka REST when you have an existing Kafka cluster?
This is not made clear on their website. I have CentOS and was going to try:
sudo yum install confluent-platform-oss-2.11
Any help would be great....
Download the Confluent Platform tarball, extract it, (or preferrably use APT/YUM) then only configure and run the REST proxy via kafka-rest-start
I wouldn't recommend using APT/YUM to install the entire confluent platform if you already have an existing Kafka. You might be able to only install kafka-rest using it, though.
Alternatively, backup your existing Kafka and Zookeeper property files, then place the Confluent Platform on top of the existing files, keeping the original files. If your Kafka is an old release, take this as a good opportunity to schedule an upgrade. Downloading Confluent isn't going to overwrite anything for the upstream Apache projects version for the corresponding release. If anything, it's an extension