i have setup Aerospike with all the configuration required for Kakfa and m using confluent local cluster for Kafka and have installed https://www.confluent.io/hub/aerospike/kafka-connect-aerospike-source and have started confluent cluster but still connector is not starting
also i found that there is no jar in share folder of confluent is it still under development ?
It works and is fully generally available but requires extra licensing on the Aerospike side. I would not expect it to work with Community Edition.
Related
I have a Kafka cluster that I work with which is managed by my team and runs on Kubernetes. We want to install the Kafka connect via helm into our cluster to work with our Kafka. This Kafka we are running is NOT the confluent platform Kafka. Is there a good way to do this? I was wondering if this would work cp-helm-charts. Will using the confluentinc Kafka connect container be compatible with my Kafka cluster that is on non-confluent platform?
Kafka Connect has never been labelled as a Confluent Platform exclusive product.
The Framework is entirely Apache 2.0 Licensed and Open Source.
Similarly, "Confluent Platform Kafka" is just Apache Kafka
Which is the better way to install kafka, zookeeper and schema registry? I already have installed kafka and zookeeper from their official site, but I noticed that on the confluent schema registry package it comes with kafka and zookeeper too, so will I need download the package and only use schema registry? or there is another link where I can download schema registry without the kafka and zookeper part?, thanks in advance
I recommend to install it all from the same place. Confluent offers the Confluent Platform community edition that includes:
Zookeeper
Kafka
Schema Registry
Kafka Connect
REST Proxy
KSQL Server
That way you will avoid any version incompatibilities. I followed the instructions from Confluent to get the entire suite installed and running.
Together with the command line tool you will be able to start and stop all necessary services with only one command:
confluent local start
They also offer Docker installation which might be of interest to you as well.
I would integrate kafka-rest-proxy confluent solution with apache kafka 2.0.0
Could some one explain how I install only kafka rest proxy for my cluster with 3 nodes kafka and 3 znodes ?
All of the Confluent tools work with Apache Kafka.
There is no individual download of the REST Proxy, so you would have to use Docker or download the full Confluent platform.
If not using Docker, you can find the kafka-rest.properties in the etc/kafka-rest folder, and so you would edit it with at least the bootstrap servers.
Find other config options here
Then run this to start it from the extract Confluent platform download
./bin/kafka-rest-start ./etc/kafka-rest/kafka-rest.properties
I'm new in Confluent world, and I know how to start kafka, zookeepers from confluent, but it's not that what I need.
I have already 3 kafka nodes and 2 zookeepers installed by Ambari. Afterwards I downloaded 3.0.0 version of Confluent and now I want to connect Confluent with already running Kafka and zookeeper. I don't want to instance new kafka server or zookeeper server which confluent is giving.
Does anyone has an idea how to accomplish that, what to actually run from Confluent and what to change.
By now I was only chaning files in ./etc/kafka or ./etc/zookeeper which are in Confluent dir. Thank you!
clarify some basics about Confluent and how manage communication between Confluent and Kafka
First things first, there is no single application called "Confluent" that can be started all on its own.
There's is nothing to configure for Kafka or Zookeeper. The Confluent Platform doesn't add anything on top of the existing Apache Kafka you have (presumably, via Hortonworks or Cloudera).
In fact, those companies do add patches to Kafka that would be slightly different than the base Apache versions you would get from Confluent.
That being said, if you read through each of extra services that Confluent provides, you'll notice either a Zookeeper or a Bootstrap server configuration option. Fill out those fields, start the respective services, and you're good to go.
what to actually run from Confluent
Look in the bin directory, you can find all the start scripts. From the comments, looks like you're trying to use Connect Distributed (which should already be installed by any recent Kafka installation, it's not Confluent specific), and Schema Registry. You'll have to be more specific about the errors that you get, but the config files are all in the etc path.
Unless you're using KSQL, REST Proxy or Control Center, there's not much to run because, as mentioned, Kafka Connect is included with the base Apache Kafka project and Hortonworks is maintaining their own Schema Registry project
2 zookeepers installed by Ambari
This is a highly non-recommended setup. Please install an odd number of Zookeepers. 3 or 5, preferably
I'm trying to install Confluent over HDP for Kafka Streams which think may not be possible could you people suggest me what to do
It seems like you are trying to install Confluent Platform using Ambari. If that's the case then you want to use a custom service install or you will need to wait for HDP to support Kafka 0.10 which includes Kafka Streams. The alternative route is to install Confluent Platform in parallel with HDP and just not activate the Kafka version that ships with HDP. This will require that you monitor and manage the Confluent Platform independently.
for Kafka Streams
Kafka Streams is a client library that can use with any Kafka cluster above version 0.10. You do not specifically require "Confluent Platform"
That being said, if the version that HDP/Cloudera provide of Kafka is not useful for you, then you should provision external infrastructure for it.