What is the difference between Kafka brokerprefix and Kafka bootstrapprefix in confluent Kafka for kubernetes.
Related
Currently we are using Apache NiFi to consume messages via Kafka consumer. Output of kafka consumer is connected to hive processor.
I'm looking into how to run kafka consumer instance on a nifi cluster.
I have 3 nodes of nifi cluster and a kafka topic which have 3 partitions, I want the kafka consumer to be able run on each node so each consumer can poll message from one of topic partitions.
After I started the kafka consumer processor ,i can only see that the kafka consumer always run on a single node but not all nodes.
Is there any configuration that I missed?
NiFi uses the Apache Kafka client which is what performs the assignment of consumers to partitions. When you start the processor, assuming you have it set to 1 concurrent task, then you should have 1 consumer on each node of your cluster, and each consumer should get assigned a different partition.
https://bryanbende.com/development/2016/09/15/apache-nifi-and-apache-kafka
I wanted use Apache Spark Structured Streaming along with Kafka, Spark Structured Streaming Supports Kafka 0.10 and above and my Kafka cluster uses kafka version 0.8.2.1 . I want to replicate some of the topics from current kafka 0.8.2.1 cluster to new Kafka Cluster which is based on 2.2.0.
To do this i tried using kafka-console-consumer on Kafka 2.2.0 cluster to listen the messages from kafka cluster 0.8.2.1 and piped the result of kafka-console-consumer to kafka-console-producer on the the kafka 2.2.0 cluster. But that didn't kafka-console-consumer on Kafka 2.2.0 cluster was not able to receive any messages.
As of now I have solved this problem by reading the data from kafka 0.8.2.1 cluster using the Java Client APIs and I am writing the data read from older kafka cluster(0.8.2.1) to newer kafka cluster(2.2.0) using the client APIs.
Can anyone suggest some better ways to mirror two kafka clusters running different versions of Kafka?
What are the advantages of using Apache Storm's KafkaBolt in apache storm 1.2.2 instead of using the kafka producer apis directly from the bolt in topology to publish to downstream kafka topics?
As from the latest consumer versions of Kafka, the consumers aren't dependent on ZooKeeper. But "https://kafka.apache.org/" says Kafka requires Zookeeper, so start zookeeper server. why is it so?. Once a topic has been created, even though I terminate Zookeeper it works. So the purpose of Zookeeper is only for creating a Topic? If so why not move creating Topic also to be independent of zookeeper
Kafka topics (still) require Zookeeper for electing a leader, communicating server failure, and storing the list of topics, plus some extra metadata such as replica location and topic configurations.
Kafka Wiki - How does Kafka depend on Zookeeper
Confluent and the Kafka community are trying to move away from the Zookeeper dependency. For example, the Confluent Schema Registry can now use Kafka for leader election. Related blog from Confluent - https://www.confluent.io/blog/how-to-prepare-for-kip-500-kafka-zookeeper-removal-guide/
And in Confluent Cloud, Amazon MSK, and other hosted Kafka offerings, you generally have no access to Zookeeper at all.
The consumers are not dependent on Zookeeper as they are client-side. Likewise with the producers.
Zookeeper is required for the Kafka brokers themselves. Kafka brokers use Zookeeper to co-ordinate and synchronise themselves.
With Kafka Nodejs client (nodejs is kafka producer), we could find kafka-node with zookeeper integration. Is there similar package for Java as Kafka producer with integration of zookeeper. The purpose is that Java could talk to zookeeper only and let zookeeper to find kafka brokers so that even one broker is down, zookeeper would advice producer client to talk with the newly selected primary broker.