There are different valid values for inter.broker.protocol.version in the documentation on the Kafka project and the Confluent documentation. The value 2.7on the Kafka page should be either 2.7-IV1 or 2.7-IV2 according to the docs on confluent.io.
Is this something specific to the confluentinc/cp-kafka image? Which one should I use when using the confluentinc/cp-kafka image?
Confluent is only a distribution of Apache Kafka. Those values can be the same.
As for the container, it really doesn't matter which Kafka image you use, as server.properties is still used, regardless. Related - Connect to Kafka running in Docker
Related
I was working with Kafka. I downloaded the zookeeper, extracted and started it.
Then I downloaded Kafka, extracted the zipped file and started Kafka. Everything was working good. I created few topics and I was able to send and receive messages. After that I stopped Kafka and Zookeeper. Then I read that Kafka itself provides Zookeeper. So I started Zookeeper that was provided with Kafka. However the data directory for it was different, and then I started Kafka from same configuration file and same data directory location. However after starting Kafka I could not find the topics that I had created.
I just want to know that, does this mean the meta data about the topics is maintained by Zookeeper. I searched Kafka documentation, however, I could not find anything in detail.
https://kafka.apache.org/documentation/
Check this documentation provided by confluent. According to this Apache Kafka® uses ZooKeeper to store persistent cluster metadata and is a critical component of the Confluent Platform deployment. For example, if you lost the Kafka data in ZooKeeper, the mapping of replicas to Brokers and topic configurations would be lost as well, making your Kafka cluster no longer functional and potentially resulting in total data loss.
So, the answer to your question is, yes, the purpose of zookeeper is to store relevant metadata about the kafka brokers, topics, etc,.
Also, since you have just started working on Kafka and Zookeeper, I would like to mention this. By default, Kafka stored it's data in a temp location which get's deleted on system reboot, so you should change that as well.
the answer to your question tag is yes,
1)Initially you started standalone zookeeper from zip file and you stopped the zookeeper, which means the topics that are created are stored in the zookeeper standalone are lost.Now you persistent cluster metadata related to Kafka is lost .
2)second time you started the zookeeper from the package that comes along with Kafka, now the new zookeeper instance does not have any topics information that you created previously, so you need to create newly .
3) suppose in case 1: if you close the terminal and start again the zookeeper from standalone , you no need to create the Topic again ,but if you stopped the zookeeper server from standalone then topics are lost.
in simple : you created two separate zookeeper instances, where topics will not be shared between them .
I have installed kafka_2.11-1.1.0 and set advertised listener to advertised.listeners=PLAINTEXT://<my-ip>:9092 (in $KAFKA_HOME/config/server.properties).
I can connect and write to my kafka using java code and see my cluster via kafka-tool from another server but I can't write messages to my topic from my local machine (the one that I have installed kafka cluster on it).
I have also tried to set listeners value to listeners = PLAINTEXT://:9092 but there is no change. What should I do to my kafka to make it reachable and writable from both outside and inside of the localhost?
In the server.properties use these two following properties
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://<your ip>:9092
I finally solved the issue by changing my code's org.apache.kafka library from version 1.1.0 to version 2.1.0.
I mention that all of these libraries were imported (downloaded) and used via mvnrepository.com.
Also, our kafka producer and consumer code pattern were written using this article:
https://dzone.com/articles/kafka-producer-and-consumer-example.
Have a look in the below following links, it may be helpful for your scenario,
Kafka access inside and outside docker
Kafka Listeners - Explained
My company is about to introduce kafka. However, i was not able to conprehend why either zookeeper or kafka confinguration, does not require to specify one or another existence.
For example, i neither find definition of kafka ip in zookeeper nor in kakfa definition of zookeeper ip in their config.
Can someone explain ?
for kafka server you should have server.properties file. It contains property zookeeper.connect
official documentation: https://kafka.apache.org/documentation/#brokerconfigs
I am looking at using Kafka but the documentation states that I need to set up Zookeeper. I already have a service discovery set up, I am using Consul. I don't want to have to look after Zookeeper as well.
Can you use Consul instead of Zookeeper to run Kafka? If so is there any documentation on how to do this anywhere?
Apache Kafka currently supports only Apache Zookeeper. It doesn't have any out of the box support for Consul.
This long-standing Kafka issue tracks adding Zookeeper alternatives. It may never happen, so in the meantime, there's the option of running a Zookeeper proxy like zetcd or parkeeper in front of etcd or Consul.
Kafka only uses Zookeeper for broker and topic discovery. Short of adding Consul support to the code base yourself, there is no other alternatives.
kafka has a zookeeper.
Is it ok to use it on production?
bin/zookeeper-server-start.sh
I want to use SASL with kafka. However I cann't find a way to chieve it with the offical zookeeper. I did make it work with the kafka zookeeper. Therefore I want to know if it's ok to use the zookeeper which is in kafka on production environment.
Yes the zookeeper that comes bundled with Apache Kafka is great for production use. No need to install any different version of zookeeper from anywhere else.