I am trying to set up the following on a single remote AWS EC2 machine.
1 Kafka Zookeeper
3 Kafka brokers
1 Kafka topic
I have done the following:
I have created two copies of the server.properties file and named them server_1.properties and server_2.properties, where I changed the following values:
broker.id=0 to broker.id=1 and broker.id=2 respectively
advertised.listeners=PLAINTEXT://your.host.name:9092 to advertised.listeners=PLAINTEXT://3.72.250.103:9092
advertised.listeners=PLAINTEXT://3.72.250.103:9093
advertised.listeners=PLAINTEXT://3.72.250.103:9094 in the three config files
changed log.dirs=/tmp/kafka-logs to
log.dirs=/tmp/kafka-logs_1 and log.dirs=/tmp/kafka-logs_2 in the respective files
All three brokers start up fine, and
bin/zookeeper.sh localhost:2181 ls /brokers/ids
shows all three brokers
But when I try to create a topic like:
bin/kafka-topics.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --create -- partitions 5 --replication-factor 1 --topic cars
the command times out with the error message: Timed out waiting for node assignment
What have I tried:
I tried creating only one broker - that worked fine
I tried creating the topic with --bootstrap-server localhost:9092 only - that did not work
I tried changing listeners=PLAINTEXT://:9092 to listeners=PLAINTEXT://3.72.250.103:9092 and for the 2nd and 3rd broker respectively - I was not even able to start the brokers with this configuration.
I am not sure what to try next
Related
I have created two kafka brokers in a kafka cluster. When one broker is down I am not able to get any data to kafka consumer.
I am using this command to read messages from consumer:
bin/kafka-console-consumer.sh --topic test_kafka_cluster \
--bootstrap-server 127.0.0.1:9092,127.0.0.2:9092 --from-beginning
Here as per your console consumer configuration, IP address used here are 127.0.0.1 and 127.0.0.2 and two bootstrap servers are configured as 9092.
Verify both the ip's are reachable
bin/kafka-console-consumer.sh --topic test_kafka_cluster \
--bootstrap-server 127.0.0.1:9092,127.0.0.2:9092 --from-beginning
Ideally when we run tow kafka broker instance it will be running in two different ports.
Presuming Kafka is running in local
Eg: localhost:9092 localhost:9093
Kafka instances running on two different host:
Eg: 127.0.0.3:9092, 127.0.0.2:9092
If Kafka is running on docker/docker toolbox:
Console consumer on Docker toolbox:
docker exec <container-name> kafka-console-consumer --bootstrap-server 192.168.99.100:9093 --topic <topic-name> --from-beginning
Console consumer on Docker:
docker exec <container-name> kafka-console-consumer --bootstrap-server localhost:9093 localhost 9092 --topic <topic-name> --from-beginning
There are two parameters that affect topics availability for a consumer:
min.insync.replicas (minISR): Minimum number of in-sync partition's replicas.
replication.factor (RF): Total number of partition's replicas.
If you want your consumer to survive a broker outage, then you must have RF > minISR. With RF=2 and minISR=2 you can't tolerate any broker down, with RF=3 and minISR=2 you can tolerate 1 broker down, with RF=5 and minISR=2 you can tolerate 3 brokers down, and so on.
Note that the internal __consumer_offsets topic is used to store consumer offsets and has a default RF value of 3, which is not achievable in your cluster of 2 nodes. So you also need to set offsets.topic.replication.factor=1 at the cluster level.
We use Kafka in Docker container. We create topics automatically if the topic does not exist when producing or consuming messages. We want 3 partitions for the topics, so set
num.partitions=3
in file /etc/kafka/server.properties in the Kafka container. However, it does not take effect. After doing the setting and restarting the container, then try subscribing or publishing on some non-existential topics, the topics are created, but only with one partition.
We tried this on containers created from image confluentinc/cp-kafka:5.1.0 and also on containers created from image confluentinc/cp-enterprise-kafka:5.3.1, and the behaviors were the same.
We tested creating topics with command:
kafka-topics --create --topic my_topic --zookeeper zookeeper:2181 --replication-factor 1 --partitions 3
This correctly created the topic with three partitions. But we need Kafka to create multi-partition topics automatically.
What could cause the problem? Or how to make Kafka auto-create multi-partition topics?
We do not have any dynamic configs. This is verified by running the following commands:
kafka-configs --bootstrap-server kafka:9092 --entity-type brokers --entity-default --describe
kafka-configs --bootstrap-server kafka:9092 --entity-type brokers --entity-name 0 (or other ids) --describe
Those commands return empty results.
This answer comes kinda late, but I've been struggling with the same thing using the docker image: confluentinc/cp-enterprise-kafka:5.3.4.
The solution for me was adding a new environment variable in my docker-compose:
KAFKA_NUM_PARTITIONS: 3 (or the partitions you want)
This will automatically add the property num.partitions in your kafka.properties file under /etc/kafka/ directory.
Modifying the property num.partitions in /etc/kafka/server.properties didn't work for me neither.
Docker containers are ephemeral, meaning that once you stopped them, all those changes that you've applied are lost.
If you want to overwrite the default settings you have to mount the property file:
Create a server.properties file on your machine
Fill it in with the properties that you need ( including the original ones )
Mount this file and in order to replace the original one from the container:
docker run ... -v /path/to/custom/server.properties:/etc/kafka/server.properties ...
I've a local Apache Kafka setup and there are total 2 broker (id - 0 and 1) on port 9092 and 9093.
I created a topic and published the messages using this command:
bin/kafka-console.-producer.sh --broker-list localhost:9092 --topic test
Then I consumed the messages on other terminal using the command:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
Till now everything is fine.
But when i type command -
bin/kafka-console.-producer.sh --broker-list localhost:9093 --topic test
and write some messages it is showing in the 2nd terminal where I've typed this command -
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
Why port 9093 messages are publishing to 9092?
Your cluster contains of two brokers. It is not important which host you use for initial connection. Using kafka client you don't specify from which broker you consume or to which your produce messages. Those hostname are only to discover whole list of kafka brokers (cluster)
According to documentation:
https://kafka.apache.org/documentation/#producerconfigs
https://kafka.apache.org/documentation/#consumerconfigs
bootstrap.servers::
A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers.
I had a Kafka cluster with three brokers.
I killed two of them by mistake.
I restarted them with the same server.properties config files that was used in running them the first time, but it is not functioning correctly.
By this I mean when I run bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic <topic_name> --from-beginning, it prints nothing, although when I replace the localhost with the address of broker server that had not been killed in --zookeeper localhost:2181, it has all the messages.
It seems like the two restarted Kafka brokers are not finding which cluster they should belong.
How can I fix this?
Also, how do brokers in a cluster recognize each other's address? By zookeeper.connect field in server.properties file?
I am trying to setup multi node Kafka-0.8.2.2 cluster with 1 Producer, 1 consumer and 3 brokers all on different machines.
While creating topic on producer, I am getting error as org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids. Complete console output is available here. There is no error in Kafka Producer's log.
Command I am using to run Kafka is:
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic edwintest
Note: Zookeeper service is running on all the servers, and all three brokers are having Kafka servers running on them (Only Brokers need Kafka Server. Right?).
Configuration of my producer.properties is as:
metadata.broker.list=<IP.OF.BROKER.1>:9092,<IP.OF.BROKER.2>:9092,<IP.OF.BROKER.3>:9092
producer.type=sync
compression.codec=none
serializer.class=kafka.serializer.DefaultEncoder
Below are some of the many articles I was using as reference:
Zookeeper & Kafka Install : A single node and a multiple broker cluster - 2016
Step by Step of Installing Apache Kafka and Communicating with Spark
At the very first glance it seems like you're calling create topic to a local zookeeper which is not aware of any of your kafka-brookers. You should call ./bin/kafka-topics.sh --create --zookeeper <IP.OF.BROKER.1>:2181
The issue was because I was trying to connect to the zookeeper of localhost. My understanding was zookeeper is needed to be running on producer, consumer, and the Kafka brokers, and the communication is done between producer -> broker and broker -> consumer via zookeeper. But that was incorrect. Actually:
Zookeeper and Kafka servers should be running only on on broker servers. While creating the topic or publishing the content to the topic, public DNS of any of the Kafka broker should be passed with --zookeeper option. There is no need to run Kafka server on producer or consumer instance.
Correct command will be:
./bin/kafka-topics.sh --create --zookeeper <Public-DNS>:<PORT> --replication-factor 1 --partitions 3 --topic edwintest
where: Public-DNS is the DNS of any of the Kafka broker and PORT is the port of the zookeeper service.