I have created two kafka brokers in a kafka cluster. When one broker is down I am not able to get any data to kafka consumer.
I am using this command to read messages from consumer:
bin/kafka-console-consumer.sh --topic test_kafka_cluster \
--bootstrap-server 127.0.0.1:9092,127.0.0.2:9092 --from-beginning
Here as per your console consumer configuration, IP address used here are 127.0.0.1 and 127.0.0.2 and two bootstrap servers are configured as 9092.
Verify both the ip's are reachable
bin/kafka-console-consumer.sh --topic test_kafka_cluster \
--bootstrap-server 127.0.0.1:9092,127.0.0.2:9092 --from-beginning
Ideally when we run tow kafka broker instance it will be running in two different ports.
Presuming Kafka is running in local
Eg: localhost:9092 localhost:9093
Kafka instances running on two different host:
Eg: 127.0.0.3:9092, 127.0.0.2:9092
If Kafka is running on docker/docker toolbox:
Console consumer on Docker toolbox:
docker exec <container-name> kafka-console-consumer --bootstrap-server 192.168.99.100:9093 --topic <topic-name> --from-beginning
Console consumer on Docker:
docker exec <container-name> kafka-console-consumer --bootstrap-server localhost:9093 localhost 9092 --topic <topic-name> --from-beginning
There are two parameters that affect topics availability for a consumer:
min.insync.replicas (minISR): Minimum number of in-sync partition's replicas.
replication.factor (RF): Total number of partition's replicas.
If you want your consumer to survive a broker outage, then you must have RF > minISR. With RF=2 and minISR=2 you can't tolerate any broker down, with RF=3 and minISR=2 you can tolerate 1 broker down, with RF=5 and minISR=2 you can tolerate 3 brokers down, and so on.
Note that the internal __consumer_offsets topic is used to store consumer offsets and has a default RF value of 3, which is not achievable in your cluster of 2 nodes. So you also need to set offsets.topic.replication.factor=1 at the cluster level.
Related
I am trying to set up the following on a single remote AWS EC2 machine.
1 Kafka Zookeeper
3 Kafka brokers
1 Kafka topic
I have done the following:
I have created two copies of the server.properties file and named them server_1.properties and server_2.properties, where I changed the following values:
broker.id=0 to broker.id=1 and broker.id=2 respectively
advertised.listeners=PLAINTEXT://your.host.name:9092 to advertised.listeners=PLAINTEXT://3.72.250.103:9092
advertised.listeners=PLAINTEXT://3.72.250.103:9093
advertised.listeners=PLAINTEXT://3.72.250.103:9094 in the three config files
changed log.dirs=/tmp/kafka-logs to
log.dirs=/tmp/kafka-logs_1 and log.dirs=/tmp/kafka-logs_2 in the respective files
All three brokers start up fine, and
bin/zookeeper.sh localhost:2181 ls /brokers/ids
shows all three brokers
But when I try to create a topic like:
bin/kafka-topics.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --create -- partitions 5 --replication-factor 1 --topic cars
the command times out with the error message: Timed out waiting for node assignment
What have I tried:
I tried creating only one broker - that worked fine
I tried creating the topic with --bootstrap-server localhost:9092 only - that did not work
I tried changing listeners=PLAINTEXT://:9092 to listeners=PLAINTEXT://3.72.250.103:9092 and for the 2nd and 3rd broker respectively - I was not even able to start the brokers with this configuration.
I am not sure what to try next
I've a local Apache Kafka setup and there are total 2 broker (id - 0 and 1) on port 9092 and 9093.
I created a topic and published the messages using this command:
bin/kafka-console.-producer.sh --broker-list localhost:9092 --topic test
Then I consumed the messages on other terminal using the command:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
Till now everything is fine.
But when i type command -
bin/kafka-console.-producer.sh --broker-list localhost:9093 --topic test
and write some messages it is showing in the 2nd terminal where I've typed this command -
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
Why port 9093 messages are publishing to 9092?
Your cluster contains of two brokers. It is not important which host you use for initial connection. Using kafka client you don't specify from which broker you consume or to which your produce messages. Those hostname are only to discover whole list of kafka brokers (cluster)
According to documentation:
https://kafka.apache.org/documentation/#producerconfigs
https://kafka.apache.org/documentation/#consumerconfigs
bootstrap.servers::
A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers.
Kafka Quickstart
Using Kafka v2.1.0 on RHEL v6.9
Consumer fails to receive data when one of the Kafka brokers is down.
Steps performed:
1. Start zookeeper
2. Start Kafka-Server0 (localhost:9092, kafkalogs1)
3. Start Kafka-Server1 (localhost:9094, kafkalog2)
4. Create topic "test1", num of partitions = 1, replication factor = 2
5. Run producer for topic "test1"
6. Run consumer
7. Send messages from the producer
8. Receive messages on the consumer side.
All the above steps worked without any issues.
When I shutdown Kafka-Server0, the consumer stops getting data from Producer.
When I bring back up Kafka-Server0, the consumer starts to get messages from where it left off.
These are the commands used
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test1
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test1
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 1 --topic test1
The behavior is the same (no message received on the consumer side) when I run the consumer with two servers specified in the --bootstrap-server option.
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9094 --topic test1
Any idea why the consumer stops getting messages when server0 is down even though the replication factor for the topic test1 was set to 2?
There is a similar question already but it was not answered completely
Kafka 0.10 quickstart: consumer fails when "primary" broker is brought down
If the offsets topic is unavailable, you cannot consume.
Look at the server.properties file for these, and see the comment above, and increase accordingly (only applies if topic doesn't already exist)
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
According to your previous question, looks like it only has one replica
See how you can increase replication factor for an existing topic
In initial versions of Kafka, offset was being managed at zookeeper, but Kafka has continuously evolved over the time introducing lot of new features. Now Kafka manages the offset in a topic __consumer_offsets.
You can think of a scenario where you created a topic with a replication factor of 1. In case the broker goes down the data is only on that Kafka node which is down. So you can't get this data. Same analogy applies to __consumer_offsets topic.
You need to revisit the server.properties in order to get features you are expecting. But in case you still wanna consume the messages from the replica partition, you may need to re-start the console consumer with --from-beginning true
I only changed the paths to logs and zookeeper-data in configs.
Running zookeeper:
zookeeper-server-start.bat D:\__programs\kafka_2.12-2.1.0\config\zookeeper.properties
Running kafka:
kafka-server-start.bat D:\__programs\kafka_2.12-2.1.0\config\server.properties
Running consumer:
kafka-console-consumer.bat -bootstrap-server localhost:2181 -topic mytopic
Running producer:
kafka-console-producer.bat -broker-list localhost:9092 -topic mytopic
So, consumer can only get messages when it's on the same port(9092) with a producer.
What's the problem here?
Consumers and producers are clients of Kafka server so they both can use same port as we configured as a client port within server.properties file.
Port is configured within config/server.properties file with either one of these parameters
advertised.listeners
listeners
Depends on configuration and protocol consumers and producers can use multiple ports.
You can find details of broker parameters here
Bootstrap server and Broker list, both should point to the same
Since you are producing events on localhost:9092, you should use the same in consumer as below:
kafka-console-producer.bat --broker-list localhost:9092 --topic mytopic
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic mytopic
P.S. Usually 2181 port is assigned to zookeeper process. And here localhost:2181 will refer to Zookeeper.
I am trying to setup multi node Kafka-0.8.2.2 cluster with 1 Producer, 1 consumer and 3 brokers all on different machines.
While creating topic on producer, I am getting error as org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids. Complete console output is available here. There is no error in Kafka Producer's log.
Command I am using to run Kafka is:
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic edwintest
Note: Zookeeper service is running on all the servers, and all three brokers are having Kafka servers running on them (Only Brokers need Kafka Server. Right?).
Configuration of my producer.properties is as:
metadata.broker.list=<IP.OF.BROKER.1>:9092,<IP.OF.BROKER.2>:9092,<IP.OF.BROKER.3>:9092
producer.type=sync
compression.codec=none
serializer.class=kafka.serializer.DefaultEncoder
Below are some of the many articles I was using as reference:
Zookeeper & Kafka Install : A single node and a multiple broker cluster - 2016
Step by Step of Installing Apache Kafka and Communicating with Spark
At the very first glance it seems like you're calling create topic to a local zookeeper which is not aware of any of your kafka-brookers. You should call ./bin/kafka-topics.sh --create --zookeeper <IP.OF.BROKER.1>:2181
The issue was because I was trying to connect to the zookeeper of localhost. My understanding was zookeeper is needed to be running on producer, consumer, and the Kafka brokers, and the communication is done between producer -> broker and broker -> consumer via zookeeper. But that was incorrect. Actually:
Zookeeper and Kafka servers should be running only on on broker servers. While creating the topic or publishing the content to the topic, public DNS of any of the Kafka broker should be passed with --zookeeper option. There is no need to run Kafka server on producer or consumer instance.
Correct command will be:
./bin/kafka-topics.sh --create --zookeeper <Public-DNS>:<PORT> --replication-factor 1 --partitions 3 --topic edwintest
where: Public-DNS is the DNS of any of the Kafka broker and PORT is the port of the zookeeper service.