I only changed the paths to logs and zookeeper-data in configs.
Running zookeeper:
zookeeper-server-start.bat D:\__programs\kafka_2.12-2.1.0\config\zookeeper.properties
Running kafka:
kafka-server-start.bat D:\__programs\kafka_2.12-2.1.0\config\server.properties
Running consumer:
kafka-console-consumer.bat -bootstrap-server localhost:2181 -topic mytopic
Running producer:
kafka-console-producer.bat -broker-list localhost:9092 -topic mytopic
So, consumer can only get messages when it's on the same port(9092) with a producer.
What's the problem here?
Consumers and producers are clients of Kafka server so they both can use same port as we configured as a client port within server.properties file.
Port is configured within config/server.properties file with either one of these parameters
advertised.listeners
listeners
Depends on configuration and protocol consumers and producers can use multiple ports.
You can find details of broker parameters here
Bootstrap server and Broker list, both should point to the same
Since you are producing events on localhost:9092, you should use the same in consumer as below:
kafka-console-producer.bat --broker-list localhost:9092 --topic mytopic
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic mytopic
P.S. Usually 2181 port is assigned to zookeeper process. And here localhost:2181 will refer to Zookeeper.
Related
I have created two kafka brokers in a kafka cluster. When one broker is down I am not able to get any data to kafka consumer.
I am using this command to read messages from consumer:
bin/kafka-console-consumer.sh --topic test_kafka_cluster \
--bootstrap-server 127.0.0.1:9092,127.0.0.2:9092 --from-beginning
Here as per your console consumer configuration, IP address used here are 127.0.0.1 and 127.0.0.2 and two bootstrap servers are configured as 9092.
Verify both the ip's are reachable
bin/kafka-console-consumer.sh --topic test_kafka_cluster \
--bootstrap-server 127.0.0.1:9092,127.0.0.2:9092 --from-beginning
Ideally when we run tow kafka broker instance it will be running in two different ports.
Presuming Kafka is running in local
Eg: localhost:9092 localhost:9093
Kafka instances running on two different host:
Eg: 127.0.0.3:9092, 127.0.0.2:9092
If Kafka is running on docker/docker toolbox:
Console consumer on Docker toolbox:
docker exec <container-name> kafka-console-consumer --bootstrap-server 192.168.99.100:9093 --topic <topic-name> --from-beginning
Console consumer on Docker:
docker exec <container-name> kafka-console-consumer --bootstrap-server localhost:9093 localhost 9092 --topic <topic-name> --from-beginning
There are two parameters that affect topics availability for a consumer:
min.insync.replicas (minISR): Minimum number of in-sync partition's replicas.
replication.factor (RF): Total number of partition's replicas.
If you want your consumer to survive a broker outage, then you must have RF > minISR. With RF=2 and minISR=2 you can't tolerate any broker down, with RF=3 and minISR=2 you can tolerate 1 broker down, with RF=5 and minISR=2 you can tolerate 3 brokers down, and so on.
Note that the internal __consumer_offsets topic is used to store consumer offsets and has a default RF value of 3, which is not achievable in your cluster of 2 nodes. So you also need to set offsets.topic.replication.factor=1 at the cluster level.
I've a local Apache Kafka setup and there are total 2 broker (id - 0 and 1) on port 9092 and 9093.
I created a topic and published the messages using this command:
bin/kafka-console.-producer.sh --broker-list localhost:9092 --topic test
Then I consumed the messages on other terminal using the command:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
Till now everything is fine.
But when i type command -
bin/kafka-console.-producer.sh --broker-list localhost:9093 --topic test
and write some messages it is showing in the 2nd terminal where I've typed this command -
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
Why port 9093 messages are publishing to 9092?
Your cluster contains of two brokers. It is not important which host you use for initial connection. Using kafka client you don't specify from which broker you consume or to which your produce messages. Those hostname are only to discover whole list of kafka brokers (cluster)
According to documentation:
https://kafka.apache.org/documentation/#producerconfigs
https://kafka.apache.org/documentation/#consumerconfigs
bootstrap.servers::
A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers.
./bin/kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --topic test
./bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
Why to consume kakfa needed zookeeper?
could I consume kafka from brokers ?
--zookeeper is an outdated option (for the old Consumer).
Use the newer --bootstrap-server option with kafka-console-consumer.
More info: http://kafka.apache.org/documentation/#newconsumerconfigs
To launch a Kafka consumer I have to run it with the following params.
kafka-console-consumer.bat --zookeeper localhost:2181 --topic MyTopic
Was wondering why it needs to have a zookeeper as a param, if I use the broker param will it not work similar to how the producer is launched. Consumer needs to be aware of the broker/cluster and not the zookeeper location.
.\kafka-console-producer.bat --broker --list localhost:9092 --topic MyTopic
This is on windows, am not sure how its in Unix flavors.
-Chandra
You're invoking the old Kafka Consumer. Old consumer requires zookeeper to co-ordinate from which broker to fetch data.
The new consumer don't requires the zookeeper parameter.
.\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic MyTopic --new-consumer
Optionally, you can add --from-beginning argument to read the old records in the topic.
I am trying to setup multi node Kafka-0.8.2.2 cluster with 1 Producer, 1 consumer and 3 brokers all on different machines.
While creating topic on producer, I am getting error as org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids. Complete console output is available here. There is no error in Kafka Producer's log.
Command I am using to run Kafka is:
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic edwintest
Note: Zookeeper service is running on all the servers, and all three brokers are having Kafka servers running on them (Only Brokers need Kafka Server. Right?).
Configuration of my producer.properties is as:
metadata.broker.list=<IP.OF.BROKER.1>:9092,<IP.OF.BROKER.2>:9092,<IP.OF.BROKER.3>:9092
producer.type=sync
compression.codec=none
serializer.class=kafka.serializer.DefaultEncoder
Below are some of the many articles I was using as reference:
Zookeeper & Kafka Install : A single node and a multiple broker cluster - 2016
Step by Step of Installing Apache Kafka and Communicating with Spark
At the very first glance it seems like you're calling create topic to a local zookeeper which is not aware of any of your kafka-brookers. You should call ./bin/kafka-topics.sh --create --zookeeper <IP.OF.BROKER.1>:2181
The issue was because I was trying to connect to the zookeeper of localhost. My understanding was zookeeper is needed to be running on producer, consumer, and the Kafka brokers, and the communication is done between producer -> broker and broker -> consumer via zookeeper. But that was incorrect. Actually:
Zookeeper and Kafka servers should be running only on on broker servers. While creating the topic or publishing the content to the topic, public DNS of any of the Kafka broker should be passed with --zookeeper option. There is no need to run Kafka server on producer or consumer instance.
Correct command will be:
./bin/kafka-topics.sh --create --zookeeper <Public-DNS>:<PORT> --replication-factor 1 --partitions 3 --topic edwintest
where: Public-DNS is the DNS of any of the Kafka broker and PORT is the port of the zookeeper service.