I'm writing a Go service that works with Kafka. I have a problems with bad commits when broker rebalances. I want to do an experiment forcing Kafka to rebalance and to see how the service behaves.
What I do:
running Kafka in Docker locally (broker, zookeeper, schema registry and control center)
created a topic with 2 partition
running producer that sends messages to both partitions
Then I'm running two consumers with the same groupID, after that I'm closing one. It seems to me that broker should start rebalancing this moment. Or no? Who's logs should I check for it?
You can check that by running the following commands:
bin/kafka-consumer-groups --bootstrap-server host:9092 --list
and to describe:
bin/kafka-consumer-groups --bootstrap-server host:9092 --describe --group foo
Full documentation could be found here: Kafka consumer group
Who's logs should I check for it?
The running consumer's log should be checked, depending on if the library you're using actually logs such information.
Related
for consumer
bin/windows/kafka-console-consumer.bat --bootstrap-server localhost:9092 --top
irer
and for producer
bin/windows/kafka-console-producer.bat --broker-list localhost:9092 --topic test
because I think for producer, we can also only choose one broker, from this broker server, we can find the all the partition information and find the leader, so I don't think broker-list is really needed.
This is for historical reasons, in either case, the provided brokers are only used to bootstrap and discover the full cluster.
Before Kafka 0.9, the consumer was still using Zookeeper to bootstrap. At that time, the producer was already using --broker-list.
In 0.9, when the "new" consumer was added, the flag to specify the broker was named --bootstrap-server for good reason as it is exactly what it is. Since then, the tools have used different flag name even though they are the same thing.
This was annoying and finally in 2.5.0, released just a few weeks ago, all tools have been updated to use --bootstrap-server!
When trying to create topic using --bootstrap-server,
I am getting exception "Error while executing Kafka topic command: Timed out waiting for a node" :-
kafka-topics --bootstrap-server localhost:9092 --topic boottopic --replication-factor 3 --partitions
However following works fine, using --zookeeper :-
kafka-topics --zookeeper localhost:2181--topic boottopic --replication-factor 3 --partitions
I am using Kafka version 2.5 and as per knowledge since version >2.2, all the offsets and metadata are stored on the broker itself. So, while creating topic there's no need to connect to zookeeper.
Please help to understand this behaviour
Note - I have set up a Zookeeper quorum and Kafka broker cluster each containing 3 instance on a single machine (for dev purposes)
Old question, but Ill answer anyways for the sake of internet wisdom.
You probably have auth set, when using --bootstrap-server you need to also specify your credentials with --command-config
since version >2.2, all the ... metadata are stored on the broker itself
False. Topic metadata is still stored on Zookeeper until KIP-500 is completed.
The AdminClient.createTopics() method, however that is used internally will delegate to Zookeeper from the Controller broker node in the cluster.
Hard to say what the error is, but most common issue is that Kafka is not running, you have SSL enabled and the certs are wrong, or the listeners are misconfigured.
For example, in the listeners, the default broker port on a Cloudera Kafka installation would be 6667, not 9092
each containing 3 instance on a single machine
Running 3 instances on one machine does not improve resiliency or performance unless you have 3 CPUs and 3 separate HDDs on that one motherboard.
"Error while executing Kafka topic command: Timed out waiting for a
node"
This seems like your broker is down or is inaccessible from where you are running those commands or it hasn't started yet (perhaps still starting).
Sometimes the broker startup takes long because it performs some cleaning operations. You may want to check your Kafka broker startup logs and see if it is ready and then try creating the topics by giving in the bootstrap servers.
There could also be some errors during your Kafka broker startup like Too many open files or wrong zookeeper url, zookeeper not being accessible by your broker, to name a few.
If you are able to create topics by passing in your Zookeeper URL means that zookeeper is up but does not necessarily mean that your Kafka broker(s) are also up and running.
Since a zookeeper can start without a broker but not vice-versa.
I have a Kafka cluster consisting on 3 servers all connected through Zookeeper. But when I delete a topic that has some information and create the topic again with the same name, the offset does not start from zero.
I tried restarting both Kafka and Zookeeper and deleting the topics directly from Zookeeper.
What I expect is to have a clean topic When I create it again.
I found the problem. A consumer was consuming from the topic and the topic was never actually deleted. I used this tool to have a GUI that allowed me to see the topics easily https://github.com/tchiotludo/kafkahq. Anyway, the consumers can be seen running this:
bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
I've spent some hours to figure out what was going on but didn't manage to find the solution.
Here is my set up on a single machine:
1 zookeeper running
3 broker running (on port 9092/9093/9094)
1 topic with 3 partitions and 3 replications (each partition are properly assigned between brokers)
I'm using kafka console producer to insert messages. If i check the replication offset (cat replication-offset-checkpoint), I see that my messages are properly ingested by Kafka.
Now I use the kafka console consumer (new):
sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic testTopicPartitionned2
I dont see anything consumed. I tried to delete my logs folder (/tmp/kafka-logs-[1,2,3]), create new topics, still nothing.
However when I use the old kafka consumer:
sudo bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic testTopicPartitionned2
I can see my messages.
Am I missing something big here to make this new consumer work ?
Thanks in advance.
Check to see what setting the consumer is using for auto.offset.reset property
This will affect what a consumer group without a previously committed offset will do in terms of setting where to start reading messages from a partition.
Check the Kafka docs for more on this.
Try providing all your brokers to --bootstrap-server argument to see if you notice any differnce:
sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --from-beginning --topic testTopicPartitionned2
Also, your topic name is rather long. I assume you've already made sure you provide the correct topic name.
I am newbie to Kafka, few days back from producer few topics are created ( automatically) i.e. with partition 1 , reflection fact -1 and ISR -1.
It worked fine , used to consume all topics messages fine.
Today i.e. after two days I ran my producer and consumer program and vice-versa too, but my consumer not able to consume/read message from the topic.
I checked all logs , no clue found what went wrong.
What is going wrong ?
Will the topics become stale after some time?
Is there any property value i need to check in kafka-server properties ?
Please help me.
Thank you.
~Shyam
There are several ways you can check the health of kafka cluster with the various tools provided.
Use the ConsumerOffsetChecker class provided to validate if there is any lag between the producer and consumer.
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect <zk host/ip>:<zk port> --group <consumer group name>
Use the JMX metrics such as belowto verify if the messages are been produced at the cluster level and there are additional metrics.
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec
kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower}
Use the Console Consumer to validate if the messages are present on the topic
bin/kafka-console-consumer.sh --zookeeper <zk host/ip>:<zk port> --topic test --from-beginning
Verify the log.retention.XXX values in kafka configuration(server.properties file)
Additional JMX monitoring details and configurations are available in documentation link
The last point is a little complicated to explain but I will try. Look at the blog link on confluent.io,it talks about the producers buffering the message before sending them to broker in the section More Partitions May Require More Memory In the Client. Not sure if your problem is related.