I'm playing around with Kafka and using my own local single instance of zookeeper + kafka and running into this error that I don't seem to understand how to resolve.
I started a simple server per the Apache Kafka Quickstart Guide
$ bin/zookeeper-server-start.sh config/zookeeper.properties
$ bin/kafka-server-start.sh config/server.properties
Then utilizing kafkacat (installed via Homebrew) I started a Producer that will just echo messages that I type into the console
$ kafkacat -P -b localhost:9092 -t TestTopic -T
test1
test1
But when I try to consume those messages I get an error:
$ kafkacat -C -b localhost:9092 -t TestTopic
% ERROR: Topic TestTopic error: Broker: Leader not available
And similarly when I try to list its' metadata
$ kafkacat -L -b localhost:9092 -t TestTopic
Metadata for TestTopic (from broker -1: localhost:9092/bootstrap):
0 brokers:
1 topics:
topic "TestTopic" with 0 partitions: Broker: Leader not available (try again)
My questions:
Is this an issue with my running instance of zookeeper and/or kafkacat - I ask this because I've been constantly shutting them down and restarting them, after deleting the /tmp/zookeeper and /tmp/kafka-logs directories
Is there some simple setting that I need to try? I tried adding auto.leader.rebalance.enable=true in Kafka's server.properties settings file, but that didn't fix this particular issue
How do I do a fresh restart of zookeeper/kafka. Is shutting them down, deleting the /tmp/zookeeper and /tmp/kafka-logs directories and then restarting zookeeper and then kafka the way to go? (Well maybe the way to go is to build a docker container that I can stand-up and tear down, I was going to use the spotify/docker-kafka container but that is not on Kafka 0.9.0.0 and I haven't taking the time to build my own)
It might be, but probably is not. My guess is the topic isn't created, so kafkacat echoes the massage on screen but doesn't really send it to kafka. All the topics are probably deleted after you delete the /tmp/kafka-logs
No. I don't think this is the way to look for a solution.
Having a docker container is definitely the way to go - you'll soon end up running kafka on multiple brokers, examining the replication behavior, high availability scenarios etc.. Having it dockerised helps a lot.
Related
I'm starting with Zookeeper and Kafka - after executing zookeeper-shell.sh commands I'm still getting EndOfStreamException. So far it doesn't cause broker failures (still getting valid responses) but I would like to resolve it before it starts. I operate on single cluster with 1 Zookeeper and 3 Kafka brokers. Commands I used so far are simplest possible:
/bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids
/bin/zookeeper-shell.sh localhost:2181 get /brokers/ids/{id}
Zookeeper command
Zookeeper logs
You're running a non-interactive shell command, then it exits successfully with code 0 and ends the session.
I have deleted topic directly from Zookeeper using the below command and did not execute deletion from Kafka before:
zookeeper-shell.sh localhost:2181 rmr /brokers/topics/<topic_name>
Now what I see is that the topic shows up in the log.dirs of at least one broker in the cluster. Is there a way that can be deleted as well.
When i attempt to delete from kafka now it throws the below error
Error while executing topic command : Topic <topic_name> does not exist on ZK path <zookeeper_server_list:2181>
I think you have missed a couple of steps. In order to manually delete a topic you need to follow these steps:
1) Stop Kafka server
2) On each broker, you have you have to delete all the topic's log files under logs.dirs:
rm -rf path/to/logs/topic_name/
3) Remove topic directory from Zookeeper:
> zookeeper-shell.sh localhost:2181
> ls /brokers/topics
> rmr /brokers/topics/topic_name
4) Restart Kafka server
Note that the suggested way for deleting a topic is
/bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic topic_name
assuming that delete.topic.enable=true.
Sometimes its necessary to fresh-start a kafka cluster with no data. when running a kafka inside docker containers this behavior is achieved for free.
How to do it with kafka process ? can i delete /var/log/kafka* and restart it ? is it ok to do so ?
BTW - i am using something like this :
# bash shell
# tl is a list of all topics
for t in $(cat tl); do
./kafka-topics.sh --zookeeper $ZOO --delete --topic $t
done
there are 2 problem with the above :
if hdd usage is 100%, then i got error when trying the kafka-topics.sh
its very inefficient if i have many topics
looking for a fast and clean way to do in dev envs.
seems like this do the job
$ ###### stop and clear all brokers
$ sudo systemctl stop kafka.service zookeeper.service
$ sudo rm -rf /var/log/kafka-logs/*
$ ###### continue ONLY after finish the above on all brokers
$ sudo systemctl start zookeeper.service
$ sleep 10s # make sure zookeeper is ready
$ sudo systemctl start kafka.service
I would suggest you to temporary set log.retention.ms to low value, lets say 1000. This way you tell kafka to keep messages only one second before deleting it forever. You wait a little bit, kafka will delete all of your messages(all but not messages on active segments) and when that is done, you can revert that log.retention.ms settings.
I have installed a kafka that has integrated zookeeper.
I have seen that to enter an independent Zookeeper installation, you can run the following command to enter the zookeeper console:
bin/ZkCli.sh
ls /zookeeper/quota
But in Kafka's scripts I only have:
zookeeper-security-migration.sh
zookeeper-server-start.sh
zookeeper-server-stop.sh
zookeeper-shell.sh
I have tried to do the following:
./zookeeper-shell.sh -server 127.0.0.1:2181 ls /zookeeper/quota
But it doesn't work, it doesn't do anything
How can I get into the Zookeeper that is integrated in Kafka?
After starting Zookeeper, you can connect to it using the zookeeper-shell.sh tool.
To get into the shell:
./zookeeper-shell IP:2181
Then you can execute commands, like:
ls /
You can use cd to move within the nodes and get to print the content of nodes.
You can also use this script to just run commands and return (without getting into the shell)
./zookeeper-shell.sh localhost:2181 get /controller
/zookeeper/quota is not a path used by Kafka, Quotas are stored under /config
I installed kafka on my machine using brew.
brew install kafka
I opened the following file
/usr/local/etc/kafka/server.properties
By default kafka brokers run on 9092. but in my case this port is already taken by elastic search.
So I made the following entry in this file
listeners=PLAINTEXT://abhisheks-mini:9093
Now i restarted kafka using brew services restart kafka
but now if I try to do
./kafka-topics.sh --create --zookeeper localhost:2181 --partitions 1 --replication-factor 1 --topic test
I get an error
Error while executing topic command : replication factor: 1 larger than available brokers: 0
[2017-09-30 12:39:18,076] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: replication factor: 1 larger than available brokers: 0
(kafka.admin.TopicCommand$)
I think this is because I changed the port and the utility does not know about the new port.
Currently kafka-topics.sh only talks to zookeeper and according to ZK you don't have a broker running.
I suggest you check the broker logs (/usr/local/var/log/kafka) to see why it failed starting.
Also if you only want to change the port, you don't need to provide a hostname/ip, you just need:
listeners=PLAINTEXT://:9093