On a couple of my clusters I'm seeing a discrepancy between the list of topics returned by zookeeper as compared to the broker i.e the following commands return different (fewer in the case of the broker) results
kafka-topics.sh --zookeeper $zookeeper --list
kafka-topics.sh --bootstrap-server $broker --command-config $clientProperties --list
I've seen this behaviour with multiple client versions which leads me to assume that the issue is on the server side, but I have no idea what the root cause is or how to fix it.
It causes an issue for me because I'm using some code that uses the brokers for GET operations like listing topics, and zookeeper for SET operations (create/updating topics). If the broker doesn't return a topic in a listing, then the code path leads to a CREATE action against zookeeper and that will be rejected (it will fail). Unfortunately, I don't control the code so I can't apply a fix there.
Nonetheless, surely the list of topics in zookeeper should be identical to the list in the broker?
I'm using Kafka (Amazon MSK) version 2.2.1
Thanks for the suggestions in this post. This is the explanation and solution:
The command "kafka-topics.sh --zookeeper" and "kafka-topics.sh --bootstrap-server" return two different outputs because the latter takes into account the configured ACLs which, in this case, prevent access to the topic metadata. Hence, the command through zookeeper provides the full list of topics, whereas the command through the broker provides only the topics for which ACLs are not configured.
In order to ensure the second command works as expected, you need to explicitly add to the ACL list of the affected topics access to the "DESCRIBE" operation
(^^ kudos to AWS Support for figuring this out)
Related
I'm trying to run some tests to understand MM2 behavior. As part of that I had the following questions:
How to correctly pass a custom consumer group for MM2 in mm2.properties?
Based on this question, tried passing <alias>.group.id=temp_cons_group in mm2.properties and on restarting the MM2 instance could see the consumer group mentioned in the MM2 logs.
However, when I try listing consumer groups registered in the source broker, the group doesn't show up?
How to test if the property <alias>.consumer.auto.offset.reset works?
Here, I want to consume the same messages again so in reference to the question, tried setting <source_alias>.consumer.auto.offset.reset to earliest and restarted MM2.
I was able to see the property set correctly in MM2 logs but did not get the messages from the beginning in the target cluster topic.
How do I start a MM2 instance to start consuming messages from a specific offset for a topic present in the source cluster?
MirrorMaker does not use a consumer group to run and instead uses the assign() API, so it's expected that you don't see a group.
It's hard to "test". One way to verify this configuration was picked up is to check it's present in the logs when MirrorMaker starts its consumers.
This is currently not trivial to do. There's a KIP in progress to improve the process but at the moment it requires manually updating the internal offset topic from your Connect instance. At a very high level, here's the process:
First, ensure MirrorMaker is not running. Then you need to find the offset records for MirrorMaker in the offsets topic using a command like:
./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
--topic <CONNECT_OFFSET_TOPIC \
--from-beginning \
--property print.key=true | grep <SOURCE_CONNECTOR_NAME>
You will see records with offsets for each partition MirrorMaker handles. To update the offsets, you need to produce new records to this topic with the offsets you want. For each partition, ensure your record has the same key as the existing message so it replaces the existing stored offsets.
When trying to create topic using --bootstrap-server,
I am getting exception "Error while executing Kafka topic command: Timed out waiting for a node" :-
kafka-topics --bootstrap-server localhost:9092 --topic boottopic --replication-factor 3 --partitions
However following works fine, using --zookeeper :-
kafka-topics --zookeeper localhost:2181--topic boottopic --replication-factor 3 --partitions
I am using Kafka version 2.5 and as per knowledge since version >2.2, all the offsets and metadata are stored on the broker itself. So, while creating topic there's no need to connect to zookeeper.
Please help to understand this behaviour
Note - I have set up a Zookeeper quorum and Kafka broker cluster each containing 3 instance on a single machine (for dev purposes)
Old question, but Ill answer anyways for the sake of internet wisdom.
You probably have auth set, when using --bootstrap-server you need to also specify your credentials with --command-config
since version >2.2, all the ... metadata are stored on the broker itself
False. Topic metadata is still stored on Zookeeper until KIP-500 is completed.
The AdminClient.createTopics() method, however that is used internally will delegate to Zookeeper from the Controller broker node in the cluster.
Hard to say what the error is, but most common issue is that Kafka is not running, you have SSL enabled and the certs are wrong, or the listeners are misconfigured.
For example, in the listeners, the default broker port on a Cloudera Kafka installation would be 6667, not 9092
each containing 3 instance on a single machine
Running 3 instances on one machine does not improve resiliency or performance unless you have 3 CPUs and 3 separate HDDs on that one motherboard.
"Error while executing Kafka topic command: Timed out waiting for a
node"
This seems like your broker is down or is inaccessible from where you are running those commands or it hasn't started yet (perhaps still starting).
Sometimes the broker startup takes long because it performs some cleaning operations. You may want to check your Kafka broker startup logs and see if it is ready and then try creating the topics by giving in the bootstrap servers.
There could also be some errors during your Kafka broker startup like Too many open files or wrong zookeeper url, zookeeper not being accessible by your broker, to name a few.
If you are able to create topics by passing in your Zookeeper URL means that zookeeper is up but does not necessarily mean that your Kafka broker(s) are also up and running.
Since a zookeeper can start without a broker but not vice-versa.
Where does Zookeeper(or Kafka) keep its ACL list?
When you run scripts like kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --list --topic test, where does Zookeeper (or Kafka) get its list?
I am trying to find a file that stores all the ACLs.
You can access Zookeeper using the zookeeper-shell.sh script.
There is a znode called kafka-acl where information about ACLs for group, topic, cluster and so on are stored.
You can list for example information about ACLs on topics ls /kafka-acl/Topic.
Then getting information about a specific topic with get /kafka-acl/Topic/test.
Since I landed here searching for the same information and eventually stubbled my way to the answer, I thought to add some additional information. Since Apache Kafka ver 2.0, for topics with patternType=PREFIXED, the acls are stored under a zookeeper node /kafka-acl-extended, this is in addition to the /kafka-acl node which hold topic details of patterType=LITERAL.
For more details, read - KAFKA-KIP-290
For additional reference :
If you look at your Zookeeper configuration file (zoo.cfg or zookeeper.properties), you will see the dataDir parameter, which tells you where the zookeeper stores its data.
For example,
dataDir=/tmp/confluent.iSAdMTvO/zookeeper/data
So, Kafka ACL list will be stored there, but in order to control it or view it, use
zookeeper-shell script. Because if you open the data to see, you won't be able to recognize it. See below for those who are curious.
I was using a kafka topic, and it's metadata as well in my application. I hard deleted the topic from the zookeeper shell, by deleting the directories corresponding to that topic. After creating the topic again, I described the topic and found that no leaders have been assigned to this newly created topic. In the consumer, I can see repeated logs printing LEADER_NOT_AVAILABLE. Any reason as to what am I doing wrong? Or maybe is there a way to delete the metadata related to the kafka topic as well that I'm unaware of? Thanks in advance!
Deleting topics in Kafka hasn't been straightforward until recently. In general, you shouldn't attempt to delete Kafka topics by deleting metadata in Zookeeper. You should always use the included command line utilities.
First you need to make sure that deleting topics is enabled in the server.properties file on all brokers, and do a rolling restart if needed:
delete.topic.enable=true
After you restart the brokers to enable topic deletion, you should issue the delete command using the command line utilities:
./kafka-topics.sh —zookeeper <zookeeper_host>:2181 —delete —topic <topic_name>
If at this point, it's still stuck, try to run these two commands from the zookeeper shell to make sure and remove all metadata for that particular topic:
rmr /brokers/topics/<topic_name>
rmr /admin/delete_topics/<topic_name>
A few more details here:
https://medium.com/#contactsunny/manually-delete-apache-kafka-topics-424c7e016ff3
I am newbie to Kafka, few days back from producer few topics are created ( automatically) i.e. with partition 1 , reflection fact -1 and ISR -1.
It worked fine , used to consume all topics messages fine.
Today i.e. after two days I ran my producer and consumer program and vice-versa too, but my consumer not able to consume/read message from the topic.
I checked all logs , no clue found what went wrong.
What is going wrong ?
Will the topics become stale after some time?
Is there any property value i need to check in kafka-server properties ?
Please help me.
Thank you.
~Shyam
There are several ways you can check the health of kafka cluster with the various tools provided.
Use the ConsumerOffsetChecker class provided to validate if there is any lag between the producer and consumer.
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect <zk host/ip>:<zk port> --group <consumer group name>
Use the JMX metrics such as belowto verify if the messages are been produced at the cluster level and there are additional metrics.
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec
kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower}
Use the Console Consumer to validate if the messages are present on the topic
bin/kafka-console-consumer.sh --zookeeper <zk host/ip>:<zk port> --topic test --from-beginning
Verify the log.retention.XXX values in kafka configuration(server.properties file)
Additional JMX monitoring details and configurations are available in documentation link
The last point is a little complicated to explain but I will try. Look at the blog link on confluent.io,it talks about the producers buffering the message before sending them to broker in the section More Partitions May Require More Memory In the Client. Not sure if your problem is related.