How to check Kafka server status or details? - apache-kafka

Is there a command to show the details of Kafka server or the status of Kafka server? (I am not trying to find out if the kafka server is running.)
I can only find information on topic, partition, producer, and consumer CLI commands.

If you are looking for the Kafka cluster broker status, you can use zookeeper cli to find the details for each broker as given below:
ls /brokers/ids returns the list of active brokers IDs on the cluster.
get /brokers/ids/<id> returns the details of the broker with the given ID.
Example :
kafka_2.12-1.1.1 % ./bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids
Connecting to localhost:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[0]
kafka_2.12-1.1.1 % ./bin/zookeeper-shell.sh localhost:2181 get /brokers/ids/0
Connecting to localhost:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://localhost:9092"],"jmx_port":-1,"host":"localhost","timestamp":"1558428038778","port":9092,"version":4}
cZxid = 0x116
ctime = Tue May 21 08:40:38 UTC 2019
mZxid = 0x116
mtime = Tue May 21 08:40:38 UTC 2019
pZxid = 0x116
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x16ad9830f16000b
dataLength = 188
numChildren = 0
You can put these steps in some shell script to get the details for all brokers.

You can activate JMX metrics by setting environment variable JMX_PORT.
$ export JMX_PORT=9010
$ ./bin/kafka-server-start.sh ./config/server.properties
Then, you can use jconsole or Java Mission Control to display cluster metrics.

Related

how to remaine the kafka retentions-bytes and kafka retention-segment even after kafka machine reboot [duplicate]

This question already exists:
kafka + how to set specific retention byets for specific topic as permamnet
Closed 2 years ago.
we set retention bytes value - 104857600 for topic - topic_test
[root#confluent01 ~]# kafka-topics --zookeeper localhost:2181 --alter --topic topic_test --config retention.bytes=104857600
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
Going forward, please use kafka-configs.sh for this functionality
Updated config for topic "topic_test".
Now we verify the retention bytes from the zookeeper:
[root#confluent01 ~]# zookeeper-shell confluent01:2181 get /config/topics/topic_test
Connecting to confluent1:2181
{"version":1,"config":{"retention.bytes":"104857600"}}
cZxid = 0xb30a00000038
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
ctime = Mon Jun 29 11:42:30 GMT 2020
mZxid = 0xb31100008978
mtime = Wed Jul 22 19:22:20 GMT 2020
pZxid = 0xb30a00000038
cversion = 0
dataVersion = 7
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 54
numChildren = 0
now we performed reboot to the kafka confluent01 machine
after machines started and kafka service start successfully , we checked again the retention-bytes from zookeeper:
but now ( after machine reboot ) we can see that the retention bytes isn't configured in zookeeper
[root#confluent01 ~]#zookeeper-shell confluent01:2181 get /config/topics/topic_test
Connecting to confluent1:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null  no retention bytes value
{"version":1,"config":{}}
cZxid = 0xb30a00000038
ctime = Mon Jun 29 11:42:30 GMT 2020
mZxid = 0xb3110000779b
mtime = Wed Jul 22 14:09:19 GMT 2020
pZxid = 0xb30a00000038
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 25
numChildren = 0
the question is - how to remain the retention bytes even after restart of kafka machine ?
NOTE - we not want to use the retention bytes from server.properties
because we set different retention bytes to each topic
Zookeeper and Kafka default to store data in /tmp
If you reboot the machines, /tmp is cleared
Otherwise, if you use confluent start command, this is not permanent data.
If you use Docker/Kubernetes without mounting any volumes, this is also not permanent.
You sould also be using kafka-topics --describe command rather than zookeeper-shell as Zookeeper will be removed completely in upcoming Kafka releases

how to capture the retention bytes values from zookeeper meta data

we are trying to capture the retention bytes values from the topic - topic_test
we try the following example , but seems this isnt the right path from zookeeper
zookeeper-shell kafka1:2181,kafka2:2181,kafka3:2181 <<< "ls /brokers/topics/topic_test/partitions/88/state"
Connecting to kafka1:2181,kafka2:2181,kafka3:2181
Welcome to ZooKeeper!
JLine support is disabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[]
any idea where are the values of retention bytes per topic that can capture from zookeeper?
I did the following , but not see the retention bytes ( what is wrong here ) , we have kafka confluent version - 0.1
zookeeper-shell confluent1:2181 get /config/topics/test_topic
Connecting to kafka1:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
{"version":1,"config":{}}
cZxid = 0xb30a00000038
ctime = Mon Jun 29 11:42:30 GMT 2020
mZxid = 0xb30a00000038
mtime = Mon Jun 29 11:42:30 GMT 2020
pZxid = 0xb30a00000038
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 25
numChildren = 0
Configurations are stored in Zookeeper under the /config path.
For example, for the topic topic_test:
# Create topic
./bin/kafka-topics.sh --bootstrap-server localhost:9092 --create \
--topic topic_test --partitions 1 --replication-factor 1 --config retention.bytes=12345
# Retrieve configs from Zookeeper
./bin/zookeeper-shell.sh localhost get /config/topics/topic_test
Connecting to localhost
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
{"version":1,"config":{"retention.bytes":"12345"}}
Note in most cases, you should not rely on direct access to Zookeeper but instead use the Kafka API to retrieve these values.
Using:
kafka-topics.sh:
./bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic topic_test
kafka-configs.sh:
./bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type topics --entity-name topic_test
The Admin API using describeConfigs()

How to see a topic creation and alteration timestamp in Kafka

Or at least one of them? I don't get it when I use kafka-topics.sh --list or --describe, perhaps I'm missing the option for verbosity, although I don't see them in the attribute list for topic configuration at all. Is it not sensible information with Kafka?
You can see the Kafka topic creation time(ctime) and last modified time(mtime) in zookeeper stat.
First login to zookeeper shell
kafka % bin/zookeeper-shell.sh localhost:2181 stat /brokers/topics/test-events
It will return below details:
Connecting to localhost:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
cZxid = 0x1007ac74c
ctime = Thu Nov 01 10:38:39 UTC 2018
mZxid = 0x4000f6e26
mtime = Mon Jan 07 05:22:25 UTC 2019
pZxid = 0x1007ac74d
cversion = 1
dataVersion = 8
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 112
numChildren = 1
You can refer this to understand the attributes : https://zookeeper.apache.org/doc/current/zookeeperProgrammers.html#sc_zkStatStructure
Kafka does not publicly state the date of topic creation/alteration.
The timing data itself is not required by Kafka to work. The current topic config values are kept by the Zookeeper ensemble that the whole Kafka cluster requires to function, so it's kept in-sync by the underlying zookeeper process, and for the part that Kafka is required to syncrhonize, only the offsets within the topic are required to partially-order the messages as they come, the timestamp is not required information.
If you want to keep topic modifications actionable, maybe your best bet is to have a Kafka topic to save such modifications so that you can later read it.

How do you know when was a topic created on a Kafka broker?

How do you know when was a topic created in Kafka?
It seems that a few of the topics were created with a wrong number of partitions. Is there a way to know the date the topic was created? Supposedly, a topic with the name "test" was created with n number of partitions. How can I find the date and time when this "test" topic was created on Kafka?
You can see the Kafka topic creation time(ctime) and last modified time(mtime) in zookeeper stat.
First login to zookeeper shell and add command "stat "
kafka % bin/zookeeper-shell.sh localhost:2181 stat /brokers/topics/test-events
It will return below details:
Connecting to localhost:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
cZxid = 0x1007ac74c
ctime = Thu Nov 01 10:38:39 UTC 2018
mZxid = 0x4000f6e26
mtime = Mon Jan 07 05:22:25 UTC 2019
pZxid = 0x1007ac74d
cversion = 1
dataVersion = 8
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 112
numChildren = 1
You can refer this to understand the attributes : https://zookeeper.apache.org/doc/current/zookeeperProgrammers.html#sc_zkStatStructure
You can tell the topic creation time by checking the zookeeper node creation time for the topic. Given that "zookeeper001:2181/foo" is the Kafka zookeeper connection string, and "test_topic" is the topic name, you can check the stat of znode to get the topic creation time:
/foo/brokers/topics/test_tpopic
I don't think that there is a way to check number of partitions at the topic creation time. You can always increase the topic partition number by using :
kafka-topics.sh --alter ...

storm: KafkaSpout - where is my consumer group?

I need to see the offset values that storm is using to read via its KafkaSpout. Here is the config I pass in:
SpoutConfig kafkaConfig = new SpoutConfig(brokerHosts, "some_values",
"/storm/env_values", "storm_DEBUG");
I've tried searching about with some of the kafka tools but haven't found anything useful yet:
kafka.tools.ExportZkOffsets
kafka.tools.ConsumerOffsetChecker
Are there better tools to use to find my offset?
Take a look to https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
in Managment consoles there is a list of tools for monitoring your topics and offsets. I remember i have use https://github.com/otoolep/stormkafkamon a python monitoring tool.
But what exactly do you want?, if you only tried to see your offset you can see your offset in zookeeper, only connect in your zookeeper where your offsets are stored.
Example:
bin/zookeeper-shell.sh localhost:2000
Connecting to localhost:2000
Welcome to ZooKeeper! JLine support is disabled
ls /
[storm, brokers, zookeeper]
ls /brokers
[kafka-spout]
ls /brokers/kafka-spout
[partition_0]
get /brokers/kafka-spout/partition_0
{"topology":{"id":"a9be1962-6b4e-4ed4-ae68-155a1948a1f6","name":"consolidate_reports"},"offset":4426029,"partition":0,"broker":{"host":"localhost","port":9092},"topic":"bid_history"}
cZxid = 0x50 ctime = Thu May 21 11:00:48 BRT 2015 mZxid = 0x50 mtime =
Thu May 21 11:00:48 BRT 2015 pZxid = 0x50 cversion = 0 dataVersion = 0
aclVersion = 0 ephemeralOwner = 0x0 dataLength = 182 numChildren = 0
KafkaSpout maintains the offset information on Storm's zookeeper by default under {root path}/{id}/{partition-id}. Read this for more information.
You can use the bundled zookeeper shell in Kafka (bin/zookeeper-shell.sh zookeeperHost:port) and browse to the location using the CLI. Issuing a get against the offset path will give you the stored value.
Turns out that I was looking at the wrong zookeeper. According to this doc:
The Kafka spout stores its offsets in the same instance of Zookeeper used by Apache Storm.
So looking at the kafka zookeeper isn't going to be v helpful.