Metadata information from kafka - apache-kafka

Question

I am new to Confluent/Kafka and I want to find metadata information from kafka
I want to know
list of producers
list of topics
schema information for topic
Confluent version is 5.0
What are classes (methods) that can give this information?
Are there any Rest API's for the same
Also is zookeeper connection necessary to get this information.

Answer 1

1) I don't think that Kafka brokers are aware of producers that produce messages in topics and therefore there is no command line tool for listing them. However, an answer to this SO question suggests that you can list producers by viewing the MBeans over JMX.
2) In order to list the topics you need to run:
kafka-topics --zookeeper localhost:2181 --list
Otherwise, if you want to list the topics using a Java client, you can call listTopics() method of KafkaConsumer.
You can also fetch the list of topics through ZooKeeper
ZkClient zkClient = new ZkClient("zkHost:zkPort");
List<String> topics = JavaConversions.asJavaList(ZkUtils.getAllTopics(zkClient));
3) To get the schema information for a topic you can use Schema Registry API
In particular, you can fetch all subjects by calling:
GET /subjects HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json
which should give a response similar to the one below:
HTTP/1.1 200 OK
Content-Type: application/vnd.schemaregistry.v1+json
["subject1", "subject2"]
You can then get all the versions of a particular subject:
GET /subjects/subject-name/versions HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json
And finally, you can get a specific version of the schema registered under this subject
GET /subjects/subject_name/versions/1 HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json
Or just the latest registered schema:
GET /subjects/subject-name/versions/latest HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json
In order to perform such actions in Java, you can either prepare your own GET requests (see how to do it here) or use Confluent's Schema Registry Java Client. You can see the implementation and the available methods in their Github repo.
Regarding your question about Zookeeper, note that ZK is a requirement for Kafka.
Kafka uses ZooKeeper so you need to first start a ZooKeeper server if
you don't already have one. You can use the convenience script
packaged with kafka to get a quick-and-dirty single-node ZooKeeper
instance.