Confluent Kafka how to get specific topic metadata without subscribe - apache-kafka

Question

I use some code for determination difference between read offsets and max values. It needed for inner diagnostics, for example if difference below "base line" value - so, read speed is good.
Provided code sample works good, but I should to know partition count for topic. And, later get max offset value for each partition in a topic.
How to get topic metadata without subscribing to a topic?
Something like: topic "TestTopic" have 4 partitions.
public async Task MaxOffsetValues()
{
// just for tests
// ToDo: use config from settings
while (true)
{
var topicName = "testTopic";
var config = new ConsumerConfig
{
BootstrapServers = "192.168.1.1:9092,192.168.1.2:9092,192.168.1.3:9092",
GroupId = Guid.NewGuid().ToString(),
ClientId = Dns.GetHostName(),
EnableAutoCommit = false,
};
using (var consumer = new ConsumerBuilder<Ignore, string>(config).Build())
{
var offsetBorders = consumer.QueryWatermarkOffsets(new TopicPartition(topicName, 0), TimeSpan.FromSeconds(10));
_log.Debug($"[Diagnostic] Topic: ({topicName}), Partition: ({0}) Minimal offset: ({offsetBorders.Low}) Maximum offset: ({offsetBorders.High})");
}
await Task.Delay(TimeSpan.FromSeconds(60));
}
}

Answer 1

If you are looking for programmatic & non-programmatic way to get metadata then there are 3 ways you can get information about a topic without subscribing it.
Admin client (https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html)
If you have access to confluent-control-center then it does show metadata about a topic.
You can also use Kafka-topics.sh CLI tool

Answer 2

For my goal was enought (code part reduced for clarity):
// ...
var brokerServers = _diagnosticSettings.Value.BrokerHosts;
var brokerDelayInSeconds = TimeSpan.FromSeconds(_diagnosticSettings.Value.BrokerDelayInSeconds);
var adminClientConfig = new AdminClientConfig
{
BootstrapServers = brokerServers,
ClientId = Dns.GetHostName(),
};
adminClient = new AdminClientBuilder(adminClientConfig).Build();
Metadata topicMetadata = null;
topicMetadata = adminClient.GetMetadata(topicExternalName, brokerDelayInSeconds);
partitionCount = topicMetadata.Topics[0].Partitions.Count;
// ...