Get all Kafka Source Connectors writing to a specific topic - apache-kafka

Question

I have the name of a Kafka Topic. I would like to know what Connectors are using this topic. Specifically, I need the Source Connector name so I can modify the source query. I only have access to the Confluent Control Center. We have hundreds of Connectors and I cannot search through them manually.
Thank you in advance!

Answer 1

You'd need to write a script.
Given: Connect REST endpoint
Do: (psuedocode)
to_find = 'your_topic_name'
topic_used_by = []
connectors = GET /connectors
for each connector_name in connectors:
connector_config = GET /connectors/{connector_name}
connector_config = connector_config['config']
if 'topics.regex' in connector_config:
// pattern match against your topic
if to_find.matches(connector_config['topics.regex']):
topic_used_by.add(connector_name)
else if 'topics' in connector_config:
// split these on commas
if to_find in connector_config['topics'].split(','):
topic_used_by.add(connector_name)
print(topic_used_by)
From that, your HTTP client should be easy to extend to update any config value, then POST /connectors/{connector_name}/config
Keep in mind that neither 'topics' nor 'topics.regex' are source connector properties (they are sink connector properties). Therefore, you'd need to modify that to use whatever connector properties you do have (ideally, filtering by the connector class name). For example, the table.whitelist property of the JDBC Source, or collection in the MongoDB source, determine the topic name.
And unless you filter by class name, that would return both sources and sinks, so unless there is "Source", or similar, in the name of the connector class, this is the best you can do
Also, you'd need to consider that some topic names may be set by various transform configurations, which could have any JSON key value in the config, so there'd be no straightforward way to search on those without at least transform_config_value.contains('your_topic_name')

Answer 2

You can use feature implemented through KIP-558: Track the set of actively used topics by connectors in Kafka Connect.
E.g. Get the set of active topics from a connector called 'some-source':
curl -s 'http://localhost:8083/connectors/some-source/topics' | jq
{
"some-source": {
"topics": [
"foo",
"bar",
"baz",
]
}
}