How to update MirrorMaker 2 topics whitelist? - apache-kafka

Question

I run MirrorMaker2 with the high-level driver, as documented here, with ./bin/connect-mirror-maker.sh mm2.properties running in 3 pods in a k8s deployment.
The mm2.properties file looks like this:
clusters = source, dest
source.bootstrap.servers = ***:9092
dest.bootstrap.servers = ***:9092
source->dest.enabled = true
dest->source.enabled = false
source->dest.topics = event\.PROD\.some_id.*
replication.factor=3
checkpoints.topic.replication.factor=3
heartbeats.topic.replication.factor=3
offset-syncs.topic.replication.factor=3
offset.storage.replication.factor=3
status.storage.replication.factor=3
config.storage.replication.factor=3
sync.topic.acls.enabled = false
This works fine, with all topics matching the event\.PROD\.some_id.* regex being replicated.
Now, when I need to add other topic the whitelisting, I expected to be able to simply scale everything down, update the regex, and scale everything up again.
When I update the whitelist regex to source->dest.topics = event\.PROD\.(some_id|another_id).* , the topics matching "another_id" are created in the dest cluster, but no data is replicated, and mirrormaker seems to be lost commiting offsets:
[2020-05-28 20:33:19,496] INFO WorkerSourceTask{id=MirrorHeartbeatConnector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424)
[2020-05-28 20:33:19,496] INFO WorkerSourceTask{id=MirrorHeartbeatConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441)
[2020-05-28 20:33:19,499] INFO WorkerSourceTask{id=MirrorHeartbeatConnector-0} Finished commitOffsets successfully in 3 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:523)
Is this a limitation of the high level driver, or am I doing something wrong? From my understanding, being able to dynamically add topics to the whitelist was one of the motivations for MM2.

Answer 1

I am playing with mmv2 as well. Can you try setting these configurations? I had to enable the sync.topic.configs.enabled parameter so my mmv2 would detect the new topics and their data.
refresh.topics.enabled = true
sync.topic.configs.enabled = true
refresh.topics.interval.seconds = 10
Pd.- I am adding my reply as an answer because I wanted to paste come configs.