Confluent Replicator End to End Latency Metrics Required - apache-kafka

Question

Problem Statement: I am trying to figure out end to end latency for replicator from On-prem to AWS data replication. We found in consumer group for replicator we have the option to display the End-to-End latency but it’s not showing any data in the control center as shown in below screenshots. I tried few things as explained below but it’s not working.
What we need to know is:
Which Metrics to be configured for this?
How can we configure these metrics?
Am I exploring the right thing as mentioned below?
Is anyone have any experience or faced similar issues
We tested the consumer group shows the overall messages running behind in the destination cluster.
It’s not showing any data in following screen. I tried figuring out why?
In Replicator connector I added following property to enable interceptor:
What confluent says:
“To monitor production and consumption in Control Center, installed the Confluent Monitoring Interceptors with your Apache Kafka® applications and configure your applications to use the interceptors on the Kafka messages produced and consumed, that are then sent to Control Center.”
We installed this in Replicator as mentioned below:
{
"name":"replicator",
"config":{
....
"src.consumer.interceptor.classes": "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor",
....
}
}
}
Then I Checked the broker for Confluent Metrics Reporter which is already configured.
Thanks :)

Answer 1

Following are the configuration which worked for me.
FYI..... Control center/Replicator(connect worker) is running on my source cluster.
consumer.properties
zookeeper.connect=src-node1:2181,src-node2:2181,src-node3:2181
bootstrap.servers=src-node1:9092,src-node2,src-node3:9092
interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
producer.properties
zookeeper.connect=dst-node1:2181,dst-node2:2181,dst-node3:2181
bootstrap.servers=dst-node1:9092,dst-node2:9092,dst-node3:9092
interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
replicator.properties
#Replication configuration
name=replicator-onprem-to-aws
topic.rename.format=${topic}.replica
replication.factor=1
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
confluent.topic.replication.factor=1
If you look into above configuration its the bare basic configuration to run the replicator. I am not using anything fancy here.
then you can run the following command:
[root#src-node1 ~]$ /app/confluent-5.3.1/bin/replicator --cluster.id 1 --consumer.config config/consumer.properties --producer.config config/producer.properties --replication.config config/replicator.properties --whitelist 'test-topic' > replicator.log 2>&1 &