I am trying to replicate Kafka cluster with MirrorMaker 2.0.
I am using following mm2.properties:
# mm2.properties
clusters=source, dest
source.bootstrap.servers=kafka-1:9092,kafka-2:19092,kafka-3:29092
dest.bootstrap.servers=kafka-4:39092,kafka-5:49092,kafka-6:59092
source->dest.enabled=true
I can see that when MirrorMaker starts, it creates its own topics. For example on source cluster it creates: mm2-configs.dest.internal with following content:
{
"properties":{
"connector.class":"org.apache.kafka.connect.mirror.MirrorSourceConnector",
"source.cluster.producer.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
"source.cluster.alias":"dest",
"source.cluster.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
"target.cluster.producer.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
"enabled":"false",
"target.cluster.admin.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
"target.cluster.alias":"source",
"target.cluster.consumer.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
"name":"MirrorSourceConnector",
"target.cluster.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
"source.cluster.admin.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
"source.cluster.consumer.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092"
}
}
and at destination cluster mm2-configs.source.internal topic with following content:
{
"properties":{
"connector.class":"org.apache.kafka.connect.mirror.MirrorSourceConnector",
"source.cluster.producer.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
"source.cluster.alias":"source",
"source.cluster.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
"target.cluster.producer.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
"enabled":"true",
"target.cluster.admin.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
"target.cluster.alias":"dest",
"target.cluster.consumer.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
"name":"MirrorSourceConnector",
"target.cluster.bootstrap.servers":"kafka-4:39092,kafka-5:49092,kafka-6:59092",
"source.cluster.admin.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092",
"source.cluster.consumer.bootstrap.servers":"kafka-1:9092,kafka-2:19092,kafka-3:29092"
}
}
But when I am trying to send messages to source cluster (kafka-1,kafka-2,kafka-3) - nothing copied to the second cluster.
What do I miss?
Appreciate any help
Thx,
mm2 works well in standalone ( single node ) mode, but when it comes to the cluster there is a bug.
The details have been reported here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters
Your config files look ok. However, those bootstrap server ports look off. 19092, 29092, etc. Those are typically 9092 only. Can you try changing those back to 9092?
I think, I found (with help of my friend) what was the problem. In case when topic does NOT exist before starting MirrorMaker, replication won't happen. Topic should exist in source Kafka cluster.
I used Java producer with topic auto create enabled. Seems, automatic topic creation, doesn't trigger MirrorMaker to start replication.
I don't really know if it's as designed behavior when defining source->dest.enabled=true or maybe related to solved few days ago bug: https://github.com/apache/kafka/pull/7745.
Related
I have a recent Kafka cluster which uses Kraft. I am facing some problems with it and it is possibly due to use of Kraft. I wish to switch to Zookeeper without losing data. Downtime is okay. How do I go about it?
I'm afraid there isn't a documented process to downgrade a cluster from KRaft to ZooKeeper.
If you've found an issue with KRaft, you should report it to the Kafka project via a Jira ticket so it can get fixed.
Assuming your KRaft cluster is somewhat functional, a way to preserve your data is to create a new cluster (running ZooKeeper) and use a tool like MirrorMaker to migrate your data.
Problem Statement: I am trying to figure out end to end latency for replicator from On-prem to AWS data replication. We found in consumer group for replicator we have the option to display the End-to-End latency but it’s not showing any data in the control center as shown in below screenshots. I tried few things as explained below but it’s not working.
What we need to know is:
Which Metrics to be configured for this?
How can we configure these metrics?
Am I exploring the right thing as mentioned below?
Is anyone have any experience or faced similar issues
We tested the consumer group shows the overall messages running behind in the destination cluster.
It’s not showing any data in following screen. I tried figuring out why?
In Replicator connector I added following property to enable interceptor:
What confluent says:
“To monitor production and consumption in Control Center, installed the Confluent Monitoring Interceptors with your Apache Kafka® applications and configure your applications to use the interceptors on the Kafka messages produced and consumed, that are then sent to Control Center.”
We installed this in Replicator as mentioned below:
{
"name":"replicator",
"config":{
....
"src.consumer.interceptor.classes": "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor",
....
}
}
}
Then I Checked the broker for Confluent Metrics Reporter which is already configured.
Thanks :)
Following are the configuration which worked for me.
FYI..... Control center/Replicator(connect worker) is running on my source cluster.
consumer.properties
zookeeper.connect=src-node1:2181,src-node2:2181,src-node3:2181
bootstrap.servers=src-node1:9092,src-node2,src-node3:9092
interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
producer.properties
zookeeper.connect=dst-node1:2181,dst-node2:2181,dst-node3:2181
bootstrap.servers=dst-node1:9092,dst-node2:9092,dst-node3:9092
interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
replicator.properties
#Replication configuration
name=replicator-onprem-to-aws
topic.rename.format=${topic}.replica
replication.factor=1
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
confluent.topic.replication.factor=1
If you look into above configuration its the bare basic configuration to run the replicator. I am not using anything fancy here.
then you can run the following command:
[root#src-node1 ~]$ /app/confluent-5.3.1/bin/replicator --cluster.id 1 --consumer.config config/consumer.properties --producer.config config/producer.properties --replication.config config/replicator.properties --whitelist 'test-topic' > replicator.log 2>&1 &
I was hoping to get some clarification on the kafka connect configuration properties here https://docs.confluent.io/current/connect/userguide.html
We were having issues connecting to our confluent connect cluster to our kafka connect instance. We had all our settings configured correctly from what i could tell and didn’t have any luck.
After extensive googling some discovered that prefixing the configuration properties with “consumer.” seems to fix the issue. There is a mention of that prefix here https://docs.confluent.io/current/connect/userguide.html#overriding-producer-and-consumer-settings
I am having a hard time understanding wrapping my head around the prefix and how the properties are picked up by connect and used. It was my assumption that the java api client used by kafka connect will pick up the connection properties from the properties file. It might have some hard coded configuration properties that can be overridden by specifying the values in the properties file. But, this is not correct? The doc linked above mentions
All new producer configs and new consumer configs can be overridden by prefixing them with producer. or consumer.
What are the new configs? The link on that page just takes me to the list of all the configs. The doc mentions
Occasionally, you may have an application that needs to adjust the default settings. One example is a standalone process that runs a log file connector
that as the use case for using the prefix override, but this is connect cluster, how does that use case apply? Appreciate your time if you have read thus far
The new prefix is probably misleading. Apache Kafka is currently at version 2.3, and back in 0.8 and 0.9 a "new" producer and consumer API was added. These are now just the standard producer and consumer, but the new prefix has hung around.
In terms of overriding configuration, it is as you say; you can prefix any of the standard consumer/producer configs in the Kafka Connect worker with consumer. (for a sink) or producer. (for a source).
Note that as of Apache Kafka 2.3 you can also override these per connector, as detailed in this post : https://www.confluent.io/blog/kafka-connect-improvements-in-apache-kafka-2-3
The Post is too old, but I'll answer it for people who will face he same difficulty:
New properties, they would like to say : any custom consumer or producer configs.
And there is two levels :
Worker side : the worker has a consumer to read configs, status and offsets of each connector and has a producer (to write status and offsets) [not confuse with __consumer_offsets topics : offset topic is only for source connector], so to override those configs:
consumer.* (example: consumer.max.poll.records=10)
producer.* (example: producer.batch.size=10000)
Connector Side : this one will inherit the worker config by default, and to override consumer/producer configs, we should use :
consumer.override.* (example: consumer.override.max.poll.records=100)
producer.override* (example: producer.override.batch.size=20000)
We are using a kafka broker setup with a kafka streams application that runs using Spring cloud stream kafka. Although it seems to run fine, we do get the following error statements in our log:
2019-02-21 22:37:20,253 INFO kafka-coordinator-heartbeat-thread | anomaly-timeline org.apache.kafka.clients.FetchSessionHandler [Consumer clientId=anomaly-timeline-56dc4481-3086-4359-a8e8-d2dae12272a2-StreamThread-1-consumer, groupId=anomaly-timeline] Node 2 was unable to process the fetch request with (sessionId=1290440723, epoch=2089): INVALID_FETCH_SESSION_EPOCH.
I searched the internet but there is not much information on this error. I guessed that it could have something to do with a difference in time settings between the broker and the consumer, but both machines have the same timeserver settings.
Any idea how this can be resolved?
There is a concept of fetch session, introduced within KIP-227 since 1.1.0 release: https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability
Kafka brokers, which are replica followers, fetch messages from the leader. In order to avoid sending full metadata each time for all partitions, only those partitions which changed are sent within the same fetch session.
When we look into Kafka's code, we can see an example, when this is returned:
if (session.epoch != expectedEpoch) {
info(s"Incremental fetch session ${session.id} expected epoch $expectedEpoch, but " +
s"got ${session.epoch}. Possible duplicate request.")
new FetchResponse(Errors.INVALID_FETCH_SESSION_EPOCH, new FetchSession.RESP_MAP, 0, session.id)
} else {
src: https://github.com/axbaretto/kafka/blob/ab2212c45daa841c2f16e9b1697187eb0e3aec8c/core/src/main/scala/kafka/server/FetchSession.scala#L493
In general, if you don't have thousands of partitions and, at the same time, this doesn't happen very often, then it shouldn't worry you.
It seems as this might be caused by Kafka-8052 issue, which was fixed for Kafka 2.3.0
Indeed, you can have this message when rolling or retention-based deletion occurs, as zen pointed out in comments. It's not a problem if it doesn't happen all the time. If it does, check your log.roll and log.retention configurations.
Updating the client version to 2.3 (same version from broker) fix it for me.
In our case, The root cause was kafka Broker - client incompatibility. If your cluster is behind the client version you might see all kinds of odd problems such as this.
Our kafka broker is on 1.x.x and our kafka-consumer was on 2.x.x. As soon as we downgraded our spring-cloud-dependencies to Finchley.RELEASE our problem was solved.
dependencyManagement {
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Finchley.RELEASE"
}
}
I am trying to create a Samza job that as closely resembles the Wikipedia example job as I can make it. However in the "WikipediaFeed" object I am trying to get data from a different Kafka broker than the Kafka broker that is running when you start the Hello-Samza grid.
Do I have to create a thread safe Kafka consumer inside the "WikipediaFeed" object to consume data from a different Kafka cluster or is there another way I'm not seeing?
Edit 1:
Here is a link to their Wikipedia example.
https://github.com/apache/samza-hello-samza/tree/master/src/main
Thanks
In your example you need change this config (https://github.com/apache/samza-hello-samza/blob/master/src/main/config/wikipedia-feed.properties) :
systems.kafka.consumer.zookeeper.connect=KAFKA_CLUSTER_FRONTING:2181
systems.kafka.producer.bootstrap.servers=KAFKA_CLUSTER_FRONTING:9092
task.inputs=kafka.topic1,kafka.topic2,kafka.topic3
Change the config with your Fronting Kafka cluster
and add your topic in task.inputs separated with ","
Edit:
Just to be clear, you can deploy your Samza into a Cluster 1 and consume a Kafka topic from another cluster. You need change the config in your Samza properties.
To see more information : Samza config
Then if you need send your message after process to another Kafka cluster you will need create another system in your config.
See more information : https://samza.apache.org/learn/documentation/0.13/api/overview.html