How to add a new Kafka brokers machines dynamically to a cluster - apache-kafka

Question

We have Kafka confluent cluster , cluster include 3 Kafka brokers ,
Version details:
Kafka machines are installed on rhel version 7.2
Kafka confluent version is 0.1x
Zookeeper version: 3.4.10
schema-registry version: 4.0.0
Each Kafka broker machine include the following services
Kafka broker
Zookeeper server
Schema registry
Now we want to add a additionally new 3 Kafka brokers machines to the current Kafka cluster ( the Additional Kafka machines are – kafka04/05/06 with the same Kafka version – 0.1X , )
So cluster should be finally with:
6 Kafka brokers machines - kafka01, kafka02 , kafka03 , kafka04 , kakfa05 , kafka05
3 zookeeper servers - kafka01, kafka02 , kafka03
3 schema registry services - kafka01, kafka02 , kafka03
In order to connect the new 3 Kafka brokers to the existing Kafka cluster we need to change the Configuration on all Kafka Machines ( the old Kafka machines and the new Kafka machines )
We are not sure what are exactly the configuration files in Kafka brokers that should be changed, But From my understanding We should change the Kafka and zookeeper settings as the following
I will happy to get remarks/notes about the following procedure
Edit the server.properties file on the new Kafka brokers - kafka04/05/06 , and change the broker.id parameter as the following
On kafka04 - broker.id=4
On kafka05 - broker.id=5
On kafka06 - broker.id=6
Edit the server.properties on all Kafka machines - kafka01/02/03/04/05/06
And change the following parameters to the number of total nodes in the cluster
offsets.topic.replication.factor=6
transaction.state.log.replication.factor=6
On the new Kafka machines - Kafka 04/05/06 edit the server.properties and update the parameter -- zookeeper.connect with the zookeeper server ip’s that located on kafka01 , kafka02 , kafka03
Example
zookeeper.connect=10.10.10.1:2181,10.10.10.2:2181,10.10.10.3:2181
On Kafka machines – kafka 04/05/06 , edit the file - zookeeper.properties as the
Following
server.1=10.10.10.1:2888:3888
server.2=10.10.10.2:2888:3888
server.3=10.10.10.3:2888:3888
Edit the file - myid file on Kafka 04/05/06 , and change the parameter broker.id as the
Following
on kafka04 set:
broker.id=4
on kafka05 set:
broker.id=5
on kafka06 set:
broker.id=6
After settings as above , restart all Kafka brokers services on kafka01/02/03/04/05/06 and Restart the zookeeper servers on kafka01/02/03
And verify all Kafka services and zookeeper service started successfully
Reference info - https://www2.microstrategy.com/producthelp/current/PlatformAnalytics/en-us/Content/Add_kafka_node_to_kafka_cluster.htm

Answer 1

When adding brokers, you don't need to change the configuration of existing brokers, nor restart them. The same applies to ZooKeeper, if you are not adding new ZooKeeper servers.
On the new brokers, you just need to set a different broker.id value.
I don't recommend increasing the replication factor of the topics (including internal) beyond 3.
Once your new brokers are started, you may want to rebalance your existing data. There are many tools to do that, including the kafka-reassign-partitions.sh tool. The Kafka docs has a section detailing the process: https://kafka.apache.org/documentation/#basic_ops_cluster_expansion