I have 3 node Zookeeper cluster and a 10 node Kafka cluster. After launching Zookeeper daemon on 3 nodes, I then proceed to launch Kafka daemon by using the command
./kafka-server-start.sh "config/server.properties"
And my server.properties is correctly configured containing the proper Zookeeper connection string eg:
zookeeper.connect=192.168.140.23:2181,192.168.140.24:2181,192.168.140.25:2181
My question is do I need to start the Kafka daemon on all of the 10 broker nodes using ./kafka-server-start.sh "config/server.properties" or starting it on just one of them will suffice ?
For reference:
producers.properties
metadata.broker.list=192.168.140.23:9092,192.168.140.24:9092,192.168.140.25:9092,192.168.140.26:9092,192.168.140.27:9092,192.168.140.11:9092,192.168.140.12:9092,192.168.140.13:9092,192.168.140.14:9092
consumer.properties
zookeeper.connect=192.168.140.23:2181,192.168.140.24:2181,192.168.140.25:2181
You have to start all the Kafka servers on those 10 nodes by issuing "./kafka-server-start.sh ...". An automatic tool might be a good way to do this.
Related
Steps
I have used two kafka brokers and I have started zookeeper,kafka server and kafka connect services.
I have one source type kafka connector which can be used for getting data from Database.
If i start the connector[connector 1] by using the rest API, then it will hit any one kafka server [Server 1] using load balancer.After that server 1 will store and running the connector.But server 2 does not know the connector [connector 1] which is running in the server 1.
Expectation
So if the kafka server 1 is down, then the another kafka server 2 should be able to run the connector in the failed kafka server 1.
While starting the connector, kafka server should know how many connectors are in running, so that if any one broker failed to do the job then another server will be able to continue the job.
Reality
Another Kafka server 2 which is not doing the job as per the requirement.
is there any thing to make it by configuration setup with kafka?.
Kindly suggest me some ideas.
Kafka Server 1
Kafka Server 2
It appears that you have started all processes in single pods.
You should run Kafka, Zookeeper, and Connect all as separate services in different pods.
I suggest you refer the Confluent or Strimzi sites to find Kafka Kubernetes Helm Charts / Operators
But to answer the question - You could give one or more broker to connect-distributed.properties bootstrap.server value. Then each broker is connected to as part of the Kafka cluster, and will reconnect in the event that one broker is unavailable
"Kakfa servers" (brokers) do not run Connectors
If you want to run a cluster of connect workers, you also need to setup their rest.advertised.listener address so that they can communicate with each other.
I have a 3 node cluster for Kafka and zookeeper. Both Kafka and zookeeper cluster use same 3 machines. When I start the Kafka service, it fails to start if all 3 zookeeper servers are not up. Why is Kafka waiting for zookeeper cluster to be fully available ? I am not sure if I am overlooking g some setting. I am trying to launch zookeeper followed by Kafka one machine at a time. Error recieved Kafka.zookeeper.ZooKeeperClientTimeoutException
we have with 3 kafka machine and 3 zookeper servers
while kafka machines are not co-hosted with zookeper server ( kafka are on different machines , OS is redhat version 7.x )
in order to get the brokers id , we do the following on the zookeper servers
cd /usr/hdp/current/zookeeper-server/bin
./zkCli.sh
ls /brokers/ids
results should be the three brokers id's as
1011 1012 1013
my question is - in which way zookeper know that broker is up?
or to be more specific
which cli zookeper execute in order to identify that kafka broker is up ?
Zookeeper is basically a distributed key-value store. Upon startup, a Kafka broker connects to Zookeeper (using the zookeeper.connect setting) and create a znode (a key-value pair) with its own broker.id under /brokers/ids. Kafka brokers then stay connected to Zookeeper while they are running.
The znode is created as "Ephemeral" (this is a feature of Zookeeper). It means that Zookeeper will delete it if the broker disconnects.
This way, Zookeeper knows at any time which brokers are alive (it does not necessarily mean the broker is healthy!). This is used by brokers to discover the other brokers in a cluster.
I have a problem with Apache kafka
I have 4 clusters where I want to install kafka instances. On 3 clusters its works, they can product, and consume messages between each other, zookepers work fine. But on 4th cluster I can't run zookeeper connected with others zookeepers. If I set in zoo.cfg only local server (0.0.0.0:2888:3888) zookeeper runs in mode standalone, but if I add others servers I get error
./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Error contacting service. It is probably not running.
How I can fix this error? I will add, that I can ping servers, so they can see each other.
Is one of them more recommended/preferred to use than the other?
Kafka uses Zookeeper so you must start a Zookeeper server before starting the Kafka broker. Zookeeper and Kafka broker are two distinct things and both of them are required in order to run a Kafka Cluster. Kafka is a distributed system and is built to use Zookeeper which is responsible for controller election, topic configuration, clustering etc.
In order to run Zookeeper you need to set the parameters in the configuration file config/zookeeper.properties and then start the ZK server using
bin/zookeeper-server-start.sh config/zookeeper.properties
Then you need to run at least one Kafka broker which can be configured in config/server.properties file and then start it using
bin/kafka-server-start.sh config/server.properties
Zookeeper-server-start.sh is to start your zookeeper server which by default runs on port 2181.
To use kafka brokers, topics and partition you need to have your zookeeper server running, zookeeper works as manager for kafka brokers.
Kafka-server-start.sh is to start your kafka broker.
Zookeeper-server-start.shtakes a
zookeeper.propertiesfile for the configuration
Kafka-server-start takes a Kafka
server.properties file for configuration