I have a 3 node cluster for Kafka and zookeeper. Both Kafka and zookeeper cluster use same 3 machines. When I start the Kafka service, it fails to start if all 3 zookeeper servers are not up. Why is Kafka waiting for zookeeper cluster to be fully available ? I am not sure if I am overlooking g some setting. I am trying to launch zookeeper followed by Kafka one machine at a time. Error recieved Kafka.zookeeper.ZooKeeperClientTimeoutException
Related
Steps
I have used two kafka brokers and I have started zookeeper,kafka server and kafka connect services.
I have one source type kafka connector which can be used for getting data from Database.
If i start the connector[connector 1] by using the rest API, then it will hit any one kafka server [Server 1] using load balancer.After that server 1 will store and running the connector.But server 2 does not know the connector [connector 1] which is running in the server 1.
Expectation
So if the kafka server 1 is down, then the another kafka server 2 should be able to run the connector in the failed kafka server 1.
While starting the connector, kafka server should know how many connectors are in running, so that if any one broker failed to do the job then another server will be able to continue the job.
Reality
Another Kafka server 2 which is not doing the job as per the requirement.
is there any thing to make it by configuration setup with kafka?.
Kindly suggest me some ideas.
Kafka Server 1
Kafka Server 2
It appears that you have started all processes in single pods.
You should run Kafka, Zookeeper, and Connect all as separate services in different pods.
I suggest you refer the Confluent or Strimzi sites to find Kafka Kubernetes Helm Charts / Operators
But to answer the question - You could give one or more broker to connect-distributed.properties bootstrap.server value. Then each broker is connected to as part of the Kafka cluster, and will reconnect in the event that one broker is unavailable
"Kakfa servers" (brokers) do not run Connectors
If you want to run a cluster of connect workers, you also need to setup their rest.advertised.listener address so that they can communicate with each other.
We have Kafka cluster with 3 broker nodes. When all are up and running, consumer able to read data from Kafka. However if I stop all Kafka server and brings up only 2 Kafka server except the one which stopped last then Consumer unable to connect to Kafka cluster.
What could be the reason behind this? Thanks in advance.
I would guess that the problem could be the offsets.topic.replication.factor in the broker that by default is 3 while you are now running a cluster with 2 brokers only.
This is the topic where the consumers store the offsets when consuming and it was created with replication factor of 3 on the first run.
When, on the second run, you start only 2 broker, it could be the problem now.
we have with 3 kafka machine and 3 zookeper servers
while kafka machines are not co-hosted with zookeper server ( kafka are on different machines , OS is redhat version 7.x )
in order to get the brokers id , we do the following on the zookeper servers
cd /usr/hdp/current/zookeeper-server/bin
./zkCli.sh
ls /brokers/ids
results should be the three brokers id's as
1011 1012 1013
my question is - in which way zookeper know that broker is up?
or to be more specific
which cli zookeper execute in order to identify that kafka broker is up ?
Zookeeper is basically a distributed key-value store. Upon startup, a Kafka broker connects to Zookeeper (using the zookeeper.connect setting) and create a znode (a key-value pair) with its own broker.id under /brokers/ids. Kafka brokers then stay connected to Zookeeper while they are running.
The znode is created as "Ephemeral" (this is a feature of Zookeeper). It means that Zookeeper will delete it if the broker disconnects.
This way, Zookeeper knows at any time which brokers are alive (it does not necessarily mean the broker is healthy!). This is used by brokers to discover the other brokers in a cluster.
I have a problem with Apache kafka
I have 4 clusters where I want to install kafka instances. On 3 clusters its works, they can product, and consume messages between each other, zookepers work fine. But on 4th cluster I can't run zookeeper connected with others zookeepers. If I set in zoo.cfg only local server (0.0.0.0:2888:3888) zookeeper runs in mode standalone, but if I add others servers I get error
./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Error contacting service. It is probably not running.
How I can fix this error? I will add, that I can ping servers, so they can see each other.
I have 3 node Zookeeper cluster and a 10 node Kafka cluster. After launching Zookeeper daemon on 3 nodes, I then proceed to launch Kafka daemon by using the command
./kafka-server-start.sh "config/server.properties"
And my server.properties is correctly configured containing the proper Zookeeper connection string eg:
zookeeper.connect=192.168.140.23:2181,192.168.140.24:2181,192.168.140.25:2181
My question is do I need to start the Kafka daemon on all of the 10 broker nodes using ./kafka-server-start.sh "config/server.properties" or starting it on just one of them will suffice ?
For reference:
producers.properties
metadata.broker.list=192.168.140.23:9092,192.168.140.24:9092,192.168.140.25:9092,192.168.140.26:9092,192.168.140.27:9092,192.168.140.11:9092,192.168.140.12:9092,192.168.140.13:9092,192.168.140.14:9092
consumer.properties
zookeeper.connect=192.168.140.23:2181,192.168.140.24:2181,192.168.140.25:2181
You have to start all the Kafka servers on those 10 nodes by issuing "./kafka-server-start.sh ...". An automatic tool might be a good way to do this.