I am trying to setup Kafka referring to the quick start guide on the apache kafka page and while starting the zookeeper server it gets stuck at this step...Would appreciate if someone can help guide me through on how to start the zookeeper server
[2015-05-26 15:41:39,216] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-05-26 15:41:39,216] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-05-26 15:41:39,216] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-05-26 15:41:39,235] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
The terminal getting stuck at that step actually means that the server is running. So, while you may have thought you were getting an error, it actually means that the server has started.
I'm running through the same setup. Since you've opened the port in that terminal, you can proceed with the rest of the instructions from another terminal. You'll have to open multiple terminals throughout the setup, and eventually will be entering messages in one terminal (the producer), and seeing them appear in another terminal (the consumer).
My logs were stuck at this line, meaning zookeper could not start properly. I could not start kafka too.
0.0.0.0 might mean something is wrong, sometimes. I renewed my DHCP lease on my mac and (system preferences -> Advanced -> TCP/IP -> Renew DHCP lease), restarted zookeeper and it started working.
based on the screenshot posted, below line tells that zookeeper is running on the port 2181
[2015-05-26 15:41:39,235] INFO binding to port 0.0.0.0/0.0.0.0:2181
it seems like you have started the zookeeper server successfully but without running the Kafka server you are trying to run Kafka commands and because of you are getting an error of broker is not available.
in mac run the following two commands in two tabs:
start-zookeeper:
zookeeper-start-server /usr/local/kafka/zookeeper.properties
start kafka server:
kafka-server-start /usr/local/etc/kafka/server.properties
open another terminal and try out the Kafka commands, commands will run without any issues
Related
I installed kafka on windows subsystem linux and started using the command confluent local services start and all services are up. Now, when I'm trying to run my kafka-spring application from windows, it is showing the following error:- connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
My server.properties are:-
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://localhost:9092
Where I'm going wrong???
You need to run the Spring app within WSL2 session such that it's on the same network bridge as the hypervisor
sorry but i am racking my brains to figure this out.
My kafka broker says waiting to connect to zookeper and then shuts down (time out).I checked broker config / port did netstat etc also confirmed zookeper settings. But then if i restart my ubuntu kernel (where kafka broker is running) all is well. Zookeeper is on Linux mint. Please can someone advice where what log files can give me more detail - i did the usual netstat and lsof no luck.
its windows 10 for some reason it is not recognizing my partitioned ip.
Fixed it was a config item on my end.
I am facing below error message when i was trying to connect and see the topic/consumer details of one of my kafka clusters we have.
we have 3 brokers in the cluster which I able to see but the topic and its partitions.
Note : I have kafka 1.0 and kafka tool version is 2.0.1
I had the same issue on my MacBook Pro. The tool was using "tshepo-mbp" as the hostname which it could not resolve. To get it to work I added 127.0.0.1 tshepo-mbp to the /etc/hosts file.
kafka tool is most likely using the hostname to connect to the broker and cannot reach it. You maybe connecting to the zookeeper host by IP address but make sure you can connect/ping the host name of the broker from the machine running the kafka tool.
If you cannot ping the broker either fix the network issues or as a workaround edit the host file on your client to let it know how to reach the broker by its name
This issue occurs if you have not set listeners and advertised.listeners property in server.properties file.
For Ex:
config/server.properties
...
listeners=PLAINTEXT://:9092
...
advertised.listeners=PLAINTEXT://<public-ip/host-name>:9092
...
To fix this issue, we need to change the server.properties file.
$ vim /usr/local/etc/kafka/server.properties
Here update the listeners value from
listeners=PLAINTEXT://:9092
to
listeners=PLAINTEXT://localhost:9092
source:https://medium.com/#Ankitthakur/apache-kafka-installation-on-mac-using-homebrew-a367cdefd273
For better visibility (even already commented the same in early days thread)
In my case, I got to know when I used Kafkatool from my local machine, tool tris to find out Kafka broker port which was blocked from my cluster admins for my local machine, that is the reason I was not able to connect.
Resolution:
Either ask the admin to open the port for intranet if they can, if they can not you can use tunnelling for your testing purpose or time being for your port.
Hope this would help a few.
I'm getting started with Confluent Platform which requires to run Zookeeper (zookeeper-server-start /etc/kafka/zookeeper.properties) and then Kafka (kafka-server-start /etc/kafka/server.properties). I am writing an Upstart script that should run both Kafka and Zookeeper. The issue is that Kafka should block until Zookeeper is ready (because it depends on it) but I can't find a reliable way to know when Zookeeper is ready. Here are some attempts in pseudo-code after running the Zookeeper server start:
Use a hardcoded block
sleep 5
Does not work reliably on slower computers and/or waits longer than needed.
Check when something (hopefully Zookeeper) is running on port 2181
wait until $(echo stat | nc localhost ${port}) is not none
This did not seem to work as it doesn't wait long enough for Zookeeper to accept a Kafka connection.
Check the logs
wait until specific string in zookeeper log is found
This is sketchy and there isn't even a string that cannot also be found on error too (e.g. "binding to port [...]").
Is there a reliable way to know when Zookeeper is ready to accept a Kafka connection? Otherwise, I will have to resort to a combination of 1 and 2.
I found that using a timer is not reliable. the second option (waiting for the port) worked for me:
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties && \
while ! nc -z localhost 2181; do sleep 0.1; done && \
bin/kafka-server-start.sh -daemon config/server.properties
The Kafka error message from your comment is definitely relevant:
FATAL [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/0. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
This indicates that ZooKeeper is up and running, and Kafka was able to connect to it. As I would have expected, technique #2 was sufficient for verifying that ZooKeeper is ready to accept connections.
Instead, the problem appears to be on the Kafka side. It has registered a ZooKeeper ephemeral node to represent the starting Kafka broker. An ephemeral node is deleted automatically when the client's ZooKeeper session expires (e.g. the process terminates so it stops heartbeating to ZooKeeper). However, this is based on timeouts. If the Kafka broker restarts rapidly, then after restart, it sees that a znode representing that broker already exists. To the new process start, this looks like there is already a broker started and registered at that path. Since brokers are expected to have unique IDs, it aborts.
Waiting for a period of time past the ZooKeeper session expiration is an appropriate response to this problem. If necessary, you could potentially tune the session expiration to happen faster as discussed in the ZooKeeper Administrator's Guide. (See discussion of tickTime, minSessionTimeout and maxSessionTimeout.) However, tuning session expiration to something too rapid could cause clients to experience spurious session expirations during normal operations.
I have less knowledge on Kafka, but perhaps there is also something that can be done on the Kafka side. I know that some management tools like Apache Ambari take steps to guarantee assignment of a unique ID to each broker on provisioning.
I am facing issue while starting zookeeper.
zoo.cfg file is
# The number of milliseconds of each tick
tickTime=2000
dataDir=/Users/admin/Documents/delete/zookeeper/zookeeper-3.4.6/zookeeperdata/1
clientPort=2181
initLimit=5
syncLimit=2
server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890
I don't see any error while starting zookeeper :
nohup ./bin/zkServer.sh start zoo.cfg
JMX enabled by default
Using config: /Users/admin/Documents/delete/zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
could see a new process Id also :
cat /Users/admin/Documents/delete/zookeeper/zookeeper-3.4.6/zookeeperdata/1/zookeeper_server.pid
14120
But while checking the status of the process, getting below error:
bin/zkServer.sh status
JMX enabled by default
Using config: /Users/admin/Documents/delete/zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
Error contacting service. It is probably not running
Could you please help.
server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890
This means that you're setting up a ZooKeeper ensemble, and one of the rules for a zk-ensamble is that the servers needs to form a majority before they are allowed to answer requests. This means that a zk-server is not running until it has formed a majority.
To get an informative answer from status you need to have at least 2 out of 3 servers running for a 3 server ensemble. Either remove these lines from your config or start another server. (And make sure the servers have different 'dataDir' and 'myid'.)
(Something a lot of people misunderstand is that the majority that is necessary is not a majority among the started servers but a majority among the servers in the configuration.)