i tried to start zookeeper with:
/opt/Kafka/kafka_2.10-0.10.0.1/bin/zookeeper-server-start.sh
/opt/Kafka/kafka_2.10-0.10.0.1/config/zookeeper.properties
but I got
ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.net.BindException: Address already in use
I restarted the computer and I got the same problem
I tried netstat -nlp|grep 2181 the result:
tcp6 0 0 :::2181 :::* LISTEN
kill -9 2181 --> bash: kill: (2181) - No such process
./zookeeper-server-stop.sh # no result so it works but
./zookeeper-server-start.sh /opt/Kafka/kafka_2.10-0.10.0.1/config/zookeeper.properties # the same exception
what can I do?
Note: when starting kafka server there is no exception
kill -9 2181 --> bash: kill: (2181) - No such process
You're trying to kill a port number, not a process ID.
Address already in use
Okay, so you clearly already have something on that port.
Options:
Edit the zookeeper.properties to bind to a different port. Also edit kafka properties to use the new Zookeeper connection.
You already have a running Zookeeper, which would explain why Kafka would work when starting.
P.S. I would recommend using a later version of Kafka anyway. Lots of good upgrades.
Related
I tried following the basic instructions to run a node over here https://geode.apache.org/docs/guide/114/getting_started/15_minute_quickstart_gfsh.html, but I am getting the following issue:
gfsh>start locator --name=locator1
Starting a Geode Locator in /home/thiago/geode/locator1...
........
Locator in /home/thiago/geode/locator1 on 192.168.50.225[10334] as locator1 is currently online.
Process ID: 28137
Uptime: 5 seconds
Geode Version: 1.14.0
Java Version: 16.0.2
Log File: /home/thiago/geode/locator1/locator1.log
JVM Arguments: -Dgemfire.enable-cluster-configuration=true -Dgemfire.load-cluster-configuration-from-dir=false -Dgemfire.launcher.registerSignalHandlers=true -Djava.awt.headless=true -Dsun.rmi.dgc.server.gcInterval=9223372036854775806
Class-Path: /home/thiago/apache-geode-1.14.0/lib/geode-core-1.14.0.jar:/home/thiago/apache-geode-1.14.0/lib/geode-dependencies.jar
Unable to auto-connect (Security Manager may be enabled). Please use "connect --locator=192.168.50.225[10334]" to connect Gfsh to the locator.
Failed to connect; unknown cause: Exception caused JMX Manager startup to fail because: 'java.rmi.server.ExportException: Port already in use: 1099; nested exception is:
java.net.BindException: Failed to create server socket on 127.0.1.1[1099]'
Before starting I made sure nothing was running on port 1099, but after running the command I checked that what uses port 1099 is the spawned java process itself:
❯ lsof -i:1099
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 28137 thiago 206u IPv6 262648 0t0 TCP *:rmiregistry (LISTEN)
Sounds like the issue is because I was using java 16 which does not seems to be supported. Using java 1.8 fixes it.
Same thing still happens with lastest geode and JDK 17. JDK 11 is the latest stable release that will still work
First try downgrading or switching to the stable Java version 1.8 or 11
locator need 10334 port to start and the need 1099 for cluster connection.
To check the port availability:
lsof -I:[port-number]
Then take the PID form the above result to free up the port
To free up the port:
sudo kill PID
if it doesn't work then try
sudo kill -2 PID
or
sudo kill -1 PID
or
sudo kill -9 PID
Kafka was working just fine for me until about 30min ago. I'm using windows and the commands I use are:
c:\kafka>.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
to start up zookeeper and then
c:\kafka>.\bin\windows\kafka-server-start.bat .\config\server.properties
to start up the kafka server.
Right now zookeeper isn't working for some reason. That is when I run the command to start zookeeper, nothing happens, a line is skipped in my terminal and I just have:
C:\Kafka>.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
C:\Kafka>
Has this ever happened to anyone? Might someone know how to resolve this?
(1) kafka version which I am using in production kafka_2.11-2.1.1
(2) OS in which I deployed Kafka
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
Required steps to start your kafka Service :-
Starting zookeeper-server
Start your Zookeeper
[root#ie1 kafka_2.11-2.1.1]# ./bin/zookeeper-server-start.sh
config/zookeeper.properties
check port is running or not ( Default port used by zookeeper :2181 )
linux commnad to check port is LISTEN or NOT
[myhostname#ie1 kafka_2.11-2.1.1]$ netstat -telnup | grep '2181'
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::2181 :::* LISTEN 0 1205756192 -
if you are getting this it mean all are ok, now start follow further steps.
Starting kafka-server
Start your kafka-server
[root#ie1 kafka_2.11-2.1.1]# ./bin/kafka-server-start.sh
config/server.properties
check port is running or not ( Default port used by zookeeper :9094 )
linux commnad to check port is LISTEN or NOT
[myhostname#ie1 kafka_2.11-2.1.1]$ netstat -telnup | grep '9094'
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::9094 :::* LISTEN 0 1205756192 -
if you are getting this it mean all are ok, now you are ready to go.
Note:- Both services should be run in background otherwise as you will kill you CMD both services would be down.
for Information Please look into log file `
kafka_2.11-2.1.1/logs/
`
Happy Reading (' ')
Thanks
I am trying to start up Zookeeper via the CLI with the command:
bin/zookeeper-server-start.sh ../config/zookeeper.properties
And it hums along for a second with what all seems to be correct until it says this:
INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
and then the below loops indefinitely until I exit:
[2018-08-10 15:07:48,223] INFO Accepted socket connection from /172.31.39.32:46374 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-08-10 15:07:48,228] WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running (org.apache.zookeeper.server.NIOServerCnxn)
[2018-08-10 15:07:48,228] INFO Closed socket connection for client /172.31.39.32:46374 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
This is a single server and I believe a single node test server, so there isn't a quorum or other pieces running. My zookeeper config is basic, it only contains this:
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
The weird thing is, my zookeeper had been running fine, and I had made NO changes to the config. Pulled it down to try to fix something else to do a quick restart on the zookeeper, and it won't budge. I've checked, and nothing else is running on port 2181.
I see this question has been asked several times with no answers, any ideas?
This might be happening because of some corruption in zookeeper data. You should not set dataDir to /tmp/*. If your computer purges some data of /tmp, it will be difficult for zookeeper to restore the state upon restart. If you check the zookeeper logs, you should see some kind of exception there.
Since you mentioned this zookeeper instance is for test purpose only. You should set
dataDir to anything but /tmp and try restart.
I'm using confluent platform, the zookeeper is active with status lookup. but when I try to start kafka with confluent it shows zookeeper is down.
$ sudo service zookeeper status
Redirecting to /bin/systemctl status zookeeper.service
● zookeeper.service - Zookeeper
Loaded: loaded (/etc/systemd/system/zookeeper.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2017-08-08 17:25:34 PDT; 16h ago
Docs: http://kafka.apache.org/documentation.html
Process: 3774 ExecStop=/var/www/confluent/bin/zookeeper-server-stop (code=exited, status=1/FAILURE)
Main PID: 3785 (java)
CGroup: /system.slice/zookeeper.service
└─3785 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/var/log...
zookeeper[3785]: [2017-08-08 17:26:09,005] INFO Processed session termination for sessionid: 0x15dc460fd0c0000 (org.apache.zooke...Processor)
zookeeper[3785]: [2017-08-08 17:26:39,000] INFO Expiring session 0x15dc4364baf0004, timeout of 60000ms exceeded (org.apache.zook...perServer)
zookeeper[3785]: [2017-08-08 17:26:39,000] INFO Expiring session 0x15dc4364baf0002, timeout of 60000ms exceeded (org.apache.zook...perServer)
zookeeper[3785]: [2017-08-08 17:26:39,000] INFO Expiring session 0x15dc4364baf0003, timeout of 60000ms exceeded (org.apache.zook...perServer)
zookeeper[3785]: [2017-08-08 17:26:39,001] INFO Processed session termination for sessionid: 0x15dc4364baf0004 (org.apache.zooke...Processor)
zookeeper[3785]: [2017-08-08 17:26:39,002] INFO Processed session termination for sessionid: 0x15dc4364baf0002 (org.apache.zooke...Processor)
zookeeper[3785]: [2017-08-08 17:26:39,002] INFO Processed session termination for sessionid: 0x15dc4364baf0003 (org.apache.zooke...Processor)
zookeeper[3785]: [2017-08-09 09:56:26,711] INFO Accepted socket connection from /127.0.0.1:46446 (org.apache.zookeeper.server.NI...xnFactory)
zookeeper[3785]: [2017-08-09 09:59:14,796] WARN Exception causing close of session 0x0 due to java.io.IOException: Len error -72...erverCnxn)
zookeeper[3785]: [2017-08-09 09:59:14,796] INFO Closed socket connection for client /127.0.0.1:46446 (no session established for...erverCnxn)
Hint: Some lines were ellipsized, use -l to show in full.
$ confluent start kafka
Starting zookeeper
|Zookeeper failed to start
zookeeper is [DOWN]
Cannot start Kafka, Zookeeper is not running. Check your deployment
This is because zookeeper is already running, you can check the process with
ps aux|grep zookeeper
and kill the process manually and it is gonna work.
The most common cause for the message you are seeing when running:
confluent start kafka
and informs you that zookeeper is down, is that there's another zookeeper instance that is currently running, and the new zookeeper instance can not bind to its required port (by default this port is 2181).
A few options at your disposal to figure out what's the other zookeeper instance that is currently running when you try to issue confluent start kafka are:
run jps to see the running java processes. Zookeeper is the process named QuorumPeerMain next to its process ID. (equivalent to running ps xuaww | grep -i zookeeper or equivalent).
run lsof -i :2181 to figure out what the process that is running and has reserved the default zookeeper port (in this example 2181, but might be different in your system).
Try running confluent start kafka again after stopping the above process.
I received the same message. In my case I didn't set $JAVA_HOME variable properly.
You are mixing two installations.
confluent start kafka would depend on you running confluent start zookeeper.
Rather, it seems you already have systemctl running Zookeeper, so you should ideally just configure your server.properties and use the regular kafka-server-start script. And/or create a systemctl file for Kafka.
run $ confluent log zookeeper you will be able to see the log for any errors
there is high chance zookeeper is already running and using the port 2181,
use $ sudo lsof -i :1-2181 to see which process is using that port and try to kill and try again or
run $ sudo netstat -plten | grep java to see processes and ports they are on.
run kill -9 <pid> to kill the process.
I have been trying to test a kafka installation and using the guide created a producer and consumer. When trying to retrieve a message I get the following error:
WARN Session 0x0 for server null, unexpected error, closing socket connection and
attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1146)
[2014-03-04 18:01:20,628] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-03-04 18:01:21,315] INFO Opening socket connection to server kafka-test/192.xxxxxx.110:2182 (org.apache.zookeeper.ClientCnxn)
[2014-03-04 18:01:21,418] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
Exception in thread "main" org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:151)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:112)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:123)
at kafka.consumer.Consumer$.create(ConsumerConnector.scala:89)
at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:178)
at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
[2014-03-04 18:01:21,419] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
Kafka
Looks like you're not connecting to Zookeeper correctly. I'm not sure of your setup (multi-machine, VMs, containers) so it's hard to say what's wrong. From the debug output I see the following line hinting at your expected Zookeeper IP:
[2014-03-04 18:01:21,315] INFO Opening socket connection to server kafka-test/192.xxxxxx.110:2182 (org.apache.zookeeper.ClientCnxn)
Kafka looks for Zookeeper at the address specified by the zookeeper.connect configuration property in the $KAFKA_HOME/config/server.properties file. Be sure to edit that before starting Kafka. Also, try giving the actual public IP of your Zookeeper instance, not just 127.0.0.1 as that solves a lot of confusion if you're running in containers. In your case it looks like it would be:
zookeeper.connect=192.xxxxxx.110:2182
Also relevant to the Kafka config if you're running on AWS or operating in a container, don't forget to update the following two configuration properties to make sure clients who connect to Kafka see the correct public IP
advertised.host.name
advertised.port
and Kafka sees the correct internal IP
host.name
port
Zookeeper
Zookeeper has some gotchas when setting it up as well. On your Zookeeper instance, don't forget to edit the server configuration property in the zoo.cfg (usually in /etc/zookeeper/conf) file to point to the correct IP for your Zookeeper instance. In your case probably the following:
server.1=192.xxxxxx.110:2888:3888
Those last two ports (2888 3888) are only needed if you're running a Zookeeper cluster (for followers to connect to the leader and Zookeeper leader election, respectively, so be sure to unblock them on firewallish things if you have multiple Zookeeper servers).
Check your zookeeper connection with telnet command:
telnet 192.xxxxxx.110 2181
You probably get an error, in which case check that the process is running:
ps -ef | grep "zookeeper.properties"
If it's not running, start it by going into kafka home directory:
bin/zookeeper-server-start.sh config/zookeeper.properties &
Something wrong with your Zookeper configuration. Make sure your zookeeper is up and running. The default port it runs on is 2181
Bit more info and some code could be useful I believe.
I hit the same issue and the problem was the max client connections property in zookeeper config.
if you see something like "maxClientCnxns = 20" in the config file in /etc/zookeeper/conf, comment it out and restart zookeeper.
You may also check if the all the connections available have already been exhausted. If you are using an API to connect to ZK, make sure you free up the connection after you're done.
I also meet the problem. When I shutdown the firewall of the zk node, it will work.