bin/zookeeper-shell.sh config/zookeeper.properties
Connecting to config/zookeeper.properties
Exception in thread "main" java.net.UnknownHostException: config: Name or service not known
To run the zookeeper shell you need to provide the machine on which it is running as parameter, not the properties file:
./bin/zookeeper-shell.sh localhost:2181
Related
I am new to kafka and i started doing the Zookeeper cluster setup by refering https://codeforgeek.com/how-to-setup-zookeeper-cluster-for-kafka/
This has three zookeeper instance, I am able to start the fist zookeeper instance but in the second node zookeeper is not started.
I used this command to start zookeeper - bin/zookeeper-server-start.sh config/zookeeper.properties
I also created myid file with number 2 written in it.
Throwing the below error,
[root#kafka2 kafka_2.12-2.2.2]# bin/zookeeper-server-start.sh config/zookeeper.properties
[2020-09-06 07:25:27,761] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-09-06 07:25:27,773] INFO Resolved hostname: 0.0.0.0 to address: /0.0.0.0 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2020-09-06 07:25:27,773] INFO Resolved hostname: 172.16.10.45 to address: /172.16.10.45 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2020-09-06 07:25:27,775] INFO Resolved hostname: 172.16.10.44 to address: /172.16.10.44 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2020-09-06 07:25:27,776] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing config/zookeeper.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:156)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:104)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Caused by: java.lang.IllegalArgumentException: Missing election port for server: 1
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:369)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:152)
... 2 more
Invalid config, exiting abnormally
I found the answers finally. Actually i missed to mention the port(2888) for server.1 in config/zookeeper.properties.
Issue
server.1=<ip of first server>:3888
server.2=0.0.0.0:2888:3888
server.3=<ip of third server>:2888:3888
Working
server.1=<ip of first server>:2888:3888
server.2=0.0.0.0:2888:3888
server.3=<ip of third server>:2888:3888
I ran into an issue while trying to set up kafaka (version 2.12-2.4.0) on my local machine by following:
https://kafka.apache.org/quickstart
I created a very simple spring-boot app that had a producer and consumer by following some online tutorials. When I would start up my app it would spin for 30 seconds and then start throwing errors in the logs trying to create a topic with connection errors.
I thought maybe my spring-boot app was misconfigured so I tried creating a topic from the command line but I got a similar error:
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic testing
Error while executing topic command : org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
[2020-02-11 21:57:06,545] ERROR java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:225)
at kafka.admin.TopicCommand$TopicService.createTopic(TopicCommand.scala:194)
at kafka.admin.TopicCommand$TopicService.createTopic$(TopicCommand.scala:189)
at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:217)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:61)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
Kafka and zookeeper were up and running but nothing could connect to kafka.
I noticed that the kafka logs said it was listening on 0.0.0.0:9092:
INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
I went into the server.properties and changed all the localhost values to:
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://127.0.0.1:9092
This solved the problem. I don't know if this is an issue with my laptop but I wanted to save other people time. I didn't have to change my spring-boot connection configuration, connecting to localhost still worked.
I am new to Kafka and seem to be having several issues with the 'Quickstart' guide for Apache Kafka found here:
https://kafka.apache.org/quickstart#quickstart_kafkaconnect
Ultimately I am trying to learn how to load a kafka queue with many kafka messages and so the Step 7 part of this Quickstart guide seemed relevant.
I installed the binary download (Scala 2.11 - kafka_2.11-1.1.0.tgz ) found here:
https://kafka.apache.org/downloads
I had initially tried to jump straight to step 7 but realised after finding this question (Kafka Connect implementation errors) I had to do the few steps prior to that
Therefore I followed the first step successfully:
tar -xzf kafka_2.11-1.1.0.tgz
cd kafka_2.11-1.1.0
Then I followed step 2:
bin/zookeeper-server-start.sh config/zookeeper.properties
But I get the error
ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:90)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:117)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:87)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
But when I run the next command in that same step:
bin/kafka-server-start.sh config/server.properties
The Kafka server seems to run successfully?
So then I tried to continue to step 3 to create a topic:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
But this produces the error:
Error while executing topic command : Replication factor: 1 larger than available brokers: 0.
[2018-04-09 14:13:26,908] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
(kafka.admin.TopicCommand$)
Then trying step 4:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This seems to work and I can write a message but then I get a connection error (which is probably due to the fact previous steps haven't worked successfully)
kafka_2.11-1.1.0 user$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>This is a message
[2018-04-09 14:17:52,631] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-04-09 14:17:52,687] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Does anyone know why these issues are occurring and how I can fix them? I can't find anymore inforomation in that tutorial about these problems
As the error suggests, you have something running on the default port for ZK. Either close it or change the zookeeper properties file to use another port.
Address localhost:2181 is already in use. Since Zookeeper cannot start, then Kafka brokers won't start too. replication-factor must be less or equal to the number of available brokers, and since no broker is available then the following error will be reported (even if you are using --replication-factor 1).
Error while executing topic command : Replication factor: 1 larger than available brokers: 0.
[2018-04-09 14:13:26,908] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
(kafka.admin.TopicCommand$)
You either need stop the process which is running in 2181 or change the ZK default port to a port which is not currently in use.
To see what is running (PID) in port 2181, run
lsof -i -n -P | grep 2181
If you want to kill that process, then run
kill -9 PID
where PID is the process ID which you can get from lsof command.
Otherwise, you need to change the port in the zookeeper.properties file by modifying the parameter clientPort=2181. And finally, you need to change zookeeper.connect=localhost:2181 parameter in the server.properties file accordingly.
I'm following this tutorial trying to set up kafka,
https://kafka.apache.org/quickstart
And I get this error when doing the part on zookeeper.
It certainly has something to do with the config, but I don't understand to what extend.
bin/zookeeper-server-start.sh config/server.properties
[2018-03-17 03:27:11,623] INFO Reading configuration from: config/server.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-03-17 03:27:11,628] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing config/server.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:154)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.lang.NumberFormatException: For input string: "initial.rebalance.delay.ms"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:242)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:150)
... 2 more
Invalid config, exiting abnormally
If you look closely, you'll notice that the command to start Zookeeper is:
bin/zookeeper-server-start.sh config/zookeeper.properties
Using the zookeeper.properties file.
It looks like you copied the command to start Kafka which is using server.properties.
I fixed this on Windows by doing below.
Make sure to specify full paths.
First, go to the bin\windows folder of your kafka installation:
C:\kafka_2.12-2.0.0\bin\windows
Then, run the following command:
.\zookeeper-server-start.bat C:\kafka_2.12-2.0.0\config\zookeeper.properties
When done, you will see the following output:
[2020-07-06 20:25:13,703] INFO Server environment:user.dir=C:\kafka_2.12-2.0.0\bin\windows (org.apache.zookeeper.server.ZooKeeperServer)
[2020-07-06 20:25:13,716] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-07-06 20:25:13,717] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-07-06 20:25:13,717] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-07-06 20:25:13,787] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2020-07-06 20:25:13,791] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
You are using following to start the zookeeper:
bin/zookeeper-server-start.sh config/server.properties
Zookeeper configuration settings are placed in zookeeper.properties file.
So, to start zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
And for starting KAFKA server:
bin/kafka-server-start.sh config/server.properties
Keep in mind which file comes with which server.
I am facing same issue because I am directly giving step2 command in my terminal,which is wrong. We have to run command of step2 in our Kafka folder.
Follow below steps to run zookeeper successfully.
Step1: Goto your kafka folder or simply do cd $KAFKA_HOME
STEP2: bin/zookeeper-server-start.sh ./config/zookeeper.properties
check for "dataDir=/tmp/zookeeper1", if not present then create a directory and one file inside this directory myid
mkdir -p /tmp/zookeeper1
chmod -R 777 /tmp/zookeeper1
vi /tmp/zookeeper1/myid
(add value as 1, this is server-id from server.1=)
Try to remove config/ , and use the file directly.
e.g.
bin/zookeeper-server-start.sh zookeeper.properties
I setup kafka cluster in vm.
but i have a problem.
My kafka version is kafka 2.11-0.0.0
$ bin/kafka-server-start.sh config/zookeeper.properties
[2017-01-31 09:15:55,216] FATAL (kafka.Kafka$)
org.apache.kafka.common.config.ConfigException: Missing required configuration "zookeeper.connect" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:148)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:49)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:56)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:702)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:691)
at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
please help me .
my zookeeper.properties (down)
zookeeper.properties
dataDir=/home/kafka01/zookeeper-data
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
initLimit=5
syncLimit=2
server.1=kafka01:2888:3888
server.2=kafka02:2888:3888
server.3=kafka03:2888:3888
server.properties
zookeeper.connect=kafka01:2181,kafka02:2181,kafka03:2181
what is the problem??
You are trying to start a kafka server but you are passing the wrong config file.
Usually you want to start a kafka server using the following command:
./kafka-server-start.sh ../config/server.properties
And is in this file where you specify the address of the zookeeper:
zookeeper.connect=kafka01:2181,kafka02:2181,kafka03:2181
The quickstart guide in the kafka official documentation is quite good, I recommend you to have a look at it. You can find it here.
You are trying to start kafka server with zookeeper config properties file and that's why you are getting the above error.
Use the commands below instead:
Zookeeper:
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Kafka:
.\bin\windows\kafka-server-start.bat .\config\server.properties
You stared the wrong script. It should be:
bin/zookeeper-server-start.sh config/zookeeper.properties
There are two ways you can start Kafka server:
Start the zookeeper server separately(on your local or another machine)
Start the zookeeper server which is embedded with Kafka server (kafka_2.12/libs/zookeeper-3.4.13.jar)
In first case:
I am assuming you have started the zookeeper server already and that is running on localhost:2181
So now you are good to start kafka server using following command
bin\windows\kafka-server-start.bat ..\..\config\server.properties
In server.properties file you have to tell Kafka server where is/are the zookeeper(s) running. Using property:
## zookeeper.connect=localhost:2181 [default]
zookeeper.connect=zk1:2181, zk2:2181, localhost:2181
In second case
Start the zookeeper first (embedded with the Kafka server)
bin\windows\zookeeper-server-start.bat ..\..\config\zookeeper.properties
Then, start the Kafka server:
bin\windows\kafka-server-start.bat ..\..\config\server.properties
This is the error because there is an issue with your command.
To start zookeeper run -
zookeeper-server-start.sh config/zookeeper.properties
To start server run -
kafka-server-start.sh config/zookeeper.properties
you need to start the zookeeper with
zookeeper-server-start.sh ~/kafka/config/zookeeper.properties
then you
need to start the kafka-server using
kafka-server-start.sh ~/kafka/config/server.properties
Now you kafka will run smoothly ....