I am running Kafka via a Confluent platform on 3 nodes but when i running confluent start get this error :
[2018-04-09 10:54:25,995] INFO Reading configuration from: /tmp/confluent.SVNfiLFU/zookeeper/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-04-09 10:54:26,011] INFO Resolved hostname: 0.0.0.0 to address: /0.0.0.0 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2018-04-09 10:54:26,011] INFO Resolved hostname: 192.168.0.36 to address: /192.168.0.36 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2018-04-09 10:54:26,011] INFO Resolved hostname: 192.168.0.22 to address: /192.168.0.22 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2018-04-09 10:54:26,011] INFO Defaulting to majority quorums (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-04-09 10:54:26,012] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing /tmp/confluent.SVNfiLFU/zookeeper/zookeeper.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:154)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.lang.IllegalArgumentException: /tmp/confluent.SVNfiLFU/zookeeper/data/myid file is missing
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:406)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:150)
... 2 more
this is zookeeper.properties :
dataDir=/var/lib/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
tickTime=2000
server.1=192.168.0.21:2888:3888
server.2=192.168.0.22:2888:3888
server.3=192.168.0.36:2888:3888
also, I created myid file that contains integer id in /var/lib/zookeeper/ directory
You need to run zookeeper-server-start.sh zookeeper.properties individual on each server.
confluent command is only for single node testing (emphasis added)
meant for development purposes only and is not suitable for a production environment. The data that are produced are transient and are intended to be temporary
That explains why you're getting errors about files in /tmp
Related
I have installed zookeeper and kafka,zookeeper is ruuning fine but as per my requirement I want to run Kafka-connect standalone to connect kafka and cassandsra. I am using following command to start kafka server kafka/bin/connect-standalone.sh kafka/config/connect-standalone.properties kafka-connect-cassandra-sink-1.4.0/conf/cassandra-sink-standalone.properties . The connect-standalone.properties configuration file is given below:
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/home/shamama/kafka-connect-cassandra-sink-1.4.0/kafka-connect-cassandra-sink-1.4.0.jar
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000
listeners=PLAINTEXT://localhost:9092
#advertised.listeners=PLAINTEXT://localhost:9092
The following error is occurring after starting kafka standalone server:
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:82)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1628583433705, tries=1, nextAllowedTryMs=1628583433806) timed out at 1628583433706 after 1 attempt(s)
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
I cannot figure out my mistake.Please help.
The error indicates Kafka isn't running.
Based on your previous question, you're running kafka-server-start with the wrong property file
Specifically, zookeeper.connect, listeners, and advertised.listeners should not be part of connect-standalone.properties, and you need broker.id, at a minimum, to start a broker (already part of server.properties)
I am new to kafka and i started doing the Zookeeper cluster setup by refering https://codeforgeek.com/how-to-setup-zookeeper-cluster-for-kafka/
This has three zookeeper instance, I am able to start the fist zookeeper instance but in the second node zookeeper is not started.
I used this command to start zookeeper - bin/zookeeper-server-start.sh config/zookeeper.properties
I also created myid file with number 2 written in it.
Throwing the below error,
[root#kafka2 kafka_2.12-2.2.2]# bin/zookeeper-server-start.sh config/zookeeper.properties
[2020-09-06 07:25:27,761] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-09-06 07:25:27,773] INFO Resolved hostname: 0.0.0.0 to address: /0.0.0.0 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2020-09-06 07:25:27,773] INFO Resolved hostname: 172.16.10.45 to address: /172.16.10.45 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2020-09-06 07:25:27,775] INFO Resolved hostname: 172.16.10.44 to address: /172.16.10.44 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2020-09-06 07:25:27,776] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing config/zookeeper.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:156)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:104)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Caused by: java.lang.IllegalArgumentException: Missing election port for server: 1
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:369)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:152)
... 2 more
Invalid config, exiting abnormally
I found the answers finally. Actually i missed to mention the port(2888) for server.1 in config/zookeeper.properties.
Issue
server.1=<ip of first server>:3888
server.2=0.0.0.0:2888:3888
server.3=<ip of third server>:2888:3888
Working
server.1=<ip of first server>:2888:3888
server.2=0.0.0.0:2888:3888
server.3=<ip of third server>:2888:3888
I am trying to start a Zookeper instance and i keep getting an error that says the zookeeper.properties file does not exist..when in fact it does.
$ ./script.sh [2019-06-30 07:40:00,844] INFO Reading configuration
from: .\config\zookepeer.properties
(org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2019-06-30
07:40:00,844] ERROR Invalid config, exiting abnormally
(org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException:
Error processing ./config/zookepeer.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:156)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:104)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Caused by: java.lang.IllegalArgumentException:
.\config\zookepeer.properties file is missing
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:140)
... 2 more Invalid config, exiting abnormally
Folder structure
-root
-config
-zookeeper.properties
-bin
-zookeeper-server-start.sh
-script.sh
Script
./bin/zookeeper-server-start.sh ./config/zookepeer.properties
Zookeeper.properties
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
Please check the properties file name in script.
./bin/zookeeper-server-start.sh ./config/zookepeer.properties .
Change it as,
./bin/zookeeper-server-start.sh ./config/zookeeper.properties
and check.
Try this
./zookeeper-server-start.sh ../config/zookeeper.properties
Whereby you are located in the bin directory.
I'm following this tutorial trying to set up kafka,
https://kafka.apache.org/quickstart
And I get this error when doing the part on zookeeper.
It certainly has something to do with the config, but I don't understand to what extend.
bin/zookeeper-server-start.sh config/server.properties
[2018-03-17 03:27:11,623] INFO Reading configuration from: config/server.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-03-17 03:27:11,628] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing config/server.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:154)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.lang.NumberFormatException: For input string: "initial.rebalance.delay.ms"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:242)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:150)
... 2 more
Invalid config, exiting abnormally
If you look closely, you'll notice that the command to start Zookeeper is:
bin/zookeeper-server-start.sh config/zookeeper.properties
Using the zookeeper.properties file.
It looks like you copied the command to start Kafka which is using server.properties.
I fixed this on Windows by doing below.
Make sure to specify full paths.
First, go to the bin\windows folder of your kafka installation:
C:\kafka_2.12-2.0.0\bin\windows
Then, run the following command:
.\zookeeper-server-start.bat C:\kafka_2.12-2.0.0\config\zookeeper.properties
When done, you will see the following output:
[2020-07-06 20:25:13,703] INFO Server environment:user.dir=C:\kafka_2.12-2.0.0\bin\windows (org.apache.zookeeper.server.ZooKeeperServer)
[2020-07-06 20:25:13,716] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-07-06 20:25:13,717] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-07-06 20:25:13,717] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-07-06 20:25:13,787] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2020-07-06 20:25:13,791] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
You are using following to start the zookeeper:
bin/zookeeper-server-start.sh config/server.properties
Zookeeper configuration settings are placed in zookeeper.properties file.
So, to start zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
And for starting KAFKA server:
bin/kafka-server-start.sh config/server.properties
Keep in mind which file comes with which server.
I am facing same issue because I am directly giving step2 command in my terminal,which is wrong. We have to run command of step2 in our Kafka folder.
Follow below steps to run zookeeper successfully.
Step1: Goto your kafka folder or simply do cd $KAFKA_HOME
STEP2: bin/zookeeper-server-start.sh ./config/zookeeper.properties
check for "dataDir=/tmp/zookeeper1", if not present then create a directory and one file inside this directory myid
mkdir -p /tmp/zookeeper1
chmod -R 777 /tmp/zookeeper1
vi /tmp/zookeeper1/myid
(add value as 1, this is server-id from server.1=)
Try to remove config/ , and use the file directly.
e.g.
bin/zookeeper-server-start.sh zookeeper.properties
Well..every service in the world can connect to my zookeeper expect kafka. Below is my connection string in server.properties file
zk.connect=1.dzk.syd.druid.neo.com:2181, 2.dzk.syd.druid.neo.com:2181
Have have all ports on the two zookeeper servers ....total promiscuous mode. I can even telnet into the zookeeper server from the kafka server..
telnet 2.dzk.syd.druid.neo.com 2181
Trying 54.252.183.218...
Connected to 2.dzk.syd.druid.neo.com.
Escape character is '^]'.
So....rather confused on why kafka will not connect to zookeeper?
I am using ubuntu 12.04 and kafka 0.7.2
[2013-07-16 04:36:49,915] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2013-07-16 04:36:49,915] INFO Client environment:user.dir=/etc/sv/kafka (org.apache.zookeeper.ZooKeeper)
[2013-07-16 04:36:49,916] INFO Initiating client connection, connectString=1.dzk.syd.druid.neo.com:2181, 2.dzk.syd.druid.neo.com:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#39cc65b1 (org.apache.zookeeper.ZooKeeper)
[2013-07-16 04:36:49,935] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2013-07-16 04:36:49,938] FATAL Fatal error during KafkaServerStable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
org.I0Itec.zkclient.exception.ZkException: Unable to connect to 1.dzk.syd.druid.neo.com:2181, 2.dzk.syd.druid.neo.com:2181
at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:66)
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:872)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.server.KafkaZooKeeper.startup(KafkaZooKeeper.scala:44)
at kafka.log.LogManager.<init>(LogManager.scala:93)
at kafka.server.KafkaServer.startup(KafkaServer.scala:58)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:47)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.net.UnknownHostException: 2.dzk.syd.druid.neo.com: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:894)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1286)
at java.net.InetAddress.getAllByName0(InetAddress.java:1239)
at java.net.InetAddress.getAllByName(InetAddress.java:1155)
at java.net.InetAddress.getAllByName(InetAddress.java:1091)
at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:387)
at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:332)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:383)
at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64)
... 9 more
[2013-07-16 04:36:49,942] INFO Shutting down Kafka server (kafka.server.KafkaServer)
[2013-07-16 04:36:49,943] INFO shutdown scheduler kafka-logcleaner- (kafka.utils.KafkaScheduler)
[2013-07-16 04:36:49,944] INFO Kafka server shut down completed (kafka.server.KafkaServer)
In your kafka/config/server.properties, there should be a property
#host.name=localhost
if you have uncommented this, or set this to another name, then that name should be in the /etc/hosts file
It's been a while since this has been answered, but in case it could help someone here is how i fixed it :
Actually i am using an Ansible playbook to install Kafka cluster and the params generated in zookeeper.properties file were not correctly ordered :
server.1=0.0.0.0:2888:3888
server.2=kafka-4:2888:3888
server.3=kafka-5:2888:3888
server.4=kafka-3:2888:3888
server.5=kafka-2:2888:3888
Putting them in the right order,
server.1=0.0.0.0:2888:3888
server.2=kafka-2:2888:3888
server.3=kafka-3:2888:3888
server.4=kafka-4:2888:3888
server.5=kafka-5:2888:3888
Then restart Kafka service, fixed it.
Change this in zookeeper.properties
maxClientCnxns=0 to maxClientCnxns=1