Zookeeper issue in setting kafka - apache-zookeeper

To install kafka , I downloaded the kafka tar folder. To start the server I tried this command :
bin/zookeeper-server-start.sh config/zookeeper.properties
The following error occured on entering the above command:
INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2014-08-21 11:53:55,748] FATAL Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing config/zookeeper.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:110)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:99)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:76)
Caused by: java.lang.IllegalArgumentException: config/zookeeper.properties file is missing
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:94)
... 2 more
Invalid config, exiting abnormally
Is it that I need to setup zookeeper separately? How could I resolve this?

For Windows:
Go to kafka_2.11-2.0.0\bin\windows folder
Then run zookeeper-server-start.bat ../../config/zookeeper.properties

This is basically because of this
java.lang.IllegalArgumentException: config/zookeeper.properties file is missing
it would be really useful if you could share what exactly have you done so far. Also check if the same file exists at the said location and you are running the command from the correct location .. it is supposed to be run from your $KAFKA_HOME folder (where you've extracted the tar file)

I too faced the same issue when I installed kafka from Brew on Macbook
This is happening because the zookeeper.properties file is not in config of bin.
Follow these step.
Enter the command---> cd /usr/local/Cellar/kafka/2.3.0
Enter the command ---->cd libex
Now enter the command--->zookeeper-server-start config/zookeeper.properties
You will get the INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory) Message.
Earlier I was getting this error:
$ zookeeper-server-start config/zookeeper.properties
[2019-10-02 14:35:20,159] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2019-10-02 14:35:20,160] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing config/zookeeper.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:156)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:104)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Caused by: java.lang.IllegalArgumentException: config/zookeeper.properties file is missing
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:140)
... 2 more
Invalid config, exiting abnormally

I saw when you run the above command it doesn't take config file. So if you Put complete path like c:\Kafka\config\zookeeper.properties... this works.

I faced the exact same error, and after a while I realized that the reason for the error was, I wasn't able to find the zookeeper.properties file, and that was because the path wasn't correct, I installed kafka through brew so the config folder was created inside libexec, so find where the config directory is and check for zookeeper.properties inside it and give that path.

Had the same issue.
I was following this guide and step 2 mentions to run this command:
bin/zookeeper-server-start.sh config/zookeeper.properties
I had 2 problems, that first was that I wasn't inside the root directory of the file you untar and the second was that I didn't copy the complete command. Make sure both of them are correct and try again.

Just make sure that whether /config folder exist or not.
Try to type properties directly. e.g. zookeeper-server-start zookeeper.properties
I installed it with homebrew, it works.

This happens because bin/windows is added to the path but kafka/config is not.
Just navigate to your kafka folder and then try to run.
I am adding screenshot if it can help.
Before
After

You can use Powershell as an alternative to CMD.
Consider myKafka is your kafka home directory, Extract your kafka tar file here.
Extracted folder(KafkaDir) will be having ./bin,/config, etc. internal folders.
Now, open Powershell prompt, go to myKafka folder.
Run below command:
.\kafkaDir\bin\windows\zookeeper-server-start.bat
.\kafkaDir\config\zookeeper.properties
Zookeeper will get start.

You need to fix the absolute path to:
$KAFKA_HOME/config/zookeeper.properties
For me I used:
$KAFKA_HOME = /usr/local/kafka

In \bin\windows\kafka-run-class.bat add the file content
rem Classpath addition for release
for %%i in ("%BASE_DIR%\libs\*") do (
call :concat "%%i"
)
// Section to Add
rem Classpath addition for LSB style path
if exist %BASE_DIR%\share\java\kafka\* (
call :concat %BASE_DIR%\share\java\kafka\*
)
**
// Above Section to Add
rem Classpath addition for core
for %%i in ("%BASE_DIR%\core\build\libs\kafka_%SCALA_BINARY_VERSION%*.jar") do (
call :concat "%%i"

Have to run in from Kafka home directory, but you are running from the bin.

Related

Invalid Config, Exiting abnormally when launching zookeper from Kafka

while doing setup facing the following issue
C:\ProgramData\confluent-6.2.1\bin\windows>zookeeper-server-start.bat zookeeper.properties
[2021-10-12 18:16:00,983] INFO Reading configuration from: zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-10-12 18:16:00,985] WARN zookeeper.properties is relative. Prepend .\ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-10-12 18:16:00,986] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing zookeeper.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:157)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:113)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Caused by: java.lang.IllegalArgumentException: zookeeper.properties file is missing
at org.apache.zookeeper.server.util.VerifyingFileFactory.doFailForNonExistingPath(VerifyingFileFactory.java:51)
at org.apache.zookeeper.server.util.VerifyingFileFactory.validate(VerifyingFileFactory.java:45)
at org.apache.zookeeper.server.util.VerifyingFileFactory.create(VerifyingFileFactory.java:40)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:142)
... 2 more
Invalid config, exiting abnormally
Does anybody know how to solve this?
It seems that you did not configure the location of the properties file (zookeeper.properties file is missing) in a correct manner. You can fist double check if it exists and then supply the file at launch as in following example from Linux command line (and for a service it will be the same):
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
Here the folder is relative to the current one, but an absolute path is safer. Beware that in windows the slashes will be inverted, so in your case, the correct command should be something like:
C:\ProgramData\confluent-6.2.1\bin\windows>zookeeper-server-start.bat C:\ProgramData\confluent-6.2.1\config\zookeeper.properties

Unable to start zookeeper getting exception saying that unable to load

I am unable to start zookeeper in my local
getting exception
Error: Could not find or load main class software\kafka_2.12-2.3.0\libs\activation-1.1.1.jar;
I have got the same error.
Finally I found the reason.
I put kafka in a folder which it's name contains space like this
C:\Open source\kafka\kafka_2.12-2.3.0
Then I copied all kafka_2.12-2.3.0 to a new folder without space in name, such asc C:\kafka_2.12-2.3.0
and kafka can run
NOte that the command line is: zookeeper-server-start.bat ../../config/zookeeper.properties
NOT: zookeeper-server-start.bat config/zookeeper.properties
Here's my result

Kafka Server - Could not find a 'KafkaServer' in JAAS

I have a standalone kafka broker that I'm trying to configure SASL for. Configurations are below. I'm trying to set up SASL_PLAIN authentication on the broker.
My understanding is that with the listener.name... configuration in the server.properties, I shouldn't need the jaas file. But I've experimented with one to see if that might be a better approach.
I have experimented with each of these commands, but both result in the same exception.
sudo bin/kafka-server-start etc/kafka/server.properties
sudo -Djava.security.auth.login.config=etc/kafka/kafka_server_jaas.conf bin/kafka-server-start etc/kafka/server.properties
the exception displayed is:
Fatal error during KafkaServer startup. Prepare to shutdown... Could
not find a 'KafkaServer' or 'sasl_plaintext.KafkaServer' entry in the
JAAS configuration. System property 'java.security.auth.login.config'
is not set
server.properties:
listeners=SASL_PLAINTEXT://0.0.0.0:9092
listener.security.protocol.map: SASL_PLAINTEXT:SASL_PLAINTEXT
listener.name.SASL_PLAINTEXT.plain.sasl.jaas.config:
org.apache.kafka.common.security.plain.PlainLoginModule required /
username="username" /
password="Password" /
user_username="Password";
advertised.listeners=SASL_PLAINTEXT://[ipaddress]:9092
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
secutiy.inter.broker.protocol=SASL_PLAINTEXT
kafka_server_jaas.conf:
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="username"
password="Password"
user_username="Password";
};
I've spent a day looking at this already - has anyone else had experience with this problem?
You need to export a variable, not in-line the config to kafka-server-start (or sudo).
export KAFKA_OPTS="-Djava.security.auth.login.config=/path/to/kafka_server_jaas.conf"
bin/kafka-server-start /path/to/server.properties
Ref. Confluent's sections on Kafka security
Putting my mistakes here for austerity:
Don't do your startup commands from the cli, put them in a .sh file and run from there:
For example, something like this:
zkstart
export KAFKA_OPTS="-Djava.security.auth.login.config=etc/kafka/zookeeper_jaas.conf"
bin/zookeeper-server-start etc/kafka/zookeeper.properties &
kafkastart
export KAFKA_OPTS=-Djava.security.auth.login.config=etc/kafka/kafka_server_jaas.conf
bin/kafka-server-start etc/kafka/server.properties
If you still encounter an error related to the configs, check your _jaas files to ensure all the configuration sections in the error messages are present. If they are, it's likely the format isn't quite correct - check for the two semi-colons in each section and if that fails, try recreating the file entirely from scratch (or from a c&p from the documentation).
edit
So, the final solution for me was to add the export.... lines to the beginning of the corresponding kafka-server-start and zookeeper-server-start files. It took me a while before the 'everything is a file' finally clicked and I realized the script files were the actual basis for the services.

Hawq init failed -- "postgres" is needed by initdb

After I build incubator-hawq on Centos7.1, I tried to init it. But the error below occurs:
20160516:18:10:43:002036 hawqinit.sh:host-172-16-0-105:hawqadmin-[INFO]:-Loading hawq_toolkit...
ALTER ROLE
20160516:18:10:44:001766 hawq_init:host-172-16-0-105:hawqadmin-[INFO]:-20160516:18:10:43:002036 hawqinit.sh:host-172-16-0-105:hawqadmin-[INFO]:-Loading hawq_toolkit...
20160516:18:10:44:001766 hawq_init:host-172-16-0-105:hawqadmin-[INFO]:-Master init successfully
20160516:18:10:44:001766 hawq_init:host-172-16-0-105:hawqadmin-[INFO]:-Init segments in list: ['hawq-master']
20160516:18:10:44:001766 hawq_init:host-172-16-0-105:hawqadmin-[DEBUG]:-Start to init segment on node 'hawq-master'
20160516:18:10:44:001766 hawq_init:host-172-16-0-105:hawqadmin-[INFO]:-Total segment number is: 1
fgets failure: Success
The program "postgres" is needed by initdb but was either not found in the same directory as "/usr/hawq/bin/initdb" or failed unexpectedly.
Check your installation; "postgres -V" may have more information.
20160516:18:10:45:002318 hawqinit.sh:host-172-16-0-105:hawqadmin-[ERROR]:-Postgres initdb failed
20160516:18:10:45:002318 hawqinit.sh:host-172-16-0-105:hawqadmin-[ERROR]:-Segment init failed on host-172-16-0-105
20160516:18:10:45:001766 hawq_init:host-172-16-0-105:hawqadmin-[INFO]:-20160516:18:10:45:002318 hawqinit.sh:host-172-16-0-105:hawqadmin-[ERROR]:-Postgres initdb failed
20160516:18:10:45:002318 hawqinit.sh:host-172-16-0-105:hawqadmin-[ERROR]:-Segment init failed on host-172-16-0-105
20160516:18:10:45:001766 hawq_init:host-172-16-0-105:hawqadmin-[ERROR]:-HAWQ init failed on hawq-master
20160516:18:10:46:001766 hawq_init:host-172-16-0-105:hawqadmin-[INFO]:-0 of 1 segments init successfully
20160516:18:10:46:001766 hawq_init:host-172-16-0-105:hawqadmin-[ERROR]:-Segments init failed, exit
When I type the command, the below shows:
[hawqadmin#host-172-16-0-105 hawqAdminLogs]$ postgres -V
postgres (HAWQ) 8.2.15
Any advice? Thanks!
If "postgres -V" works, that means the postgres binary is good.
Before you do "hawq init cluster", please make sure:
1) $GPHOME in greenplum_path.sh is correctly set to the directory of hawq binary, i.e, /usr/hawq in your case
2) source $GPHOME/greenplum_path.sh
3) check if initdb and postgres binary is in $GPHOME/bin
From the error you pasted above, 2 possible causes:
(1) The binary postgres called is not /usr/hawq/bin/postgres, You can use which postgres to check the path.
(2) The dependent lib for postgres may be wrong. You can use ldd for linux or otool for mac to print all dependent lib paths, and check them.
Moreover, if any error when init hawq, please check log in ~/hawqAdminLogs/, you may find out the specific error message.
Hope it will help you to find out the root cause.
Recently I faced same error while initializing cluster.
Postgres -V showed correct version, which postgres showed /usr/local/hawq/bin/postgres, also the path was already set, still faced above error.
Finally resolved by setting LD_LIBRARY_PATH to /usr/local/hawq/lib/ and sourced it via .bashrc file.
Looks like you might have installed hawq binaries in different directory . Please check the following
1.Make sure you have all the right PATH set
Check hawq initdb binaries are there in /usr/hawq/bin/ directory
make sure you have successed compile hawq and install them
check postgres is in the same dir with initdb
if there are more than 1 postgres in your pc, make sure the path of postgres(the same dir with initdb) is in your PATH.

Cloudera Kafka can not run

I installed Zookeeper and tried to install Kafka0.8.0 on Cloudera5.4.4. It successfully deployed, but when I ran it, it failed. The error log as following:
[Errno 2] No such file or directory: '/var/log/kafka/server.log'
I really have no any idea.
Thanks in advance!
I met this problem before, the default maximum memory of brokers were set to 0MB by Cloudera(it seems a bug of Cloudeara), it caused Kafka could not get run, and the parameter fetch.message.max.bytes also was set to low by default. Check the stderr installation log, search keyword ERROR, otherwise the log too messy to check. You would find the root error message. The message above [Errno 2] No such file or directory: '/var/log/kafka/server.log' is not the root exception.