I have installed zookeeper on my Linux server ubuntu 12.04 in some folder like abc/zookeeper/zkserver/bin/zkserver.sh start. this works fine and starts the server as expected but when i put this zkserver.sh file in /etc/init.d folder and copy the file in rc2.d folder for the reason to start-up zookeeper on system reboot. and when i run this command /etc/init.d/zkserver.sh start this is giving errors like:
JMX enabled by default
Using config: /etc/init.d/../etc/zookeeper/zoo.cfg
grep: /etc/init.d/../etc/zookeeper/zoo.cfg: No such file or directory
mkdir: cannot create directory `': No such file or directory
Starting zookeeper ... STARTED
The zkServer.sh is dependent on a certain directory structure and certain files being present. It is not supposed to be moved in isolation like that. It is also not supposed to be used as an init script.
Check if your zk download comes with the init script. Try looking at src/packages/rpm/init.d/zookeeper or similar, and use that one instead.
Related
I'm trying to install Kafka message queue on Windows for testing purposes (not for production).
I found this article on how to install Apache Kafka 0.8 on Windows: http://janschulte.wordpress.com/2013/10/13/apache-kafka-0-8-on-windows/
It's a good article, but it's outdated, unfortunately.
Does someone know a way to achieve that?
Ok, it's finally not complicated :)
The only steps are:
Edit (2021/10) Latest versions of Kafka end up with an AccessDeniedException upon startup, and even when successfully started, throw the same error during log rotation. Kafka utilizes Java methods designed for Unix filesystem libraries, so installing Kafka in WSL2 is recommended (link to Confluent blog, but steps also applicable to Apache Kafka direct installation).
Old answer
Download Kafka and uncompress it somewhere nice (let's say C:/Kafka)
Install Cygwin
If using Cygwin, edit \bin\kafka-run-class.sh and at the end of the file, change
`exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"`
to
``exec java $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp `cygpath -wp $CLASSPATH` $KAFKA_OPTS "$#"``
In Environment Variables, Add java to your Path System Variable:
That's it.. you can now run ZooKeeper and Kafka servers and start playing with topics and stuff..
These are the steps I followed to run kafka on Windows
Install Zookeeper first (I downloaded v3.3.6) zookeeper-3.3.6.tar.gz
Extract Zookeeper and run this command in powershell/cmd \zookeeper-3.3.6\bin> .\zkServer.cmd Now this should up a Zookeeper instance on localhost:2181
Download Kafka binary version (I downloaded v0.10.0.1)kafka_2.10-0.10.0.1.tgz
Extract Kafka, time to modify some configs
Inside Kafka extraction you can find .\config\server.properties
In .\config\server.properties replace log.dirs=c:/kafka/kafka-logs
Note: Make sure to create those folders in relevant paths
Happy news: Now Kafka ships with windows .bat scripts, You can find these files inside ./bin/windows folder
Start powershell/cmd and run this command to start Kafka broker .\bin\windows\kafka-server-start.bat .\config\server.properties
DONE!, Now you have a running Zookeeper instance and a Kafka broker.
You will probably find that Kafka comes with Windows .bat files under the bin folder to help you run Kafka under windows. However, those bat files are not updated since version 0.8.0 and some of Kafka's logic has changed (regarding topics, etc.).
I have tried two ways to solve this problem. One is to use Cygwin or MinGW to simulate Linux environment and run the Linux shell script, but there would be tons of other problems related to path names. The other solution, which is simpler and come with less trouble, is to use the corrected version of Windows bat files.
Please refer to this blog post.
Download any binary version(I've downloaded kafka_2.11-0.10.2.0) of Kafka from https://kafka.apache.org/downloads
Extract it to any folder(I've extracted to "C:\Kafka")
Open Command Prompt
Go to the folder that you extract Kafka(C:\Kafka\kafka_2.11-0.10.2.0)
Run this command to start Zookeeper
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Run this command to start Kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties
Now it works!
In version 0.9.0. there are scripts in "bin" folder for windows. You can use them.
Running From A windows Shell
As of April 2019, downloading Kafka from their website worked on Windows almost right out of the box
Downloading and using the windows version .bat files is described here:
https://kafka.apache.org/quickstart
I ran into two problems when I did this:
1) JAVA_HOME was set to an unsupported JDK which led to this kind of error
Exception in thread "main" java.lang.VerifyError: Uninitialized object
exists on backward branch 209
Replacing with JDK 11 solved the problem.
2) The JAVA_HOME must not contain spaces which caused a 'cannot find the specified file' error. To fix this I used a shortened path like set JAVA_HOME=C:\Progra~1\Java\jdk-11.0.1
Running From Cygwin
If instead of the .bat files, you want to run the .sh files from cygwin, there is actually quite a bit that needs to be done, and even after that there may be problems that come up later. I can't really recommend this option, but I do use it as it's pretty convenient for some purposes.
If your JAVA_HOME path contains a space e.g. "C:\Program Files\Java\Jdk..." you will see something like this:
bin/kafka-run-class.sh: line 305: exec: C:\Program: not found
One solution is to copy the jdk to a path without spaces, and change the Java home accordingly.
If you don't want to change the JDK location, you can change the cygwin env variable as follows:
JAVA_HOME="/cygdrive/c/Program Files/Java/jdk-11.0.1"
and change the line
exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
to
exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
There was another problem with the log parameters in kafka-run-class.sh and I had to replace the line
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
with
KAFKA_GC_LOG_OPTS="-Xlog:gc*"
And even after all these changes I occasionally run into problems of Kafka shutting down because of an incompatible windows style path in the logs directories as described here:
Kafka 1.0 stops with FATAL SHUTDOWN error. Logs directory failed
In short, you may be better off running the Kafka .bat scripts from the windows directory...
With Chocolatey and Powershell:
chocolatey.exe install kafka -> installs Kafka into C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ - your version might differ of course
add C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ to your powershell path using notepad $PROFILE
After restarting powershell kafka-console-consumer should work as a command.
You also need to install ssl.properties with keystore and truststore. Put them e.g. into C:\ProgramData\kafka and put that into ssl.properties (please note the escaping of the backslashes):
security.protocol=SSL
ssl.truststore.location=C:\\ProgramData\\kafka\\kafka-truststore.jks
ssl.truststore.password=PASSWORD
ssl.keystore.location=C:\\ProgramData\\kafka\\kafka-keystore.jks
ssl.keystore.password=PASSWORD
ssl.key.password=PASSWORD
client.id=console-test
Now things like kafka-consumer-groups --bootstrap-server SERVERNAME:9094 --command-config C:\ProgramData\kafka\ssl.properties --list should first pop up a firewall warning which you'd need to accept and then output a list of groups.
Kafka & Zookeeper have inbuilt Windows specific scripts as verified in the latest version. As mentioned in other other answers, the directory for windows scripts will be under kafka_2.12-2.4.0\bin\windows for Kafka. Zookeeper scripts are under /bin only but ending with "*.cmd"
There will be few minor additional steps you will have to do with ZooKeeper & Kafka like
Creating zoo.cfg for ZooKeeper
Configuring windows specific data directory for ZeeKeeper
Kafka comes with default configuration file & it has very good OS specific handling i.e. log.dirs=/tmp/kafka-logs but it will create that directory structure under C:/ directory seamlessly without failing.
It is also easy to quickly setup a Kafka Cluster with multiple Kafka Brokers on Windows with minor configurations. You can also create topics, publish/consume messages, verify topics/partitions etc. all from Windows Command Prompt. Here is a good detailed reference for the windows specific setup Kafka Setup on Windows OS | Basic Installation, Setup, Verification, Cluster Setup, Storage
I had configured Zookeeper and Apache Kafka on my Windows machine.
For Zookeeper, simply configure ZOOKEEPER_HOME and set the PATH also and then rename the zoo.sample.cfg to zoo.cfg and also change the path for dataDir in the zoo.cfg
For Apache Kafka, simply unzip and start the server from bin folder and start the Producer/Consumer from windows folder under the Kafka directory
Ok, so I am trying to move the /var/lib/tftpboot folder the "proper" way to a dedicated partition. To accomplish this goal I have setup a separate partition called /app and moved the tftpboot folder there.
Issue 1: Symlink
After I moved the folder I created a symlink from the new directory to the old directory using the ln -s /app/tftpboot /var/lib/ command. After doing this I am unable to successfully restart the service using systemctl restart tftp. However, if I just update the path listed in the service file and the config file the service boots fine.
I'm trying to install Kafka message queue on Windows for testing purposes (not for production).
I found this article on how to install Apache Kafka 0.8 on Windows: http://janschulte.wordpress.com/2013/10/13/apache-kafka-0-8-on-windows/
It's a good article, but it's outdated, unfortunately.
Does someone know a way to achieve that?
Ok, it's finally not complicated :)
The only steps are:
Edit (2021/10) Latest versions of Kafka end up with an AccessDeniedException upon startup, and even when successfully started, throw the same error during log rotation. Kafka utilizes Java methods designed for Unix filesystem libraries, so installing Kafka in WSL2 is recommended (link to Confluent blog, but steps also applicable to Apache Kafka direct installation).
Old answer
Download Kafka and uncompress it somewhere nice (let's say C:/Kafka)
Install Cygwin
If using Cygwin, edit \bin\kafka-run-class.sh and at the end of the file, change
`exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"`
to
``exec java $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp `cygpath -wp $CLASSPATH` $KAFKA_OPTS "$#"``
In Environment Variables, Add java to your Path System Variable:
That's it.. you can now run ZooKeeper and Kafka servers and start playing with topics and stuff..
These are the steps I followed to run kafka on Windows
Install Zookeeper first (I downloaded v3.3.6) zookeeper-3.3.6.tar.gz
Extract Zookeeper and run this command in powershell/cmd \zookeeper-3.3.6\bin> .\zkServer.cmd Now this should up a Zookeeper instance on localhost:2181
Download Kafka binary version (I downloaded v0.10.0.1)kafka_2.10-0.10.0.1.tgz
Extract Kafka, time to modify some configs
Inside Kafka extraction you can find .\config\server.properties
In .\config\server.properties replace log.dirs=c:/kafka/kafka-logs
Note: Make sure to create those folders in relevant paths
Happy news: Now Kafka ships with windows .bat scripts, You can find these files inside ./bin/windows folder
Start powershell/cmd and run this command to start Kafka broker .\bin\windows\kafka-server-start.bat .\config\server.properties
DONE!, Now you have a running Zookeeper instance and a Kafka broker.
You will probably find that Kafka comes with Windows .bat files under the bin folder to help you run Kafka under windows. However, those bat files are not updated since version 0.8.0 and some of Kafka's logic has changed (regarding topics, etc.).
I have tried two ways to solve this problem. One is to use Cygwin or MinGW to simulate Linux environment and run the Linux shell script, but there would be tons of other problems related to path names. The other solution, which is simpler and come with less trouble, is to use the corrected version of Windows bat files.
Please refer to this blog post.
Download any binary version(I've downloaded kafka_2.11-0.10.2.0) of Kafka from https://kafka.apache.org/downloads
Extract it to any folder(I've extracted to "C:\Kafka")
Open Command Prompt
Go to the folder that you extract Kafka(C:\Kafka\kafka_2.11-0.10.2.0)
Run this command to start Zookeeper
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Run this command to start Kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties
Now it works!
In version 0.9.0. there are scripts in "bin" folder for windows. You can use them.
Running From A windows Shell
As of April 2019, downloading Kafka from their website worked on Windows almost right out of the box
Downloading and using the windows version .bat files is described here:
https://kafka.apache.org/quickstart
I ran into two problems when I did this:
1) JAVA_HOME was set to an unsupported JDK which led to this kind of error
Exception in thread "main" java.lang.VerifyError: Uninitialized object
exists on backward branch 209
Replacing with JDK 11 solved the problem.
2) The JAVA_HOME must not contain spaces which caused a 'cannot find the specified file' error. To fix this I used a shortened path like set JAVA_HOME=C:\Progra~1\Java\jdk-11.0.1
Running From Cygwin
If instead of the .bat files, you want to run the .sh files from cygwin, there is actually quite a bit that needs to be done, and even after that there may be problems that come up later. I can't really recommend this option, but I do use it as it's pretty convenient for some purposes.
If your JAVA_HOME path contains a space e.g. "C:\Program Files\Java\Jdk..." you will see something like this:
bin/kafka-run-class.sh: line 305: exec: C:\Program: not found
One solution is to copy the jdk to a path without spaces, and change the Java home accordingly.
If you don't want to change the JDK location, you can change the cygwin env variable as follows:
JAVA_HOME="/cygdrive/c/Program Files/Java/jdk-11.0.1"
and change the line
exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
to
exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
There was another problem with the log parameters in kafka-run-class.sh and I had to replace the line
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
with
KAFKA_GC_LOG_OPTS="-Xlog:gc*"
And even after all these changes I occasionally run into problems of Kafka shutting down because of an incompatible windows style path in the logs directories as described here:
Kafka 1.0 stops with FATAL SHUTDOWN error. Logs directory failed
In short, you may be better off running the Kafka .bat scripts from the windows directory...
With Chocolatey and Powershell:
chocolatey.exe install kafka -> installs Kafka into C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ - your version might differ of course
add C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ to your powershell path using notepad $PROFILE
After restarting powershell kafka-console-consumer should work as a command.
You also need to install ssl.properties with keystore and truststore. Put them e.g. into C:\ProgramData\kafka and put that into ssl.properties (please note the escaping of the backslashes):
security.protocol=SSL
ssl.truststore.location=C:\\ProgramData\\kafka\\kafka-truststore.jks
ssl.truststore.password=PASSWORD
ssl.keystore.location=C:\\ProgramData\\kafka\\kafka-keystore.jks
ssl.keystore.password=PASSWORD
ssl.key.password=PASSWORD
client.id=console-test
Now things like kafka-consumer-groups --bootstrap-server SERVERNAME:9094 --command-config C:\ProgramData\kafka\ssl.properties --list should first pop up a firewall warning which you'd need to accept and then output a list of groups.
Kafka & Zookeeper have inbuilt Windows specific scripts as verified in the latest version. As mentioned in other other answers, the directory for windows scripts will be under kafka_2.12-2.4.0\bin\windows for Kafka. Zookeeper scripts are under /bin only but ending with "*.cmd"
There will be few minor additional steps you will have to do with ZooKeeper & Kafka like
Creating zoo.cfg for ZooKeeper
Configuring windows specific data directory for ZeeKeeper
Kafka comes with default configuration file & it has very good OS specific handling i.e. log.dirs=/tmp/kafka-logs but it will create that directory structure under C:/ directory seamlessly without failing.
It is also easy to quickly setup a Kafka Cluster with multiple Kafka Brokers on Windows with minor configurations. You can also create topics, publish/consume messages, verify topics/partitions etc. all from Windows Command Prompt. Here is a good detailed reference for the windows specific setup Kafka Setup on Windows OS | Basic Installation, Setup, Verification, Cluster Setup, Storage
I had configured Zookeeper and Apache Kafka on my Windows machine.
For Zookeeper, simply configure ZOOKEEPER_HOME and set the PATH also and then rename the zoo.sample.cfg to zoo.cfg and also change the path for dataDir in the zoo.cfg
For Apache Kafka, simply unzip and start the server from bin folder and start the Producer/Consumer from windows folder under the Kafka directory
I am trying to install the zookeeper in my laptop. Using the file, which is bin.tar.gz 3.6.1 version.https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.6.1/apache-zookeeper-3.6.1-bin.tar.gz
Following the instructions which are:
make a folder logs in the main folder.
change the file name in conf folder, from Zoo_sample.cfg to zoo.cfg
change the dataDir path inside the zoo.cfg to where the logs situated.
put the bin path to the environment variables following the post here:https://medium.com/#shaaslam/installing-apache-zookeeper-on-windows-45eda303e835
Using Windows 10 with Bash, the feedback:
alex0#DESKTOP-AGJ32D1 MINGW64 /c/Tools/apache-zookeeper-3.6.1-bin/bin
$ ./zkServer.sh
ZooKeeper JMX enabled by default
Using config: C:\Tools\apache-zookeeper-3.6.1-bin\conf\zoo.cfg
grep: C:\Tools\apache-zookeeper-3.6.1-bin\conf\zoo.cfg: No such file or directory
grep: C:\Tools\apache-zookeeper-3.6.1-bin\conf\zoo.cfg: No such file or directory
mkdir: cannot create directory ‘’: No such file or directory
Usage: ./zkServer.sh [--config <conf-dir>] {start|start-foreground|stop|version|restart|status|print-cmd}
I am sure that the name of cfg file is properly changed, and the dataDir path is set as well. What could the problem I've made? Any help is highly appreciated.
I've figured it out. It is simply the file name issue.
When I changed it into zoo.cfg, the name of the file is zoo.cfg. Therefore, just change it to zoo should solve the problem.
when you setup zookeeper for the first time at your linux system, the default configuration file that have been created is zoo_sample.cfg then when your run the server or the zkServer.sh file, the file came by default to check on the config file named as zoo.cfg, so there the source of the issue.
zookeeper config dir
in my case what I did is that I renamed the file in the zookeeper dir from zoo_simple.cfg to zoo.cfg by using the following command on terminal.
~$ mv zoo_sample.cfg zoo.cfg
in case you still facing an error of zookeeper failed to start as you see in this error example
Try to run Zookeeper with the config file full path, here's the command:
~$ sudo ./zkServer.sh start /[path_to_zookeeper_config_dir]/zoo.cfg
then the server will start
I make index on one Sphinx compatible xml document.It runs success fully.
Then i make service using searchd command.Following is command to make service
searchd --install --config "path for config file" --servicename "servicename" --port "portnumber".
If Sphinx config file is put inside Sphinx directory then service create and start successfully.
But if config file is not inside Sphinx directory then service created successfully but it can not start successfully.
Should i change something inside searchd block in config file?
According the the documentation (http://sphinxsearch.com/docs/current.html#ref-searchd) all parameters specified when creating the service is called when starting the service.
--install installs searchd as a service into the Microsoft Management Console (Control Panel / Administrative Tools / Services). Any other parameters specified on the command line, where --install is specified will also become part of the command line on future starts of the service. For example, as part of calling searchd, you will likely also need to specify the configuration file with --config, and you would do that as well as specifying --install. Once called, the usual start/stop facilities will become available via the management console, so any methods you could use for starting, stopping and restarting services would also apply to searchd. Example:
C:\WINDOWS\system32> C:\Sphinx\bin\searchd.exe --install --config C:\Sphinx\sphinx.conf
The above command means that the config file must always exist at C:\Sphinx\sphinx.conf.
If your "path for config file" exists and the service still does not start, then I would suggest creating a bug at http://sphinxsearch.com/bugs/main_page.php so it cab be resolved.
In the meantime, put the config file in the Sphinx directory where it works.
Make sure you have a log folder in the sphinx folder. If you check the searchd section of the config file, it will have an option for where the log files and pid are to be placed. If the directory does not exist, then the service will not start.