I am trying to start Confluent 5.3.0 on Ubuntu and need some help.
The first thing I tried was navigating to confluent-5.3.0/bin and running sudo ./confluent start (because that's the way I've been starting Confluent 5.2.1).
This gave an error message: sudo: ./confluent: command not found.
I then checked the folder confluent-5.3.0/bin and found it's missing a shell script named confluent, which is included in 5.2.1.
I checked the Release Notes and the Quickstart. Both of them say I should start Confluent with confluent local start instead of confluent start. (See screenshot at bottom). However, I still get the same error message because the shell script is still missing: sudo: ./confluent: command not found
For those of you who are able to start Confluent 5.3.0, how are you doing it? Can you paste your command line here?
Did you have to write your own confluent shell script for 5.3.0?
Can I just copy my confluent shell script from 5.2.1? I'm assuming it must be incompatible, since they removed it from 5.3.0.
From 5.3.0 you will have to install CLI separately
https://docs.confluent.io/current/cli/installing.html#cli-install
curl -L https://cnfl.io/cli | sh -s -- -b /path-to-directory/bin
Then you can run
confluent local start --path <path-to-directory>
Related
I'm using http://packages.confluent.io/archive/7.3/ > confluent-7.3.0.zip
confluent local services start
getting the error below even though the file is there. Please help!
Using CONFLUENT_CURRENT: C:\Users\user\AppData\Local\Temp\confluent.727246
Starting ZooKeeper
Error: exec: "D:\\Development\\tools\\confluent\\bin\\zookeeper-server-start": file does not exist
bin\zookeeper-server-start.sh script is intended for Unix execution.
Confluent Platform does not support Windows
If you use WSL2, it may work there. How to install Kafka on Windows?
I'm trying to install Kafka message queue on Windows for testing purposes (not for production).
I found this article on how to install Apache Kafka 0.8 on Windows: http://janschulte.wordpress.com/2013/10/13/apache-kafka-0-8-on-windows/
It's a good article, but it's outdated, unfortunately.
Does someone know a way to achieve that?
Ok, it's finally not complicated :)
The only steps are:
Edit (2021/10) Latest versions of Kafka end up with an AccessDeniedException upon startup, and even when successfully started, throw the same error during log rotation. Kafka utilizes Java methods designed for Unix filesystem libraries, so installing Kafka in WSL2 is recommended (link to Confluent blog, but steps also applicable to Apache Kafka direct installation).
Old answer
Download Kafka and uncompress it somewhere nice (let's say C:/Kafka)
Install Cygwin
If using Cygwin, edit \bin\kafka-run-class.sh and at the end of the file, change
`exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"`
to
``exec java $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp `cygpath -wp $CLASSPATH` $KAFKA_OPTS "$#"``
In Environment Variables, Add java to your Path System Variable:
That's it.. you can now run ZooKeeper and Kafka servers and start playing with topics and stuff..
These are the steps I followed to run kafka on Windows
Install Zookeeper first (I downloaded v3.3.6) zookeeper-3.3.6.tar.gz
Extract Zookeeper and run this command in powershell/cmd \zookeeper-3.3.6\bin> .\zkServer.cmd Now this should up a Zookeeper instance on localhost:2181
Download Kafka binary version (I downloaded v0.10.0.1)kafka_2.10-0.10.0.1.tgz
Extract Kafka, time to modify some configs
Inside Kafka extraction you can find .\config\server.properties
In .\config\server.properties replace log.dirs=c:/kafka/kafka-logs
Note: Make sure to create those folders in relevant paths
Happy news: Now Kafka ships with windows .bat scripts, You can find these files inside ./bin/windows folder
Start powershell/cmd and run this command to start Kafka broker .\bin\windows\kafka-server-start.bat .\config\server.properties
DONE!, Now you have a running Zookeeper instance and a Kafka broker.
You will probably find that Kafka comes with Windows .bat files under the bin folder to help you run Kafka under windows. However, those bat files are not updated since version 0.8.0 and some of Kafka's logic has changed (regarding topics, etc.).
I have tried two ways to solve this problem. One is to use Cygwin or MinGW to simulate Linux environment and run the Linux shell script, but there would be tons of other problems related to path names. The other solution, which is simpler and come with less trouble, is to use the corrected version of Windows bat files.
Please refer to this blog post.
Download any binary version(I've downloaded kafka_2.11-0.10.2.0) of Kafka from https://kafka.apache.org/downloads
Extract it to any folder(I've extracted to "C:\Kafka")
Open Command Prompt
Go to the folder that you extract Kafka(C:\Kafka\kafka_2.11-0.10.2.0)
Run this command to start Zookeeper
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Run this command to start Kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties
Now it works!
In version 0.9.0. there are scripts in "bin" folder for windows. You can use them.
Running From A windows Shell
As of April 2019, downloading Kafka from their website worked on Windows almost right out of the box
Downloading and using the windows version .bat files is described here:
https://kafka.apache.org/quickstart
I ran into two problems when I did this:
1) JAVA_HOME was set to an unsupported JDK which led to this kind of error
Exception in thread "main" java.lang.VerifyError: Uninitialized object
exists on backward branch 209
Replacing with JDK 11 solved the problem.
2) The JAVA_HOME must not contain spaces which caused a 'cannot find the specified file' error. To fix this I used a shortened path like set JAVA_HOME=C:\Progra~1\Java\jdk-11.0.1
Running From Cygwin
If instead of the .bat files, you want to run the .sh files from cygwin, there is actually quite a bit that needs to be done, and even after that there may be problems that come up later. I can't really recommend this option, but I do use it as it's pretty convenient for some purposes.
If your JAVA_HOME path contains a space e.g. "C:\Program Files\Java\Jdk..." you will see something like this:
bin/kafka-run-class.sh: line 305: exec: C:\Program: not found
One solution is to copy the jdk to a path without spaces, and change the Java home accordingly.
If you don't want to change the JDK location, you can change the cygwin env variable as follows:
JAVA_HOME="/cygdrive/c/Program Files/Java/jdk-11.0.1"
and change the line
exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
to
exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
There was another problem with the log parameters in kafka-run-class.sh and I had to replace the line
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
with
KAFKA_GC_LOG_OPTS="-Xlog:gc*"
And even after all these changes I occasionally run into problems of Kafka shutting down because of an incompatible windows style path in the logs directories as described here:
Kafka 1.0 stops with FATAL SHUTDOWN error. Logs directory failed
In short, you may be better off running the Kafka .bat scripts from the windows directory...
With Chocolatey and Powershell:
chocolatey.exe install kafka -> installs Kafka into C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ - your version might differ of course
add C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ to your powershell path using notepad $PROFILE
After restarting powershell kafka-console-consumer should work as a command.
You also need to install ssl.properties with keystore and truststore. Put them e.g. into C:\ProgramData\kafka and put that into ssl.properties (please note the escaping of the backslashes):
security.protocol=SSL
ssl.truststore.location=C:\\ProgramData\\kafka\\kafka-truststore.jks
ssl.truststore.password=PASSWORD
ssl.keystore.location=C:\\ProgramData\\kafka\\kafka-keystore.jks
ssl.keystore.password=PASSWORD
ssl.key.password=PASSWORD
client.id=console-test
Now things like kafka-consumer-groups --bootstrap-server SERVERNAME:9094 --command-config C:\ProgramData\kafka\ssl.properties --list should first pop up a firewall warning which you'd need to accept and then output a list of groups.
Kafka & Zookeeper have inbuilt Windows specific scripts as verified in the latest version. As mentioned in other other answers, the directory for windows scripts will be under kafka_2.12-2.4.0\bin\windows for Kafka. Zookeeper scripts are under /bin only but ending with "*.cmd"
There will be few minor additional steps you will have to do with ZooKeeper & Kafka like
Creating zoo.cfg for ZooKeeper
Configuring windows specific data directory for ZeeKeeper
Kafka comes with default configuration file & it has very good OS specific handling i.e. log.dirs=/tmp/kafka-logs but it will create that directory structure under C:/ directory seamlessly without failing.
It is also easy to quickly setup a Kafka Cluster with multiple Kafka Brokers on Windows with minor configurations. You can also create topics, publish/consume messages, verify topics/partitions etc. all from Windows Command Prompt. Here is a good detailed reference for the windows specific setup Kafka Setup on Windows OS | Basic Installation, Setup, Verification, Cluster Setup, Storage
I had configured Zookeeper and Apache Kafka on my Windows machine.
For Zookeeper, simply configure ZOOKEEPER_HOME and set the PATH also and then rename the zoo.sample.cfg to zoo.cfg and also change the path for dataDir in the zoo.cfg
For Apache Kafka, simply unzip and start the server from bin folder and start the Producer/Consumer from windows folder under the Kafka directory
I'm trying to install Kafka message queue on Windows for testing purposes (not for production).
I found this article on how to install Apache Kafka 0.8 on Windows: http://janschulte.wordpress.com/2013/10/13/apache-kafka-0-8-on-windows/
It's a good article, but it's outdated, unfortunately.
Does someone know a way to achieve that?
Ok, it's finally not complicated :)
The only steps are:
Edit (2021/10) Latest versions of Kafka end up with an AccessDeniedException upon startup, and even when successfully started, throw the same error during log rotation. Kafka utilizes Java methods designed for Unix filesystem libraries, so installing Kafka in WSL2 is recommended (link to Confluent blog, but steps also applicable to Apache Kafka direct installation).
Old answer
Download Kafka and uncompress it somewhere nice (let's say C:/Kafka)
Install Cygwin
If using Cygwin, edit \bin\kafka-run-class.sh and at the end of the file, change
`exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"`
to
``exec java $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp `cygpath -wp $CLASSPATH` $KAFKA_OPTS "$#"``
In Environment Variables, Add java to your Path System Variable:
That's it.. you can now run ZooKeeper and Kafka servers and start playing with topics and stuff..
These are the steps I followed to run kafka on Windows
Install Zookeeper first (I downloaded v3.3.6) zookeeper-3.3.6.tar.gz
Extract Zookeeper and run this command in powershell/cmd \zookeeper-3.3.6\bin> .\zkServer.cmd Now this should up a Zookeeper instance on localhost:2181
Download Kafka binary version (I downloaded v0.10.0.1)kafka_2.10-0.10.0.1.tgz
Extract Kafka, time to modify some configs
Inside Kafka extraction you can find .\config\server.properties
In .\config\server.properties replace log.dirs=c:/kafka/kafka-logs
Note: Make sure to create those folders in relevant paths
Happy news: Now Kafka ships with windows .bat scripts, You can find these files inside ./bin/windows folder
Start powershell/cmd and run this command to start Kafka broker .\bin\windows\kafka-server-start.bat .\config\server.properties
DONE!, Now you have a running Zookeeper instance and a Kafka broker.
You will probably find that Kafka comes with Windows .bat files under the bin folder to help you run Kafka under windows. However, those bat files are not updated since version 0.8.0 and some of Kafka's logic has changed (regarding topics, etc.).
I have tried two ways to solve this problem. One is to use Cygwin or MinGW to simulate Linux environment and run the Linux shell script, but there would be tons of other problems related to path names. The other solution, which is simpler and come with less trouble, is to use the corrected version of Windows bat files.
Please refer to this blog post.
Download any binary version(I've downloaded kafka_2.11-0.10.2.0) of Kafka from https://kafka.apache.org/downloads
Extract it to any folder(I've extracted to "C:\Kafka")
Open Command Prompt
Go to the folder that you extract Kafka(C:\Kafka\kafka_2.11-0.10.2.0)
Run this command to start Zookeeper
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Run this command to start Kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties
Now it works!
In version 0.9.0. there are scripts in "bin" folder for windows. You can use them.
Running From A windows Shell
As of April 2019, downloading Kafka from their website worked on Windows almost right out of the box
Downloading and using the windows version .bat files is described here:
https://kafka.apache.org/quickstart
I ran into two problems when I did this:
1) JAVA_HOME was set to an unsupported JDK which led to this kind of error
Exception in thread "main" java.lang.VerifyError: Uninitialized object
exists on backward branch 209
Replacing with JDK 11 solved the problem.
2) The JAVA_HOME must not contain spaces which caused a 'cannot find the specified file' error. To fix this I used a shortened path like set JAVA_HOME=C:\Progra~1\Java\jdk-11.0.1
Running From Cygwin
If instead of the .bat files, you want to run the .sh files from cygwin, there is actually quite a bit that needs to be done, and even after that there may be problems that come up later. I can't really recommend this option, but I do use it as it's pretty convenient for some purposes.
If your JAVA_HOME path contains a space e.g. "C:\Program Files\Java\Jdk..." you will see something like this:
bin/kafka-run-class.sh: line 305: exec: C:\Program: not found
One solution is to copy the jdk to a path without spaces, and change the Java home accordingly.
If you don't want to change the JDK location, you can change the cygwin env variable as follows:
JAVA_HOME="/cygdrive/c/Program Files/Java/jdk-11.0.1"
and change the line
exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
to
exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
There was another problem with the log parameters in kafka-run-class.sh and I had to replace the line
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
with
KAFKA_GC_LOG_OPTS="-Xlog:gc*"
And even after all these changes I occasionally run into problems of Kafka shutting down because of an incompatible windows style path in the logs directories as described here:
Kafka 1.0 stops with FATAL SHUTDOWN error. Logs directory failed
In short, you may be better off running the Kafka .bat scripts from the windows directory...
With Chocolatey and Powershell:
chocolatey.exe install kafka -> installs Kafka into C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ - your version might differ of course
add C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ to your powershell path using notepad $PROFILE
After restarting powershell kafka-console-consumer should work as a command.
You also need to install ssl.properties with keystore and truststore. Put them e.g. into C:\ProgramData\kafka and put that into ssl.properties (please note the escaping of the backslashes):
security.protocol=SSL
ssl.truststore.location=C:\\ProgramData\\kafka\\kafka-truststore.jks
ssl.truststore.password=PASSWORD
ssl.keystore.location=C:\\ProgramData\\kafka\\kafka-keystore.jks
ssl.keystore.password=PASSWORD
ssl.key.password=PASSWORD
client.id=console-test
Now things like kafka-consumer-groups --bootstrap-server SERVERNAME:9094 --command-config C:\ProgramData\kafka\ssl.properties --list should first pop up a firewall warning which you'd need to accept and then output a list of groups.
Kafka & Zookeeper have inbuilt Windows specific scripts as verified in the latest version. As mentioned in other other answers, the directory for windows scripts will be under kafka_2.12-2.4.0\bin\windows for Kafka. Zookeeper scripts are under /bin only but ending with "*.cmd"
There will be few minor additional steps you will have to do with ZooKeeper & Kafka like
Creating zoo.cfg for ZooKeeper
Configuring windows specific data directory for ZeeKeeper
Kafka comes with default configuration file & it has very good OS specific handling i.e. log.dirs=/tmp/kafka-logs but it will create that directory structure under C:/ directory seamlessly without failing.
It is also easy to quickly setup a Kafka Cluster with multiple Kafka Brokers on Windows with minor configurations. You can also create topics, publish/consume messages, verify topics/partitions etc. all from Windows Command Prompt. Here is a good detailed reference for the windows specific setup Kafka Setup on Windows OS | Basic Installation, Setup, Verification, Cluster Setup, Storage
I had configured Zookeeper and Apache Kafka on my Windows machine.
For Zookeeper, simply configure ZOOKEEPER_HOME and set the PATH also and then rename the zoo.sample.cfg to zoo.cfg and also change the path for dataDir in the zoo.cfg
For Apache Kafka, simply unzip and start the server from bin folder and start the Producer/Consumer from windows folder under the Kafka directory
I installed confluent community platform Manual Install using Systemd on Ubuntu and Debian
I can start:
sudo systemctl start confluent-zookeeper
sudo systemctl start confluent-kafka
...
but where is my confluent path?
I do not have confluent cli
please help
I encountered same problem when setting up elastisearch for confluent. In my system /usr/bin acted as /bin and remaining set up was successfull . So try using /usr as path to confluent if following Manual Install using Systemd on Ubuntu and Debian.
You can manually install Confluent CLI:
curl -L https://cnfl.io/cli | sh -s -- -b /<path-to-directory>/bin
Run this script to install the Confluent CLI. This command creates a
bin directory in your designated location (<path-to-directory>/bin).
The location must be in your PATH (e.g. /usr/local/bin).
When i try to setup Apache Samza, i get nc command not found when starting zookeeper.
I'm running the command:
bin/grid bootstrap
i get
/c/..../hello-samza/bin/grid: line 170: nc: command not found
I'm writing the command in git bash console on windows 10 Education.
I'm new to samza and git bash in general. How can i fix this or is there a way to integrate nc commands in git bash basic commands?
or am i running the commands in the wrong console ?