I'm starting with Zookeeper and Kafka - after executing zookeeper-shell.sh commands I'm still getting EndOfStreamException. So far it doesn't cause broker failures (still getting valid responses) but I would like to resolve it before it starts. I operate on single cluster with 1 Zookeeper and 3 Kafka brokers. Commands I used so far are simplest possible:
/bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids
/bin/zookeeper-shell.sh localhost:2181 get /brokers/ids/{id}
Zookeeper command
Zookeeper logs
You're running a non-interactive shell command, then it exits successfully with code 0 and ends the session.
How to start Zookeeper and Kafka broker on remote target with Ansible Playbook.
Following commands work fine locally.
Start Zookeeper:
cd /opt/kafka
./bin/zookeeper-server-start.sh config/zookeeper.properties
Start Kafka Broker: (setting 2 env variables while starting)
JMX_PORT="5555" KAFKA_HEAP_OPTS="-Xms500M -Xmx550M" ./bin/kafka-server-start.sh config/server.properties
I'd recommend creating a systemd service file that you can start/stop appropriately using the Ansible service module.
Confluent offers cp-ansible repo that you can either copy from, or use directly to install a Kafka + Zookeeper cluster (and more)
You can use either the modules shell, command
Using command, I think something like that should work:
- name: Start Zookeeper
command:
cmd: ./bin/zookeeper-server-start.sh config/zookeeper.properties
chdir: /opt/kafka
- name: Start Kafka broker
command:
cmd: 'JMX_PORT="5555" KAFKA_HEAP_OPTS="-Xms500M -Xmx550M" ./bin/kafka-server-start.sh config/server.properties'
chdir: /opt/kafka
My project needs to kafka process. therefore, I downloded kafka_2.11-2.0.0 in my local machine. As you know, there are a lot of scripts for make this in bin directory, such as:
kafka-run-class.sh
kafka-topics.sh
kafka-server-start.sh
Which scripts are required to standing up Kafka ?
Unzip the tar.
Then issue these commands from inside the unzipped folder (you need to start zookeeper server first, then start kafka server):
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
This will start Apache Kafka with default configurations.
I have installed kafka and zookeeper on my local C:\ drive folder now i am trying to run the zookeeper and kafka server so i can create topics but its throwing below error, any idea what is going wrong here ?
kafka comand Line:
C:\kafka_2.11-0.10.0.0>.\bin\windows\kafka-server-start.bat .\config\server.pro
erties
'#' is not recognized as an internal or external command,
operable program or batch file.
The syntax of the command is incorrect.
Error: missing `server' JVM at `C:\Program Files (x86)\Java\jre8\bin\server\jvm
dll'.
Please install or use the JRE or JDK that contains these missing components.
zookeeper.config
tickTime=2000
initLimit=10
syncLimit=5
dataDir=\zookeeper-3.4.8
clientPort=2181
Have you set up ZOOKEEPER_HOME in system variable ?
Please follow this video
https://www.youtube.com/watch?v=OJKesEpO6ok
In the latest version kafka_2.12-3.1.0 (2022) after unzipping and setting the properties and logs. keep the Kafka folder on the C drive and always run the command prompt with 'run as administrator'.
You can configure zookeeper with below commands
The .bat file is for windows
Terminal 1 zookeeper
C:\kafka\bin\windows>zookeeper-server-start.bat
..\..\config\zookeeper.properties
Terminal 2 Server
C:\kafka\bin\windows>kafka-server-start.bat
..\..\config\server.properties
Terminal 3 Creating a Topic
C:\kafka\bin\windows>kafka-topics.bat --create --topic tutorialGB
--bootstrap-server localhost:9092
Created topic tutorialGB.
To checklist of topic created
C:\kafka\bin\windows>kafka-topics.bat --list --bootstrap-server
localhost:9092
tutorialGB
I just downloaded Kafka 2.8.0 from Apache website, and I am trying to setup using the instructions given on the website. But when I try to start zookeper server, I am getting below error:
Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain
My environment is Windows 7 64 bit. I tried to follow below e-mail chain: Apache Email Chain . But still it's having same issue. Can anyone guide me in this? As I am very new to this and couldn't find many information on Google/Apache Kafka email chain.
Run these commands from your Kafka root folder:
cd bin\windows
Then run Zookeper server:
zookeeper-server-start.bat ..\..\config\zookeeper.properties
Then run Kafka server:
kafka-server-start.bat ..\..\config\server.properties
The gotcha here is to run the .bat files from the /bin/windows folder, so after you run your servers with the steps above and want to follow up with the tutorial, make sure you are running the correct batch files to create topics and whatnot, e.g.:
Create a topic:
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
List topics:
kafka-topics.bat --list --zookeeper localhost:2181
This problem happened because you have downloaded wrong version of Kafka. On the official page
you have selected Source download. If you will download the binary, everything will work as expected.
I also got the same problem in Windows platform due to keeping the kafka_2.12-2.3.0 folder in the Program Files. I changed the Kafka directory from Program Files to C:\ drive root directory. It worked fine after executing the following commands:
First, go to the Kafka root folder:
cd C:\kafka_2.12-2.3.0
Run Zookeeper server:
bin\windows\zookeeper-server-start.bat config\zookeeper.properties
Then Run Kafka Server:
bin\windows\kafka-server-start.bat config\server.properties
Had the same error message even though I had the correct Kafka download for my Scala version.
It ended up I was running the non-windows sh file (needed to run the bat in the windows sub-folder) and was running from within the bin folder instead of the root. Moving up to the root Kafka folder and running from there appears to have worked.
C:\kafka_2.10-0.8.2.0>bin\windows\zookeeper-server-start.bat config\zookeeper.properties
I'm running on OS X and I found that having spaces anywhere in the path leading to my Kafka base directory led to this exact error message. Renaming things without spaces solved the problem with no other fix.
As an example, my Kafka install was sitting at /Users/me/Kafka Demo/kafka_2.11-0.9.0.1 which caused that error when I tried to start Zookeeper. All I did was rename things as /Users/me/KafkaDemo/kafka_2.11-0.9.0.1 and everything worked fine.
The solution for windows OS is to do the following:
Download the binary version instead of the source version from the official website of Apache Kafka.
Make sure the JAVA_HOME environment variable does not have a space in it (by default it is installed in Program files), follow the official installation documentation for this.
https://docs.oracle.com/cd/E39271_01/general.300/eid_getting_started/src/tgs_install_jdk.html.
Make sure the JRE path and KAFKA_HOME path do not have spaces in them as well.
Run the zookeeper before starting Kafka.
First Thing: Remove all blank space from the folder name.
Second Thing: Use Powershell
Wrong Path:
d:\kafka node> .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Correct Path
d:\kafka_node> .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Thanks & Regards
Jaiswar Vipin Kumar R.
Just download the binary and follow the steps mentioned in below link. http://kafka.apache.org/07/quickstart.html
if you are facing errors like
Error: Could not find or load main class org.apache.zookeeper.server.quorum.Quorumpeermain
copy zookeeper.properties from config folder to bin folder and bin/windows folder.
Now start the like
server : run
zookeeper-server-start.bat zookeeper.properties
from bin or bin/windows(if you are in window machine). It should work.
If you are still facing problem, below link would be helpful
http://blog.anilot.tk/2014/08/06/Set-Up-Kafka-On-Windows/
Now try staring the Kafka server.
use the below command if you are using windows: Goto bin/windows and then run
kafka-server-start.bat ../../config/server.properties
if you are trying from shell prompt, goto bin and run below command
kafka-server-start.bat ../config/server.properties
**Make sure server.properties is present in the config folder
Yes, I too faced the same problem and tried this way which was suggested in some comments above,
First I had my Kafka in C:\Program Files, Then I moved the kafka_2.11-2.1.0 folder to C:\ directory and tried the following command and it worked
.\bin\windows\kafka-server-start.bat config\server.properties
P.S - ran the zookeper server before starting kafka
I downloaded latest stable Kafka from Apache and instruction mentioned by Matheus Felipe worked as expected.
I downloaded binary option - Scala 2.11 # http://kafka.apache.org/downloads
I had same problem used zookeeper-3.4.12.
bin directory do not has .bat files and windows directory.
Just exist .com and .sh files.
.com contain operation zookeeper execute.
As you can see, that has to find %JAVA%.
So I just set JAVA=[my java.exe path]. And it works.
I am using below mention command in windows environment.
Confluent_home\bin\windows> zookeeper-server-start.bat ..\..\etc\kafka\zookeeper.properties
if still you are getting the same error, then please verify you folder structure, Is there any space between folder name if yes then replace with either "_" or removed completely.
I was also facing the same error.
Its bit weird. But replacing the root folder name to Kafka resolved my issue. It might help one or others.
Kafka docker for Windows OS
Download the Zookeeper from page https://zookeeper.apache.org/releases.html (3.5.8)
Download the Kafka from page https://kafka.apache.org/downloads (kafka_2.13-2.6.0)
Start Zookeeper
Clone file /conf/zoo_sample.cfg to /conf/zoo.cfg
Configure Zookeeper Admin (in /conf/zoo.cfg file) netty server port by adding this line admin.serverPort=8099 at the end of the zoo.cfg file. This is to avoid the App default port 8080 conflict later.
Run command to start ./zkServer.cmd
Stop Zookeeper
Run command to stop: ./zkServer.cmd stop
Start Kafka
Configure Kafka (in /config/zookeeper.properties file) with the Zookeeper admin port (e.g admin.serverPort=8099)
Run command to start ./kafka-server-start.bat ../../config/server.properties
Stop Kafka
Run command to stop: ./kafka-server-stop.bat
Topics
Run command to create a topic: ./kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test_1
Run command to list all topics: ./kafka-topics.bat --list --zookeeper localhost:2181
Test Topics
Run command to verify if a topic was created: ./kafka-topics.bat --describe --topic test_1 --bootstrap-server localhost:9092
Open terminal to create an event to the topic: ./kafka-console-producer.bat --topic test_1 --bootstrap-server localhost:9092
Open terminal to read the event from the topic: ./kafka-console-consumer.bat --topic test_1 --from-beginning --bootstrap-server localhost:9092
Practical
Write some events into the Topic (at kafka-console-producer terminal): This is an event
Will see the events printed out from reading by the consumer (at kafka-console-consumer terminal) => This is an event
Happy coding!
I was facing the same issue.
Change the JAVA_HOME system variable to the shortened pathname so it contains no space.
Instead of: C:\Program Files\Java\jdk1.8.0_241
Set: C:\Progra~1\Java\jdk1.8.0_241
This error can happen if any kafka .bat file references a malformed JAVA_HOME variable that contains space.
Setting the shortened pathname to the variable avoids the change to another path that contains no space and solves the problem.
I used to meet the issue by running below cmd via git bash tool.
bin/zookeeper-server-start.sh config/zookeeper.properties
Solution: I switched to cygwin tool (http://cygwin.com/) to run the same cmd and I did not see the issue any more.