I'm using testcontainers kafka image which is confluent cp-kafka, I need to amend kafka config using command:
./kafka-configs.sh --alter --entity-name <TOPIC_NAME> --entity-type topics --add-config message.timestamp.type=LogAppendTime --zookeeper <HOST>:<PORT>
I have an issue exucuting this command using KafkaContainer::execInContainer which yields no such file or directory.
I've looked at github image and confluent installation on ubuntu where it should be located, and tried various options with no success.
How to properly do this ?
./kafka-configs.sh assumes that kafka-configs.sh is available in a current directory, which might not be the case (try running cwd or ls -la instead and check the output).
But if you remove ./ before kafka-configs.sh, it should use the command from the PATH variable, which is supposed to be configured inside the Kafka image.
I am trying to create a topic in Kafka, I installed a fresh copy of Kafka by downloading the .tar from official Apache mirror site.
I used the tar -xvf command to unpack the bundle and started the server, which ran ok.
Now I am trying the command:
bin/kafka-create-topic.sh --zookeeper localhost:2181 --replica 1 --partition 1 --topic test
I tried to dig the problem and actually check if the file was present, somehow the result is negative:
You are in the bin directory so the command should be :
./kafka-create-topic.sh .....
and not
bin/kafka-create-topic.sh
maybe a copy & paste error from the documentation web site ;)
[UPDATE]
even if the directory was wrong, in this case the new Kafka version has changed the script name so the right command is :
kafka-topic.sh --create ...
Use the following command to create a topic in Kafka,
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topic_name
For further clarification, visit https://kafka.apache.org/quickstart
I have installed kafka and zookeeper on my local C:\ drive folder now i am trying to run the zookeeper and kafka server so i can create topics but its throwing below error, any idea what is going wrong here ?
kafka comand Line:
C:\kafka_2.11-0.10.0.0>.\bin\windows\kafka-server-start.bat .\config\server.pro
erties
'#' is not recognized as an internal or external command,
operable program or batch file.
The syntax of the command is incorrect.
Error: missing `server' JVM at `C:\Program Files (x86)\Java\jre8\bin\server\jvm
dll'.
Please install or use the JRE or JDK that contains these missing components.
zookeeper.config
tickTime=2000
initLimit=10
syncLimit=5
dataDir=\zookeeper-3.4.8
clientPort=2181
Have you set up ZOOKEEPER_HOME in system variable ?
Please follow this video
https://www.youtube.com/watch?v=OJKesEpO6ok
In the latest version kafka_2.12-3.1.0 (2022) after unzipping and setting the properties and logs. keep the Kafka folder on the C drive and always run the command prompt with 'run as administrator'.
You can configure zookeeper with below commands
The .bat file is for windows
Terminal 1 zookeeper
C:\kafka\bin\windows>zookeeper-server-start.bat
..\..\config\zookeeper.properties
Terminal 2 Server
C:\kafka\bin\windows>kafka-server-start.bat
..\..\config\server.properties
Terminal 3 Creating a Topic
C:\kafka\bin\windows>kafka-topics.bat --create --topic tutorialGB
--bootstrap-server localhost:9092
Created topic tutorialGB.
To checklist of topic created
C:\kafka\bin\windows>kafka-topics.bat --list --bootstrap-server
localhost:9092
tutorialGB
I'm trying out Kafka (0.8.2.1) in a VM, but am having trouble with it: though everything is fine while the machine remains on (even if I restart ZK/Kafka), if I reboot the machine (after gracefully shutting down ZK/Kafka) it seems all Kafka topics go lost.
I'm probably missing something basic here, since this is probably not supposed to happen. What might it be?
cd /vagrant/kafka_2.11-0.8.2.1
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 10 --topic foo
bin/kafka-topics.sh --list --zookeeper localhost:2181
# foo
# ^C then resume ZooKeeper, Kafka, or both
bin/kafka-topics.sh --list --zookeeper localhost:2181
# foo
# ^C both, reboot machine, boot ZK/Kafka again
bin/kafka-topics.sh --list --zookeeper localhost:2181
# no topics
Looks like the default location for logs is in the /tmp directory which gets wiped on reboot. Change that location in the config to a more permanent location.
Go to kafka installation folder > config> server.properties
search for log.dirs in that file, change path from /tmp/logs to local directory. Restart kafka server and you will see topics created will be saved in that local folder we have changed in config file.
This happens because the tmp folder get cleared out on reboot.
To fix this issue, do the following.
Go to you kafka installation directory and search for the file server.properties. You should see a section as below
A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs
Change the logs.dir to something more local or a custom dir like this.
log.dirs=/Users/xxx/yyy/software/confluent-5.3.1/mydata
Reboot your kafka cluster for the changes to take effect.
Reboot your system and you will see the Topics are still present.
I just downloaded Kafka 2.8.0 from Apache website, and I am trying to setup using the instructions given on the website. But when I try to start zookeper server, I am getting below error:
Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain
My environment is Windows 7 64 bit. I tried to follow below e-mail chain: Apache Email Chain . But still it's having same issue. Can anyone guide me in this? As I am very new to this and couldn't find many information on Google/Apache Kafka email chain.
Run these commands from your Kafka root folder:
cd bin\windows
Then run Zookeper server:
zookeeper-server-start.bat ..\..\config\zookeeper.properties
Then run Kafka server:
kafka-server-start.bat ..\..\config\server.properties
The gotcha here is to run the .bat files from the /bin/windows folder, so after you run your servers with the steps above and want to follow up with the tutorial, make sure you are running the correct batch files to create topics and whatnot, e.g.:
Create a topic:
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
List topics:
kafka-topics.bat --list --zookeeper localhost:2181
This problem happened because you have downloaded wrong version of Kafka. On the official page
you have selected Source download. If you will download the binary, everything will work as expected.
I also got the same problem in Windows platform due to keeping the kafka_2.12-2.3.0 folder in the Program Files. I changed the Kafka directory from Program Files to C:\ drive root directory. It worked fine after executing the following commands:
First, go to the Kafka root folder:
cd C:\kafka_2.12-2.3.0
Run Zookeeper server:
bin\windows\zookeeper-server-start.bat config\zookeeper.properties
Then Run Kafka Server:
bin\windows\kafka-server-start.bat config\server.properties
Had the same error message even though I had the correct Kafka download for my Scala version.
It ended up I was running the non-windows sh file (needed to run the bat in the windows sub-folder) and was running from within the bin folder instead of the root. Moving up to the root Kafka folder and running from there appears to have worked.
C:\kafka_2.10-0.8.2.0>bin\windows\zookeeper-server-start.bat config\zookeeper.properties
I'm running on OS X and I found that having spaces anywhere in the path leading to my Kafka base directory led to this exact error message. Renaming things without spaces solved the problem with no other fix.
As an example, my Kafka install was sitting at /Users/me/Kafka Demo/kafka_2.11-0.9.0.1 which caused that error when I tried to start Zookeeper. All I did was rename things as /Users/me/KafkaDemo/kafka_2.11-0.9.0.1 and everything worked fine.
The solution for windows OS is to do the following:
Download the binary version instead of the source version from the official website of Apache Kafka.
Make sure the JAVA_HOME environment variable does not have a space in it (by default it is installed in Program files), follow the official installation documentation for this.
https://docs.oracle.com/cd/E39271_01/general.300/eid_getting_started/src/tgs_install_jdk.html.
Make sure the JRE path and KAFKA_HOME path do not have spaces in them as well.
Run the zookeeper before starting Kafka.
First Thing: Remove all blank space from the folder name.
Second Thing: Use Powershell
Wrong Path:
d:\kafka node> .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Correct Path
d:\kafka_node> .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Thanks & Regards
Jaiswar Vipin Kumar R.
Just download the binary and follow the steps mentioned in below link. http://kafka.apache.org/07/quickstart.html
if you are facing errors like
Error: Could not find or load main class org.apache.zookeeper.server.quorum.Quorumpeermain
copy zookeeper.properties from config folder to bin folder and bin/windows folder.
Now start the like
server : run
zookeeper-server-start.bat zookeeper.properties
from bin or bin/windows(if you are in window machine). It should work.
If you are still facing problem, below link would be helpful
http://blog.anilot.tk/2014/08/06/Set-Up-Kafka-On-Windows/
Now try staring the Kafka server.
use the below command if you are using windows: Goto bin/windows and then run
kafka-server-start.bat ../../config/server.properties
if you are trying from shell prompt, goto bin and run below command
kafka-server-start.bat ../config/server.properties
**Make sure server.properties is present in the config folder
Yes, I too faced the same problem and tried this way which was suggested in some comments above,
First I had my Kafka in C:\Program Files, Then I moved the kafka_2.11-2.1.0 folder to C:\ directory and tried the following command and it worked
.\bin\windows\kafka-server-start.bat config\server.properties
P.S - ran the zookeper server before starting kafka
I downloaded latest stable Kafka from Apache and instruction mentioned by Matheus Felipe worked as expected.
I downloaded binary option - Scala 2.11 # http://kafka.apache.org/downloads
I had same problem used zookeeper-3.4.12.
bin directory do not has .bat files and windows directory.
Just exist .com and .sh files.
.com contain operation zookeeper execute.
As you can see, that has to find %JAVA%.
So I just set JAVA=[my java.exe path]. And it works.
I am using below mention command in windows environment.
Confluent_home\bin\windows> zookeeper-server-start.bat ..\..\etc\kafka\zookeeper.properties
if still you are getting the same error, then please verify you folder structure, Is there any space between folder name if yes then replace with either "_" or removed completely.
I was also facing the same error.
Its bit weird. But replacing the root folder name to Kafka resolved my issue. It might help one or others.
Kafka docker for Windows OS
Download the Zookeeper from page https://zookeeper.apache.org/releases.html (3.5.8)
Download the Kafka from page https://kafka.apache.org/downloads (kafka_2.13-2.6.0)
Start Zookeeper
Clone file /conf/zoo_sample.cfg to /conf/zoo.cfg
Configure Zookeeper Admin (in /conf/zoo.cfg file) netty server port by adding this line admin.serverPort=8099 at the end of the zoo.cfg file. This is to avoid the App default port 8080 conflict later.
Run command to start ./zkServer.cmd
Stop Zookeeper
Run command to stop: ./zkServer.cmd stop
Start Kafka
Configure Kafka (in /config/zookeeper.properties file) with the Zookeeper admin port (e.g admin.serverPort=8099)
Run command to start ./kafka-server-start.bat ../../config/server.properties
Stop Kafka
Run command to stop: ./kafka-server-stop.bat
Topics
Run command to create a topic: ./kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test_1
Run command to list all topics: ./kafka-topics.bat --list --zookeeper localhost:2181
Test Topics
Run command to verify if a topic was created: ./kafka-topics.bat --describe --topic test_1 --bootstrap-server localhost:9092
Open terminal to create an event to the topic: ./kafka-console-producer.bat --topic test_1 --bootstrap-server localhost:9092
Open terminal to read the event from the topic: ./kafka-console-consumer.bat --topic test_1 --from-beginning --bootstrap-server localhost:9092
Practical
Write some events into the Topic (at kafka-console-producer terminal): This is an event
Will see the events printed out from reading by the consumer (at kafka-console-consumer terminal) => This is an event
Happy coding!
I was facing the same issue.
Change the JAVA_HOME system variable to the shortened pathname so it contains no space.
Instead of: C:\Program Files\Java\jdk1.8.0_241
Set: C:\Progra~1\Java\jdk1.8.0_241
This error can happen if any kafka .bat file references a malformed JAVA_HOME variable that contains space.
Setting the shortened pathname to the variable avoids the change to another path that contains no space and solves the problem.
I used to meet the issue by running below cmd via git bash tool.
bin/zookeeper-server-start.sh config/zookeeper.properties
Solution: I switched to cygwin tool (http://cygwin.com/) to run the same cmd and I did not see the issue any more.