I am running Confluent's kafka-avro-console-consumer as described in the quickstart tutorial:
kafka-avro-console-consumer --topic test --zookeeper localhost:2181 --from-beginning
However, I have created a separate user kafka to run this command. I get the error message
mkdir: cannot create directory '/usr/bin/../logs': Permission denied
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /usr/bin/../logs/schema-registry.log (No such file or directory)
[...]
I would rather not give the kafka write access to the entire /usr/bin directory. Where exactly is Kafka trying to create the logs directory?
The kafka-avro-console-consumer respects the LOG_DIR environment variable so if you set it, the log file will be written to this directory instead.
I am trying to create a topic in Kafka, I installed a fresh copy of Kafka by downloading the .tar from official Apache mirror site.
I used the tar -xvf command to unpack the bundle and started the server, which ran ok.
Now I am trying the command:
bin/kafka-create-topic.sh --zookeeper localhost:2181 --replica 1 --partition 1 --topic test
I tried to dig the problem and actually check if the file was present, somehow the result is negative:
You are in the bin directory so the command should be :
./kafka-create-topic.sh .....
and not
bin/kafka-create-topic.sh
maybe a copy & paste error from the documentation web site ;)
[UPDATE]
even if the directory was wrong, in this case the new Kafka version has changed the script name so the right command is :
kafka-topic.sh --create ...
Use the following command to create a topic in Kafka,
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topic_name
For further clarification, visit https://kafka.apache.org/quickstart
I want to clear steps of how to install zookeeper, kafka and storm in Ubuntu
It will guide through sequence of steps :
Kafka binary file already has built-in Zookeeper with it, so you don't need to download it separately. Download Kafka at the below link.
Download Kafka version 0.8.2.0 from http://kafka.apache.org/downloads.html
Release and Un-tar the zip file using the below commands
tar -xzf kafka_2.9.1-0.8.2.0.tgz
Go into the extracted folder
cd kafka_2.9.1-0.8.2.0
Start the Zookeeper Server(which listens on port 2181 for kafka server requests)
bin/zookeeper-server-start.sh config/zookeeper.properties
Now start the Kafka Server in a new terminal window
bin/kafka-server-start.sh config/server.properties
Now let us test if the zookeeper-kafka configuration is working.
Open a new terminal and Create a topic test:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
bin/kafka-topics.sh --list --zookeeper localhost:2181
Use a producer to Kafka's topic test:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message(You have to enter these messages)
This is another message
Use Kafka's Consumer to see the messages produced :
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
The above command should list all the messages you typed earlier. That's it. You successfully configured your Zookeeper-Kafka Single Broker.
To configure multil-broker use the link following, refer the official site kafka.apache.org
Now Let's install Apache Storm :
Download tar.gz file from mirrorShttp://mirrors.ibiblio.org/apache/storm/apache-storm-0.9.2-incubating/
Extract it : $tar xzvf apache-storm-0.9.2-incubating.tar.gz
Create a data directory
sudo mkdir /var/stormtmp
sudo chmod -R 777 /var/stormtmp
sudo gedit apache-storm-0.9.2-incubating/conf/storm.yaml
Edit the opened file such that it should have following properties set(JAVA_HOME path, you can use jdk7 or higher versions. Java must be installed in your system) :
storm.zookeeper.servers: - "localhost"
storm.zookeeper.port: 2181
nimbus.host: "localhost"
storm.local.dir: "/var/stormtmp"
java.library.path: "/usr/lib/jvm/java-7-openjdk-amd64"
supervisor.slots.ports:
-6700
-6701
-6702
-6703
worker.childopts: "-Xmx768m"
nimbus.childopts: "-Xmx512m"
supervisor.childopts: "-Xmx256m"
If everything goes fine, you are now ready with apache-zookeeper-kafka-storm, you can restart the system, That's it.
I'm trying out Kafka (0.8.2.1) in a VM, but am having trouble with it: though everything is fine while the machine remains on (even if I restart ZK/Kafka), if I reboot the machine (after gracefully shutting down ZK/Kafka) it seems all Kafka topics go lost.
I'm probably missing something basic here, since this is probably not supposed to happen. What might it be?
cd /vagrant/kafka_2.11-0.8.2.1
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 10 --topic foo
bin/kafka-topics.sh --list --zookeeper localhost:2181
# foo
# ^C then resume ZooKeeper, Kafka, or both
bin/kafka-topics.sh --list --zookeeper localhost:2181
# foo
# ^C both, reboot machine, boot ZK/Kafka again
bin/kafka-topics.sh --list --zookeeper localhost:2181
# no topics
Looks like the default location for logs is in the /tmp directory which gets wiped on reboot. Change that location in the config to a more permanent location.
Go to kafka installation folder > config> server.properties
search for log.dirs in that file, change path from /tmp/logs to local directory. Restart kafka server and you will see topics created will be saved in that local folder we have changed in config file.
This happens because the tmp folder get cleared out on reboot.
To fix this issue, do the following.
Go to you kafka installation directory and search for the file server.properties. You should see a section as below
A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs
Change the logs.dir to something more local or a custom dir like this.
log.dirs=/Users/xxx/yyy/software/confluent-5.3.1/mydata
Reboot your kafka cluster for the changes to take effect.
Reboot your system and you will see the Topics are still present.
I just downloaded Kafka 2.8.0 from Apache website, and I am trying to setup using the instructions given on the website. But when I try to start zookeper server, I am getting below error:
Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain
My environment is Windows 7 64 bit. I tried to follow below e-mail chain: Apache Email Chain . But still it's having same issue. Can anyone guide me in this? As I am very new to this and couldn't find many information on Google/Apache Kafka email chain.
Run these commands from your Kafka root folder:
cd bin\windows
Then run Zookeper server:
zookeeper-server-start.bat ..\..\config\zookeeper.properties
Then run Kafka server:
kafka-server-start.bat ..\..\config\server.properties
The gotcha here is to run the .bat files from the /bin/windows folder, so after you run your servers with the steps above and want to follow up with the tutorial, make sure you are running the correct batch files to create topics and whatnot, e.g.:
Create a topic:
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
List topics:
kafka-topics.bat --list --zookeeper localhost:2181
This problem happened because you have downloaded wrong version of Kafka. On the official page
you have selected Source download. If you will download the binary, everything will work as expected.
I also got the same problem in Windows platform due to keeping the kafka_2.12-2.3.0 folder in the Program Files. I changed the Kafka directory from Program Files to C:\ drive root directory. It worked fine after executing the following commands:
First, go to the Kafka root folder:
cd C:\kafka_2.12-2.3.0
Run Zookeeper server:
bin\windows\zookeeper-server-start.bat config\zookeeper.properties
Then Run Kafka Server:
bin\windows\kafka-server-start.bat config\server.properties
Had the same error message even though I had the correct Kafka download for my Scala version.
It ended up I was running the non-windows sh file (needed to run the bat in the windows sub-folder) and was running from within the bin folder instead of the root. Moving up to the root Kafka folder and running from there appears to have worked.
C:\kafka_2.10-0.8.2.0>bin\windows\zookeeper-server-start.bat config\zookeeper.properties
I'm running on OS X and I found that having spaces anywhere in the path leading to my Kafka base directory led to this exact error message. Renaming things without spaces solved the problem with no other fix.
As an example, my Kafka install was sitting at /Users/me/Kafka Demo/kafka_2.11-0.9.0.1 which caused that error when I tried to start Zookeeper. All I did was rename things as /Users/me/KafkaDemo/kafka_2.11-0.9.0.1 and everything worked fine.
The solution for windows OS is to do the following:
Download the binary version instead of the source version from the official website of Apache Kafka.
Make sure the JAVA_HOME environment variable does not have a space in it (by default it is installed in Program files), follow the official installation documentation for this.
https://docs.oracle.com/cd/E39271_01/general.300/eid_getting_started/src/tgs_install_jdk.html.
Make sure the JRE path and KAFKA_HOME path do not have spaces in them as well.
Run the zookeeper before starting Kafka.
First Thing: Remove all blank space from the folder name.
Second Thing: Use Powershell
Wrong Path:
d:\kafka node> .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Correct Path
d:\kafka_node> .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Thanks & Regards
Jaiswar Vipin Kumar R.
Just download the binary and follow the steps mentioned in below link. http://kafka.apache.org/07/quickstart.html
if you are facing errors like
Error: Could not find or load main class org.apache.zookeeper.server.quorum.Quorumpeermain
copy zookeeper.properties from config folder to bin folder and bin/windows folder.
Now start the like
server : run
zookeeper-server-start.bat zookeeper.properties
from bin or bin/windows(if you are in window machine). It should work.
If you are still facing problem, below link would be helpful
http://blog.anilot.tk/2014/08/06/Set-Up-Kafka-On-Windows/
Now try staring the Kafka server.
use the below command if you are using windows: Goto bin/windows and then run
kafka-server-start.bat ../../config/server.properties
if you are trying from shell prompt, goto bin and run below command
kafka-server-start.bat ../config/server.properties
**Make sure server.properties is present in the config folder
Yes, I too faced the same problem and tried this way which was suggested in some comments above,
First I had my Kafka in C:\Program Files, Then I moved the kafka_2.11-2.1.0 folder to C:\ directory and tried the following command and it worked
.\bin\windows\kafka-server-start.bat config\server.properties
P.S - ran the zookeper server before starting kafka
I downloaded latest stable Kafka from Apache and instruction mentioned by Matheus Felipe worked as expected.
I downloaded binary option - Scala 2.11 # http://kafka.apache.org/downloads
I had same problem used zookeeper-3.4.12.
bin directory do not has .bat files and windows directory.
Just exist .com and .sh files.
.com contain operation zookeeper execute.
As you can see, that has to find %JAVA%.
So I just set JAVA=[my java.exe path]. And it works.
I am using below mention command in windows environment.
Confluent_home\bin\windows> zookeeper-server-start.bat ..\..\etc\kafka\zookeeper.properties
if still you are getting the same error, then please verify you folder structure, Is there any space between folder name if yes then replace with either "_" or removed completely.
I was also facing the same error.
Its bit weird. But replacing the root folder name to Kafka resolved my issue. It might help one or others.
Kafka docker for Windows OS
Download the Zookeeper from page https://zookeeper.apache.org/releases.html (3.5.8)
Download the Kafka from page https://kafka.apache.org/downloads (kafka_2.13-2.6.0)
Start Zookeeper
Clone file /conf/zoo_sample.cfg to /conf/zoo.cfg
Configure Zookeeper Admin (in /conf/zoo.cfg file) netty server port by adding this line admin.serverPort=8099 at the end of the zoo.cfg file. This is to avoid the App default port 8080 conflict later.
Run command to start ./zkServer.cmd
Stop Zookeeper
Run command to stop: ./zkServer.cmd stop
Start Kafka
Configure Kafka (in /config/zookeeper.properties file) with the Zookeeper admin port (e.g admin.serverPort=8099)
Run command to start ./kafka-server-start.bat ../../config/server.properties
Stop Kafka
Run command to stop: ./kafka-server-stop.bat
Topics
Run command to create a topic: ./kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test_1
Run command to list all topics: ./kafka-topics.bat --list --zookeeper localhost:2181
Test Topics
Run command to verify if a topic was created: ./kafka-topics.bat --describe --topic test_1 --bootstrap-server localhost:9092
Open terminal to create an event to the topic: ./kafka-console-producer.bat --topic test_1 --bootstrap-server localhost:9092
Open terminal to read the event from the topic: ./kafka-console-consumer.bat --topic test_1 --from-beginning --bootstrap-server localhost:9092
Practical
Write some events into the Topic (at kafka-console-producer terminal): This is an event
Will see the events printed out from reading by the consumer (at kafka-console-consumer terminal) => This is an event
Happy coding!
I was facing the same issue.
Change the JAVA_HOME system variable to the shortened pathname so it contains no space.
Instead of: C:\Program Files\Java\jdk1.8.0_241
Set: C:\Progra~1\Java\jdk1.8.0_241
This error can happen if any kafka .bat file references a malformed JAVA_HOME variable that contains space.
Setting the shortened pathname to the variable avoids the change to another path that contains no space and solves the problem.
I used to meet the issue by running below cmd via git bash tool.
bin/zookeeper-server-start.sh config/zookeeper.properties
Solution: I switched to cygwin tool (http://cygwin.com/) to run the same cmd and I did not see the issue any more.