I´m using zookeeper 3.4.8 in single node and try to use kafka.
When I run this command:
zookeeper-server-start.sh /usr/local/kafka_2.9.2-0.8.2.2 /config/zookeeper.properties
I get the below error:
[2016-02-22 17:32:41,661] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.io.IOException: Unable to create data directory /var/zookeeper/version-2
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:85)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:104)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Any advice?
One reason could be the inappropriate path specified to zoo.config file.
A lot of solutions on the web specifies the path as ":\zookeeper-3.4.7\data".
Instead of the above mentioned format, specify the address as full path from your C: drive to the data folder. It worked for me. (Don't forget to put double slash \ instead of one in case you're on windows)
I got this problem for this setting on Windows PC:
dataDir=c:/data/zoo/
and thus this error:
2016-12-02 15:29:25,327 [myid:] - ERROR [main:ZooKeeperServerMain#64] - Unexpected exception, exiting abnormally
java.io.IOException: Unable to create data directory ??:\data\zoo\version-2
Problem was solved by changing (I have ZooKeeper on C disk unpackaged)
dataDir=/data/zoo/
Also run command line tool as Administrator if needed
I faced the same issue, and this works with
sudo bin/zookeeper-server-start.sh config/zookeeper.properties
You probably don't have permission to write to the directory log.dirs (see zookeeper.properties). Change the directory to a different one, change the permission setting of the current log.dirs directory or run Kafka as different user. You can use the command ls -l /var/zookeeper to see the current permissions and then chmod to change the permissions.
The reason is that zookeeper has no permission. Trying to use the administrator role to install it.
For window's machine
Solved : Use double slashes inside the path while defining the dataDir path
dataDir=E:\\tools\\zookeeperdata\\data
And in my windows 10 system, using zookeeper 3.4.10. the dataDir attribute should setting like :\\\\zookeeper\\\\data, not d:\zookeeper\data. it also can setting as linux file system separator(d:/zookeeper/data). then this problem should be ok. And in linux, I think it permission problem. also it can come across when dataDir is under driver C in windows system.
If you're running the zookeeper in the Windows 10 machine we need to specify the dataDir property something like this
"dataDir=C:\zookeeper-3.4.13\data"
In my windows 10 system, using zookeeper 3.4.13, the following example path is working:
"dataDir=C:\\dev\\tools\\zookeeper-3.4.13\\data"
You have to use double backslashes.
on zoo.cfg you need to change directory to above or anything similar:
dataDir=C:/zookeeper-3.4.14/zookeeper-3.4.14/data
For windows, set dataDir to full path where you have no access restrictions - with no quotes("")
dataDir=C:\\your-path\
dataDir=C:\\zk\tmp\
Note: I have observed the command to fail for some of the path(though full access) and running command prompt as administrator has solved it.
For windows the below too works:
dataDir=C:\\zookeeper-3.4.14\\zookeeper-3.4.14\\data
Related
I'm trying to install Kafka message queue on Windows for testing purposes (not for production).
I found this article on how to install Apache Kafka 0.8 on Windows: http://janschulte.wordpress.com/2013/10/13/apache-kafka-0-8-on-windows/
It's a good article, but it's outdated, unfortunately.
Does someone know a way to achieve that?
Ok, it's finally not complicated :)
The only steps are:
Edit (2021/10) Latest versions of Kafka end up with an AccessDeniedException upon startup, and even when successfully started, throw the same error during log rotation. Kafka utilizes Java methods designed for Unix filesystem libraries, so installing Kafka in WSL2 is recommended (link to Confluent blog, but steps also applicable to Apache Kafka direct installation).
Old answer
Download Kafka and uncompress it somewhere nice (let's say C:/Kafka)
Install Cygwin
If using Cygwin, edit \bin\kafka-run-class.sh and at the end of the file, change
`exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"`
to
``exec java $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp `cygpath -wp $CLASSPATH` $KAFKA_OPTS "$#"``
In Environment Variables, Add java to your Path System Variable:
That's it.. you can now run ZooKeeper and Kafka servers and start playing with topics and stuff..
These are the steps I followed to run kafka on Windows
Install Zookeeper first (I downloaded v3.3.6) zookeeper-3.3.6.tar.gz
Extract Zookeeper and run this command in powershell/cmd \zookeeper-3.3.6\bin> .\zkServer.cmd Now this should up a Zookeeper instance on localhost:2181
Download Kafka binary version (I downloaded v0.10.0.1)kafka_2.10-0.10.0.1.tgz
Extract Kafka, time to modify some configs
Inside Kafka extraction you can find .\config\server.properties
In .\config\server.properties replace log.dirs=c:/kafka/kafka-logs
Note: Make sure to create those folders in relevant paths
Happy news: Now Kafka ships with windows .bat scripts, You can find these files inside ./bin/windows folder
Start powershell/cmd and run this command to start Kafka broker .\bin\windows\kafka-server-start.bat .\config\server.properties
DONE!, Now you have a running Zookeeper instance and a Kafka broker.
You will probably find that Kafka comes with Windows .bat files under the bin folder to help you run Kafka under windows. However, those bat files are not updated since version 0.8.0 and some of Kafka's logic has changed (regarding topics, etc.).
I have tried two ways to solve this problem. One is to use Cygwin or MinGW to simulate Linux environment and run the Linux shell script, but there would be tons of other problems related to path names. The other solution, which is simpler and come with less trouble, is to use the corrected version of Windows bat files.
Please refer to this blog post.
Download any binary version(I've downloaded kafka_2.11-0.10.2.0) of Kafka from https://kafka.apache.org/downloads
Extract it to any folder(I've extracted to "C:\Kafka")
Open Command Prompt
Go to the folder that you extract Kafka(C:\Kafka\kafka_2.11-0.10.2.0)
Run this command to start Zookeeper
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Run this command to start Kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties
Now it works!
In version 0.9.0. there are scripts in "bin" folder for windows. You can use them.
Running From A windows Shell
As of April 2019, downloading Kafka from their website worked on Windows almost right out of the box
Downloading and using the windows version .bat files is described here:
https://kafka.apache.org/quickstart
I ran into two problems when I did this:
1) JAVA_HOME was set to an unsupported JDK which led to this kind of error
Exception in thread "main" java.lang.VerifyError: Uninitialized object
exists on backward branch 209
Replacing with JDK 11 solved the problem.
2) The JAVA_HOME must not contain spaces which caused a 'cannot find the specified file' error. To fix this I used a shortened path like set JAVA_HOME=C:\Progra~1\Java\jdk-11.0.1
Running From Cygwin
If instead of the .bat files, you want to run the .sh files from cygwin, there is actually quite a bit that needs to be done, and even after that there may be problems that come up later. I can't really recommend this option, but I do use it as it's pretty convenient for some purposes.
If your JAVA_HOME path contains a space e.g. "C:\Program Files\Java\Jdk..." you will see something like this:
bin/kafka-run-class.sh: line 305: exec: C:\Program: not found
One solution is to copy the jdk to a path without spaces, and change the Java home accordingly.
If you don't want to change the JDK location, you can change the cygwin env variable as follows:
JAVA_HOME="/cygdrive/c/Program Files/Java/jdk-11.0.1"
and change the line
exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
to
exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
There was another problem with the log parameters in kafka-run-class.sh and I had to replace the line
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
with
KAFKA_GC_LOG_OPTS="-Xlog:gc*"
And even after all these changes I occasionally run into problems of Kafka shutting down because of an incompatible windows style path in the logs directories as described here:
Kafka 1.0 stops with FATAL SHUTDOWN error. Logs directory failed
In short, you may be better off running the Kafka .bat scripts from the windows directory...
With Chocolatey and Powershell:
chocolatey.exe install kafka -> installs Kafka into C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ - your version might differ of course
add C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ to your powershell path using notepad $PROFILE
After restarting powershell kafka-console-consumer should work as a command.
You also need to install ssl.properties with keystore and truststore. Put them e.g. into C:\ProgramData\kafka and put that into ssl.properties (please note the escaping of the backslashes):
security.protocol=SSL
ssl.truststore.location=C:\\ProgramData\\kafka\\kafka-truststore.jks
ssl.truststore.password=PASSWORD
ssl.keystore.location=C:\\ProgramData\\kafka\\kafka-keystore.jks
ssl.keystore.password=PASSWORD
ssl.key.password=PASSWORD
client.id=console-test
Now things like kafka-consumer-groups --bootstrap-server SERVERNAME:9094 --command-config C:\ProgramData\kafka\ssl.properties --list should first pop up a firewall warning which you'd need to accept and then output a list of groups.
Kafka & Zookeeper have inbuilt Windows specific scripts as verified in the latest version. As mentioned in other other answers, the directory for windows scripts will be under kafka_2.12-2.4.0\bin\windows for Kafka. Zookeeper scripts are under /bin only but ending with "*.cmd"
There will be few minor additional steps you will have to do with ZooKeeper & Kafka like
Creating zoo.cfg for ZooKeeper
Configuring windows specific data directory for ZeeKeeper
Kafka comes with default configuration file & it has very good OS specific handling i.e. log.dirs=/tmp/kafka-logs but it will create that directory structure under C:/ directory seamlessly without failing.
It is also easy to quickly setup a Kafka Cluster with multiple Kafka Brokers on Windows with minor configurations. You can also create topics, publish/consume messages, verify topics/partitions etc. all from Windows Command Prompt. Here is a good detailed reference for the windows specific setup Kafka Setup on Windows OS | Basic Installation, Setup, Verification, Cluster Setup, Storage
I had configured Zookeeper and Apache Kafka on my Windows machine.
For Zookeeper, simply configure ZOOKEEPER_HOME and set the PATH also and then rename the zoo.sample.cfg to zoo.cfg and also change the path for dataDir in the zoo.cfg
For Apache Kafka, simply unzip and start the server from bin folder and start the Producer/Consumer from windows folder under the Kafka directory
I have been working with Kafka 2.4.0 (2.11) and yesterday I had to forcefully terminate the process for some unknown reason. Since then I haven't been unable to start Zookeeper due to the following error:
[2020-01-11 11:12:43,783] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.io.IOException: No snapshot found, but there are log entries. Something is broken!
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:222)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
And as soon as I searched for this problem I found issue ZOOKEEPER-3513 reported, which may or may not explain the problem. However, what I'm finding strange is that if I delete the Kafka/Zookeeper directory and download it again from scratch, the problem persists. Does anyone know how I can solve this?
Thank you for your help
Check for the tmp/zookeeper folder on the drive where you have kafka folder (lets say D:/), and delete the folder tmp, which will create automatically for you once run the zookeeper again.
Try changing your zookeeper data directory.
Your zookeeper data directory is defined in zookeeper.properties (I think the default is /tmp/zookeeper).
Perhaps you're not deleting the correct zookeeper directory?
I had the same problem, and this solution worked.
NOTE: I'm experimenting with Kafka, and not using it in production. I have no idea what else the above does, apart from fix this error...
I've faced the same issue with Zookeeper after updating from version 3.4.x to 3.5.6. As described here. I've:
added empty snapshot.0 file in data directory
added a property 'zookeeper.snapshot.trust.empty=true' to Zookeeper configuration file (default is zoo.cfg)
On windows ->
Go to the tmp folder where the zookeeper details are stored
and delete the existing log files
Directory path = d:\tmp\zookeeper\version-2
On Linux ->
Path = /tmp/zookeeper/version-2
And remove all the existing log files using rm -r log.1
The log files will be created automatically again and will resolve the issue.
Faced same issue in macOS.
Solution: In kafka dir, path cd /tmp/zookeeper/version-2 deleted the log.1 file. It worked for me
if you are on windows make sure you escape the location of the zookeeper temp directory.
dataDir=d:\tmp\zookeeper
Created a new dir for logs and configured the same path in zoo.cfg.
It worked:)
I use macOS and my solution was to delete everything in the dataDir, the default value should be /usr/local/var/lib/zookeeper.
For those who are using docker, I'll share my experience:
I've been running zookeeper confluentinc/cp-zookeeper:5.2.1 as it follows:
docker run \
--network kafka-net --name=zookeeper \
-e ALLOW_ANONYMOUS_LOGIN=yes \
-e ZOOKEEPER_CLIENT_PORT=2181 \
-v /tmp/zookeeper-data:/var/lib/zookeeper/data \
-v /tmp/zookeeper-txn-logs:/var/lib/zookeeper/log \
-p 2181:2182 confluentinc/cp-zookeeper:5.2.1
As expected, I can see a few files placed in /tmp/zookeeper-txn-logs and /tmp/zookeeper-data on host. After cleaning up /tmp/zookeeper-data and running again, I've got the error No snapshot found, but there are log entries.
In my case, I just had to purge the data on /tmp/zookeeper-txn-logs. For a dev/production environment, I'd recommend following the docs https://access.redhat.com/documentation/en-us/red_hat_amq/6.3/html/fabric_guide/ensemble-purgetxnlog
I am using Windows 32-bit machine and tried to start MongoDB service from Windows > services as shown below.
However, I am unable to start the MongoDB service from it and throws the following error.
When I try using cmd prompt, I am getting the following error:
Network Failed to connect to 127.0.0.1:27017, reason: errno:10061 No
connection could be made because the target machine actively refused
it.
Error: Couldn't connect to server 127.0.0.1:27017 <127.0.0.1>,
connection attempt failed.
I had same an issue.
Try to remove mongod.lock file from your Mongo data directory.
For example mine is "C:\Program Files\MongoDB\Data\mongod.lock" and after deleting file start the MongoDB service and it's work like charm.
In case someone else is running into this problem, just read your Log files and you will be able to find the problem, for me after trying to install it inside wamp directory when I run the MongoDB service it gave me the same error message, I went to the logs and find out that I was missing a directory inside my data directory which is called db, once I have created this directory the service run perfectly.
MongoDB uses a default folder to store its files. On Windows, the default location is C:\data\db.
Maybe that folder doesn´t exist. In that case just creat it or change the default location of Mongo service using the --dbpath command-line flag.
So I just had the same problem, running on Windows 10. The reason why MongoDB didn't start was because the path to the data and logs was not correctly set. This has already been pointed out, but my solution is different. Look in C:\Program Files\MongoDB\Server\4.0\bin (or wherever your mongoDB is installed). There is a config file called mongod.cfg. Check that
storage:
dbPath:
and
systemLog:
path:
Is set to what you want. In my case, it was using environment variables %MONGODBPATH% or similar that was not set by Windows. By default, the log and data should point to C:\Program Files\MongoDB\Server\4.0\data and C:\Program Files\MongoDB\Server\4.0\log\mongod.log respectively.
There was a npm: in last line of mongodb configuration file which is located in the installation folder in the bin\mongod.cfg
I commented out that line and started the service and it is working like charm.
I concluded this by running the mongodb service command from windows command line(cmd) and I got an error.
I ran this to spot the error:
C:\Program Files\MongoDB\Server\4.2\bin\mongod.exe --config "C:\Program Files\MongoDB\Server\4.2\bin\mongod.cfg" --service
mongod.lock deletion did not helped me, repair did not help either. In my case it was due to one of database happened to be corrupted, I moved all dbs to another directory and then copied them back one by one and re-starting mongodb service to figure out what db file is corrupted. It's definitely MongoDb bug
I had the same error message. Try to locate the mongodb log files and look at the last entries. My issue was clearly stated there, a missing directory :
2019-01-29T16:59:44.424+0100 I STORAGE [initandlisten] exception in
initAndListen: NonExistentPath: Data directory
C:\wamp64\bin\mongodb\mongodb-win32-x86_64-2008plus-ssl-3.6.10\data\db
not found., terminating
The advice of checking the log was what helped me. In this case:
The MongoDB service could not be started. A service specific error occurred: 100
turns out I had a problem with some databases created with WiredTiger while the mongod.cfg specified engine was: mmapv1
So I basically removed the content of the folder c:/data/db/ and then used the command net start MondoBD --repair and worked. Uffff it´s been 2 days.
I'm here a bit late, very late actually. But may it works something out for the ones facing this issue now. Mongodb configuration file in Windows OS is under 'C:\Program Files\MongoDB\Server\%YOUR MONGO VERSION%'.
I had changed this file and manipulated the bindip field, so I was getting the same error. It should be 127.0.0.1 or your machine's IP address which you can find it by 'ipconfig/all' command in cmd. So I fixed bindip and the service starts with no problems.
stuck on the same issue, but got the solution by hit and trial, just create a new folder for path "C:\data\db" then go to your command prompt and type 'mongod', your database server will start.
For me it was a port problem :
just search and kill the process using the port 27017
for linux : https://bobcares.com/blog/mongodb-error-code-48/
for windows : How do I kill the process currently using a port on localhost in Windows?
I have found out that Visual C++ Redistributable was missing in my Windows 7 Machine. After installing it worked.
For Windows 10 users
specify database location, if don't know create the below-mentioned directory and always use this
open cmd
mkdir C:\users\{username}\data
cd C:\users\{username}\data
mongod --dbpath .
start mongodb server
open cmd
mongod --dbpath C:\users\{username}\data
stop mongodb
open cmd
mongo
if server is running, run:
use admin
db.shutdownServer()
quit()
In my case, this happened because I did not stop MongoDB from docker. after I stopped the process the error was gone.
In my case, it was the docker with MongoDB running on the same port. So after I stopped the container, the service is then successfully starting.
I have installed zookeeper on my Linux server ubuntu 12.04 in some folder like abc/zookeeper/zkserver/bin/zkserver.sh start. this works fine and starts the server as expected but when i put this zkserver.sh file in /etc/init.d folder and copy the file in rc2.d folder for the reason to start-up zookeeper on system reboot. and when i run this command /etc/init.d/zkserver.sh start this is giving errors like:
JMX enabled by default
Using config: /etc/init.d/../etc/zookeeper/zoo.cfg
grep: /etc/init.d/../etc/zookeeper/zoo.cfg: No such file or directory
mkdir: cannot create directory `': No such file or directory
Starting zookeeper ... STARTED
The zkServer.sh is dependent on a certain directory structure and certain files being present. It is not supposed to be moved in isolation like that. It is also not supposed to be used as an init script.
Check if your zk download comes with the init script. Try looking at src/packages/rpm/init.d/zookeeper or similar, and use that one instead.
So I'm trying to to get started with Accumulo. I installed Hadoop and it runs w/o problems but when I try to start Zookeeper I get:
JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
-n Starting zookeeper ...
/opt/zookeeper/bin/zkServer.sh: line 103: /tmp/zookeeper/zookeeper_server.pid: No such file or directory
FAILED TO WRITE PID
I've looked around can't seem to find an answer.
I have had the same problem. In my case was useful to start Zookeeper and directly specify a configuration file:
/bin/zkServer.sh start conf/zoo.conf
I have never heard of zookeeper, but it could be a permissions issue trying to write the file zookeeper_server.pid or perhaps the directory /tmp/zookeeper/ doesn't exist and the shell script isn't accounting for that possibility. Check the permissions and existence of those directories.
zookeeper distributed with default conf, uses /tmp/zookeeper as dataDir for just example sake. It is suggested changing this value in /path/to/zookeeper/conf/zoo.cfg to /var/lib/zookeeper.
Creating /var/lib/zookeeper needs root access, so sudo is required. This directory when created will have following permissions.
ls -al /var/lib/zookeeper/
drwxr-xrwx 4 root wheel 128 May 9 14:03 .
When zookeeper is started without root permission, it cannot write to this directory. hence fails with error
... /usr/local/zookeeper/bin/zkServer.sh: line 169: /var/lib/zookeeper/zookeeper_server.pid: Permission denied
FAILED TO WRITE PID
You need to give write permissions to allow user starting zookeeper to write to /var/lib/zookeeper. In my case, as I am using it in local, I used the following command and it worked
sudo chmod o+w /var/lib/zookeeper