I am trying to install the zookeeper in my laptop. Using the file, which is bin.tar.gz 3.6.1 version.https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.6.1/apache-zookeeper-3.6.1-bin.tar.gz
Following the instructions which are:
make a folder logs in the main folder.
change the file name in conf folder, from Zoo_sample.cfg to zoo.cfg
change the dataDir path inside the zoo.cfg to where the logs situated.
put the bin path to the environment variables following the post here:https://medium.com/#shaaslam/installing-apache-zookeeper-on-windows-45eda303e835
Using Windows 10 with Bash, the feedback:
alex0#DESKTOP-AGJ32D1 MINGW64 /c/Tools/apache-zookeeper-3.6.1-bin/bin
$ ./zkServer.sh
ZooKeeper JMX enabled by default
Using config: C:\Tools\apache-zookeeper-3.6.1-bin\conf\zoo.cfg
grep: C:\Tools\apache-zookeeper-3.6.1-bin\conf\zoo.cfg: No such file or directory
grep: C:\Tools\apache-zookeeper-3.6.1-bin\conf\zoo.cfg: No such file or directory
mkdir: cannot create directory ‘’: No such file or directory
Usage: ./zkServer.sh [--config <conf-dir>] {start|start-foreground|stop|version|restart|status|print-cmd}
I am sure that the name of cfg file is properly changed, and the dataDir path is set as well. What could the problem I've made? Any help is highly appreciated.
I've figured it out. It is simply the file name issue.
When I changed it into zoo.cfg, the name of the file is zoo.cfg. Therefore, just change it to zoo should solve the problem.
when you setup zookeeper for the first time at your linux system, the default configuration file that have been created is zoo_sample.cfg then when your run the server or the zkServer.sh file, the file came by default to check on the config file named as zoo.cfg, so there the source of the issue.
zookeeper config dir
in my case what I did is that I renamed the file in the zookeeper dir from zoo_simple.cfg to zoo.cfg by using the following command on terminal.
~$ mv zoo_sample.cfg zoo.cfg
in case you still facing an error of zookeeper failed to start as you see in this error example
Try to run Zookeeper with the config file full path, here's the command:
~$ sudo ./zkServer.sh start /[path_to_zookeeper_config_dir]/zoo.cfg
then the server will start
Below the steps I did to get this issue :
Launch ZooKeeper
Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties
And at the second step the error happens :
ERROR Fatal error during KafkaServer startup. Prepare to shutdown
(kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID
Reu8ClK3TTywPiNLIQIm1w doesn't match stored clusterId
Some(BaPSk1bCSsKFxQQ4717R6Q) in meta.properties. The broker is trying
to join the wrong cluster. Configured zookeeper.connect may be wrong.
at kafka.server.KafkaServer.startup(KafkaServer.scala:220)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
When I trigger .\bin\windows\kafka-server-start.bat .\config\server.properties zookeeper console returns :
INFO [SyncThread:0:FileTxnLog#216] - Creating new log file: log.1
How to fix this issue to get kafka running ?
Edit You can access to the proper issue on the right site (serverfault) here
Edit Here is the Answer
I managed to Solve this issue with the following steps :
Just Delete all the log/Data file created (or generated) into
zookeeper and kafka.
Run Zookeper
Run Kafka
[Since this post is open again I post my answer there so you got all on the same post]
** 1. The easiest solution is to remove kafka logs and start again.
** 2. But the root cause is Kafka saved failed cluster ID in meta.properties.**
Try to delete kafka-logs/meta.properties from your tmp folder, which is located in C:/tmp folder by default on windows, and /tmp/kafka-logs on Linux
if kafka is running in docker containers, the log path may be specified by volume config in the docker-compose - see docs.docker.com/compose/compose-file/compose-file-v2/#volumes -- Chris Halcrow
** 3. How to find Kafka log path:**
Open server server.properties file which is located in your kafka folder kafka_2.11-2.4.0\config\server.properties (considering your version of kafka, folder name could be kafka_<kafka_version>):
Then search for entry log.dirs to check where logs locate
log.dirs=/tmp/kafka-logs
For mac, the following steps are needed.
Stop kafka service: brew services stop kafka
open kafka server.properties file: vim /usr/local/etc/kafka/server.properties
find value of log.dirs in this file. For me, it is /usr/local/var/lib/kafka-logs
delete path-to-log.dirs/meta.properties file
start kafka service brew services start kafka
No need to delete the log/data files on Kafka. Check the Kafka error logs and find the new cluster id. Update the meta.properties file with cluster-ID then restart the Kafka.
/home/kafka/logs/meta.properties
To resolve this issue permanently follow below.
Check your zookeeper.properties file and look for dataDirpath and change the path tmp location to any other location which should not be removed after server restart.
/home/kafka/kafka/config/zookeeper.properties
Copy the zookeeper folder and file to the new(below or non tmp) location then restart the zookeeper and Kafka.
cp -r /tmp/zookeeper /home/kafka/zookeeper
Now server restart won’t affect the Kafka startup.
If you use Embedded Kafka with Testcontainers in your Java project like myself, then simply delete your build/kafka folder and Bob's your uncle.
The mentioned meta.properties can be found under build/kafka/out/embedded-kafka.
I had some old volumes lingering around. I checked the volumes like this:
docker volume list
And pruned old volumes:
docker volume prune
And also removed the ones that were kafka:
example:
docker volume rm test_kafka
I deleted the following directories :-
a.) logs directory from kafka-server's configured location i.e. log.dir property path.
b.) tmp directory from kafka broker's location.
log.dirs=../tmp/kafka-logs-1
I was using docker-compose to re-set up Kafka on a Linux server, with a known, working docker-compose.config that sets up a number of Kafka components (broker, zookeeper, connect, rest proxy), and I was getting the issue described in the OP. I fixed this for my dev server instance by doing the following
docker-compose down
backup kafka-logs directory using cp kafka-logs -r kafka-logs-bak
delete the kafka-logs/meta.properties file
docker-compose up -d
Note for users of docker-compose:
My log files weren't in the default location (/tmp/kafka-logs). If you're running Kafka in Docker containers, the log path can be specified by volume config in the docker-compose e.g.
volumes:
- ./kafka-logs:/tmp/kafka-logs
This is specifying SOURCE:TARGET. ./kafka-logs is the source (i.e. a directory named kafka-logs, in the same directory as the docker-compose file). This is then targeted to /tmp/kafka-logs as the mounted volume within the kafka container). So the logs can either be deleted from the source folder on the host machine, or by deleting them from the mounted volume after doing a docker exec into the kafka container.
see https://docs.docker.com/compose/compose-file/compose-file-v2/#volumes
For me, meta.properties was in /usr/local/var/lib/kafka-logs
By removing it, the kafka started working.
I also deleted all the content of the folder containing all data generated by Kafka. I could find the folder in my .yml file:
kafka:
image: confluentinc/cp-kafka:7.0.0
ports:
- '9092:9092'
environment:
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_BROKER_ID: 1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: "true"
volumes:
- ./kafka-data/data:/var/lib/kafka/data
depends_on:
- zookeeper
networks:
- default
Under volumes: stays the location. So, in my case I deleted all files of the data folder located under kafka-data.
I've tried deleting the meta.properties file but didn't work.
In my case, it's solved by deleting legacy docker images.
But the problem with this is that deletes all previous data.
So be careful if you want to keep the old data this is not the right solution for you.
docker rm $(docker ps -q -f 'status=exited')
docker rmi $(docker images -q -f "dangling=true")
My project needs to kafka process. therefore, I downloded kafka_2.11-2.0.0 in my local machine. As you know, there are a lot of scripts for make this in bin directory, such as:
kafka-run-class.sh
kafka-topics.sh
kafka-server-start.sh
Which scripts are required to standing up Kafka ?
Unzip the tar.
Then issue these commands from inside the unzipped folder (you need to start zookeeper server first, then start kafka server):
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
This will start Apache Kafka with default configurations.
I have installed zookeeper on my Linux server ubuntu 12.04 in some folder like abc/zookeeper/zkserver/bin/zkserver.sh start. this works fine and starts the server as expected but when i put this zkserver.sh file in /etc/init.d folder and copy the file in rc2.d folder for the reason to start-up zookeeper on system reboot. and when i run this command /etc/init.d/zkserver.sh start this is giving errors like:
JMX enabled by default
Using config: /etc/init.d/../etc/zookeeper/zoo.cfg
grep: /etc/init.d/../etc/zookeeper/zoo.cfg: No such file or directory
mkdir: cannot create directory `': No such file or directory
Starting zookeeper ... STARTED
The zkServer.sh is dependent on a certain directory structure and certain files being present. It is not supposed to be moved in isolation like that. It is also not supposed to be used as an init script.
Check if your zk download comes with the init script. Try looking at src/packages/rpm/init.d/zookeeper or similar, and use that one instead.
So I'm trying to to get started with Accumulo. I installed Hadoop and it runs w/o problems but when I try to start Zookeeper I get:
JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
-n Starting zookeeper ...
/opt/zookeeper/bin/zkServer.sh: line 103: /tmp/zookeeper/zookeeper_server.pid: No such file or directory
FAILED TO WRITE PID
I've looked around can't seem to find an answer.
I have had the same problem. In my case was useful to start Zookeeper and directly specify a configuration file:
/bin/zkServer.sh start conf/zoo.conf
I have never heard of zookeeper, but it could be a permissions issue trying to write the file zookeeper_server.pid or perhaps the directory /tmp/zookeeper/ doesn't exist and the shell script isn't accounting for that possibility. Check the permissions and existence of those directories.
zookeeper distributed with default conf, uses /tmp/zookeeper as dataDir for just example sake. It is suggested changing this value in /path/to/zookeeper/conf/zoo.cfg to /var/lib/zookeeper.
Creating /var/lib/zookeeper needs root access, so sudo is required. This directory when created will have following permissions.
ls -al /var/lib/zookeeper/
drwxr-xrwx 4 root wheel 128 May 9 14:03 .
When zookeeper is started without root permission, it cannot write to this directory. hence fails with error
... /usr/local/zookeeper/bin/zkServer.sh: line 169: /var/lib/zookeeper/zookeeper_server.pid: Permission denied
FAILED TO WRITE PID
You need to give write permissions to allow user starting zookeeper to write to /var/lib/zookeeper. In my case, as I am using it in local, I used the following command and it worked
sudo chmod o+w /var/lib/zookeeper