I tried to setup kafka and zookeeper on windows.
Initially I created topics, producers and consumers. It was working fine.
Then I deleted one topic by using the below command:
kafka-run-class.bat kafka.admin.TopicCommand --delete --topic junk --zookeeper localhost:2181
Now every time I re-run the kafka, it gets terminated with the below error:
java.nio.file.AccessDeniedException: C:\kafka_2.12-2.8.0\kafka_2.12-2.8.0kafka-logs\junk-0 -> C:\kafka_2.12-2.8.0\kafka_2.12-2.8.0kafka-logs\junk-0.305f67a1260f4cccb87d9367c6619fd2-delete
I tried to remove the zookeper and kafka directory and use a fresh directory for both. But somehow its retaining the previous saved topics and logs (I don't know what location they're stored at).
Could anyone tell me how to fix this?
I also faced the same issue with new version kafka_2.12-3.0.0. Sorted out by using lower version kafka_2.12-2.8.1
Login as admin, and try the below topic path/location the logs and
delete it manually or delete all your logs (both kafka & zookeeper logs) if you want to try fresh
/tmp/kafka-logs/[yourTopics] // Delete *** Kafka Logs
Now go back and try again. If you are still running into the problem then disable the cleaner
log.cleaner.enable = false
Next, I would recommend stopping all services and then type %temp% in windows command, and delete all the temp files as well
in linux
// find stale files older than for more than `7 days`
// and deletes those, not folders.
sudo find /tmp -type f -atime +7-delete
Finally, you will need to delete the zookeeper logs and kill the running zoo keeper process see this answer here are you using the confluent stack?
// kill the zookeeper process
ps aux | grep zookeeper
sudo kill -9 <PID> // or windows admin
// find and delete **** ZooKeeper logs
ps -ef | grep zookeeper | grep zookeeper.log.dir --color
lsof -p <pid of zookeeper> | grep log
lsof -p <pid of zookeeper> | grep out
Update to comment below - yes you can run on windows, with the WSL 2 subsystem/Linux 2 link from official confluent site
I am assuming that your Kafka server is being tied to Zookeeper. If true then we get this error when any topic was deleted.
To resolve this issue, we need to delete data directory folder configured in apache-zookeeper-home\conf\zoo.cfg file
Related
I have been working with Kafka 2.4.0 (2.11) and yesterday I had to forcefully terminate the process for some unknown reason. Since then I haven't been unable to start Zookeeper due to the following error:
[2020-01-11 11:12:43,783] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.io.IOException: No snapshot found, but there are log entries. Something is broken!
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:222)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
And as soon as I searched for this problem I found issue ZOOKEEPER-3513 reported, which may or may not explain the problem. However, what I'm finding strange is that if I delete the Kafka/Zookeeper directory and download it again from scratch, the problem persists. Does anyone know how I can solve this?
Thank you for your help
Check for the tmp/zookeeper folder on the drive where you have kafka folder (lets say D:/), and delete the folder tmp, which will create automatically for you once run the zookeeper again.
Try changing your zookeeper data directory.
Your zookeeper data directory is defined in zookeeper.properties (I think the default is /tmp/zookeeper).
Perhaps you're not deleting the correct zookeeper directory?
I had the same problem, and this solution worked.
NOTE: I'm experimenting with Kafka, and not using it in production. I have no idea what else the above does, apart from fix this error...
I've faced the same issue with Zookeeper after updating from version 3.4.x to 3.5.6. As described here. I've:
added empty snapshot.0 file in data directory
added a property 'zookeeper.snapshot.trust.empty=true' to Zookeeper configuration file (default is zoo.cfg)
On windows ->
Go to the tmp folder where the zookeeper details are stored
and delete the existing log files
Directory path = d:\tmp\zookeeper\version-2
On Linux ->
Path = /tmp/zookeeper/version-2
And remove all the existing log files using rm -r log.1
The log files will be created automatically again and will resolve the issue.
Faced same issue in macOS.
Solution: In kafka dir, path cd /tmp/zookeeper/version-2 deleted the log.1 file. It worked for me
if you are on windows make sure you escape the location of the zookeeper temp directory.
dataDir=d:\tmp\zookeeper
Created a new dir for logs and configured the same path in zoo.cfg.
It worked:)
I use macOS and my solution was to delete everything in the dataDir, the default value should be /usr/local/var/lib/zookeeper.
For those who are using docker, I'll share my experience:
I've been running zookeeper confluentinc/cp-zookeeper:5.2.1 as it follows:
docker run \
--network kafka-net --name=zookeeper \
-e ALLOW_ANONYMOUS_LOGIN=yes \
-e ZOOKEEPER_CLIENT_PORT=2181 \
-v /tmp/zookeeper-data:/var/lib/zookeeper/data \
-v /tmp/zookeeper-txn-logs:/var/lib/zookeeper/log \
-p 2181:2182 confluentinc/cp-zookeeper:5.2.1
As expected, I can see a few files placed in /tmp/zookeeper-txn-logs and /tmp/zookeeper-data on host. After cleaning up /tmp/zookeeper-data and running again, I've got the error No snapshot found, but there are log entries.
In my case, I just had to purge the data on /tmp/zookeeper-txn-logs. For a dev/production environment, I'd recommend following the docs https://access.redhat.com/documentation/en-us/red_hat_amq/6.3/html/fabric_guide/ensemble-purgetxnlog
Below the steps I did to get this issue :
Launch ZooKeeper
Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties
And at the second step the error happens :
ERROR Fatal error during KafkaServer startup. Prepare to shutdown
(kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID
Reu8ClK3TTywPiNLIQIm1w doesn't match stored clusterId
Some(BaPSk1bCSsKFxQQ4717R6Q) in meta.properties. The broker is trying
to join the wrong cluster. Configured zookeeper.connect may be wrong.
at kafka.server.KafkaServer.startup(KafkaServer.scala:220)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
When I trigger .\bin\windows\kafka-server-start.bat .\config\server.properties zookeeper console returns :
INFO [SyncThread:0:FileTxnLog#216] - Creating new log file: log.1
How to fix this issue to get kafka running ?
Edit You can access to the proper issue on the right site (serverfault) here
Edit Here is the Answer
I managed to Solve this issue with the following steps :
Just Delete all the log/Data file created (or generated) into
zookeeper and kafka.
Run Zookeper
Run Kafka
[Since this post is open again I post my answer there so you got all on the same post]
** 1. The easiest solution is to remove kafka logs and start again.
** 2. But the root cause is Kafka saved failed cluster ID in meta.properties.**
Try to delete kafka-logs/meta.properties from your tmp folder, which is located in C:/tmp folder by default on windows, and /tmp/kafka-logs on Linux
if kafka is running in docker containers, the log path may be specified by volume config in the docker-compose - see docs.docker.com/compose/compose-file/compose-file-v2/#volumes -- Chris Halcrow
** 3. How to find Kafka log path:**
Open server server.properties file which is located in your kafka folder kafka_2.11-2.4.0\config\server.properties (considering your version of kafka, folder name could be kafka_<kafka_version>):
Then search for entry log.dirs to check where logs locate
log.dirs=/tmp/kafka-logs
For mac, the following steps are needed.
Stop kafka service: brew services stop kafka
open kafka server.properties file: vim /usr/local/etc/kafka/server.properties
find value of log.dirs in this file. For me, it is /usr/local/var/lib/kafka-logs
delete path-to-log.dirs/meta.properties file
start kafka service brew services start kafka
No need to delete the log/data files on Kafka. Check the Kafka error logs and find the new cluster id. Update the meta.properties file with cluster-ID then restart the Kafka.
/home/kafka/logs/meta.properties
To resolve this issue permanently follow below.
Check your zookeeper.properties file and look for dataDirpath and change the path tmp location to any other location which should not be removed after server restart.
/home/kafka/kafka/config/zookeeper.properties
Copy the zookeeper folder and file to the new(below or non tmp) location then restart the zookeeper and Kafka.
cp -r /tmp/zookeeper /home/kafka/zookeeper
Now server restart won’t affect the Kafka startup.
If you use Embedded Kafka with Testcontainers in your Java project like myself, then simply delete your build/kafka folder and Bob's your uncle.
The mentioned meta.properties can be found under build/kafka/out/embedded-kafka.
I had some old volumes lingering around. I checked the volumes like this:
docker volume list
And pruned old volumes:
docker volume prune
And also removed the ones that were kafka:
example:
docker volume rm test_kafka
I deleted the following directories :-
a.) logs directory from kafka-server's configured location i.e. log.dir property path.
b.) tmp directory from kafka broker's location.
log.dirs=../tmp/kafka-logs-1
I was using docker-compose to re-set up Kafka on a Linux server, with a known, working docker-compose.config that sets up a number of Kafka components (broker, zookeeper, connect, rest proxy), and I was getting the issue described in the OP. I fixed this for my dev server instance by doing the following
docker-compose down
backup kafka-logs directory using cp kafka-logs -r kafka-logs-bak
delete the kafka-logs/meta.properties file
docker-compose up -d
Note for users of docker-compose:
My log files weren't in the default location (/tmp/kafka-logs). If you're running Kafka in Docker containers, the log path can be specified by volume config in the docker-compose e.g.
volumes:
- ./kafka-logs:/tmp/kafka-logs
This is specifying SOURCE:TARGET. ./kafka-logs is the source (i.e. a directory named kafka-logs, in the same directory as the docker-compose file). This is then targeted to /tmp/kafka-logs as the mounted volume within the kafka container). So the logs can either be deleted from the source folder on the host machine, or by deleting them from the mounted volume after doing a docker exec into the kafka container.
see https://docs.docker.com/compose/compose-file/compose-file-v2/#volumes
For me, meta.properties was in /usr/local/var/lib/kafka-logs
By removing it, the kafka started working.
I also deleted all the content of the folder containing all data generated by Kafka. I could find the folder in my .yml file:
kafka:
image: confluentinc/cp-kafka:7.0.0
ports:
- '9092:9092'
environment:
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_BROKER_ID: 1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: "true"
volumes:
- ./kafka-data/data:/var/lib/kafka/data
depends_on:
- zookeeper
networks:
- default
Under volumes: stays the location. So, in my case I deleted all files of the data folder located under kafka-data.
I've tried deleting the meta.properties file but didn't work.
In my case, it's solved by deleting legacy docker images.
But the problem with this is that deletes all previous data.
So be careful if you want to keep the old data this is not the right solution for you.
docker rm $(docker ps -q -f 'status=exited')
docker rmi $(docker images -q -f "dangling=true")
Sometimes its necessary to fresh-start a kafka cluster with no data. when running a kafka inside docker containers this behavior is achieved for free.
How to do it with kafka process ? can i delete /var/log/kafka* and restart it ? is it ok to do so ?
BTW - i am using something like this :
# bash shell
# tl is a list of all topics
for t in $(cat tl); do
./kafka-topics.sh --zookeeper $ZOO --delete --topic $t
done
there are 2 problem with the above :
if hdd usage is 100%, then i got error when trying the kafka-topics.sh
its very inefficient if i have many topics
looking for a fast and clean way to do in dev envs.
seems like this do the job
$ ###### stop and clear all brokers
$ sudo systemctl stop kafka.service zookeeper.service
$ sudo rm -rf /var/log/kafka-logs/*
$ ###### continue ONLY after finish the above on all brokers
$ sudo systemctl start zookeeper.service
$ sleep 10s # make sure zookeeper is ready
$ sudo systemctl start kafka.service
I would suggest you to temporary set log.retention.ms to low value, lets say 1000. This way you tell kafka to keep messages only one second before deleting it forever. You wait a little bit, kafka will delete all of your messages(all but not messages on active segments) and when that is done, you can revert that log.retention.ms settings.
I have installed a kafka that has integrated zookeeper.
I have seen that to enter an independent Zookeeper installation, you can run the following command to enter the zookeeper console:
bin/ZkCli.sh
ls /zookeeper/quota
But in Kafka's scripts I only have:
zookeeper-security-migration.sh
zookeeper-server-start.sh
zookeeper-server-stop.sh
zookeeper-shell.sh
I have tried to do the following:
./zookeeper-shell.sh -server 127.0.0.1:2181 ls /zookeeper/quota
But it doesn't work, it doesn't do anything
How can I get into the Zookeeper that is integrated in Kafka?
After starting Zookeeper, you can connect to it using the zookeeper-shell.sh tool.
To get into the shell:
./zookeeper-shell IP:2181
Then you can execute commands, like:
ls /
You can use cd to move within the nodes and get to print the content of nodes.
You can also use this script to just run commands and return (without getting into the shell)
./zookeeper-shell.sh localhost:2181 get /controller
/zookeeper/quota is not a path used by Kafka, Quotas are stored under /config
Every time I stop the kafka server and start it again it doesn't start properly and I have to restart my whole machine and start the kafka server.
Does anybody know how I can restart kafka server without having to restart my machine?
Actually I would like to terminate the consumer from last session.
Thank you,
Zeinab
If your Kafka broker is running as a service (found under /lib/systemd/system/) from a recent Confluent Platform release, you can stop it using:
systemctl stop confluent-kafka.service
or if you'd like to restart the service,
systemctl restart confluent-kafka.service
Otherwise, you can stop your broker using
./bin/kafka-server-stop.sh
and re-start it:
./bin/kafka-server-start.sh config/server.properties
If you want to stop a specific consumer, simply find the corresponding process id:
ps -ef | grep consumer_name
and kill that process:
kill -9 process_id
Or simply:
sudo systemctl restart kafka