Every time I stop the kafka server and start it again it doesn't start properly and I have to restart my whole machine and start the kafka server.
Does anybody know how I can restart kafka server without having to restart my machine?
Actually I would like to terminate the consumer from last session.
Thank you,
Zeinab
If your Kafka broker is running as a service (found under /lib/systemd/system/) from a recent Confluent Platform release, you can stop it using:
systemctl stop confluent-kafka.service
or if you'd like to restart the service,
systemctl restart confluent-kafka.service
Otherwise, you can stop your broker using
./bin/kafka-server-stop.sh
and re-start it:
./bin/kafka-server-start.sh config/server.properties
If you want to stop a specific consumer, simply find the corresponding process id:
ps -ef | grep consumer_name
and kill that process:
kill -9 process_id
Or simply:
sudo systemctl restart kafka
Related
I am currently start a kafka Connector in --daemon mode below:
bin/connect-standalone.sh -daemon \
/kafka/config/connect-standalone.properties \
/kafka/config/custom-connector.properties
How do I stop this connector process gracefully?
I am currenlty using top command to locate a java process and use kill -15 pid to stop it. I found this quite not practical because I cannot specify the connector by some properties to stop it.
Is there any way to stop a kafka connector in a way like executing a command below? Or any better alternatives?
kafka/bin/kafka-connect-stop.sh \
/kafka/config/connect-standalone.properties
To stop a connector, and not the worker, use PUT /connectors/{connector}/pause REST API endpoint.
https://kafka.apache.org/documentation/#connect_rest
Otherwise, yes, to stop the worker, you can use kill, or you can wrap it in SystemD script, and use systemctl stop to do the same.
Thanks #OneCricketeer's answer.
I wrap my command using systemd script below.
Create a kafka-connector.servce file in /etc/systemd/system as below
[Unit]
Description=Kafka Connector
[Service]
User=root
Type=simple
ExecStart=/bin/sh -c "/kafka/bin/connect-standalone.sh /kafka/config/connect-standalone.properties /kafka/config/my-connector.properties"
Start the kafka connector using
sudo systemctl start kafka-connector
Stop the kafka connector using
sudo systemctl stop kafka-connector
Check the status of the kafka connector using
sudo systemctl status kafka-connector
I'm aware, that the kafka server can be shutdown using shell script kafka-server-stop.sh and zookeeper can be shutdown using zookeeper-server-stop.sh
But, how do we stop connect-distributed.sh gracefully, I didn't find any stop shell script for connect-distributed.
Unfortunately, there isn't a stop script.
Best options other than kill command would be to use systemctl to manage the service, or use pre-built Docker images to run the server that can be stopped.
I tried to setup kafka and zookeeper on windows.
Initially I created topics, producers and consumers. It was working fine.
Then I deleted one topic by using the below command:
kafka-run-class.bat kafka.admin.TopicCommand --delete --topic junk --zookeeper localhost:2181
Now every time I re-run the kafka, it gets terminated with the below error:
java.nio.file.AccessDeniedException: C:\kafka_2.12-2.8.0\kafka_2.12-2.8.0kafka-logs\junk-0 -> C:\kafka_2.12-2.8.0\kafka_2.12-2.8.0kafka-logs\junk-0.305f67a1260f4cccb87d9367c6619fd2-delete
I tried to remove the zookeper and kafka directory and use a fresh directory for both. But somehow its retaining the previous saved topics and logs (I don't know what location they're stored at).
Could anyone tell me how to fix this?
I also faced the same issue with new version kafka_2.12-3.0.0. Sorted out by using lower version kafka_2.12-2.8.1
Login as admin, and try the below topic path/location the logs and
delete it manually or delete all your logs (both kafka & zookeeper logs) if you want to try fresh
/tmp/kafka-logs/[yourTopics] // Delete *** Kafka Logs
Now go back and try again. If you are still running into the problem then disable the cleaner
log.cleaner.enable = false
Next, I would recommend stopping all services and then type %temp% in windows command, and delete all the temp files as well
in linux
// find stale files older than for more than `7 days`
// and deletes those, not folders.
sudo find /tmp -type f -atime +7-delete
Finally, you will need to delete the zookeeper logs and kill the running zoo keeper process see this answer here are you using the confluent stack?
// kill the zookeeper process
ps aux | grep zookeeper
sudo kill -9 <PID> // or windows admin
// find and delete **** ZooKeeper logs
ps -ef | grep zookeeper | grep zookeeper.log.dir --color
lsof -p <pid of zookeeper> | grep log
lsof -p <pid of zookeeper> | grep out
Update to comment below - yes you can run on windows, with the WSL 2 subsystem/Linux 2 link from official confluent site
I am assuming that your Kafka server is being tied to Zookeeper. If true then we get this error when any topic was deleted.
To resolve this issue, we need to delete data directory folder configured in apache-zookeeper-home\conf\zoo.cfg file
Kafka server failed to start on confluent start command.
command lines:
~]# sudo confluent start
zookeeper is already running. Try restarting if needed
Starting kafka
-Kafka failed to start
kafka is [DOWN]
Cannot start Schema Registry, Kafka Server is not running. Check your deployment
Run
confluent log kafka
to see the log from Kafka trying to start, and see what the error is.
I'm new to storm I submitted topology but I want to shutdown storm.
First I killed topology by bin/storm kill topology_name
I want to close zookeeper, nimbus, supervisor?
if you ran it terminal without supervisor, just close terminal.
if you ran it under supervisor,
to stop zookeeper run "service zookeeper stop"
to stop supervisor run "service supervisord stop"
when supervisor shutdown, your storm will also shutdown
You can just kill the processes.
You can figure out the process ID with ps command:
ps -ef
Look for a java process that shows backtype.storm.ui.core (Nimbus) and backtype.storm.daemon.supervisor in its command line arguments?
For Zookeeper, you can just call bin/zkServer.sh stop.