I am currently start a kafka Connector in --daemon mode below:
bin/connect-standalone.sh -daemon \
/kafka/config/connect-standalone.properties \
/kafka/config/custom-connector.properties
How do I stop this connector process gracefully?
I am currenlty using top command to locate a java process and use kill -15 pid to stop it. I found this quite not practical because I cannot specify the connector by some properties to stop it.
Is there any way to stop a kafka connector in a way like executing a command below? Or any better alternatives?
kafka/bin/kafka-connect-stop.sh \
/kafka/config/connect-standalone.properties
To stop a connector, and not the worker, use PUT /connectors/{connector}/pause REST API endpoint.
https://kafka.apache.org/documentation/#connect_rest
Otherwise, yes, to stop the worker, you can use kill, or you can wrap it in SystemD script, and use systemctl stop to do the same.
Thanks #OneCricketeer's answer.
I wrap my command using systemd script below.
Create a kafka-connector.servce file in /etc/systemd/system as below
[Unit]
Description=Kafka Connector
[Service]
User=root
Type=simple
ExecStart=/bin/sh -c "/kafka/bin/connect-standalone.sh /kafka/config/connect-standalone.properties /kafka/config/my-connector.properties"
Start the kafka connector using
sudo systemctl start kafka-connector
Stop the kafka connector using
sudo systemctl stop kafka-connector
Check the status of the kafka connector using
sudo systemctl status kafka-connector
Related
I am trying to install Kafka following a tutorial from DigitalOcean.com here.
I am doing this on Windows WSL2 with Ubuntu. So, after creating the zookeeper.service and kafka.service as per the tutorial, I do this command (the tutorial uses sudo systemctl start kafka instead), following advice from this thread:
sudo service kafka start
I received :
kafka: unrecognized service
When I do service --status-all to see if kafka is in the list, it is not there.
What am I missing?
There is lack of support in WSL for systemd
why systemd is disabled in WSL?
I have Kafka running with a Sql connector. Currently this is running on a Linux server and I am using a Putty connection to configure and run everything. I can start the worker process in standalone mode fine and everything works as expected. However, it feels like I should be able to leave my terminal session and keep the worker running like a service. Currently I end my terminal session and just reconnect with putty but again, this doesn't feel like the right approach. Does anyone know how to get the worker to run like a service?
./connect-standalone.sh '../config/worker.properties' '../config/connector.properties'
First I would like to tell you that, running Kafka-connect in a standalone mode is not a good choice, and you can also run Kafka-connect in distributed mode on a single machine.
If you don't want to create a service then you can use the screen utility.
Ex.
Create a Screen
screen -S kafka-connect
Run kafka-connect command
./connect-standalone.sh '../config/worker.properties' '../config/connector.properties'
Detech the screen using ctrl+A+D
List and Resume screen
screen -ls
screen -r kafka-connect
Type exit inside the screen to the terminal attached screen.
Service
Create a new file kafka-connect.service inside /etc/systemd/system directory.
kafka-connect.service
[Unit]
Description=Kakfka-connect
After=network.target
[Service]
User=ubuntu
Group=ubuntu
Environmet="KAFKA_HEAP_OPTS=-Xmx1G -Xms1G"
Environment="KAFKA_OPTS=-javaagent:/home/ubuntu/prometheus/jmx_prometheus_javaagent-0.15.0.jar=8080:/home/ubuntu/prometheus/kafka-connect.yml"
ExecStart=/home/ubuntu/kafka_2.13-2.7.0/bin/connect-distributed.sh /home/ubuntu/kafka_2.13-2.7.0/config/connect-distributed.properties
[Install]
WantedBy=multi-user.target
In ExecStart that is the command to start Kafka-connect service as distributed mode, you can change that also If you haven't install jmx_exporter then you can remove Environment="KAFKA_OPTS=-javaagent:/home/ubuntu/prometheus/jmx_prometheus_javaagent-0.15.0.jar=8080:/home/ubuntu/prometheus/kafka-connect.yml" this line from service.
Paths are correct, I don't know why I can't start kafka service, all lines, log files, etc and do not say what the Issue is... :/
I'm trying to install kafka on my Centos 7/8 and there is no Issue description I can figure out.
zookeeper.service file:
[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simpleUser=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.shRestart=on-abnormal
[Install]
WantedBy=multi-user.target
kafka.service file:
[Unit]
Requires=zookeeper.service
After=zookeeper.service
[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.shRestart=on-abnormal
[Install]
WantedBy=multi-user.target
UPDATE: I was trying to solve kafka Issue when I sould check zookeeper service first. Please check the workaround below.
Ok I found the solution and I'm posting it because I saw a lot of questions regarding this Issue hoping I can also help them.
So, please check out the new command line ExecStart:
zookeeper.service file:
[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simpleUser=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties'
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
On Centos (7/8/ X Version) it needs to specify the batch like this (using /bin/sh -c ''):
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties'
After this, you can run the kafka service! :)
From Kafka docs Quickstart
Kafka uses ZooKeeper so you need to first start a ZooKeeper server if
you don't already have one. You can use the convenience script
packaged with kafka to get a quick-and-dirty single-node ZooKeeper
instance.
Start zookeeper server with default settings
$ bin/zookeeper-server-start.sh config/zookeeper.properties
Start Kakfa server with default settings
$ bin/kafka-server-start.sh config/server.properties
If you had already done this step and still getting this error, Kafka server is unable to reach zookeeper service. Please check zookeeper server is running and listening on port mentioned (clientPort, default port: 2181) in zookeeper.properties file using anyone of the following commands - netstat, lsof, and telnet
Every time I stop the kafka server and start it again it doesn't start properly and I have to restart my whole machine and start the kafka server.
Does anybody know how I can restart kafka server without having to restart my machine?
Actually I would like to terminate the consumer from last session.
Thank you,
Zeinab
If your Kafka broker is running as a service (found under /lib/systemd/system/) from a recent Confluent Platform release, you can stop it using:
systemctl stop confluent-kafka.service
or if you'd like to restart the service,
systemctl restart confluent-kafka.service
Otherwise, you can stop your broker using
./bin/kafka-server-stop.sh
and re-start it:
./bin/kafka-server-start.sh config/server.properties
If you want to stop a specific consumer, simply find the corresponding process id:
ps -ef | grep consumer_name
and kill that process:
kill -9 process_id
Or simply:
sudo systemctl restart kafka
I used to run my pyramid server as a daemon with the pserve --daemon command.
Given that it's deprecated, I'm looking for the best replacement. This link recommends to run it with screen or tmux, but it seems too heavy to just run a web server. Another idea would be to launch it with setsid.
What would be a good way to run it ?
Create a service file in /etc/systemd/system. Here a example (pyramid.service):
[Unit]
Description=pyramid_development
After=network.target
[Service]
# your Working dir
WorkingDirectory=/srv/www/webgis/htdocs/app
# your pserve path with ini
ExecStart=/srv/www/app/env/bin/pserve /srv/www/app/development.ini
[Install]
WantedBy=multi-user.target
Enable the service:
systemctl enable pyramid.service
Start/Stop/Restart the service with:
systemctl start pyramid.service
systemctl restart pyramid.service
systemctl stop pyramid.service
The simplest option is to install supervisord and setup a conf file for the service. The program would just be env/bin/pserve production.ini. There are countless examples online of how to do this.
The best option is to integrate with your system's process manager (systemd usually, but maybe also upstart or sysvinit or openrc). It is very easy to write a systemd unit file for starting pserve and then it will be started/stopped along with the rest of your system. Log files are even handled automatically in these cases.