Paths are correct, I don't know why I can't start kafka service, all lines, log files, etc and do not say what the Issue is... :/
I'm trying to install kafka on my Centos 7/8 and there is no Issue description I can figure out.
zookeeper.service file:
[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simpleUser=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.shRestart=on-abnormal
[Install]
WantedBy=multi-user.target
kafka.service file:
[Unit]
Requires=zookeeper.service
After=zookeeper.service
[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.shRestart=on-abnormal
[Install]
WantedBy=multi-user.target
UPDATE: I was trying to solve kafka Issue when I sould check zookeeper service first. Please check the workaround below.
Ok I found the solution and I'm posting it because I saw a lot of questions regarding this Issue hoping I can also help them.
So, please check out the new command line ExecStart:
zookeeper.service file:
[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simpleUser=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties'
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
On Centos (7/8/ X Version) it needs to specify the batch like this (using /bin/sh -c ''):
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties'
After this, you can run the kafka service! :)
From Kafka docs Quickstart
Kafka uses ZooKeeper so you need to first start a ZooKeeper server if
you don't already have one. You can use the convenience script
packaged with kafka to get a quick-and-dirty single-node ZooKeeper
instance.
Start zookeeper server with default settings
$ bin/zookeeper-server-start.sh config/zookeeper.properties
Start Kakfa server with default settings
$ bin/kafka-server-start.sh config/server.properties
If you had already done this step and still getting this error, Kafka server is unable to reach zookeeper service. Please check zookeeper server is running and listening on port mentioned (clientPort, default port: 2181) in zookeeper.properties file using anyone of the following commands - netstat, lsof, and telnet
Related
I am currently start a kafka Connector in --daemon mode below:
bin/connect-standalone.sh -daemon \
/kafka/config/connect-standalone.properties \
/kafka/config/custom-connector.properties
How do I stop this connector process gracefully?
I am currenlty using top command to locate a java process and use kill -15 pid to stop it. I found this quite not practical because I cannot specify the connector by some properties to stop it.
Is there any way to stop a kafka connector in a way like executing a command below? Or any better alternatives?
kafka/bin/kafka-connect-stop.sh \
/kafka/config/connect-standalone.properties
To stop a connector, and not the worker, use PUT /connectors/{connector}/pause REST API endpoint.
https://kafka.apache.org/documentation/#connect_rest
Otherwise, yes, to stop the worker, you can use kill, or you can wrap it in SystemD script, and use systemctl stop to do the same.
Thanks #OneCricketeer's answer.
I wrap my command using systemd script below.
Create a kafka-connector.servce file in /etc/systemd/system as below
[Unit]
Description=Kafka Connector
[Service]
User=root
Type=simple
ExecStart=/bin/sh -c "/kafka/bin/connect-standalone.sh /kafka/config/connect-standalone.properties /kafka/config/my-connector.properties"
Start the kafka connector using
sudo systemctl start kafka-connector
Stop the kafka connector using
sudo systemctl stop kafka-connector
Check the status of the kafka connector using
sudo systemctl status kafka-connector
I tried to install apache-kafka several times but I always had this problem. I'm using ubuntu on my virtual machine. When I'm trying to activate kafka service using sudo systemctl start kafka
and then controlling if it's working at first, the output is "active (running)", but if I double-check it and the output is "failed (Result: exit-code) ". And I tried sudo systemctl enable kafka but it didn't work.
This is the output:
● kafka.service
Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2021-05-26 05:40:22 PDT; 3s ago
Process: 8098 ExecStart=/bin/sh -c /home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/co>
Main PID: 8098 (code=exited, status=1/FAILURE)
May 26 05:40:19 ubuntu systemd[1]: Started kafka.service.
May 26 05:40:22 ubuntu systemd[1]: kafka.service: Main process exited, code=exited, status=1/FAILURE
May 26 05:40:22 ubuntu systemd[1]: kafka.service: Failed with result 'exit-code'.
You can see the full output attached
I also tried journalctl -xe and it recommended using ./gradlew jar -PscalaVersion=2.13.5, and I download it, at first it seemed to work, but the following day I had the same problem ( kafka.service: Failed with result 'exit-code'.). And if I tried journalctl -xe I had an output that you can see attached.
With zookeeper I had no problem, it's always active.
Thank you in advance.
Open the file meta.properties.
In my case, it was located at the path /home/kafka/logs/meta.properties
Just comment the the cluster.id with a #
Restart zookeeper and kafka.
I had the same issue by following the tutorial from well known site. I fixed the problem by doing all from the scratch this way.
sudo apt update
sudo apt install default-jdk
I downloaded latest BINARY release from here https://kafka.apache.org/downloads. I used https://dlcdn.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
sudo wget https://dlcdn.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
Unpack and move
tar xzf kafka_2.13-3.0.0.tgz
mv kafka_2.13-3.0.0 /usr/local/kafka
edit zookeeper unit file
sudo vi /etc/systemd/system/zookeeper.service
add this content
[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
Edit Kafka systemd unit file
sudo vi /etc/systemd/system/kafka.service
and add the content below. Note: You must change JAVA_HOME=path to your path
[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
[Service]
Type=simple
Environment="JAVA_HOME=REPLACE-THIS-WITH-YOUR-PATH"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target
Reload the systemd daemon to apply new changes.
sudo systemctl daemon-reload
Start zookeeper and kafka
sudo systemctl start zookeeper
sudo systemctl start kafka
check kafka status now, it should be running
sudo systemctl status kafka
All you need to do is to build kafka project before running it:
./gradlew jar -PscalaVersion=2.13.6
Note that you need to have Java installed
tried to install apache-kafka several times
Kafka doesn't come with Systemd scripts. Follow the official Apache Kafka website to see how you start it without systemctl
If you want to install on Ubuntu, Confluent Community edition allows you to do apt-get install to get both Kafka and Zookeeper
Your error shows an InconsistentClusterIdException, which means you need to wipe the data directories for Zookeeper and Kafka so that the broker will start in a fresh state
For me, I found out that the system actually has 2 folder kafka so when the service started, it said "exit-code"
-> My solution for my problem is delete 1 folder and keep folder /home/kafka
In my case Kafka didn't start in the first place, I reassigned a different logs folder to server.properties files and provided necessary rights to the folder, and restarted both the zookeeper and Kafka services, and then they seem to work.
in my case, I was using a Source Download
which I was : kafka-3.3.1-src.tgz
use binary version
Scala 2.13 - kafka_2.13-3.3.1.tgz
you can download it from https://kafka.apache.org/downloads
I have a Kuberenetes cluster and telegraf is running on each node. Telegraf is collecting data and storing into InfluxDB. Now I want to run another instance of telegraf which will use one of the pods namespace and collect stats from Apache server running inside the pod and store the stats in the same InfluxDB storage.
I followed this link (https://community.influxdata.com/t/multiple-telegraf-configs/245/6) but couldn't figure out how can I implement this in my setup.
I am using Debian GNU/Linux 9 (stretch) and telegraf_1.12.5-1.
I created two service file as follows:
cat /usr/lib/telegraf/scripts/telegraf.service
[Unit]
Description=The plugin-driven server agent for reporting metrics into InfluxDB
Documentation=https://github.com/influxdata/telegraf
After=network.target
[Service]
EnvironmentFile=-/etc/default/telegraf
User=telegraf
ExecStart=/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d $TELEGRAF_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartForceExitStatus=SIGPIPE
KillMode=control-group
[Install]
WantedBy=multi-user.target
cat /usr/lib/telegraf/scripts/telegraf_xyz.service
[Unit]
Description=The plugin-driven server agent for reporting metrics into InfluxDB
Documentation=https://github.com/influxdata/telegraf
After=network.target
[Service]
EnvironmentFile=-/etc/default/telegraf_xyz
User=telegraf
ExecStart=/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d $TELEGRAF_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartForceExitStatus=SIGPIPE
KillMode=control-group
[Install]
WantedBy=multi-user.target
But when I try to run the second instance it's giving error:
Failed to start telegraf_xyz.service: Unit telegraf_xyz.service not found.
What else changes I need to do as part of this? I see the telegraf.service file in many other locations (/sys/), I am not sure where else I need to configure the second telegraf instance. I am very new in this.
Is there any other better way to implement this in my setup?
NOTE: I have created two service file and able run that in my host. Now the real challange is running the instance in another net namespace. Can anyone help me to implement this?
Please put your service file here: /etc/systemd/system/. Then reload systemd with systemctl daemon-reload. Your service should now be found.
I have a DO droplet running Ubuntu 16.04.1x64 and I'm trying to run IPFS as a systemd service. I've gone ahead and created a user "connor" and installed IPFS following the instructions here. I'm storing the service as "ipfs.service" in ~/.config/systemd/user/ipfs.service which looks like this:
[Unit]
Description=IPFS Daemon
[Service]
Type=simple
ExecStart=/usr/local/bin/ipfs daemon
ExecStop=/usr/bin/pkill ipfs
Restart=always
User=Connor
[Install]
WantedBy=default.target
What's odd is that if I run systemctl --user start ipfs it starts up just fine. However, running systemctl --user daemon-reload and then
systemctl --user enable ipfs I get the error:
Failed to execute operation: No such file or directory
However, if I run systemctl enable /home/connor/.config/systemd/user/ipfs.service -f it runs just fine. I can reboot and run IPFS commands just fine. I'd like to run it as a user though, and would also like to understand what I'm doing wrong.
Please, check that you are executing the commands with connor user, you may run whoami to see the user executing the command. (running the command with sudo changes the user to root)
In addition, I see that the user in the service file is capitalized (Connor instead of connor), this could bring other problems, and it is not needed, as a simple configuration like the one proposed by Arch Linux wiki works for user daemons.
Please find bellow the configuration I used for my ipfs daemon, (without User= and with a different Restart=, since Restart=always gave me problems while starting the daemon):
[Unit]
Description=IPFS daemon
After=network.target
[Service]
ExecStart=/usr/local/bin/ipfs daemon
Restart=on-failure
[Install]
WantedBy=default.target
I'm having trouble getting a Docker container to stay up when it's started by systemd. When I start it manually with sudo docker start containername, it stays up without trouble, but when it's started via systemd with sudo systemctl start containername, it stays up for 10 seconds then mysteriously dies, leaving messages in syslog something like the following:
Mar 13 14:01:09 hostname docker[329]: time="2015-03-13T14:01:09Z" level="info" msg="POST /v1.17/containers/containername/stop?t=10"
Mar 13 14:01:09 hostname docker[329]: time="2015-03-13T14:01:09Z" level="info" msg="+job stop(containername)"
I am making the assumption that it's systemd killing the process, but I can't work out why it might be happening. The systemd unit file (/etc/systemd/system/containername.service) is pretty simple, as follows:
[Unit]
Description=MyContainer
After=docker.service
Requires=docker.service
[Service]
ExecStart=/usr/bin/docker start containername
ExecStop=/usr/bin/docker stop containername
[Install]
WantedBy=multi-user.target
Docker starts fine on boot, and it looks like it does even start the docker container, but no matter if on boot or manually, it then quits after exactly 10 seconds. Help gratefully received!
Solution: The start command seems to need the -a (attach) parameter as described in the documentation when used in a systemd script. I assume this is because it by default forks to the background, although the systemd expect daemon feature doesn't appear to fix the issue.
from the docker-start manpage:
-a, --attach=true|false
Attach container's STDOUT and STDERR and forward all signals to the process. The default is false.
The whole systemd script then becomes:
[Unit]
Description=MyContainer
After=docker.service
Requires=docker.service
[Service]
ExecStart=/usr/bin/docker start -a containername
ExecStop=/usr/bin/docker stop containername
[Install]
WantedBy=multi-user.target