I have Kafka running with a Sql connector. Currently this is running on a Linux server and I am using a Putty connection to configure and run everything. I can start the worker process in standalone mode fine and everything works as expected. However, it feels like I should be able to leave my terminal session and keep the worker running like a service. Currently I end my terminal session and just reconnect with putty but again, this doesn't feel like the right approach. Does anyone know how to get the worker to run like a service?
./connect-standalone.sh '../config/worker.properties' '../config/connector.properties'
First I would like to tell you that, running Kafka-connect in a standalone mode is not a good choice, and you can also run Kafka-connect in distributed mode on a single machine.
If you don't want to create a service then you can use the screen utility.
Ex.
Create a Screen
screen -S kafka-connect
Run kafka-connect command
./connect-standalone.sh '../config/worker.properties' '../config/connector.properties'
Detech the screen using ctrl+A+D
List and Resume screen
screen -ls
screen -r kafka-connect
Type exit inside the screen to the terminal attached screen.
Service
Create a new file kafka-connect.service inside /etc/systemd/system directory.
kafka-connect.service
[Unit]
Description=Kakfka-connect
After=network.target
[Service]
User=ubuntu
Group=ubuntu
Environmet="KAFKA_HEAP_OPTS=-Xmx1G -Xms1G"
Environment="KAFKA_OPTS=-javaagent:/home/ubuntu/prometheus/jmx_prometheus_javaagent-0.15.0.jar=8080:/home/ubuntu/prometheus/kafka-connect.yml"
ExecStart=/home/ubuntu/kafka_2.13-2.7.0/bin/connect-distributed.sh /home/ubuntu/kafka_2.13-2.7.0/config/connect-distributed.properties
[Install]
WantedBy=multi-user.target
In ExecStart that is the command to start Kafka-connect service as distributed mode, you can change that also If you haven't install jmx_exporter then you can remove Environment="KAFKA_OPTS=-javaagent:/home/ubuntu/prometheus/jmx_prometheus_javaagent-0.15.0.jar=8080:/home/ubuntu/prometheus/kafka-connect.yml" this line from service.
Related
I'm aware, that the kafka server can be shutdown using shell script kafka-server-stop.sh and zookeeper can be shutdown using zookeeper-server-stop.sh
But, how do we stop connect-distributed.sh gracefully, I didn't find any stop shell script for connect-distributed.
Unfortunately, there isn't a stop script.
Best options other than kill command would be to use systemctl to manage the service, or use pre-built Docker images to run the server that can be stopped.
telnet is necessary in order to maintain compatibility with older software in this case. I'm working with the Yocto Rocko 2.4.2 distribution. when I try to telnet to the board I'm getting the oh so detailed message "connection refused".
Using the method here and the options here I modified the busybox configuration per suggestion. When the board is booted up and logged in, if you execute: telnet, it spits out usage info and a quick directory check shows that telnet is installed to /usr/bin/telnet. My guess is that the telnet client is installed but the telnet server is not running?
I need to get telnetd to start manually at least so I know it will work with an init script in place. The second reference link there suggests that 'telnetd will not be started automatically though...' and that there will need to be an init script. How can I start telnetd manually for testing?
systemctl enable telnetd
returns: Unit telnetd.service could not be found
UPDATE
telnetd in located in /usr/sbin/telnetd. I was able to manually start the telnetd service for testing from there. After manually starting the service telnet login now works. looking into writing a systemd init script to auto start the telnetd service, so I suppose this issue is closed. unless anyone would like to offer up detailed telnet busybox configuration and setup steps as an answer to 'How to configure telnet service for yocto image'
update
Perhaps there is something more? I created a unit file that looks like this:
[Unit]
Description=auto start telnetd
[Service]
ExecStart=/usr/sbin/telnetd
[Install]
WantedBy=multi-user.target
on reboot, systemd indicates the process executed and succeeded:
systemctl status telnetd
.
.
.
Process: 466 ExecStart=/usr/sbin/telnetd (code=exited, status=0/SUCCESS)
.
.
.
The service is not running however. netstat -l does not list it and telnet login fails. Something I'm missing?
last update...i think
so following this post, I managed to get telnet.socket service to startup on reboot.
systemctl status telnet.socket
shows that it is running and listening on 23. Now however, when I try to remote in with telnet I'm getting
Connection closed by foreign host
Everything I've read so far has been talking about xinetd service (which I do not have...). What is confusing is that, if I just navigate to /usr/sbin/ and execute telnetd, the server is up and running and I can telnet into the board, so I do not believe I'm missing any utilities or services (like the above mentioned xinetd), but something is still not being configured correctly. any ideas?
Using Postgres 9.5 and the libpqxx c++ bindings, I want to launch a copy of postgres that is not installed on the users machine, but is instead packaged up in my application directory.
Currently, I am using pg_ctl.exe to start and stop the server, however when we do this, pg_ctl.exe seems to launch postgres.exe in a new terminal window.
I want it to launch postgres.exe in a headless state, but can't work out how.
I have tried enabling/disabling the logging collector, setting the logging method to a csv file (instead of stdout/stderr), and a couple of other logging related things, but I don't think the issue is the logging.
I have also tried running postgres.exe manually (without pg_ctl) and can get that to run headless by spawning it as a background process and redirecting the logs, but I would prefer to use the "pg_ctl start" api for the "wait for startup" (-w), and "timeout" (-t) options that it provides.
I believe you won't be able to do that with pg_ctl.
It is perfectly fine to start PostgreSQL directly through the server executable postgres.exe. Alternatively, you can use pg_ctl register to create a service and start the service.
In my use case, I was able to resolve the issue by running pg_ctl.exe using
CreateProcess, and providing the dwCreationFlags CREATE_NEW_PROCESS_GROUP | CREATE_NO_WINDOW.
I was originally using CREATE_NEW_PROCESS_GROUP | DETACHED_PROCESS, but DETACHED_PROCESS still allowed a postgres terminal to appear. This is because DETACHED_PROCESS will spawn the pg_ctl without a console, but any process that inherits stdin/stdout from pg_ctl will try to use it's console, and since there isn't one, one will be spawned. CREATE_NO_WINDOW however will launch the process with a conhost.exe, however the console will have no window. When the executables spawned by pg_ctl try to write to the terminal, they will successfully write to the console created by the conhost.exe which has no window.
I am now able to run pg_ctl from code with no console appearing.
I've developed a Netty application that allows connections through TCP from various devices. However i'm not entirely sure what is the best way to deploy the application for production use. Right now i package it up in a JAR file and run a screen session on the target server like so:
screen -S Nettyjava -jar Server-Netty.jar
Is this the recommended way to deploy it or is screen the best option available?
screen is not the right tool to run a service in production. If the system has to reboot, you will have to relaunch the service by hand. On most current linux distributions, you can handle this with a systemd service unit file. This allows you to define the working directory, the user, the command to run... Here is an example taken from the Unix & Linux StackExchange question configure java daemon with systemd
[Unit]
Description=Some job
After=network.target
[Service]
WorkingDirectory=/home/user/tmp/testout
SyslogIdentifier=SocketTest
ExecStart=/bin/sh -c "exec java -jar /home/user/programming/tests/java/core/SocketTest/SocketTest.jar"
User=dlt
Type=simple
[Install]
WantedBy=multi-user.target
A good practice consists in creating a specific user for running the service and to restrain his right on the filesystem.
I used to run my pyramid server as a daemon with the pserve --daemon command.
Given that it's deprecated, I'm looking for the best replacement. This link recommends to run it with screen or tmux, but it seems too heavy to just run a web server. Another idea would be to launch it with setsid.
What would be a good way to run it ?
Create a service file in /etc/systemd/system. Here a example (pyramid.service):
[Unit]
Description=pyramid_development
After=network.target
[Service]
# your Working dir
WorkingDirectory=/srv/www/webgis/htdocs/app
# your pserve path with ini
ExecStart=/srv/www/app/env/bin/pserve /srv/www/app/development.ini
[Install]
WantedBy=multi-user.target
Enable the service:
systemctl enable pyramid.service
Start/Stop/Restart the service with:
systemctl start pyramid.service
systemctl restart pyramid.service
systemctl stop pyramid.service
The simplest option is to install supervisord and setup a conf file for the service. The program would just be env/bin/pserve production.ini. There are countless examples online of how to do this.
The best option is to integrate with your system's process manager (systemd usually, but maybe also upstart or sysvinit or openrc). It is very easy to write a systemd unit file for starting pserve and then it will be started/stopped along with the rest of your system. Log files are even handled automatically in these cases.