Script hangs only when started via SystemD - apache-kafka

I am trying to start the Kafka Connect component alongside Cloudera and want to wrap the Kafka Connect startup script in a systemd service file such that it can start on boot once the Cloudera services start.
For some strange reason if I start this script outside of systemd it works just fine, but when I start it via systemctl start kafka-connect it just hangs at the following log entry lines.
[Unit]
Description=Kafka Connect - Distributed
After=network.target cloudera-scm-agent
[Service]
Type=forking
ExecStart=/bin/bash -c "/app/cloudera/parcels/KAFKA/lib/kafka/bin/connect-distributed.sh -daemon /app/cloudera/kafka-connect/connect-distributed.properties"
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
2019-07-18 13:40:27,962 INFO (main) [ConnectDistributed(main:69)] Scanning for plugin classes. This might take a moment ...
2019-07-18 13:40:27,992 INFO (main) [DelegatingClassLoader(registerPlugin:184)] Loading plugin from: /app/cloudera/kafka-connect/plugins/splunk-kafka-connect.jar
2019-07-18 13:40:27,993 DEBUG (main) [DelegatingClassLoader(registerPlugin:191)] Loading plugin urls: [file:/app/cloudera/kafka-connect/plugins/splunk-kafka-connect.jar]
2019-07-18 13:40:29,206 DEBUG (main) [VersionUtils(getVersionFromProperties:63)] found git version string=v1.1.0 in version.properties file
2019-07-18 13:40:29,219 INFO (main) [DelegatingClassLoader(scanUrlsAndAddPlugins:207)] Registered loader: PluginClassLoader{pluginLocation=file:/app/cloudera/kafka-connect/plugins/splunk-kafka-connect.jar}
2019-07-18 13:40:29,220 INFO (main) [DelegatingClassLoader(addPlugins:136)] Added plugin 'com.splunk.kafka.connect.SplunkSinkConnector'
2019-07-18 13:40:29,220 INFO (main) [DelegatingClassLoader(addPlugins:136)] Added plugin 'org.apache.kafka.connect.storage.StringConverter'
My next thought was to try something a bit simpler and just create an init.d script at /etc/init.d/kafka-connect which works fine because it is nothing more than a shell script wrapper UNTIL I source in the /etc/init.d/functions file which causes this to be started via systemd again.
Two main questions -
What does systemctl do differently than a regular shell script that would cause this shell script (which in turn launches a java process) to hang at that exact same step everytime.
If my /etc/init.d/kafka-connect script works, is it okay to use on RHEL7? If so, is there a way to load that init.d script on boot of the server?
Thanks in advance!

Related

Apache Geode can't run with basic instructions

I tried following the basic instructions to run a node over here https://geode.apache.org/docs/guide/114/getting_started/15_minute_quickstart_gfsh.html, but I am getting the following issue:
gfsh>start locator --name=locator1
Starting a Geode Locator in /home/thiago/geode/locator1...
........
Locator in /home/thiago/geode/locator1 on 192.168.50.225[10334] as locator1 is currently online.
Process ID: 28137
Uptime: 5 seconds
Geode Version: 1.14.0
Java Version: 16.0.2
Log File: /home/thiago/geode/locator1/locator1.log
JVM Arguments: -Dgemfire.enable-cluster-configuration=true -Dgemfire.load-cluster-configuration-from-dir=false -Dgemfire.launcher.registerSignalHandlers=true -Djava.awt.headless=true -Dsun.rmi.dgc.server.gcInterval=9223372036854775806
Class-Path: /home/thiago/apache-geode-1.14.0/lib/geode-core-1.14.0.jar:/home/thiago/apache-geode-1.14.0/lib/geode-dependencies.jar
Unable to auto-connect (Security Manager may be enabled). Please use "connect --locator=192.168.50.225[10334]" to connect Gfsh to the locator.
Failed to connect; unknown cause: Exception caused JMX Manager startup to fail because: 'java.rmi.server.ExportException: Port already in use: 1099; nested exception is:
java.net.BindException: Failed to create server socket on 127.0.1.1[1099]'
Before starting I made sure nothing was running on port 1099, but after running the command I checked that what uses port 1099 is the spawned java process itself:
❯ lsof -i:1099
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 28137 thiago 206u IPv6 262648 0t0 TCP *:rmiregistry (LISTEN)
Sounds like the issue is because I was using java 16 which does not seems to be supported. Using java 1.8 fixes it.
Same thing still happens with lastest geode and JDK 17. JDK 11 is the latest stable release that will still work
First try downgrading or switching to the stable Java version 1.8 or 11
locator need 10334 port to start and the need 1099 for cluster connection.
To check the port availability:
lsof -I:[port-number]
Then take the PID form the above result to free up the port
To free up the port:
sudo kill PID
if it doesn't work then try
sudo kill -2 PID
or
sudo kill -1 PID
or
sudo kill -9 PID

Ionic Serve Not Serving

When I run ionic serve, it builds the app but the server isn't run. Here's the trail:
Starting app-scripts server: --address 0.0.0.0 --port 8100 --livereload-port 35729
--dev-logger-port 53703 --nobrowser - Ctrl+C to cancel
[17:55:35] watch started ...
[17:55:35] build dev started ...
[17:55:36] clean started ...
[17:55:36] clean finished in 2 ms
[17:55:36] copy started ...
[17:55:36] deeplinks started ...
[17:55:37] deeplinks finished in 589 ms
[17:55:37] transpile started ...
[17:55:44] transpile finished in 7.38 s
[17:55:44] preprocess started ...
[17:55:44] copy finished in 8.56 s
[17:55:44] preprocess finished in 206 ms
[17:55:44] webpack started ...
After webpack started the cli just exits and i see the normal terminal prompt.
Trying out the npm run ionic:serve command instead results into this error:
TypeError: Cannot read property 'substr' of undefined
So far I've re-installed ionic, same result. A bit unsure how to deal with this one...
It turns out that this is caused when creating multiple component classes in one ts file. Splitting those classes each into their own file solves the problem.
Related to this https://github.com/ionic-team/ionic-app-scripts/issues/796

How do I use runit with zookeeper

How do I use runit with zookeeper? Its running but I get those nasty logs.....
Here is my run file
more /etc/sv/zookeeper/run
#!/bin/sh
exec 2>&1
exec /var/chef/cache/zookeeper-3.4.5/bin/zkServer.sh start >> /tmp/zookeeper.log 2>&1
below is the tail of my log file
tail -f zookeeper.log
Using config: /var/chef/cache/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 32701.
JMX enabled by default
Using config: /var/chef/cache/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 32701.
JMX enabled by default
Using config: /var/chef/cache/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 32701.
JMX enabled by default
Using config: /var/chef/cache/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 32701.
JMX enabled by default
Using config: /var/chef/cache/zookeeper-3.4.5/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 32701.
From the look of it, zookeeper is already running in your system. Or there is a lock file which didn't get deleted due to an unclean shutdown. Check using zkServer.sh status to see if it is running properly.

Supervisord does not start killed processes

I have supervisord installed on my Ubuntu 10.04 and it runs a Java process continuously and supposed to heal (reload) process when it somehow dies or crashes.
On my htop I send SIGKILL, SIGTERM, SIGHUP, SIGSEGV signals to that Java process and watch /etc/logs/supervisord.log file and it says.
08:09:46,182 INFO success: myprogram entered RUNNING state,[...]
08:38:10,043 INFO exited: myprogram (exit status 0; expected)
At 08:38 I kill the process with SIGSEGV. How come it is exited with code 0 and why does not supervisord restart it at all?
All my supervisord.conf about this specific program is as follows:
[program:play-9000]
command=play run /var/www/myprogram/ --%%prod
stderr_logfile = /var/log/supervisord/myprogram-stderr.log
stdout_logfile = /var/log/supervisord/myprogram-stdout.log
Process works really fine when I launch supervisord, however does not get healed.
By the way any ideas how to start supervisord as a service so that it automatically launches when the whole system reboots?
Try setting autorestart=true. By default, autorestart is set to "unexpected" which means it will only restart a process if it exits with an unexpected exit code. By default, exit code 0 is expected.
http://supervisord.org/configuration.html#program-x-section-settings
You can use the chkconfig program to make sure that supervisor starts on reboot.
$ sudo apt-get install chkconfig
$ chkconfig -l supervisor
supervisor 0:off 1:off 2:on 3:on 4:on 5:on 6:off
You can see that it's enabled for runlevels 2-5 by default when I installed it.
$ man 7 runlevel
for more info on run levels.

Unable to start JBoss from within Eclipse

I am unable to start JBoss server 5.1.0.GA version from eclipse Indigo.
Eclipse shows me message box saying 'Server JBoss v5.0 at localhost was unable to start within 500 seconds. If the server requires more time, try increasing the timeout in the server editor.' but in the console window I can see that JBoss has been actually started.
here is some part of log which I can see in console window of eclipse :
SecureDeploymentManager/remote - EJB3.x Default Remote Business Interface
SecureDeploymentManager/remote-org.jboss.deployers.spi.management.deploy.DeploymentManager - EJB3.x Remote Business Interface
15:14:20,212 INFO [SessionSpecContainer] Starting jboss.j2ee:jar=profileservice-secured.jar,name=SecureManagementView,service=EJB3
15:14:20,212 INFO [EJBContainer] STARTED EJB: org.jboss.profileservice.ejb.SecureManagementView ejbName: SecureManagementView
15:14:20,222 INFO [JndiSessionRegistrarBase] Binding the following Entries in Global JNDI:
SecureManagementView/remote - EJB3.x Default Remote Business Interface
SecureManagementView/remote-org.jboss.deployers.spi.management.ManagementView - EJB3.x Remote Business Interface
15:14:20,252 INFO [SessionSpecContainer] Starting jboss.j2ee:jar=profileservice-secured.jar,name=SecureProfileService,service=EJB3
15:14:20,262 INFO [EJBContainer] STARTED EJB: org.jboss.profileservice.ejb.SecureProfileServiceBean ejbName: SecureProfileService
15:14:20,272 INFO [JndiSessionRegistrarBase] Binding the following Entries in Global JNDI:
SecureProfileService/remote - EJB3.x Default Remote Business Interface
SecureProfileService/remote-org.jboss.profileservice.spi.ProfileService - EJB3.x Remote Business Interface
15:14:20,362 INFO [TomcatDeployment] deploy, ctxPath=/admin-console
15:14:20,412 INFO [config] Initializing Mojarra (1.2_12-b01-FCS) for context '/admin-console'
15:14:23,486 INFO [TomcatDeployment] deploy, ctxPath=/BannedListSearch
15:14:27,532 INFO [TomcatDeployment] deploy, ctxPath=/IWorkWebApp
15:14:27,813 INFO [TomcatDeployment] deploy, ctxPath=/
15:14:29,155 INFO [TomcatDeployment] deploy, ctxPath=/TestWebProject
15:14:30,036 INFO [TomcatDeployment] deploy, ctxPath=/displaytag-examples-1.2
15:14:30,136 INFO [TomcatDeployment] deploy, ctxPath=/jmx-console
15:14:30,276 INFO [TomcatDeployment] deploy, ctxPath=/HelloWebService
15:14:30,407 ERROR [EngineConfigurationFactoryServlet] Unable to find config file. Creating new servlet engine config file: /WEB-INF/server-config.wsdd
15:14:30,687 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on http-127.0.0.1-8081
15:14:30,707 INFO [AjpProtocol] Starting Coyote AJP/1.3 on ajp-127.0.0.1-8009
15:14:30,707 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221053)] Started in 48s:110ms
I have increased start Timeout of server to 500 seconds then also I am getting same error. I have not changed anything else.
I am able to start JBoss from command prompt successfully but same server is not getting started from eclipse.
Please help me to start the JBoss server.
Sound to me like the http port you are configured in JBoss is different to the port you have in the Eclipse configuration for JBoss.
Eclipse uses the port configuration to listen to JBoss' port so that it can determine that JBoss has actually started. If they differ, Eclipse thinks JBoss has never started although it actually has according to the log console. Make the ports match and it will probably work.
Updated: According to your log, JBoss is using port 8081 for HTTP:
Starting Coyote HTTP/1.1 on http-127.0.0.1-8081
Now you have to tell Eclipse to listen to that port so that it can figure out whether JBoss has started (default is 8080 and therefore Eclipse will never be aware of it!). Go to your servers view, double click on your JBoss server, and the configuration screen will come up:
You have to edit the HTTP port (in the 'Port' box) and set it to 8081 so that it matches your server's.