CATALINA_PID was set but the specified file does not exist - ubuntu-16.04

I was following this guide to setup tomcat on Ubuntu 16.04.2 LTS Xenial (https://www.digitalocean.com/community/tutorials/how-to-install-apache-tomcat-8-on-ubuntu-16-04).
Got to the point of the first launch:
sudo systemctl start tomcat
and received an error:
~$ sudo systemctl start tomcat
Job for tomcat.service failed because the control process exited with error code. See "systemctl status tomcat.service" and "journalctl -xe" for details.
journalctl -xe
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit tomcat.service has begun starting up.
Feb 27 15:23:42 76672.local startup.sh[20551]: Tomcat started.
Feb 27 15:23:42 76672.local shutdown.sh[20563]: $CATALINA_PID was set but the specified file does not exist. Is Tomcat running? Stop aborted.
Feb 27 15:23:42 76672.local systemd[1]: tomcat.service: Control process exited, code=exited status=1
Feb 27 15:23:42 76672.local systemd[1]: Failed to start Tomcat 9 servlet container.
-- Subject: Unit tomcat.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit tomcat.service has failed.
--
-- The result is failed.
CATALINA_PID expected to be in a directory /opt/tomcat/latest/temp/tomcat.pid
I gave rights to do everyone for everything for this dir /opt/tomcat and all subfolders/files:
~$ namei -l /opt/tomcat
f: /opt/tomcat
drwxr-xr-x root root /
drwxr-xr-x root root opt
drwxrwxrwx tomcat tomcat tomcat
Does anyone can point me to the possible solutions?
UPD:
Using bin/startup.sh Tomcat successfully starts.

When you use CATALINA_PID in tomcat.service file please make sure to use the path inside of the double-quotes.
Environment="CATALINA_PID=/opt/tomcat/latest/temp/tomcat.pid"
Like below
[Unit]
Description=Tomcat 9 servlet container
After=network.target
[Service]
Type=forking
User=tomcat
Group=tomcat
Environment="/usr/lib/jvm/java-1.8.0-openjdk-amd64"
Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom -Djava.awt.headless=true"
Environment="CATALINA_BASE=/opt/tomcat/"
Environment="CATALINA_HOME=/opt/tomcat/"
Environment="CATALINA_PID=/opt/tomcat/temp/tomcat.pid"
Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"
ExecStart=/opt/tomcat/bin/startup.sh
ExecStop=/opt/tomcat/bin/shutdown.sh
[Install]
WantedBy=multi-user.target
congratulations!

Related

Service starting gunicorn failing with "Start request repeated too quickly"

Trying to start a service to run gunicorn as backend server for Flask, not working. Running nginx as frontend server for React, working.
Server:
Virtualization: vmware
Operating System: Red Hat Enterprise Linux 8.4 (Ootpa)
CPE OS Name: cpe:/o:redhat:enterprise_linux:8.4:GA
Kernel: Linux 4.18.0-305.3.1.el8_4.x86_64
Architecture: x86-64
Service file in /etc/systemd/system/myservice.service:
[Unit]
Description="Description"
After=network.target
[Service]
User=root
Group=root
WorkingDirectory=/home/project/app/api
ExecStart=/home/project/app/api/venv/bin/gunicorn -b 127.0.0.1:5000 api:app
Restart=always
[Install]
WantedBy=multi-user.target
/app/api:
-rwxr-xr-x. 1 root root 2018 Jun 9 20:06 api.py
drwxrwxr-x+ 5 root root 100 Jun 7 10:11 venv
Error message:
● myservice.service - "Description"
Loaded: loaded (/etc/systemd/system/myservice.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2021-06-10 19:01:01 CEST; 5s ago
Process: 18307 ExecStart=/home/project/app/api/venv/bin/gunicorn -b 127.0.0.1:5000 api:app (code=exited, status=203/EXEC)
Main PID: 18307 (code=exited, status=203/EXEC)
Jun 10 19:01:01 xxxx systemd[1]: myservice.service: Service RestartSec=100ms expired, scheduling restart.
Jun 10 19:01:01 xxxx systemd[1]: myservice.service: Scheduled restart job, restart counter is at 5.
Jun 10 19:01:01 xxxx systemd[1]: Stopped "Description".
Jun 10 19:01:01 xxxx systemd[1]: myservice.service: Start request repeated too quickly.
Jun 10 19:01:01 xxxx systemd[1]: myservice.service: Failed with result 'exit-code'.
Jun 10 19:01:01 xxxx systemd[1]: Failed to start "Description".
Tried, not working:
Adding Environment="PATH=/home/project/app/api/venv/bin" under [Service]
$ systemctl reset-failed myservice.service
$ systemctl daemon-reload
Reboot, ofc.
Tried, working:
Running (as root) /home/project/app/api/venv/bin/gunicorn -b 127.0.0.1:5000 api:app while in /app/api directory
Does anyone know how to fix this problem?
Typically enough, I figured it out shortly after posting this issue.
SELinux is messing with permissions for files and directories, so for anyone experiencing the same issue, make sure to test with the following alterings (as root):
$ setsebool -P httpd_can_network_connect on
$ chcon -Rt httpd_sys_content_t /path/to/your/Flask/dir
In my case: $ chcon -Rt httpd_sys_content_t /home/project/app/api
While this is NOT a permanent fix, it's worth a try. Check out the SELinux docs for more permanent solutions.

Should systemctl show output when start/stop fails?

I've googled every variation of this question I can think of, but I'm just getting questions about failed services, not about how systemctl treats them. I have a service that I've been running as an init.d script. We're using systemctl now, fine. I created a service file that's a lightly modified version of the file automatically generated by systemd-sysv-generator. For ExecStart and ExecStop, it calls a bash script that returns 0 if the start/stop was successful, and non-zero if it was not.
I understand that there's no output from "systemctl start/stop" if it was successful. But I also don't get any output if either of the calls failed. The return code of the systemctl start/stop command is always 0 even if the return code of the source script is not. It's very clear it did fail because it shows as failed when I run the status command.
Is that expected behavior? Should it not give any indication that something failed unless you run a separate status command? And if that's not how it should behave, how can I make it indicate that a failure occurred?
Service file below.
[Unit]
SourcePath=/my/service/script.sh
[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/my/service/script.sh start
ExecStop=/my/service/script.sh stop
Working fine here, CentOS 7. Maybe double check that script.sh is really returning non zero?
$pwd
/etc/systemd/system
$cat me.service
[Unit]
[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/systemd/system/die.sh
$cat die.sh
#!/bin/bash
echo "dying"
exit 1
$sudo systemctl start me
Job for me.service failed because the control process exited with error code. See "systemctl status me.service" and "journalctl -xe" for details.
$sudo systemctl status me.service
● me.service
Loaded: loaded (/etc/systemd/system/me.service; static; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2019-09-12 17:55:02 GMT; 8s ago
Process: 19758 ExecStart=/etc/systemd/system/die.sh (code=exited, status=1/FAILURE)
Sep 12 17:55:02 dpsdev-wkr01 systemd[1]: Starting me.service...
Sep 12 17:55:02 dpsdev-wkr01 die.sh[19758]: dying
Sep 12 17:55:02 dpsdev-wkr01 systemd[1]: me.service: control process exited, code=exited status=1
Sep 12 17:55:02 dpsdev-wkr01 systemd[1]: Failed to start me.service.
Sep 12 17:55:02 dpsdev-wkr01 systemd[1]: Unit me.service entered failed state.
Sep 12 17:55:02 dpsdev-wkr01 systemd[1]: me.service failed.

Mongodb fails to start -> presents weird error logs

When I moved my environment from my local (mac) to my server (ubuntu) I unzipped my directory and the server installed with npm install with no errors or warnings, but my database was failing so I decided to reinstall it based on this tutorial (well, apt-remove mongo* first)
https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04
but then I get a
Job for mongodb.service failed because the control process exited with error code. See "systemctl status mongodb.service" and "journalctl -xe" for details.
Does anyone know what any of this means?
-- Unit mongodb.service has begun starting up.
Jun 20 03:54:18 ip-172-31-16-163 mongodb[25271]: * Starting database mongodb
Jun 20 03:54:19 ip-172-31-16-163 mongodb[25271]: ...fail!
Jun 20 03:54:19 ip-172-31-16-163 systemd[1]: mongodb.service: Control process exited, code=exited status=1
Jun 20 03:54:19 ip-172-31-16-163 sudo[25268]: pam_unix(sudo:session): session closed for user root
Jun 20 03:54:19 ip-172-31-16-163 systemd[1]: Failed to start LSB: An object/document-oriented database.
-- Subject: Unit mongodb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mongodb.service has failed.
--
-- The result is failed.
Jun 20 03:54:19 ip-172-31-16-163 systemd[1]: mongodb.service: Unit entered failed state.
Jun 20 03:54:19 ip-172-31-16-163 systemd[1]: mongodb.service: Failed with result 'exit-code'.
Looks familiar. Check ownership of files. Files in dbPath, mongod.run -lock file, keyfile...
Basically all those files what are listed at your /etc/mongod.conf
Run the following command and Its works for me
sudo apt-get install --reinstall mongodb

Cannot create systemd script for Puma

I created a service script named "puma.service" in /etc/systemd/system/ with the following contents:
[Unit]
Description=Puma HTTP Server
After=network.target
[Service]
Type=simple
User=ubuntu
WorkingDirectory=/home/username/appdir/current
ExecStart=/bin/bash -lc "/home/username/appdir/current/sbin/puma -C /home/username/appdir/current/config/puma.rb /home/username/appdir/current/config.ru"
Restart=always
[Install]
WantedBy=multi-user.target
I enabled the service and when started, I'm getting the following log from systemctl:
● puma.service - Puma HTTP Server
Loaded: loaded (/etc/systemd/system/puma.service; enabled; vendor preset: enabled)
Active: inactive (dead) (Result: exit-code) since Wed 2016-12-14 10:09:46 UTC; 12min ago
Process: 16889 ExecStart=/bin/bash -lc cd /home/username/appdir/current && bundle exec puma -C /home/username/appdir..
Main PID: 16889 (code=exited, status=127)
Dec 14 10:09:46 ip-172-31-29-40 systemd[1]: puma.service: Main process exited, code=exited, status=127/n/a
Dec 14 10:09:46 ip-172-31-29-40 systemd[1]: puma.service: Unit entered failed state.
Dec 14 10:09:46 ip-172-31-29-40 systemd[1]: puma.service: Failed with result 'exit-code'.
Dec 14 10:09:46 ip-172-31-29-40 systemd[1]: puma.service: Service hold-off time over, scheduling restart.
Dec 14 10:09:46 ip-172-31-29-40 systemd[1]: Stopped Puma HTTP Server.
Dec 14 10:09:46 ip-172-31-29-40 systemd[1]: puma.service: Start request repeated too quickly.
Dec 14 10:09:46 ip-172-31-29-40 systemd[1]: Failed to start Puma HTTP Server.
Although, when I give the command in SSH terminal the server started and was running perfect. Is there any changes I have to make in the service file?
Note:
I have changed the dirnames for your convenience.
I did some research and the cause of status 127 is due to the executable not in the path. But, I guess that won't be a problem.
Can you shed some light?
I found the problem and changed the ExecStart as mentioned below and it worked like a charm:
ExecStart=/home/username/.rbenv/shims/bundle exec puma -e production -C ./config/puma.rb config.ru
PIDFile=/home/username/appdir/shared/tmp/pids/puma.pid
bundle should be taken from the rbenv shims and also the puma's config file (config/puma.rb) and application's config file (config.ru) can be given in relative path.
One way to solve it is to specify a PID file and systemd will take a look at that file to check on service status.
Here's how we use this in our scripts (adapted to your given sample)
ExecStart=/bin/bash -lc '/home/username/appdir/current/sbin/puma -C /home/username/appdir/current/config/puma.rb /home/username/appdir/current/config.ru --pidfile /home/username/appdir/current/tmp/pids/puma.pid'
PIDFile=/home/username/appdir/current/tmp/pids/puma.pid
Take note that you might have to configure --pidfile via your -C puma.rb file instead of passing it an as parameter. I'm just showing it here to illustrate that --pidfile (in puma config) should be the same as PIDFile in the service file.
As for why the error message is that way, I'm not sure myself and is interested in the answer too.
For rvm users try with
[Unit]
Description=Puma HTTP Server
After=network.target
[Service]
Type=simple
User=user_name
Group=user_name
WorkingDirectory=/home/user_name/apps/app_name/current
Environment=RAILS_ENV=production
ExecStart=/home/user_name/.rvm/bin/rvm ruby-3.1.2#app_name do bundle exec --keep-file-descriptors puma -C /home/user_name/apps/app_name/current/config/puma/production.rb
ExecReload=/bin/kill -USR1 $MAINPID
StandardOutput=append:/home/user_name/apps/app_name/current/log/puma_access.log
StandardError=append:/home/user_name/apps/app_name/current/log/puma_error.log
SyslogIdentifier=app_name-puma
Restart=always
KillMode=process
[Install]
WantedBy=multi-user.target
if you are not using a gemset change
ExecStart=/home/user_name/.rvm/bin/rvm ruby-3.1.2#app_name ....
to
ExecStart=/home/user_name/.rvm/bin/rvm default

postmaster.pid permission denied CentOS 7

Postgres 9.2 on CentOS 7.
After "su - postgres" I installed using
pg-ctl initdb -D /var/lib/pgsql/data
which ran fine.
[root#server ~]# systemctl start postgresql
Job for postgresql.service failed. See 'systemctl status postgresql.service' and 'journalctl -xn' for details.
[root#server ~]# systemctl status postgresql.service
postgresql.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; disabled)
Active: failed (Result: exit-code) since Fri 2015-11-27 13:48:57 EST; 9s ago
Process: 3262 ExecStart=/usr/bin/pg_ctl start -D ${PGDATA} -s -o -p ${PGPORT} -w -t 300 (code=exited, status=1/FAILURE)
Process: 3256 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
Nov 27 13:48:57 server.company.network systemd[1]: Starting PostgreSQL database server...
Nov 27 13:48:57 server.company.network pg_ctl[3262]: pg_ctl: could not open PID file "/var/lib/pgsql/data/postmaster.pid": Permission denied
Nov 27 13:48:57 server.company.network systemd[1]: postgresql.service: control process exited, code=exited status=1
Nov 27 13:48:57 server.company.network systemd[1]: Failed to start PostgreSQL database server.
Nov 27 13:48:57 server.company.network systemd[1]: Unit postgresql.service entered failed state.
[root#server ~]# journalctl -xn
-- Logs begin at Fri 2015-11-27 13:29:37 EST, end at Fri 2015-11-27 13:48:57 EST. --
Nov 27 13:48:35 server.company.network sudo[3228]: pam_unix(sudo:auth): conversation failed
Nov 27 13:48:35 server.company.network sudo[3228]: pam_unix(sudo:auth): auth could not identify password for [myuserid]
Nov 27 13:48:46 server.company.network sudo[3230]: myuserid : TTY=pts/0 ; PWD=/home/myuserid ; USER=root ; COMMAND=/bin/su -
Nov 27 13:48:46 server.company.network su[3234]: (to root) myuserid on pts/0
Nov 27 13:48:46 server.company.network su[3234]: pam_unix(su-l:session): session opened for user root by myuserid(uid=0)
Nov 27 13:48:57 server.company.network systemd[1]: Starting PostgreSQL database server...
-- Subject: Unit postgresql.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql.service has begun starting up.
Nov 27 13:48:57 server.company.network pg_ctl[3262]: pg_ctl: could not open PID file "/var/lib/pgsql/data/postmaster.pid": Permission denied
Nov 27 13:48:57 server.company.network systemd[1]: postgresql.service: control process exited, code=exited status=1
Nov 27 13:48:57 server.company.network systemd[1]: Failed to start PostgreSQL database server.
-- Subject: Unit postgresql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql.service has failed.
--
-- The result is failed.
Nov 27 13:48:57 server.company.network systemd[1]: Unit postgresql.service entered failed state.
When I "su - postgres" I can "touch" the file, "ls" the file, "rm" /var/lib/pgsql/data/postmaster.pid. Permissions on data are 700 postgres:postgres. pgsql is a symlink to /data0/postgres and postgres is 700 postgres:postgres.
ADDITIONS:
I forgot to mention that after having this problem, I replaced the commands for ExecStartPre and ExecStart with shell scripts that wrote the user, primary group, PGDATA, and PGPORT values to a file. They were all correct. The start still died on postmaster.pid .
The postgresql.service file:
[root#server /]# cat /usr/lib/systemd/system/postgresql.service
# It's not recommended to modify this file in-place, because it will be
# overwritten during package upgrades. If you want to customize, the
# best way is to create a file "/etc/systemd/system/postgresql.service",
# containing
# .include /lib/systemd/system/postgresql.service
# ...make your changes here...
# For more info about custom unit files, see
# http://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add_a_custom_unit_file.3F
# For example, if you want to change the server's port number to 5433,
# create a file named "/etc/systemd/system/postgresql.service" containing:
# .include /lib/systemd/system/postgresql.service
# [Service]
# Environment=PGPORT=5433
# This will override the setting appearing below.
# Note: changing PGPORT or PGDATA will typically require adjusting SELinux
# configuration as well; see /usr/share/doc/postgresql-*/README.rpm-dist.
# Note: do not use a PGDATA pathname containing spaces, or you will
# break postgresql-setup.
# Note: in F-17 and beyond, /usr/lib/... is recommended in the .include line
# though /lib/... will still work.
[Unit]
Description=PostgreSQL database server
After=network.target
[Service]
Type=forking
User=postgres
Group=postgres
# Port number for server to listen on
Environment=PGPORT=5432
# Location of database directory
Environment=PGDATA=/var/lib/pgsql/data
# Where to send early-startup messages from the server (before the logging
# options of postgresql.conf take effect)
# This is normally controlled by the global default set by systemd
# StandardOutput=syslog
# Disable OOM kill on the postmaster
OOMScoreAdjust=-1000
ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGDATA}
ExecStart=/usr/bin/pg_ctl start -D ${PGDATA} -s -o "-p ${PGPORT}" -w -t 300
ExecStop=/usr/bin/pg_ctl stop -D ${PGDATA} -s -m fast
ExecReload=/usr/bin/pg_ctl reload -D ${PGDATA} -s
# Give a reasonable amount of time for the server to start up/shut down
TimeoutSec=300
[Install]
WantedBy=multi-user.target
I figured it out. After running initdb, I copied the data directory to the other drive. With SELinux, the FILETYPE switches to the target parent directory FILETYPE. I tried to semanage the directory, but that wasn't working. So I started over again and moved the data directory instead, which maintained the FILETYPE.