Abnormal termination error running supervisor [closed] - supervisord

Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
I'm trying to use supervisor with nginx, but I haven't been able to get to the nginx part yet because I'm screwing something up with supervisor.
I can run
sudo supervisorctl reread
and it will say "site_demo: available". Then I run
sudo supervisorctl update
and it says "site_demo: updated process group". So that seems fine. But then I try to run
sudo supervisorctl start site_demo
and I get the error
site_demo: ERROR (abnormal termination)
Here is my supervisor.conf file:
; supervisor config file
[unix_http_server]
file=/var/run//supervisor.sock ; (the path to the socket file)
chmod=0700 ; sockef file mode (default 0700)
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run//supervisor.sock ; use a unix:// URL for a unix socket
; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
[include]
files = /etc/supervisor/conf.d/*.conf
And here is demo.conf, the one file included under [include]:
[program:site_demo]
directory=/home/snorthway/demo/website
environment=PYTHONPATH=/home/snorthway/python-envs/demo
command=/home/snorthway/python-envs/demo/bin/python /home/snorthway/demo/website/manage.py runfcgi method=prefork socket=/home/snorthway/eft/demo/fcgi/socket pidfile=/home/snorthway/eft/demo/fcgi/demo.pid minspare=4 maxspare=10 daemonize=false
redirect_stderr=true
stdout_logfile=/home/snorthway/eft/demo/logs/demo.fcgi.log
autostart=true
user=www-data
I checked that all of the paths in there exist and whatnot, so I don't think that's the problem.
My log file looks like this:
2013-06-24 15:40:23,548 CRIT Supervisor running as root (no user in config file)
2013-06-24 15:40:23,566 INFO RPC interface 'supervisor' initialized
2013-06-24 15:40:23,566 WARN cElementTree not installed, using slower XML parser for XML-RPC
2013-06-24 15:40:23,566 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2013-06-24 15:40:23,566 INFO daemonizing the supervisord process
2013-06-24 15:40:23,566 INFO supervisord started with pid 8896
2013-06-24 16:08:27,262 INFO spawned: 'site_demo' with pid 9221
2013-06-24 16:08:27,464 INFO exited: site_demo (exit status 0; not expected)
2013-06-24 16:08:28,469 INFO spawned: 'site_demo' with pid 9226
2013-06-24 16:08:28,667 INFO exited: site_demo (exit status 0; not expected)
2013-06-24 16:08:30,672 INFO spawned: 'site_demo' with pid 9231
2013-06-24 16:08:30,863 INFO exited: site_demo (exit status 0; not expected)
2013-06-24 16:08:33,870 INFO spawned: 'site_demo' with pid 9240
2013-06-24 16:08:34,077 INFO exited: site_demo (exit status 0; not expected)
2013-06-24 16:08:35,079 INFO gave up: site_demo entered FATAL state, too many start retries too quickly

Well, if I had looked at demo.fcgi.log I would have figured this out a long time ago. It consisted entirely of this error:
ERROR: No module named flup
Unable to load the flup package. In order to run django
as a FastCGI application, you will need to get flup from
http://www.saddi.com/software/flup/ If you've already
installed flup, then make sure you have it in your PYTHONPATH.
So I did
pip install flup
sudo supervisorctl reread # This may not have been necessary
sudo supervisorctl update
sudo supervisorctl start site_demo
and it worked!

Related

Daphne keeps exiting without any error under supervisor

I am running daphne under supervisord following the official docs.It keeps failing for some reason wihout any clear error
Below is the supervisor log
2021-07-18 15:35:01,110 INFO Creating socket tcp://localhost:8000
2021-07-18 15:35:01,112 INFO spawned: 'asgi0' with pid 15075
2021-07-18 15:35:01,116 INFO exited: asgi0 (exit status 0; not expected)
2021-07-18 15:35:01,116 INFO Closing socket tcp://localhost:8000
2021-07-18 15:35:04,128 INFO Creating socket tcp://localhost:8000
2021-07-18 15:35:04,130 INFO spawned: 'asgi0' with pid 15079
2021-07-18 15:35:04,134 INFO exited: asgi0 (exit status 0; not expected)
2021-07-18 15:35:04,134 INFO Closing socket tcp://localhost:8000
2021-07-18 15:35:05,136 INFO gave up: asgi0 entered FATAL state, too many start retries too quickly
Here's the supervisor config file
[fcgi-program:asgi]
# TCP socket used by Nginx backend upstream
socket=tcp://localhost:8000
# environment=DJANGO_SETTINGS_MODULE=mysite.dev_settings
# Directory where your site's project files are located
directory=/home/jaga/C42
# Each process needs to have a separate socket file, so we use process_num
# Make sure to update "mysite.asgi" to match your project name
command=bash -c 'source /etc/environment' && daphne -u /run/daphne/daphne%(process_num)d.sock --fd 10 --access-log - --proxy-headers mysite.asgi:application
# Number of processes to startup, roughly the number of CPUs you have
numprocs=1
# Give each process a unique name so they can be told apart
process_name=asgi%(process_num)d
# Automatically start and recover processes
autostart=true
autorestart=true
# Choose where you want your log to go
stdout_logfile=/var/log/asgi.log
redirect_stderr=true
Same happened when there were 4 processes.

Supervisord (exit status 2; not expected) ubuntu

I'm trying to run Celery with Supervisord on Ubuntu, but am getting:
INFO exited: celery (exit status 2; not expected)
INFO spawned: 'celery' with pid 15517
INFO gave up: celery entered FATAL state, too many start retries too
quickly
This is the Supervisord script:
cd into the directory and activate the virtual environment
celery -A [APP_NAME].celery worker -E -l info --concurrency=2
If I run this script manually, Celery starts up without any issues. But running sudo supervisorctl start celery errors out with the error messages above.

Eclipse using MinGW make fails without msys

As background, I'm trying to build ChibiOS for STM32 on a Windows 8.1 host. This works perfectly well if I simply run make in the demo directory in the msys.bat command prompt. The toolchain and paths should thus be fine.
Now, if I simply set up an Eclipse project, it will try to run make.exe directly and fails. The output is similar to running make (either make.exe or mingw32-make.exe) from a plain cmd prompt.
make all
0 [main] sh 5524 sync_with_child: child 2444(0x188) died before initialization with status code 0xC0000142
22 [main] sh 5524 sync_with_child: *** child state waiting for longjmp
/usr/bin/sh: fork: Resource temporarily unavailable
0 [main] sh 188 sync_with_child: child 1152(0x188) died before initialization with status code 0xC0000142
26 [main] sh 188 sync_with_child: *** child state waiting for longjmp
/usr/bin/sh: fork: Resource temporarily unavailable
0 [main] sh 5096 sync_with_child: child 3200(0x18C) died before initialization with status code 0xC0000142
25 [main] sh 5096 sync_with_child: *** child state waiting for longjmp
/usr/bin/sh: fork: Resource temporarily unavailable
0 [main] sh 5232 sync_with_child: child 3820(0x184) died before initialization with status code 0xC0000142
25 [main] sh 5232 sync_with_child: *** child state waiting for longjmp
/usr/bin/sh: fork: Resource temporarily unavailable
make: Nothing to be done for 'all'.
20:39:33 Build Finished (took 4s.171ms)
I've seen some info saying this is some aspect of Windows 8.1. Can I convince Eclipse to use msys somehow or is there another known clean way to make a make (any make) work without it?
Possibly related:
http://forum.chibios.org/phpbb/viewtopic.php?p=16023
http://sourceforge.net/p/mingw/bugs/1013/?page=0
Bizarrely, the issue was solved for me by replacing msys-1.0.dll in WinAVR directory with the msys one. I'm guessing there is an ancient version there that somehow gets loaded while it's not in the system path as far as I can tell.
The links in the question refer to updating the dll or replacing it with a patched one.

dotcloud supervisord twistd exiting early

I'm trying to set up a twistd daemon on dotcloud:
My supervisord.conf file:
[program:apnsd]
command=/home/dotcloud/env/bin/twistd --logfile /var/log/supervisor/apnsd.log apnsd -c gp_config.py
directory=/home/dotcloud/current/apnsd
However, it looks like the command 'exits early', which then prompts Supervisor to try and restart, which then fails because the twistd dameon is running in the background.
From the supervisord log:
more supervisord.log
2012-05-19 03:07:52,723 CRIT Set uid to user 1000
2012-05-19 03:07:52,723 WARN Included extra file "/etc/supervisor/conf.d/uwsgi.c
onf" during parsing
2012-05-19 03:07:52,723 WARN Included extra file "/home/dotcloud/current/supervi
sord.conf" during parsing
2012-05-19 03:07:52,922 INFO RPC interface 'supervisor' initialized
2012-05-19 03:07:52,922 WARN cElementTree not installed, using slower XML parser
for XML-RPC
2012-05-19 03:07:52,923 CRIT Server 'unix_http_server' running without any HTTP
authentication checking
2012-05-19 03:07:52,932 INFO daemonizing the supervisord process
2012-05-19 03:07:52,934 INFO supervisord started with pid 144
2012-05-19 03:07:53,941 INFO spawned: 'apnsd' with pid 147
2012-05-19 03:07:53,949 INFO spawned: 'uwsgi' with pid 149
2012-05-19 03:07:54,706 INFO exited: apnsd (exit status 0; not expected)
2012-05-19 03:07:55,712 INFO spawned: 'apnsd' with pid 175
2012-05-19 03:07:55,712 INFO success: uwsgi entered RUNNING state, process has s
tayed up for > than 1 seconds (startsecs)
2012-05-19 03:07:56,261 INFO exited: apnsd (exit status 1; not expected)
2012-05-19 03:07:58,267 INFO spawned: 'apnsd' with pid 176
2012-05-19 03:07:58,783 INFO exited: apnsd (exit status 1; not expected)
2012-05-19 03:08:01,790 INFO spawned: 'apnsd' with pid 177
2012-05-19 03:08:02,840 INFO success: apnsd entered RUNNING state, process has s
tayed up for > than 1 seconds (startsecs)
From the apnsd log:
dotcloud#hack-default-www-0:/var/log/supervisor$ more apnsd-stderr---supervisor
-m7GnKV.log
INFO:root:Reactor Type: <twisted.internet.pollreactor.PollReactor object at 0x10
a09d0>
DEBUG:root:Creating listener: apnsd.listeners.line.LineProtocolFactory
INFO:root:Listening on Line Protocol on :1055
DEBUG:root:Listener Created: <apnsd.listeners.line.LineProtocolFactory instance
at 0x12fc8c0>
DEBUG:root:Creating App Factory: apnsd.daemon.APNSFactory
INFO:root:Connecting to APNS Server, App: apns_dev:AAA.com.company.www
INFO:root:apns_dev:AAA.com.company.www -> Started connecting to APNS con
nector...
INFO:root:Registering Application: apns_dev:GoParcel...
DEBUG:root:Creating App Factory: apnsd.daemon.APNSFactory
INFO:root:Connecting to APNS Server, App: apns_dev:T365ED94A9.com.appitems.parce
ls
INFO:root:apns_dev:T365ED94A9.com.appitems.parcels -> Started connecting to APNS
connector...
INFO:root:Registering Application: apns_dev:GoParcelVictor...
Another twistd server is running, PID 172
This could either be a previously started instance of your application or a
different application entirely. To start a new one, either run it in some other
directory, or use the --pidfile and --logfile parameters to avoid clashes.
Another twistd server is running, PID 172
--More--(42%)
Status of worker is failed:
./dotcloud run hack.worker supervisorctl status
USER PATH IS: C:\Users\Taras/.dotcloud\dotcloud.conf
# supervisorctl status
apnsd FATAL Exited too quickly (process log may
have details)
But the twistd process is there (ps -ef):
dotcloud 171 1 0 03:13 ? 00:00:00 /home/dotcloud/env/bin/python /home/dotcloud/env/bin/twistd --logfile /var/log/supervisor/apnsd.log apnsd -c gp_config.py
I am having a similar problem when trying to start the process through a wrapper script (and using exec so that a child process isn't created). What am I doing wrong?
Supervisor expects the controlled process to remain in the foreground, but twistd forks to the background by default. Supervisor therefore thinks that it has exited, and tries to start it again.
You should start twistd with the --nodaemon option: twistd will remain in the foreground, and Supervisor will be happy!

Supervisord does not start killed processes

I have supervisord installed on my Ubuntu 10.04 and it runs a Java process continuously and supposed to heal (reload) process when it somehow dies or crashes.
On my htop I send SIGKILL, SIGTERM, SIGHUP, SIGSEGV signals to that Java process and watch /etc/logs/supervisord.log file and it says.
08:09:46,182 INFO success: myprogram entered RUNNING state,[...]
08:38:10,043 INFO exited: myprogram (exit status 0; expected)
At 08:38 I kill the process with SIGSEGV. How come it is exited with code 0 and why does not supervisord restart it at all?
All my supervisord.conf about this specific program is as follows:
[program:play-9000]
command=play run /var/www/myprogram/ --%%prod
stderr_logfile = /var/log/supervisord/myprogram-stderr.log
stdout_logfile = /var/log/supervisord/myprogram-stdout.log
Process works really fine when I launch supervisord, however does not get healed.
By the way any ideas how to start supervisord as a service so that it automatically launches when the whole system reboots?
Try setting autorestart=true. By default, autorestart is set to "unexpected" which means it will only restart a process if it exits with an unexpected exit code. By default, exit code 0 is expected.
http://supervisord.org/configuration.html#program-x-section-settings
You can use the chkconfig program to make sure that supervisor starts on reboot.
$ sudo apt-get install chkconfig
$ chkconfig -l supervisor
supervisor 0:off 1:off 2:on 3:on 4:on 5:on 6:off
You can see that it's enabled for runlevels 2-5 by default when I installed it.
$ man 7 runlevel
for more info on run levels.