Supervisorctl not respecting my configuration - supervisord

I have set the following in /home/david/conf/supervisor.conf:
[unix_http_server]
file=/home/david/tmp/supervisor.sock
[supervisord]
logfile=/home/david/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/home/david/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200
childlogdir=/home/david/tmp
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///home/david/tmp/supervisor.sock
And started supervisord:
$ supervisord -c /home/david/conf/supervisor.conf
However how come supervisorctl still uses the default http://localhost:9001 as the serverurl?
$ supervisorctl
http://localhost:9001 refused connection
supervisor>
I checked /home/david/tmp and the files supervisord.log and supervisord.pid do exist.

You should run supervisorctl with -c as well. From the documentation (my emphasis):
The Supervisor configuration file is conventionally named
supervisord.conf. It is used by both supervisord and supervisorctl. If
either application is started without the -c option (the option which
is used to tell the application the configuration filename
explicitly), the application will look for a file named
supervisord.conf within the following locations, in the specified
order. It will use the first file it finds.
$CWD/supervisord.conf
$CWD/etc/supervisord.conf
/etc/supervisord.conf

In MacOS, use brew to install
brew install supervisor
Then go to /usr/local/etc/supervisord.ini and comment these lines:
;[unix_http_server]
;file=/usr/local/var/run/supervisor.sock ; the path to the socket file
and uncomment these lines:
[inet_http_server] ; inet (TCP) server disabled by default
port=127.0.0.1:9001 ; ip_address:port specifier, *:port for all iface
Finally restart the daemon:
brew services restart supervisor
That's all you need.

To add to the valid answer above make sure you are putting your config files for the apps you want to monitor under supervisor's config folder as a subfolder called conf.d. This will depend of what method you use to install supervisor, the default package manager or easy_install.

As solution you can make symbolic link to the config file.
Like this for Mac OS:
sudo ln -sv /usr/local/etc/supervisord.ini /etc/supervisord.conf

Related

Supervisord not releasing log file during rotation

I've got a pair of CentOS 7 servers that are running SupervisorD with one program. Supervisord is not set to handle any log rotations.. supervisord.conf's lines related to size and backups both = 0.
supervisord.conf
[unix_http_server]
file=/var/run/dir1/supervisor.sock ; the path to the socket file
[supervisord]
logfile=/var/log/dir1/supervisord.log ; main log file; default $CWD/supervisord.log
logfile_maxbytes=0
; logfile_maxbytes=50MB ; max main logfile bytes b4 rotation; default 50MB
logfile_backups=0
;logfile_backups=10 ; # of main logfile backups; 0 means none, default 10
loglevel=trace ; log level; default info; others: debug,warn,trace
pidfile=/var/run/dir1/supervisord.pid ; supervisord pidfile; default supervisord.pid
nodaemon=false ; start in foreground if true; default false
minfds=786068
minprocs=200 ; min. avail process descriptors;default 200
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/dir1/supervisor.sock ; use a unix:// URL for a unix socket
[include]
files = /etc/supervisord.d/*.conf
program1.conf
[program:jobengine]
; Set full path to Job Engine program if using virtualenv
command=python /opt/program1/jobengine/jobengine.pyc --namespace=worker%(process_num)02d
environment=PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:
directory=/opt/program1
user=root
numprocs=4
process_name=%(process_num)01d
stdout_logfile=/var/log/dir1/jobengine.log
stderr_logfile=/var/log/dir1/jobengine.log
autostart=true
autorestart=true
startsecs=10
startretries=999
; Below lines added to ensure supervisord does not perform any log handling, in favor of logrotate
stdout_logfile_maxbytes=0
stderr_logfile_maxbytes=0
stdout_logfile_backups=0
stderr_logfile_backups=0
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; Causes supervisor to send the termination signal (SIGTERM) to the whole process group.
stopasgroup=false
; default (999)
priority=999
logrotate/program1.conf
# Track main logs, jobengine logs, etc.
/var/log/dir1/*log {
# Run as the Apache user
su apache apache
# CloudBolt will create the log file
missingok
# Do not rotate empty logs
notifempty
# Do not compress logs
nocompress
# Use `*.log.1` naming instead of `*.log-YYYYMMDD` format
nodateext
# Keep 5 archived logs
rotate 5
# Rotate files larger than this size
size 5M
}
So what happens is logrotate kicks off at it's appropriate time, however supervisord refuses to move from the old to the new log file. I'm used to the logrotate having a postscript for a restart signal however I'm 100% new to supervisord so I've been trying to verify how to gracefully restart supervisord and it's child processes. The way I understand it, 'stopwaitsecs' force supervisord to wait the specified time if busy and when I test the 4 child pids are killed via SIGTERM immediately.
What I need is supervisord to only restart once the child pids have completed w/e they're doing.
I also, on a whim, tried sending supervisord a SIGUSR2, and also a supervisorctl reload.. everything seems to kill the processes immediately.
What am I doing wrong here?

How to auto-reload (start) docker container with supervisor

I'm stuck with a problem where I can't perfectly start my container via supervisor.
I have a mongodb container, which I want to run under supervisor.
Supervisor config itself:
; supervisor config file
[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
chmod=0700 ; sockef file mode (default 0700)
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
[include]
files = /etc/supervisor/conf.d/*.conf
and mongodb container conf file:
[program:mongo]
command=/usr/bin/docker start b1b -DFOREGROUND
autostart=true
autorestart=true
startretries=10
exitcodes=0
startsecs=0
When I do "supervisorctl restart mongo" it will start container, but after I stop it it will not relaunch it automatically.
Any advice on it?
Docker already has the facility to restart stopped/failed containers, you do not need supervisor.
docker run --restart=always nginx

Finding source of unidentified node name for celery worker (seen in DuplicateNodenameWarning when trying flower)

For a Django project running celery tasks, I'm running celery flower with the command: python manage.py celery flower --address=0.0.0.0 --port=5555
The server starts correctly, but I get the warning:
[W 161223 17:18:02 control:36] /home/myuser/myenv/myproj/local/lib/python2.7/site-packages/celery/app/control.py:36: DuplicateNodenameWarning: Received multiple replies from node name: 'names'.
Look at the very end of that warning. It says my node is named names. I don't get it. I'm running celery tasks via supervisor, and none of the nodes there are named names.
What's going on? How do I diagnose this problem? I'm essentially trying to solve this warning message and have it go away.
p.s. no mention of it in /etc/default/celeryd either
Here's supervisord.conf:
; supervisor config file
[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
chmod=0700 ; sockef file mode (default 0700)
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)
environment=ON_AZURE="1"
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
[include]
files = /etc/supervisor/conf.d/*.conf
celery.conf is:
[program:celeryworker1]
command=python manage.py celery worker -l info -n celeryworker1
directory = /home/myuser/myproject
environment=PATH="/home/myuser/myenvs/projenv/bin",VIRTUAL_ENV="/home/myuser/myenvs/projenv",PYTHONPATH="/home/myuser/myenvs/projenv/lib/python2.7/site-packages"
user=myuser
password=mypassword
process_name=%(program_name)s%(process_num)d#%(host_node_name)s
numprocs=4
stdout_logfile = /etc/supervisor/logs/celery-worker.log
stderr_logfile = /etc/supervisor/logs/celery-worker.log
autostart = true
autorestart = true
startsecs=10
stopwaitsecs = 600
killasgroup = true
priority = 998

supervisord Error: Invalid user name

I am trying to configure supervisord with golang web app according to the article. But got an error when I run command sudo supervisord -c /etc/supervisor/supervisord.conf:
Error: Invalid username deploy user # the user your app should run as (i.e. *not* root!)
I created user deployuser and added it to group supervisor
sudo adduser deployser supervisor
Supervisord config file is
[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
chmod=0770 ; sockef file mode (default 0700)
chown=root:supervisor ; add our group
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
[include]
files = /etc/supervisor/conf.d/*.conf
I added user deployuser to my app config file:
[program:myGolangApp]
command=/root/path/to/my/go/lang/app/myGolangApp # the location of your app
autostart=true
autorestart=true
startretries=10
user=deployuser # the user your app should run as (i.e. *not* root!)
directory=/root/path/to/my/go/lang/app/ # where your application runs from
environment=APP_SETTINGS="/srv/www/yourapp.com/prod.toml" # environmental variables
redirect_stderr=true
stdout_logfile=/var/log/supervisor/myGolangApp.log # the name of the log file.
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
My os is ubuntu server. And I am doing all the configuration logined as root.
Try changing the comment style in the second comment ( # -> ; )to match the first style. I can't find it in the docs, but they don't seem to use "#" for a comment.
It seems really suspicious that the entire configuration line is in the error. If it had parsed correctly, it would just contain an error value, not the comment.
It seems like you made a typo: deployser vs. deployuser

supervisorctl will not tail a running process

Problem
I can't get supervisor to tail running processes properly. I'm running version 3.0a8.
Here's my supervisord.conf file
[inet_http_server]
port=9011
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
;serverurl=unix:///var/run//supervisor.sock ; use a unix:// URL for a unix socket
serverurl=http://localhost:9011
; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
[include]
files = /etc/supervisor/conf.d/*.conf
And here's the most basic supervisor config file, kept in /etc/supervisord/conf.d/duder.conf
[program: duder]
command=/bin/cat
However, this is what my terminal session with supervisor looks like
vagrant#lucid64:/etc/supervisor/conf.d$ sudo supervisorctl
duder RUNNING pid 31795, uptime 0:03:32
supervisor> tail duder
duder: ERROR (no such process name)
any ideas?
Remove the space in "program: duder", as your program is called " duder" now (with an initial space).
Also, upgrade supervisord ;)