Supervisor can't run a docker-compose up command - docker-compose

I'm trying to monitor multiple docker process using supervisor but I have tried everything I can but no results. for some reason, supervisor always say that it can't find a command docker-compose . I have pointed to the bin of the docker-compose on my machine, but still, it didn't work, I would appreciate if someone can help.
this is my supervisord.conf file
; Sample supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html
;
; Note: shell expansion ("~" or "$HOME") is not supported. Environment
; variables can be expanded using this syntax: "%(ENV_HOME)s".
[unix_http_server]
file=/tmp/supervisor.sock ; (the path to the socket file)
chmod=0700 ; socket file mode (default 0700)
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
username=socialhub ; (default is no username (open server))
password=socialhub ; (default is no password (open server))
[inet_http_server] ; inet (TCP) server disabled by default
port=127.0.0.1:9001 ; (ip_address:port specifier, *:port for all iface)
username=socialhub ; (default is no username (open server))
password=socialhub ; (default is no password (open server))
[supervisord]
logfile=supervisord/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=0 ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=supervisord/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
nocleanup=false ; (don't clean up tempfiles at start;default false)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
; The below sample program section shows all possible program subsection values,
; create one or more 'real' program: sections to be able to control them under
; supervisor.
[program:mongo]
command=/usr/local/bin/docker-compose up mongodb ; the program (relative uses PATH, can take args)
process_name=%(program_name)s ; process_name expr (default %(program_name)s)
numprocs=1 ; number of processes copies to start (def 1)
directory=supervisord ; directory to cwd to before exec (def no cwd)
priority=1000 ; the relative start priority (default 999)
autostart=true ; start at supervisord start (default: true)
autorestart=unexpected ; whether/when to restart (default: unexpected)
startsecs=60 ; number of secs prog must stay running (def. 1)
startretries=5 ; max # of serial start failures (default 3)
;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=1 ; max num secs to wait b4 SIGKILL (default 10)
stopasgroup=false ; send stop signal to the UNIX process group (default false)
killasgroup=false ; SIGKILL the UNIX process group (def false)
stdout_logfile=supervisord/ ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=0 ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile=supervisord/ ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=10MB ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=5 ; # of stderr logfile backups (default 10)
stderr_events_enabled=true ; emit events on stderr writes (default false)
; The below sample eventlistener section shows all possible
; eventlistener subsection values, create one or more 'real'
; eventlistener: sections to be able to handle event notifications
; sent by supervisor.
[eventlistener:process_fail]
command=/usr/local/bin/docker-compose rm -f ; the program (relative uses PATH, can take args)
process_name=%(program_name)s ; process_name expr (default %(program_name)s)
numprocs=1 ; number of processes copies to start (def 1)
events=PROCESS_LOG_STDERR ; event notif. types to subscribe to (req'd)
buffer_size=10 ; event buffer queue size (default 10)
directory=supervisord/ ; directory to cwd to before exec (def no cwd)
priority=-1 ; the relative start priority (default -1)
autostart=true ; start at supervisord start (default: true)
autorestart=unexpected ; whether/when to restart (default: unexpected)
startsecs=1 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=1 ; max num secs to wait b4 SIGKILL (default 10)
stopasgroup=false ; send stop signal to the UNIX process group (default false)
killasgroup=false ; SIGKILL the UNIX process group (def false)
;user=chrism ; setuid to this UNIX account to run the program
redirect_stderr=false ; redirect proc stderr to stdout (default false)
stdout_logfile=NONE ; stdout log path, NONE for none; default AUTO

The directory that you have used is not correct
directory=supervisord
You should use the complete path of the directory where your docker-compose.yml file is present.
Edit - 1:
So I had to dig a bit deeper for you question. Spaces in command are not passed properly till version 3.0b2.
If you upgrade to 3.0.1 or higher the command with spaces would work fine. In your version it is taking full value of command and trying to locate that as a file, that us why you see a command not found error
You can easily upgrade to latest version of supervisor using pip
pip install supervisor==3.3.3
In case you can't upgrade your supervisor for some reason (which i strongly recommend as they recently had XML-RPC vulnerability which allows command execution), then you would need to create a shell script
start_mongo.sh
#!/bin/bash
cd /home/user/socialhub/
exec docker-compose up
Make sure to chmod +x start_mongo.sh and change the command to /path/to/start_mongo.sh

Related

Supervisord not releasing log file during rotation

I've got a pair of CentOS 7 servers that are running SupervisorD with one program. Supervisord is not set to handle any log rotations.. supervisord.conf's lines related to size and backups both = 0.
supervisord.conf
[unix_http_server]
file=/var/run/dir1/supervisor.sock ; the path to the socket file
[supervisord]
logfile=/var/log/dir1/supervisord.log ; main log file; default $CWD/supervisord.log
logfile_maxbytes=0
; logfile_maxbytes=50MB ; max main logfile bytes b4 rotation; default 50MB
logfile_backups=0
;logfile_backups=10 ; # of main logfile backups; 0 means none, default 10
loglevel=trace ; log level; default info; others: debug,warn,trace
pidfile=/var/run/dir1/supervisord.pid ; supervisord pidfile; default supervisord.pid
nodaemon=false ; start in foreground if true; default false
minfds=786068
minprocs=200 ; min. avail process descriptors;default 200
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/dir1/supervisor.sock ; use a unix:// URL for a unix socket
[include]
files = /etc/supervisord.d/*.conf
program1.conf
[program:jobengine]
; Set full path to Job Engine program if using virtualenv
command=python /opt/program1/jobengine/jobengine.pyc --namespace=worker%(process_num)02d
environment=PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:
directory=/opt/program1
user=root
numprocs=4
process_name=%(process_num)01d
stdout_logfile=/var/log/dir1/jobengine.log
stderr_logfile=/var/log/dir1/jobengine.log
autostart=true
autorestart=true
startsecs=10
startretries=999
; Below lines added to ensure supervisord does not perform any log handling, in favor of logrotate
stdout_logfile_maxbytes=0
stderr_logfile_maxbytes=0
stdout_logfile_backups=0
stderr_logfile_backups=0
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; Causes supervisor to send the termination signal (SIGTERM) to the whole process group.
stopasgroup=false
; default (999)
priority=999
logrotate/program1.conf
# Track main logs, jobengine logs, etc.
/var/log/dir1/*log {
# Run as the Apache user
su apache apache
# CloudBolt will create the log file
missingok
# Do not rotate empty logs
notifempty
# Do not compress logs
nocompress
# Use `*.log.1` naming instead of `*.log-YYYYMMDD` format
nodateext
# Keep 5 archived logs
rotate 5
# Rotate files larger than this size
size 5M
}
So what happens is logrotate kicks off at it's appropriate time, however supervisord refuses to move from the old to the new log file. I'm used to the logrotate having a postscript for a restart signal however I'm 100% new to supervisord so I've been trying to verify how to gracefully restart supervisord and it's child processes. The way I understand it, 'stopwaitsecs' force supervisord to wait the specified time if busy and when I test the 4 child pids are killed via SIGTERM immediately.
What I need is supervisord to only restart once the child pids have completed w/e they're doing.
I also, on a whim, tried sending supervisord a SIGUSR2, and also a supervisorctl reload.. everything seems to kill the processes immediately.
What am I doing wrong here?

How to auto-reload (start) docker container with supervisor

I'm stuck with a problem where I can't perfectly start my container via supervisor.
I have a mongodb container, which I want to run under supervisor.
Supervisor config itself:
; supervisor config file
[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
chmod=0700 ; sockef file mode (default 0700)
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
[include]
files = /etc/supervisor/conf.d/*.conf
and mongodb container conf file:
[program:mongo]
command=/usr/bin/docker start b1b -DFOREGROUND
autostart=true
autorestart=true
startretries=10
exitcodes=0
startsecs=0
When I do "supervisorctl restart mongo" it will start container, but after I stop it it will not relaunch it automatically.
Any advice on it?
Docker already has the facility to restart stopped/failed containers, you do not need supervisor.
docker run --restart=always nginx

Finding source of unidentified node name for celery worker (seen in DuplicateNodenameWarning when trying flower)

For a Django project running celery tasks, I'm running celery flower with the command: python manage.py celery flower --address=0.0.0.0 --port=5555
The server starts correctly, but I get the warning:
[W 161223 17:18:02 control:36] /home/myuser/myenv/myproj/local/lib/python2.7/site-packages/celery/app/control.py:36: DuplicateNodenameWarning: Received multiple replies from node name: 'names'.
Look at the very end of that warning. It says my node is named names. I don't get it. I'm running celery tasks via supervisor, and none of the nodes there are named names.
What's going on? How do I diagnose this problem? I'm essentially trying to solve this warning message and have it go away.
p.s. no mention of it in /etc/default/celeryd either
Here's supervisord.conf:
; supervisor config file
[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
chmod=0700 ; sockef file mode (default 0700)
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)
environment=ON_AZURE="1"
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
[include]
files = /etc/supervisor/conf.d/*.conf
celery.conf is:
[program:celeryworker1]
command=python manage.py celery worker -l info -n celeryworker1
directory = /home/myuser/myproject
environment=PATH="/home/myuser/myenvs/projenv/bin",VIRTUAL_ENV="/home/myuser/myenvs/projenv",PYTHONPATH="/home/myuser/myenvs/projenv/lib/python2.7/site-packages"
user=myuser
password=mypassword
process_name=%(program_name)s%(process_num)d#%(host_node_name)s
numprocs=4
stdout_logfile = /etc/supervisor/logs/celery-worker.log
stderr_logfile = /etc/supervisor/logs/celery-worker.log
autostart = true
autorestart = true
startsecs=10
stopwaitsecs = 600
killasgroup = true
priority = 998

Supervisorctl not respecting my configuration

I have set the following in /home/david/conf/supervisor.conf:
[unix_http_server]
file=/home/david/tmp/supervisor.sock
[supervisord]
logfile=/home/david/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/home/david/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200
childlogdir=/home/david/tmp
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///home/david/tmp/supervisor.sock
And started supervisord:
$ supervisord -c /home/david/conf/supervisor.conf
However how come supervisorctl still uses the default http://localhost:9001 as the serverurl?
$ supervisorctl
http://localhost:9001 refused connection
supervisor>
I checked /home/david/tmp and the files supervisord.log and supervisord.pid do exist.
You should run supervisorctl with -c as well. From the documentation (my emphasis):
The Supervisor configuration file is conventionally named
supervisord.conf. It is used by both supervisord and supervisorctl. If
either application is started without the -c option (the option which
is used to tell the application the configuration filename
explicitly), the application will look for a file named
supervisord.conf within the following locations, in the specified
order. It will use the first file it finds.
$CWD/supervisord.conf
$CWD/etc/supervisord.conf
/etc/supervisord.conf
In MacOS, use brew to install
brew install supervisor
Then go to /usr/local/etc/supervisord.ini and comment these lines:
;[unix_http_server]
;file=/usr/local/var/run/supervisor.sock ; the path to the socket file
and uncomment these lines:
[inet_http_server] ; inet (TCP) server disabled by default
port=127.0.0.1:9001 ; ip_address:port specifier, *:port for all iface
Finally restart the daemon:
brew services restart supervisor
That's all you need.
To add to the valid answer above make sure you are putting your config files for the apps you want to monitor under supervisor's config folder as a subfolder called conf.d. This will depend of what method you use to install supervisor, the default package manager or easy_install.
As solution you can make symbolic link to the config file.
Like this for Mac OS:
sudo ln -sv /usr/local/etc/supervisord.ini /etc/supervisord.conf

supervisorctl will not tail a running process

Problem
I can't get supervisor to tail running processes properly. I'm running version 3.0a8.
Here's my supervisord.conf file
[inet_http_server]
port=9011
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
;serverurl=unix:///var/run//supervisor.sock ; use a unix:// URL for a unix socket
serverurl=http://localhost:9011
; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
[include]
files = /etc/supervisor/conf.d/*.conf
And here's the most basic supervisor config file, kept in /etc/supervisord/conf.d/duder.conf
[program: duder]
command=/bin/cat
However, this is what my terminal session with supervisor looks like
vagrant#lucid64:/etc/supervisor/conf.d$ sudo supervisorctl
duder RUNNING pid 31795, uptime 0:03:32
supervisor> tail duder
duder: ERROR (no such process name)
any ideas?
Remove the space in "program: duder", as your program is called " duder" now (with an initial space).
Also, upgrade supervisord ;)