I've got a pair of CentOS 7 servers that are running SupervisorD with one program. Supervisord is not set to handle any log rotations.. supervisord.conf's lines related to size and backups both = 0.
supervisord.conf
[unix_http_server]
file=/var/run/dir1/supervisor.sock ; the path to the socket file
[supervisord]
logfile=/var/log/dir1/supervisord.log ; main log file; default $CWD/supervisord.log
logfile_maxbytes=0
; logfile_maxbytes=50MB ; max main logfile bytes b4 rotation; default 50MB
logfile_backups=0
;logfile_backups=10 ; # of main logfile backups; 0 means none, default 10
loglevel=trace ; log level; default info; others: debug,warn,trace
pidfile=/var/run/dir1/supervisord.pid ; supervisord pidfile; default supervisord.pid
nodaemon=false ; start in foreground if true; default false
minfds=786068
minprocs=200 ; min. avail process descriptors;default 200
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/dir1/supervisor.sock ; use a unix:// URL for a unix socket
[include]
files = /etc/supervisord.d/*.conf
program1.conf
[program:jobengine]
; Set full path to Job Engine program if using virtualenv
command=python /opt/program1/jobengine/jobengine.pyc --namespace=worker%(process_num)02d
environment=PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:
directory=/opt/program1
user=root
numprocs=4
process_name=%(process_num)01d
stdout_logfile=/var/log/dir1/jobengine.log
stderr_logfile=/var/log/dir1/jobengine.log
autostart=true
autorestart=true
startsecs=10
startretries=999
; Below lines added to ensure supervisord does not perform any log handling, in favor of logrotate
stdout_logfile_maxbytes=0
stderr_logfile_maxbytes=0
stdout_logfile_backups=0
stderr_logfile_backups=0
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; Causes supervisor to send the termination signal (SIGTERM) to the whole process group.
stopasgroup=false
; default (999)
priority=999
logrotate/program1.conf
# Track main logs, jobengine logs, etc.
/var/log/dir1/*log {
# Run as the Apache user
su apache apache
# CloudBolt will create the log file
missingok
# Do not rotate empty logs
notifempty
# Do not compress logs
nocompress
# Use `*.log.1` naming instead of `*.log-YYYYMMDD` format
nodateext
# Keep 5 archived logs
rotate 5
# Rotate files larger than this size
size 5M
}
So what happens is logrotate kicks off at it's appropriate time, however supervisord refuses to move from the old to the new log file. I'm used to the logrotate having a postscript for a restart signal however I'm 100% new to supervisord so I've been trying to verify how to gracefully restart supervisord and it's child processes. The way I understand it, 'stopwaitsecs' force supervisord to wait the specified time if busy and when I test the 4 child pids are killed via SIGTERM immediately.
What I need is supervisord to only restart once the child pids have completed w/e they're doing.
I also, on a whim, tried sending supervisord a SIGUSR2, and also a supervisorctl reload.. everything seems to kill the processes immediately.
What am I doing wrong here?
Related
CentOS v.7
Logrotate v.3.8.6
I set logrotate to rotate when file reaches 5M but it ignores it, if I add daily it will rotate daily regardless of size, i tried with size, minsize and maxsize all the same the only difference is with "size" it doesnt even refer to it in the output, here is my config and output of logrotate -vdf /etc/logrotate.d/maillog
(the actual log file size when running the following tests was 45K)
(the conf file is the same for all tests only the size parameter changed)
/var/log/maillog {
size 5M
rotate 50
create 644 root root
dateext
dateformat -%Y-%m-%d_%H_%s
notifempty
postrotate systemctl restart rsyslog
systemctl restart postfix
endscript }
SIZE:
logrotate -vdf /etc/logrotate.d/maillog
reading config file /etc/logrotate.d/maillog
Allocating hash table for state file, size 15360 B
Handling 1 logs
rotating pattern: /var/log/maillog forced from command line (50 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/maillog
log needs rotating
rotating log /var/log/maillog, log->rotateCount is 50
Converted ' -%Y-%m-%d_%H_%s' -> '-%Y-%m-%d_%H_%s'
dateext suffix '-2017-12-19_13_1513689486'
glob pattern '-[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]_[0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
fscreate context set to unconfined_u:object_r:var_log_t:s0
renaming /var/log/maillog to /var/log/maillog-2017-12-19_13_1513689486
creating new /var/log/maillog mode = 0644 uid = 0 gid = 0
running postrotate script
running script with arg /var/log/maillog: "
systemctl restart rsyslog
systemctl restart postfix
"
No reason for "log needs rotating" is given.
MINSIZE:
logrotate -vdf /etc/logrotate.d/maillog
reading config file /etc/logrotate.d/maillog
Allocating hash table for state file, size 15360 B
Handling 1 logs
rotating pattern: /var/log/maillog forced from command line (50 rotations)
empty log files are not rotated, only log files >= 5242880 bytes are rotated, old logs are removed
considering log /var/log/maillog
log needs rotating
rotating log /var/log/maillog, log->rotateCount is 50
Converted ' -%Y-%m-%d_%H_%s' -> '-%Y-%m-%d_%H_%s'
dateext suffix '-2017-12-19_13_1513689869'
glob pattern '-[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]_[0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
fscreate context set to unconfined_u:object_r:var_log_t:s0
renaming /var/log/maillog to /var/log/maillog-2017-12-19_13_1513689869
creating new /var/log/maillog mode = 0644 uid = 0 gid = 0
running postrotate script
running script with arg /var/log/maillog: "
systemctl restart rsyslog
systemctl restart postfix
"
Here it shows, "only log files >= are rotated" but no reason for "log needs rotating" is given.
MAXSIZE:
reading config file /etc/logrotate.d/maillog
Allocating hash table for state file, size 15360 B
Handling 1 logs
rotating pattern: /var/log/maillog forced from command line (50 rotations)
empty log files are not rotated, log files >= 5242880 are rotated earlier, old logs are removed
considering log /var/log/maillog
log needs rotating
rotating log /var/log/maillog, log->rotateCount is 50
Converted ' -%Y-%m-%d_%H_%s' -> '-%Y-%m-%d_%H_%s'
dateext suffix '-2017-12-19_13_1513690859'
glob pattern '-[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]_[0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
fscreate context set to unconfined_u:object_r:var_log_t:s0
renaming /var/log/maillog to /var/log/maillog-2017-12-19_13_1513690859
creating new /var/log/maillog mode = 0644 uid = 0 gid = 0
running postrotate script
running script with arg /var/log/maillog: "
systemctl restart rsyslog
systemctl restart postfix
"
Here it shows, "log files => are rotated" but no reason for "log needs rotating" is given.
Why is it ignoring file size when rotating?
You're running logrotate with -f, in that scenario it's always going to force a rotation with a complete lack of regard for your other options:
https://manpages.debian.org/jessie/logrotate/logrotate.8.en.html
-f, --force
Tells logrotate to force the rotation, even if it doesn't think this is necessary.
It gives you no reason because you in fact were the reason-giver.
I'm trying to monitor multiple docker process using supervisor but I have tried everything I can but no results. for some reason, supervisor always say that it can't find a command docker-compose . I have pointed to the bin of the docker-compose on my machine, but still, it didn't work, I would appreciate if someone can help.
this is my supervisord.conf file
; Sample supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html
;
; Note: shell expansion ("~" or "$HOME") is not supported. Environment
; variables can be expanded using this syntax: "%(ENV_HOME)s".
[unix_http_server]
file=/tmp/supervisor.sock ; (the path to the socket file)
chmod=0700 ; socket file mode (default 0700)
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
username=socialhub ; (default is no username (open server))
password=socialhub ; (default is no password (open server))
[inet_http_server] ; inet (TCP) server disabled by default
port=127.0.0.1:9001 ; (ip_address:port specifier, *:port for all iface)
username=socialhub ; (default is no username (open server))
password=socialhub ; (default is no password (open server))
[supervisord]
logfile=supervisord/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=0 ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=supervisord/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
nocleanup=false ; (don't clean up tempfiles at start;default false)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
; The below sample program section shows all possible program subsection values,
; create one or more 'real' program: sections to be able to control them under
; supervisor.
[program:mongo]
command=/usr/local/bin/docker-compose up mongodb ; the program (relative uses PATH, can take args)
process_name=%(program_name)s ; process_name expr (default %(program_name)s)
numprocs=1 ; number of processes copies to start (def 1)
directory=supervisord ; directory to cwd to before exec (def no cwd)
priority=1000 ; the relative start priority (default 999)
autostart=true ; start at supervisord start (default: true)
autorestart=unexpected ; whether/when to restart (default: unexpected)
startsecs=60 ; number of secs prog must stay running (def. 1)
startretries=5 ; max # of serial start failures (default 3)
;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=1 ; max num secs to wait b4 SIGKILL (default 10)
stopasgroup=false ; send stop signal to the UNIX process group (default false)
killasgroup=false ; SIGKILL the UNIX process group (def false)
stdout_logfile=supervisord/ ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=0 ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile=supervisord/ ; stderr log path, NONE for none; default AUTO
stderr_logfile_maxbytes=10MB ; max # logfile bytes b4 rotation (default 50MB)
stderr_logfile_backups=5 ; # of stderr logfile backups (default 10)
stderr_events_enabled=true ; emit events on stderr writes (default false)
; The below sample eventlistener section shows all possible
; eventlistener subsection values, create one or more 'real'
; eventlistener: sections to be able to handle event notifications
; sent by supervisor.
[eventlistener:process_fail]
command=/usr/local/bin/docker-compose rm -f ; the program (relative uses PATH, can take args)
process_name=%(program_name)s ; process_name expr (default %(program_name)s)
numprocs=1 ; number of processes copies to start (def 1)
events=PROCESS_LOG_STDERR ; event notif. types to subscribe to (req'd)
buffer_size=10 ; event buffer queue size (default 10)
directory=supervisord/ ; directory to cwd to before exec (def no cwd)
priority=-1 ; the relative start priority (default -1)
autostart=true ; start at supervisord start (default: true)
autorestart=unexpected ; whether/when to restart (default: unexpected)
startsecs=1 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=1 ; max num secs to wait b4 SIGKILL (default 10)
stopasgroup=false ; send stop signal to the UNIX process group (default false)
killasgroup=false ; SIGKILL the UNIX process group (def false)
;user=chrism ; setuid to this UNIX account to run the program
redirect_stderr=false ; redirect proc stderr to stdout (default false)
stdout_logfile=NONE ; stdout log path, NONE for none; default AUTO
The directory that you have used is not correct
directory=supervisord
You should use the complete path of the directory where your docker-compose.yml file is present.
Edit - 1:
So I had to dig a bit deeper for you question. Spaces in command are not passed properly till version 3.0b2.
If you upgrade to 3.0.1 or higher the command with spaces would work fine. In your version it is taking full value of command and trying to locate that as a file, that us why you see a command not found error
You can easily upgrade to latest version of supervisor using pip
pip install supervisor==3.3.3
In case you can't upgrade your supervisor for some reason (which i strongly recommend as they recently had XML-RPC vulnerability which allows command execution), then you would need to create a shell script
start_mongo.sh
#!/bin/bash
cd /home/user/socialhub/
exec docker-compose up
Make sure to chmod +x start_mongo.sh and change the command to /path/to/start_mongo.sh
For a Django project running celery tasks, I'm running celery flower with the command: python manage.py celery flower --address=0.0.0.0 --port=5555
The server starts correctly, but I get the warning:
[W 161223 17:18:02 control:36] /home/myuser/myenv/myproj/local/lib/python2.7/site-packages/celery/app/control.py:36: DuplicateNodenameWarning: Received multiple replies from node name: 'names'.
Look at the very end of that warning. It says my node is named names. I don't get it. I'm running celery tasks via supervisor, and none of the nodes there are named names.
What's going on? How do I diagnose this problem? I'm essentially trying to solve this warning message and have it go away.
p.s. no mention of it in /etc/default/celeryd either
Here's supervisord.conf:
; supervisor config file
[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
chmod=0700 ; sockef file mode (default 0700)
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)
environment=ON_AZURE="1"
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
[include]
files = /etc/supervisor/conf.d/*.conf
celery.conf is:
[program:celeryworker1]
command=python manage.py celery worker -l info -n celeryworker1
directory = /home/myuser/myproject
environment=PATH="/home/myuser/myenvs/projenv/bin",VIRTUAL_ENV="/home/myuser/myenvs/projenv",PYTHONPATH="/home/myuser/myenvs/projenv/lib/python2.7/site-packages"
user=myuser
password=mypassword
process_name=%(program_name)s%(process_num)d#%(host_node_name)s
numprocs=4
stdout_logfile = /etc/supervisor/logs/celery-worker.log
stderr_logfile = /etc/supervisor/logs/celery-worker.log
autostart = true
autorestart = true
startsecs=10
stopwaitsecs = 600
killasgroup = true
priority = 998
I have set the following in /home/david/conf/supervisor.conf:
[unix_http_server]
file=/home/david/tmp/supervisor.sock
[supervisord]
logfile=/home/david/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/home/david/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200
childlogdir=/home/david/tmp
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///home/david/tmp/supervisor.sock
And started supervisord:
$ supervisord -c /home/david/conf/supervisor.conf
However how come supervisorctl still uses the default http://localhost:9001 as the serverurl?
$ supervisorctl
http://localhost:9001 refused connection
supervisor>
I checked /home/david/tmp and the files supervisord.log and supervisord.pid do exist.
You should run supervisorctl with -c as well. From the documentation (my emphasis):
The Supervisor configuration file is conventionally named
supervisord.conf. It is used by both supervisord and supervisorctl. If
either application is started without the -c option (the option which
is used to tell the application the configuration filename
explicitly), the application will look for a file named
supervisord.conf within the following locations, in the specified
order. It will use the first file it finds.
$CWD/supervisord.conf
$CWD/etc/supervisord.conf
/etc/supervisord.conf
In MacOS, use brew to install
brew install supervisor
Then go to /usr/local/etc/supervisord.ini and comment these lines:
;[unix_http_server]
;file=/usr/local/var/run/supervisor.sock ; the path to the socket file
and uncomment these lines:
[inet_http_server] ; inet (TCP) server disabled by default
port=127.0.0.1:9001 ; ip_address:port specifier, *:port for all iface
Finally restart the daemon:
brew services restart supervisor
That's all you need.
To add to the valid answer above make sure you are putting your config files for the apps you want to monitor under supervisor's config folder as a subfolder called conf.d. This will depend of what method you use to install supervisor, the default package manager or easy_install.
As solution you can make symbolic link to the config file.
Like this for Mac OS:
sudo ln -sv /usr/local/etc/supervisord.ini /etc/supervisord.conf
Problem
I can't get supervisor to tail running processes properly. I'm running version 3.0a8.
Here's my supervisord.conf file
[inet_http_server]
port=9011
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor ; ('AUTO' child log dir, default $TEMP)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
;serverurl=unix:///var/run//supervisor.sock ; use a unix:// URL for a unix socket
serverurl=http://localhost:9011
; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
[include]
files = /etc/supervisor/conf.d/*.conf
And here's the most basic supervisor config file, kept in /etc/supervisord/conf.d/duder.conf
[program: duder]
command=/bin/cat
However, this is what my terminal session with supervisor looks like
vagrant#lucid64:/etc/supervisor/conf.d$ sudo supervisorctl
duder RUNNING pid 31795, uptime 0:03:32
supervisor> tail duder
duder: ERROR (no such process name)
any ideas?
Remove the space in "program: duder", as your program is called " duder" now (with an initial space).
Also, upgrade supervisord ;)