Supervisor socket error issue [closed] - supervisord

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 3 years ago.
Improve this question
$ supervisorctl reread
error: <class 'socket.error'>, [Errno 111] Connection refused: file: /usr/lib64/python2.6/socket.py line: 567
I'm trying to configure supervisor on my production system, but am hitting this error. The supervisor log file is empty.
When I just type supervisorctl, it complains:
http://localhost:9001 refused connection
Nothing is currently listening on port 9001, AFACT: lsof | grep TCP returns nothing.

You have to start supervisord before you can use supervisorctl. In my case:
sudo supervisord -c /etc/supervisor/supervisord.conf
sudo supervisorctl -c /etc/supervisor/supervisord.conf

If you started supervisord with a custom path, like this:
sudo supervisord -c /mypath/supervisord.conf
Then you should also start supervisorctl with the same parameter, like this:
sudo supervisorctl -c /mypath/supervisord.conf

You can get this issue if you start supervisor with a relative url for -c and then try run supervisorctl from a different directory.
For example:
supervisord -c supervisord.conf
supervisorctl start someapp ## works
cd somedirectory
supervisorctl restart someapp ## http://localhost:9001 refused connection
Solution would be to always use the absolute path. e.g.:
Good:
supervisord -c /etc/supervisor/supervisord.conf
Bad:
supervisord -c supervisord.conf

This issue also occurs when an old vesion of supervisord.conf file is used.
Newer version of supervisor uses different configuration file.
The above solutions don't work.
In this case you just have to regenerate the configuration file with
echo_supervisord_conf > /etc/supervisord.conf
Hope it helps someoone.

please do the following :
sudo service supervisor start
post which everything is normal as b4 - there is a problem with 1b version
sudo supervisorctl reload

In Ubuntu 18.04 with distribution's package
You probably did the same mistake as me an created the config file /etc/supervisord.conf while my service manager (systemd) was using the config file /etc/supervisor/supervisord.conf
sudo rm /etc/supervisord.conf
Or
sudo mv /etc/supervisord.conf /etc/supervisor/supervisord.conf
if you want to keep it
Now you can run sudo supervisorctl
Why?
when you run supervisorctl it first searches for the config file located at /etc/supervisord.conf, if it's not present, it will search for the package's default file /etc/supervisor/supervisord.conf this is the one systemd actually runs.
Systemd always use the file /etc/supervisor/supervisord.conf regardless of the other file's existence.
You can check which file is using systemd by running sudo systemctl status supervisor
You can see in the last line the command where the config file is hardcoded

both supervisord and supervisorctl use -c with absolute config file path, make sure both commands startup with same config.

Related

Failed to restart containerd.service: Unit not found [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Improve this question
I have installed containerd 1.5.4 by following below steps in CentOS 7.9:
wget -c https://github.com/containerd/containerd/releases/download/v1.5.4/containerd-1.5.4-linux-amd64.tar.gz
tar -zxvf containerd-1.5.4-linux-amd64.tar.gz -C /
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
I have followed the docs from here and also created the config according to them. But when I try to start containerd:
[root#iZuf62lgwih3vksz3640gnZ sysctl.d]# systemctl start containerd
Failed to start containerd.service: Unit not found.
What should I do to fix this problem?
The main issue is that you are only copying binary files, you are not creating any systemd service.
Be careful when using -C / flag with the tar command. On my CentOS 7 machine, two first commands:
wget -c https://github.com/containerd/containerd/releases/download/v1.5.4/containerd-1.5.4-linux-amd64.tar.gz
tar -zxvf containerd-1.5.4-linux-amd64.tar.gz -C /
led to overwrite the /bin directory which destroyed the OS.
Back to the question, it seems like you are mixing two different instructions for installing containerd package. The instructions from official Kubernetes wiki that you mentioned in your question are pretty-forward and good to follow. Try them:
Step 1. Install the containerd.io package from the official Docker repositories:
yum install -y yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y containerd.io
Step 2. Configure containerd:
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
Step 3. Restart containerd:
systemctl restart containerd

MongoDB - can't run because of SELinux is preventing mongod from open access on the file /proc/sys/net/ipv4/tcp_fastopen

After install, my mongod server was running well. I have created user and restart the server without issue.
But now when modified gonfi file by adding 0.0.0.0 to bindip, server wont restart.
Error message are
Jan 24 11:59:53 localhost.localdomain setroubleshoot[4656]: failed to retrieve rpm info for /proc/sys/net/ipv4/tcp_fastopen
Jan 24 11:59:54 localhost.localdomain setroubleshoot[4656]: SELinux is preventing mongod from open access on the file /proc/sys/net/ipv4/tcp_fastopen. For complete SELinux messag>
Jan 24 11:59:54 localhost.localdomain setroubleshoot[4656]: SELinux is preventing mongod from open access on the file /proc/sys/net/ipv4/tcp_fastopen.
***** Plugin catchall (100. confidence) suggests **************************
If you believe that mongod should be allowed open access on the tcp_fastopen file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'mongod' --raw | audit2allow -M my-mongod
# semodule -X 300 -i my-mongod.pp
ausearch -c 'mongod' --raw | audit2allow -M my-mongod
semodule -X 300 -i my-mongod.pp
does not solved the problem.
Mongodb doc say that version 4 activate by default use of tcp_fastopen
I can't find out how to apply semanage permissive to use tcp_fastopen.
Thanks in advance
If you're running on RHEL, CentOS, or Oracle Linux, follow the instructions for the official SELinux policy for the MongoDB server:
sudo yum install -y git make checkpolicy policycoreutils selinux-policy-devel
git clone https://github.com/mongodb/mongodb-selinux
cd mongodb-selinux
make
sudo make install
Verify your operating system is supported by MongoDB.
Install a vanilla version of your operating system, do not change any settings, use published MongoDB documentation to install MongoDB and get it working.
Identify differences between your current installation and the vanilla installation.
Update your question with findings.
I couldn't find an answer to this exact error. I was starting to pull my hair out. I ended up looking at the mongod.log file. It had a permission denied error in there. journalctl showed the tcp_fastopen, so I was troubleshooting SELinux while it was actually a permission denied error. Hopefully this will help someone else running into this error.
I had the same problem after upgrading mongod to 4.4.6.
I ended up applying what is suggested here, compiling the module manually.
Now it works!
I did many tries, so I am not 100% sure that what I did is more than necessary.
The audit2allow command does not include the rule
allow mongod_t sysctl_net_t:file { getattr read open };
# cat > mongodb_sysctl_net.te << EOF
module mongodb_sysctl_net 1.0;
require {
type mongod_t;
type sysctl_net_t;
class dir search;
class file { getattr read open };
}
#============= mongod_t ==============
allow mongod_t sysctl_net_t:dir search;
allow mongod_t sysctl_net_t:file { getattr read open };
EOF
# checkmodule -M -m -o mongodb_sysctl_net.mod mongodb_sysctl_net.te
# semodule_package -o mongodb_sysctl_net.pp -m mongodb_sysctl_net.mod
# semodule -i mongodb_sysctl_net.pp
# systemctl start mongod.service
NOTE: I already had policy modules installed from the previous installation written according to the mongodb documentation
I had the same issue (semodule error about accessing tcp_fastopen) with a mongo 4.4 replica-set configuration. It couldn't be an os (oracle linux 8) issue, since I had the error on just one of three identical replica-set nodes (same update status).
The system already had the selinux configuration as suggested in the official documentation, with
semodule -l | grep mongo
returning
mongodb
mongodb_cgroup_memory
mongodb_proc_net
Digging inside mongod.log I finally found:
"Failed to unlink socket file","attr":{"path":"/tmp/mongodb-27017.sock","error":"Operation not permitted"}}
I don't know which condition led to this status, but when I unlinked the socket (as root), the mongo daemon started again without errors so far.

PostgreSQL strange behaviour in Ubuntu 18.04 [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 3 years ago.
Improve this question
I installed PostgreSQL using:
sudo apt install libpq-dev postgresql postgresql-contrib
Everything is working fine at the beginning, but I need also remote connection,
so
I need to modify:
pg_hba.conf and postgresql.conf
but I make backups of them, before modifying.
Restart - sudo systemctl restart posgresql
Sometimes it works perfect
but in other cases, when I try sudo -u postgres psql I get the following error:
psql: colud not connect to the server: No such file or directory. Is
the server running locally and accepting connections on Unix domain
socket "/var/run/postgresql/.s.PGSQL.5432
It is very strange because, I change just the IP address in pg_hba.conf to allow remote connection and sometimes works with no errors and sometimes I receive the error. Also remote stop working.
I go back to the backup files, restart server(so no changes for remote in files), the error remains.
I check the service: sudo systemctl status postgresql
Is Active and working.
I have no idea what is wrong, because returned to initial files from backups I expected to fix the error. Please help
I found the errors asked multiple times, but in my case the server is active, and even returned back to backup and is not working.
I manage to solve this by the following method.
Check postgresql logs
> tail -f /var/log/postgresql/<what-ever-postgresql-log-name>.logs
If you log is showing FATAL: could not remove old lock file as follow. Then go for step 2.
2019-09-06 01:49:13.477 UTC [5439] LOG: database system is shut down
pg_ctl: another server might be running; trying to start server anyway
2019-09-06 01:51:17.668 UTC [1039] FATAL: could not remove old lock file "postmaster.pid": Permission denied
2019-09-06 01:51:17.668 UTC [1039] HINT: The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again.
pg_ctl: could not start server
Examine the log output.
Remove postmaster.pid at data_directory path.
You can check your data_directory path via
cat /etc/postgresql/*/main/postgresql.conf
Confirm your data_directory path - then issue the command below.
rm /var/lib/postgresql/10/main/postmaster.pid
Set permission for postgres to data_directory path. At my case is at /var/lib/postgresql/ -
Honestly I am still looking for a (why we still need to set permission) where by default it is already have permission for postgres user.
sudo chmod 765 /var/run/postgresql
sudo chown postgres /var/run/postgresql
Then restart service
sudo service postgresql restart
Test whether is working.
sudo -u postgres psql
Note: I am using Postgresql 10

Running PostgreSQL with Supervisord

I want to run PostgreSQL 9.1 using Supervisor on Ubuntu 10.04. At the moment, I manually start PostgreSQL using the init script:
/etc/init.d/postgresql start
According to this post: http://nicksergeant.com/using-postgresql-with-supervisor-on-ubuntu-1010/, I need to modify the PostgreSQL config to make it run on TCP port instead of Unix socket, in order to make PostgreSQL work with Supervisor.
I have two questions regarding this approach:
Considering this is more of hack, is there any implication (e.g. security/permissions, performance, etc) of doing this?
Why cannot we just run the same init script postgresql in Supervisor config? Instead, as shown in the link above, it runs postmaster?
UPDATE:
Thanks to the useful suggestions from both answers below, I have setup a script for Supervisor to invoke PostgreSQL directly:
#!/bin/sh
# This script is run by Supervisor to start PostgreSQL 9.1 in foreground mode
if [ -d /var/run/postgresql ]; then
chmod 2775 /var/run/postgresql
else
install -d -m 2775 -o postgres -g postgres /var/run/postgresql
fi
exec su postgres -c "/usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main -c config_file=/etc/postgresql/9.1/main/postgresql.conf"
I also set the config: /etc/postgresql/9.1/main/start.conf to manual so that PostgreSQL does not start automatically on boot (however, it's not clear to me whether this config is loaded). And then I setup the Supervisor config for postgres as:
[program:postgres]
user=root
group=root
command=/usr/local/bin/run_postgresql.sh
autostart=true
autorestart=true
stderr_logfile=/home/www-data/logs/postgres_err.log
stdout_logfile=/home/www-data/logs/postgres_out.log
redirect_stderr=true
stopsignal=QUIT
So now, I can start PostgreSQL in supervisorctl by doing start postgres, which runs fine. However, after I issue stop postgres, although supervisorctl declares postgres is stopped, the server apparently is still running as I can psql into it.
I wonder if this is a Supervisor config issue, or a PostgreSQL issue. Any suggestion welcome!
The blog post is rather badly written. There is no "TCP mode": the post's suggested method will still listen on a Unix socket, just in a different directory. Comments in the post such as "external pid file - not needed for TCP mode" are very misleading.
postmaster is the traditional name for the postgresql executable (to distinguish the master dispatching process from the backend slaves). For some time now there has been no separate executable, and now it is installed as simply "postgres".
Assuming that Supervisor is broadly simliar to the qmail/daemontools supervise scheme, it would be entirely possible (in fact, quite normal) to have it run a script that sets up the directories and environment, and then execs postgres with the requisite arguments (or propagates arguments given to the wrapper script, which would be unusual with supervise but makes more sense when you have a config file to put arguments into).
The way supervise worked (and I'm going to continue to assume "Supervisor" is the same) is to have the supervisor process run a subprocess as specified, and simply relaunch a new subprocess if it exits. This is based on the idea that the process being launched is a long-lived daemon process that only exits when something goes badly wrong, and that simply restarting it is a valid fix. By contrast, init scripts such as in /etc/init.d run the subprocess and detach it, and return control to their caller- if the subprocess exits, nothing special happens, and it must be restarted manually. If you tried to simply run /etc/init.d/postgresql start from supervise, it would continuously keep spawning postgresql daemons, as the return from the init script would be interpreted as the daemon process having exited, when in fact it had been started and detached.
To avoid auto-starting the service with the /etc/init.d scripts, the package for postgresql 9.1 provides a file /etc/postgresql/9.1/main/start.conf that contains:
# Automatic startup configuration
# auto: automatically start/stop the cluster in the init script
# manual: do not start/stop in init scripts, but allow manual startup with
# pg_ctlcluster
# disabled: do not allow manual startup with pg_ctlcluster (this can be easily
# circumvented and is only meant to be a small protection for
# accidents).
auto
This is the file to modify to avoid auto-start as opposed to moving away /etc/init.d/postgresql as the blog post suggests.
Also, changing unix sockets parameters for lack of /var/run/postgresql doesn't look like the best idea, because it's the default for any program linked with libpq, and because there's no difficulty in creating that directory with the proper permissions, just like it's done by the package's start sequence in /usr/share/postgresql-common/init.d-functions:
# create socket directory
if [ -d /var/run/postgresql ]; then
chmod 2775 /var/run/postgresql
else
install -d -m 2775 -o postgres -g postgres /var/run/postgresql
fi
And although the default value shouldn't cause a problem, note that whether postmaster ultimately stays in foreground or forks and runs in background is controlled by the silent_mode parameter in postgresql.conf. Make sure that it is off.
I am trying to make both tomcat and postgres run under supervisor, and found some hints here : https://serverfault.com/questions/425132/controlling-tomcat-with-supervisor
Here is my modified run_postgresql.sh, using bash :
#!/bin/bash
# This script is run by Supervisor to start PostgreSQL 9.1 in foreground mode
function shutdown()
{
echo "Shutting down PostgreSQL"
pkill postgres
}
if [ -d /var/run/postgresql ]; then
chmod 2775 /var/run/postgresql
else
install -d -m 2775 -o postgres -g postgres /var/run/postgresql
fi
# Allow any signal which would kill a process to stop PostgreSQL
trap shutdown HUP INT QUIT ABRT KILL ALRM TERM TSTP
exec sudo -u postgres /usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main --config-file=/etc/postgresql/9.1/main/postgresql.conf
With this script postgresql stops correctly after supervisorctl stop postgres.

I have installed PostgreSQL via MacPorts, but cannot access it

As I said in title, I've installed PostgreSQL usind MacPorts, but cannot access it.
The installation process was
$ sudo port install postgresql83-server
$ sudo mkdir -p /opt/local/var/db/postgresql83/webcraft
$ sudo chown postgres:postgres /opt/local/var/db/postgresql83/webcraft
$ sudo su postgres -c '/opt/local/lib/postgresql83/bin/initdb -D /opt/local/var/db/postgresql83/webcraft'
$ sudo launchctl load -w /Library/LaunchDaemons/org.macports.postgresql83-server.plist
My PATH is
/opt/local/lib/postgresql83/bin:/opt/local/lib/mysql5/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
I try to connect the server using psql client
$ psql
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
Here is some info
$ ps ax | grep postgres | grep -v grep
52 ?? Ss 0:00.00 /opt/local/bin/daemondo --label=postgresql83-server --start-cmd /opt/local/etc/LaunchDaemons/org.macports.postgresql83-server/postgresql83-server.wrapper start ; --stop-cmd /opt/local/etc/LaunchDaemons/org.macports.postgresql83-server/postgresql83-server.wrapper stop ; --restart-cmd /opt/local/etc/LaunchDaemons/org.macports.postgresql83-server/postgresql83-server.wrapper restart ; --pid=none
Did you try running:
which psql
I imagine psql is still referencing /usr/bin/psql, and the macports version of psql is suffixed with the version number, in your case psql83. You can alias psql to psql83 as a simple workaround. Better would be to change the default:
sudo port select --set postgresql postgresql83
That will do the proper routing.
There is a very easy solution to this, but it's not well documented in my opinion:
MacPorts encourages installing their *_select ports to manage potentially multiple versions of software (say you want Postgres93 and Postgres94 at the same time). It's a great feature, but it adds an extra step that is for some reason rarely mentioned in the docs:
$ sudo port install postgresql94-server
Many failed attempts at starting the server later..
$ sudo port install postgresql_select
$ sudo port select postgresql
Available versions for postgresql:
none (active)
postgresql94
Well that can't be good!
$ sudo port select postgresql postgresql94
$ sudo port load postgresql94-server
You're kidding me. Now it's running?
Simply installing Postgres doesn't fully setup symlinks to make it easily runnable. Installing postrgresql_select gives MacPorts the information it needs to do that via port select. Once you've selected the active version of your choice, starting the Posgres server via luanchctl is as easy as port load postgresqlXX-server.
I know this is a very late answer and doesn't answer your full question, but launchctl will show different results depending on if you are superuser or not.
Try doing:
sudo launchctl list | grep postgres
I had exactly the same problem on my MacBook Pro. I could resolve the problem after I rode this blog post here and all the comments:
http://benscheirman.com/2010/06/installing-postgresql-for-rails-on-mac-os-x
The Problem is that postgres is not really running. I recognized this after I did a port scan to my own machine and realized that nothing is running on Port 5432.
I created a small script "start_pg_server.sh":
#!/bin/sh
sudo su postgres -c 'pg_ctl start -D /opt/local/var/db/postgresql83/defaultdb/'
after executing this script the server was running and I could connect me with pgAdmin. I was also able to run my ruby stuff with rake db:create and rake db:migrate.
After I restored using Timemachine I had the same problem.
The reason was that the permissions were mangled and postgres could not write the pid file.
Running this solved it for me:
sudo chown -R postgres:postgres /opt/local/var/db/postgresql91/
sudo port unload postgresql91-server
sudo port load postgresql91-server
Did you by any chance create your postgres user with a shell of /usr/bin/false? If so, the startup script won't work because it uses su which passes commands you send it through the shell.
If you did set it to /usr/bin/false, try changing it to /bin/bash and that might fix things.