Everyday I get an error email from my python flask app with the following error:
WriteError: error 5 from memcached_set: (54648992) SERVER HAS FAILED
AND IS DISABLED UNTIL TIMED RETRY, host: 127.0.0.1:11211 ->
libmemcached/connect.cc:612
Memcached is running and the address is set up to 127.0.0.1 on both the client and the server. When I manually check connection it seems just fine. It only happens periodically.
~ cat /etc/memcached.conf:
# memcached default config file
# 2003 - Jay Bonci <jaybonci#debian.org>
# This configuration file is read by the start-memcached script provided as
# part of the Debian GNU/Linux distribution.
# Run memcached as a daemon. This command is implied, and is not needed for the
# daemon to run. See the README.Debian that comes with this package for more
# information.
-d
# Log memcached's output to /var/log/memcached
logfile /var/log/memcached.log
# Be verbose
# -v
# Be even more verbose (print client commands as well)
# -vv
# Start with a cap of 64 megs of memory. It's reasonable, and the daemon default
# Note that the daemon will grow to this size, but does not start out holding this much
# memory
-m 64
# Default connection port is 11211
-p 11211
# Run the daemon as root. The start-memcached will default to running as root if no
# -u command is present in this config file
-u memcache
# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
-l 127.0.0.1
# Limit the number of simultaneous incoming connections. The daemon default is 1024
# -c 1024
# Lock down all paged memory. Consult with the README and homepage before you do this
# -k
# Return error when memory is exhausted (rather than removing items)
# -M
# Maximize core file limit
# -r
/var/log/memcached.log is empty.
I have classic situation. Need to configure monit for memcached on CentOS7. The problem is, that all configs i can find in google contains this row:
check process memcached with pidfile /var/run/memcached/memcached.pid
However, There is no memcached.pid file in /var/run and no /var/run/memcached folder. I've checked /usr/lib/systemd/system/memcached.service
[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/memcached
ExecStart=/usr/bin/memcached -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
So, there is no path to .pid file.
The question is can I check memcached without .pid file?
The second question - can be this .pid file in another location?
Replace in your monit config
check process memcached with pidfile /var/run/memcached/memcached.pid
with
check process memcached with match memcached
My config for memcached:
check process memcached with match memcached
start program = "/usr/bin/systemctl start memcached"
stop program = "/usr/bin/systemctl stop memcached"
if failed host 127.0.0.1 port 11211 protocol MEMCACHE then restart
if cpu > 70% for 2 cycles then alert
if cpu > 98% for 5 cycles then restart
if 2 restarts within 3 cycles then timeout
Centos 7, monit 5.14
I'm trying to provision a development box with Vagrant and a CentOS 6.5 base box. I want memcached to automatically start at system boot/vagrant up.
I have tried adding memcached -d -l localhost -p11211 to /etc/rc.d/rc.local and this does not work.
I have also tried adding to /etc/init/vagrant-mounted.conf
start on vagrant-mounted
memcached -d -l localhost -p11211
[EDIT]
I've updated /etc/rc.d/rc.local to now use the following
chkconfig memcached on
service memcached start
I'm not seeing anything in the /var/log/boot.log. It looks like rc.local is not being run at all. It has ugo+x permissions; so the file is definitely executable, but it doesn't appear to run at all.
Does memcached -d -l localhost -p11211 exit immediately or spawn a process?
If it keeps running, try: nohup memcached -d -l localhost -p11211 &
Also, try putting it in /etc/rc.local as
memcached -d -l localhost -p11211 >/var/log/memcached.log 2>&1
That will give you a log file with possible errors.
Lastly, does your install of memcached not have an init.d file in /etc/init.d ?
if it does, simply do chkconfig servicename on && service servicename start
I am trying to create a docker image for Couchbase and i get the following error with the dockerfile on CentOS image.
# expose default port
EXPOSE 8091
ENV PATH $PATH:/opt/couchbase/bin
RUN cd /var/tmp &&\
wget http://packages.couchbase.com/releases/2.5.0/couchbase-server-enterprise_2.5.0_x86_64.rpm &&\
rpm -ivh couchbase-server-enterprise_2.5.0_x86_64.rpm &&\
chkconfig couchbase-server on &&\
service couchbase-server start
#add start script to the container
ADD start /start
#make the script executable
RUN chmod 0755 /start
EXPOSE 11211 11210 11209 4369 8092 18091 18092 11214 11215
#start mysql service and launch the command console
CMD ["/start"]
When building it, i am getting the following error ..
ulimit: open files: cannot modify limit: Operation not permitted
ulimit: max locked memory: cannot modify limit: Operation not permitted
I saw in a forum that we can set these values in docker.conf files. But i tried creating this file /etc/init/docker.conf and put the following lines in that file-
limit memlock unlimited unlimited
limit nofile 262144
but still i get the same error.
If i follow the same steps manually on CentOS VM, it works. So i guess i need to set something on Docker CentOS image.
Resolved the issue by setting the ulimit on the Docker host using the following command:
ulimit -l unlimited
ulimit -n 10240
ulimit -c unlimited
Then restart the Docker service on the CentOS. That fixed the issue since these values of the host will be inherited by the container.
Only root can increase the hard upper limit, and you have to be logged in as root, sudo won't work.
su root
ulimit -n 10240
I want to run PostgreSQL 9.1 using Supervisor on Ubuntu 10.04. At the moment, I manually start PostgreSQL using the init script:
/etc/init.d/postgresql start
According to this post: http://nicksergeant.com/using-postgresql-with-supervisor-on-ubuntu-1010/, I need to modify the PostgreSQL config to make it run on TCP port instead of Unix socket, in order to make PostgreSQL work with Supervisor.
I have two questions regarding this approach:
Considering this is more of hack, is there any implication (e.g. security/permissions, performance, etc) of doing this?
Why cannot we just run the same init script postgresql in Supervisor config? Instead, as shown in the link above, it runs postmaster?
UPDATE:
Thanks to the useful suggestions from both answers below, I have setup a script for Supervisor to invoke PostgreSQL directly:
#!/bin/sh
# This script is run by Supervisor to start PostgreSQL 9.1 in foreground mode
if [ -d /var/run/postgresql ]; then
chmod 2775 /var/run/postgresql
else
install -d -m 2775 -o postgres -g postgres /var/run/postgresql
fi
exec su postgres -c "/usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main -c config_file=/etc/postgresql/9.1/main/postgresql.conf"
I also set the config: /etc/postgresql/9.1/main/start.conf to manual so that PostgreSQL does not start automatically on boot (however, it's not clear to me whether this config is loaded). And then I setup the Supervisor config for postgres as:
[program:postgres]
user=root
group=root
command=/usr/local/bin/run_postgresql.sh
autostart=true
autorestart=true
stderr_logfile=/home/www-data/logs/postgres_err.log
stdout_logfile=/home/www-data/logs/postgres_out.log
redirect_stderr=true
stopsignal=QUIT
So now, I can start PostgreSQL in supervisorctl by doing start postgres, which runs fine. However, after I issue stop postgres, although supervisorctl declares postgres is stopped, the server apparently is still running as I can psql into it.
I wonder if this is a Supervisor config issue, or a PostgreSQL issue. Any suggestion welcome!
The blog post is rather badly written. There is no "TCP mode": the post's suggested method will still listen on a Unix socket, just in a different directory. Comments in the post such as "external pid file - not needed for TCP mode" are very misleading.
postmaster is the traditional name for the postgresql executable (to distinguish the master dispatching process from the backend slaves). For some time now there has been no separate executable, and now it is installed as simply "postgres".
Assuming that Supervisor is broadly simliar to the qmail/daemontools supervise scheme, it would be entirely possible (in fact, quite normal) to have it run a script that sets up the directories and environment, and then execs postgres with the requisite arguments (or propagates arguments given to the wrapper script, which would be unusual with supervise but makes more sense when you have a config file to put arguments into).
The way supervise worked (and I'm going to continue to assume "Supervisor" is the same) is to have the supervisor process run a subprocess as specified, and simply relaunch a new subprocess if it exits. This is based on the idea that the process being launched is a long-lived daemon process that only exits when something goes badly wrong, and that simply restarting it is a valid fix. By contrast, init scripts such as in /etc/init.d run the subprocess and detach it, and return control to their caller- if the subprocess exits, nothing special happens, and it must be restarted manually. If you tried to simply run /etc/init.d/postgresql start from supervise, it would continuously keep spawning postgresql daemons, as the return from the init script would be interpreted as the daemon process having exited, when in fact it had been started and detached.
To avoid auto-starting the service with the /etc/init.d scripts, the package for postgresql 9.1 provides a file /etc/postgresql/9.1/main/start.conf that contains:
# Automatic startup configuration
# auto: automatically start/stop the cluster in the init script
# manual: do not start/stop in init scripts, but allow manual startup with
# pg_ctlcluster
# disabled: do not allow manual startup with pg_ctlcluster (this can be easily
# circumvented and is only meant to be a small protection for
# accidents).
auto
This is the file to modify to avoid auto-start as opposed to moving away /etc/init.d/postgresql as the blog post suggests.
Also, changing unix sockets parameters for lack of /var/run/postgresql doesn't look like the best idea, because it's the default for any program linked with libpq, and because there's no difficulty in creating that directory with the proper permissions, just like it's done by the package's start sequence in /usr/share/postgresql-common/init.d-functions:
# create socket directory
if [ -d /var/run/postgresql ]; then
chmod 2775 /var/run/postgresql
else
install -d -m 2775 -o postgres -g postgres /var/run/postgresql
fi
And although the default value shouldn't cause a problem, note that whether postmaster ultimately stays in foreground or forks and runs in background is controlled by the silent_mode parameter in postgresql.conf. Make sure that it is off.
I am trying to make both tomcat and postgres run under supervisor, and found some hints here : https://serverfault.com/questions/425132/controlling-tomcat-with-supervisor
Here is my modified run_postgresql.sh, using bash :
#!/bin/bash
# This script is run by Supervisor to start PostgreSQL 9.1 in foreground mode
function shutdown()
{
echo "Shutting down PostgreSQL"
pkill postgres
}
if [ -d /var/run/postgresql ]; then
chmod 2775 /var/run/postgresql
else
install -d -m 2775 -o postgres -g postgres /var/run/postgresql
fi
# Allow any signal which would kill a process to stop PostgreSQL
trap shutdown HUP INT QUIT ABRT KILL ALRM TERM TSTP
exec sudo -u postgres /usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main --config-file=/etc/postgresql/9.1/main/postgresql.conf
With this script postgresql stops correctly after supervisorctl stop postgres.