Redis fails to start with error: redis-server.service: Failed at step NAMESPACE spawning /usr/bin/redis-server: Stale file handle - redis-server

After upgrading Debian, it has an issue starting redis-server.service.
In the output of journalctl -xe I see the following:
redis-server.service: Failed at step NAMESPACE spawning /usr/bin/redis-server: Stale file handle.
I can't start the redis-server.service and in the output of the systemctl start redis-server I have:
Job for redis-server.service failed because the control process exited with error code.
See "systemctl status redis-server.service" and "journalctl -xe" for details.
In the output of systemctl status redis-server I have:
● redis-server.service - Advanced key-value store
Loaded: loaded (/lib/systemd/system/redis-server.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2018-01-29 10:29:08 MSK; 58s ago
Docs: http://redis.io/documentation,
man:redis-server(1)
Process: 11701 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=226/NAMESPACE)
Process: 11720 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=226/NAMESPACE)
Main PID: 10193 (code=exited, status=0/SUCCESS)
Jan 29 10:29:08 xxx systemd[1]: redis-server.service: Service hold-off time over, scheduling restart.
Jan 29 10:29:08 xxx systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 5.
Jan 29 10:29:08 xxx systemd[1]: Stopped Advanced key-value store.
My question how to fix this issue and start redis-server.service?

Found a workaround solution:
I've played around with the /lib/systemd/system/redis-server.service editing service file as root, commenting out different fields trying to find where the failure happens and restarting systemd (via systemctl daemon-reload, systemctl stop redis-server, systemctl start redis-server)
For me the issue was the following line in the redis-server.service file:
ReadOnlyDirectories=/
which I have commented out and that allowed redis-server to start succesfully.
So my current /lib/systemd/system/redis-server.service is:
[Unit]
Description=Advanced key-value store
After=network.target
Documentation=http://redis.io/documentation, man:redis-server(1)
[Service]
Type=forking
ExecStart=/usr/bin/redis-server /etc/redis/redis.conf
ExecStop=/bin/kill -s TERM $MAINPID
PIDFile=/var/run/redis/redis-server.pid
TimeoutStopSec=0
Restart=always
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=2755
UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
#Modified 20180129 to avoid issue to start redis
#redis-server.service: Failed at step NAMESPACE spawning /usr/bin/redis-server: Stale file handle
#ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/redis
ReadWriteDirectories=-/var/log/redis
ReadWriteDirectories=-/var/run/redis
NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
MemoryDenyWriteExecute=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
# redis-server can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
ProtectSystem=true
ReadWriteDirectories=-/etc/redis
[Install]
WantedBy=multi-user.target
Alias=redis.service

I've just encountered this issue after running sudo apt -y dist-upgrade and got this slightly different error in /var/log/syslog
redis-server.service: Failed at step NAMESPACE spawning /usr/bin/redis-server: Invalid argument
The solution, Launchpad Bug 1638410, is:
sudo systemctl edit redis-server
[Service]
ProtectHome=no
Save and exit the editor and complete the upgrade:
sudo apt install -f
cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
$ apt policy redis-server
redis-server:
Installed: 5:5.0.0-3chl1~xenial1
Candidate: 5:5.0.0-3chl1~xenial1
Version table:
*** 5:5.0.0-3chl1~xenial1 500
500 http://ppa.launchpad.net/chris-lea/redis-server/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
2:3.0.6-1 500
500 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages

If you're using Ubuntu, in
/etc/redis/redis.conf
you should have :
supervised systemd

Related

Grafana-server Init Failed - Ubuntu 20.04

Using Grafana 8.3 in Ubuntu 20.04. When I went to the desktop environment to use Grafana, it wouldn’t connect. In the command line, when I type:
sudo netstat -lp
I noticed port 3000 wasn’t listed anymore.
starting the grafana server i get this below
a#a:~$ grafana-server
Grafana-server Init Failed: Could not find config defaults, make sure homepath command line parameter is set or working directory is homepath
when checking the status, I get this:
● grafana-server.service - Grafana instance
Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2022-01-05 16:06:11 MST; 11min ago
Docs: http://docs.grafana.org
Process: 4621 ExecStart=/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=deb cfg:default.paths.lo>
Main PID: 4621 (code=exited, status=1/FAILURE)
Jan 05 16:06:11 antonio- systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5.
Jan 05 16:06:11 antonio- systemd[1]: Stopped Grafana instance.
Jan 05 16:06:11 antonio- systemd[1]: grafana-server.service: Start request repeated too quickly.
Jan 05 16:06:11 antonio- systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Jan 05 16:06:11 antonio- systemd[1]: Failed to start Grafana instance.
thte only way I can get grafana to work in the desktop envionment is if I use and keep this open in the terminal:
$ sudo grafana-server -homepath /usr/share/grafana
Why has this happened and how can I use grafana without have to do this terminal step everytime?
thanks, I basically solved the problem by not solving the problem.
sudo apt-get purge grafana
sudo apt-get remove grafana
sudo apt-get remove --auto-remove grafana
then I deleted the config folder (contains the config file)
rm -R /etc/grafana/
then I reinstalled grafana
$ sudo apt-get install -y gnupg2 curl
$ curl https://packages.grafana.com/gpg.key | sudo apt-key add -
$ sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
$ sudo apt-get update
$ sudo apt-get -y install grafana
$ sudo systemctl start grafana-server
Now its running as before, with default port 3000 on startup

Using MongoDB on RaspberryPi via SSH connection in VS Code

I installed MongoDB via the official documentation on my Raspberry Pi, which I access using SSH remote connection in VSCode from my actual Desktop.
The installation looked all good, but I can't get the service to start using
sudo systemctl start mongod.
This is what I get from
sudo systemctl status mongod
mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: failed (Result: core-dump) since Wed 2021-08-04 12:42:10 UTC; 1s ago
Docs: https://docs.mongodb.org/manual
Process: 4442 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=dumped, signal=ILL)
Main PID: 4442 (code=dumped, signal=ILL)
Aug 04 12:42:10 ubuntu systemd[1]: Started MongoDB Database Server.
Aug 04 12:42:10 ubuntu systemd[1]: mongod.service: Main process exited, code=dumped, status=4/ILL
Aug 04 12:42:10 ubuntu systemd[1]: mongod.service: Failed with result 'core-dump'.
I tried changing all kinds of permissions and unfortunately no log files are created in the process. My last idea would be that I have to change the ipbind setting in order to make it work, but I already tried commenting it out in the config file, as well as replacing it with 0.0.0.0
At this point I don't know how to progress, so any help would be welcome!
Turns out the latest version (5.0) is not compatible with Raspberry Pi 4.
I got it to work by installing MongoDB 4.4.8 using the command
sudo apt-get install -y mongodb-org=4.4.8 mongodb-org-server=4.4.8 mongodb-org-shell=4.4.8 mongodb-org-mongos=4.4.8 mongodb-org-tools=4.4.8
I also then locked the versions using
echo "mongodb-org hold" | sudo dpkg --set-selections && echo "mongodb-org-server hold" | sudo dpkg --set-selections && echo "mongodb-org-shell hold" | sudo dpkg --set-selections && echo "mongodb-org-mongos hold" | sudo dpkg --set-selections && echo "mongodb-org-tools hold" | sudo dpkg --set-selections
to prevent apt-get from updating.
More information: https://www.mongodb.com/community/forums/t/core-dump-on-mongodb-5-0-on-rpi-4/115291/14

supervisord not starting automatically on Ubuntu 20.04

I intalled latest superviosrd
pip install supervisord==4.2.1
and then I copied supervisord init script to /etc/init.d/supervisord
scp /path/to/init_script.sh /etc/init.d/supervisord
sudo update-rc.d supervisord defaults
and then I run:
sudo service supervisord start && sudo service supervisord status
got:
● supervisord.service - LSB: Starts supervisord - see http://supervisord.org
Loaded: loaded (/etc/init.d/supervisord; generated)
Active: active (exited) since Sat 2020-08-29 13:03:57 UTC; 19min ago
Docs: man:systemd-sysv-generator(8)
Tasks: 0 (limit: 1074)
Memory: 0B
CGroup: /system.slice/supervisord.service
Aug 29 13:03:57 vagrant systemd[1]: Starting LSB: Starts supervisord - see http://supervisord.org...
Aug 29 13:03:57 vagrant systemd[1]: Started LSB: Starts supervisord - see http://supervisord.org.
and then I check
supervisorctl
got
unix:///tmp/supervisor.sock no such file
supervisor>
However, if I manually run
supervisord
it will start:
redis STARTING
supervisor>
Any ideas on how I can make it run automatically on start? and why sudo service supervisord start does not start it?
To answer my own question, it is because the init script I was using has the following:
NAME=supervisord
DAEMON=/usr/bin/$NAME
SUPERVISORCTL=/usr/bin/supervisorctl
however, supervisord is installed in /usr/local/bin/, so I changed the path, and everything works now.

Elasticsearch status is active and running but cant able to connect using curl

I get the following info using status command:
sudo /etc/init.d/elasticsearch status
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2017-08-04 19:19:13 IST; 49s ago
Docs: http://www.elastic.co
Process: 9974 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 9978 (java)
CGroup: /system.slice/elasticsearch.service
└─9978 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOcc...
Aug 04 19:19:13 joy systemd[1]: Starting Elasticsearch...
Aug 04 19:19:13 joy systemd[1]: Started Elasticsearch.
But curl GET request to it fails:
curl -XGET "localhost:9200"
curl -XGET 'localhost:9200/?pretty'
curl: (7) Failed to connect to localhost port 9200: Connection refused
You could try this if find, "No java runtime was found." in your log files.
Jul 30 18:28:13 dimik elasticsearch[10266]: [warning] /etc/init.d/elasticsearch: No java runtime was found
Here's my solution to the problem.
Check elasticsearch init file
sudo nano /etc/init.d/elasticsearch
search for
. /usr/share/java-wrappers/java-wrappers.sh
find_java_runtime openjdk8 oracle8 openjdk7 oracle7 openjdk6 sun6 default
export JAVA_HOME
Check java-wrappers.sh file
sudo nano /usr/share/java-wrappers/java-wrappers.sh
Now you could see the warning comes from
#Displays a warning
java_warning() {
echo "[warning] $0: $#" >&2;
}
Somehow, java directories are not listed in jvm-list.sh files
Now edit the jvm-list.sh file
sudo nano /usr/lib/java-wrappers/jvm-list.sh
Edit the line add your java directories files, in my case add /usr/lib/jvm/java-8-oracle*
__jvm_oracle8="/usr/lib/jvm/jdk-8-oracle-* /usr/lib/jvm/jre-8-oracle-* /usr/lib/jvm/java-8-oracle*"
Now restart the service and check elasticsearch services
sudo systemctl restart elasticsearch
sudo systemctl elasticsearch status
curl -X GET "http://localhost:9200"

Mongodb not able to start in Ubuntu 15.04

I have installed MongoDB 3.0.6 in Ubuntu 15.04 using the following commands.
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
echo "deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.0 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org
In the end in the installation process I got this:
Job for mongod.service failed. See "systemctl status mongod.service" and "journalctl -xe" for details.
invoke-rc.d: initscript mongod, action "start" failed.
dpkg: error processing package mongodb-org-server (--configure):
subprocess installed post-installation script returned error exit status 1
Setting up mongodb-org-mongos (3.0.6) ...
Setting up mongodb-org-tools (3.0.6) ...
dpkg: dependency problems prevent configuration of mongodb-org:
mongodb-org depends on mongodb-org-server; however:
Package mongodb-org-server is not configured yet.
dpkg: error processing package mongodb-org (--configure):
dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
Processing triggers for systemd (219-7ubuntu6) ...
Processing triggers for ureadahead (0.100.0-19) ...
Errors were encountered while processing:
mongodb-org-server
mongodb-org
E: Sub-process /usr/bin/dpkg returned an error code (1)
After that when I start the server using the below command I get an error and the server is not started.
sudo service mongod start
And the error is
Job for mongod.service failed. See "systemctl status mongod.service" and "journalctl -xe" for details.
Now when I run:
sudo systemctl status mongod.service
I get the following:
mongod.service - LSB: An object/document-oriented database
Loaded: loaded (/etc/init.d/mongod)
Active: failed (Result: exit-code) since Thu 2015-10-08 13:46:08 IST; 22s ago
Docs: man:systemd-sysv-generator(8)
Process: 6604 ExecStart=/etc/init.d/mongod start (code=exited, status=1/FAILURE)
Oct 08 13:46:07 gariya-GA-A55M systemd[1]: Starting LSB: An object/document-oriented database...
Oct 08 13:46:07 gariya-GA-A55M mongod[6604]: * Starting database mongod
Oct 08 13:46:08 gariya-GA-A55M mongod[6604]: ...fail!
Oct 08 13:46:08 gariya-GA-A55M systemd[1]: mongod.service: control process exited, code=exited status=1
Oct 08 13:46:08 gariya-GA-A55M systemd[1]: Failed to start LSB: An object/document-oriented database.
Oct 08 13:46:08 gariya-GA-A55M systemd[1]: Unit mongod.service entered failed state.
Oct 08 13:46:08 gariya-GA-A55M systemd[1]: mongod.service failed.
Someone please help, I am new to Ubuntu and MongoDB and I am not getting what is the problem.
There is a log file: /var/log/mongodb/mongod.log
In the log I've found the following:
Failed to unlink socket file /tmp/mongodb-27017.sock errno: Operation not permitted
Most probably this was from previous MongoDB version. I deleted the file (rm -rf /tmp/mongodb-27017.sock) and finished the installation:
apt-get install mongodb-org
Now it works. :-)
Recently i ended up with same issue and i tried all the possible solutions explained up here. But it didn't worked out for me. I was getting the below error.
# sudo service mongod restart
Restarting mongod (via systemctl): Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details. [FAILED]
Eventually i figured out the solution which worked for me.
I went to /run/mongodb folder and deleted the file called mongod.pid. Then tried to restart the mongo using # sudo service mongod restart and it's worked!
My log said:
[initandlisten] Found an invalid index { v: 2, key: { version: 1 }, name: "incompatible_with_version_32", ns: "admin.system.version"
so:
$cd /var/lib
$sudo rm -rf ./mongodb
$sudo mkdir mongodb
$sudo chown -R mongodb.mongodb mongodb/
$sudo service mongod restart
and then all worked fine.
sudo nano /etc/systemd/system/mongodb.service
[Unit]
Description=High-performance, schema-free document-oriented database
After=network.target
[Service]
User=mongodb
ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf
[Install]
WantedBy=multi-user.target
sudo systemctl start mongodb
sudo systemctl status mongodb
sudo systemctl enable mongodb
Is job!
The first thing you should check is a mongodb log file located at /var/log/mongodb/mongod.log
Look for the line with status E containing words like 'failed', 'fatal', etc.
Make sure that the /data/db directory exists
Make sure your user have write privileges in that directory:
$ sudo chown id -u /data/db
Execute mongod
If you still have problems and if the file /data/db/mongod.lock exists, remove it and execute mongod --repair.
I removed the mongodb lock file-
sudo /data/db/mongod.lock
Then restarted the mongodb
sudo service mongod restart
This solved the issue.
I was having the same issue, after searching I got the answer that, when I executed
sudo service mongod restart
I changed the permissions from user mongodb to root of the database folder, so the service was not able to read the data.
So to solve I just reverted the permissions back to the user mongodb
sudo chown -R mongodb:mongodb /path-to-db-folder
sudo service mongod restart