When I try to do a mongodump :
mongodump -u aaa -p abc123 --authenticationDatabase admin -d TestDb --gzip --out /var/backups/dump-25-05-22/mybackup.gz
inside my mongodb pod (kubectl exec -t <pod_name> --bash> I am getting an error :
Failed: error dumping metadata: error creating directory for metadata file /var/backups/...: permission denied
For context I do not have access to the host machine where the k8s instance is running, I can only do kubectl commands so the issue can only be fixed from K8S side.
These are the current permissions on the mount path (/var/backups) :
ls -la
drwxr-xr-x 2 root root 4096 Jun 11 2021 .
drwxr-xr-x 1 root root 4096 Jun 11 2021 ..
I have also tried the following :
1.
sudo chmod 777 -R /var/backups/
bash: sudo: command not found
chmod 777 -R /var/backups/
error changing permissions of '/var/backups/': Operation not permitted
Tried to enable sudo mode (Requires password and I dont know it)
su -
Having done a lot of digging around this the best solution I have is to tweak the mongo Dockerfile and build my own custom image.
The only feasible solution I found is at this link and it says "Configure mongo to run as root via editing the mongo config and copying it during the docker build process. This assumes you're using a docker file to build that image. Then it will have no problem accessing the attached volume."
My question(s):
Given an example Dockerfile as below how would I make the so-called mongod user run as root ? :
FROM mongo:3.2
COPY entrypoint.sh /root/entrypoint.sh
RUN chown -R mongodb:mongodb /var/log /data/db
USER mongodb
ENTRYPOINT ["/root/entrypoint.sh"]
At which level is this mongodb user accessible ? I have checked the users in the mongodb container but they is no entry for such (Unless Im missing something obvious) :
cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats
Related
I'm using the postgres docker image, and after months of using databases running in docker images, now I'm getting the behaviour where after a certain period of time, they simply just hang. I can exec with bin/bash but can't do anything with postgres at all; the commands don't return and the containers can't be brought down. Even docker kill -s SIGKILL <container_id> doesn't work; needs a reboot of the docker server to stop them.
The only smoking gun I can see is the message:
WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
on all containers. Anyone has any ideas I'd be really appreciative as this is killing things at the moment.
This is happening due to a user permission mismatch in the docker container.
Listing the relevant files in the container:
$ docker exec <container> ls -l /var/lib/postgresql/data/pg_stat_tmp
-rw------- 1 root root [...] db_0.stat
-rw------- 1 root root [...] db_1.stat
-rw------- 1 root root [...] db_2.stat
-rw------- 1 postgres postgres [...] global.stat
we can see that all the db_*.stat files are owned by root:root, while global.stat is owned by postgres:postgres.
Checking the docker user gives us:
$ docker exec <container> whoami
root
So, we'd like all of these files to be owned by the postgres user.
Luckily, this is quite easy! Just set user to postgres, and restart!
In a dockerfile:
USER postgres
Using docker-compose:
services:
postgres:
image: postgres:13
user: postgres
As Norling and Dharman's answer did not work for me, I've tried another way: leaving the temporary file in container - and that worked. I've changed with inline command the Postgresql config:
postgresdb:
image: 'postgres'
command: postgres -c stats_temp_directory=/tmp
.
.
.
I'm having problems lauching mongod as a service:
How is it possible that it works when I do sudo mongod -f /etc/mongod.conf but when launching it with sudo service mongod start I get an error in the log
Assertion: 28595:13: Permission denied src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 267
I'm running mongodb on ubuntu 16
I followed exactly the instructions in the mongodb documentation for installation of that version, so is this a bug? Any suggestions how to solve this are appreciated.
Additional information:
The mongodb service startup script looks like this and runs it as user mongodb, could this be connected to the error?
lib/systemd/system/mongodb.service:
[Unit]
Description=MongoDB Database Service
Wants=network.target
After=network.target
[Service]
ExecStart=/usr/bin/mongod --config /etc/mongod.conf
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
User=mongodb
Group=mongodb
StandardOutput=syslog
StandardError=syslog
[Install]
WantedBy=multi-user.target
I'm having problems lauching mongod as a service: How is it possible that it works when I do sudo mongod -f /etc/mongod.conf but when launching it with sudo service mongod start I get an error in the log
The sudo command starts mongod with root permissions (aka superuser access). If you run mongod as a service the user and group are configured in the service definition (mongodb for both in your example).
There is no need to run the mongod process as the root user, and this is strongly discouraged as per the common security practice of Principle of least privilege.
If you want to test a configuration from the command-line, you could use sudo to run with a specified user instead of the default (root) user.
For example:
sudo -u mongodb mongod -f /etc/mongod.conf
In general, it's best to use a service configuration rather than running mongod manually. With manual invocation you will also have to remember to include parameters like the config file path (as there is no default config path). Without a configuration file, mongod also uses default options such as a dbPath of /data/db.
Assertion: 28595:13: Permission denied src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 267
The likely cause of your permission errors is having previously started mongod as the root user. Some directories and files may now be owned by the root user, so the mongodb user cannot access those. Your specific error relates to accessing files in the data directory (i.e. the configured storage.dbPath in mongod.conf).
Assuming you haven't changed the default paths in your mongod.conf file, you should be able to recursively adjust permissions to match what the mongod.service definition expects.
First, ensure you have stopped your mongod instance if it is currently running.
Then, recursively adjust permissions to the expected user and group:
# storage.dbPath
sudo chown -R mongodb:mongodb /var/lib/mongodb
# systemLog.path
sudo chown -R mongodb:mongodb /var/log/mongodb
Now you should be able to start mongod as a service. If the service fails to start, there should be further detail in the mongod log file (assuming the log file is writable by the mongodb service user).
Have same problem.
What been in /var/log/mongodb/mongod.log:
2017-05-13T13:46:41.152+0700 E STORAGE [initandlisten] WiredTiger error (13) [1494658001:152518][15821:0x7fb843803cc0], connection: /var/lib/mongodb/journal/WiredTigerPreplog.0000000002: file-remove: unlink: Permission denied
2017-05-13T13:46:41.159+0700 I - [initandlisten] Assertion: 28595:13: Permission denied src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 267
So wee see that something can't remove file "WiredTigerPreplog.0000000002" in /var/lib/mongodb/journal/
So id just gave permissions, i just did:
sudo chmod 764 /var/lib/mongodb/journal/
If not help, try:
sudo chown -R mongodb:mongodb /var/lib/mongodb/ && sudo chmod 764 /var/lib/mongodb/journal/
There are three set-ups that triggers this kind of problem :
MongoDB installation is configured to create database files at a given path and this path does not exist on your current system. This path is called dbpath in mongo.
In your case, check if /data/db exist. If it doesn't or if it is empty, mongod is trying the wrong dbpath. You need to find it, it's usually under /var/lib/mongodb.
Once you found it there's two thing you can do. First, copy all the file from there to /data/db. Second, change your dbpath under the mongod.conf file, which is located (in linux) at /etc/mongod.conf. Make sure to start mongod with the --config specifying the configuration file.
MongoDB does not have the permission to read one or more files or directories corresponding to its dbpath.
chown mongodb:mongodb dbpath -R.
MongoDB is missing WiredTiger.wt . This can happen if you remove files under the dbpath or if there's a device failure. We do it for testing a recovery strategy for example.
If you're sure dbpath is correct and that there's no instance of WiredTiger.wt there. Your database is broken. There are no ways to ensure integrity if you lose this file. Reinstall mongodb by :
sudo apt-get purge mongodb-org*
sudo rm -r dbpath
sudo apt-get install mongodb-org
Edit :
Or copy dbpath from one of your replicas.
I had same problem in my mongod.log:
2021-09-16T16:06:43.782+0200 F STORAGE [initandlisten] Reason: 13: Permission denied
2021-09-16T16:06:43.782+0200 F - [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 789
So I looked in my dbPath folder (specified in mongodb config file /etc/mongod.conf, section storage:dbPath) and found that same file was owned by root:
WiredTiger.turtle
WiredTigerLAS.wt
journal/WiredTigerLog.0000000112
From dbPath folder, using the command below, I change owner and group to mongodb:
sudo chown mongodb:mongodb WiredTiger.turtle WiredTigerLAS.wt journal/WiredTigerLog.0000000112
After this, I can start mongodb server as
"service mongod start"
and check its status as
"service mongod status" with output:
mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-09-16 16:19:47 CEST; 54min ago
Docs: https://docs.mongodb.org/manual
Main PID: 8459 (mongod)
CGroup: /system.slice/mongod.service
└─8459 /usr/bin/mongod --config /etc/mongod.conf
Sep 16 16:19:47 svi5-ubu16 systemd[1]: Started MongoDB Database Server.
Note that above commands are working as normal user, not as root.
My configuration: Ubuntu 16.04.2 LTS, MongoDB shell version v4.2.1.
Other file or folder owned by mongodb (user and group):
/var/log/mongodb
/var/lib/mongodb
/tmp/mongodb-27028.sock
I spent a lot of time in solving this issue, thank you very match to every one has posted here and guided me to solution.
I wish append a comment to the previous answer, but unfortunately I cannot yet.
I completely agree with the explanation of Stennie. It is exactly what was happened to me.
I've always run mongod as a service but today, because some changes that I've made, I've tried to run the process using sudo mongod --auth --dbpath /data/mongodb/.. to test authorizations and db changing location.
After that the mongod service didn't run anymore, due to this permissions problem.
I've to say that the command sudo chown -R mongodb:mongodb /data/mongodb/ didn't immediately fix the problem as expected. I've had to reboot several times, remove the mongod.lock file under /data/mongodb/, reissue the sudo chown command again.. and finally everything gone well.
I have a similar issue but with custom log path and data dir. updating the owner and group access to them did not fix the issue for running as a service
updating the group and owner to mongodb:mongodb works on running mongod by itself as stated earlier
sudo -u mongodb mongod -f /etc/mongod.conf
or
sudo mongod -f /etc/mongod.conf
If you find yourself with the same problem on CentOS, but the permissions look correct, it could be because of SELinux policies. On my system, MongoDB files use a specific SELinux context. The mongod server failed to start, outputting the same permissions errors, until I corrected the SELinux contexts.
If it doesn't exist, create the MongoDB data folder (either /var/lib/mongo or /var/lib/mongodb, check your /etc/mongod.conf file):
$ mkdir -p /var/lib/mongodb
Then try to restore the SELinux contexts:
$ restorecon -v /var/lib/mongodb
$ restorecon -v /var/lib/mongodb/*
If that still doesn't work, try to apply the contexts directly:
$ chcon system_u:object_r:mongod_var_lib_t:s0 /var/lib/mongodb
$ chcon system_u:object_r:mongod_var_lib_t:s0 /var/lib/mongodb/*
Verify the contexts are correct:
$ ls -lZ -d /var/lib/mongodb
drwxr-xr-x. mongod mongod system_u:object_r:mongod_var_lib_t:s0 /var/lib/mongodb
sudo chmod -R 666 /var/lib/mongodb
is another solution...
After running docker run --name mongo -p 27017:27017 -v ~/Documents/store/mongo/:/data/db -d mongo --smallfiles
[initandlisten] exception in initAndListen: 98 Unable to create/open
lock file: /data/db/mongod.lock errno:13 Permission denied Is a mongod
instance already running?, terminating
In store directory, ls -l
drwxr-xr-x 4 MeoBeoI staff 136 Dec 8 17:11 mongo
drwxr-xr-x 2 MeoBeoI staff 68 Dec 9 10:20 redis
I use OSX 10.11.1
Looking at the mongo Dockerfile:
a user mongodb is created
/data/db is created and protected for mongodb:mongodb
So try and make sure to start your container with that user:
docker run --user mongodb ...
If that is not working, then fallback to the original docker run command (which runs as root by default) and define run.sh (that needs to be copied in your own image) which will:
do the right chown -R mongodb:mongodb /data/db
call entrypoint.sh mongod
That is:
docker run ... run.sh
That would be an approach similar to issue 7198 (Make uid & gid configurable for shared volumes).
I have postgres docker image, which can be deployed on Bluemix Containers. It works fine. But when I attached volume, container fails with permission error.
I am using $PGDATA as /var/lib/postgresql/data.
Entry point script, I have mentioned, sudo chown -R postgres /var/lib/postgresql/data. Also I have mounted volume using option -v data1:/var/lib/postgresql/data
But when I start container, chown always fails with 'Permission Error'.
I have added postgres user as part of root group.
But it still gives me same error.
chown: changing ownership of ?/var/lib/postgresql/data?: Permission denied
How do I fix this issue?
I found a way arround adding postgress to root group (which is a security flaw in my eyes).
At first you make the volume writable for everyone, then add an folder in the volume with the user you want to run your daemon with (in your case postgres). After this you can reset the volumes access right to default again.
I use this snippet in my entrypoint scripts on setup time:
chsh -s /bin/bash www-data
chmod 777 /var/www
su -c "mkdir -p /var/www/html" www-data
chmod 755 /var/www
usermod -s /bin/false www-data
Instead of chown volume directory to postgres user, change its permission to allow group write:
$ chmod g+w $PGDATA
Since you already added root group to user postgres it should work now.
Am running MongoDB 2.2 on Ubuntu and if I run:
sudo mongod
I get an error that it can't find /data/db, which is not where the database is. In mongod.conf the database path is specified as the Ubuntu 10gen default /var/lib/mongodb which is where the db is located. Seems like mongod is not finding the conf file. So when I run:
sudo mongod -f /etc/mongodb.conf
The server starts up fine and output is logged to the log file: /var/log/mongodb/mongodb.log. All is happy. I can switch to another shell, log into mongo shell, see the databases and run queries.
So, I cancel out of that and try to run as a service:
> sudo status mongodb
mongodb stop/waiting
> sudo start mongodb
mongodb start/running, process 10468
Looks good so far, but the mongo server did not start. Running another:
> sudo status mongodb
mongodb stop/waiting
> mongo
MongoDB shell version: 2.2.0
connecting to: test
Sat Sep 1 19:07:43 Error: couldn't connect to server 127.0.0.1:27017 src/mongo/shell/mongo.js:91
exception: connect failed
"test" is not the correct database, and nothing appears in the log file.
I am at a loss as to what could be wrong. I checked the upstart scripts and they seem fine. /etc/init/mongodb.conf runs:
mongodb --exec /usr/bin/mongod -- --config /etc/mongodb.conf
OK, this all comes down to permissions, but let's take it step by step. When you run sudo mongod it does not load a config file at all, it literally starts with the compiled in defaults - port 27017, database path of /data/db etc. - that is why you got the error about not being able to find that folder. The "Ubuntu default" is only used when you point it at the config file (if you start using the service command, this is done for you behind the scenes).
Next you ran it like this:
sudo mongod -f /etc/mongodb.conf
If there weren't problems before, then there will be now - you have run the process, with your normal config (pointing at your usual dbpath and log) as the root user. That means that there are going to now be a number of files in that normal MongoDB folder with the user:group of root:root.
This will cause errors when you try to start it as a normal service again, because the mongodb user (which the service will attempt to run as) will not have permission to access those root:root files, and most notably, it will probably not be able to write to the log file to give you any information.
Therefore, to run it as a normal service, we need to fix those permissions. First, make sure MongoDB is not currently running as root, then:
cd /var/log/mongodb
sudo chown -R mongodb:mongodb .
cd /var/lib/mongodb
sudo chown -R mongodb:mongodb .
That should fix it up (assuming the user:group is mongodb:mongodb), though it's probably best to verify with an ls -al or similar to be sure. Once this is done you should be able to get the service to start successfully again.
First confirm that the mongodb user/group has permission to write to both the data directory and log file:
$ sudo chown -R mongodb:mongodb /var/lib/mongodb/.
$ sudo chown -R mongodb:mongodb /var/log/mongodb.log
Start up MongoDB as a Daemon (background process) using the following command:
$ mongod --fork --dbpath /var/lib/mongodb/ --smallfiles --logpath
/var/log/mongodb.log --logappend
To Shut Down MongoDB enter the Mongo CLI, access the admin and issue the shutdown command:
$ ./mongo
> use admin
> db.shutdownServer()
Ref: http://www.mongodb.org/display/DOCS/Starting+and+Stopping+Mongo
I too had the same problem. So I went to cd /var/lib/mongodb/ and deleted the mongod.lock file
Then it worked for me.
After checking all permission in the data, journal and log folders as suggested by #nelsonic, my problem was solved by giving permission to lock file in the /tmp folder
sudo chown mongod:mongod mongodb-27017.sock
I was running it as a AWS Amazon Linux instance.
I figured that out by executing as the mongod user as below, and then, researching the error code. It might be useful for other troubleshooting.
sudo -S -u mongod mongod -f /etc/mongod.conf
Just try this command:
sudo chown mongodb /tmp/mongodb-27017.sock
Nothing worked for me, then I've found that it was a permissions problem on /tmp directory:
sudo chmod 1777 /tmp
sudo chown root:root /tmp
None of the above answers worked for me. I finally figured it out by debugging the init script with:
sudo bash -x /etc/init.d/mongodb start
And seeing it was passing the wrong config path to mongod. I simply changed the line in /etc/init.d/mongodb from "CONF=/etc/mongodb.conf" to "CONF=/etc/mongod.conf". Version 2 uses the former, and installing version 3 added /etc/mongod.conf with the new format but apparently did not update the init script.
UPDATE: I now have a much stranger problem where the init script works, but only if I run it with "sudo bash -x /etc/init.d/mongodb start" and not with "sudo service mongodb start". Same thing for stop.
My mongodb was starting when launched from the command line as the mongod user, but not as a service with User=mongod.
After an hour checking permissions, definition of the service, sockets... it was SElinux !
In /etc/selinux/config I switched from enforcing to permissive and reboot. It is now ok.
After none of the above answers worked for me, deleting my log file brought Mongo back to life.
These days this error can occur if you've updated mongod and you are running and old database. Mongod will be using the wiredTiger engine by default and you'll have a mmapv1 database
edit the engine setting in /etc/mongod.conf
# engine: wiredTiger
engine: mmapv1
Careful - YAML is whitespace sensitive
journalctl/systemd won't see this problem. Check the mongod log in /var/log/mongodb/mongod.log
I presume you can convert the database with something like the steps outlined here
https://docs.mongodb.com/manual/tutorial/change-standalone-wiredtiger/