While trying to run MongoDB on a mounted disk I found that its user (mongodb) has no permissions to the folder it's configured to (/media/dutzi/hdd/mongodb).
I then tried running:
sudo -H -u mongodb bash -c 'touch /media/dutzi/hdd/mongodb/asd'
And got:
touch: cannot touch '/media/dutzi/hdd/mongodb/asd': Permission denied
When trying with a different user (root, or my own) I was able to touch that file.
I tried (recursively) allowing all possible permissions, currently it looks like this:
❯ ll /media/dutzi/hdd/
total 20K
drwxrwxrwx 2 root root 16K Nov 7 17:25 lost+found/
drwsrwsrwx 4 mongodb mongodb 4.0K Nov 7 17:38 mongodb/
dutzi in dutzipc in ~
❯ ll /media/dutzi/hdd/mongodb/
total 204K
-rwxrwxrwx 1 mongodb mongodb 47 Nov 7 17:29 WiredTiger*
-rwxrwxrwx 1 mongodb mongodb 21 Nov 7 17:29 WiredTiger.lock*
-rwxrwxrwx 1 mongodb mongodb 1.3K Nov 7 17:29 WiredTiger.turtle*
-rwxrwxrwx 1 mongodb mongodb 44K Nov 7 17:29 WiredTiger.wt*
-rwxrwxrwx 1 mongodb mongodb 4.0K Nov 7 17:29 WiredTigerHS.wt*
-rwxrwxrwx 1 mongodb mongodb 20K Nov 7 17:29 _mdb_catalog.wt*
-rwxrwxrwx 1 mongodb mongodb 0 Nov 7 17:31 as2d*
-rwxrwxrwx 1 mongodb mongodb 0 Nov 7 17:38 asd*
-rwxrwxrwx 1 mongodb mongodb 20K Nov 7 17:29 collection-0--7241342955860792272.wt*
-rwxrwxrwx 1 mongodb mongodb 20K Nov 7 17:29 collection-2--7241342955860792272.wt*
-rwxrwxrwx 1 mongodb mongodb 4.0K Nov 7 17:29 collection-4--7241342955860792272.wt*
drwxrwxrwx 2 mongodb mongodb 4.0K Nov 7 17:29 diagnostic.data/
-rwxrwxrwx 1 mongodb mongodb 20K Nov 7 17:29 index-1--7241342955860792272.wt*
-rwxrwxrwx 1 mongodb mongodb 20K Nov 7 17:29 index-3--7241342955860792272.wt*
-rwxrwxrwx 1 mongodb mongodb 4.0K Nov 7 17:29 index-5--7241342955860792272.wt*
-rwxrwxrwx 1 mongodb mongodb 4.0K Nov 7 17:29 index-6--7241342955860792272.wt*
drwxrwxrwx 2 mongodb mongodb 4.0K Nov 7 17:29 journal/
-rwxrwxrwx 1 mongodb mongodb 0 Nov 7 17:29 mongod.lock*
-rwxrwxrwx 1 mongodb mongodb 20K Nov 7 17:29 sizeStorer.wt*
-rwxrwxrwx 1 mongodb mongodb 114 Nov 7 17:29 storage.bson*
dutzi in dutzipc in ~
❯ sudo -H -u mongodb bash -c 'touch /media/dutzi/hdd/mongodb/asdasd'
touch: cannot touch '/media/dutzi/hdd/mongodb/asdasd': Permission denied
Pretty sure one of these will work, but let me know in comments:
Check the permissions under the remaining directories
journal, diagnostics.data
Most of the DB seems empty, if that's the case, and previous idea doesn't resolve, then:
Wipe it out and start from scratch.
You don't need those files to be executable. I believe 444 is enough.
Related
I am having issues with postgresql 14 and 2 tablespaces:
db=# select * from stock_move;
ERROR: could not open file "pg_tblspc/32192/PG_14_202107181/16384/32197": No such file or directory
db=#
db=# select * from stock_picking;
ERROR: could not open file "pg_tblspc/32191/PG_14_202107181/16384/32194": No such file or directory
db=#
Everything was normal, but we decided to scale the server and when we restarted it came up with the error described above. There is also another tablespace, but it is working normally.
Tablespaces:
postgres#database-server:/mnt$ ls -l
total 12
drwx------ 4 postgres postgres 4096 Jan 9 05:33 stock_move
drwx------ 4 postgres postgres 4096 Jan 9 05:34 stock_move_line
drwx------ 4 postgres postgres 4096 Jan 9 05:34 stock_picking
postgres#database-server:/mnt$
These tablespaces are put in a partition for each one, each partition is mounted in /etc/fstab
/dev/nvme1n1 /mnt/stock_move ext4 defaults 0 1
/dev/nvme2n1 /mnt/stock_move_line ext4 defaults 0 1
/dev/nvme3n1 /mnt/stock_picking ext4 defaults 0 1
It seems that postgres tries to search for these files but they do not exist (32197, 32194).
root#database-server:/mnt/stock_picking/PG_14_202107181/16384# ll
total 1939392
drwx------ 2 postgres postgres 4096 Jan 13 01:06 ./
drwx------ 3 postgres postgres 4096 Jan 9 05:39 ../
-rw------- 1 postgres postgres 1073741824 Jan 11 17:26 32197
-rw------- 1 postgres postgres 911597568 Jan 12 00:01 32197.1
-rw------- 1 postgres postgres 507904 Jan 11 23:16 32197_fsm
-rw------- 1 postgres postgres 65536 Jan 11 17:17 32197_vm
-rw------- 1 postgres postgres 0 Jan 9 05:39 32198
-rw------- 1 postgres postgres 8192 Jan 9 05:39 32199
root#database-server:/mnt/stock_picking/PG_14_202107181/16384#
and
root#database-server:/mnt/stock_move/PG_14_202107181/16384# ll
total 1200276
drwx------ 2 postgres postgres 4096 Jan 13 01:10 ./
drwx------ 3 postgres postgres 4096 Jan 9 05:36 ../
-rw------- 1 postgres postgres 38191104 Jan 12 00:01 32194
-rw------- 1 postgres postgres 32768 Jan 11 19:26 32194_fsm
-rw------- 1 postgres postgres 8192 Jan 9 17:39 32194_vm
-rw------- 1 postgres postgres 0 Jan 9 05:36 32195
-rw------- 1 postgres postgres 8192 Jan 9 05:36 32196
-rw------- 1 postgres postgres 1073741824 Jan 11 16:55 32203
-rw------- 1 postgres postgres 116727808 Jan 12 00:01 32203.1
-rw------- 1 postgres postgres 311296 Jan 11 23:43 32203_fsm
-rw------- 1 postgres postgres 40960 Jan 11 23:47 32203_vm
-rw------- 1 postgres postgres 0 Jan 9 05:44 32204
-rw------- 1 postgres postgres 8192 Jan 9 05:44 32205
root#database-server:/mnt/stock_move/PG_14_202107181/16384#
and pg_tblspc links
root#database-server:/var/lib/postgresql/14/main/pg_tblspc# ll
total 8
drwx------ 2 postgres postgres 4096 Jan 9 05:34 ./
drwx------ 19 postgres postgres 4096 Jan 13 01:15 ../
lrwxrwxrwx 1 postgres postgres 18 Jan 9 05:33 32191 -> /mnt/stock_picking/
lrwxrwxrwx 1 postgres postgres 15 Jan 9 05:34 32192 -> /mnt/stock_move/
lrwxrwxrwx 1 postgres postgres 20 Jan 9 05:34 32193 -> /mnt/stock_move_line/
root#database-server:/var/lib/postgresql/14/main/pg_tblspc#
From here on I don't know what to do.
The only good explanation is that someone or something deleted the files that make up this table. This is the time to restore your backup.
Here is content of MongoDB /data/db folder content. Where one of the collections is 723GB. And other collections just few KB.
-rw------- 1 lxd docker 723G Dec 5 10:15 collection-0-1080408413244540209.wt
-rw------- 1 lxd docker 36K Dec 5 10:15 collection-0-3112968025499504303.wt
-rw------- 1 lxd docker 36K Dec 5 10:15 collection-2-1080408413244540209.wt
-rw------- 1 lxd docker 4.0K Dec 5 10:14 collection-4-1080408413244540209.wt
-rw------- 1 lxd docker 20K Dec 5 10:15 collection-7-1080408413244540209.wt
-rw------- 1 lxd docker 0 Dec 5 10:14 .dbshell
drwx------ 2 lxd docker 90 Dec 5 10:15 diagnostic.data
-rw------- 1 lxd docker 8.1G Dec 5 10:15 index-1-1080408413244540209.wt
-rw------- 1 lxd docker 20K Dec 5 10:14 index-1-3112968025499504303.wt
-rw------- 1 lxd docker 36K Dec 5 10:15 index-3-1080408413244540209.wt
-rw------- 1 lxd docker 4.0K Dec 5 10:14 index-5-1080408413244540209.wt
-rw------- 1 lxd docker 4.0K Dec 5 10:14 index-6-1080408413244540209.wt
-rw------- 1 lxd docker 20K Dec 5 10:14 index-8-1080408413244540209.wt
-rw------- 1 lxd docker 20K Dec 5 10:15 index-9-1080408413244540209.wt
drwx------ 2 lxd docker 110 Dec 5 10:15 journal
-rw------- 1 lxd docker 36K Dec 5 10:15 _mdb_catalog.wt
-rw------- 1 lxd docker 0 Dec 5 10:15 mongod.lock
-rw------- 1 lxd docker 36K Dec 5 10:15 sizeStorer.wt
-rw------- 1 lxd docker 114 Dec 5 10:14 storage.bson
-rw------- 1 lxd docker 50 Dec 5 10:14 WiredTiger
-rw------- 1 lxd docker 4.0K Dec 5 10:15 WiredTigerHS.wt
-rw------- 1 lxd docker 21 Dec 5 10:14 WiredTiger.lock
-rw------- 1 lxd docker 1.5K Dec 5 10:15 WiredTiger.turtle
-rw------- 1 lxd docker 84K Dec 5 10:15 WiredTiger.wt
I simply backup whole docker volume to another server via rsync.
I never change content of the 723GB collection, but for some reason MongoDB update the file approximately once in a week.
Because of that rsync also update that file remotely. And because I'm using snapshots, every week new snapshot add another 723GB to the storage, that is unacceptable and cause me the problems.
To resolve that problem, I simply added 723GB collection into rsync exception and do not upload it anymore. Is that fine? May I after 1 year still use my backup to restore the server if I do not update collection-0-1080408413244540209.wt file any more?
By default, rsync only copies new or changed files from a source to destination so you dont need to add the file to exception list if you copy the mounted snapshot files. From the other hand the wiredTiger is a bit sensitive and generate checksum in the data root folder based on checkpoints from all collections so in case the file differ there is a big chance the mongod process to not be able to start from the restored snapshot. So I would suggest to not exclude the file completely but leave to rsync to check every time if file is same or differ and need to be copied again or not.
P.S.
Note that the scenario you have described was valid with the deprecated previous mongoDB storage engine mmapv1 where it was even possible to copy the collection inside running instance on the fly , unfortunately the wiredTiger do not allow it , but offer other advantages.
I have a DOCKER container built from a few different images using a .yml, Dockerfile(s), etc. Everything builds and runs fine so far, except for this one issue that I am seeing mentioned in the title:
index-db_1 | 2021-02-22 23:18:33.388 UTC [31] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
That Database Index is mapped to a Folder on the host in the root of the Docker Package, and everything else seems to work fine as far as the database is concerned. I am using a Mac, but if I list permission from CLI for the DB folder I get:
-rw-------# 1 sscotti staff 3 Feb 22 11:01 PG_VERSION
drwx------# 6 sscotti staff 192 Feb 22 11:54 base
drwx------# 60 sscotti staff 1920 Feb 22 16:00 global
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_commit_ts
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_dynshmem
-rw-------# 1 sscotti staff 4782 Feb 22 11:02 pg_hba.conf
-rw-------# 1 sscotti staff 1636 Feb 22 11:01 pg_ident.conf
drwx------# 5 sscotti staff 160 Feb 22 17:46 pg_logical
drwx------# 4 sscotti staff 128 Feb 22 11:01 pg_multixact
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_notify
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_replslot
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_serial
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_snapshots
drwx------# 2 sscotti staff 64 Feb 22 16:00 pg_stat
drwx------# 5 sscotti staff 160 Feb 22 17:50 pg_stat_tmp
drwx------# 3 sscotti staff 96 Feb 22 11:01 pg_subtrans
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_tblspc
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_twophase
drwx------# 4 sscotti staff 128 Feb 22 11:01 pg_wal
drwx------# 3 sscotti staff 96 Feb 22 11:01 pg_xact
-rw-------# 1 sscotti staff 88 Feb 22 11:01 postgresql.auto.conf
-rw-------# 1 sscotti staff 28073 Feb 22 11:01 postgresql.conf
-rw-------# 1 sscotti staff 36 Feb 22 16:00 postmaster.opts
-rw------- 1 sscotti staff 94 Feb 22 16:00 postmaster.pid
pg_stat folder is actually empty.
and pg_stat_temp has:
-rw------- 1 sscotti staff 1952 Feb 22 17:54 db_0.stat
-rw------- 1 sscotti staff 20360 Feb 22 17:54 db_13395.stat
-rw------- 1 sscotti staff 1151 Feb 22 17:54 global.stat
The .yml file has this:
index-db:
image: postgres
restart: unless-stopped
volumes:
- ./OrthancIndex:/var/lib/postgresql/data
Is that something that can just be ignored given that it is a Docker container.
Adding a comment about the same setup on UBUNTU.
Database Folder:
drwx------ 19 systemd-coredump root 4096 Jun 30 13:12 OrthancIndex
Database:
drwx------ 6 systemd-coredump systemd-coredump 4096 Jun 11 13:00 base
drwx------ 2 systemd-coredump systemd-coredump 4096 Jun 30 13:12 global
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_commit_ts
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_dynshmem
-rw------- 1 systemd-coredump systemd-coredump 4782 Mar 12 16:12 pg_hba.conf
-rw------- 1 systemd-coredump systemd-coredump 1636 Mar 12 16:12 pg_ident.conf
drwx------ 4 systemd-coredump systemd-coredump 4096 Jul 1 13:27 pg_logical
drwx------ 4 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_multixact
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_notify
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_replslot
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_serial
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_snapshots
drwx------ 2 systemd-coredump systemd-coredump 4096 Jun 30 13:12 pg_stat
drwx------ 2 systemd-coredump systemd-coredump 4096 Jul 1 13:29 pg_stat_tmp
drwx------ 2 systemd-coredump systemd-coredump 4096 Jun 24 21:04 pg_subtrans
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_tblspc
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_twophase
-rw------- 1 systemd-coredump systemd-coredump 3 Mar 12 16:12 PG_VERSION
drwx------ 3 systemd-coredump systemd-coredump 4096 Jul 1 12:37 pg_wal
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_xact
-rw------- 1 systemd-coredump systemd-coredump 88 Mar 12 16:12 postgresql.auto.conf
-rw------- 1 systemd-coredump systemd-coredump 28073 Mar 12 16:12 postgresql.conf
-rw------- 1 systemd-coredump systemd-coredump 36 Jun 30 13:12 postmaster.opts
-rw------- 1 systemd-coredump systemd-coredump 94 Jun 30 13:12 postmaster.pid
pg_stat_temp
-rw------- 1 systemd-coredump systemd-coredump 2660 Jul 1 13:30 db_0.stat
-rw------- 1 systemd-coredump systemd-coredump 31157 Jul 1 13:30 db_13395.stat
-rw------- 1 systemd-coredump systemd-coredump 1151 Jul 1 13:30 global.stat
I actually get the same error on UBUNTU:
postgres_index-db_1 | 2021-07-01 18:06:45.140 UTC [266] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
postgres_index-db_1 | 2021-07-01 18:13:45.583 UTC [273] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
postgres_index-db2_1 | 2021-07-01 18:19:43.716 UTC [282] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
postgres_index-db2_1 | 2021-07-01 18:21:43.749 UTC [284] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
Although here the user and group are systemd-coredump.
Simple and dumb
I've tried another way: leaving the temporary file in container, where you need not deal with user permissions - and that worked. I've changed with inline command the Postgresql config in docker-compose.yml:
postgresdb:
image: 'postgres'
command: postgres -c stats_temp_directory=/tmp
.
.
.
This is quick and simple solution, but it does not set multiple postgres config parameters. Through the time I found another solution.
Advanced (added on 22/01/14 09:00PM CET)
The advanced solution consists in these steps:
Prepare your desired postgresql.conf - here you can set whichever postgres server parameters you need. The necessary one for solving problem in this thread is:
stats_temp_directory = '/tmp/stat_temporary'
Create bash file, that you will start using docker-entrypoint-initdb.d in docker-compose.yml with which you will copy your configuration to postgres config directory in docker and create your temporary folder.
If you setup the docker-compose.yml this way:
postgres-db:
image: postgres
container_name: 'dbcontainer'
volumes:
# point to your postgres init scripts (folder or file)
- $PWD/db_init:/docker-entrypoint-initdb.d
# if you need persistent data, point data to local folder
- $PWD/database:/var/lib/postgresql/data
# if you want some data restore, point to your backup
- $PWD/db_backup:/db_backup
# point docker to directory with your config
- $PWD/db_config:/db_config
Then database will init with scripts in db_init directory. (Or you can point directly to one sql or bash file. If you point to directory, init script will run all files in alphabetical order.
Your file with init bash script, that you will place in your db_init folder can look like this and will be started inside docker:
echo starting script...
# create temporary directory for postgres in docker
mkdir /tmp/stat_temporary
# copy your postgresql.conf to postgresql config location in docker
cp /db_config/postgresql.conf var/lib/postgresql/data/postgresql.conf
# alternatively you can run any pg script you need, here I restore backup to new database
pg_restore -U postgres -v -Fc -d analysis /db_backup/analysis_backup.bak
echo finished
After this script is run, the database service on docker is restarted and loads with new parameters you set up.
Writing this down as much for my future self as to answer the question: looking over at the documentation and taking #Fide's solution above, the following seems to work on MacOSX Monterey running a PostGIS image (which is just built on top of the Postgres images)...
Note that this is taken from a shell script to launch a Postgres+PostGIS image with a persistent local store on the host system:
WORK_DIR="$HOME/Documents/git/project"
DOCKER_NM="postgis"
DOCKER_IMG="postgis/postgis"
POSTGRES_PWD="XXX"
PORT_NO="5432"
docker run --name $DOCKER_NM \
-e POSTGRES_PASSWORD=$POSTGRES_PWD \
-e PGDATA=/var/lib/postgresql/data/pgdata \
-v "$WORK_DIR/data/postgres":/var/lib/postgresql/data \
-p "$PORT_NO:5432" -d $DOCKER_IMG \
postgres -c stats_temp_directory=/tmp
If you were running from the Terminal directly (i.e. not via zsh/bash) then you'd need to export each of the parameters instead of just specifying them.
With this the global.stat error seems to have gone away... though I should note that I've only just figured this out and have been running the container for less than 60 minutes. The alternative would be to follow the instructions on that page for creating and using a persistent custom conf file (extending #SScotti's comment above) as mounted from the local FS:
-v "$PWD/my-postgres.conf":/etc/postgresql/postgresql.conf
Mounting with a local volume requires user mapping within the postgres container. There is a section in the https://hub.docker.com/_/postgres README titled Arbitrary --user Notes which describes solutions to the problem.
One recommendation listed is to use a docker volume, initialize the contents of volume mounted, shutdown the container image, run a recursive ownership change and then restart the container normally.
$ docker volume create pgdata
$ docker run -it --rm -v pgdata:/var/lib/postgresql/data -e POSTGRES_PASSWORD=mysecretpassword postgres
The files belonging to this database system will be owned by user "postgres".
...
( once it's finished initializing successfully and is waiting for connections, stop it )
$ docker run -it --rm -v pgdata:/var/lib/postgresql/data bash chown -R 1000:1000 /var/lib/postgresql/data
$ docker run -it --rm --user 1000:1000 -v pgdata:/var/lib/postgresql/data postgres
Not an answer to the original question, but hence the lack of reputation points I can't add my two cents as a comment. So here is a working docker-compose.yml file using the tmpfs solution as an answer (for #SScotti and anyone else, who wants to try this without cluttering the default command or config):
Environment file (.env):
POSTGRES_VERSION=13.6
DATABASE_NAME=dbname
DATABASE_USER=dbuser
DATABASE_PASS=dbpass
Compose file (docker-compose.yml):
version: '3.6'
services:
postgres:
image: postgres:${POSTGRES_VERSION}-alpine
restart: always
container_name: postgres
environment:
POSTGRES_DB: ${DATABASE_NAME}
POSTGRES_USER: ${DATABASE_USER}
POSTGRES_PASSWORD: ${DATABASE_PASS}
volumes:
- /etc/timezone:/etc/timezone:ro
- .volumes/pgdata:/var/lib/postgresql/data
- type: tmpfs
target: /var/lib/postgresql/data/pg_stat_tmp
tmpfs:
# 256 Mb - beware the default, it's infinity(!)
size: 268435456
Edit: Note on first run
On first run, postgres tries to initialise the database and fails, because of non empty data directory (the existing/mounted tmpfs pg_stats_tmp in the data directory).
To avoid that, you have to run the service at first without the tmpfs section in the compose file, and when postgres finished initialising take the service down and enable the tmpfs section in the compose file before bringen up the service again.
I ran into this same problem - Looked into it in detail and finally found a simple solution.
First some basic background - your pg_stat_temp_directory location is a subdirectory under your data_directory.
show data_directory ;
If it returns the default value for Unix based systems like
/var/lib/postgresql/data
And your command to show stats_temp
show stats_temp_directory ;
Also returns the default value
pg_stat_tmp
Then your full path to stats_temp_directory is /var/lib/postgresql/data/pg_stat_tmp
I was using bind mapping to map the PG container data files to my host so that I can keep the data even after the container is stopped.
docker run -d --name gutlo-postgres -p 5434:5432 -v /data/postgres-docker/14:/var/lib/postgresql/data -e POSTGRES_PASSWORD=<PASSWORD> postgres:latest
Now everytime I start the container, it automatically creates files under stats_temp directory and once that it has a problem with is called global.stat
ls -la global.stat
-rw------- 1 ssengupta admin 3648 May 13 16:24 global.stat
ls -la . | head -2 | tail -1
drwxr-xr-x# 14 ssengupta admin 448 May 13 16:26 .
Now you see, where is the problem. The container is able to create the file global.stat ( its directory file "." has open rwx permissions. But the file itself is created without open permission hence my container cannot access its own created file.
From docker manual, there is a concept of temp mount. So simply create a temp mount point for the stats_temp directory and the docker will use tempfs / RAM - that eliminates permission problem.
--mount type=tmpfs,destination=/var/lib/postgresql/data/pg_stat_tmp
In addition it would improve performance too although you may not detect it.
My attempts...
Attach to the db container:
docker compose run --no-deps -it db bash
and do ls on the data directory:
root#dfc6399e2981:/var/lib/postgresql/data# ls -l
total 60
drwx------ 7 root root 224 Sep 8 23:38 base
drwx------ 60 root root 1920 Sep 8 23:38 global
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_commit_ts
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_dynshmem
-rw------- 1 postgres postgres 4821 Sep 8 23:35 pg_hba.conf
-rw------- 1 postgres postgres 1636 Sep 8 23:35 pg_ident.conf
drwx------ 5 root root 160 Sep 8 23:43 pg_logical
drwx------ 4 postgres postgres 128 Sep 8 23:35 pg_multixact
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_notify
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_replslot
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_serial
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_snapshots
drwx------ 2 root root 64 Sep 8 23:35 pg_stat
drwx------ 6 root root 192 Sep 9 00:20 pg_stat_tmp
drwx------ 3 postgres postgres 96 Sep 8 23:35 pg_subtrans
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_tblspc
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_twophase
-rw------- 1 postgres postgres 3 Sep 8 23:35 PG_VERSION
drwx------ 4 postgres postgres 128 Sep 8 23:35 pg_wal
drwx------ 3 postgres postgres 96 Sep 8 23:35 pg_xact
-rw------- 1 postgres postgres 88 Sep 8 23:35 postgresql.auto.conf
-rw------- 1 postgres postgres 28835 Sep 8 23:35 postgresql.conf
-rw------- 1 root root 36 Sep 8 23:35 postmaster.opts
-rw------- 1 postgres postgres 94 Sep 8 23:35 postmaster.pid
Obviously, note that some things are owned by root. Also, compare and contrast, from outside the container I see this:
% ls -l db/data
total 120
-rw-------# 1 pedz staff 3 Sep 8 18:35 PG_VERSION
drwx------# 7 pedz staff 224 Sep 8 18:38 base
drwx------# 60 pedz staff 1920 Sep 8 18:38 global
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_commit_ts
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_dynshmem
-rw-------# 1 pedz staff 4821 Sep 8 18:35 pg_hba.conf
-rw-------# 1 pedz staff 1636 Sep 8 18:35 pg_ident.conf
drwx------# 5 pedz staff 160 Sep 8 18:43 pg_logical
drwx------# 4 pedz staff 128 Sep 8 18:35 pg_multixact
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_notify
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_replslot
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_serial
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_snapshots
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_stat
drwx------# 6 pedz staff 192 Sep 8 19:27 pg_stat_tmp
drwx------# 3 pedz staff 96 Sep 8 18:35 pg_subtrans
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_tblspc
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_twophase
drwx------# 4 pedz staff 128 Sep 8 18:35 pg_wal
drwx------# 3 pedz staff 96 Sep 8 18:35 pg_xact
-rw-------# 1 pedz staff 88 Sep 8 18:35 postgresql.auto.conf
-rw-------# 1 pedz staff 28835 Sep 8 18:35 postgresql.conf
-rw-------# 1 pedz staff 36 Sep 8 18:35 postmaster.opts
-rw-------# 1 pedz staff 94 Sep 8 18:35 postmaster.pid
So, while in the db container I did:
chown -R postgres:postgres /var/lib/postgresql/data
I can now see updates into pg_stat_tmp from outside of the container and it has been up for 30 minutes with no Warning messages so I think its "fixed" although I don't know enough about Docker's images to know for sure that the issue won't come back later. Perhaps someone with more Docker knowledge can chime in.
Update: this does NOT fix the issue. Rather than delete this answer I think I will leave it here. It is odd that some files are owned by root. And even after I did chown -R there are still some files owned by root:
root#ec24992481d1:/var/lib/postgresql# find . -user 0 -exec ls -ld {} +
-rw------- 1 root root 8 Sep 9 00:34 ./data/pg_logical/replorigin_checkpoint
-rw------- 1 root root 2225 Sep 9 01:12 ./data/pg_stat_tmp/db_0.stat
-rw------- 1 root root 6665 Sep 9 01:12 ./data/pg_stat_tmp/db_13757.stat
-rw------- 1 root root 7035 Sep 9 01:12 ./data/pg_stat_tmp/db_16384.stat
-rw------- 1 root root 94 Sep 9 00:29 ./data/postmaster.pid
root#ec24992481d1:/var/lib/postgresql#
This might be a clue for others more savvy with Docker.
We are setting up a MongoDB server for the production environment on Amazon EC2 instance, but could not able to start the service. I've followed this documentation for setup. Here are the steps, I've taken for setting up the server:
Added following to /etc/yum.repos.d/mongodb-org-3.0.repo
[mongodb-org-3.0]
name=MongoDB Repository
baseurl=http://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/
gpgcheck=0
enabled=1
And installed MongoDB 3.0.2 using sudo yum install -y mongodb-org-3.0.2
Created three partitions for data, journal & log:
sudo mkdir /mongo
sudo mkdir /mongo/data
sudo mkdir /mongo/log
sudo mkdir /mongo/journal
Created file system for three separate partitions:
sudo mkfs.ext4 /dev/xvdb
sudo mkfs.ext4 /dev/xvdc
sudo mkfs.ext4 /dev/xvdd
Created entry in fstab for reboot:
echo '/dev/xvdb /mongo/data ext4 defaults,auto,noatime,noexec 0 0
/dev/xvdc /mongo/journal ext4 defaults,auto,noatime,noexec 0 0
/dev/xvdd /mongo/log ext4 defaults,auto,noatime,noexec 0 0' | sudo tee -a /etc/fstab
And mounted the partitions:
sudo mount /mongo/data
sudo mount /mongo/journal
sudo mount /mongo/log
Given the permissions and created link
sudo chown mongod:mongod /mongo/data /mongo/journal /mongo/log
sudo ln -s /mongo/journal /mongo/data/journal
Configured ulimit & read ahead settings as given in the documentation link above. Verified permissions and partitions:
[deployer#prod-mongo ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 8.0G 1.3G 6.8G 16% /
devtmpfs 3.6G 0 3.6G 0% /dev
tmpfs 3.5G 0 3.5G 0% /dev/shm
tmpfs 3.5G 57M 3.4G 2% /run
tmpfs 3.5G 0 3.5G 0% /sys/fs/cgroup
/dev/xvdc 7.8G 36M 7.3G 1% /mongo/journal
/dev/xvdb 150G 51M 149G 1% /mongo/data
/dev/xvdd 3.9G 16M 3.6G 1% /mongo/log
Permissions:
[deployer#prod-mongo ~]$ ll /
total 32
lrwxrwxrwx. 1 root root 7 Sep 29 2014 bin -> usr/bin
dr-xr-xr-x. 4 root root 4096 Sep 29 2014 boot
drwxr-xr-x. 17 root root 2860 May 11 12:11 dev
lrwxrwxrwx. 1 root root 7 Sep 29 2014 lib -> usr/lib
lrwxrwxrwx. 1 root root 9 Sep 29 2014 lib64 -> usr/lib64
drwxr-xr-x. 2 root root 6 Jun 10 2014 mnt
drwxr-xr-x. 5 mongod mongod 41 May 11 05:06 mongo
drwxr-xr-x. 21 root root 660 May 11 12:47 run
lrwxrwxrwx. 1 root root 8 Sep 29 2014 sbin -> usr/sbin
Inside /mongo
[deployer#prod-mongo ~]$ ll /mongo/
total 12
drwxr-xr-x. 3 mongod mongod 4096 May 11 07:33 data
drwxr-xr-x. 3 mongod mongod 4096 May 11 07:31 journal
drwxr-xr-x. 3 mongod mongod 4096 May 11 08:58 log
After changing the configurations inside /etc/mongodb.conf
logpath=/mongo/log/mongod.log
dbpath=/mongo/data
and when I'm doing: sudo service mongod start, I'm getting this error:
Starting mongod (via systemctl): Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
[FAILED]
Further logging:
[deployer#prod-mongo ~]$ sudo systemctl status mongod.service
mongod.service - SYSV: Mongo is a scalable, document-oriented database.
Loaded: loaded (/etc/rc.d/init.d/mongod)
Active: failed (Result: exit-code) since Tue 2015-05-12 04:42:10 UTC; 42s ago
Process: 22881 ExecStart=/etc/rc.d/init.d/mongod start (code=exited, status=1/FAILURE)
May 11 04:42:10 ip-xx-xx-xx-xx.local runuser[22887]: pam_unix(runuser:session): session opened for user mongod by (uid=0)
May 11 04:42:10 ip-xx-xx-xx-xx.localdomain runuser[22887]: pam_unix(runuser:session): session closed for user mongod
May 11 04:42:10 ip-xx-xx-xx-xx.local mongod[22881]: Starting mongod: [FAILED]
May 11 04:42:10 ip-xx-xx-xx-xx.local systemd[1]: mongod.service: control process exited, code=exited status=1
May 11 04:42:10 ip-xx-xx-xx-xx.local systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..
May 11 04:42:10 ip-xx-xx-xx-xx.local systemd[1]: Unit mongod.service entered failed state.
I've followed various articles and blog posts and StackExchange answers but didn't get any solution. Am I missing something?
Update: If I'm directly running the mongodb service from the normal user something like this: sudo mongod --logpath ~/mongod.log --dbpath ~/mongodata, then this service is starting properly.
We tried changing the path of the pid file to another directory, that didn't help either.
I'm guessing you're running a flavour of Linux that uses SELinux (RHEL or CentOS 7, perhaps?)
If so, the issue is that you don't have a permissive policy on your /mongo/ directory that permits access to daemons (like the mongod service.)
From Wikipedia:
SELinux can potentially control which activities a system allows each
user, process and daemon, with very precise specifications. However,
it is mostly used to confine daemons[citation needed] like database
engines or web servers that have more clearly defined data access and
activity rights. This limits potential harm from a confined daemon
that becomes compromised. Ordinary user-processes often run in the
unconfined domain, not restricted by SELinux but still restricted by
the classic Linux access rights
To check whether this is the issue, try this at the shell:
sudo setenforce 0
This should disable SELinux policies and allow the service to run.
For a more permanent solution, see https://wiki.centos.org/HowTos/SELinux
I ran into this problem and actually found a solution for me.
In short, mongodb 3.2 uses the user 'mongod' while older versions use 'mongodb'. Some of the files and directories were owned by 'mongodb' (the older user). Once I chmod'd them to the 'mongod' user, I was able to use systemctl to control the mongod process.
More specifically, it was the "/var/log/mongodb/*" files that had the wrong user ownership.
root#<HOST>:# ls -alh /var/log/mongodb
total 664K
drwxr-xr-x 2 mongod mongod 4.0K Oct 27 12:08 .
drwxr-xr-x. 22 root root 4.0K Oct 27 11:51 ..
-rw-r--r-- 1 mongodb mongodb 3.8K Oct 27 11:48 mongod.log
-rw-r--r-- 1 mongodb mongodb 19K Apr 14 2016 mongod.log.2016-04-14T18-29-34
-rw-r--r-- 1 mongodb mongodb 2.8K Apr 14 2016 mongod.log.2016-04-14T18-30-13
-rw-r--r-- 1 mongodb mongodb 12K Apr 14 2016 mongod.log.2016-04-14T22-27-27
-rw-r--r-- 1 mongodb mongodb 11K Apr 14 2016 mongod.log.2016-04-14T22-29-12
-rw-r--r-- 1 mongodb mongodb 5.6K Apr 18 2016 mongod.log-20160418.gz
-rw-r--r-- 1 mongodb mongodb 0 Apr 18 2016 mongod.log.2016-09-09T17-33-48
-rw-r--r-- 1 mongodb mongodb 3.6K Sep 9 11:34 mongod.log.2016-09-09T17-34-52
-rw-r--r-- 1 mongodb mongodb 23K Sep 9 11:49 mongod.log.2016-09-09T17-49-49
-rw-r--r-- 1 mongodb mongodb 5.0K Sep 9 11:55 mongod.log.2016-09-09T17-55-15
-rw-r--r-- 1 mongodb mongodb 5.0K Sep 9 12:02 mongod.log.2016-09-09T18-02-26
-rw-r--r-- 1 mongodb mongodb 5.0K Sep 9 12:13 mongod.log.2016-09-09T18-13-17
-rw-r--r-- 1 mongodb mongodb 5.0K Sep 9 12:25 mongod.log.2016-09-09T18-25-01
-rw-r--r-- 1 mongodb mongodb 5.2K Sep 9 12:47 mongod.log.2016-09-09T18-47-54
-rw-r--r-- 1 mongodb mongodb 5.0K Sep 9 12:52 mongod.log.2016-09-09T18-52-16
-rw-r--r-- 1 mongodb mongodb 5.0K Sep 9 12:54 mongod.log.2016-09-09T18-54-49
-rw-r--r-- 1 mongodb mongodb 5.0K Sep 9 13:01 mongod.log.2016-09-09T19-01-22
-rw-r--r-- 1 mongodb mongodb 3.0K Sep 9 13:03 mongod.log.2016-09-09T19-03-21
-rw-r--r-- 1 mongodb mongodb 215K Sep 9 14:25 mongod.log.2016-09-09T20-25-59
-rw-r--r-- 1 mongodb mongodb 281K Sep 10 03:42 mongod.log-20160910
-rw-r--r-- 1 mongodb mongodb 0 Sep 10 03:42 mongod.log.2016-10-27T17-42-42
-rw-r----- 1 mongod mongod 0 Sep 29 22:03 mongod.log.rpmnew
Notice the owner of the directory is 'mongod' (the new user) while the log files are all owned by 'mongodb' (the old user).
In case, anyone encountered the same issue with MongoDB startup, here is the thread of comments https://jira.mongodb.org/browse/SERVER-18439. This is scheduled to be fixed in 3.1.
I have a GridFS MongoDB database that I need to manage the size of. It has been running very well since it was created, but I have never really looked at its disk size until now.
Judging by this outout from the db.stats() command
> db.stats()
{
"db" : "documents",
"collections" : 4,
"objects" : 10967,
"avgObjSize" : 52491.573994711405,
"dataSize" : 575675092,
"storageSize" : 595255296,
"numExtents" : 24,
"indexes" : 4,
"indexSize" : 686784,
"fileSize" : 2080374784,
"nsSizeMB" : 16,
"ok" : 1
}
it seems the database itself is roughly 600MB. This size makes sense to me as it is the same size as the database backups I get from mongodump. The file size is far larger though, and it gets worse when I look in the data directory itself in /var/lib/mongodb:
root#deathstar:/var/lib/mongodb# ls -la
total 2474036
drwxr-xr-x 5 mongodb mongodb 4096 Apr 15 09:28 .
drwxr-xr-x 62 root root 4096 Mar 4 07:48 ..
drwxr-xr-x 2 mongodb mongodb 4096 Apr 13 11:48 documents
-rw------- 1 mongodb mongodb 67108864 Apr 15 09:16 documents.0
-rw------- 1 mongodb mongodb 134217728 Apr 13 11:48 documents.1
-rw------- 1 mongodb mongodb 268435456 Apr 13 11:48 documents.2
-rw------- 1 mongodb mongodb 536870912 Apr 15 09:16 documents.3
-rw------- 1 mongodb mongodb 1073741824 Apr 13 11:50 documents.4
-rw------- 1 mongodb mongodb 16777216 Apr 15 09:16 documents.ns
drwxr-xr-x 2 mongodb mongodb 4096 Apr 13 11:50 journal
-rwxr-xr-x 1 mongodb mongodb 5 Apr 13 11:46 mongod.lock
drwxr-xr-x 2 mongodb mongodb 4096 Apr 15 09:28 _tmp
-rw------- 1 mongodb mongodb 67108864 Apr 15 09:28 -v.0
-rw------- 1 mongodb mongodb 67108864 Apr 15 09:28 v.0
-rw------- 1 mongodb mongodb 134217728 Apr 15 09:28 -v.1
-rw------- 1 mongodb mongodb 134217728 Apr 15 09:28 v.1
-rw------- 1 mongodb mongodb 16777216 Apr 15 09:28 -v.ns
-rw------- 1 mongodb mongodb 16777216 Apr 15 09:28 v.ns
And this in /var/lib/mongodb/journal:
root#deathstar:/var/lib/mongodb/journal# ls -la
total 3145752
drwxr-xr-x 2 mongodb mongodb 4096 Apr 13 11:50 .
drwxr-xr-x 5 mongodb mongodb 4096 Apr 15 09:28 ..
-rw------- 1 mongodb mongodb 1073741824 Apr 15 09:28 j._2
-rw------- 1 mongodb mongodb 88 Apr 15 09:28 lsn
-rw------- 1 mongodb mongodb 1073741824 May 5 2012 prealloc.1
-rw------- 1 mongodb mongodb 1073741824 May 5 2012 prealloc.2
Now correct me if I'm wrong, but I am basically looking at 5.5GB disk size for a 600MB database. That is pretty inefficient.
How can I reduce the disk size? Is there a similar command to OPTIMIZE TABLE in MySQL?
I don't know whether GridFS is a different beast from a regular database, but I tried running compact but it didn't do anything to the disk size.
And how about the journal files? Can I somehow reduce the disk size of all journal files?
The issue with large files is not specific to GridFS.
Journal is there to provide durability, and MongoDB always preallocates files before it needs them. I would recommend not changing anything here - i.e. continue using journaling to protect your files in case of an unexpected crash of the server.
You see much smaller files with mongodump because you don't get the preallocated data files nor journal files.
If you want to have smaller DB directory, I recommend looking at --smallfiles and --noprealloc options to mongod. Both affect one when space is allocated and how much is allocated at a time.